Setting device, setting method and program

The described setting device and method use reinforcement learning to calculate weights and adjust parameter values for simulators, addressing inefficiencies in simulating systems with multiple outputs and improving accuracy, including in scenarios with sensor failures.

JP7586298B2Active Publication Date: 2024-11-19NEC CORP
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2023515941
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-04-20
Publication Date
2024-11-19
Estimated Expiration
2041-04-20

AI Technical Summary

Technical Problem

Existing methods struggle to effectively learn parameter values for simulators when multiple output values are obtained from a simulation target, particularly in systems with multiple sensors, leading to inefficiencies in simulation accuracy.

Method used

A setting device and method that utilizes reinforcement learning to set parameter values for simulators by calculating weights for each output item based on an evaluation function, using a learning control mechanism to adjust these values and improve simulation accuracy.

Benefits of technology

Enables effective learning of parameter settings for simulators, ensuring high accuracy in simulating complex systems with multiple output values, even in the presence of sensor failures or dynamic changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586298000034
    Figure 0007586298000034
  • Figure 0007586298000035
    Figure 0007586298000035
  • Figure 0007586298000036
    Figure 0007586298000036
Patent Text Reader

Abstract

This setting device is provided with a parameter value setting means for setting a parameter value of a simulator for simulating a simulation subject, a weight calculating means for calculating weights of respective output items on the basis of an output value of the simulator and a reference output value that is subject to simulation according to the output value of the simulator, and a learning control means for using an evaluation function based on the output value of the simulator, the reference output value, and the weights to control learning of the setting of the parameter value of the simulator done by the parameter value setting means.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a setting device, a setting method, and program Regarding. [Background technology]

[0002] It has been proposed to learn how to calculate the simulator's parameter values ​​using reinforcement learning. For example, Patent Document 1 describes that parameters (numerical values ​​that characterize a plant model of a process model) of a waste incineration plant simulator are learned by reinforcement learning (Q-Learning, TD learning method, etc.). Patent Document 1 also describes that rewards are determined according to the difference between actual measurements and calculated state quantities. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2004-178492 Summary of the Invention [Problem to be solved by the invention]

[0004] It is possible that output values ​​of multiple items can be obtained from the simulated object, such as when a plant has multiple sensors, etc. In this case, it is preferable to be able to effectively learn the settings of the simulator's parameter values.

[0005] One example of the object of the present invention is to provide a setting device, a setting method, and program The purpose of this project is to provide [Means for solving the problem]

[0006] According to a first aspect of the present invention, a setting device includes a parameter value setting means for setting parameter values ​​of a simulator that simulates a simulation target, and a setting device for setting a parameter value of the simulator based on an output value of the simulator and a reference output value that is to be simulated by the output value of the simulator. This is an item for each output value of the simulator. The system includes a weight calculation means for calculating a weight for each output item, and a learning control means for controlling learning of the setting of the parameter values ​​of the simulator by the parameter value setting means using an evaluation function based on the output value of the simulator, the reference output value, and the weight.

[0007] According to a second aspect of the present invention, a setting device includes a parameter value setting means for setting parameter values ​​of a first simulator that simulates a simulation object, and a parameter value setting means for setting parameter values ​​of a first simulator that simulates the simulation object. The output value is used as a reference output value to be simulated by the output value of the first simulator. and a learning control means for controlling learning of the setting of the parameter values ​​of the first simulator by the parameter value setting means, using an evaluation function based on an output value of a second simulator and an output value of the first simulator when an input value identical to an input value to the second simulator is input to the first simulator.

[0008] According to a third aspect of the present invention, in a setting method, a computer sets parameter values ​​of a simulator that simulates a simulation target, and based on an output value of the simulator and a reference output value that is to be simulated by the output value of the simulator, This is an item for each output value of the simulator. A weight is calculated for each output item, and learning of the parameter value settings of the simulator is controlled using an evaluation function based on the output value of the simulator, the reference output value, and the weight.

[0009] According to a fourth aspect of the present invention, a setting method includes a computer setting parameter values ​​of a first simulator that simulates a simulated object, and simulating the simulated object. The output value is used as a reference output value to be simulated by the output value of the first simulator. Learning of parameter value settings of the first simulator is controlled using an evaluation function based on the output value of a second simulator and the output value of the first simulator when the first simulator is input with an input value that is the same as the input value to the second simulator.

[0010] According to a fifth aspect of the present invention, a program includes the steps of: setting parameter values ​​of a simulator that simulates a simulation target; and calculating, based on an output value of the simulator and a reference output value that is to be simulated by the output value of the simulator, This is an item for each output value of the simulator. This is a program for executing the following: calculating a weight for each output item; and controlling learning of the parameter value settings of the simulator using an evaluation function based on the output value of the simulator, the reference output value, and the weight.

[0011] According to a sixth aspect of the present invention, a program includes a step of setting parameter values ​​of a first simulator that simulates a simulation object in a computer, and executing a program for simulating the simulation object. The output value is used as a reference output value to be simulated by the output value of the first simulator. and controlling learning of parameter value settings of the first simulator by using an evaluation function based on an output value of a second simulator and an output value of the first simulator when the same input value as the input value to the second simulator is input to the first simulator. Effect of the Invention

[0012] According to the present invention, it is expected that when output values ​​of a plurality of items can be obtained from a simulated object, learning of the setting of parameter values ​​of a simulator can be effectively performed. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating an example of a device configuration of a simulation system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a setting device during learning in the first embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of input and output of data during operation of the simulation system according to the first embodiment. [Figure 4] FIG. 11 is a diagram illustrating an example of the configuration of a simulation system according to a second embodiment. [Diagram 5]FIG. 11 is a diagram showing an example of input and output of data during learning in the simulation system according to the second embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of a configuration for calculating an evaluation value in a setting device according to the second embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of the gradient of an error in a sensor value according to the second embodiment. [Figure 8] FIG. 11 is a diagram illustrating an example of a normalized error calculated by a weight calculation unit according to the second embodiment. [Figure 9] 13 is a diagram showing an example of a degree of coincidence calculated by a weight calculation unit according to the second embodiment; FIG. [Figure 10] FIG. 11 is a diagram illustrating an example of weights calculated by a weight calculation unit according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing a first example of a distribution of weighted errors in the second embodiment. [Figure 12] FIG. 11 is a diagram showing a second example of the distribution of weighted errors in the second embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of updating weights by a learning control unit according to the second embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of a processing procedure in which the setting device according to the second embodiment updates weights of errors. [Figure 15] FIG. 13 is a diagram illustrating an example of the configuration of a simulation system according to a third embodiment. [Figure 16] FIG. 13 is a diagram showing an example of input and output of data during learning in the simulation system according to the third embodiment. [Figure 17] FIG. 13 is a diagram illustrating an example of a gradient vector in a space of parameter values ​​in the third embodiment. [Figure 18] FIG. 13 is a diagram illustrating an example of a processing procedure in which a gradient vector calculation unit according to the third embodiment estimates an error differential dLi(θ) / dθ. [Figure 19] FIG. 11 is a diagram illustrating an example of a processing procedure in which the setting device according to the third embodiment updates error weights. [Figure 20] FIG. 13 is a diagram illustrating an example of the configuration of a simulation system according to a fourth embodiment. [Figure 21] FIG. 13 is a diagram showing an example of input and output of data during learning in the simulation system according to the fourth embodiment. [Figure 22] FIG. 13 is a diagram illustrating an example of the configuration of a simulation system according to a fifth embodiment. [Figure 23] FIG. 13 is a diagram showing an example of input and output of data during learning in the simulation system according to the fifth embodiment. [Figure 24] FIG. 13 is a diagram illustrating an example of the configuration of a simulation system according to a sixth embodiment. [Diagram 25] FIG. 13 is a diagram showing an example of input and output of data during learning in the simulation system according to the sixth embodiment. [Figure 26] FIG. 23 is a diagram illustrating an example of the configuration of a setting device according to the seventh embodiment. [Figure 27] FIG. 23 is a diagram illustrating an example of the configuration of a setting device according to the eighth embodiment. [Figure 28] FIG. 13 is a diagram showing an example of a processing procedure in a setting method according to the ninth embodiment. [Figure 29] FIG. 23 is a diagram showing an example of a processing procedure in a setting method according to the tenth embodiment. [Diagram 30] FIG. 1 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0015] First Embodiment FIG. 1 is a diagram illustrating an example of a device configuration of a simulation system according to the first embodiment. 1, the simulation system 1 includes a setting device 100 and a simulator 200. Also shown in Fig. 1 is a simulation target 900. The simulation target 900 may be included in the simulation system 1, or may be an external configuration of the simulation system 1.

[0016] The simulated object 900 is a subject of simulation by the simulation system 1. The simulated object 900 can be various objects capable of executing a simulation, and is not limited to a specific object. The simulated object 900 may be a system including multiple devices, a single device, or a part of a device. For example, the simulated object 900 may be, but is not limited to, a chemical plant, a distillation column, an air traffic control system, or a railroad system.

[0017] The simulation system 1 simulates a simulation target 900 . Specifically, simulator 200 includes a simulation model of simulation target 900. The simulation model of simulation target 900 included in simulator 200 has parameters corresponding to the state of simulation target 900. Setting device 100 sets values ​​for the parameters of the simulation model, and simulator 200 simulates the behavior of simulation target 900 in the state of simulation target 900 corresponding to the set parameter values. Both the setting device 100 and the simulator 200 may be configured using a computer. Alternatively, the setting device 100 and the simulator 200 may be configured as a single device including both of them.

[0018] The simulation model of the simulation target 900 provided in the simulator 200 is also referred to simply as a simulation model, or as a simulation model of the simulation target 900. The parameters of the simulation model of the simulation target 900 included in the simulator 200 are also referred to simply as parameters or parameters of the simulator 200. The parameter values ​​of the simulation model of the simulation target 900 included in the simulator 200 are also referred to simply as parameter values ​​or parameter values ​​of the simulator 200.

[0019] The setting device 100 acquires a reference output value. The reference output value here is a value to be simulated by the output value of the simulator 200. The setting device 100 sets parameter values ​​so that the output value of the simulator 200 matches the reference output value. Specifically, the setting device 100 sets the parameter values ​​so that the output value of the simulator 200 becomes as close as possible to the reference output value.

[0020] For example, a value that is the same as an actual input value to the simulation target 900 may be input to the simulator 200. Then, the setting device 100 may obtain an actual output value of the simulation target 900 for this input value as a reference output value, and set the parameter value so that the output value of the simulator 200 approaches the reference output value as much as possible. This allows the simulator 200 to simulate the simulation target 900 with high accuracy according to the state of the simulation target 900. Also, by referring to the parameter values ​​set by the setting device 100, the state of the simulation target 900 can be analyzed.

[0021] When the simulation system 1 acquires actual input values ​​and output values ​​of the simulation target 900, the input values ​​and output values ​​of the simulation target 900 may be acquired in real time. In this case, the simulator 200 can simulate the simulation target 900 in real time. Alternatively, the simulation system 1 may acquire historical data of input values ​​and output values ​​of the simulation target 900. In this case, the simulator 200 can reproduce past operations of the simulation target 900 by simulation. In this case, the setting device 100 and the simulation target 900 do not need to be communicatively connected.

[0022] An input value to the simulation target 900, or a value equivalent to an input value to the simulation target 900, is also referred to as a control input value. A value equivalent to an input value to the simulation target 900 may be a value that can be used as an input value to the simulation target 900, but is not actually intended to be input to the simulation target 900.

[0023] The simulation system 1 may simulate the simulated target 900 based on virtual data of a control input value and a reference output value instead of actual data of the simulated target 900. For example, the simulation system 1 may simulate the simulated target 900 based on planned values ​​of input and output of the simulated target 900. In this way, by referring to the parameter values ​​set by the setting device 100, it is possible to analyze the state of the simulated target 900 suitable for implementing the plan. In this case, the setting device 100 and the simulation target 900 do not need to be communicatively connected.

[0024] The output value of the simulation target 900 is not limited to a specific one as long as it is a value that can be acquired by the simulation system 1. In the following, a means for acquiring the output value of the simulation target 900 is treated as a sensor. The output value of the simulation target 900 is also referred to as a sensor value. The output value of the simulated target 900 may include a sensor value of the simulated target 900 itself, such as the pressure in a tank provided in the simulated target 900. The output value of the simulated target 900 may also include a sensor value related to the surrounding environment of the simulated target 900, such as the room temperature of the room in which the simulated target 900 is installed.

[0025] When the simulated target 900 has multiple outputs, each output of the simulated target 900 is also referred to as an output item of the simulated target 900, or simply as an output item. The output value of the simulated target 900 may be represented as a vector having an element for each output item. When the output values ​​of all output items of the simulated target 900 are to be distinguished from the output values ​​for each output item, the output values ​​of all output items are also referred to as an output value vector. The terms "output item" and "output value vector" are used for the simulator 200 that simulates the simulated object 900, just as in the case of the simulated object 900. The terms "output item" and "output value vector" are also used for the reference output values. For example, the reference output values ​​of all output items are also referred to as the "reference output value vector."

[0026] The input values ​​to the simulated target 900 may include values ​​input as signals to the simulated target 900, such as a command value for the flow rate of liquid in a pipe provided in the simulated target 900. The input values ​​to the simulated target 900 may also include values ​​input by operations on the simulated target 900, such as the opening of a control valve operated by an operator of the simulated target 900.

[0027] When there are multiple inputs to the simulated target 900, each input to the simulated target 900 is also referred to as an input item to the simulated target 900. The input value to the simulated target 900 may be represented as a vector having an element for each input item. When the input values ​​of all input items of the simulated target 900 are to be distinguished from the output values ​​for each input item, the input values ​​of all input items are also referred to as an input value vector.

[0028] The setting device 100 may update the parameter values ​​during the simulation in response to changes in the state of the simulation target 900 over time. Furthermore, the setting device 100 may set parameter values ​​so that the simulation model exhibits the same behavior as the simulated target 900 or a behavior as similar as possible to that of the simulated target 900 even if some of the sensors of the simulated target 900 fail.

[0029] The simulation model of the simulated object 900 may have multiple parameters, and the values ​​of these multiple parameters may be represented by a vector. When all parameter values ​​of the simulation model of the simulated object 900 are to be distinguished from individual parameter values, all parameter values ​​are also referred to as a parameter value vector.

[0030] 2 is a diagram showing an example of the configuration of the setting device 100 during learning. In the configuration shown in FIG. The setting device 100 acquires a calculation method for the simulation model parameter values ​​by reinforcement learning.

[0031] Reinforcement learning is a type of machine learning. In reinforcement learning, a "policy," which is a criterion for determining an "action" by an "agent" in an "environment" observing a "state," is the subject of updating through learning. When updating the policy, a "reward" indicating an evaluation of the effect of the action on the environment is presented to the agent. In addition to the policy, the method of calculating the reward may also be subject to updating through learning. Also, a so-called "loss," in which the smaller the value, the higher the evaluation, may be presented to the agent as a reward. Learning is also called training.

[0032] In the simulation system 1, the setting device 100 corresponds to an example of an agent. The simulator 200 and an output source of the control input value and the reference output value, such as the simulation target 900, correspond to an example of an environment. The control input value, the reference output value, and the output value of the simulator 200 correspond to an example of a state observed by the setting device 100.

[0033] The parameter value setting section 120 sets parameter values ​​of the simulator 200. Specifically, the parameter value setting section 120 has a function for calculating parameter values, and calculates the parameter values ​​using this function. The parameter value setting section 120 transmits the calculated parameter values ​​to the simulator 200, thereby setting the parameter values ​​in the simulator 200. The parameter value setting unit 120 corresponds to an example of a parameter value setting means.

[0034] The criteria by which the parameter value setting unit 120 determines the parameter values ​​may be configured in a format other than a function. For example, the criteria by which the parameter value setting unit 120 determines the parameter values ​​may be shown in the format of an algorithm. The parameter value set by the parameter value setting unit 120 corresponds to an example of an action, and the criterion by which the parameter value setting unit 120 determines the parameter value corresponds to an example of a measure.

[0035] The criterion for the parameter value setting unit 120 to determine the parameter value is configured to be updatable. For example, when the criterion for the parameter value setting unit 120 to determine the parameter value is configured in the form of a function, the coefficient value of the function and the value of a constant term may be updatable. When the parameter value setting unit 120 is configured in the form of an algorithm, an external variable whose value can be updated by learning may be provided in the algorithm.

[0036] In the following, an example will be described in which the criterion for parameter value setting unit 120 to determine parameter values ​​is configured in the form of a function. This function is also called a policy function. The policy function may be represented by "π". The arguments of the policy function π may be, but are not limited to, the control input value, the reference output value, and the output value of the simulator 200. The policy function π may also take data for a plurality of time periods as arguments. For example, the policy function π may take the control input value, the reference output value, and the history information of the simulator 200 as arguments.

[0037] The learning control unit 110 uses an evaluation function based on the output value and the reference output value of the simulator 200 to control learning of the parameter value setting unit 120. The learning control unit 110 corresponds to an example of a learning control means. Specifically, the learning control unit 110 has an evaluation function for calculating an evaluation value for the parameter value set by the parameter value setting unit 120, using the output value of the simulator 200 and the reference output value as arguments. The learning control unit 110 calculates the evaluation value using this evaluation function. For example, the learning control unit 110 may use an evaluation function that calculates the magnitude of the error of the output value of the simulator 200 with respect to the reference output value as an evaluation value.

[0038] The criteria by which the learning control unit 110 obtains an evaluation value may be configured in a format other than a function. For example, the criteria by which the learning control unit 110 obtains an evaluation value may be indicated in the format of an algorithm. The evaluation value acquired by the learning control unit 110 corresponds to an example of a reward. The learning control unit 110 may acquire, as the reward, a so-called loss, in which the smaller the value, the higher the evaluation, like the error described above. In the following, an example will be described in which the learning control unit 110 calculates an evaluation value using an evaluation function. This evaluation function is also called a reward function.

[0039] Based on the calculated evaluation value, the learning control unit 110 controls the learning of parameter value setting by the parameter value setting unit 120. Specifically, the learning of parameter value setting by the parameter value setting unit 120 here is the updating of the policy function π.

[0040] The learning control unit 110 updates the policy function π so that the evaluation value decreases. As a result, the learning control unit 110 updates the policy function π so that the output value of the simulator 200 approaches the reference output value. Instead of the learning control unit 110 directly updating the policy function π, the learning control unit 110 may cause the parameter value setting unit 120 to update the policy function π.

[0041] Here, a known method can be used as a method for updating the policy function π. For example, if the gradient of the policy function π can be calculated by some method, such as differentiation or difference approximation of the policy function π, the learning control unit 110 may use a gradient method to determine the update content of the policy function π.

[0042] FIG. 3 is a diagram showing an example of data input and output during operation of the simulation system 1. As shown in FIG. In the example of Fig. 3, the parameter value setting unit 120 is shown among the units of the setting device 100. Among the units of the setting device 100 shown in Fig. 2, the learning control unit 110 is not necessary during operation. During operation, the setting device 100 does not need to include the learning control unit 110. Alternatively, even during operation, the setting device 100 may be configured as in the example of Fig. 2, and the learning control unit 110 may be in a stopped state.

[0043] 3, the parameter value setting unit 120 acquires a control input value, a reference output value, and an output value of the simulator 200. The parameter value setting unit 120 inputs each acquired value to a learned policy function π to calculate parameter values, and sets the calculated parameter values ​​in the simulator 200.

[0044] The simulator 200 executes a simulation of the simulation target 900 based on the set parameter values, and calculates an output value for a control input value. In the following, time is represented by a consecutive number for each time step, such as time 0, 1, 2, .... The parameter value setting unit 120 calculates and sets the parameter value for each time step. Then, the simulator 200 calculates an output value for a control input value every time a parameter value is set. The input and output of the simulator 200 are expressed as in equation (1).

[0045]

number

[0046] x (t) denotes an input value vector to the simulator 200 at time t. (t) denotes the parameter value vector of the simulator 200 at time t. f is a function that indicates the behavior of the simulation model. In the function f, the state of the simulation model changes depending on the parameter value θ (t) is shown in the argument. The simulation model uses the parameter value θ (t)In this case, the function f may have a state other than the parameter value θ (t) Alternatively, the state variable value may be taken as an argument. y^ (t+1) indicates the output value vector of the simulator 200 at time t+1. Simulator 200, x (t) and θ (t) Based on this, the output value vector y^ of the simulator 200 at time t is (t) may be calculated.

[0047] The start time of the simulation is 0, and the current time in the simulation is τ. The time series data of the control input value vector from time 0 to τ is expressed as shown in equation (2).

[0048]

number

[0049] The time series data of the output value of the simulation target 900 from time 0 to τ is expressed as in equation (3).

[0050]

number

[0051] The time series data of the output value of the simulator 200 from time 0 to τ is expressed as in equation (4).

[0052]

number

[0053] The setting device 100 may use all or a part of the time series data of the control input value, the time series data of the output value of the simulation target 900, and the time series data of the output value of the simulator 200 as state observation data in learning the setting of parameter values. Instead of the time series data of the output value of the simulation target 900, the setting device 100 may use virtual data such as time series data of the output plan value of the simulation target 900 as state observation data.

[0054] <Second embodiment> FIG. 4 is a diagram illustrating an example of the configuration of a simulation system according to the second embodiment. 4, the simulation system 2 includes a setting device 101 and a simulator 200. The setting device 101 includes a learning control unit 111, a parameter value setting unit 120, and a weight calculation unit 310. 4, parts having similar functions to those of the parts in FIG. 2 are given the same reference numerals (120, 200), and detailed description thereof will be omitted here.

[0055] The simulation system 2 corresponds to an example of the simulation system 1. The setting device 101 corresponds to an example of the setting device 100. The learning control unit 111 corresponds to an example of the learning control unit 110. Therefore, the learning control unit 111 corresponds to an example of the learning control means. The weight calculation unit 310 calculates a weight for the error for each output item of the output value of the simulator 200 with respect to the output value of the simulation target 900. The learning control unit 111 calculates an error obtained by weighting and summing the errors for each output item as an evaluation value using an evaluation function based on the output value of the simulator 200 and the reference output value as well as the weights calculated by the weight calculation unit 310. In other respects, the simulation system 2 is similar to the simulation system 1.

[0056] Fig. 5 is a diagram showing an example of input and output of data during learning of the simulation system 2. In the example of Fig. 5, the weight calculation unit 310 receives input of the reference output value and the output value of the simulator 200, and calculates a weight for each output item.

[0057] The learning control unit 111 receives as input the reference output value, the output value of the simulator 200, and the weight calculated by the weight calculation unit 310. The learning control unit 111 calculates the error of the output value of the simulator 200 relative to the reference output value for each output item, and weights and sums the errors for each output item using the weight calculated by the weight calculation unit 310. The learning control unit 111 uses the weighted sum of the errors as an evaluation value, and controls the learning of the parameter value setting by the parameter value setting unit 120 so as to reduce the evaluation value.

[0058] The parameter value setting unit 120 receives the control input value, the reference output value, and the output value of the simulator 200. The parameter value setting unit 120 inputs each of the obtained values ​​to a policy function π to calculate parameter values, and sets the calculated parameter values ​​in the simulator 200. The simulator 200 executes a simulation of the simulation target 900 based on the set parameter values, and calculates an output value for a control input value.

[0059] FIG. 6 is a diagram showing an example of a configuration for calculating an evaluation value in the setting device 101. As shown in FIG. In the example of FIG. 6, the state of the simulated object 900 is “θ t The parameter value of the simulator 200 corresponding to the state of the simulated object 900 is written as "θ^ t " is written. S t indicates the output of the sensor of the simulated object 900 and the simulator 200 that simulates the sensor of the simulated object 900. S t A white circle (◯) indicates that the sensor or output shown in is normal, and a black circle (●) indicates that it is faulty. Fig. 6 shows an example in which one of the sensors of the simulated object 900 is faulty.

[0060] The learning control unit 111 calculates an evaluation value as a weighted sum of squares of differences for each output item between the output value of the simulator 200 and the output value of the simulation target 900, and updates the policy function π to minimize the evaluation value. The square of the difference for each output item between the output value of the simulator 200 and the output value of the simulation target 900 is also called a squared error. 6 indicates the difference between the output of the simulation target 900 and the output of the simulator 200 for each output item. 2 " indicates a square. "w" indicates weighting by the weight calculated by weight calculation section 310. "Σ" indicates a sum.

[0061] As in the example of Figure 6, if some of the sensors of the simulated object 900 are faulty, the learning control unit 111 may control the learning so that the output value of the simulator 200 matches the output value of the faulty sensor, which may result in a decrease in the accuracy of the simulation by the simulator 200. Furthermore, if some of the sensors of the simulated object 900 are out of order, it may be impossible to make the error calculated as the evaluation value 0. If the error cannot be made 0, it may take a long time for learning.

[0062] Therefore, the weight calculation unit 310 sets a weight for each output item. This weight is used as a weight indicating the degree to which the error of the output value of the simulator 200 with respect to the output value of the simulation target 900 is reflected in the evaluation value. When the weight of all the output items is to be distinguished from the weight for each output item, the weight of all the output items is also referred to as a weight vector. The weight calculation unit 310 corresponds to an example of a weight calculation means. The weight calculation unit 310 may set the weight for each output item in the range of 0 to 1. In this case, the weight vector w is expressed as in equation (5).

[0063]

number

[0064] P is an integer indicating the number of output items. In formula (5), "[0,1]" indicates the range from 0 to 1. The parameter value setting section 120 calculates the parameter value based on, for example, equation (6).

[0065]

number

[0066] θ represents time-series data of the parameter value vector. If the start time of the simulation is 0 and the current time in the simulation is τ, then θ in equation (6) is expressed as in equation (7).

[0067]

number

[0068] The parameter value setting unit 120 selects the parameter value vector θ (τ-1) Let θ be the parameter value vector. (τ) may be set in the simulator 200 as

[0069] In equation (6), y i (t+1) is the output value vector y of the simulated object 900 at time t+1. (t+1) indicates the i-th element of f i (x (t) ,θ (t) ) is the function f(x (t) ,θ (t) ), the output value y^ of the simulator 200 i (t+1) This shows the part where y^ is calculated. i (t+1) is the output value vector y^ of the simulator 200 at time t+1. (t+1) indicates the i-th element of w i denotes the i-th element of the weight vector. argmin θ(t) is the parameter value vector θ that minimizes the value of the function given as an argument. (t) This is a function that outputs

[0070] f i (x (t) ,θ (t) )=y^ i (t+1) This can be expressed as "w i (y i (t+1) -f i (x (t) ,θ (t) )) 2 " indicates the weighted sum of squared errors for each output item of the output value of the simulator 200 with respect to the output value of the simulation target 900. Equation (6) indicates that the time series data of the weight vector is calculated so that the sum of the weighted squared errors for the output items and weights is minimized.

[0071] In addition, in equation (6), all weights w i If w is set to 0, the argument of argmin will always be 0, and it will not be possible to use it as an evaluation function in reinforcement learning. Therefore, the weight calculation unit 310 sets all the weights w i The sum of all the weights w i The weight w i Calculate.

[0072] Here, as an example of a case where some of the sensors of the simulated object 900 are broken, consider a case where the simulated object 900 has two sensors and one of the two sensors is broken. The output value of the simulated object 900 is y = [y1, y2] = [5.2, 1.2]. It is assumed that the first sensor, i.e. the sensor with output value y1 = 5.2, is broken.

[0073] The output value of the simulator 200 is y^=[y^1, y^2], where y^1=1.2 regardless of the parameter values. y^2 is assumed to take a value in the range of 0.0 or more and 10.0 or less. The error of the first sensor is a fixed value of y1-y^1=5.2-1.2=4.0, and the error function value changes depending on the value of y^2.

[0074] The parameter value vector θ set by the parameter value setting section 120 is expressed as in equation (8).

[0075]

number

[0076] Error ε of the i-th sensor i is expressed as equation (9).

[0077]

number

[0078] FIG. 7 is a diagram showing an example of the gradient of the error in the sensor value. The horizontal axis of the graph in FIG. 7 represents the output value y^2 of the simulator 200. The vertical axis represents the error Σ i (w i (y i -f i (x,θ) 2 ) is shown. Line L1 shows the relationship between the output value y^2 and the error when the weight vector is w=[0.5,0.5]. Line L2 shows the relationship between the output value y^2 and the error when the weight vector is w=[0.05,0.95].

[0079] In the example of Figure 7, the error ε1 of the first sensor has a fixed value of 9. Therefore, the weighted error w1ε1 has a fixed value, and the magnitude of the error gradient is determined by the magnitude of the weighted error w2ε2 of the second sensor. Between lines L1 and L2, line L2, which has a larger value of weight w2, has a larger error gradient. For example, in Figure 7, the change in weighted error for the same change in y^2 is larger for D2 than for D1. With a larger error gradient, it is expected that the learning time for learning using gradient methods, for example, will be relatively short.

[0080] Therefore, the weight calculation unit 310 calculates the weight based on the normality probability of the sensor value. The normality probability here is the degree to which the sensor value is expected to be accurate. The higher the normality probability, the higher the possibility that the sensor is accurate. The accuracy of the sensor here may mean that the sensor value matches the actual value to such an extent that the error between the sensor value and the actual value can be negligible.

[0081] Specifically, the weight calculation unit 310 normalizes the difference between the output value of the simulation target 900 for each output item and the output value of the simulator 200 so that it is possible to compare them. Then, the weight calculation unit 310 determines the magnitude of the weight so that the larger the magnitude of the normalized difference is, the smaller the weight of the output item is.

[0082] As learning progresses and the magnitude of the difference between the output value of the simulation target 900 and the output value of the simulator 200 becomes smaller, it is conceivable that the difference in output items with a low probability of being normal, such as the output item of a faulty sensor or the output item of a sensor with significantly lower accuracy than other sensors, will become relatively large. By having the weight calculation unit 310 determine the magnitude of the weight so that the magnitude of the weight is smaller for output items with a larger magnitude of normalized difference, it is expected that the weight of output items with a low probability of being normal will be reduced, thereby reducing the impact on learning.

[0083] The weight calculation unit 310 may set the initial value of the weight to the same value for all output items. For example, the weight calculation unit 310 may initialize the weight vector w as shown in equation (10).

[0084]

number

[0085] After learning has progressed to a certain extent, the learning control unit 111 may weight the error. For example, the weight calculation unit 310 may keep the value of each weight at the initial value shown in equation (10) until learning has been performed a predetermined number of times.

[0086] FIG. 8 is a diagram showing an example of the normalized error calculated by the weight calculation section 310. As shown in FIG. The weight calculation unit 310 converts the output value of the simulation target 900 for each output item and the output value of the simulator 200 for each output item, for example, so that the range of each sensor is aligned to the range from 0 to 1 ([0,1]). However, the normalization performed by weight calculation section 310 is not limited to a specific method as long as it makes the errors comparable.

[0087] The weight calculation unit 310 calculates, for each output item, the squared error of the output value of the normalized simulator 200 with respect to the output value of the normalized simulation target 900. The squared error of the output value of the normalized simulator 200 with respect to the output value of the normalized simulation target 900 is also referred to as a normalized squared error or a comparison error. Figure 8 shows an example where the number of output items is three. The comparison errors for each output item are 0.7, 0.2, and 0.1.

[0088] FIG. 9 is a diagram showing an example of the degree of coincidence calculated by weight calculation section 310. As shown in FIG. The degree of match used is a value that increases as the comparison error decreases. In the example of FIG. 9, the weight calculation unit 310 calculates the degree of match by subtracting the comparison error from 1 for each output item. The degrees of match for each output item are 0.3, 0.8, and 0.9. The degree of match for each output item is R i i is an identification number for identifying the output item, and can be a consecutive number starting from 1, for example.

[0089] However, the method by which weight calculation section 310 calculates the degree of coincidence is not limited to the above method. For example, the weight calculation unit 310 calculates the degree of match r for each training sample and each output item using, for example, equation (11). i may be calculated.

[0090]

number

[0091] The training sample referred to here may be any one that serves as a unit of learning for setting parameter values ​​of the simulator 200. For example, the training sample may be configured to include a combination of time series data of one control input value vector and time series data of one reference output value vector. The training data may be configured to include a plurality of training samples. When learning is repeatedly performed using one training data, one learning using one training data may be referred to as one epoch of learning. In equation (11), ε i denotes the normalized squared error. The weight calculation unit 310 divides the normalized squared error for each output item by the sum of the normalized squared errors using equation (11) to calculate the individual matching scores r i is a value in the range of 0 or more and 1 or less, and the degree of match r i The total value for all output items will be 1. The weight calculation unit 310 calculates the degree of agreement R for each output item by averaging the degree of agreement for each training sample and each output item for a predetermined number of training samples using, for example, equation (12). i may be calculated.

[0092]

number

[0093] D is a constant indicating a predetermined number. ε k (d) denotes the normalized squared error of the kth output item and the dth training sample.i (d) / Σ k=1 P ε k (d) " is the degree of agreement for each training sample and each output item, r i , the average over D training samples.

[0094] Individual agreement score R i is a value in the range of 0 or more and 1 or less, and the degree of agreement R i The total value for all output items will be 1. In formula (12), D may be 1. Therefore, the weight calculation unit 310 calculates the degree of match r i The degree of agreement for each output item is R i It may be calculated as follows.

[0095] FIG. 10 is a diagram showing an example of weights calculated by weight calculation section 310. As shown in FIG. The weight calculation unit 310 may multiply the degree of coincidence for each output item obtained in the example of FIG. For example, the weight calculation unit 310 calculates the weight w for each output item using the formula (13). i Calculate.

[0096]

number

[0097] The weight calculation unit 310 calculates the weight for each output item using equation (13), and the individual weights w i is a value in the range 0 or more and 1 or less, and the weight w i The total value for all output items will be 1.

[0098] Alternatively, the weight calculation unit 310 calculates the weight w for each output item using equation (14) instead of equation (13). i may be calculated.

[0099]

number

[0100] e denotes Napier's constant. The weight calculation unit 310 calculates the weight for each output item using equation (14), thereby obtaining the individual matching scores w i is a value in the range 0 or more and 1 or less, and the weight w i The total value for all output items will be 1.

[0101] In addition, the weight calculation unit 310 calculates the degree of agreement R by calculating the power of a base larger than 1, such as Napier's constant, as in Equation (14). i The weights w are used to emphasize the relative differences in the size of i Specifically, when the weight calculation unit 310 uses the formula (14), the degree of agreement R i The larger the weight w i will be calculated to be larger.

[0102] Alternatively, the weight calculation unit 310 may use the degree of coincidence as the weight as it is. For example, the weight calculation unit 310 may use the degree of coincidence R i , the degree of agreement r calculated using Eq. (11) i , or the degree of agreement R calculated using Eq. (12) i The weight w i It may be used as:

[0103] The learning control unit 111 may calculate an error index value indicating the magnitude of the weighted error between the output value of the simulator 200 and the reference output value for all output items for each unit of learning, such as for each epoch. If the error index value after one unit of learning indicates a larger error than the error index value before that unit of learning, the learning control unit 111 may return the weights to the weights before that unit of learning.

[0104] FIG. 11 is a diagram showing a first example of the distribution of weighted errors. 11 indicates the class in the frequency distribution of the magnitude of the weighted error. The vertical axis indicates the number of sensors for each class of the magnitude of the weighted error. As the weighted error, a value obtained by multiplying the square error for each output item of the output value of simulator 200 with respect to the output value of simulation target 900 by a coefficient for each output item calculated by weight calculation unit 310 can be used. In the example of FIG. 11, the frequency of classes on the side of the horizontal axis of the graph where the error is relatively large (the right side of FIG. 11) is relatively large.

[0105] FIG. 12 is a diagram showing a second example of the distribution of weighted errors. The horizontal axis of the graph in Fig. 12 indicates the ranks in the frequency distribution of the magnitude of the weighted error, and the vertical axis indicates the number of sensors for each rank of the magnitude of the weighted error. Compared to the example in Figure 11, the example in Figure 12 has a relatively large frequency of classes on the side of the horizontal axis of the graph where the error is relatively small (the left side of Figure 12). It can be evaluated that the example in Figure 12 has an improved weighted error compared to the example in Figure 11.

[0106] Various index values ​​indicating the degree of improvement of the weighted error can be used as the error index value calculated by the learning control unit 111. For example, the learning control unit 111 may calculate, as the error index value, the mode, the average, or the median of the squared error of the output value of the simulator 200 with respect to the output value of the simulation target 900 for all output items, but is not limited to these.

[0107] Fig. 13 is a diagram showing an example of updating the weights by the learning control unit 111. In the example of Fig. 12, the learning control unit 111 learns the setting of parameter values ​​for each time step, calculates an error index value, and updates the weights. In the example of FIG. 13, the learning control unit 111 uses the difference obtained by subtracting the previous evaluation value from the current evaluation value as the error index value. t " indicates the evaluation value at time t. "E t-E t-1 ” denotes the error index value at time t. "w (t) " indicates the weight calculated by weight calculation section 310 at time t. As described above, the evaluation value here can be an error obtained by weighting and summing the errors for each output item.

[0108] As described above, the learning control unit 111 can use various index values ​​as the error index value. For example, the learning control unit 111 may calculate, as the error index value, the mode, average, or median of the squared error of the output value of the simulator 200 with respect to the output value of the simulation target 900 for all output items. Alternatively, the learning control unit 111 may calculate, as the error index value, "|E t -E t-1 |-|E t-1 -E t-2 A higher-order difference in the evaluation value, such as "|", may be used.

[0109] At time t-1, the error index value E t-1 -E t-2 ≧0, it can be evaluated that the weighted sum error has been improved. Therefore, the learning control unit 111 calculates the weight w (t-1) We have adopted the following. On the other hand, at time t, the error index value E t -E t-1 <0, and it can be evaluated that the weighted sum error has not been improved. Therefore, at time t, the learning control unit 111 controls the weight calculation unit 310 to use the weight w calculated by the weight calculation unit 310 at time t-2. (t-2) It is returning to.

[0110] If the error index value indicates that the weighted error has not improved, it is possible that the error has not been weighted appropriately. By the learning control unit 111 restoring the weights as described above, it is possible to cancel weights that may be inappropriate and restore the original weights. If the error index value indicates that the weighted error has not been improved, the learning control unit 111 may cancel not only the weights but also the learning itself, and return the parameter values, etc. to the values ​​before learning at that time.

[0111] FIG. 14 is a diagram showing an example of a processing procedure in which the setting device 101 updates the weight of the error. 14, the learning control unit 111 normalizes the reference output value for each output item and the output value of the simulator 200 for each output item, and sets weights (step S111). The output value of the simulator 200 is also referred to as the actual value of the output value by simulation. As mentioned above, the initial value of the weight for each output item can be set to the same value for all output items, for example 1 / P.

[0112] Next, the learning control unit 111 performs learning for one epoch using the set weights (step S112). However, the amount of learning in step S112 is not limited to one repetition of learning using one training data, and can be various amounts that are units of learning.

[0113] Next, the weight calculation unit 310 calculates the normalized error of the output value of the simulator 200 with respect to the reference output value for each output item (step S113). As described above, the weight calculation unit 310 may calculate the normalized square error as the normalized error.

[0114] Next, the weight calculation unit 310 calculates the degree of agreement of the output value of the simulator 200 with respect to the reference output value for each output item (step S114). As described above, the weight calculation unit 310 may calculate the degree of agreement by subtracting the normalized square error for each output item from 1. Alternatively, the weight calculation unit 310 may calculate the degree of agreement based on equation (11). Alternatively, the weight calculation unit 310 may calculate the degree of agreement based on equations (11) and (12).

[0115] Next, the learning control unit 111 updates the weight calculation unit 310 (step S115). Specifically, the learning control unit 111 updates the weight to be applied to the error to the weight calculated by the weight calculation unit 310. As described above, the weight calculation unit 310 may calculate the weight for each output item using formula (13) or formula (14). Alternatively, the weight calculation unit 310 may use the degree of coincidence calculated for each output item as the weight for each output item as it is.

[0116] As described above, the learning control unit 111 may calculate the error index value. If the error index value indicates that the error is larger than the error index value before one unit of learning is executed, the learning control unit 111 may return the weight to the weight before the execution of that unit of learning.

[0117] Next, the learning control unit 111 determines whether or not a learning end condition has been met (step S116). The learning end condition here is not limited to a specific condition. For example, the learning control unit 111 may determine that the learning end condition has been met when the learning has been repeated a predetermined number of times. Alternatively, the learning control unit 111 may determine that the learning end condition has been met when the evaluation value calculated by the learning control unit 111 indicates that the parameter value is more accurate than a predetermined condition.

[0118] In step S116, if the learning control unit 111 determines that the learning end condition is not satisfied (step S116: NO), the process returns to step S112. On the other hand, if the learning control unit 111 determines in step S116 that the learning end condition is met (step S116: YES), the setting device 101 ends the process of FIG.

[0119] As described above, the parameter value setting unit 120 sets the parameter values ​​of the simulator 200 that simulates the simulation target. The weight calculation unit 310 calculates the weight for each output item based on the output value of the simulator 200 and the reference output value that is the target of simulation by the output value of the simulator 200. The learning control unit 111 controls learning of the setting of the parameter values ​​of the simulator by the parameter value setting means, using an evaluation function based on the output value of the simulator 200, the reference output value, and the weight.

[0120] According to the setting device 101, it is possible to reduce the influence on the evaluation value by setting a small weight for an output item whose reference output value has a low probability of being normal due to a sensor failure, etc. In this respect, according to the setting device 101, when output values ​​for multiple items can be obtained from the simulation target, it is expected that the setting of the parameter values ​​of the simulator can be effectively learned.

[0121] Furthermore, the weight calculation section 310 calculates a weight for each output item so that the weight for an output item having a larger normalized error after normalizing the error for each output item is smaller. According to the setting device 101, when the upper and lower limit values ​​of the sensor range, etc. differ for each output item, the upper and lower limit values ​​can be aligned by normalization. In this respect, according to the setting device 101, it is possible to more appropriately compare the magnitude of error for each output item and more appropriately set the weights, and it is expected that learning of the setting of the simulator parameter values ​​can be performed effectively.

[0122] Furthermore, the learning control unit 111 calculates an error index value for each unit of learning. The error index value is an index value that indicates the magnitude of the weighted error between the output value of the simulator 200 and the reference output value for all output items. If the error index value after one unit of learning is executed indicates that the error is larger than the error index value before that unit of learning is executed, the learning control unit 111 restores the weight to the weight before that unit of learning is executed.

[0123] According to the setting device 101, when the error index value indicates that the weight setting may be inappropriate, the weight setting can be restored to the original setting. In this respect, according to the setting device 101, it is expected that the weight setting can be set with higher accuracy and the setting of the parameter values ​​of the simulator can be effectively learned.

[0124] <Third embodiment> FIG. 15 is a diagram illustrating an example of the configuration of a simulation system according to the third embodiment. As shown in FIG. 15, the simulation system 3 includes a setting device 102 and a simulator 200. The setting device 102 includes a learning control unit 112, a parameter value setting unit 120, a weight calculation unit 311, and a gradient vector calculation unit 320. 15, parts having similar functions to those in FIG. 2 are given the same reference numerals (120, 200), and detailed description thereof will be omitted here.

[0125] The simulation system 3 corresponds to an example of the simulation system 1. The setting device 102 corresponds to an example of the setting device 100. The learning control unit 112 corresponds to an example of the learning control unit 110. Therefore, the learning control unit 112 corresponds to an example of the learning control means.

[0126] The weight calculation unit 311 calculates a weight for the error for each output item of the output value of the simulator 200 with respect to the output value of the simulation target 900. The learning control unit 112 calculates an error obtained by weighting and summing the errors for each output item as an evaluation value using an evaluation function based on the output value of the simulator 200 and the reference output value as well as the weights calculated by the weight calculation unit 311. In other respects, the simulation system 3 is similar to the simulation system 1.

[0127] The gradient vector calculation unit 320 calculates, for each output item, a gradient vector indicating the gradient of the error between the output value of the simulator 200 and the reference output value in the space of parameter values. The gradient vector calculation unit 320 corresponds to an example of a gradient vector calculation means. Weight calculation unit 311 calculates a weight for each output item such that the smaller the difference between the gradient vector and an average gradient vector obtained by averaging the gradient vectors for all output items, the larger the weight. Weight calculation unit 311 and weight calculation unit 310 in Fig. 4 have different weight calculation methods. In other respects, weight calculation unit 311 is similar to weight calculation unit 310. Weight calculation unit 311 corresponds to an example of weight calculation means. 4, the specific learning process differs depending on the difference between the weight calculation methods of weight calculation unit 311 and weight calculation unit 310. In other respects, learning control unit 112 is similar to learning control unit 111. Learning control unit 112 corresponds to an example of a learning control means.

[0128] Fig. 16 is a diagram showing an example of input and output of data during learning of the simulation system 3. In the example of Fig. 16, the gradient vector calculation unit 320 receives input of the reference output value and the output value of the simulator 200, and calculates a gradient vector for each output item. The weight calculation unit 311 receives the gradient vector for each output item calculated by the gradient vector calculation unit 320, and calculates a weight for each output item.

[0129] The input and output of data in the learning control unit 112 is the same as that in the learning control unit 111 in Fig. 5. The input and output of data in the parameter value setting unit 120 is the same as that in the parameter value setting unit 120 in Fig. 5. The input and output of data in the simulator 200 is the same as that in the simulator 200 in Fig. 5.

[0130] FIG. 17 is a diagram showing an example of a gradient vector in the space of parameter values. Fig. 17 shows an example in which simulator 200 has two parameters, θ1 and θ2. The horizontal axis of the graph in Fig. 17 indicates the value of parameter θ1. The vertical axis indicates the value of parameter θ2. The orthogonal coordinate space formed by these horizontal and vertical axes corresponds to an example of the space of parameter values.

[0131] Fig. 17 shows an example in which the number of output items is four, and the first of the four sensors of the simulated object 900 is faulty. Therefore, in the example of Fig. 17, among the elements of the output value vector y = [y1, y2, y3, y4] of the simulated object 900, the output value y1 is different from the normal value of the sensor. On the other hand, the output values ​​y2, y3, and y4 are all the normal values ​​of the sensor.

[0132] As shown in equation (15), the square error ε1 of the output value of the simulator 200 with respect to the output value y1 of the simulation target 900 is expressed as L1(θ) as a function with the parameter θ as an argument.

[0133]

number

[0134] Also, as shown in equation (16), the output value f i (x;θ) to y^ i It can also be written as:

[0135]

number

[0136] The derivative of the error L1(θ) with respect to the parameter θ is expressed as in equation (17).

[0137]

number

[0138] Because “y1” in “2(f1(x;θ)-y1)” on the right-hand side of equation (17) indicates a value different from when the sensor is normal, the differential dL1(θ) / dθ of the error L1(θ) due to the parameter θ indicates a value different from when the sensor is normal.

[0139] Also, as in equation (18), the square error ε2 of the output value of the simulator 200 with respect to the output value y2 of the simulation target 900 is expressed as L2(θ) as a function with the parameter θ as an argument.

[0140]

number

[0141] Similarly, the squared errors ε3 and ε4 of the output values ​​of the simulator 200 with respect to the output values ​​y3 and y4 of the simulation target 900 are expressed as L3(θ) and L4(θ), respectively. Alternatively, the normalized squared error ε i The error L i It may be expressed as (θ). In the example of FIG.

[0142] In FIG. 17, the current parameter value in the simulation is represented by "θ". Also, in FIG. 17, the parameter values ​​that minimize the errors L1(θ), L2(θ), L3(θ), and L4(θ) are respectively represented by "argmin θ L1(θ), argmin θ L2(θ) and argmin θ L3(θ) and argmin θ It is expressed as "L4(θ)".

[0143] argmin θ L2(θ), argmin θ L3(θ) and argmin θ L2(θ), L3(θ), and L4(θ) are shown close to each other in the space of parameter values. It is considered that the values ​​of L2(θ), L3(θ), and L4(θ) are all small when the simulator 200 simulates the simulation target 900 with high accuracy.

[0144] In contrast, the first sensor of the simulated object 900 is broken, so argmin θ L1(θ) is argmin θ L2(θ), argminθ L3(θ) and argmin θ It is shown at a position away from L4(θ). Even if the simulator 200 simulates the simulation target 900 with high accuracy, it is possible that the value of the error L1(θ) will not become very small due to a sensor failure.

[0145] Output value y of simulated target 900 i The output value y^ of the simulator 200 for i Error L i The parameter value that minimizes (θ) is θ^ (i) It can also be written as: The parameter value θ^ that minimizes the error L1(θ) of the output value y^1 of the simulator 200 with respect to the output value y1 of the simulated object 900 due to a faulty sensor. (1) and the parameter value θ^ that minimizes the error L2(θ) of the output value y^2 of the simulator 200 with respect to the output value y2 of the simulated object 900 by a normal sensor. (1) The difference between these is shown in equation (19).

[0146]

number

[0147] In addition, in Fig. 17, the gradient vectors of the errors L1(θ), L2(θ), L3(θ), and L4(θ) at the current parameter values ​​in the simulation, ∇ θ L1", "∇ θ L2", "∇ θ L3" and "∇ θ L4" is shown. Furthermore, in Figure 17, the gradient vector ∇ θ L1, ∇ θ L2, ∇ θ L3 and ∇ θ The average gradient vector obtained by averaging L4, ∇ θ "L" is shown. Here, the average error L(θ) obtained by averaging the errors L1(θ), L2(θ), ... is expressed as equation (20).

[0148]

number

[0149] N indicates the number of output items. In the example of FIG. 17, N=4. Average gradient vector ∇ θ L is expressed as in equation (21).

[0150]

number

[0151] In the example of Figure 17, argmin θ L2(θ), argmin θ L3(θ) and argmin θ L4(θ) are close to each other in the parameter value space, so the gradient vector ∇ θ L2", "∇ θ L3" and "∇ θ L4” is a similar vector, i.e., the gradient vector “∇ θ L2", "∇ θ L3" and "∇ θ The difference between the L4 is small.

[0152] On the other hand, argmin θ L1(θ) is argmin θ L2(θ), argmin θ L3(θ) and argmin θ In response to the distance from L4(θ), the gradient vector "∇ θ L1 is the gradient vector ∇ θ L2", "∇ θ L3" and "∇ θ L4" has a different vector. Gradient vector "∇ θ L1 is the gradient vector ∇ θ L2", "∇ θ L3" and "∇ θ The difference from "L4" can also be expressed using differentiation as in equation (21).

[0153]

number

[0154] Average gradient vector ∇ θ In relation to L, the gradient vector "∇ θ L2", "∇ θ L3" and "∇ θ L4" is the average gradient vector ∇ θ The difference from L is small. On the other hand, the gradient vector "∇ θ L1" is the average gradient vector ∇ θ The difference with L is the gradient vector "∇ θ L2", "∇ θ L3" and "∇ θ It is larger than the L4 case.

[0155] In this way, for output items that are unlikely to be normal, such as the output items of a faulty sensor or the output items of a sensor whose accuracy is significantly lower than that of other sensors, the difference between the error gradient vector and the average gradient vector is likely to be large. The difference between the error gradient vector and the average gradient vector can also be expressed as in equation (23).

[0156]

number

[0157] The larger the difference shown in equation (23), the lower the probability that the sensor is normal. Therefore, the weight calculation unit 311 calculates a weight for each output item such that the smaller the difference between the gradient vector and the average gradient vector, the larger the weight. For example, the weight calculation unit 311 calculates the gradient deviation E of the output value of the simulation target 900 for each output item using the formula (24). i may be calculated.

[0158]

number

[0159] Gradient deviation E i is the error for each output item L i Gradient of (θ) dL i It shows the degree of difference between (θ) / dθ and the average gradient of the error for all output items dL(θ) / dθ. Gradient deviation E i The larger it is, the lower the probability of normality is considered to be. The weight calculation unit 311 may calculate the weight vector w using equation (25).

[0160]

number

[0161] “T” denotes the transpose of a vector or a matrix. In equation (25), the weight vector w is represented as a vertical vector. However, the weights calculated by the weight calculation unit 311 are not limited to those shown in equation (25). The weights calculated by the weight calculation unit 311 can be various weights, in which the smaller the difference between the gradient vector and the average gradient vector, the larger the weight.

[0162] In equation (24), "dL i Regarding "(θ) / dθ", it is generally difficult to explicitly obtain a function that indicates the operation of the simulator, and there may be cases where it is difficult to directly obtain the derivative of the simulator 200. Therefore, the gradient vector calculation unit 320 may calculate the gradient vector based on the amount of change in error when the parameter value is randomly changed. For example, the gradient vector calculation unit 320 may repeat the process of randomly changing the parameter value and performing difference approximation of the differential dLi(θ) / dθ as shown in Equation (26) multiple times, and estimate the gradient by taking the average of the multiple times.

[0163]

number

[0164] When it is difficult to randomly change the state of the actual machine of the simulation target 900, the operation of the simulation target 900 may also be simulated using the simulator 200.

[0165] FIG. 18 shows the gradient vector calculation unit 320 calculating the differential dL i 18 is a diagram showing an example of a processing procedure for estimating (θ) / dθ. The scope of application of the processing in FIG. 18 may be limited to cases where the change in the input or output is small, such as the steady state of the simulated object 900. In the process of FIG. 18, gradient vector calculation unit 320 initializes the gradient as shown in equation (27) (step S211).

[0166]

number

[0167] ∇^θL i is the error L i This represents an estimate of the gradient of . The estimate of the gradient is also denoted as the estimated gradient. Next, the gradient vector calculation unit 320 samples random numbers for each parameter and calculates the change in the parameter value Δθ i is set (step S212). For example, the gradient vector calculation unit 320 samples random numbers from an appropriate distribution such as R=U(-1,1) for the number of parameters. If a sample value becomes 0, it is sampled again.

[0168] Next, the gradient vector calculation unit 320 calculates the change Δθ i is set as the sample value to construct the perturbation vector Δθ shown in equation (28) (step S213).

[0169]

number

[0170] p indicates the number of parameters of the simulation model of the simulated object 900. For example, when the gradient vector calculation unit 320 samples a random number from R=U(−1,1), Δθ i Each of these is set to increase or decrease the parameter value by one. Next, the gradient vector calculation unit 320 calculates the error function value L for each of i=1, 2, . . . p. i (θ;x) and L i (θ+Δθ; x) is calculated B times each (step S214). B is an integer constant such that B≧1.

[0171] Next, the gradient vector calculation unit 320 calculates the error function value L obtained in step S214 for each of i=1, 2, . . . p. i (θ;x) and L i (θ+Δθ;x) into Equation 29 to obtain the estimated gradient ∇^ θ L i is updated (step S215).

[0172]

number

[0173] The gradient vector calculation unit 320 calculates the error function value L B times in step S214. i (θ;x) and L i (θ+Δθ;x) into equation (29), the estimated gradient ∇^ θ L i is updated B times. Estimated gradient after B updates ∇^ θ L i is the average gradient of B estimated gradients. The gradient vector calculation unit 320 calculates the estimated gradient ∇^ after B updates. θ L i The differential of the error dL i We adopt (θ) / dθ as the approximate gradient. After step S215, the gradient vector calculation unit 320 ends the process of FIG.

[0174] FIG. 19 is a diagram showing an example of a processing procedure in which the setting device 102 updates the weight of the error. In the process of FIG. 19, the reward function r(y) is defined as shown in Equation (30).

[0175]

number

[0176] 19, the weight calculation unit 311 initializes the weight vector w (step S311). For example, the weight calculation unit 311 sets the initial value of the weight vector w as shown in Equation (31).

[0177]

number

[0178] N is a positive integer indicating the number of output items. The number of elements in the weight vector w is N, and the sum of all elements in the weight vector w is 1. Next, the setting device 102 starts a loop L11 for repeating the process a number of times determined as the number of updates of the policy function π (step S312).

[0179] In the process of loop L11, the setting device 102 starts loop L12, which repeats the process a number of times determined as the number of episodes used for one update of the policy function π (step S313). An episode here is a time series from the start to the end of one simulation of the simulation target 900 by the simulator 200 in learning.

[0180] In the process of loop L12, the learning control unit 112 causes the simulator 200 to simulate the simulation target 900, and calculates the output value y^ of the simulator 200 shown in equation (32). t+1 , and the reward function value according to the reward function r(y) in equation (30) is saved (step S314).

[0181]

number

[0182] z is a vector indicating the control input value, the reference input value, and the output value of the simulator 200. α indicates the parameter value of the policy function π.

[0183] Furthermore, the gradient vector calculation unit 320 determines whether or not a condition for a timing determined as the update timing for the estimated gradient is satisfied (step S315). If it is determined that the update timing condition is met (step S315: YES), the gradient vector calculation unit 320 calculates the error L i Gradient of (θ) dL i The estimated value of (θ) / dθ is updated (step S316). For example, the gradient vector calculation unit 320 may perform the process of FIG.

[0184] Furthermore, the weight calculation unit 311 determines whether the number of episodes reaches a number determined as the weight calculation timing (step S317). When it is determined that the number of episodes reaches the number of times of weight calculation timing (step S317: YES), the weight calculation unit 311 calculates the weight and reflects the calculated weight in the reward function of equation (30) (step S318). For example, the weight calculation unit 311 calculates the gradient deviation E of the output value of the simulation target 900 for each output item using the formula (33). i may be calculated.

[0185]

number

[0186] The gradient of the error for each output item "∇^ θ L i " and the average error gradient for all output items "1 / pΣ k=1 p ∇^ θ L iThe larger the difference between the gradient deviation E i The value of becomes larger. The weight calculation unit 311 calculates the gradient deviation E obtained by the formula (33). i may be input into equation (25) to calculate the weight vector w.

[0187] After step S318, the learning control unit 112 performs a termination process of the loop L12 (step S319). Specifically, the learning control unit 112 determines whether or not the process of the loop L12 has been repeated a predetermined number of episodes. If the learning control unit 112 determines that the process of the loop L12 has not yet been repeated a predetermined number of episodes, the process returns to step S314, and the setting device 102 continues to repeat the process of the loop L12. On the other hand, if it is determined that the processing of loop L12 has been repeated a predetermined number of episodes, the learning control unit 112 ends loop L12.

[0188] When the loop L12 is ended in step S319, the learning control unit 112 updates the policy parameter α by using the stored episode data, thereby updating the policy function π (step S320).

[0189] After step S320, the learning control unit 112 performs a termination process of the loop L11 (step S321). Specifically, the learning control unit 112 determines whether or not the process of the loop L11 has been repeated a predetermined number of updates. If the learning control unit 112 determines that the process of the loop L11 has not yet been repeated a predetermined number of updates, the process returns to step S313, and the setting device 102 continues to repeat the process of the loop L11. On the other hand, if it is determined that the process of the loop L11 has been repeated a predetermined number of updates, the learning control unit 112 ends the loop L11. When the loop L11 is ended in step S321, the setting device 102 ends the processing in FIG.

[0190] As described above, gradient vector calculation unit 320 calculates a gradient vector indicating the gradient in the parameter value space of the error between the output value of simulator 200 and the reference output value for each output item of the simulator. Weight calculation unit 311 calculates a weight for each output item such that the weight increases as the difference between the gradient vector and an average gradient vector obtained by averaging the gradient vectors for all output items decreases.

[0191] For output items where the difference between the gradient vector and the average gradient vector is small, the reference output value is expected to be accurate. According to the setting device 102, it is possible to set a large weight for an output item where the reference output value is expected to be accurate, and this is expected to enable effective learning of parameter value setting.

[0192] Furthermore, gradient vector calculation section 320 calculates a gradient vector based on the amount of change in error when parameter values ​​are randomly changed. This allows the setting device 102 to estimate the gradient of the error even when the differential of the error cannot be directly calculated.

[0193] <Fourth embodiment> FIG. 20 is a diagram illustrating an example of the configuration of a simulation system according to the fourth embodiment. As shown in FIG. 20, the simulation system 4 includes a setting device 103, a first simulator 200A, and a second simulator 200B. The setting device 103 includes a reinforcement learning unit 300 and a parameter value changing unit 330. The reinforcement learning unit 300 includes a learning control unit 110 and a parameter value setting unit 120. Among the components in FIG. 20, those that correspond to the components in FIG. 2 and have similar functions are given the same reference numerals (110, 120), and detailed description thereof will be omitted here.

[0194] The simulation system 4 corresponds to an example of the simulation system 1. The setting device 103 corresponds to an example of the setting device 100. Both the first simulator 200A and the second simulator 200B simulate a simulation target 900. The first simulator 200A corresponds to an example of the simulator 200 in FIG.

[0195] On the other hand, the second simulator 200B is used in place of the simulation target 900. The output value of the second simulator 200B is used as the reference output value. The parameter value changing section 330 changes the parameter values ​​of the second simulator 200B. The parameter value changing section 330 may change the parameter values ​​of the second simulator 200B while the second simulator 200B is executing a simulation.

[0196] Fig. 21 is a diagram showing an example of input and output of data during learning of the simulation system 4. In the example of Fig. 21, the parameter value change unit 330 sets and changes the parameter values ​​of the second simulator 200B. The second simulator 200B receives an input of a control input value, executes a simulation of the simulation target 900 according to the parameter values ​​set and changed by the parameter value change unit, and calculates an output value. As described above, the output value of the second simulator 200B is used as a reference output value.

[0197] The learning control unit 110 receives as input the output value of the first simulator 200A and the output value of the second simulator 200B. The learning control unit 110 calculates an evaluation value using an evaluation function that gives a higher evaluation as the error of the output value of the first simulator 200A from the output value of the second simulator 200B is smaller. For example, the learning control unit 110 may calculate the error of the output value of the first simulator 200A from the output value of the second simulator 200B for each output item, and sum up the errors for each output item.

[0198] The learning control unit 110 controls the learning of the parameter value setting by the parameter value setting unit 120 so that the evaluation indicated by the evaluation value becomes higher. In this way, the learning control unit 110 controls the learning of the parameter value setting by the parameter value setting unit 120 so that the output value of the first simulator 200A approaches the output value of the second simulator 200B.

[0199] The parameter value setting unit 120 receives the control input value, the output value of the first simulator 200A, and the output value of the second simulator 200B. The parameter value setting unit 120 inputs each of the obtained values ​​to a policy function π to calculate parameter values, and sets the calculated parameter values ​​in the first simulator 200A. The first simulator 200A executes a simulation of the simulation target 900 based on the set parameter values, and calculates an output value for a control input value.

[0200] As described above, the output value of the second simulator 200B that simulates the simulation target 900 is used as the reference output value. The learning control unit 110 controls the learning of the setting of the parameter values ​​of the first simulator by the parameter value setting unit 120, using an evaluation function based on the output value of the first simulator 200A when the same input value as the input value to the second simulator 200B is input to the first simulator 200A, which is a simulator for which the parameter value setting unit 120 sets parameter values, and the reference output value.

[0201] In this way, by using the output value of the second simulator 200B as the reference output value, the reinforcement learning unit 300 can learn the setting of parameter values ​​by the parameter value setting unit 120 even if there is little data available on the actual machine of the simulation target 900. Furthermore, by using the output value of the second simulator 200B as a reference output value, the reinforcement learning unit 300 can learn how to respond to abnormal states and other conditions that do not occur often in the actual machine of the simulation target 900.

[0202] Furthermore, the parameter value changing section 330 changes the parameter values ​​of the second simulator 200 B. The reference output value includes the output value of the second simulator when the parameter value changing section 330 changes the parameter values ​​of the second simulator. In this way, the parameter value changing unit 330 changes the parameter values ​​of the second simulator 200B while the second simulator 200B is executing a simulation, thereby obtaining a reference output value corresponding to a dynamic change in the state of the simulation target 900. The reinforcement learning unit 300 uses this reference output value to learn the parameter value setting by the parameter value setting unit 120, thereby making it possible to set the parameter values ​​of the first simulator 200A so that the first simulator 200A simulates the simulation target 900 in response to the dynamic change in the state of the simulation target 900.

[0203] Furthermore, the parameter value setting unit 120 sets parameter values ​​of the first simulator 200A that simulates the simulation target. The learning control unit 110 controls learning of the setting of the parameter values ​​of the first simulator 200A by the parameter value setting unit 120, using an evaluation function based on the output value of the second simulator 200B that simulates the simulation target and the output value of the first simulator when the same input value as the input value to the second simulator 200B is input to the first simulator 200A.

[0204] In this way, by using the output value of the second simulator 200B as the reference output value, the reinforcement learning unit 300 can learn the setting of parameter values ​​by the parameter value setting unit 120 even if there is little data available on the actual machine of the simulation target 900. Furthermore, by using the output value of the second simulator 200B as a reference output value, the reinforcement learning unit 300 can learn how to respond to abnormal states and other conditions that do not occur often in the actual machine of the simulation target 900.

[0205] The second embodiment and the fourth embodiment may be combined and implemented. In this case, in the configuration of Fig. 20, the reinforcement learning unit 300 may further include the weight calculation unit 310 of Fig. 5, and may include the learning control unit 111 of Fig. 5 instead of the learning control unit 110.

[0206] The third and fourth embodiments may be combined and implemented. In this case, the reinforcement learning unit 300 in the configuration of Fig. 20 may further include the weight calculation unit 311 and the gradient vector calculation unit 320 in Fig. 15, and may include the learning control unit 112 in Fig. 15 instead of the learning control unit 110.

[0207] <Fifth embodiment> FIG. 22 is a diagram illustrating an example of the configuration of a simulation system according to the fifth embodiment. As shown in FIG. 22, the simulation system 5 includes a setting device 104, a first simulator 200A, and a second simulator 200B. The setting device 104 includes a first reinforcement learning unit 300A and a second reinforcement learning unit 300B. The first reinforcement learning unit 300A includes a first learning control unit 110A and a first parameter value setting unit 120A. The second reinforcement learning unit 300B includes a second learning control unit 110B, a second parameter value setting unit 120B, and a scaling unit 340.

[0208] 20. The setting device 104 corresponds to an example of the setting device 103. The first reinforcement learning unit 300A corresponds to an example of the reinforcement learning unit 300. The first learning control unit 110A corresponds to an example of the learning control unit 110. The first learning control unit 110A corresponds to an example of a first learning means. The first parameter value setting section 120A corresponds to an example of the parameter value setting section 120. Therefore, the first parameter value setting section 120A corresponds to an example of the parameter value setting means.

[0209] Both the first simulator 200A and the second simulator 200B simulate a simulation target 900. The first simulator 200A corresponds to the example of the first simulator 200A in Fig. 20, and is a target for which parameter values ​​are set by the first parameter value setting unit 120A. 20, and is used in place of the simulation target 900. The output value of the second simulator 200B is used as the reference output value.

[0210] The first parameter value setting unit 120A sets the parameter values ​​of the first simulator 200A by learning under the control of the first learning control unit 110A so that the output value of the first simulator 200A approaches the output value of the second simulator 200A. The first learning control unit 110A controls the learning of the setting of the parameter values ​​of the first simulator 200A by the first parameter value setting unit 120A so as to reduce the difference between the output value of the first simulator 200A and the output value of the second simulator 200B used as a reference output value.

[0211] In response to this, the second parameter value setting unit 120B sets the parameter values ​​of the second simulator 200B through learning under the control of the second learning control unit 110B so that the output value of the second simulator 200A differs from the output value of the first simulator 200A. The second parameter value setting section 120B corresponds to an example of a parameter value changing means.

[0212] The second learning control unit 110B controls the learning of the setting of the parameter values ​​of the second simulator 200B by the second parameter value setting unit 120B so as to increase the difference between the output value of the first simulator and the reference output value, using a second evaluation function based on the output value of the first simulator 200A when the same input value as the input value to the second simulator 200B is input to the first simulator 200A and the output value of the second simulator 200B used as a reference output value. The second learning control unit 110B corresponds to an example of a second learning control means.

[0213] This allows the first reinforcement learning unit 300A to learn how to set parameter values ​​by the first parameter value setting unit 120A, using control input values ​​and reference output values ​​that prevent the first parameter value setting unit 120A from appropriately setting parameter values ​​for the first simulator 200A. This is expected to enable the first parameter value setting unit 120A to appropriately set parameter values ​​for the first simulator 200A for various control input values ​​and reference output values.

[0214] The scaling unit 340 performs scaling to increase the value of the second evaluation function when the learning under the control of the first learning control means progresses beyond a predetermined condition. The predetermined condition here is not limited to a specific upper limit. For example, the predetermined condition may be that learning is performed for a predetermined number of time steps or more, but is not limited thereto.

[0215] In the initial stage of learning, the scaling section 340 gives priority to the setting of the parameter values ​​of the first simulator 200A by the first parameter value setting section 120A over the setting of the parameter values ​​of the second simulator 200B by the second parameter value setting section 120B.

[0216] In the initial stage of learning, if the output value of the second simulator 200A is maintained different from the output value of the first simulator 200A, it is considered that the learning of the first reinforcement learning unit 300A does not progress. Therefore, the scaling unit 340 prioritizes the setting of the parameter values ​​of the first simulator 200A by the first parameter value setting unit 120A over the setting of the parameter values ​​of the second simulator 200B by the second parameter value setting unit 120B. As a result, the output value of the first simulator 200A approaches the output value of the second simulator 200A, and it is expected that the learning of the first reinforcement learning unit 300A will progress. The scaling unit 340 corresponds to an example of a scaling means.

[0217] FIG. 23 is a diagram showing an example of data input and output during learning in the simulation system 5. In FIG. The first parameter value setting unit 120A receives the control input value, the output value of the first simulator 200A, and the output value of the second simulator 200B as inputs, and calculates these values ​​as a first policy function π A to calculate parameter values ​​for the first simulator 200A. The first parameter value setting unit 120A sets the calculated parameter values ​​in the first simulator 200A.

[0218] The first learning control unit 110A receives an input of an output value of the first simulator 200A and an output value of the second simulator 200B. The first learning control unit 110A calculates an evaluation value using a first evaluation function in which the evaluation is higher as the error of the output value of the first simulator 200A with respect to the output value of the second simulator 200B is smaller. The first learning control unit 110A controls the learning of the setting of the parameter value of the first simulator 200A by the first parameter value setting unit 120A so that the evaluation indicated by the evaluation value becomes higher. Thus, the first learning control unit 110A controls the learning of the setting of the parameter value by the first parameter value setting unit 120A so that the output value of the first simulator 200A approaches the output value of the second simulator 200B. The first simulator 200A executes a simulation of the simulation target 900 based on the set parameter values, and calculates an output value for a control input value.

[0219] The first parameter value setting unit 120A receives the control input value, the output value of the first simulator 200A, and the output value of the second simulator 200B as inputs, and calculates these values ​​as a first policy function π A to calculate parameter values ​​for the first simulator 200A. The first parameter value setting unit 120A sets the calculated parameter values ​​in the first simulator 200A.

[0220] The second parameter value setting unit 120B receives the control input value, the output value of the first simulator 200A, and the output value of the second simulator 200B, and calculates these values ​​as a second policy function π Bto calculate parameter values ​​for the second simulator 200B. The second parameter value setting unit 120B sets the calculated parameter values ​​in the second simulator 200B.

[0221] The second learning control unit 110B receives an input of the output value of the first simulator 200A and the output value of the second simulator 200B. The second learning control unit 110B calculates an evaluation value using a second evaluation function in which the evaluation is higher as the difference between the output value of the first simulator 200A and the output value of the second simulator 200B is larger. The second learning control unit 110B controls the learning of the parameter value setting of the second simulator 200B by the second parameter value setting unit 120B so that the evaluation indicated by the evaluation value is higher. Thus, the second learning control unit 110B controls the learning of the parameter value setting by the second parameter value setting unit 120B so that the output value of the second simulator 200B is different from the output value of the first simulator 200A. The second simulator 200B executes a simulation of the simulation target 900 based on the set parameter values, and calculates an output value for a control input value.

[0222] When the learning under the control of the first learning control means progresses beyond a predetermined condition, the scaling unit 340 performs scaling to increase the value of the second evaluation function. As described above, in the early stage of learning, the scaling unit 340 prioritizes the setting of the parameter values ​​of the first simulator 200A by the first parameter value setting unit 120A over the setting of the parameter values ​​of the second simulator 200B by the second parameter value setting unit 120B.

[0223] As described above, the second learning control unit 110B controls the learning of the setting of the parameter values ​​of the second simulator 200B by the second parameter value setting unit 120B so as to increase the difference between the output value of the first simulator 200A and the reference output value, using a second evaluation function based on the output value of the first simulator 200A when the same input value as the input value to the second simulator 200B is input to the first simulator 200A and the output value of the second simulator 200B used as the reference output value. The first learning control unit 110A controls learning of the setting of the parameter values ​​of the first simulator 200A by the first parameter value setting unit 120A so as to reduce the difference between the output value of the first simulator 200A and the reference output value.

[0224] This allows the first reinforcement learning unit 300A to learn how to set parameter values ​​by the first parameter value setting unit 120A, using control input values ​​and reference output values ​​that prevent the first parameter value setting unit 120A from appropriately setting parameter values ​​for the first simulator 200A. This is expected to enable the first parameter value setting unit 120A to appropriately set parameter values ​​for the first simulator 200A for various control input values ​​and reference output values.

[0225] When the learning under the control of the first learning control unit 110A progresses beyond a predetermined condition, the scaling unit 340 performs scaling to increase the value of the second evaluation function. If the output value of the second simulator 200A is kept different from the output value of the first simulator 200A in the initial stage of learning, it is considered that the learning of the first reinforcement learning unit 300A does not progress. Therefore, the scaling unit 340 prioritizes the setting of the parameter values ​​of the first simulator 200A by the first parameter value setting unit 120A over the setting of the parameter values ​​of the second simulator 200B by the second parameter value setting unit 120B in the initial stage of learning by the above-mentioned scaling. As a result, the output value of the first simulator 200A approaches the output value of the second simulator 200A, and it is expected that the learning of the first reinforcement learning unit 300A will progress.

[0226] The second embodiment and the fifth embodiment may be combined and implemented. In this case, in the configuration of Fig. 22, the first reinforcement learning unit 300A may further include the weight calculation unit 310 of Fig. 4, and may include the learning control unit 111 of Fig. 4 instead of the first learning control unit 110A.

[0227] The third embodiment and the fifth embodiment may be implemented in combination. In this case, the first reinforcement learning unit 300A in the configuration of Fig. 22 may further include the weight calculation unit 311 and the gradient vector calculation unit 320 in Fig. 15, and may include the learning control unit 112 in Fig. 15 instead of the first learning control unit 110A.

[0228] Sixth Embodiment FIG. 24 is a diagram illustrating an example of the configuration of a simulation system according to the sixth embodiment. 24, the simulation system 6 includes a setting device 105, a first simulator 200A, and a second simulator 200B. The setting device 105 includes a reinforcement learning unit 300, a parameter value changing unit 330, and a fault setting unit 350. The reinforcement learning unit 300 includes a learning control unit 110 and a parameter value setting unit 120. 24, parts having similar functions to those in FIG. 20 are given the same reference numerals (110, 120, 200A, 200B, 300, 330), and detailed description thereof will be omitted here.

[0229] The simulation system 6 differs from the simulation system 4 in that the setting device 105 further includes a fault setting section 350 in addition to the components of the setting device 103 in FIG. In other respects, simulation system 6 is similar to simulation system 4 .

[0230] The fault setting unit 350 sets a fault of a sensor in the simulation model in the second simulator 200B. The fault setting unit 350 may randomly select a sensor to set a fault. The fault setting unit 305 may switch between the presence and absence of a fault in the sensor during the execution of the simulation by the second simulator 200B. This allows the reinforcement learning unit 300 to learn the setting of parameter values ​​by the parameter value setting unit 120 so that the first simulator 200A can simulate the simulation target 900 with high accuracy even if a sensor of the simulation target 900 fails. The fault setting unit 350 corresponds to an example of the fault setting means.

[0231] When the failure setting unit 350 sets a sensor failure, the learning control unit 110 may use an evaluation function in which the output value of the sensor for which the failure has been set is excluded from the reference targets in the calculation of the evaluation function. For example, a coefficient may be set for each output item in the calculation formula of the evaluation function. Then, the learning control unit 110 may set the coefficient value of the output item corresponding to the sensor for which the failure has been set to 0. In this way, by the learning control unit 110 excluding the output value of a sensor that has been set as faulty from the references in the calculation of the evaluation function, it is expected that the reinforcement learning unit 300 will be able to perform learning efficiently.

[0232] Fig. 25 is a diagram showing an example of input and output of data during learning of the simulation system 6. In the example of Fig. 25, the parameter value changing unit 330 sets and changes the parameter values ​​of the second simulator 200B. The fault setting section 350 sets a fault in a sensor in the simulation model in the second simulator 200B. The second simulator 200B receives the control input value, executes a simulation of the simulation target 900 according to the parameter values ​​set and changed by the parameter value changing unit and the sensor failure setting by the failure setting unit 350, and calculates an output value. As described above, the output value of the second simulator 200B is used as the reference output value.

[0233] The learning control unit 110 receives as input the output value of the first simulator 200A and the output value of the second simulator 200B. The learning control unit 110 calculates an evaluation value using an evaluation function that gives a higher evaluation as the error of the output value of the first simulator 200A from the output value of the second simulator 200B is smaller. For example, the learning control unit 110 may calculate the error of the output value of the first simulator 200A from the output value of the second simulator 200B for each output item, and sum up the errors for each output item.

[0234] The learning control unit 110 controls the learning of the parameter value setting by the parameter value setting unit 120 so that the evaluation indicated by the evaluation value becomes higher. In this way, the learning control unit 110 controls the learning of the parameter value setting by the parameter value setting unit 120 so that the output value of the first simulator 200A approaches the output value of the second simulator 200B.

[0235] The parameter value setting unit 120 receives the control input value, the output value of the first simulator 200A, and the output value of the second simulator 200B. The parameter value setting unit 120 inputs each of the obtained values ​​to a policy function π to calculate parameter values, and sets the calculated parameter values ​​in the first simulator 200A. The first simulator 200A executes a simulation of the simulation target 900 based on the set parameter values, and calculates an output value for a control input value.

[0236] As described above, the fault setting section 350 sets a fault in the sensor in the simulation model in the second simulator 200B. This allows the reinforcement learning unit 300 to learn the setting of parameter values ​​by the parameter value setting unit 120 so that the first simulator 200A can simulate the simulation target 900 with high accuracy even if a sensor of the simulation target 900 fails.

[0237] Furthermore, the learning control unit 110 uses an evaluation function in which the output value of a sensor in which a fault has been set is excluded from the objects to be referred to in the calculation of the evaluation function. In this way, by the learning control unit 110 excluding the output value of a sensor that has been set as faulty from the references in the calculation of the evaluation function, it is expected that the reinforcement learning unit 300 will be able to perform learning efficiently.

[0238] The second embodiment and the sixth embodiment may be combined and implemented. In this case, the setting device 101 may further include a fault setting unit 350 in the configuration of FIG.

[0239] The third embodiment and the sixth embodiment may be combined and implemented. In this case, the setting device 102 may further include a fault setting section 350 in the configuration of FIG.

[0240] The fifth and sixth embodiments may be combined and implemented. In this case, the setting device 104 may further include a fault setting section 350 in the configuration of FIG.

[0241] Furthermore, the second embodiment, the fifth embodiment, and the sixth embodiment may be implemented in combination. The third embodiment, the fifth embodiment, and the sixth embodiment may be implemented in combination.

[0242] In any one of the first to sixth embodiments, or a combination thereof, the learning control means may be configured to use an evaluation function based on the output values ​​of the simulator at a time in the simulation of the object simulated by the simulator that includes a time in the future than the time at which the parameter value setting means sets the parameter value. This is expected to allow the parameter value setting means to set the parameter values ​​of the simulator so that the simulator will simulate the controlled object with high accuracy at a future time.

[0243] In the first embodiment, the learning control unit 110 corresponds to an example of the learning control means here. The simulator 200 corresponds to an example of the simulator here. The parameter value setting unit 120 corresponds to an example of the parameter value setting means here. In the second embodiment, the learning control unit 111 corresponds to an example of the learning control means here. The simulator 200 corresponds to an example of the simulator here. The parameter value setting unit 120 corresponds to an example of the parameter value setting means here.

[0244] In the third embodiment, the learning control unit 112 corresponds to an example of the learning control means here. The simulator 200 corresponds to an example of the simulator here. The parameter value setting unit 120 corresponds to an example of the parameter value setting means here. In the fourth embodiment, the learning control unit 110 corresponds to an example of the learning control means here. The first simulator 200A corresponds to an example of the simulator here. The parameter value setting unit 120 corresponds to an example of the parameter value setting means here.

[0245] In the fifth embodiment, the first learning control unit 110A corresponds to an example of the learning control means here. The first simulator 200A corresponds to an example of the simulator here. The first parameter value setting unit 120A corresponds to an example of the parameter value setting means here. In the sixth embodiment, the learning control unit 110 corresponds to an example of the learning control means here. The first simulator 200A corresponds to an example of the simulator here. The parameter value setting unit 120 corresponds to an example of the parameter value setting means here.

[0246] Seventh embodiment FIG. 26 is a diagram illustrating an example of the configuration of a setting device according to the seventh embodiment. In the configuration shown in FIG. 26, a setting device 610 includes a parameter value setting section 611, a weight calculation section 612, and a learning control section 613. A parameter value setting unit 611 sets parameter values ​​of a simulator that simulates a simulation target. A weight calculation unit 612 calculates a weight for each output item based on an output value of the simulator and a reference output value that is a target for simulation by the output value of the simulator. A learning control unit 613 controls learning of the setting of the parameter values ​​of the simulator by the parameter value setting unit 611 using an evaluation function based on the output value of the simulator, the reference output value, and the weight. The parameter value setting unit 611 corresponds to an example of a parameter value setting means, the weight calculation unit 612 corresponds to an example of a weight calculation means, and the learning control unit 613 corresponds to an example of a learning control means.

[0247] The setting device 610 can set a small weight for an output item whose reference output value has a low probability of being normal due to a sensor failure, etc., thereby reducing the impact on the evaluation value. In this respect, the setting device 610 is expected to be able to effectively learn the setting of the simulator's parameter values ​​when output values ​​for multiple items can be obtained from the simulation target.

[0248] The parameter value setting unit 611 can be realized, for example, by using the functions of the parameter value setting unit 120 shown in Fig. 4. The weight calculation unit 612 can be realized, for example, by using the functions of the weight calculation unit 310 shown in Fig. 4. The learning control unit 613 can be realized, for example, by using the functions of the learning control unit 111 shown in Fig. 4.

[0249] Eighth embodiment FIG. 27 is a diagram illustrating an example of the configuration of a setting device according to the eighth embodiment. In the configuration shown in FIG. 27, a setting device 620 includes a parameter value setting section 621 and a learning control section 622. The parameter value setting unit 621 sets parameter values ​​of a first simulator that simulates the simulation target. The learning control unit 622 controls learning of the setting of the parameter values ​​of the first simulator by the parameter value setting unit 621 using an evaluation function based on an output value of a second simulator that simulates the simulation target and an output value of the first simulator when the same input value as the input value to the second simulator is input to the first simulator. The parameter value setting unit 621 corresponds to an example of a parameter value setting means, and the learning control unit 622 corresponds to an example of a learning control means.

[0250] According to the setting device 620, by performing learning using the output values ​​of the second simulator, even if there is little data available from the actual device to be simulated, it is possible to learn the setting of the parameter values ​​by the parameter value setting unit 621. Furthermore, the setting device 620 can learn how to respond to abnormal conditions and other situations that do not occur often in the actual machine being simulated by performing learning using the output values ​​of the second simulator.

[0251] The parameter value setting unit 621 can be realized, for example, by using the functions of the parameter value setting unit 120 shown in Fig. 20. The learning control unit 622 can be realized, for example, by using the functions of the learning control unit 110 shown in Fig. 20.

[0252] <Ninth embodiment> FIG. 28 is a diagram illustrating an example of a processing procedure in the setting method according to the ninth embodiment. The setting method shown in FIG. 28 includes setting parameter values ​​(step S611), calculating weights (step S612), and controlling learning (step S613). In setting parameter values ​​(step S611), parameter values ​​of a simulator that simulates a simulation target are set. In calculating weights (step S612), weights for each output item are calculated based on the output values ​​of the simulator and the reference output values ​​that are the targets of simulation by the output values ​​of the simulator. In controlling learning (step S613), learning of the setting of the parameter values ​​of the simulator is controlled using an evaluation function based on the output values ​​of the simulator, the reference output values, and the weights. According to the setting method shown in Fig. 28, it is possible to reduce the impact on the evaluation value by setting a small weight for an output item whose reference output value has a low probability of being normal due to a sensor failure, etc. In this respect, according to the setting method shown in Fig. 28, when output values ​​for multiple items can be obtained from the simulated object, it is expected that learning to set the parameter values ​​of the simulator can be performed effectively.

[0253] Tenth embodiment FIG. 29 is a diagram illustrating an example of a processing procedure in the setting method according to the tenth embodiment. The setting method shown in FIG. 29 includes setting parameter values ​​(step S621) and controlling learning (step S622). In setting parameter values ​​(step S621), parameter values ​​of a first simulator that simulates the simulated object are set. In controlling learning (step S622), learning of setting parameter values ​​of the first simulator is controlled using an evaluation function based on an output value of a second simulator that simulates the simulated object and an output value of the first simulator when the same input value as the input value to the second simulator is input to the first simulator.

[0254] According to the setting method shown in FIG. 29, by performing learning using the output values ​​of the second simulator, it is possible to learn the settings of parameter values ​​even when there is little data available from the actual device to be simulated. Furthermore, according to the setting method shown in FIG. 29, by performing learning using the output values ​​of the second simulator, it is possible to learn how to respond to abnormal conditions and other situations that do not occur often in the actual machine being simulated.

[0255] FIG. 30 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment. In the configuration shown in FIG. 30, a computer 700 includes a CPU 710, a main memory device 720, an auxiliary memory device 730, an interface 740, and a non-volatile recording medium 750.

[0256] Any one or more of the setting devices 100, 101, 102, 103, 104, 105, 610, and 620 or a part thereof may be implemented in the computer 700. In this case, the operation of each of the above-mentioned processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, develops it in the main storage device 720, and executes the above-mentioned processing according to the program. The CPU 710 also secures a storage area corresponding to each of the above-mentioned storage units in the main storage device 720 according to the program. Communication between each device and other devices is executed by the interface 740 having a communication function and performing communication according to the control of the CPU 710. The interface 740 also has a port for the non-volatile recording medium 750, and reads information from the non-volatile recording medium 750 and writes information to the non-volatile recording medium 750.

[0257] When the setting device 100 is implemented in a computer 700, the operations of the learning control unit 110 and the parameter value setting unit 120 are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0258] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the processing performed by the setting device 100 in accordance with the program. The communication between the setting device 100 and other devices is carried out by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the setting device 100 and a user is carried out by the interface 740 having an input device and an output device, presenting information to the user via the output device under the control of the CPU 710, and receiving user operations via the input device.

[0259] When the setting device 101 is implemented in a computer 700, the operations of the learning control unit 111, the parameter value setting unit 120, and the weight calculation unit 310 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0260] Furthermore, the CPU 710 reserves a storage area in the main storage device 720 for the processing performed by the setting device 101 in accordance with the program. The communication between the setting device 101 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the setting device 101 and a user is performed by the interface 740 having an input device and an output device, presenting information to the user via the output device under the control of the CPU 710, and accepting user operations via the input device.

[0261] When the setting device 102 is implemented in a computer 700, the operations of the learning control unit 112, the parameter value setting unit 120, the weight calculation unit 311, and the gradient vector calculation unit 320 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing according to the program.

[0262] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the processing performed by the setting device 102 in accordance with the program. The communication between the setting device 102 and other devices is carried out by the interface 740 having a communication function and operating under the control of the CPU 710 . Interaction between the setting device 102 and a user is performed by the interface 740 having an input device and an output device, presenting information to the user via the output device under the control of the CPU 710, and accepting user operations via the input device.

[0263] When the setting device 103 is implemented in a computer 700, the operations of the reinforcement learning unit 300, the parameter value change unit 330, and each of the units are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it in the main storage device 720, and executes the above-mentioned processing according to the program.

[0264] Furthermore, the CPU 710 reserves a storage area in the main storage device 720 for the processing performed by the setting device 103 in accordance with the program. The communication between the setting device 103 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the setting device 103 and the user is carried out by the interface 740 having an input device and an output device, presenting information to the user via the output device under the control of the CPU 710, and accepting user operations via the input device.

[0265] When the setting device 104 is implemented in a computer 700, the operations of the first reinforcement learning unit 300A, the second reinforcement learning unit 300B and each of the units are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it in the main storage device 720, and executes the above-mentioned processing according to the program.

[0266] Furthermore, the CPU 710 reserves a storage area in the main storage device 720 for the processing performed by the setting device 104 in accordance with the program. The communication between the setting device 104 and other devices is carried out by the interface 740 having a communication function and operating under the control of the CPU 710 . Interaction between the setting device 104 and the user is carried out by the interface 740 having an input device and an output device, presenting information to the user via the output device under the control of the CPU 710, and accepting user operations via the input device.

[0267] When the setting device 105 is implemented in a computer 700, the reinforcement learning unit 300, the parameter value changing unit 330, the fault setting unit 350, and the operations of each unit are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it in the main storage device 720, and executes the above-mentioned processing according to the program.

[0268] Furthermore, the CPU 710 reserves a storage area in the main storage device 720 for the processing performed by the setting device 105 in accordance with the program. The communication between the setting device 105 and other devices is carried out by the interface 740 having a communication function and operating under the control of the CPU 710 . Interaction between the setting device 105 and a user is carried out by the interface 740 having an input device and an output device, presenting information to the user via the output device under the control of the CPU 710, and accepting user operations via the input device.

[0269] When the setting device 610 is implemented in the computer 700, the operations of the parameter value setting unit 611, the weight calculation unit 612, and the learning control unit 613 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0270] Furthermore, the CPU 710 reserves a storage area in the main storage device 720 for the processing performed by the setting device 610 in accordance with the program. Communication between the setting device 610 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the setting device 610 and a user is carried out by the interface 740 having an input device and an output device, presenting information to the user via the output device under the control of the CPU 710, and accepting user operations via the input device.

[0271] When the setting device 620 is implemented in the computer 700, the operations of the parameter value setting unit 621 and the learning control unit 622 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0272] Furthermore, the CPU 710 reserves a storage area in the main storage device 720 for the processing performed by the setting device 620 in accordance with the program. Communication between the setting device 620 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the setting device 620 and a user is carried out by the interface 740 having an input device and an output device, presenting information to the user via the output device under the control of the CPU 710, and accepting user operations via the input device.

[0273] Any one or more of the above-mentioned programs may be recorded in non-volatile recording medium 750. In this case, interface 740 may read the program from non-volatile recording medium 750. Then, CPU 710 may directly execute the program read by interface 740, or may temporarily store the program in main storage device 720 or auxiliary storage device 730 and execute it.

[0274] Note that a program for executing all or part of the processing performed by the setting devices 100, 101, 102, 103, 104, 105, 610, and 620 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing of each part. Note that the term "computer system" here includes hardware such as the OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into computer systems. The above-mentioned program may be for realizing part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

[0275] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included.

[0276] Some or all of the above embodiments may be described as follows, but are not limited to the following:

[0277] (Appendix 1) a parameter value setting means for setting parameter values ​​of a simulator that simulates a simulation target; a weight calculation means for calculating a weight for each output item based on an output value of the simulator and a reference output value to be simulated by the output value of the simulator; a learning control means for controlling learning of the setting of the parameter values ​​of the simulator by the parameter value setting means, using an evaluation function based on the output value of the simulator, the reference output value, and the weight; A setting device comprising:

[0278] (Appendix 2) the weight calculation means calculates a weight for each output item such that an output item having a larger post-normalization error after normalization of the error for each output item has a smaller weight. 2. The setting device according to claim 1.

[0279] (Appendix 3) the learning control means calculates, for each unit of learning, an error index value indicating the magnitude of a weighted error between the output value of the simulator and the reference output value for all of the output items, and when the error index value after execution of one unit of learning indicates a larger error than the error index value before execution of that unit of learning, restores the weight to the weight before execution of that unit of learning. 3. The setting device according to claim 2.

[0280] (Appendix 4) a gradient vector calculation means for calculating, for each output item of the simulator, a gradient vector indicating the gradient of the error between the output value of the simulator and the reference output value in the space of the parameter values; Further equipped with The weight calculation means calculates a weight for each of the output items such that the weight is larger as the difference between the gradient vector and an average gradient vector obtained by averaging the gradient vectors for all of the output items is smaller. 2. The setting device according to claim 1.

[0281] (Appendix 5) the gradient vector calculation means calculates the gradient vector based on an amount of change in the error when the parameter value is randomly changed. 5. The setting device according to claim 4.

[0282] (Appendix 6) the learning control means uses the evaluation function based on an output value of the simulator at a time in the simulation of the object to be simulated by the simulator, the time including a time in the future than a time at which the parameter value setting means sets the parameter value; 6. A setting device according to any one of claims 1 to 5.

[0283] (Appendix 7) the reference output value is an output value of a second simulator that simulates the simulation target, the learning control means controls learning of the setting of the parameter values ​​of the first simulator by the parameter value setting means, using an evaluation function based on an output value of the first simulator when an input value identical to an input value to the second simulator is input to the first simulator, which is the simulator for which the parameter value setting means sets the parameter values, the reference output value, and the weight. 7. A setting device according to any one of claims 1 to 6.

[0284] (Appendix 8) a parameter value changing means for changing a parameter value of the second simulator; Further equipped with the reference output value includes an output value of the second simulator when the parameter value changing means changes the parameter value of the second simulator. 8. The setting device according to claim 7.

[0285] (Appendix 9) a second learning control means for controlling learning of the setting of the parameter values ​​of the second simulator by the parameter value changing means, using a second evaluation function based on the output value of the first simulator when the same input value as the input value to the second simulator is input to the first simulator and the reference output value, so as to increase the difference between the output value of the first simulator and the reference output value. Further equipped with the learning control means for controlling learning of the parameter values ​​of the first simulator is a first learning control means for controlling learning of setting of the parameter values ​​of the first simulator by the parameter value setting means so as to reduce a difference between the output value of the first simulator and the reference output value; 9. The setting device according to claim 8.

[0286] (Appendix 10) a scaling means for performing scaling to increase the value of the second evaluation function when learning under the control of the first learning control means progresses beyond a predetermined condition; 10. The setting device according to claim 9, further comprising:

[0287] (Appendix 11) The second simulator further includes a fault setting means for setting a fault of a sensor in a simulation model. 11. The setting device according to any one of appendixes 8 to 10.

[0288] (Appendix 12) the weight calculation means sets the weight for the output value of the sensor in which a fault has been set to a value indicating exclusion from the calculation of the evaluation function. 12. The setting device of claim 11.

[0289] (Appendix 13) a parameter value setting means for setting parameter values ​​of a first simulator that simulates a simulation target; a learning control means for controlling learning of the setting of parameter values ​​of the first simulator by the parameter value setting means, using an evaluation function based on an output value of a second simulator which simulates the simulation target and an output value of the first simulator when the first simulator is input with an input value that is the same as an input value to the second simulator; A setting device comprising:

[0290] (Appendix 14) a parameter value changing means for changing a parameter value of the second simulator; Further equipped with the output value of the second simulator includes an output value when the parameter value changing means changes the parameter value of the second simulator; 14. The setting device of claim 13.

[0291] (Appendix 15) a second learning control means for controlling learning of the setting of the parameter values ​​of the second simulator by the parameter value changing means, using a second evaluation function based on an output value of the first simulator when the same input value as the input value to the second simulator is input to the first simulator and an output value of the second simulator, so as to increase a difference between the output value of the first simulator and the output value of the second simulator. Further equipped with the learning control means for controlling learning of the parameter values ​​of the first simulator is a first learning control means for controlling learning of setting of the parameter values ​​of the first simulator by the parameter value setting means so as to reduce a difference between an output value of the first simulator and an output value of the second simulator. 15. The setting device of claim 14.

[0292] (Appendix 16) a scaling means for performing scaling to increase the value of the second evaluation function when learning under the control of the first learning control means progresses beyond a predetermined condition; 16. The setting device of claim 15, further comprising:

[0293] (Appendix 17) The second simulator further includes a fault setting means for setting a fault of a sensor in a simulation model. 17. The setting device according to any one of appendixes 14 to 16.

[0294] (Appendix 18) the learning control means uses the evaluation function in which the output value of the sensor in which a fault has been set is excluded from references in the calculation of the evaluation function. 18. The setting device of claim 17.

[0295] (Appendix 19) The computer Set parameter values ​​of a simulator that simulates the simulated object; calculating a weight for each output item based on an output value of the simulator and a reference output value to be simulated by the output value of the simulator; controlling learning of the parameter value settings of the simulator using an evaluation function based on the output value of the simulator, the reference output value, and the weights; How to set it up.

[0296] (Appendix 20) The computer setting parameter values ​​of a first simulator that simulates a simulated object; controlling learning of parameter value settings of the first simulator by using an evaluation function based on an output value of a second simulator that simulates the simulation target and an output value of the first simulator when the same input value as the input value to the second simulator is input to the first simulator; How to set it up.

[0297] (Appendix 21) On the computer, Setting parameter values ​​of a simulator that simulates a simulation target; Calculating a weight for each output item based on an output value of the simulator and a reference output value to be simulated by the output value of the simulator; Controlling learning of parameter value settings of the simulator using an evaluation function based on the output value of the simulator, the reference output value, and the weights; A recording medium for recording a program for executing the above.

[0298] (Appendix 22) On the computer, Setting parameter values ​​of a first simulator that simulates a simulation object; controlling learning of parameter value settings of the first simulator by using an evaluation function based on an output value of a second simulator that simulates the simulation target and an output value of the first simulator when the first simulator is input with an input value that is the same as an input value to the second simulator; A recording medium for recording a program for executing the above. [Industrial Applicability]

[0299] The embodiment of the present invention may be applied to a setting device, a setting method, and a recording medium. [Explanation of symbols]

[0300] 1, 2, 3, 4, 5, 6 Simulation System 100, 101, 102, 103, 104, 105, 610, 620 Setting device 110, 111, 112, 613, 622 Learning control unit 110A First learning control unit 110B Second learning control unit 120,611,621 Parameter value setting section 120A 1st parameter value setting section 120B second parameter value setting unit 130 Weight setting section 200 Simulator 200A No. 1 Simulator 200B No. 2 Simulator 300 Reinforcement Learning Department 300A Reinforcement Learning Division 1 300B Second Reinforcement Learning Department 310, 311, 612 Weight calculation unit 320 Gradient Vector Calculation Unit 330 Parameter value change section 340 Scaling section 350 Fault setting section 900 Simulated Target

Claims

1. a parameter value setting means for setting parameter values ​​of a simulator that simulates a simulation target; a weight calculation means for calculating a weight for each output item, which is an item for each output value of the simulator, based on an output value of the simulator and a reference output value to be simulated by the output value of the simulator; a learning control means for controlling learning of the setting of the parameter values ​​of the simulator by the parameter value setting means, using an evaluation function based on the output value of the simulator, the reference output value, and the weight; A setting device comprising:

2. the weight calculation means calculates a weight for each of the output items such that an output item having a larger post-normalization error obtained by normalizing the error for each of the output items has a smaller weight. The setting device according to claim 1 .

3. a gradient vector calculation means for calculating, for each of the output items, a gradient vector indicating the gradient of the error between the output value of the simulator and the reference output value in the space of the parameter values; Further equipped with The weight calculation means calculates a weight for each of the output items such that the weight is larger as the difference between the gradient vector and an average gradient vector obtained by averaging the gradient vectors for all of the output items is smaller. The setting device according to claim 1 .

4. the learning control means uses the evaluation function based on an output value of the simulator at a time in the simulation of the object to be simulated by the simulator, the time including a time in the future than a time at which the parameter value setting means sets the parameter value; The setting device according to any one of claims 1 to 3.

5. a parameter value setting means for setting parameter values ​​of a first simulator that simulates a simulation target; a learning control means for controlling learning of the setting of parameter values ​​of the first simulator by the parameter value setting means, using an evaluation function based on an output value of a second simulator which simulates the simulation target and is used as a reference output value to be simulated by the output value of the first simulator, and on an output value of the first simulator when an input value identical to an input value to the second simulator is input to the first simulator; A setting device comprising:

6. a parameter value changing means for changing a parameter value of the second simulator; Further equipped with the output value of the second simulator includes an output value when the parameter value changing means changes the parameter value of the second simulator. The setting device according to claim 5 .

7. The computer Set parameter values ​​of a simulator that simulates the simulated object; calculating a weight for each output item, which is an item for each output value of the simulator, based on an output value of the simulator and a reference output value to be simulated by the output value of the simulator; controlling learning of the parameter value settings of the simulator using an evaluation function based on the output value of the simulator, the reference output value, and the weights; How to set it up.

8. The computer Setting parameter values ​​of a first simulator that simulates a simulation target; a second simulator that simulates the simulation target and controls learning of parameter value settings of the first simulator using an evaluation function based on an output value of the second simulator, the output value of which is used as a reference output value to be simulated by the output value of the first simulator, and an output value of the first simulator when an input value identical to an input value to the second simulator is input to the first simulator; How to set it up.

9. On the computer, Setting parameter values ​​of a simulator that simulates a simulation target; Calculating a weight for each output item, which is an item for each output value of the simulator, based on an output value of the simulator and a reference output value to be simulated by the output value of the simulator; Controlling learning of parameter value settings of the simulator using an evaluation function based on the output value of the simulator, the reference output value, and the weights; A program for executing.

10. On the computer, Setting parameter values ​​of a first simulator that simulates a simulation object; simulating the simulation target, and controlling learning of parameter value settings of the first simulator by using an evaluation function based on output values ​​of a second simulator, the output value of which is used as a reference output value to be simulated by the output value of the first simulator, and on the output value of the first simulator when the same input value as the input value to the second simulator is input to the first simulator; A program for executing.

Citation Information

Patent Citations

  • Self learning diagnosis for plant, estimation and device therefor

    JP1994289179A

  • Plant simulation method using enhanced learning method

    JP2004178492A

  • Equipment characteristics parameter estimation device and equipment characteristics parameter information output device

    JP2005050283A

  • Operation support device and operation support method

    JP2006344004A

  • Control device, method and program

    JP2008304970A