Control device, control method, and non-transitory computer readable medium

Abstract

A control device is provided with a model output acquisition section that acquires an operation model output that is output in correspondence with state data representing a state of a device being input to an operation model, the operation model being obtained by machine learning in a manner that outputs an action corresponding to the state of the device using simulation data from a simulator that simulates an operation of the device; an index acquisition section that acquires an index that is calculated using actual data from the device for monitoring a difference between an action of the simulator and an actual operation of the device; a correction section that corrects the operation model output based on the index; and a control section that outputs an operation amount for controlling a control target provided to the device in accordance with the corrected operation model output.

Description

Technical Field

[0001] This invention relates to control devices, control methods, and non-transitory computer-readable media. Background Technology

[0002] Patent Document 1 describes a method where "Model 45 outputs recommended control parameters representing the first type of control content recommended to improve the reward value based on the input of measurement data." Additionally, Non-Patent Document 1 describes "FKDPP (Factorial Kernel Dynamic Policy Programming)."

[0003] Patent Document 1: Japanese Patent Application Publication No. 2021-086283

[0004] Patent Document 2: Japanese Patent Application Publication No. 2020-027556

[0005] Patent Document 3: Japanese Patent Application Publication No. 2019-020885

[0006] Non-patent document 1: "Yokogawa Electric's NAIST Chemical Processing and Reinforcement Learning", Japanese Robotics March 2019 issue Summary of the Invention

[0007] In a first aspect of the present invention, a control device is provided. The control device may include a model output acquisition unit that acquires an operation model output corresponding to state data representing the state of the device, which is output in a manner that performs machine learning by inputting simulation data from a simulator that operates the device to a machine learning model to output actions corresponding to the state of the device. The control device may include an index acquisition unit that acquires an index calculated using actual data from the device for monitoring the difference between the simulator's actions and the actual operation of the device. The control device may include a correction unit that corrects the operation model output based on the index. The control device may include a control unit that outputs an operational quantity for controlling a controlled object disposed on the device based on the corrected operation model output.

[0008] The indicator acquisition unit can acquire the output of a monitoring model that performs machine learning by inputting the state data into a monitoring model that outputs the indicator corresponding to the state of the device, and use this output as the indicator.

[0009] The monitoring model can be a machine learning model that performs teacher-led learning in a way that reduces the difference between the output of the monitoring model and the teacher signal.

[0010] The control device may further include a monitoring model learning unit that generates the monitoring model through teacher-led learning.

[0011] The operational model can be a machine learning model that performs reinforcement learning by outputting actions that increase the reward value specified by the reward function.

[0012] The control device may further include an operation model learning unit, which generates the operation model through the reinforcement learning.

[0013] The indicator acquisition unit can acquire the future predicted value of the sensor contained in the state data, or the future predicted value of the sensor that can be inferred based on the state data, and use it as the indicator.

[0014] The indicator acquisition unit can acquire a value that quantifies the quality of the product or semi-finished product manufactured in the equipment and use it as the indicator.

[0015] The control device may further include a correction condition setting unit, which sets the conditions for correcting the output of the operation model.

[0016] The conditions may include a permissible range for the indicator. The correction unit can correct the output of the operating model in a manner that brings the indicator towards the range if the indicator is outside the range.

[0017] The indicator acquisition unit can acquire the calculated value calculated using the simulator and use it as the indicator.

[0018] The correction unit can use a correction function, which includes the difference between the actual measured value and the calculated value represented by the state data, in the correction terms to correct the output of the operation model.

[0019] In a second aspect of the present invention, a control method is provided. The control method can be executed by a computer. The control method may include the following steps: the computer acquires and outputs an operation model output representing the state of the device, which is input to an operation model that performs machine learning using simulation data from a simulator of the device's operation to output actions corresponding to the state of the device. The control method may include the following steps: the computer acquires an index calculated using actual data from the device for monitoring the difference between the simulator's actions and the actual operation of the device. The control method may include the following steps: the computer corrects the operation model output based on the index. The control method may include the following steps: the computer outputs an operational quantity for controlling a controlled object located on the device, based on the corrected operation model output.

[0020] In a third aspect of the invention, a non-transitory computer-readable medium is provided for recording a control program. The control program can be executed by a computer. The control program can cause the computer to function as a model output acquisition unit, which acquires operation model outputs corresponding to state data representing the state of the device, generated by machine learning inputs to an operation model that performs machine learning using simulation data from a simulator operating on the device, outputting actions corresponding to the state of the device. The control program can also cause the computer to function as an indicator acquisition unit, which acquires an indicator calculated using actual data from the device for monitoring the difference between the simulator's actions and the actual operation of the device. The control program can further cause the computer to function as a correction unit, which corrects the operation model output based on the indicator. Finally, the control program can also cause the computer to function as a control unit, which outputs an operational quantity for controlling a controlled object located on the device based on the corrected operation model output.

[0021] Furthermore, the above summary of the invention does not list all the essential features of the invention. Additionally, sub-components of the above-described feature set can also constitute the invention. Attached Figure Description

[0022] Figure 1 This is an example of a block diagram showing a control system 1 that may include the control device 100 involved in this embodiment.

[0023] Figure 2 An example of a block diagram showing the operation model management device 200.

[0024] Figure 3This is an example of a data table that an operational model can have.

[0025] Figure 4 This represents an example of an action table used to determine actions based on an operational model.

[0026] Figure 5 This is an example illustrating the generation process of an operational model based on reinforcement learning.

[0027] Figure 6 An example of a block diagram representing the index calculation device 300.

[0028] Figure 7 This represents an example of teacher-generated learning data used to monitor the model.

[0029] Figure 8 This is an example of the control flow based on the control device 100 involved in this embodiment.

[0030] Figure 9 This is an example of a block diagram showing a variation of the control device 100 according to this embodiment.

[0031] Figure 10 Examples of computers 9900 that can embody all or part of the various embodiments of the present invention are shown. Detailed Implementation

[0032] The present invention will now be described through embodiments thereof, which do not limit the scope of the invention as defined in the claims. Furthermore, not all combinations of the features described in the embodiments are essential to the solution of the invention.

[0033] Figure 1 This is an example of a block diagram showing a control system 1 that may include the control device 100 involved in this embodiment. Furthermore, these modules are functionally separate modules and may not correspond to the actual device structure. That is, they may be shown as a single module in this figure, but may not be composed of a single device. Additionally, they may be shown as different modules in this figure, but may not be composed of different devices. The same applies to the block diagrams thereafter.

[0034] The control system 1 may include device 10, simulator 20, database 30, control device 100, operation model management device 200, and index calculation device 300.

[0035] Equipment 10 is a facility or device that has a controlled object 15. For example, equipment 10 can be a factory or a composite device composed of multiple pieces of equipment. As a factory, in addition to industrial factories such as chemical and biological factories, examples include factories that manage and control wellheads and their surroundings in gas fields and oil fields, factories that manage and control power generation such as hydropower, thermal power, and nuclear power, factories that manage and control power generation from environmental resources such as solar and wind power, and factories that manage and control water supply and drainage, dams, etc.

[0036] Next, as an example, we will describe the case where device 10 is a distillation apparatus as part of a process unit. Typically, in a distillation apparatus, low-boiling components are evaporated within a distillation column and discharged from the top. The discharged low-boiling component vapor is condensed in a condenser and stored in a reflux wick. Furthermore, the distillation apparatus returns a portion of the vapor stored in the reflux wick to the distillation column, allowing it to contact the vapor within the column and distill off both low-boiling and high-boiling components. In this distillation apparatus, as an example, the valve located between the reflux wick and the distillation column, whose reflux flow rate should be controlled, is used for opening and closing control.

[0037] The controlled object 15 is a device installed in the apparatus 10 and becoming the controlled object. For example, the controlled object 15 can be an actuator, such as a valve, heater, motor, fan, or switch, that controls at least one physical quantity in the process of the apparatus 10, including the quantity of matter, temperature, pressure, flow rate, speed, and pH, i.e., an operating end, performing a prescribed operation corresponding to the operating quantity. Hereinafter, as an example, the case where the controlled object 15 is a valve installed between the reflux liner and the distillation column of the distillation apparatus will be described. However, it is not limited to this. The controlled object 15 can also be a controller that controls the operating end. That is, the term "control" as used in this specification can be broadly interpreted to include not only direct control of the operating end, but also indirect control of the operating end via a controller.

[0038] One or more sensors capable of measuring various states (physical quantities) inside and outside the device 10 can be installed in the device 10 where the controlled object 15 is located. As an example, in the case that the device 10 is a distillation apparatus, the sensor can output the measured value PV (Process Variable) obtained by measuring the temperature at various locations of the distillation apparatus (e.g., top of the column, center of the column, bottom of the column, etc.) and the flow rate in various paths. The state data representing the state of the device 10 can include this measured value PV. In addition, the state data can include the operating quantity MV (Manipulated Variable) representing the opening and closing degree of the valve that is the controlled object 15. In addition to the operating data representing the operating state as a result of such control of the controlled object 15, the state data can also include consumption data representing the energy of the device 10, the consumption of raw materials, and external interference environment data representing physical quantities that can act as external disturbances to the control of the controlled object 15.

[0039] Distillation units are among the most widely used equipment in petroleum / chemical processes, characterized by strong interference between the top and bottom of the column, long time constants, and nonlinear operation. When using PID (Proportional Integral Differential) or similar methods to control the opening and closing of valves in such distillation units that require reflux flow control, it is difficult to achieve improved controllability. Furthermore, when operators manually operate these valves for multiple purposes such as quality assurance, energy saving, GHG (Green House Gas) reduction, and increased yield, the degree of valve opening and closing control largely depends on the operator's experience and intuition.

[0040] Therefore, when controlling the opening and closing of such a valve, it is considered to use a machine learning model generated through machine learning. For example, the control device 100 according to this embodiment can set the output of such a machine learning model as the correction target described later.

[0041] Simulator 20 simulates the operation of device 10. For example, simulator 20 may be designed based on the design information of device 10 and execute actions that simulate the operation of device 10. Simulator 20 acquires signals simulating the operational quantities for controlled object 15, causing environmental changes, and outputs simulated data simulating the state of device 10 (e.g., sensor predictions). As an example, simulator 20 may consist of a predictive model that predicts the state of a distillation unit and a plant control simulator. The predictive model can use modeling techniques employing deep learning for time series data to predict reactor state changes based on accumulated process data. Additionally, the plant control simulator can virtually simulate PID control for controlled object 15, deriving the operational quantity MV from the difference between the target value SV and the control quantity CV. That is, in addition to state predictions, simulator 20 can simulate the actions of device 10 itself.

[0042] The database 30 stores status data (also known as "actual data" or "historical data") obtained directly or indirectly from sensors installed on the device 10 when the device 10 is actually operated.

[0043] The control device 100 can be a PC (personal computer), tablet computer, smartphone, workstation, server computer, or general-purpose computer, or it can be a computer system composed of multiple connected computers. Such a computer system is also a computer in a broad sense. Furthermore, the control device 100 can be installed within a computer through one or more executable virtual computer environments. Alternatively, the control device 100 can be a dedicated computer designed for controlling the object 15, or it can be dedicated hardware implemented using dedicated circuitry. Additionally, when connected to the Internet, the control device 100 can be implemented through cloud computing.

[0044] The control device 100 controls the controlled object 15 using an operation model that performs machine learning with simulated data. At this time, the control device 100 according to this embodiment acquires an index for monitoring the difference (deviation) between the actions of the simulator 20 and the actual operation of the device 10, and corrects the output of the operation model based on this index. The control device 100 includes a model output acquisition unit 110, an index acquisition unit 120, a correction condition setting unit 130, a correction unit 140, and a control unit 150.

[0045] The model output acquisition unit 110 acquires the operation model output corresponding to the state data representing the state of the device 10 input to the operation model. This operation model is obtained through machine learning by using simulation data from the simulator 20 that simulates the operation of the device 10, outputting actions corresponding to the state of the device 10. For example, the model output acquisition unit 110 can acquire this operation model output from the operation model management device 200 that manages the operation models via a network. The operation model management device 200 will be described in detail below. The model output acquisition unit 110 supplies the acquired operation model output to the correction unit 140.

[0046] The indicator acquisition unit 120 acquires an indicator calculated using actual data from the device 10, used to monitor the difference between the actions of the simulator 20 and the actual operation of the device 10. For example, the indicator acquisition unit 120 can acquire this indicator via a network from an indicator calculation device 300 capable of calculating the indicator. The indicator calculation device 300 will be described in detail below. The indicator acquisition unit 120 supplies the acquired indicator to the correction unit 140.

[0047] The calibration condition setting unit 130 sets the conditions for calibrating the output of the operation model. For example, the calibration condition setting unit 130 can obtain such conditions through user input. The calibration condition setting unit 130 supplies the set conditions to the calibration unit 140.

[0048] The calibration unit 140 calibrates the operation model output acquired by the model output acquisition unit 110 based on the indicators acquired by the indicator acquisition unit 120. At this time, the calibration unit 140 calibrates the operation model output according to the conditions set by the calibration condition setting unit 130. The calibration unit 140 then supplies the calibrated operation model output to the control unit 150.

[0049] The control unit 150 outputs an operational quantity for controlling the controlled object 15 installed on the device 10 based on the corrected operating model output. For example, if the controlled object 15 is a valve, the control unit 150 can output the valve's opening and closing degree. However, it is not limited to this. The control unit 150 can also output various control inputs that can change the control quantity. That is, the term "operating quantity" as used in this specification can be broadly interpreted to include not only the so-called MV but also the target value SV and other control parameters (such as PID parameters).

[0050] Figure 2This is an example of a block diagram showing the operation model management device 200. Like the control device 100, the operation model management device 200 can be a computer or a computer system composed of multiple connected computers. Furthermore, the operation model management device 200 can be installed within a computer through one or more executable virtual computer environments. Alternatively, the operation model management device 200 can be a dedicated computer designed for the management of operation models, or dedicated hardware implemented using dedicated circuitry. Additionally, when connected to the Internet, the operation model management device 200 can be implemented via cloud computing.

[0051] The operation model management device 200 manages the operation model, which is a machine learning operation model that uses simulation data from the simulator 20 of the simulation device 10 to output actions corresponding to the state of the device 10. The operation model management device 200 includes an operation end / observation point setting unit 210, a target setting unit 220, an operation model learning unit 230, and an operation model storage unit 240.

[0052] The operator / observation point setting unit 210 sets information about the operator to be operated and information about the observation points used for learning. For example, the operator / observation point setting unit 210 can obtain the label name of the operator and the label name of the observation points through user input. At this time, the user can select a label name from a pre-prepared label list or directly input a label name. The operator / observation point setting unit 210 supplies the set information to the operation model learning unit 230.

[0053] The target setting unit 220 sets a target value and a reward function. For example, the target setting unit 220 can obtain the target value SV of the control quantity CV and a reward function used to quantify whether the selected action is good or bad through user input. As an example, the reward function may at least include a component representing the difference between the target value and the control quantity, such as a constant × |SV-CV| or a constant × (SV-CV)^2. That is, the reward function can be set such that the closer the control quantity is to the target value, the higher the reward value as the calculation result. The target setting unit 220 supplies the set information to the operation model learning unit 230.

[0054] The operation model learning unit 230 generates an operation model through machine learning. For example, the operation model learning unit 230 can obtain setting information from the operation terminal / observation point setting unit 210 and the target setting unit 220. Additionally, the operation model learning unit 230 can obtain simulation data from the simulator 20. Furthermore, the operation model learning unit 230 can generate an operation model by performing machine learning according to the setting information and using the simulation data to output actions corresponding to the state of the device 10. Here, the generated operation model can be a machine learning model that has undergone reinforcement learning by outputting actions that increase the reward value defined by the reward function. This reinforcement learning will be described in detail later.

[0055] The operation model storage unit 240 stores operation models. For example, the operation model storage unit 240 stores operation models generated by the operation model learning unit 230. Furthermore, in the above description, as an example, the operation model storage unit 240 stores operation models generated internally in the operation model management device 200, but it is not limited to this. The operation model storage unit 240 may also store operation models generated externally to the operation model management device 200.

[0056] Figure 3 This is an example of a data table that an operation model can have. The data table can consist of a combination of the state and actions of device 10, and the weights for that combination. This figure shows, as an example, a case where the label name of the operation terminal is set to "V001" and the label names of the observation points are set to "TI001", "TI002", "TI003", "FI001", and "FI002" in the operation terminal / observation point setting unit 210. In this case, the state of device 10 can be represented as state data S = (TI001, TI002, TI003, FI001, FI002, V001).

[0057] Additionally, an action can be represented as a change in the operation quantity MV at the operating end. For example, if the operation quantity MV (i.e., the value of label "V001") at time a is 70, and the operation quantity MV is 71 at time b immediately following time a, this indicates that the operation quantity MV changed from state 70 to state 71 from time a to time b. Here, the operation change ΔMV = 1 is achieved by subtracting the operation quantity MV = 70 at time a from the operation quantity MV = 71 at time b. In this case, the action at time a can be represented as action data A = 1.

[0058] Furthermore, the weights can be represented as weight data W calculated based on the reward value specified by the reward function set in the target setting unit 220. Additionally, the weight data W can initially be set to an initial value (e.g., 1), and then sequentially overwritten as learning progresses.

[0059] Thus, in the data table of the operational model, multiple sample data can be stored, consisting of combinations of state data S = (TI001, TI002, TI003, FI001, FI002, V001) and action data A = ΔMV, as well as weight data W for the combinations.

[0060] As an example, looking at the first row of this diagram, taking an action of A=1 (e.g., 1% rotation control in the direction of opening the valve, i.e., counterclockwise) with the state S=(-2.47803, -2.48413, -0.07324, 29.71191, 24.2511, 70) represents a weight of W=144.1484. The next action is determined using an operational model with this data table.

[0061] Figure 4 This diagram illustrates an example of an action table used to determine actions based on an operational model. The action table can consist of state data S and action data A that can be taken. In this diagram, as an example, it shows the following situation: the state of device 10 is S = (0.1, 0.2, 0.4, 0.3, 0.8, 0.2), and the available action options are A = [-3, -1, 0, 1, 3], which are equivalent to [3% off, 1% off, no change, 1% on, 3% on]. For example, by inputting the action table shown in this diagram into a system with… Figure 3 The operational model shown in the data table determines the next action. The process for generating this operational model is explained in detail using a workflow.

[0062] Figure 5 Here is an example of the process for generating an operational model based on reinforcement learning. In step S500, the operational model management device 200 initializes the operational model. For example, the operational model learning unit 230 creates an empty data table in the operational model storage unit 240. Furthermore, the operational model learning unit 230 reads the label names of the operational terminals and observation points set in the operational terminal / observation point setting unit 210, and defines the data table columns using these label names, actions, and weights. Additionally, the operational model learning unit 230 acquires multiple sample data as initial learning samples and appends them to each row of the data table. At this time, the operational model learning unit 230 can set the weight data W to initial values ​​for the initial learning samples.

[0063] In step S502, the operation model management device 200 acquires simulation data. For example, the operation model learning unit 230 acquires simulation data of the state of the simulation device 10 from the simulator 20, which serves as the learning environment, via a network.

[0064] In step S504, the operation model management device 200 determines an action. For example, the operation model learning unit 230 may determine the action randomly. However, it is not limited to this. The operation model learning unit 230 may utilize known AI algorithms such as FKDPP when determining the action. When using this kernel method, the operation model learning unit 230 generates a vector of state data S based on the sensor's predicted values ​​obtained from the simulated data. Next, the operation model learning unit 230 combines the state data S with all possible action data A to generate, for example... Figure 4 The action table is shown. Furthermore, the operation model learning unit 230 inputs the action table into the operation model. Correspondingly, the operation model performs kernel calculations between each row of the action table and each row of the data table (excluding the weight column), calculating the distance to each sample data point. Moreover, the operation model sequentially adds the values ​​obtained by multiplying the distances calculated for each sample data point by the values ​​of their respective weight columns to calculate the reward value (expected reward value) for each action. The operation model selects the action with the highest calculated reward value. The operation model learning unit 230 can, for example, use the updated operation model to select the action with the highest reward value to determine the action. During learning, the operation model learning unit 230 can determine the action by appropriately selecting either randomly or using the operation model.

[0065] In step S506, the operation model management device 200 instructs the simulator 20 to perform an operation. For example, the operation model learning unit 230 provides the simulator 20 with the latest operation quantity of the operation terminal of the learning environment plus the new operation quantity obtained from the action determined in step S504. As a result, the state of the learning environment changes.

[0066] In step S508, the operation model management device 200 acquires simulation data. For example, similar to step S502, the operation model learning unit 230 acquires simulation data of the state of the simulation device 10 from the simulator 20. That is, the operation model learning unit 230 acquires the state of the learning environment after the operation amount provided for the decision-based action has changed accordingly.

[0067] In step S510, the operation model management device 200 calculates the reward value. For example, the operation model learning unit 230 calculates the reward value using the reward function set in the target setting unit 220.

[0068] In step S512, the operation model management device 200 determines whether the number of steps has been exceeded. For example, the operation model learning unit 230 determines whether the processing of acquiring the state corresponding to the action decision has exceeded a specified number of steps. Furthermore, this number of steps can be specified in advance by the user or based on the learning period (e.g., 10 days). If it is determined that the processing has not exceeded the number of steps (in the case of No), the operation model management device 200 returns the processing to step S504, allowing the process to continue. The operation model management device 200 executes the acquisition processing of the state corresponding to this action decision for the specified number of steps.

[0069] In step S512, if it is determined that the above processing has exceeded the number of steps (if Yes), the operation model management device 200 causes the processing to proceed to step S514. In step S514, the operation model management device 200 updates the operation model. For example, in addition to updating the operation model... Figure 3 In addition to overwriting the values ​​in the weight column of the data table shown, the operation model learning unit 230 appends any new sample data that has not been saved up to a new row of the data table.

[0070] In step S516, the operation model management device 200 determines whether the number of repetitions has been exceeded. For example, the operation model learning unit 230 determines whether the operation model update process has exceeded the specified number of repetitions. Furthermore, this number of repetitions can be specified in advance by the user or determined based on the appropriateness of the operation model. If it is determined that the above process has not exceeded the number of repetitions (in the case of No), the operation model management device 200 returns the process to step S502, allowing the process to continue.

[0071] In step S516, if it is determined that the above process has exceeded the number of repetitions (if Yes), the operation model management device 200 terminates the process. The operation model management device 200 performs reinforcement learning, for example, by outputting actions that increase the reward value specified by the reward function, thereby generating an operation model that outputs actions corresponding to the state of the device 10.

[0072] The resulting operation model is learned from the actions of simulator 20. Therefore, even when the actions of simulator 20 differ from the actual operation of device 10, the operation model outputs actions based on the actions of simulator 20. As a result, it is possible to provide the controlled object 15 with operation quantities unsuitable for the actual operation of device 10. In this case, compared to manual operation by an operator, various problems may arise, such as reduced operational efficiency, deterioration of product quality, and increased risk of dangerous situations. Therefore, the operation model monitors the actions of simulator 20 as a premise, comparing them with the actual operation of device 10, and it is important to reflect the monitoring results to the control object 15 when discrepancies arise.

[0073] Figure 6 This is an example of a block diagram showing the index calculation device 300. Similar to the control device 100, the index calculation device 300 can be a computer or a computer system composed of multiple connected computers. Furthermore, the index calculation device 300 can be installed within a computer using one or more executable virtual computer environments. Alternatively, the index calculation device 300 can be a dedicated computer designed for index calculation or dedicated hardware implemented using dedicated circuitry. Additionally, when connected to the Internet, the index calculation device 300 can be implemented via cloud computing.

[0074] The indicator calculation device 300 calculates an indicator that measures the difference between the actions of the monitoring simulator 20 and the actual operation of the device 10, using actual data obtained during actual operation of the device 10. Here, as an example, the indicator calculation device 300 calculates this indicator using a monitoring model, which is a machine learning model. The indicator calculation device 300 includes a learning setting unit 310, a monitoring model learning unit 320, and a monitoring model storage unit 330.

[0075] The learning setting unit 310 sets information defining the state data used for learning, the hyperparameters of the learning algorithm, and information about the teacher signal. For example, the learning setting unit 310 can obtain the tag name of the sensor used for learning through user input as information defining the state data used for learning. At this time, the user can select a tag name from a pre-prepared tag list or directly input a tag name.

[0076] Furthermore, for example, when using a neural network model as a monitoring model, the learning setting unit 310 can obtain the neural network's structure, learning rate, optimization algorithm, number of iterations, and random number seed as hyperparameters of the learning algorithm via user input. Moreover, here, as an example, a case where a neural network is used as a learning algorithm for a monitoring model is shown. However, it is not limited to this. Other algorithms, such as decision trees, can also be used when learning the monitoring model.

[0077] Furthermore, the learning setting unit 310 can acquire information about any signal that the user expects the device 10 to revert to, via user input, and use it as teacher signal information. At this time, the user can input the value of the teacher signal itself, or a mathematical formula used to derive the value of the teacher signal. As an example, this teacher signal can be a future (e.g., 5 steps ahead) predicted value of a sensor contained in the state data representing the state of the device 10, or a future predicted value of a sensor capable of prediction based on the state data, i.e., a soft sensor. Alternatively, or based on this, the teacher signal can be the quality of a product or semi-finished product manufactured in the device 10 that can be predicted (numerized) based on the state data. Thus, the teacher signal only needs to be predictable based on the state data and can be any signal that the user expects the device 10 to revert to. The learning setting unit 310 supplies the setting information to the monitoring model learning unit 320.

[0078] The monitoring model learning unit 320 generates a monitoring model through machine learning. For example, the monitoring model learning unit 320 can obtain setting information from the learning setting unit 310. Additionally, the monitoring model learning unit 320 can obtain actual data obtained during actual operation of the device 10 from the database 30. Furthermore, the monitoring model learning unit 320 can generate a monitoring model by performing machine learning according to the setting information and using the actual data to output indicators corresponding to the state of the device 10. Here, the generated monitoring model can be a machine learning model that has undergone teacher-guided learning in a way that reduces the difference between the output of the monitoring model and the teacher signal. This teacher-guided learning will be described in detail later.

[0079] The monitoring model storage unit 330 stores monitoring models. For example, the monitoring model storage unit 330 stores monitoring models generated by the monitoring model learning unit 320. Furthermore, in the above description, as an example, the monitoring model storage unit 330 stores monitoring models generated internally in the index calculation device 300, but it is not limited to this. The monitoring model storage unit 330 may also store monitoring models generated externally to the index calculation device 300.

[0080] Furthermore, in this embodiment, only one indicator is used as an example for monitoring the actions of the simulator 20, but it is not limited to this. Multiple indicators can be used as such an indicator. In this case, the monitoring model storage unit 330 can store multiple monitoring models that output different indicators respectively.

[0081] Figure 7 This figure illustrates an example of teacher-trained learning data used to generate a monitoring model. As an example, in the learning setting unit 310, the tag names of the sensors used for learning are set as "TI001", "TI002", "TI003", "FI001", "FI002", and "V001". Actual data obtained from the database 30, i.e., the values ​​of the status data obtained when the device 10 is actually operated, are input into the aforementioned columns in a time sequence. Additionally, the values ​​of the teacher signals set in the learning setting unit 310 are input into the "Teacher Signal" column in a time sequence. The monitoring model learning unit 320 can, for example, use this learning data to perform teacher-trained learning and generate a monitoring model.

[0082] More specifically, the monitoring model learning unit 320 initially sets up the monitoring model based on the neural network structure, learning rate, optimization algorithm, number of iterations, and random number seed set in the learning setting unit 310. At this time, the monitoring model learning unit 320 can set the state data as S = (TI001, TI002, TI003, FI001, FI002, V001), set w as the weights of the neural network, and construct a neural network model denoted as F(S, w). Next, the monitoring model learning unit 320 can initialize the weights w using random numbers. Next, the monitoring model learning unit 320 can, for example, divide the learning data shown in this figure into columns of state data and columns of teacher signals, and generate feature quantities using the columns of state data. Next, the monitoring model learning unit 320 can calculate F(S, w) for each row of the state data S. Next, the monitoring model learning unit 320 can compare the output y of F(S, w) with the teacher signal y′, and update the weights w of the neural network using an optimization algorithm and a learning rate to reduce their difference. The monitoring model learning unit 320 can repeatedly perform this weight w update process a number of times specified by the number of iterations.

[0083] The control device 100 of this embodiment acquires the output y of a monitoring model generated in such a way as an indicator for the operation of the simulator 20, which is set as a monitoring device for the operation model, and corrects the output of the operation model. This will be explained in detail using a flowchart.

[0084] Figure 8This illustrates an example of the control flow of the control device 100 according to this embodiment. In step S802, the control device 100 acquires status data. For example, the control device 100 can acquire status data S = (TI001, TI002, TI003, FI001, FI002, V001) representing the status of the device 10 in real time from the device 10 via a network.

[0085] In step S804, the control device 100 acquires the operation model output. For example, the model output acquisition unit 110 can input the state data S acquired in step S802 into the operation model stored in the operation model management device 200. Accordingly, the operation model can use the FKDPP algorithm described above to select the action with the highest reward value and output it. The model output acquisition unit 110 can acquire the action output from the operation model as the operation model output x. For example, the model output acquisition unit 110 can acquire the operation model output x in which the state data S representing the state of the device 10 is input into the operation model that has performed machine learning by using simulation data from the simulator 20 that simulates the operation of the simulation device 10 to output actions corresponding to the state of the device 10. The model output acquisition unit 110 supplies the acquired operation model output x to the correction unit 140.

[0086] In step S806, the control device 100 acquires an indicator. For example, the indicator acquisition unit 120 can input the state data S acquired in step S802 into a monitoring model stored in the indicator calculation device 300. Correspondingly, the monitoring model can calculate F(S, w) and output y as the calculation result. The indicator acquisition unit 120 can acquire the monitoring model output y as an indicator. For example, the indicator acquisition unit 120 can acquire the monitoring model output y, which is output in response to the state data S input into a monitoring model that has undergone machine learning using actual data to output an indicator corresponding to the state of the device, and use it as an indicator.

[0087] Furthermore, as described above, machine learning is performed on the monitoring model in a manner that reduces the difference between the monitoring model output y and the teacher signal y′. Therefore, when the teacher signal y′ is set to the future predicted value of the sensor contained in the state data S, or the future predicted value of the soft sensor that can be inferred from the state data S, the index acquisition unit 120 acquires the future predicted value of the sensor contained in the state data S, or the future predicted value of the sensor that can be inferred from the state data S, as an index. Alternatively, or based on this, when the teacher signal y′ is set to the quality of the product or semi-finished product manufactured in the device 10 that can be predicted (numerized) based on the state data S, the index acquisition unit 120 acquires a value that numerizes the quality of the product or semi-finished product manufactured in the device 10 as an index.

[0088] The indicator acquisition unit 120 can acquire, for example, an indicator calculated using actual data from the device 10, used to monitor the difference between the actions of the simulator 20 and the actual operation of the device 10. The indicator acquisition unit 120 supplies the acquired indicator, here the monitoring model output y, to the correction unit 140.

[0089] In step S808, the control device 100 corrects the operation model output. For example, the correction unit 140 can correct the operation model output x obtained in step S804 based on the index obtained in step S806, here the monitoring model output y.

[0090] Here, the conditions set in the calibration condition setting unit 130 are assumed to include the range of permissible indicators. As an example, the calibration condition setting unit 130 sets an upper alarm limit and a lower alarm limit for the monitoring model output y. Furthermore, the condition formula for performing calibration is set as follows. Moreover, this condition formula is an example of a situation where a relationship is known in advance (theoretically or empirically) such that if x increases, y decreases, and if x decreases, y increases.

[0091] [Mathematical Expression 1]

[0092] Function(x, y){

[0093] If y > the alarm limit of y, then x = x + 1

[0094] Else If y < the lower alarm limit of y, then x = x-1

[0095] Else x = x

[0096] Return x

[0097] }

[0098] In this case, the correction unit 140 can correct the operation model output x according to the conditional expression. That is, when the monitoring model output y exceeds the alarm upper limit, the correction unit 140 can increment the operation model output x by 1. For example, when the operation model output x is "-3", the correction unit 140 can correct the operation model output x from "-3" to "-2", that is, change the action of closing the valve by 3% to closing the valve by 2%. Similarly, when the monitoring model output y is below the alarm lower limit, the correction unit 140 can decrement the operation model output x by 1. For example, when the operation model output x is "-3", the correction unit 140 can correct the operation model output x from "-3" to "-4", that is, change the action of closing the valve by 3% to closing the valve by 4%. In this way, when the indicator is outside the range, the correction unit 140 can correct the operation model output x in a way that brings the indicator into the range. The correction unit 140 supplies the corrected operation model output x to the control unit 150.

[0099] In step S810, the control device 100 outputs an operation quantity. For example, the control unit 150 can output an operation quantity for controlling the control object 15 installed on the device 10 based on the operation model output x corrected in step S808. As an example, the control unit 150 can provide a new operation quantity MV obtained by adding the corrected operation model output x to the most recent operation quantity MV of the operation terminal to the control object 15, which is the operation terminal.

[0100] When generating operation models through machine learning, dangerous operations may occur during the learning process if the actual equipment is operated during the learning phase. Furthermore, when using reinforcement learning to generate operation models, the learning process may not always be perfect if only historical data is used. Therefore, a technique for learning operation models based on simulation has been investigated. However, it is possible that the actions of the simulator 20, designed based on the design information of equipment 10, may differ from the actual operation of equipment 10. In this case, the output of the simulation-based operation model may not be suitable for the actual operation of equipment 10. When this occurs, compared to manual operation of equipment 10 by an operator, problems arise such as decreased operating efficiency of equipment 10, deterioration of product quality, and an increased risk of dangerous situations.

[0101] In contrast, the control device 100 of this embodiment, when controlling the controlled object 15 using an operation model learned from simulation, acquires an index for monitoring the difference between the actions of the simulator 20 and the actual operation of the device 10, and corrects the output of the operation model based on this index. Therefore, according to the control device 100 of this embodiment, even in situations where the actions of the simulator 20 differ from the actual operation of the device 10, it is possible to avoid directly using the output of the operation model learned from simulation to control the controlled object 15.

[0102] Furthermore, the control device 100 according to this embodiment can use the output of a monitoring model that has undergone machine learning as an indicator. Thus, according to the control device 100 according to this embodiment, the controlled object 15 can be controlled while taking into account the gap between simulation and reality using two machine learning models.

[0103] Furthermore, the control device 100 according to this embodiment can use a machine learning model that has undergone teacher-guided learning as a monitoring model. Therefore, according to the control device 100 according to this embodiment, the output of the monitoring model can reflect the operator's insights and past operational performance.

[0104] Furthermore, the control device 100 according to this embodiment can use a machine learning model that has undergone reinforcement learning as an operation model. Therefore, according to the control device 100 according to this embodiment, even when it is difficult to define an evaluation metric for a single action or when it is required to adapt to an unknown environment, the controlled object 15 can be flexibly controlled.

[0105] Furthermore, the control device 100 according to this embodiment can set future predicted values ​​of sensors, product quality, etc., as indicators. In this way, according to the control device 100 according to this embodiment, the operation of the simulator 20 can be monitored using various indicators that are expected to be returned to regarding the operation of the device 10.

[0106] Furthermore, the control device 100 according to this embodiment corrects the output of the operation model by bringing the indicator closer to the set range when the indicator is outside the set range. Therefore, according to the control device 100 according to this embodiment, the output of the operation model can be corrected to bridge the gap between simulation and reality.

[0107] Furthermore, in the above description, the control device 100 and the operation model management device 200 were shown as separate devices as an example, but this is not a limitation. Some or all of the functional units of the operation model management device 200 may also be integrated with the control device 100. That is, the control device 100 may also have an operation model storage unit 240. In addition, the control device 100 may also have an operation model learning unit 230. In addition, the control device 100 may also have a target setting unit 220. In addition, the control device 100 may also have an operation terminal / observation point setting unit 210.

[0108] Therefore, the control device 100 can realize the functions of controlling the controlled object 15 and storing the operation model using a single device. Thus, when acquiring the operation model output, there is no need to exchange data between devices, allowing for safe and secure control of the controlled object 15. Furthermore, since the control device 100 can realize the functions of controlling the controlled object 15 and generating the operation model using a single device, there is no need to move the generated operation model between devices, reducing communication costs.

[0109] Similarly, in the above description, the control device 100 and the index calculation device 300 were shown as separate devices as an example, but this is not a limitation. Some or all of the functional units of the index calculation device 300 may also be integrated with the control device 100. That is, the control device 100 may also have a monitoring model storage unit 330. In addition, the control device 100 may also have a monitoring model learning unit 320. Furthermore, the control device 100 may also have a learning setting unit 310.

[0110] Therefore, the control device 100 can perform the functions of controlling the controlled object 15 and storing the monitoring model using a single device. Thus, when acquiring indicators, there is no need to exchange data between devices, allowing for safe and secure control of the controlled object 15. Furthermore, since the control device 100 can perform both the functions of controlling the controlled object 15 and generating the monitoring model using a single device, there is no need to move the generated monitoring model between devices, reducing communication costs.

[0111] Figure 9 This is an example of a block diagram showing a variation of the control device 100 according to this embodiment. Figure 9 In the middle, for those with Figure 1Components with the same function and structure are labeled with the same reference numerals, and descriptions are omitted except for the differences described below. In the above embodiment, as an example, a case is shown where the control device 100 acquires the output of the monitoring model as an indicator and corrects the output of the operation model based on this monitoring model output. However, in this modified example, the control device 100 acquires the calculated value y calculated by the simulator 20. M As an indicator, based on this calculated value y M The output of the operation model is then corrected. The control device 100 involved in this modification also has an actual measurement value extraction unit 910.

[0112] In this modified example, the model output acquisition unit 110 functions in the same way as in the embodiment described above. For example, if the state data at time t is set as S(t), the model output acquisition unit 110 can acquire the operational model output x(t) that is output corresponding to the input of the state data S(t) to the operational model. The model output acquisition unit 110 supplies the acquired operational model output x(t) to the correction unit 140.

[0113] The actual measurement value extraction unit 910 extracts the calculated value y from the state data S. M The actual measurement value y of the comparison object is used. For example, if the actual measurement value at time t is set as y(t), the actual measurement value extraction unit 910 can extract the value of the object sensor contained in the state data S(t) as the actual measurement value y(t). Here, the object sensor can be one or more sensors contained in the state data S(t). The actual measurement value extraction unit 910 supplies the extracted actual measurement value y(t) to the correction unit 140.

[0114] In this variant, the index acquisition unit 120 acquires the calculated value y calculated by the simulator 20. M As an indicator. For example, if we set the state data of the time preceding time t-1 as S(t-1), the corrected operational model output at time t-1 as u(t-1), and the calculated value at time t as y M If (t), then the indicator acquisition unit 120 can acquire the state data S(t-1) and the corrected operation model output u(t-1), and input them to the simulator 20. Correspondingly, the simulator 20 can use the state data S(t-1) and the corrected operation model output u(t-1) to simulate the value y of the object sensor at time t immediately following time t-1. M (t) is used for calculation. The index acquisition unit 120 can acquire, for example, the calculated value y obtained by the simulator 20. M (t) is used as an indicator. The indicator acquisition unit 120 will acquire the calculated value y. M (t) is supplied to the calibration unit 140.

[0115] In this modified example, the correction condition setting unit 130 can set the output x(t) of the operating model, the actual measured value y(t), and the calculated value y. M (t) is a correction function calculated for the corrected operating model output u(t). As an example, the correction condition setting unit 130 can set such a correction function with P as the first coefficient as follows.

[0116]

Mathematical Formula 2

[0117] u(t)=x(t)+P*(y(t)-y M (t))

[0118] Instead, the correction condition setting unit 130 can set the following correction function with I as the second coefficient.

[0119]

Mathematical Expression 3

[0120]

[0121] Instead, the correction condition setting unit 130 can set the following correction function with D as the third coefficient.

[0122]

Mathematical Expression 4

[0123]

[0124] In this modified example, the correction unit 140 corrects the operation model output using a correction function set in the correction condition setting unit 130. For example, the correction unit 140 can correct the operation model output using equation (Mathematical Formula 2). That is, the correction unit 140 can add the actual measured value y(t) and the calculated value y(t) to the operation model output x(t). M The corrected operational model output u(t) is calculated by multiplying the difference between (t) and the coefficient.

[0125] Instead, the calibration unit 140 can use equation (Mathematical Formula 3) to correct the output of the operation model. That is, the calibration unit 140 can add the actual measured value y(t) and the calculated value y(t) to the operation model output x(t). M The value obtained by multiplying the difference (t) by a coefficient, and the difference between the actual measured value y(t) and the calculated value y M The corrected operational model output u(t) is calculated by multiplying the integral of the difference between u(t) and the coefficient.

[0126] Instead, the calibration unit 140 can use equation (Mathematical Formula 4) to correct the output of the operation model. That is, the calibration unit 140 can add the actual measured value y(t) and the calculated value y(t) to the output x(t) of the operation model.M The value obtained by multiplying the difference (t) by a coefficient, the actual measured value y(t) and the calculated value y M The value obtained by integrating the difference between y(t) and the coefficient, and the actual measured value y(t) and the calculated value y(t) M The value obtained by multiplying the deviation of (t) by the coefficient is used to calculate the corrected operating model output u(t).

[0127] In this modification, the correction unit 140 can correct the operation model output using a correction function that includes, for example, the difference between the actual measured value and the calculated value represented by state data in the correction term. Thus, the control device 100 of this modification uses the calculated value calculated by the simulator 20 as an indicator for monitoring the difference between the operation of the simulator 20 and the actual operation of the device 10. Therefore, according to the control device 100 of this modification, there is no need to prepare a separate model for calculating indicators such as the monitoring model, simplifying the process. Furthermore, the control device 100 of this modification corrects the operation model output using a correction function that includes the difference between the actual measured value and the calculated value represented by state data in the correction term. Therefore, according to the control device 100 of this modification, when there is a difference between the actual measured value and the calculated value, the operation model output can be corrected according to the magnitude of the difference.

[0128] Several possible implementation methods have been illustrated so far. However, the above implementation methods can be modified or applied in various ways. For example, in the above description, as an example, a case was shown where historical data was used to pre-learn the monitoring model used in the application during learning. However, it is conceivable that the actual data used in the application of the monitoring model is not similar to the historical data used for learning the monitoring model. In this case, the monitoring model can be relearned based on the low similarity between the data. For example, the indicator calculation device 300 can also obtain actual data from the database 30 during application. Moreover, the indicator calculation device 300 can calculate the similarity between the historical data used during learning and the actual data obtained during application, and if the similarity does not meet a predetermined range, the monitoring model can be relearned using the actual data obtained during application.

[0129] Furthermore, the above description illustrates, as an example, a case where the user manually sets correction conditions to calibrate the operation model output. However, machine learning models can also be used when setting correction conditions. For example, the control device 100 can use a machine learning model that takes the operation model output and the monitoring model output as inputs and returns the optimal calibrated operation model output to set the correction conditions.

[0130] Various embodiments of the present invention can be described with reference to flowchart and block diagrams, where a block can represent (1) a stage of a process for performing an operation, or (2) a part of a device that performs an operation. Specific stages and parts can be installed using dedicated circuits, programmable circuits supplied together with computer-readable instructions stored on a computer-readable medium, and / or processors supplied together with computer-readable instructions stored on a computer-readable medium. Dedicated circuits can include digital and / or analog hardware circuits, and can also include integrated circuits (ICs) and / or discrete circuits. Programmable circuits can include reconfigurable hardware circuits such as logic AND, logic OR, logic XOR, logic NAND, logic NOR, and other logic operations, trigger circuits, registers, field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.

[0131] A computer-readable medium can include any tangible device capable of storing instructions that can be executed using appropriate means, resulting in a product having instructions stored therein containing executable instructions for the purpose of creating means for performing operations specified in a flowchart or block diagram. Examples of computer-readable media include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable media include floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD), Blu-ray (RTM) discs, memory sticks, integrated circuit cards, etc.

[0132] Computer-readable instructions may include any combination of source code or object code written in one or more programming languages, including object-oriented programming languages ​​such as Smalltalk (registered trademark), JAVA (registered trademark), C++, and existing procedural programming languages ​​such as "C" or similar programming languages.

[0133] Computer-readable instructions can be provided to the processor or programmable circuitry of a general-purpose computer, special-purpose computer, or other programmable data processing device via a local area network (LAN), wide area network (WAN), or the Internet, and executed to create means for performing operations specified in a flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0134] Figure 10 Examples of computer 9900 that can embody all or part of the various embodiments of the present invention are shown. Programs installed in computer 9900 enable computer 9900 to function as an operation associated with or one or more parts of a device according to embodiments of the present invention, or to perform such operation or such parts, and / or to perform such processes or stages of processes according to embodiments of the present invention. Such programs may be executed by CPU 9912 to enable computer 9900 to perform specific operations associated with several or all of the modules in the flowcharts and block diagrams described in this specification.

[0135] The computer 9900 according to this embodiment includes a CPU 9912, RAM 9914, a graphics controller 9916, and a display device 9918, which are interconnected via a main controller 9910. Additionally, the computer 9900 includes input / output units such as a communication interface 9922, a hard disk drive 9924, a DVD-ROM drive 9926, and an IC card driver, which are connected to the main controller 9910 via an input / output controller 9920. Furthermore, the computer includes conventional input / output units such as a ROM 9930 and a keyboard 9942, which are connected to the input / output controller 9920 via an input / output chip 9940.

[0136] CPU 9912 executes actions according to the programs stored in ROM 9930 and RAM 9914, thereby controlling each unit. Graphics controller 9916 acquires image data provided to RAM 9914, generated by CPU 9912 in frame buffer or other locations, and displays the image data on display device 9918.

[0137] The communication interface 9922 communicates with other electronic devices via a network. The hard disk drive 9924 stores programs and data used by the CPU 9912 within the computer 9900. The DVD-ROM drive 9926 reads programs or data from the DVD-ROM 9901 and provides the programs or data to the hard disk drive 9924 via RAM 9914. The IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.

[0138] The ROM 9930 stores startup programs executed by the computer 9900 upon activation, and / or programs dependent on the hardware of the computer 9900. Additionally, the input / output chip 9940 connects various input / output units to the input / output controller 9920 via parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0139] The program is provided by a computer-readable medium such as a DVD-ROM 9901 or an IC card. The program is read from the computer-readable medium and installed on a hard disk drive 9924, RAM 9914, or ROM 9930, which are also examples of computer-readable media, and executed by the CPU 9912. The information processing described in the program is read by the computer 9900, enabling cooperation between the program and the aforementioned hardware resources of various types. An apparatus or method can be constituted by using the computer 9900 to perform information manipulation or processing.

[0140] For example, when communication is performed between the computer 9900 and external devices, the CPU 9912 can execute the communication program loaded in the RAM 9914 and issue communication processing instructions to the communication interface 9922 based on the processing described in the communication program. Under the control of the CPU 9912, the communication interface 9922 reads the transmission data stored in the transmission buffer processing area provided in the recording medium such as the RAM 9914, hard disk drive 9924, DVD-ROM 9901, or IC card, and sends the read transmission data to the network, or writes the received data received from the network to the receive buffer processing area provided on the recording medium, etc.

[0141] In addition, CPU 9912 can read all or a portion of files or databases stored on external recording media such as hard disk drive 9924, DVD-ROM drive 9926 (DVD-ROM 9901), IC card, etc., into RAM 9914, and perform various types of processing on the data in RAM 9914. CPU 9912 then writes the processed data back to the external recording media.

[0142] Various types of information, such as programs, data, tables, and databases, can be stored in recording media and processed. The CPU 9912 can perform various types of processing, including operations, information processing, conditional judgments, conditional branches, unconditional branches, and information retrieval / replacement, as specified by a program instruction sequence described at any location of this disclosure, on data read from RAM 9914, and write back the results to RAM 9914. Furthermore, the CPU 9912 can retrieve information from files, databases, etc., within the recording medium. For example, when multiple entries, each having an attribute value of a first attribute associated with a second attribute value, are stored in the recording medium, the CPU 9912 can retrieve from these multiple records an entry whose first attribute value matches a condition, read the attribute value of the second attribute stored within that entry, and obtain the attribute value of the second attribute associated with the first attribute that satisfies the pre-defined condition.

[0143] The programs or software modules described above can be stored on or near the computer 9900 on a computer-readable medium. Alternatively, a recording medium such as a hard disk or RAM provided to a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the program to the computer 9900 via the network.

[0144] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. Those skilled in the art will understand that various modifications or improvements can be made to the above embodiments. As clearly understood from the claims, such modifications or improvements can also be included within the technical scope of the present invention.

[0145] Regarding the execution order of actions, sequence, steps, and stages in the apparatus, system, program, and method shown in the claims, specification, and drawings, it should be noted that, unless explicitly stated as "earlier" or "earlier," and the output of a previous process is not used in a subsequent process, they can be implemented in any order. Even if terms such as "firstly," "nextly," etc., are used for convenience in describing the flow of actions in the claims, specification, and drawings, it does not imply that they must be performed in that order.

[0146] Explanation of the label

[0147] 1 Control System

[0148] 10 devices

[0149] 15 Controlled Objects

[0150] 20 simulator

[0151] 30 databases

[0152] 100 control device

[0153] 110 Model Output Acquisition Department

[0154] 120 Index Acquisition Department

[0155] 130 Calibration Condition Setting Section

[0156] 140 Correction Department

[0157] 150 Control Department

[0158] 200 Operation Model Management Device

[0159] 210 Operation Terminal / Observation Point Setting Unit

[0160] 220 Target Setting Department

[0161] 230 Operational Model Learning Department

[0162] 240 Operation Model Storage Unit

[0163] 300 Index Calculation Device

[0164] 310 Learning Setting Department

[0165] 320 Monitoring Model Learning Department

[0166] 330 Monitoring Model Storage Unit

[0167] 910 Actual Measurement Value Extraction Department

[0168] 9900 Computer

[0169] 9901 DVD-ROM

[0170] 9910 main controller

[0171] 9912 CPU

[0172] 9914 RAM

[0173] 9916 Graphics Controller

[0174] 9918 display device

[0175] 9920 Input / Output Controller

[0176] 9922 communication interface

[0177] 9924 Hard Drive

[0178] 9926 DVD drive

[0179] 9930ROM

[0180] 9940 Input / Output Chip

[0181] 9942 Keyboard

Claims

1. A control device, wherein The control device has: The model output acquisition unit acquires the operation model output corresponding to the state data representing the state of the device input to the operation model. The operation model is obtained by machine learning using simulation data from a simulator that simulates the operation of the device to output actions corresponding to the state of the device. The indicator acquisition unit acquires indicators calculated using actual data from the device for monitoring the difference between the simulator's actions and the device's actual operation; A calibration unit that calibrates the output of the operating model based on the stated indicators; and The control unit outputs an operational quantity for controlling the controlled object installed on the device, based on the corrected operation model.

2. The control device according to claim 1, wherein, The indicator acquisition unit acquires the monitoring model output as the indicator, and the monitoring model output is output in accordance with the monitoring model that has undergone machine learning by inputting the state data into the actual data to output the indicator corresponding to the state of the device.

3. The control device according to claim 2, wherein, The monitoring model is a machine learning model that performs teacher-led learning in a way that reduces the difference between the output of the monitoring model and the teacher signal.

4. The control device according to claim 3, wherein, The control device also includes a monitoring model learning unit, which generates the monitoring model through teacher-led learning.

5. The control device according to any one of claims 1 to 4, wherein, The operational model is a machine learning model that performs reinforcement learning by outputting actions that increase the reward value specified by the reward function.

6. The control device according to claim 5, wherein, The control device also includes an operation model learning unit, which generates the operation model through reinforcement learning.

7. The control device according to any one of claims 1 to 4, wherein, The indicator acquisition unit acquires the future predicted value of the sensor contained in the state data, or the future predicted value of the sensor that can be inferred from the state data, and uses it as the indicator.

8. The control device according to any one of claims 1 to 4, wherein, The indicator acquisition unit acquires a value that quantifies the quality of the product or semi-finished product manufactured in the equipment, and uses it as the indicator.

9. The control device according to any one of claims 1 to 4, wherein, The control device also includes a correction condition setting unit, which sets the conditions for correcting the output of the operation model.

10. The control device according to claim 9, wherein, The conditions include a permissible range for the indicators. When the indicator is outside the range, the correction unit corrects the output of the operation model in a way that brings the indicator toward the range.

11. The control device according to claim 1, wherein, The indicator acquisition unit acquires the calculated value obtained using the simulator, and uses it as the indicator.

12. The control device according to claim 11, wherein, The correction unit uses a correction function that includes the difference between the actual measured value represented by the state data and the calculated value in the correction term to correct the output of the operation model.

13. A control method executed by a computer, wherein, The control method comprises the following steps: The computer acquires and outputs operation model output corresponding to the state data representing the state of the device input into the operation model. The operation model is obtained by machine learning using simulation data from a simulator that simulates the operation of the device to output actions corresponding to the state of the device. The computer acquires an index calculated using actual data from the device to monitor the difference between the simulator's actions and the device's actual operation; The computer corrects the output of the operation model based on the indicators; as well as The computer outputs an operational quantity for controlling the controlled object set on the device, based on the corrected operation model.

14. A non-transitory computer-readable medium that records a control program executed by a computer, wherein, The control program causes the computer to function as a functional unit: The model output acquisition unit acquires the operation model output corresponding to the state data representing the state of the device input to the operation model. The operation model is obtained by machine learning using simulation data from a simulator that simulates the operation of the device to output actions corresponding to the state of the device. The indicator acquisition unit acquires indicators calculated using actual data from the device for monitoring the difference between the simulator's actions and the device's actual operation; A calibration unit that calibrates the output of the operating model based on the stated indicators; and The control unit outputs an operational quantity for controlling the controlled object installed on the device, based on the corrected operation model.

Images