Control device, control method, and control program

Abstract

To provide a control device, a control method and a control program.SOLUTION: A control device includes: a model output obtaining unit that obtains an operation model output that is outputted in response to an input of state data representing a state of a facility in an operation model having undergone machine learning using simulation data from a simulator that simulates an operation in the facility so as to output an action in accordance with the state of the facility; an indicator obtaining unit that obtains an indicator which is calculated using actual data from the facility and which is configured to monitor a difference between behavior of the simulator and actual operation in the facility; a correcting unit that corrects the operation model on the basis of the indicator; and a control unit that operates an operation quantity to control an object to be controlled and provided at the facility, in accordance with the corrected operation model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device, a control method, and a control program. [Background technology]

[0002] Patent Document 1 states that "the model 45 outputs recommended control parameters indicating the first type of control content recommended to increase the reward value in response to input of measurement data." Furthermore, Non-Patent Document 1 describes "Factorial Kernel Dynamic Policy Programming (FKDPP)." [Prior art document] [Patent documents] [Patent Document 1] JP 2021-086283 [Patent Document 2] JP 2020-027556 [Patent Document 3] JP 2019-020885 [Non-patent literature] [Non-Patent Document 1] "Yokogawa Electric and NAIST Reinforcement Learning for Chemical Plants," Nikkei Robotics, March 2019 issue Summary of the Invention

[0003] A first aspect of the present invention provides a control device. The control device may include a model output acquisition unit that acquires an operational model output in response to inputting status data indicating the status of equipment into an operational model that has been machine-learned to output behavior according to the status of the equipment using simulation data from a simulator that simulates operation of the equipment. The control device may include an index acquisition unit that acquires an index for monitoring a difference between the behavior of the simulator and the actual operation of the equipment, the index being calculated using actual data from the equipment. The control device may include a correction unit that corrects the operational model output based on the index. The control device may include a control unit that outputs a manipulated variable for controlling a controlled object provided in the equipment in response to the corrected operational model output.

[0004] The index acquisition unit may acquire, as the index, a monitoring model output that is output in response to inputting the status data into a monitoring model that has been machine-learned using the actual data to output the index according to the status of the equipment.

[0005] The monitoring model may be a machine learning model that has undergone supervised learning so that the difference between the output of the monitoring model and a teacher signal is small.

[0006] The control device may further include a surveillance model learning unit that generates the surveillance model by the supervised learning.

[0007] The operational model may be a machine learning model that has undergone reinforcement learning to output actions that increase a reward value determined by a reward function.

[0008] The control device may further include an operation model learning unit that generates the operation model by the reinforcement learning.

[0009] The index acquisition unit may acquire, as the index, a future predicted value of the sensor included in the status data or a future predicted value of the sensor that can be estimated from the status data.

[0010] The index acquisition unit may acquire, as the index, a value that quantifies the quality of a product or semi-finished product manufactured in the facility.

[0011] The control device may further include a correction condition setting unit that sets conditions for correcting the operation model output.

[0012] The condition may include an allowable range for the index. When the index is outside the allowable range, the correction unit may correct the operation model output so that the index is directed toward the allowable range.

[0013] The index acquisition unit may acquire, as the index, a calculated value calculated by the simulator.

[0014] The correction unit may correct the operational model output using a correction function that includes a difference between the calculated value and the actual measurement value indicated by the status data as a correction term.

[0015] A second aspect of the present invention provides a control method. The control method may be executed by a computer. The control method may include the computer acquiring an operational model output in response to inputting status data indicating a status of equipment into an operational model that has been trained by machine learning using simulation data from a simulator that simulates operation of the equipment to output behavior corresponding to the status of the equipment. The control method may include the computer acquiring an index for monitoring a difference between the behavior of the simulator and actual operation of the equipment, the index being calculated using actual data from the equipment. The control method may include the computer correcting the operational model output based on the index. The control method may include the computer outputting a manipulated variable for controlling a control target provided in the equipment in response to the corrected operational model output.

[0016] A third aspect of the present invention provides a control program. The control program may be executed by a computer. The control program may cause the computer to function as a model output acquisition unit that acquires an operational model output that is output in response to inputting status data indicating the status of equipment into an operational model that has been machine-learned using simulation data from a simulator that simulates operation of the equipment to output behavior corresponding to the status of the equipment. The control program may cause the computer to function as an index acquisition unit that acquires an index for monitoring a difference between the behavior of the simulator and actual operation of the equipment, the index being calculated using actual data from the equipment. The control program may cause the computer to function as a correction unit that corrects the operational model output based on the index. The control program may cause the computer to function as a control unit that outputs a manipulated variable for controlling a controlled object provided in the equipment in response to the corrected operational model output.

[0017] The above summary of the invention does not list all of the features of the present invention, and subcombinations of these features may also be inventions. [Brief explanation of the drawings]

[0018] [Figure 1] 1 shows an example of a block diagram of a control system 1 that may include a control device 100 according to the present embodiment. [Figure 2] 1 shows an example of a block diagram of an operation model management device 200. [Figure 3] 1 shows an example of a data table that an operational model may have. [Figure 4] 10 shows an example of an action table for determining actions using an operation model. [Figure 5] An example of a flow for generating an operation model using reinforcement learning is shown below. [Figure 6] 1 shows an example of a block diagram of an index calculation device 300. [Figure 7]1 shows an example of supervised training data used to generate a supervisory model. [Figure 8] 1 shows an example of a control flow by the control device 100 according to the present embodiment. [Figure 9] FIG. 10 shows an example of a block diagram of a control device 100 according to a modified example of the present embodiment. [Figure 10] 99 illustrates an example computer 9900 in which aspects of the present invention may be embodied, in whole or in part. DETAILED DESCRIPTION OF THE INVENTION

[0019] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0020] FIG. 1 shows an example of a block diagram of a control system 1 that may include a control device 100 according to this embodiment. Note that these blocks are functionally separated functional blocks and may not necessarily correspond to the actual device configuration. In other words, just because something is shown as one block in this diagram does not necessarily mean that it is composed of one device. Also, just because something is shown as separate blocks in this diagram does not necessarily mean that it is composed of separate devices. The same applies to the subsequent block diagrams.

[0021] The control system 1 may include a facility 10, a simulator 20, a database 30, a control device 100, an operation model management device 200, and an index calculation device 300.

[0022] The facility 10 is a facility or device in which a control target 15 is installed. For example, the facility 10 may be a plant, or a composite device that combines multiple devices. Examples of plants include industrial plants such as chemical and bio plants, plants that manage and control wellheads and surrounding areas of gas and oil fields, plants that manage and control power generation such as hydroelectric, thermal, and nuclear power, plants that manage and control environmental power generation such as solar and wind power, and plants that manage and control water supply and sewage systems, dams, etc.

[0023] From here on, we will explain the case where the equipment 10 is a distillation apparatus, which is one type of process equipment, as an example. Generally, a distillation apparatus evaporates low-boiling components in a distillation column and extracts them from the top of the column. The extracted low-boiling component vapor is condensed in a condenser and stored in a reflux drum. The distillation apparatus then refluxes a portion of the vapor stored in the reflux drum into the distillation column, where it comes into contact with the vapor in the distillation column and is distilled into low-boiling components and high-boiling components. In such a distillation apparatus, as an example, a valve installed between the reflux drum and the distillation column is opened and closed to control the reflux amount.

[0024] The controlled object 15 is a device installed in the equipment 10 and is the object of control. For example, the controlled object 15 may be an actuator, i.e., an operating element, such as a valve, heater, motor, fan, or switch, that controls at least one physical quantity, such as the amount, temperature, pressure, flow rate, speed, or pH, of an object in the process of the equipment 10, and performs a given operation in response to the operating amount. Hereinafter, a case will be described in which the controlled object 15 is a valve installed between the reflux drum and the distillation column in a distillation apparatus. However, this is not limiting. The controlled object 15 may also be a controller that controls an operating element. In other words, the term "control" used in this specification may be broadly interpreted to include not only direct control of an operating element, but also indirect control of an operating element via a controller.

[0025] The equipment 10 in which the control object 15 is installed may be provided with one or more sensors capable of measuring various conditions (physical quantities) inside and outside the equipment 10. As an example, if the equipment 10 is a distillation apparatus, the sensors may output measured values PV (Process Variables) measuring temperatures at various positions in the distillation apparatus (e.g., the top, center, or bottom of the column) and flow rates in various paths. Status data indicating the status of the equipment 10 may include such measured values PV. The status data may also include manipulated variables MV (Manipulated Variables) indicating the opening and closing degrees of valves, which are the control object 15. In addition to operation data indicating the operating state as a result of controlling the control object 15, the status data may also include consumption data indicating the consumption of energy and raw materials in the equipment 10, disturbance environment data indicating physical quantities that may act as disturbances to the control of the control object 15, and the like.

[0026] Distillation units are one of the most widely used devices in petroleum and chemical processes. However, they are characterized by strong mutual interference between the top and bottom of the column, a long time constant, and nonlinear operation. When valves are controlled to control the reflux amount in such distillation units using proportional integral differential (PID) or other methods, it is difficult to improve controllability. Furthermore, when such valves are manually operated by operators for multiple purposes, such as quality assurance, energy conservation, GHG (Green House Gas) reduction, and yield improvement, the extent to which the valves are controlled relies heavily on the operator's experience and intuition.

[0027] Therefore, when controlling the opening and closing of such valves, it is conceivable to use a machine learning model generated by machine learning. The control device 100 according to this embodiment may use, for example, the output of such a machine learning model as a target for correction, which will be described later.

[0028] The simulator 20 simulates the operation of the facility 10. For example, the simulator 20 may be designed based on design information for the facility 10 and executes behavior simulating the operation of the facility 10. The simulator 20 acquires signals simulating manipulated variables for the controlled object 15 to change the environment, and outputs simulation data simulating the state of the facility 10 (e.g., predicted values of sensors). As an example, the simulator 20 may be composed of a prediction model that predicts the state of a distillation apparatus and a plant control simulator. The prediction model may be capable of predicting state changes of a reactor from accumulated process data using time-series data modeling technology using deep learning. Furthermore, the plant control simulator may be capable of virtually simulating PID control for the controlled object 15, which derives a manipulated variable MV based on the difference between a target value SV and a controlled variable CV. In other words, the simulator 20 may be capable of simulating the behavior of the facility 10 itself in addition to the predicted state value.

[0029] The database 30 stores status data (also called "actual data" or "historical data") obtained directly or indirectly from sensors, etc. installed in the equipment 10 when the equipment 10 is actually operated.

[0030] The control device 100 may be a computer such as a personal computer (PC), tablet computer, smartphone, workstation, server computer, or general-purpose computer, or may be a computer system in which multiple computers are connected. Such a computer system is also a computer in the broad sense. The control device 100 may also be implemented as one or more virtual computer environments executable within a computer. Alternatively, the control device 100 may be a dedicated computer designed for controlling the control target 15, or may be dedicated hardware realized by a dedicated circuit. Furthermore, if the control device 100 is connectable to the Internet, it may be realized by cloud computing.

[0031] The control device 100 controls the controlled object 15 using an operation model that has been machine-learned using simulation data. In this case, the control device 100 according to this embodiment acquires an index for monitoring a difference (deviation) between the behavior of the simulator 20 and the actual operation of the facility 10, and corrects the output of the operation model based on the index. The control device 100 includes a model output acquisition unit 110, an index acquisition unit 120, a correction condition setting unit 130, a correction unit 140, and a control unit 150.

[0032] The model output acquisition unit 110 acquires an operational model output that is output in response to inputting status data indicating the status of the equipment 10 into an operational model that has been machine-learned to output an action corresponding to the status of the equipment 10 using simulation data from a simulator 20 that simulates operations at the equipment 10. For example, the model output acquisition unit 110 may acquire such an operational model output via a network from an operational model management device 200 that manages the operational model. Details of the operational model management device 200 will be described later. The model output acquisition unit 110 supplies the acquired operational model output to the correction unit 140.

[0033] The index acquiring unit 120 acquires indexes, which are calculated using actual data from the equipment 10, for monitoring differences between the behavior of the simulator 20 and the actual operation of the equipment 10. For example, the index acquiring unit 120 may acquire such indexes via a network from an index calculation device 300 that is capable of calculating the indexes. Details of the index calculation device 300 will be described later. The index acquiring unit 120 supplies the acquired indexes to the correction unit 140.

[0034] The correction condition setting unit 130 sets conditions for correcting the operation model output. For example, the correction condition setting unit 130 may acquire such conditions through user input. The correction condition setting unit 130 supplies the set conditions to the correction unit 140.

[0035] The correction unit 140 corrects the operation model output acquired by the model output acquisition unit 110 based on the index acquired by the index acquisition unit 120. At this time, the correction unit 140 corrects the operation model output in accordance with the conditions set by the correction condition setting unit 130. The correction unit 140 supplies the corrected operation model output to the control unit 150.

[0036] The control unit 150 outputs a manipulated variable for controlling the controlled object 15 provided in the equipment 10 in accordance with the corrected operation model output. For example, if the controlled object 15 is a valve, the control unit 150 may output the opening / closing degree of the valve. However, this is not limited to this. The control unit 150 may output various control inputs that can change the controlled variable. In other words, the term "manipulated variable" used in this specification may be broadly interpreted to include not only the so-called MV but also the target value SV and other control parameters (e.g., PID parameters).

[0037] FIG. 2 shows an example of a block diagram of the operation model management device 200. Like the control device 100, the operation model management device 200 may be a computer or a computer system to which multiple computers are connected. Furthermore, the operation model management device 200 may be implemented as a virtual computer environment in which one or more programs can be executed within a computer. Alternatively, the operation model management device 200 may be a dedicated computer designed for managing operation models, or may be dedicated hardware realized by dedicated circuits. Furthermore, if the operation model management device 200 is connectable to the Internet, it may be realized by cloud computing.

[0038] The operational model management device 200 manages an operational model that has been machine-learned to output an action according to the state of the facility 10 using simulation data from a simulator 20 that simulates operations in the facility 10. The operational model management device 200 includes an operation terminal / observation point setting unit 210, a target setting unit 220, an operational model learning unit 230, and an operational model storage unit 240.

[0039] The control terminal / observation point setting unit 210 sets information about the control terminal to be operated and information about the observation point to be used for learning. For example, the control terminal / observation point setting unit 210 may acquire the tag name of the control terminal and the tag name of the observation point via user input. In this case, the user may select the tag name from a pre-prepared tag list, or may input the tag name directly. The control terminal / observation point setting unit 210 supplies the set information to the operation model learning unit 230.

[0040] The goal setting unit 220 sets a target value and a reward function. For example, the goal setting unit 220 may acquire, via user input, a target value SV of the controlled variable CV and a reward function for quantifying the quality of the selected action. In this case, as an example, the reward function may include at least a component of the difference between the target value and the controlled variable, such as a constant × |SV - CV| or a constant × (SV - CV)^2. In other words, the reward function may be set so that the reward value, which is the calculation result, becomes higher as the controlled variable approaches the target value. The goal setting unit 220 supplies the set information to the operation model learning unit 230.

[0041] The operational model learning unit 230 generates an operational model through machine learning. For example, the operational model learning unit 230 may acquire setting information from the operation terminal / observation point setting unit 210 and the target setting unit 220. The operational model learning unit 230 may also acquire simulation data from the simulator 20. The operational model learning unit 230 may then generate an operational model by performing machine learning using the simulation data in accordance with the setting information to output an action corresponding to the state of the equipment 10. Here, the generated operational model may be a machine learning model that has undergone reinforcement learning to output an action that increases a reward value determined by a reward function. Such reinforcement learning will be described in detail below.

[0042] The operation model storage unit 240 stores an operation model. For example, the operation model storage unit 240 stores an operation model generated by the operation model learning unit 230. In the above description, an example has been given in which the operation model storage unit 240 stores an operation model generated inside the operation model management device 200, but the present invention is not limited to this. The operation model storage unit 240 may also store an operation model generated outside the operation model management device 200.

[0043] FIG. 3 shows an example of a data table that the operation model may have. The data table may be configured with combinations of the state and actions of the equipment 10, and weights for the combinations. This figure shows an example in which the tag name of the operation terminal is set to "V001" and the tag names of the observation points are set to "TI001," "TI002," "TI003," "FI001," and "FI002" in the operation terminal / observation point setting unit 210. In such a case, the state of the equipment 10 may be represented as state data S = (TI001, TI002, TI003, FI001, FI002, V001).

[0044] Furthermore, a behavior may be represented as a change in the manipulated variable MV of the controlled element. For example, if the manipulated variable MV (i.e., the value of the tag "V001") of the controlled element is 70 at time a, and the manipulated variable MV is 71 at time b following time a, this indicates that the manipulated variable MV changed from 70 to 71 from time a to time b. Here, subtracting the manipulated variable MV=70 at time a from the manipulated variable MV=71 at time b results in the manipulated variable change ΔMV=1. In this case, the behavior at time a may be represented as behavior data A=1.

[0045] Furthermore, the weight may be represented as weight data W calculated by a reward value determined by a reward function set in the goal setting unit 220. The weight data W may be initially set to an initial value (e.g., 1) and may be sequentially overwritten as learning progresses.

[0046] In this way, the data table of the operation model may store multiple sample data, each of which is a combination of status data S = (TI001, TI002, TI003, FI001, FI002, V001) and action data A = ΔMV, and weight data W for that combination.

[0047] As an example, looking at the first line in this diagram, if the state is S=(-2.47803, -2.48413, -0.07324, 29.71191, 24.2511, 70) and action A=1 (for example, controlling the valve to open by 1% counterclockwise) is taken, the weight will be W=144.1484. The next action is determined using an operation model with such a data table.

[0048] FIG. 4 shows an example of an action table for determining an action using an operation model. The action table may be composed of state data S and possible action data A. In this figure, as an example, the state of the equipment 10 is S=(0.1, 0.2, 0.4, 0.3, 0.8, 0.2), and the possible action options are A=[-3, -1, 0, 1, 3], i.e., [3% close, 1% close, no change, 1% open, 3% open]. For example, the next action is determined by inputting the action table shown in this figure into an operation model having a data table such as that shown in FIG. 3. A method for generating such an operation model will be described in detail using a flow chart.

[0049] FIG. 5 shows an example of a flow for generating an operational model using reinforcement learning. In step S500, the operational model management device 200 initializes the operational model. For example, the operational model learning unit 230 creates an empty data table in the operational model storage unit 240. Then, the operational model learning unit 230 reads the tag names of the control elements and observation points set in the control element / observation point setting unit 210, and defines columns of the data table using these tag names, actions, and weights. The operational model learning unit 230 also acquires multiple sample data as initial learning samples and adds them to each row of the data table. At this time, the operational model learning unit 230 may set the weight data W for the initial learning samples to an initial value.

[0050] In step S502, the operation model management device 200 acquires simulation data. For example, the operation model learning unit 230 acquires simulation data that simulates the state of the facility 10 from the simulator 20, which is the learning environment, via a network.

[0051] In step S504, the operational model management device 200 determines an action. For example, the operational model learning unit 230 may randomly determine an action. However, this is not limiting. The operational model learning unit 230 may use a known AI algorithm, such as FKDPP, to determine the action. When using such a kernel method, the operational model learning unit 230 generates a vector of state data S from the predicted values of the sensors obtained from the simulation data. Next, the operational model learning unit 230 combines the state data S with all possible action data A to generate an action table, such as that shown in FIG. 4. The operational model learning unit 230 then inputs the action table into the operational model. In response to this, the operational model performs kernel calculations between each row of the action table and each row of the data table excluding the weight column, and calculates the distance between each sample data. The operation model then sequentially adds up the results of multiplying the distance calculated for each sample data by the value of each weight sequence, and calculates the reward value (expected reward value) for each action. The operation model selects the action that will result in the highest reward value calculated in this way. For example, the operation model learning unit 230 may determine an action by selecting the action that will result in the highest reward value using the operation model being updated in this way. During learning, the operation model learning unit 230 may determine an action by appropriately selecting whether to determine an action randomly or to determine an action using an operation model.

[0052] In step S506, the operation model management device 200 instructs the simulator 20 to perform an operation. For example, the operation model learning unit 230 adds the action determined in step S504 to the immediately preceding operation amount of the operation terminal in the learning environment, and provides the new operation amount to the simulator 20. This changes the state of the learning environment.

[0053] In step S508, the operation model management device 200 acquires simulation data. For example, similar to step S502, the operation model learning unit 230 acquires simulation data simulating the state of the facility 10 from the simulator 20. That is, the operation model learning unit 230 acquires the state of the learning environment after it has changed in response to the application of the operation amount based on the determined action.

[0054] In step S510, the operation model management device 200 calculates a reward value. For example, the operation model learning unit 230 calculates the reward value using the reward function set in the goal setting unit 220.

[0055] In step S512, the operation model management device 200 determines whether the number of steps has been exceeded. For example, the operation model learning unit 230 determines whether the process of acquiring a state corresponding to the determined action has exceeded a specified number of steps. Note that such a number of steps may be specified in advance by the user, or may be determined based on a learning period (e.g., 10 days). If it is determined that the above process has not exceeded the number of steps (No), the operation model management device 200 returns the process to step S504 and continues the flow. The operation model management device 200 executes the process of acquiring a state corresponding to the determined action for the specified number of steps.

[0056] If it is determined in step S512 that the above-mentioned processing has exceeded the number of steps (Yes), the operation model management device 200 proceeds to step S514. In step S514, the operation model management device 200 updates the operation model. For example, the operation model learning unit 230 overwrites the values in the weight column in the data table shown in Fig. 3 and adds new sample data that has not been saved to a new row in the data table.

[0057] In step S516, the operation model management device 200 determines whether the number of repetitions has been exceeded. For example, the operation model learning unit 230 determines whether the update process of the operation model has exceeded a specified number of repetitions. Note that such a number of repetitions may be specified in advance by the user, or may be determined depending on the validity of the operation model. If it is determined that the above process has not exceeded the number of repetitions (No), the operation model management device 200 returns the process to step S502 and continues the flow.

[0058] If it is determined in step S516 that the number of repetitions of the above-mentioned process has exceeded (Yes), the operation model management device 200 ends the flow. In this way, for example, the operation model management device 200 can generate an operation model that outputs an action according to the state of the equipment 10 by performing reinforcement learning so as to output an action that increases the reward value determined by the reward function.

[0059] The operation model generated in this manner is learned based on the behavior of the simulator 20. Therefore, even if a situation arises in which the behavior of the simulator 20 differs from the actual operation of the equipment 10, the operation model outputs an action that is based on the behavior of the simulator 20. As a result, a situation may arise in which an operation amount that is not suitable for the actual operation of the equipment 10 is applied to the control object 15. In such a case, various problems may occur, such as a decrease in operational efficiency, deterioration of product quality, and an increased risk of dangerous situations, compared to when an operator manually operates the equipment 10. Therefore, it is important to monitor the behavior of the simulator 20, which is the assumption of the operation model, in light of the actual operation of the equipment 10, and, if a difference occurs between the two, to reflect the monitoring results in the control of the control object 15.

[0060] FIG. 6 shows an example of a block diagram of the index calculation device 300. Like the control device 100, the index calculation device 300 may also be a computer, or a computer system to which multiple computers are connected. The index calculation device 300 may also be implemented by a virtual computer environment in which one or more programs can be executed within a computer. Alternatively, the index calculation device 300 may be a dedicated computer designed for calculating the index, or may be dedicated hardware realized by a dedicated circuit. Furthermore, if the index calculation device 300 is connectable to the Internet, it may be realized by cloud computing.

[0061] The index calculation device 300 uses actual data obtained when the equipment 10 is actually operated to calculate an index for monitoring the difference between the behavior of the simulator 20 and the actual operation of the equipment 10. Here, a case will be described as an example in which the index calculation device 300 calculates such an index using a monitoring model, which is a machine learning model. The index calculation device 300 includes a learning setting unit 310, a monitoring model learning unit 320, and a monitoring model storage unit 330.

[0062] The learning setting unit 310 sets information defining state data used in learning, hyperparameters of the learning algorithm, and information about the teacher signal. For example, the learning setting unit 310 may acquire, via user input, the tag name of a sensor used in learning as information defining state data used in learning. In this case, the user may select the tag name from a pre-prepared tag list or may input the tag name directly.

[0063] Furthermore, for example, when a neural network model is used as the monitoring model, the learning setting unit 310 may acquire, via user input, hyperparameters of the learning algorithm, such as the neural network structure, learning rate, optimization algorithm, number of epochs, and random number seed. Here, a case where a neural network is used as the learning algorithm of the monitoring model is shown as an example. However, this is not limiting. For example, other algorithms, such as a decision tree, may be used to train the monitoring model.

[0064] The learning setting unit 310 may also acquire, via user input, information about any signal that the user desires to regress regarding operations in the facility 10 as information about the teacher signal. In this case, the user may input the value of the teacher signal itself, or may input a mathematical formula for deriving the value of the teacher signal. As an example, such a teacher signal may be a future (e.g., five-step ahead) predicted value of a sensor included in status data indicating the state of the facility 10, or a future predicted value of a sensor that can be predicted from the status data, i.e., a soft sensor. Alternatively or in addition to this, the teacher signal may be the quality of a product or semi-finished product manufactured in the facility 10, which can be predicted (quantified) from the status data. In this way, the teacher signal may be any signal that the user desires to regress regarding operations in the facility 10, as long as it can be predicted from the status data. The learning setting unit 310 supplies the set information to the monitoring model learning unit 320.

[0065] The monitoring model learning unit 320 generates a monitoring model through machine learning. For example, the monitoring model learning unit 320 may acquire setting information from the learning setting unit 310. The monitoring model learning unit 320 may also acquire actual data obtained when the equipment 10 is actually operated from the database 30. The monitoring model learning unit 320 may then generate a monitoring model by performing machine learning in accordance with the setting information and using the actual data to output an index corresponding to the state of the equipment 10. Here, the generated monitoring model may be a machine learning model that has been subjected to supervised learning so that the difference between the output of the monitoring model and a teacher signal is small. Such supervised learning will be described in detail below.

[0066] The surveillance model storage unit 330 stores a surveillance model. For example, the surveillance model storage unit 330 stores a surveillance model generated by the surveillance model learning unit 320. Note that, in the above description, a case where the surveillance model storage unit 330 stores a surveillance model generated inside the index calculation device 300 has been described as an example, but the present invention is not limited to this. The surveillance model storage unit 330 may also store a surveillance model generated outside the index calculation device 300.

[0067] In this embodiment, a case where only one index is used as an index for monitoring the behavior of the simulator 20 will be described as an example, but the present invention is not limited to this. A plurality of indexes may be used as such an index. In this case, the monitoring model storage unit 330 may store a plurality of monitoring models that each output a different index.

[0068] FIG. 7 shows an example of supervised learning data used to generate a monitoring model. This figure shows an example in which the tag names of the sensors used for learning are set to "TI001," "TI002," "TI003," "FI001," "FI002," and "V001" in the learning setting unit 310. These columns contain actual data acquired from the database 30, i.e., values of status data obtained during actual operation of the equipment 10, entered in chronological order. Furthermore, the "teacher signal" column contains values of the teacher signal set in the learning setting unit 310, entered in chronological order. The monitoring model learning unit 320 may generate a monitoring model by, for example, performing supervised learning using such learning data.

[0069] More specifically, the monitoring model training unit 320 initializes the monitoring model according to the neural network structure, learning rate, optimization algorithm, number of epochs, and random number seed set in the training setting unit 310. At this time, the monitoring model training unit 320 may construct a neural network model F(S, w) where S=(TI001, TI002, TI003, FI001, FI002, V001) is the state data and w is the weight of the neural network. Next, the monitoring model training unit 320 may initialize the weight w with a random number. Next, the monitoring model training unit 320 may divide the training data, such as that shown in this figure, into a column of state data and a column of teacher signals, and generate features using the column of state data. Next, the monitoring model training unit 320 may calculate F(S, w) for each row of the state data S. Next, the supervisory model training unit 320 may compare the output y of F(S, w) with the teacher signal y' and update the weights w of the neural network using an optimization algorithm and a learning rate so that the difference between them becomes small. The supervisory model training unit 320 may repeat this process of updating the weights w a number of times specified by the number of epochs.

[0070] The control device 100 according to this embodiment acquires, for example, the output y of the monitoring model generated in this way as an index for monitoring the behavior of the simulator 20 on which the operational model is based, and corrects the operational model output. This will be described in detail using a flow chart.

[0071] 8 shows an example of a control flow by the control device 100 according to this embodiment. In step S802, the control device 100 acquires status data. For example, the control device 100 may acquire status data S=(TI001, TI002, TI003, FI001, FI002, V001) indicating the status of the equipment 10 from the equipment 10 via a network in real time.

[0072] In step S804, the control device 100 acquires an operational model output. For example, the model output acquisition unit 110 may input the state data S acquired in step S802 into the operational model stored in the operational model management device 200. In response, the operational model may select and output the action that maximizes the reward value using the above-described FKDPP algorithm. The model output acquisition unit 110 may acquire the action output from the operational model as the operational model output x. For example, in this manner, the model output acquisition unit 110 can acquire the operational model output x that is output in response to inputting the state data S indicating the state of the equipment 10 into the operational model that has been trained by machine learning to output an action according to the state of the equipment 10 using simulation data from the simulator 20 that simulates operations in the equipment 10. The model output acquisition unit 110 supplies the acquired operational model output x to the correction unit 140.

[0073] In step S806, the control device 100 acquires an index. For example, the index acquiring unit 120 may input the status data S acquired in step S802 to a monitoring model stored in the index calculation device 300. In response to this, the monitoring model may calculate F(S, w) and output the calculation result y. The index acquiring unit 120 may acquire the monitoring model output y as an index. For example, in this way, the index acquiring unit 120 may acquire the monitoring model output y, which is output in response to inputting the status data S into a monitoring model that has been machine-learned using actual data to output an index according to the status of the equipment, as an index.

[0074] As described above, the monitoring model is machine-learned to reduce the difference between the monitoring model output y and the teacher signal y'. Therefore, when the teacher signal y' is set as a future predicted value of a sensor included in the state data S, or a future predicted value of a software sensor that can be estimated from the state data S, the index acquiring unit 120 acquires, as an index, the future predicted value of the sensor included in the state data S, or the future predicted value of a sensor that can be estimated from the state data S. Alternatively, or in addition, when the teacher signal y' is set as the quality of a product or semi-finished product manufactured in the facility 10 that can be predicted (quantified) from the state data S, the index acquiring unit 120 acquires, as an index, a value that quantifies the quality of the product or semi-finished product manufactured in the facility 10.

[0075] In this way, for example, the index acquiring unit 120 can acquire an index calculated using actual data from the equipment 10, for monitoring the difference between the behavior of the simulator 20 and the actual operation of the equipment 10. The index acquiring unit 120 supplies the acquired index, in this case the monitoring model output y, to the correcting unit 140.

[0076] In step S808, the control device 100 corrects the operation model output. For example, the correction unit 140 may correct the operation model output x acquired in step S804 based on the index acquired in step S806, here, the monitoring model output y.

[0077] Here, it is assumed that the conditions set in the correction condition setting unit 130 include a range in which the index is allowed. As an example, it is assumed that an upper alarm limit value and a lower alarm limit value for the monitoring model output y are set in the correction condition setting unit 130. It is also assumed that the conditional expression for executing the correction is set as follows. Note that such a conditional expression is an example of a case where the relationship that increases x makes y smaller, and decreases x makes y larger is known in advance (theoretically or empirically).

number

[0078] In this case, the correction unit 140 may correct the operational model output x in accordance with the conditional expression. That is, if the monitoring model output y exceeds the alarm upper limit, the correction unit 140 may increase the operational model output x by 1. For example, if the operational model output x is "-3," the correction unit 140 may correct the operational model output x from "-3" to "-2," i.e., change the action from closing the valve 3% to closing the valve 2%. Similarly, if the monitoring model output y is below the alarm lower limit, the correction unit 140 may decrease the operational model output x by 1. For example, if the operational model output x is "-3," the correction unit 140 may correct the operational model output x from "-3" to "-4," i.e., change the action from closing the valve 3% to closing the valve 4%. In this way, if the index is out of range, the correction unit 140 may correct the operational model output x so that the index is within the range. The correction unit 140 supplies the corrected operation model output x to the control unit 150.

[0079] In step S810, the control device 100 outputs a manipulated variable. For example, the control unit 150 may output a manipulated variable for controlling the controlled object 15 provided in the equipment 10, in accordance with the operation model output x corrected in step S808. As an example, the control unit 150 may provide the controlled object 15, which is the controlled end, with a new manipulated variable MV obtained by adding the corrected operation model output x to the immediately previous manipulated variable MV at the controlled end.

[0080] When generating an operation model using machine learning, learning while operating the actual equipment can lead to the possibility of dangerous operations during the learning period. Furthermore, when using reinforcement learning to generate an operation model, learning may not converge well if only historical data is used. Therefore, simulation-based learning of operation models has been considered. However, situations may arise in which the behavior of the simulator 20, which is designed based on the design information of the equipment 10, differs from the actual operation of the equipment 10. In such cases, the output of the simulation-based trained operation model may not be suitable for the actual operation of the equipment 10. When such a situation occurs, problems such as a deterioration in the operating efficiency and product quality of the equipment 10 and an increased risk of dangerous situations may arise compared to when an operator manually operates the equipment 10.

[0081] In contrast, the control device 100 according to this embodiment, when controlling the controlled object 15 using an operation model learned on a simulation basis, acquires an index for monitoring the difference between the behavior of the simulator 20 and the actual operation of the facility 10, and corrects the output of the operation model based on the index. As a result, according to the control device 100 according to this embodiment, even if a situation arises in which the behavior of the simulator 20 differs from the actual operation of the facility 10, it is possible to avoid controlling the controlled object 15 by directly adopting the output of the operation model learned on a simulation basis.

[0082] Furthermore, the control device 100 according to this embodiment may use the output of a monitoring model that has been machine-learned as an index. As a result, the control device 100 according to this embodiment can use two machine-learning models to control the control target 15 while taking into account the gap between the simulation and reality.

[0083] Furthermore, the control device 100 according to this embodiment may use a supervised machine learning model as the monitoring model. As a result, the control device 100 according to this embodiment can reflect the knowledge of workers and past operational performance in the output of the monitoring model.

[0084] Furthermore, the control device 100 according to this embodiment may use a machine learning model that has undergone reinforcement learning as an operation model. As a result, the control device 100 according to this embodiment can flexibly control the control target 15 even when it is difficult to determine an evaluation scale for a single action or when adaptation to an unknown environment is required.

[0085] Furthermore, the control device 100 according to this embodiment can set future predicted values of sensors, product quality, etc. as indices. In this way, the control device 100 according to this embodiment can monitor the behavior of the simulator 20 using various indices that are desired to be regressed regarding the operation of the facility 10.

[0086] Furthermore, when an index is outside the range of a set condition, the control device 100 according to this embodiment corrects the output of the operation model so that the index approaches the range. As a result, the control device 100 according to this embodiment can correct the output of the operation model so as to close the gap between the simulation and reality.

[0087] In the above description, the control device 100 and the operation model management device 200 are separate devices, but the present invention is not limited to this. Some or all of the functional units of the operation model management device 200 may be integrated with the control device 100. That is, the control device 100 may further include an operation model storage unit 240. The control device 100 may further include an operation model learning unit 230. The control device 100 may further include a target setting unit 220. The control device 100 may further include an operation terminal / observation point setting unit 210.

[0088] As a result, the control device 100 can realize, in a single device, the function of controlling the controlled object 15 and the function of storing the operation model, so there is no need to exchange data between devices when obtaining the operation model output, and it is possible to safely and securely control the controlled object 15. Furthermore, since the control device 100 can realize, in a single device, the function of controlling the controlled object 15 and the function of generating an operation model, there is no need to transfer the generated operation model between devices, and communication costs can be reduced.

[0089] Similarly, in the above description, the case where the control device 100 and the index calculation device 300 are separate devices has been described as an example, but the present invention is not limited to this. Some or all of the functional units of the index calculation device 300 may be configured integrally with the control device 100. That is, the control device 100 may further include a surveillance model storage unit 330. The control device 100 may also include a surveillance model learning unit 320. The control device 100 may also include a learning setting unit 310.

[0090] As a result, the control device 100 can realize, in one device, the function of controlling the control target 15 and the function of storing the monitoring model, so there is no need to exchange data between devices when acquiring the index, and it is possible to safely and securely control the control target 15. Furthermore, since the control device 100 can realize, in one device, the function of controlling the control target 15 and the function of generating a monitoring model, there is no need to move the generated monitoring model between devices, and communication costs can be reduced.

[0091] FIG. 9 shows an example of a block diagram of the control device 100 according to a modification of this embodiment. In FIG. 9, the same reference numerals are used to designate components having the same functions and configurations as those in FIG. 1, and explanations will be omitted hereinafter except for differences. In the above-described embodiment, the control device 100 acquires the monitoring model output as an index and corrects the operation model output based on the monitoring model output. However, in this modification, the control device 100 calculates the calculated value y 1 calculated by the simulator 20. M is obtained as an index, and the calculated value y M The control device 100 according to this modification further includes an actual measurement value extraction unit 910.

[0092] In this modification, the model output acquisition unit 110 functions in the same manner as in the above-described embodiment. For example, if state data at time t is S(t), the model output acquisition unit 110 may acquire an operation model output x(t) that is output in response to inputting the state data S(t) into the operation model. The model output acquisition unit 110 supplies the acquired operation model output x(t) to the correction unit 140.

[0093] The actual measurement value extraction unit 910 extracts the calculated value y from the state data S. MThe actual measurement value extraction unit 910 extracts an actual measurement value y to be compared with the actual measurement value y(t). For example, if the actual measurement value at time t is y(t), the actual measurement value extraction unit 910 may extract the value of the target sensor included in the status data S(t) as the actual measurement value y(t). Here, the target sensor may be one or more sensors included in the status data S(t). The actual measurement value extraction unit 910 supplies the extracted actual measurement value y(t) to the correction unit 140.

[0094] In this modification, the index acquisition unit 120 calculates the calculated value y M For example, the state data at time t-1 immediately before time t is S(t-1), the corrected operation model output at time t-1 is u(t-1), and the calculated value at time t is y M (t), the index acquisition unit 120 may acquire the state data S(t-1) and the corrected operation model output u(t-1) and input them to the simulator 20. In response to this, the simulator 20 may calculate the value y of the target sensor at time t following time t-1 using the state data S(t-1) and the corrected operation model output u(t-1). M The index acquiring unit 120 may calculate the calculated value y (t) calculated by the simulator 20 in this way. M The index acquiring unit 120 may acquire the acquired calculated value y M (t) is supplied to the correction unit 140.

[0095] In this modification, the correction condition setting unit 130 calculates the operation model output x(t), the actual measurement value y(t), and the calculated value y M (t), a correction function for calculating the corrected operation model output u(t) may be set. As an example, the correction condition setting unit 130 may set a correction function as shown in the following equation, with P as the first coefficient:

number

[0096] Alternatively, the correction condition setting section 130 may set a correction function such as the following equation, with I as the second coefficient.

number

[0097] Alternatively, the correction condition setting section 130 may set a correction function such as the following equation, with D as a third coefficient.

number

[0098] In this modification, the correction unit 140 corrects the operational model output using the correction function set in the correction condition setting unit 130. For example, the correction unit 140 may correct the operational model output using equation (2). That is, the correction unit 140 corrects the operational model output x(t) by applying the actual measurement value y(t) and the calculated value y(t) to the operational model output x(t). M The corrected operational model output u(t) may be calculated by adding a value obtained by multiplying the difference with (t) by a coefficient.

[0099] Alternatively, the correcting unit 140 may correct the operational model output using the formula (3). That is, the correcting unit 140 corrects the operational model output x(t) by adding the actual measured value y(t) and the calculated value y(t). M The difference between the measured value y(t) and the calculated value y(t) multiplied by a coefficient is M The corrected operational model output u(t) may be calculated by adding the integral of the difference between (t) and (t) multiplied by a coefficient.

[0100] Alternatively, the correcting unit 140 may correct the operational model output using equation 4. That is, the correcting unit 140 corrects the operational model output x(t) by adding the actual measured value y(t) and the calculated value y M The difference between the measured value y(t) and the calculated value y(t) multiplied by a coefficient is M The integral of the difference between (t) and the calculated value y(t) is multiplied by a coefficient. M The deviation from (t) may be multiplied by a coefficient and added to calculate the corrected operational model output u(t).

[0101] In this modification, the correction unit 140 can correct the operational model output using, for example, a correction function whose correction term includes the difference between the actual measured value and the calculated value indicated by the status data. In this manner, the control device 100 according to this modification uses the calculated value calculated by the simulator 20 as an index for monitoring the difference between the behavior of the simulator 20 and the actual operation of the facility 10. This eliminates the need to separately prepare a model for calculating an index such as a monitoring model, thereby simplifying processing. Furthermore, the control device 100 according to this modification corrects the operational model output using a correction function whose correction term includes the difference between the actual measured value and the calculated value indicated by the status data. This allows the control device 100 according to this modification to correct the operational model output in the event of a difference between the actual measured value and the calculated value, depending on the magnitude of the difference.

[0102] So far, several possible embodiments have been described as examples. However, the above-described embodiments may be modified or applied in various ways. For example, the above description has shown an example in which the monitoring model used during operation is trained in advance using historical data during training. However, it is conceivable that the actual data used during operation of the monitoring model is not similar to the historical data used during training of the monitoring model. In this case, the monitoring model may be retrained in response to a low degree of similarity between these data. For example, the index calculation device 300 may acquire actual data from the database 30 even during operation. Then, the index calculation device 300 may calculate the similarity between the historical data used during training and the actual data acquired during operation, and if the similarity does not satisfy a predetermined range, retrain the monitoring model using the actual data acquired during operation.

[0103] In the above description, a case where the user manually sets the correction conditions for correcting the operational model output has been described as an example. However, a machine learning model may be used to set the correction conditions. For example, the control device 100 may set the correction conditions using a machine learning model that receives the operational model output and the monitoring model output as input and returns an optimal corrected operational model output.

[0104] Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where the blocks may represent (1) stages of a process in which operations are performed or (2) sections of an apparatus responsible for performing the operations. Particular stages and sections may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable medium, and / or a processor provided with computer-readable instructions stored on a computer-readable medium. Dedicated circuitry may include digital and / or analog hardware circuitry, and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry, including logical AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, memory elements such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

[0105] A computer-readable medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that the computer-readable medium having instructions stored thereon comprises an article of manufacture containing instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable media may include electronic, magnetic, optical, electromagnetic, and semiconductor storage media. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated circuit card, and the like.

[0106] The computer readable instructions may include either assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages such as the “C” programming language or similar programming languages.

[0107] The computer-readable instructions may be provided to a processor or programmable circuitry of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, either locally or over a wide-area network (WAN) such as a local area network (LAN), the Internet, etc., which executes the computer-readable instructions to create means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0108] 10 illustrates an example of a computer 9900 in which aspects of the present invention may be embodied, in whole or in part. Programs installed on the computer 9900 may cause the computer 9900 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to embodiments of the present invention, and / or to perform a process or steps of a process according to embodiments of the present invention. Such programs may be executed by the CPU 9912 to cause the computer 9900 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0109] The computer 9900 according to this embodiment includes a CPU 9912, a RAM 9914, a graphics controller 9916, and a display device 9918, which are interconnected by a host controller 9910. The computer 9900 also includes input / output units such as a communication interface 9922, a hard disk drive 9924, a DVD drive 9926, and an IC card drive, which are connected to the host controller 9910 via an input / output controller 9920. The computer also includes legacy input / output units such as a ROM 9930 and a keyboard 9942, which are connected to the input / output controller 9920 via an input / output chip 9940.

[0110] The CPU 9912 operates according to programs stored in the ROM 9930 and RAM 9914, thereby controlling each unit. The graphics controller 9916 retrieves image data generated by the CPU 9912 into a frame buffer or the like provided in the RAM 9914 or into the graphics controller itself, and causes the image data to be displayed on the display device 9918.

[0111] The communication interface 9922 communicates with other electronic devices via a network. The hard disk drive 9924 stores programs and data used by the CPU 9912 in the computer 9900. The DVD drive 9926 reads programs or data from the DVD-ROM 9901 and provides the programs or data to the hard disk drive 9924 via the RAM 9914. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0112] The ROM 9930 stores therein a boot program or the like that is executed by the computer 9900 upon activation, and / or programs that depend on the hardware of the computer 9900. The input / output chip 9940 may also connect various input / output units to the input / output controller 9920 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0113] The programs are provided by a computer-readable medium such as a DVD-ROM 9901 or an IC card. The programs are read from the computer-readable medium, installed in the hard disk drive 9924, RAM 9914, or ROM 9930, which are also examples of computer-readable media, and executed by the CPU 9912. The information processing described in these programs is read by the computer 9900, and causes cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing information manipulation or processing in accordance with the use of the computer 9900.

[0114] For example, when communication is performed between the computer 9900 and an external device, the CPU 9912 may execute a communication program loaded into the RAM 9914 and instruct the communication interface 9922 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 9912, the communication interface 9922 reads transmission data stored in a transmission buffer processing area provided in the RAM 9914, the hard disk drive 9924, the DVD-ROM 9901, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0115] The CPU 9912 may also cause all or a necessary portion of a file or database stored on an external recording medium such as a hard disk drive 9924, a DVD drive 9926 (DVD-ROM 9901), an IC card, etc. to be read into the RAM 9914, and perform various types of processing on the data on the RAM 9914. The CPU 9912 then writes back the processed data to the external recording medium.

[0116] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 9912 may perform various types of processing on data read from the RAM 9914, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 9914. The CPU 9912 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored on the recording medium, the CPU 9912 may search for an entry that matches a condition specified by the attribute value of the first attribute from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0117] The programs or software modules described above may be stored in a computer-readable medium on or near the computer 9900. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable medium, thereby providing the programs to the computer 9900 via the network.

[0118] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0119] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order. [Explanation of symbols]

[0120] 1. Control System 10 Equipment 15 Control Objects 20 Simulator 30 databases 100 control device 110 Model output acquisition unit 120 Indicator acquisition part 130 Correction condition setting section 140 Correction unit 150 control section 200 Operational Model Management Device 210 Control terminal / observation point setting unit 220 Goal Setting Department 230 Operation Model Learning Department 240 Operational Model Memory Unit 300 Index calculation device 310 Learning Settings 320 Monitoring Model Learning Unit 330 Monitoring model storage unit 910 Actual measurement value extraction unit 9900 Computer 9901 DVD-ROM 9910 Host Controller 9912 CPU 9914 RAM 9916 Graphics Controller 9918 Display Device 9920 Input / Output Controller 9922 Communication Interface 9924 Hard Disk Drive 9926 DVD drive 9930 ROM 9940 I / O chip 9942 keyboard

Claims

1. a model output acquisition unit that acquires an operation model output that is output in response to inputting status data indicating the status of the equipment into an operation model that has been machine-learned to output an action according to the status of the equipment using simulation data from a simulator that simulates operations in the equipment; an index acquisition unit that acquires an index for monitoring a difference between the behavior of the simulator and an actual operation of the facility, the difference being calculated using actual data from the facility; a correction unit that corrects the operation model output based on the index; a control unit that outputs an operation amount for controlling a control target provided in the facility in accordance with the corrected operation model output; A control device comprising:

2. 2. The control device according to claim 1, wherein the index acquisition unit acquires, as the index, a monitoring model output that is output in response to inputting the status data into a monitoring model that has been machine-learned using the actual data to output the index according to the status of the equipment.

3. The control device according to claim 2 , wherein the monitoring model is a machine learning model that has undergone supervised learning so that a difference between an output of the monitoring model and a teacher signal becomes small.

4. The control device according to claim 3 , further comprising a supervisory model learning unit that generates the supervisory model through the supervised learning.

5. The control device according to claim 1 , wherein the operation model is a machine learning model that has undergone reinforcement learning to output an action that increases a reward value determined by a reward function.

6. The control device according to claim 5 , further comprising an operation model learning unit that generates the operation model by the reinforcement learning.

7. The control device according to claim 1 , wherein the index acquisition unit acquires, as the index, a future predicted value of the sensor included in the status data or a future predicted value of the sensor that can be estimated from the status data.

8. The control device according to claim 1 , wherein the index acquisition unit acquires, as the index, a value that quantifies the quality of a product or semi-finished product manufactured in the facility.

9. The control device according to claim 1 , further comprising a correction condition setting unit that sets conditions for correcting the operation model output.

10. The conditions include an allowable range for the index, The control device according to claim 9 , wherein the correction unit corrects the operation model output when the index is outside the range so that the index approaches the range.

11. The control device according to claim 1 , wherein the index acquisition unit acquires, as the index, a calculated value calculated by the simulator.

12. The control device according to claim 11 , wherein the correction unit corrects the operational model output using a correction function that includes a difference between the actual measurement value indicated by the status data and the calculated value in a correction term.

13. The method is executed by a computer, and the computer acquiring an operational model output that is output in response to inputting status data indicating the status of the equipment into an operational model that has been machine-learned to output an action according to the status of the equipment using simulation data from a simulator that simulates operations in the equipment; obtaining an index for monitoring a difference between the behavior of the simulator and the actual operation of the facility, the index being calculated using actual data from the facility; correcting the operational model output based on the index; outputting a manipulated variable for controlling a control target provided in the facility in accordance with the corrected operation model output; A control method comprising:

14. The method is executed by a computer, causing the computer to: a model output acquisition unit that acquires an operation model output that is output in response to inputting status data indicating the status of the equipment into an operation model that has been machine-learned to output an action according to the status of the equipment using simulation data from a simulator that simulates operations in the equipment; an index acquisition unit that acquires an index for monitoring a difference between the behavior of the simulator and an actual operation of the facility, the difference being calculated using actual data from the facility; a correction unit that corrects the operation model output based on the index; a control unit that outputs an operation amount for controlling a control target provided in the facility in accordance with the corrected operation model output; A control program that makes the device function.

Images