Estimation device, estimation method, and estimation program

Abstract

SOLUTION: There is provided an estimation apparatus including an abnormality index acquisition unit configured to acquire, as an abnormality index, an evaluation index which is output by an evaluation model according to state data being input when an abnormality occurs in a facility, among pieces of the state data indicating a state of the facility when an operation model is used to control a control target provided in the facility, the operation model being generated by reinforcement learning in which an output of the evaluation model trained by machine learning to output the evaluation index in accordance with the state of the facility, is set as at least a part of a reward, and outputting an action in accordance with the state of the facility; an estimation unit configured to estimate which of the evaluation model or the operation model is a main cause of the abnormality, based on the abnormality index; and an output unit configured to execute an output in accordance with a result of the estimate.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an estimation device, an estimation method, and an estimation program. [Background technology]

[0002] Patent Document 1 states that "the model 45 outputs a recommended control parameter indicating a first type of control content recommended to increase a reward value in response to input of measurement data." In addition, Non-Patent Document 1 describes "FKDPP (Factoral Kernel Dynamic Policy Programming)." [Prior art document] [Patent documents] [Patent Document 1] JP2021-086283 [Non-patent literature] [Non-Patent Document 1] "Yokogawa Electric and NAIST Reinforcement Learning for Chemical Plants," Nikkei Robotics, March 2019 issue Summary of the Invention

[0003] In a first aspect of the present invention, there is provided an estimation device, which includes: an abnormality index acquisition unit that acquires, as an abnormality index, the evaluation index output by the evaluation model in response to input of state data in the case where an abnormality occurs in the equipment, among state data indicating the state of the equipment when a control target provided in the equipment is controlled using an operation model that is generated by reinforcement learning using, as at least a part of a reward, an output of an evaluation model that has been machine-learned to output an evaluation index corresponding to a state of the equipment, and that outputs an action corresponding to the state of the equipment; an estimation unit that estimates, based on the abnormality index, whether the main cause of the abnormality is the evaluation model or the operation model; and an output unit that executes output according to the estimated result.

[0004] In the estimation device, the estimation unit may estimate that a main cause of the abnormality is the operation model when the abnormality indicator does not satisfy a predetermined criterion.

[0005] In any of the estimation devices, the output unit may output a command to re-learn the operational model when it is estimated that the main cause of the abnormality is the operational model.

[0006] In any of the estimation devices, the estimation unit may estimate that a main cause of the abnormality is the evaluation model when the abnormality indicator satisfies a predetermined criterion.

[0007] In any of the estimation devices, the output unit may output a command to re-learn the evaluation model when it is estimated that the main cause of the abnormality is the evaluation model.

[0008] In the estimation device, when the output unit outputs an instruction to re-learn the evaluation model, it may also output an instruction to re-learn the operation model using the output of the re-learned evaluation model as at least a part of the remuneration.

[0009] Any of the estimation devices may further include a state data acquisition unit that acquires the state data.

[0010] Any of the estimation devices may further include an abnormality detection unit that detects that an abnormality has occurred in the equipment based on the status data.

[0011] Any of the estimation devices may further include a control unit that controls the control target using the operation model.

[0012] Any of the estimation devices may further include an operation model generation unit that generates the operation model by reinforcement learning.

[0013] Any of the estimation devices may further include an evaluation model generation unit that generates the evaluation model by machine learning.

[0014] In a second aspect of the present invention, there is provided an estimation method, which is executed by a computer and includes: acquiring, as an abnormality index, the evaluation index output by the evaluation model in response to input of status data in the case where an abnormality occurs in the equipment, among status data indicating the status of the equipment when a control target provided in the equipment is controlled using an operation model that is generated by reinforcement learning using, as at least a part of a reward, an output of an evaluation model that has been machine-learned to output an evaluation index corresponding to a status of the equipment and that outputs an action corresponding to the status of the equipment; estimating, based on the abnormality index, whether the main cause of the abnormality is the evaluation model or the operation model; and executing an output corresponding to the estimated result.

[0015] In a third aspect of the present invention, there is provided an estimation program, which is executed by a computer and causes the computer to function as an abnormality index acquisition unit that acquires, as an abnormality index, the evaluation index output by the evaluation model in response to input of status data in the case where an abnormality occurs in the equipment, among status data indicating the status of the equipment when a control target provided in the equipment is controlled using an operation model that is generated by reinforcement learning using, as at least a part of a reward, an output of an evaluation model that has been machine-learned to output an evaluation index corresponding to the status of the equipment and that outputs an action corresponding to the status of the equipment, an estimation unit that estimates, based on the abnormality index, whether the main cause of the abnormality is the evaluation model or the operation model, and an output unit that executes an output corresponding to the estimated result.

[0016] The above summary of the invention does not list all of the features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0017] [Figure 1] 1 shows an example of a block diagram of a control system 1. [Diagram 2]1 shows an example of a block diagram of an evaluation model management device 200. [Diagram 3] 1 shows an example of a block diagram of an operation model management device 300. [Figure 4] 1 shows an example of a block diagram of an estimation device 400 according to the present embodiment. [Diagram 5] An example of a block diagram of a control device 500 is shown. [Figure 6] 1 shows an example of a flow diagram of an estimation method that may be executed by the estimation device 400 according to the present embodiment. [Figure 7] FIG. 1 shows an example of a block diagram of an estimation device 400 according to a first modified example. [Figure 8] FIG. 13 shows an example of a block diagram of an estimation device 400 according to a second modified example. [Figure 9] FIG. 13 shows an example of a block diagram of an estimation device 400 according to a third modified example. [Figure 10] FIG. 13 shows an example of a block diagram of an estimation device 400 according to a fourth modified example. [Figure 11] An example of a computer 9900 in which aspects of the present invention may be embodied, in whole or in part, is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0019] FIG. 1 shows an example of a block diagram of a control system 1. These blocks are functionally separated functional blocks and do not necessarily correspond to the actual device configuration. In other words, even if one block is shown in this diagram, it does not necessarily have to be composed of one device. Also, even if separate blocks are shown in this diagram, they do not necessarily have to be composed of separate devices. The same applies to the subsequent block diagrams.

[0020] In the control system 1, an evaluation model that outputs an evaluation index according to the state of the equipment 10 is generated by machine learning, and an operation model is generated by reinforcement learning using the output of the evaluation model as at least a part of the reward. Then, in the control system 1, the generated operation model is used to control the control target 15 in the equipment 10. Such control using the operation model is also called AI (Artificial Intelligence) control. In the control system 1 in which a plurality of different machine learning models, namely the evaluation model and the operation model, are used, when an abnormality occurs in the equipment 10, the estimation device 400 according to this embodiment estimates whether the main cause of the abnormality is the evaluation model or the operation model.

[0021] The control system 1 may include a facility 10, a simulator 100, an evaluation model management device 200, an operation model management device 300, an estimation device 400, and a control device 500.

[0022] The equipment 10 is a facility or device in which a control target 15 is installed. For example, the equipment 10 may be a plant, or a composite device in which a plurality of devices are combined. Examples of the plant include industrial plants such as chemical and bio plants, plants that manage and control wellheads and surrounding areas of gas fields and oil fields, plants that manage and control power generation such as hydroelectric, thermal and nuclear power, plants that manage and control environmental power generation such as solar and wind power, and plants that manage and control water supply and sewage systems, dams, etc.

[0023] From here on, a case where the equipment 10 is a distillation apparatus, which is one of the process equipment, will be described as an example. In general, a distillation apparatus evaporates low boiling components in a distillation tower and extracts them from the top of the tower, condenses the extracted vapor of the low boiling components in a condenser, and stores it in a reflux drum. The distillation apparatus then refluxes a portion of the vapor stored in the reflux drum into the distillation tower, where it comes into contact with the vapor in the distillation tower and is distilled into low boiling components and high boiling components. In such a distillation apparatus, as an example, a valve provided between the reflux drum and the distillation tower is controlled to open and close in order to control the reflux amount.

[0024] The controlled object 15 is a device provided in the equipment 10 and is a control target. For example, the controlled object 15 may be an actuator, such as a valve, heater, motor, fan, or switch, that controls at least one physical quantity, such as the amount, temperature, pressure, flow rate, speed, or pH, of an object in the process of the equipment 10, that is, an operating end, and executes a given operation according to the operating amount. From here on, a case where the controlled object 15 is a valve provided between a reflux drum and a distillation column in a distillation apparatus will be described as an example. However, this is not limited to this. The controlled object 15 may be a controller that controls the operating end. In other words, the term "control" used in this specification may be broadly interpreted to include indirect control of an operating end via a controller in addition to direct control of an operating end.

[0025] The equipment 10 in which the control object 15 is provided may be provided with one or more sensors capable of measuring various conditions (physical quantities) inside and outside the equipment 10. As an example, when the equipment 10 is a distillation apparatus, the sensor may output a measured value PV (Process Variable) measuring the temperature at various positions (e.g., the top, center, bottom, etc.) of the distillation apparatus, the flow rate in various paths, etc. The state data indicating the state of the equipment 10 may include such a measured value PV. The state data may also include a manipulated variable MV (Manipulated Variable) indicating the opening and closing degree of a valve which is the control object 15. In addition to the operation data indicating the operating state as a result of controlling the control object 15 in this way, the state data may also include consumption data indicating the consumption of energy and raw materials in the equipment 10, disturbance environment data indicating physical quantities which may act as disturbances on the control of the control object 15, and the like.

[0026] Distillation units are one of the most widely used devices in petroleum and chemical processes, but they have the characteristics of strong mutual interference between the top and bottom of the column, a long time constant, and nonlinear operation. When controlling the opening and closing of a valve to control the reflux amount in such a distillation unit using PID (Proportional Integral Differential) or other methods, it is difficult to improve controllability. In addition, when such valves are manually operated by workers for multiple purposes such as quality assurance, energy saving, GHG (Green House Gas) reduction, and yield improvement, the extent to which the valves are opened and closed is largely dependent on the experience and intuition of the workers.

[0027] Therefore, when controlling the opening and closing of such a valve, it is conceivable to generate an evaluation model that outputs an evaluation index by machine learning, generate an operation model by reinforcement learning using the output of the evaluation model as at least a part of the reward, and perform AI control using the operation model. The estimation device 400 according to this embodiment may be applied to the control system 1 in which a plurality of different machine learning models, such as an evaluation model and an operation model, are used.

[0028] The simulator 100 simulates an operation in the facility 10. For example, the simulator 100 may be designed based on design information for the facility 10, and executes behavior simulating the operation in the facility 10. The simulator 100 changes the environment by acquiring a signal simulating an operation amount for the control object 15, and outputs simulation data simulating a state in the facility 10 (for example, a predicted value of a sensor). As an example, the simulator 100 may be composed of a prediction model predicting a state of a distillation apparatus and a plant control simulator. The prediction model may be capable of predicting a state change of a reactor from accumulated process data using a time-series data modeling technique using deep learning. In addition, the plant control simulator may be capable of virtually simulating PID control for the control object 15, which derives an operation amount MV by the difference between a target value SV and a control amount CV. That is, the simulator 100 may be capable of simulating the behavior itself in the facility 10 in addition to the state prediction value.

[0029] The evaluation model management device 200 manages an evaluation model that outputs an evaluation index according to the state of the facility 10. For example, the evaluation model management device 200 may generate an evaluation model by machine learning and store the generated evaluation model in its own device. In addition, the evaluation model management device 200 may output the generated evaluation model to the operation model management device 300.

[0030] The operation model management device 300 manages an operation model that outputs an action according to the state of the facility 10. For example, the operation model management device 300 may generate an operation model by reinforcement learning using the output of the evaluation model managed by the evaluation model management device 200 as at least a part of the reward, and store the generated operation model in the device itself. In addition, the operation model management device 300 may output the generated operation model to the control device 500.

[0031] In a control system 1 in which a plurality of different machine learning models, namely, an evaluation model and an operation model, are used, when an abnormality occurs in a facility 10, the estimation device 400 estimates whether the main cause of the abnormality is the evaluation model or the operation model. Then, the estimation device 400 executes output according to the estimation result.

[0032] The control device 500 uses the operation model to control the control target 15. For example, the control device 500 may use the operation model managed by the operation model management device 300 to control the control target 15 in the facility 10.

[0033] In this way, in the control system 1, the AI ​​automatically finds bottlenecks (potential faults) in operations and generates indicators for improvement as an evaluation model. Then, the AI ​​performs trial and error based on the given indicators to generate an operation model that indicates a better operation method. As a result, the control system 1 provides an environment in which the equipment 10 can be autonomously controlled using AI technology. In the control system 1 in which a plurality of different machine learning models, such as an evaluation model and an operation model, are used, when an abnormality occurs in the equipment 10, the estimation device 400 according to this embodiment estimates whether the main cause of the abnormality is the evaluation model or the operation model. Regarding this, the details of each device will be described in order.

[0034] FIG. 2 shows an example of a block diagram of the evaluation model management device 200. The evaluation model management device 200 may be a computer such as a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, or a general-purpose computer, or may be a computer system to which multiple computers are connected. Such a computer system is also a computer in a broad sense. The evaluation model management device 200 may be implemented by a virtual computer environment in which one or more programs can be executed within a computer. Alternatively, the evaluation model management device 200 may be a dedicated computer designed for managing evaluation models, or may be dedicated hardware realized by a dedicated circuit. In addition, if the evaluation model management device 200 is connectable to the Internet, it may be realized by cloud computing.

[0035] The evaluation model management device 200 includes an evaluation model generation unit 210 , an evaluation model storage unit 220 , and an evaluation model output unit 230 .

[0036] The evaluation model generating unit 210 generates an evaluation model that outputs an evaluation index according to the state of the equipment 10. For example, the evaluation model generating unit 210 may obtain the operation target (such as a plant KPI (Key Performance Indicator)) in the equipment 10, the state data indicating the state of the equipment 10, and the teacher label, and generate labeling data based on these. Then, the evaluation model generating unit 210 may generate an evaluation model by a machine learning algorithm using the generated labeling data as learning data. When the state of the equipment 10 is evaluated in light of the set operation target, such an evaluation model may output a larger value as the evaluation index as the evaluation result is better, and may output a smaller value as the evaluation index as the evaluation result is worse. The generation process of the evaluation model itself may be arbitrary, and further details will not be described here. The evaluation model generating unit 210 supplies the generated evaluation model to the evaluation model storage unit 220.

[0037] The evaluation model storage unit 220 stores an evaluation model. For example, the evaluation model storage unit 220 may store an evaluation model generated by the evaluation model generation unit 210. In the above description, the evaluation model storage unit 220 stores an evaluation model generated inside the evaluation model management device 200 as an example, but the present invention is not limited to this. The evaluation model storage unit 220 may store an evaluation model generated outside the evaluation model management device 200. The evaluation model storage unit 220 copies the stored evaluation model and supplies it to the evaluation model output unit 230.

[0038] The evaluation model output unit 230 outputs the evaluation model. For example, the evaluation model output unit 230 may output the evaluation model copied by the evaluation model storage unit 220 to the operation model management device 300 via a network.

[0039] FIG. 3 shows an example of a block diagram of the operation model management device 300. As with the evaluation model management device 200, the operation model management device 300 may be a computer, or a computer system to which multiple computers are connected. The operation model management device 300 may be implemented by a virtual computer environment in which one or more programs can be executed within a computer. Alternatively, the operation model management device 300 may be a dedicated computer designed for managing the operation model, or may be dedicated hardware realized by a dedicated circuit. In addition, if the operation model management device 300 is connectable to the Internet, it may be realized by cloud computing.

[0040] The operation model management device 300 includes an evaluation model acquisition unit 310 , an operation model generation unit 320 , an operation model storage unit 330 , and an operation model output unit 340 .

[0041] The evaluation model acquisition unit 310 acquires an evaluation model that outputs an evaluation index according to the state of the equipment 10. For example, the evaluation model acquisition unit 310 may acquire the evaluation model output from the evaluation model output unit 230 via a network. The evaluation model acquisition unit 310 supplies the acquired evaluation model to the operation model generation unit 320.

[0042] The operation model generation unit 320 generates an operation model that outputs an action according to the state of the equipment 10 by reinforcement learning using the output of the evaluation model as at least a part of the reward. As an example, such an operation model may have a data table consisting of a combination (S, A) of S indicating a set of sampled state data and an action A taken in each state, and a weight W calculated by the reward. Note that the output of the evaluation model may be used as at least a part of the reward for calculating such weight W.

[0043] When generating such an operation model, the operation model generating unit 320 may acquire learning environment data indicating the state of the learning environment. In this case, when a simulator 100 that simulates operations in the facility 10 is used as the learning environment, the operation model generating unit 320 may acquire simulation data from the simulator 100 as the learning environment data. However, this is not limited to this. The actual facility 10 may be used as the learning environment. In this case, the operation model generating unit 320 may acquire state data indicating the state of the facility 10 as the learning environment data.

[0044] Next, the operation model generation unit 320 may determine an action randomly or by using a known AI algorithm such as FKDPP (described later) and provide an operation amount based on the action to the control target in the learning environment. The state of the learning environment changes accordingly.

[0045] Then, the operation model generating unit 320 may acquire the learning environment data again, thereby enabling the operation model generating unit 320 to acquire the state of the learning environment after it has changed in response to the application of the operation amount based on the determined action to the control target.

[0046] The operation model generator 320 may calculate the reward value based at least in part on the output of the evaluation model. As an example, in response to inputting learning environment data indicating the state of the learning environment after the change to the evaluation model, the operation model generator 320 may calculate the reward value as the index output by the evaluation model as it is.

[0047] The operation model generating unit 320 may update the operation model by overwriting the values ​​of the weight column in the data table and adding new sample data that has not been saved to a new row in the data table after repeating such a process of acquiring the state according to the decision of the action multiple times. The operation model generating unit 320 can generate the operation model by repeating such an update process multiple times. Since the generation of the operation model itself may be optional, further details will not be described here. The operation model generating unit 320 supplies the generated operation model to the operation model storage unit 330.

[0048] The operation model storage unit 330 stores an operation model. For example, the operation model storage unit 330 may store an operation model generated by the operation model generation unit 320. In the above description, the operation model storage unit 330 stores an operation model generated inside the operation model management device 300 as an example, but the present invention is not limited to this. The operation model storage unit 330 may store an operation model generated outside the operation model management device 300. The operation model storage unit 330 copies the stored operation model and supplies it to the operation model output unit 340.

[0049] The operation model output unit 340 outputs the operation model. For example, the operation model output unit 340 may output the operation model copied by the operation model storage unit 330 to the control device 500 via a network.

[0050] FIG. 4 shows an example of a block diagram of the estimation device 400 according to this embodiment. As with the evaluation model management device 200, the estimation device 400 may be a computer, or a computer system to which multiple computers are connected. The estimation device 400 may be implemented by a virtual computer environment in which one or more programs can be executed within a computer. Alternatively, the estimation device 400 may be a dedicated computer designed for estimating anomaly factors, or may be dedicated hardware realized by a dedicated circuit. In addition, if the estimation device 400 is connectable to the Internet, it may be realized by cloud computing.

[0051] The estimation device 400 includes a state data acquisition unit 410 , an abnormality index acquisition unit 420 , an estimation unit 430 , and an output unit 440 .

[0052] The state data acquisition unit 410 acquires state data indicating the state of the equipment 10. Here, the state data is data indicating the state of the equipment 10 when the control target 15 is AI-controlled using the evaluation model and the operation model. Therefore, the state data acquisition unit 410 may acquire state data indicating the state of the equipment 10 when the control target 15 provided in the equipment 10 is controlled using an operation model that is generated by reinforcement learning using as at least a part of the reward the output of an evaluation model that has been machine-learned to output an evaluation index according to the state of the equipment 10, and that outputs an action according to the state of the equipment 10. The state data acquisition unit 410 supplies the acquired state data to the abnormality index acquisition unit 420.

[0053] The abnormality index acquisition unit 420 acquires an abnormality index. For example, the abnormality index acquisition unit 420 may acquire, as an abnormality index, an evaluation index output by an evaluation model in response to input of state data in the case where an abnormality occurs in the equipment 10, out of state data indicating the state of the equipment 10 when the control target 15 provided in the equipment 10 is controlled using an operation model that is generated by reinforcement learning using, as at least a part of a reward, an output of an evaluation model that has been machine-learned to output an evaluation index according to the state of the equipment 10 and that outputs an action according to the state of the equipment 10. The abnormality index acquisition unit 420 supplies the acquired abnormality index to the estimation unit 430.

[0054] The estimation unit 430 estimates the main cause of the abnormality. For example, the estimation unit 430 may estimate whether the main cause of the abnormality is the evaluation model or the operation model based on the abnormality indicator. The estimation unit 430 supplies the estimated result to the output unit 440.

[0055] The output unit 440 executes an output. For example, the output unit 440 may execute an output according to the estimated result. At this time, the output unit 440 may output the estimated result as it is, or may output an instruction to re-learn the evaluation model or the operation model, as described later.

[0056] 5 shows an example of a block diagram of the control device 500. The control device 500 may be, for example, a controller in a distributed control system (DCS) or a medium-scale instrumentation system, or may be a real-time OS controller or the like.

[0057] The control device 500 includes an operation model acquisition unit 510 , an actual environment data acquisition unit 520 , and a control unit 530 .

[0058] The operation model acquisition unit 510 acquires an operation model. For example, the operation model acquisition unit 510 may acquire, via a network, the operation model output by the operation model output unit 340. The operation model acquisition unit 510 supplies the acquired operation model to the control unit 530.

[0059] The real environment data acquiring unit 520 acquires real environment data indicating the real environment, i.e., the state of the equipment 10. Such real environment data may be similar to the state data indicating the state of the equipment 10. The real environment data acquiring unit 520 supplies the acquired real environment data to the control unit 530.

[0060] The control unit 530 controls the control target 15 using the operation model. For example, the control unit 530 may determine the action by a known AI algorithm such as FKDPP. When using such a kernel method, the control unit 530 may generate a vector of the state S from a sensor value obtained from the acquired real environment data (state data). Next, the control unit 530 may generate a combination of the state S and all possible actions A as a behavior decision table. Then, the control unit 530 may input the behavior decision table to the operation model. In response to this, the operation model may perform kernel calculation between each row of the behavior decision table and each sample data in the data table excluding the weight column, and calculate the distance between each sample data. Next, the operation model may sequentially add up the distances calculated for each sample data multiplied by the value of each weight column, and calculate the reward expectation value for each action. Then, the operation model may output the action with the highest reward expectation value among all possible actions. The control unit 530 may determine the action, for example, by selecting the action output by the operation model in this manner.

[0061] Then, the control unit 530 may provide the control object 15 with an operation amount obtained by adding the determined action to a current value (e.g., a current valve opening) of the control object 15. In this manner, for example, the control unit 530 can perform AI control of the control object 15 by using the operation model managed by the operation model management device 300.

[0062] FIG. 6 shows an example of a flow diagram of an estimation method that may be executed by the estimation device 400 according to this embodiment.

[0063] In step S610, the estimation device 400 acquires state data. For example, the state data acquisition unit 410 may acquire various physical quantities measured by various sensors provided in the facility 10 from the facility 10 via a network as state data. However, this is not limited to this. The state data acquisition unit 410 may acquire the state data via a means other than a network, or may acquire the state data from another device other than the facility 10.

[0064] Here, the status data is data indicating the status of the equipment 10 when the control target 15 is AI-controlled using the evaluation model and the operation model. Therefore, the status data acquisition unit 410 may acquire status data indicating the status of the equipment 10 when the control target 15 provided in the equipment 10 is controlled using an operation model that is generated by reinforcement learning using, as at least a part of the reward, the output of an evaluation model that has been machine-learned to output an evaluation index according to the status of the equipment 10, and that outputs an action according to the status of the equipment 10. The status data acquisition unit 410 supplies the acquired status data to the abnormality indicator acquisition unit 420.

[0065] In step S620, the estimation device 400 determines whether an abnormality has occurred. For example, the estimation device 400 may determine whether an abnormality has been detected in the facility 10 from the outside (another device or a user). If an abnormality has not been detected (No), the estimation device 400 returns the process to step S610 and continues the flow. That is, the estimation device 400 continues the process of acquiring state data by the state data acquisition unit 410. On the other hand, if an abnormality has been detected (Yes), the estimation device 400 advances the process to step S630. That is, the estimation device 400 triggers the process of acquiring an abnormality indicator by the abnormality indicator acquisition unit 420.

[0066] In step S630, the estimation device 400 acquires an abnormality index. For example, the abnormality index acquisition unit 420 may identify the time when the abnormality occurred based on a notification that an abnormality has been detected. Next, the abnormality index acquisition unit 420 may extract state data at the time when the abnormality occurred (for example, measured at the time when the abnormality occurred) from the state data acquired in step S610. Then, the abnormality index acquisition unit 420 may supply the state data at the time when the abnormality occurred to the evaluation model management device 200 via a network. In response to this, the evaluation model managed by the evaluation model management device 200 may input the state data at the time when the abnormality occurred and output an evaluation index. Then, the abnormality index acquisition unit 420 may acquire the evaluation index output by the evaluation model as an abnormality index from the evaluation model management device 200 via a network. The abnormality index acquisition unit 420 may acquire, as an abnormality index, an evaluation index output by the evaluation model in response to input of status data in the case where an abnormality occurs in the equipment 10, from status data indicating the status of the equipment 10 when the control target 15 provided in the equipment 10 is controlled using an operation model that is generated by reinforcement learning in which the output of an evaluation model that has been machine-learned to output an evaluation index corresponding to the status of the equipment 10 is at least a part of the reward, and that outputs an action corresponding to the status of the equipment 10.

[0067] In the above description, the estimation device 400 acquires the abnormality index by itself, extracts the state data when an abnormality occurs from the acquired state data, and supplies the state data when an abnormality occurs to the evaluation model management device 200. However, the present invention is not limited to this. Another device may acquire the state data, extract the state data when an abnormality occurs from the acquired state data, and supply the state data when an abnormality occurs to the evaluation model management device 200. Then, the estimation device 400 may acquire the evaluation index output by the evaluation model in response to this as the abnormality index. That is, the processes of steps S610 and S620 do not necessarily have to be executed by the estimation device 400, but may be executed by another device. In this case, the estimation device 400 may not include the state data acquisition unit 410. The abnormality index acquisition unit 420 supplies the acquired abnormality index to the estimation unit 430.

[0068] The estimation unit 430 may estimate whether the main cause of the abnormality is the evaluation model or the operation model, based on the abnormality index acquired in step S630, for example, in the following manner.

[0069] In step S640, the estimation device 400 determines whether the abnormality index satisfies a criterion. For example, the estimation unit 430 may determine whether the abnormality index acquired in step S630 satisfies a predetermined criterion. As an example, it is assumed that the better the evaluation result is when the state of the equipment 10 is evaluated in light of the set operation target, the larger the value of the evaluation index is output, and the worse the evaluation result is, the smaller the value of the evaluation index is output. In this case, the estimation unit 430 may determine whether the abnormality index is equal to or greater than a predetermined threshold. Then, when the abnormality index is less than the threshold, the estimation unit 430 may determine that the abnormality index does not satisfy the criterion (No). In this case, the estimation device 400 advances the process to step S650.

[0070] In step S650, the estimation device 400 estimates that the main cause of the abnormality is the operation model. In this manner, for example, when it is determined in step S640 that the abnormality indicator does not satisfy the predetermined criterion, the estimation unit 430 may estimate that the main cause of the abnormality is the operation model.

[0071] Here, the operation model is generated by reinforcement learning with the evaluation index output by the evaluation model as at least a part of the reward, and outputs the behavior that maximizes the expected value of the reward. Therefore, if the operation model is generated appropriately, when state data acquired under AI control using the operation model is input to the evaluation model, the evaluation index output from the evaluation model should be a large value. Nevertheless, if the abnormality index, that is, the evaluation index output by the evaluation model in response to input of state data at the time of abnormality occurrence, is less than the threshold, there is a high probability that the reinforcement learning for generating the operation model has some problem (lack of learning data, incompatibility of the learning algorithm, or inadequate setting of initial parameters, etc.) and the operation model was not generated appropriately. Then, it can be considered that an abnormality has occurred in the equipment 10 as a result of controlling the control target 15 based on the behavior output by such an operation model. Therefore, for example, in such a case, the estimation unit 430 may estimate that the main cause of the abnormality in the equipment 10 is the operation model. The estimation unit 430 supplies the estimated result to the output unit 440.

[0072] In step S660, the estimation device 400 outputs a command to re-learn the operation model. For example, when it is estimated in step S650 that the main cause of the abnormality is the operation model, the output unit 440 may output a command to re-learn the operation model. At this time, the output unit 440 may display and output the command to re-learn the operation model on a monitor, may output a voice from a speaker, or may print out the command to re-learn the operation model on a printer. In response to this, the user may instruct the operation model management device 300 to re-learn the operation model, and cause the operation model generation unit 320 to generate a new operation model. Alternatively or in addition to this, the output unit 440 may transmit and output a message to the operation model management device 300 to instruct the operation model to be re-learned. In response to this, the operation model generation unit 320 may trigger the re-learning of the operation model, and automatically generate a new operation model (without the intervention of the user).

[0073] In addition, when re-learning the operation model, the operation model generating unit 320 may change at least one of the learning data, the learning algorithm, and the initial parameters before re-learning the operation model. Therefore, when outputting a command to re-learn the operation model, the output unit 440 may also output a command to change at least one of the learning data, the learning algorithm, and the initial parameters.

[0074] On the other hand, in step S640, if the abnormality index is equal to or greater than the threshold, the estimation unit 430 may determine that the abnormality index satisfies the criterion (Yes), In this case, the estimation device 400 advances the process to step S670.

[0075] In step S670, the estimation device 400 estimates that the main cause of the abnormality is the evaluation model. In this manner, for example, when it is determined in step S640 that the abnormality indicator satisfies a predetermined criterion, the estimation unit 430 may estimate that the main cause of the abnormality is the evaluation model.

[0076] Here, the evaluation model is machine-learned so that, when the state of the equipment 10 is evaluated, the better the evaluation result, the larger the evaluation index is output, and the worse the evaluation result, the smaller the evaluation index is output. Therefore, when the evaluation index output by the evaluation model in response to input of state data is equal to or greater than the threshold, it means that the evaluation model has evaluated the state of the equipment 10 as relatively good. Nevertheless, if an abnormality occurs in the equipment 10, there is a high probability that there is some problem in the machine learning for generating the evaluation model, and the evaluation model was not properly generated. Then, it can be considered that an abnormality occurs in the equipment 10 as a result of AI control of the control target 15 using an operation model that has been reinforced learning with the output of the evaluation model that cannot properly evaluate the state of the equipment 10 as a reward. Therefore, for example, in such a case, the estimation unit 430 may estimate that the main cause of the abnormality in the equipment 10 is the evaluation model. The estimation unit 430 supplies the estimated result to the output unit 440.

[0077] In step S680, the estimation device 400 outputs a command to re-learn the evaluation model. For example, when it is estimated in step S670 that the main cause of the abnormality is the evaluation model, the output unit 440 may output a command to re-learn the evaluation model. At this time, the output unit 440 may display and output the command to re-learn the evaluation model on a monitor, may output a voice from a speaker, or may print out the command to re-learn the evaluation model on a printer. In response to this, the user may instruct the evaluation model management device 200 to re-learn the evaluation model, and cause the evaluation model generation unit 210 to generate a new evaluation model. Alternatively or in addition to this, the output unit 440 may transmit and output a message to instruct the evaluation model management device 200 to re-learn the evaluation model. In response to this, the evaluation model generation unit 210 may trigger the re-learning of the evaluation model, and automatically generate a new evaluation model (without the intervention of the user).

[0078] In addition, when re-learning the evaluation model, the evaluation model generation unit 210 may change at least one of the learning data (including the operation target and the teacher label), the learning algorithm, and the initial parameters before re-learning the evaluation model. Therefore, when outputting an instruction to re-learn the evaluation model, the output unit 440 may also output an instruction to change at least one of the learning data, the learning algorithm, and the initial parameters.

[0079] After step S680, the estimation device 400 may proceed to step S660. That is, the output unit 440 may output an instruction to re-learn the operation model. In this way, when the output unit 440 outputs an instruction to re-learn the evaluation model, it can also output an instruction to re-learn the operation model in which the output of the re-learned evaluation model is at least a part of the remuneration.

[0080] Machine learning models are generated by machine learning training data, and output appropriate results according to input. However, it is not possible to know on what basis (logic) the machine learning model derived the output result. This type of problem is also known as the black box problem. In the past, in a system in which multiple black box machine learning models were used, when an abnormality occurred, it was difficult to determine which machine learning model was the cause of the abnormality.

[0081] In contrast, the estimation device 400 according to the present embodiment acquires, as an abnormality index, an evaluation index output by the evaluation model in response to input of state data at the time of abnormality occurrence. Then, the estimation device 400 according to the present embodiment estimates whether the main cause of the abnormality is the evaluation model or the operation model based on the abnormality index, and executes output according to the estimation result. As a result, according to the estimation device 400 according to the present embodiment, even when multiple different machine learning models, that is, an evaluation model and an operation model, are used, it is possible to estimate and distinguish whether the main cause of the abnormality is on the evaluation model side or the operation model side.

[0082] The estimation device 400 according to the present embodiment may estimate that the main cause of the abnormality is the operation model when the abnormality indicator does not satisfy a predetermined standard. As a result, the estimation device 400 according to the present embodiment can estimate that the probability that the operation model is the main cause of the abnormality is high based on objective grounds. Furthermore, when the estimation device 400 according to the present embodiment estimates that the main cause of the abnormality is the operation model, it may output an instruction to re-learn the operation model. As a result, the estimation device 400 according to the present embodiment can provide a trigger to re-learn the operation model when the probability that the operation model is the main cause of the abnormality is high.

[0083] The estimation device 400 according to the present embodiment may estimate that the main cause of the abnormality is the evaluation model when the abnormality index satisfies a predetermined criterion. As a result, the estimation device 400 according to the present embodiment can estimate that the evaluation model is highly likely to be the main cause of the abnormality based on objective grounds. In addition, the estimation device 400 according to the present embodiment may output an instruction to re-learn the evaluation model when it is estimated that the main cause of the abnormality is the evaluation model. As a result, the estimation device 400 according to the present embodiment can provide a trigger to re-learn the evaluation model when the evaluation model is highly likely to be the main cause of the abnormality. In addition, the estimation device 400 according to the present embodiment may output an instruction to re-learn the operation model in which the output of the re-learned evaluation model is at least a part of the remuneration when outputting an instruction to re-learn the evaluation model. As a result, the estimation device 400 according to the present embodiment can provide a trigger to re-learn the operation model in addition to re-learning the evaluation model.

[0084] Fig. 7 shows an example of a block diagram of an estimation device 400 according to a first modified example. In Fig. 7, the same reference numerals are used for components having the same functions and configurations as those in Fig. 4, and descriptions thereof will be omitted except for the differences. In the above-described embodiment, a case in which the estimation device 400 judges an abnormality in the facility 10 based on a notification from the outside has been shown as an example. However, in this modified example, the estimation device 400 detects an abnormality in the facility 10 by itself.

[0085] The estimation device 400 according to this modification may further include an abnormality detection unit 710. In addition, in the estimation device 400 according to this modification, the state data acquisition unit 410 may supply the acquired state data to the abnormality detection unit 710 instead of the abnormality index acquisition unit 420.

[0086] The anomaly detection unit 710 detects that an anomaly has occurred in the equipment 10 based on the state data. At this time, the anomaly detection unit 710 may detect that an anomaly has occurred in the equipment 10 using a predetermined anomaly detection function, or may detect that an anomaly has occurred in the equipment 10 using an anomaly detection model that has been machine-learned in advance. The anomaly detection process itself may be arbitrary, and further details will not be described here.

[0087] When the abnormality detection unit 710 detects that an abnormality has occurred in the equipment 10 based on the status data, it may supply the status data that led to the detection of the abnormality to the abnormality indicator acquisition unit 420 as status data at the time of the abnormality occurrence.

[0088] In this way, the estimation device 400 according to this modification detects by itself that an abnormality has occurred in the equipment 10 based on the status data. As a result, the estimation device 400 according to this modification can realize a function of detecting an abnormality and a function of estimating the main cause of the abnormality by a single device. Therefore, the estimation device 400 according to this modification can eliminate the need for an external notification for an abnormality determination that triggers acquisition of an abnormality indicator.

[0089] FIG. 8 shows an example of a block diagram of an estimation device 400 according to a second modified example. In FIG. 8, the same reference numerals are used for components having the same functions and configurations as those in FIG. 4, and descriptions thereof will be omitted except for the differences. In the above-described embodiment, the evaluation model management device 200, the operation model management device 300, the estimation device 400, and the control device 500 are provided as separate independent devices. However, these devices may be provided as a single device in which some or all of them are integrated. In this modified example, the estimation device 400 provides the functions of the control device 500 in addition to the functions of the estimation device 400 according to the above-described embodiment.

[0090] The estimation device 400 according to this modification may further include an operation model acquisition unit 510 and a control unit 530. That is, the estimation device 400 according to this modification may further include the control unit 530 that controls the control target 15 using the operation model.

[0091] Furthermore, the state data acquiring section 410 may supply the acquired state data to the control section 530 in addition to the abnormality indicator acquiring section 420. That is, in the estimation device 400 according to this modification, the state data acquiring section 410 may also function as the actual environment data acquiring section 520. Then, the control section 530 may control the control target 15 using the operation model. In this way, for example, the estimation device 400 may also provide the function of the control device 500.

[0092] Furthermore, when the estimation device 400 also provides a function as the control device 500, the output unit 440 may transmit a message to the control unit 530 to suspend control of the control target 15 in response to outputting an instruction to re-learn the evaluation model or the operation model. In this way, the output unit 440 can also temporarily suspend control of the control target 15 by the control unit 530.

[0093] In this way, the estimation device 400 according to this modification can also use the operation model to control the control target 15. As a result, the estimation device 400 according to this modification can realize, by a single device, a function of estimating the main cause of an abnormality and a function of controlling the control target 15.

[0094] Fig. 9 shows an example of a block diagram of an estimation device 400 according to a third modified example. In Fig. 9, the same reference numerals are used for components having the same functions and configurations as those in Fig. 8, and descriptions thereof will be omitted except for the differences. In the second modified example, the estimation device 400 also provides the functions of the control device 500 as an example. However, in this modified example, the estimation device 400 also provides the functions of the operation model management device 300 in addition to the functions of the control device 500.

[0095] The estimation device 400 according to this modification may further include an evaluation model acquisition unit 310. Moreover, the estimation device 400 according to this modification may further include an operation model generation unit 320 instead of the operation model acquisition unit 510. That is, the estimation device 400 according to this modification may further include an operation model generation unit 320 that generates an operation model by reinforcement learning. Then, the operation model generation unit 320 may supply the generated operation model to the control unit 530.

[0096] Furthermore, when the estimation device 400 also provides the function of the operational model management device 300, the output unit 440 can also send a message to the operational model generation unit 320 instructing re-learning of the operational model. In response to this, the operational model generation unit 320 can also generate a new operational model.

[0097] In this way, the estimation device 400 according to this modification can also generate an operation model by reinforcement learning. As a result, the estimation device 400 according to this modification can realize a function of managing (generating) an operation model, a function of estimating a main cause of an abnormality, and a function of controlling the control target 15, by a single device. As a result, the estimation device 400 according to this modification does not need to exchange an operation model between the operation model management device 300 and the control device 500, so that communication costs and time can be reduced.

[0098] Fig. 10 shows an example of a block diagram of an estimation device 400 according to a fourth modified example. In Fig. 10, the same reference numerals are used for components having the same functions and configurations as those in Fig. 9, and descriptions thereof will be omitted except for the differences. In the third modified example, the estimation device 400 provides the functions of the control device 500 and the operation model management device 300 as well. However, in this modified example, the estimation device 400 provides the functions of the evaluation model management device 200 in addition to the functions of the control device 500 and the operation model management device 300.

[0099] The estimation device 400 according to this modification may further include an evaluation model generation unit 210 instead of the evaluation model acquisition unit 310. That is, the estimation device 400 according to this modification may further include the evaluation model generation unit 210 that generates an evaluation model by machine learning. Then, the evaluation model generation unit 210 may supply the generated evaluation model to the operation model generation unit 320.

[0100] Furthermore, when the estimation device 400 also provides the function of the evaluation model management device 200, the output unit 440 can also send a message to the evaluation model generation unit 210 instructing re-learning of the evaluation model. In response to this, the evaluation model generation unit 210 can also generate a new evaluation model.

[0101] In this way, the estimation device 400 according to this modification can also generate an evaluation model by machine learning. As a result, according to the estimation device 400 according to this modification, a function for managing (generating) an evaluation model, a function for managing (generating) an operation model, a function for estimating a main cause of an abnormality, and a function for controlling the control target 15 can be realized by one device. As a result, according to the estimation device 400 according to this modification, it is not necessary to exchange an evaluation model between the evaluation model management device 200 and the operation model management device 300, so that communication costs and time can be reduced. In addition, according to the estimation device 400 according to this modification, it is not necessary to exchange state data or evaluation indexes between the evaluation model management device 200 and the estimation device 400 when acquiring an abnormality index, so that communication costs and time can be reduced.

[0102] Various embodiments of the present invention may be described with reference to flow charts and block diagrams, where the blocks may represent (1) stages of a process in which operations are performed or (2) sections of an apparatus responsible for performing the operations. Particular stages and sections may be implemented by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable medium, and / or a processor provided with computer readable instructions stored on a computer readable medium. Dedicated circuitry may include digital and / or analog hardware circuitry and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, memory elements such as flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

[0103] A computer-readable medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that the computer-readable medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray (RTM) disks, memory sticks, integrated circuit cards, and the like.

[0104] The computer readable instructions may include either assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0105] Computer readable instructions may be provided to a processor or programmable circuitry of a general purpose computer, special purpose computer, or other programmable data processing apparatus, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to execute the computer readable instructions to create means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0106] 11 illustrates an example of a computer 9900 in which aspects of the present invention may be embodied in whole or in part. Programs installed on the computer 9900 may cause the computer 9900 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to an embodiment of the present invention, and / or to perform a process or steps of a process according to an embodiment of the present invention. Such programs may be executed by the CPU 9912 to cause the computer 9900 to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0107] The computer 9900 according to this embodiment includes a CPU 9912, a RAM 9914, a graphic controller 9916, and a display device 9918, which are connected to each other by a host controller 9910. The computer 9900 also includes input / output units such as a communication interface 9922, a hard disk drive 9924, a DVD drive 9926, and an IC card drive, which are connected to the host controller 9910 via an input / output controller 9920. The computer also includes legacy input / output units such as a ROM 9930 and a keyboard 9942, which are connected to the input / output controller 9920 via an input / output chip 9940.

[0108] The CPU 9912 operates according to programs stored in the ROM 9930 and the RAM 9914, thereby controlling each unit. The graphics controller 9916 retrieves image data generated by the CPU 9912 into a frame buffer or the like provided in the RAM 9914 or into itself, and causes the image data to be displayed on the display device 9918.

[0109] The communication interface 9922 communicates with other electronic devices over a network. The hard disk drive 9924 stores programs and data used by the CPU 9912 in the computer 9900. The DVD drive 9926 reads programs or data from the DVD-ROM 9901 and provides the programs or data to the hard disk drive 9924 via the RAM 9914. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0110] The ROM 9930 stores therein a boot program or the like executed by the computer 9900 upon activation, and / or a program that depends on the hardware of the computer 9900. The input / output chip 9940 may also connect various input / output units to the input / output controller 9920 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0111] The programs are provided by a computer-readable medium such as a DVD-ROM 9901 or an IC card. The programs are read from the computer-readable medium, installed in the hard disk drive 9924, the RAM 9914, or the ROM 9930, which are also examples of computer-readable media, and executed by the CPU 9912. Information processing described in these programs is read by the computer 9900, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing information manipulation or processing according to the use of the computer 9900.

[0112] For example, when communication is performed between the computer 9900 and an external device, the CPU 9912 may execute a communication program loaded in the RAM 9914 and instruct the communication interface 9922 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 9912, the communication interface 9922 reads transmission data stored in a transmission buffer processing area provided in the RAM 9914, the hard disk drive 9924, the DVD-ROM 9901, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0113] The CPU 9912 may also cause all or a necessary portion of a file or database stored on an external recording medium such as the hard disk drive 9924, the DVD drive 9926 (DVD-ROM 9901), an IC card, etc. to be read into the RAM 9914, and perform various types of processing on the data on the RAM 9914. The CPU 9912 then writes back the processed data to the external recording medium.

[0114] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 9912 may perform various types of processing on the data read from the RAM 9914, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 9914. The CPU 9912 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 9912 may search for an entry that matches a condition, in which an attribute value of the first attribute is specified, from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0115] The above-described programs or software modules may be stored in a computer-readable medium on the computer 9900 or in the vicinity of the computer 9900. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the programs to the computer 9900 via the network.

[0116] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0117] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order. [Explanation of symbols]

[0118] 1. Control System 10 Equipment 15 Control Object 100 Simulator 200 Evaluation model management device 210 Evaluation model generation unit 220 Evaluation model memory unit 230 Evaluation model output section 300 Operational model management device 310 Evaluation Model Acquisition Department 320 Operational Model Generation Unit 330 Operational Model Memory Unit 340 Operational Model Output Section 400 Estimator 410 Status data acquisition unit 420 Abnormality index acquisition unit 430 Estimation Department 440 Output section 500 Control device 510 Operational Model Acquisition Department 520 Real-world data acquisition unit 530 Control Unit 710 Anomaly detection unit 9900 Computer 9901 DVD-ROM 9910 Host Controller 9912 CPU 9914 RAM 9916 Graphics Controller 9918 Display Device 9920 Input / Output Controller 9922 Communication Interface 9924 Hard Disk Drive 9926 DVD Drive 9930 ROM 9940 Input / Output Chip 9942 Keyboard

Claims

1. an abnormality index acquisition unit that acquires, as an abnormality index, the evaluation index output by the evaluation model in response to input of status data in the event of an abnormality occurring in the equipment, among status data indicating the status of the equipment when a control target provided in the equipment is controlled using an operation model that outputs an action in accordance with the status of the equipment, the evaluation index being generated by reinforcement learning using, as at least a part of a reward, an output of an evaluation model that has been machine-learned to output an evaluation index corresponding to the status of the equipment; an estimation unit that estimates whether a main cause of the abnormality is the evaluation model or the operation model based on the abnormality indicator; an output unit that performs output according to the estimated result; An estimation device comprising:

2. The estimation device according to claim 1 , wherein the estimation unit estimates that a main cause of the abnormality is the operation model when the abnormality indicator does not satisfy a predetermined criterion.

3. The estimation device according to claim 2 , wherein the output unit outputs a command to re-learn the operational model when it is estimated that the main cause of the abnormality is the operational model.

4. The estimation device according to claim 1 , wherein the estimation unit estimates that a main cause of the abnormality is the evaluation model when the abnormality indicator satisfies a predetermined criterion.

5. The estimation device according to claim 4 , wherein the output unit outputs an instruction to re-learn the evaluation model when it is estimated that the main cause of the abnormality is the evaluation model.

6. The estimation device according to claim 5 , wherein when the output unit outputs an instruction to re-learn the evaluation model, the output unit outputs an instruction to re-learn the operation model using an output of the re-learned evaluation model as at least a part of the remuneration.

7. The estimation device according to claim 1 , further comprising a state data acquisition unit that acquires the state data.

8. The estimation device according to claim 7 , further comprising an abnormality detection unit that detects an occurrence of an abnormality in the facility based on the status data.

9. The estimation device according to claim 7 , further comprising a control unit that controls the controlled object using the operation model.

10. The estimation device according to claim 1 , further comprising an operation model generation unit that generates the operation model by reinforcement learning.

11. The estimation device according to claim 1 , further comprising an evaluation model generation unit that generates the evaluation model by machine learning.

12. The method is executed by a computer, the computer comprising: acquiring, as an abnormality index, the evaluation index output by the evaluation model in response to inputting the status data in the case where an abnormality occurs in the equipment, among status data indicating the status of the equipment when a control target provided in the equipment is controlled using an operation model that outputs an action in response to the status data, the evaluation index being generated by reinforcement learning using, as at least a part of a reward, an output of an evaluation model that has been machine-learned to output an evaluation index corresponding to the status of the equipment; Estimating whether a main cause of the abnormality is the evaluation model or the operation model based on the abnormality indicator; Executing an output according to the estimated result; The estimation method comprises:

13. The method is executed by a computer, causing the computer to: an abnormality index acquisition unit that acquires, as an abnormality index, the evaluation index output by the evaluation model in response to input of status data in the event of an abnormality occurring in the equipment, among status data indicating the status of the equipment when a control target provided in the equipment is controlled using an operation model that outputs an action in accordance with the status of the equipment, the evaluation index being generated by reinforcement learning using, as at least a part of a reward, an output of an evaluation model that has been machine-learned to output an evaluation index corresponding to the status of the equipment; an estimation unit that estimates whether a main cause of the abnormality is the evaluation model or the operation model based on the abnormality indicator; an output unit that performs output according to the estimated result; A guessing program that makes it work.

Images