Model Selection Device, Model Selection Method, and Model Selection Program

Abstract

SOLUTION: A model selection system includes a candidate model storage unit that stores plural candidate models each of which are generated through reinforcement learning using an output of an evaluation model, which outputs an index obtained by evaluating a condition of a facility, as at least part of remuneration, and each of which can output an action dependent on the condition of the facility, a condition data acquisition unit that, when a manipulation quantity based on the output of each of the candidate models is given to an object of control in the facility, acquires plural condition data items representing the condition of the facility, an index acquisition unit that acquires plural indices which are outputted from the evaluation model in response to the respective condition data items, a model selection unit that selects an object model, which is used to control the object of control, from among the candidate models on the basis of the plural indices, and an object model output unit that outputs the object mode.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a model selection device, a model selection method, and a model selection program.

Background Art

[0002] Patent Document 1 describes that "Model 45 outputs a recommended control parameter indicating the first type of control content recommended for increasing the reward value in response to the input of measurement data." Non-Patent Document 1 describes "FKDPP (Factorial Kernel Dynamic Policy Programming)". [Prior Art Documents] [Patent Documents] [Patent Document 1] JP 2021-086283 [Non-Patent Documents] [Non-Patent Document 1] "Yokogawa Electric and NAIST Apply Reinforcement Learning to Chemical Plants", Nikkei Robotics, March 2019 issue

Summary of the Invention

[0003] In a first aspect of the present invention, a model selection device is provided. The model selection device stores a plurality of candidate models, each of which is generated by reinforcement learning using, as at least part of the reward, the output of an evaluation model that outputs an index evaluating the state of the facility, and is capable of outputting an action according to the state in the facility. The model selection device includes a candidate model storage unit, a state data acquisition unit that acquires a plurality of state data indicating the state of the facility when respective operation amounts based on the outputs of the plurality of candidate models are given to a control target in the facility, an index acquisition unit that acquires a plurality of indexes output by the evaluation model in response to input of each of the plurality of state data, a model selection unit that selects a target model for controlling the control target from among the plurality of candidate models based on the plurality of indexes, and a target model output unit that outputs the target model.

[0004] In the model selection device, the model selection unit may select, as the target model, a candidate model that outputs an action that has achieved the highest value of the index among the plurality of candidate models.

[0005] In any of the model selection devices, the model selection unit may select, as the target model, a candidate model that outputs an action that has achieved the highest statistical value of the index at a plurality of time points among the plurality of candidate models.

[0006] In any of the model selection devices, the statistical value may include at least either an average value or a minimum value.

[0007] In any of the model selection devices, the model selection unit may reselect the target model in response to the update of the evaluation model.

[0008] In any of the model selection devices, the model selection unit may reselect the target model in response to the elapse of a predetermined time.

[0009] Any of the model selection devices may further include an input unit that receives user input in response to the output of the target model.

[0010] Any of the model selection devices may further include a control unit that controls the control target using the target model.

[0011] Any of the model selection devices may further include an operation model generation unit that generates a plurality of operation models that become the plurality of candidate models by the reinforcement learning.

[0012] Any of the model selection devices may further include an evaluation model storage unit that stores the evaluation model.

[0013] Any of the model selection devices may further include an evaluation model generation unit that generates the evaluation model by machine learning.

[0014] In a second aspect of the present invention, a model selection method is provided. The model selection method is executed by a computer, and the computer stores a plurality of candidate models each generated by reinforcement learning in which at least part of the reward is the output of an evaluation model that outputs an index for evaluating the state of a facility, and the candidate models can output actions according to the state of the facility. The method further includes obtaining a plurality of state data indicating the state of the facility when respective operation amounts based on the outputs of the plurality of candidate models are applied to a control target in the facility, obtaining a plurality of indices output by the evaluation model in response to inputting each of the plurality of state data, selecting a target model for controlling the control target from among the plurality of candidate models based on the plurality of indices, and outputting the target model.

[0015] In a third aspect of the present invention, a model selection program is provided. The model selection program is executed by a computer, and the computer functions as a candidate model storage unit that stores a plurality of candidate models each generated by reinforcement learning in which at least part of the reward is the output of an evaluation model that outputs an index for evaluating the state of a facility, and the candidate models can output actions according to the state of the facility, a state data acquisition unit that obtains a plurality of state data indicating the state of the facility when respective operation amounts based on the outputs of the plurality of candidate models are applied to a control target in the facility, an index acquisition unit that obtains a plurality of indices output by the evaluation model in response to inputting each of the plurality of state data, a model selection unit that selects a target model for controlling the control target from among the plurality of candidate models based on the plurality of indices, and a target model output unit that outputs the target model.

[0016] Note that the above summary of the invention does not list all the features of the present invention. Also, sub-combinations of these feature groups can also be inventions.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0018] Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention.

[0019] FIG. 1 shows an example of a block diagram of control system 1. Note that these blocks are functionally separated functional blocks and do not necessarily match the actual device configuration. That is, in this figure, just because something is shown as one block does not mean it must necessarily be constituted by one device. Also, in this figure, just because something is shown as separate blocks does not mean they must necessarily be constituted by separate devices. The same applies to the block diagrams hereinafter.

[0020] In control system 1, an evaluation model that outputs an index evaluating the state of facility 10 is generated by machine learning, and an operation model is generated by reinforcement learning using the output of the evaluation model as at least part of the reward. Then, in control system 1, the generated operation model is used to control control target 15 in facility 10. Such control using an operation model is also called AI (Artificial Intelligence) control. Model selection device 400 according to this embodiment selects a model to be used for control from among a plurality of candidates when there are a plurality of operation models available for AI control in such a control system 1.

[0021] Control system 1 may include facility 10, simulator 100, evaluation model management device 200, operation model management device 300, model selection device 400, and control device 500.

[0022] Facility 10 is a facility or device provided with control target 15. For example, facility 10 may be a plant or a composite device combining a plurality of devices. Examples of plants include industrial plants such as chemical and bio plants, plants for managing and controlling wellheads and their surroundings in gas fields and oil fields, plants for managing and controlling power generation such as hydraulic, thermal, and nuclear power, plants for managing and controlling environmental power generation such as solar and wind power, plants for managing and controlling water supply and drainage and dams, and the like.

[0023] Hereinafter, the case where the facility 10 is a distillation apparatus which is one of the process apparatuses will be described as an example. Generally, in a distillation apparatus, low-boiling components are evaporated in a distillation column and extracted from the top of the column, and the vapor of the extracted low-boiling components is condensed by a condenser and stored in a reflux drum. Then, the distillation apparatus refluxes a part of the liquid stored in the reflux drum into the distillation column, brings it into contact with the vapor in the distillation column, and distills it into low-boiling components and high-boiling components. In such a distillation apparatus, as an example, in order to control the reflux flow rate, the valve provided between the reflux drum and the distillation column is controlled to open and close.

[0024] The control target 15 is a device provided in the facility 10 and is a device to be controlled. For example, the control target 15 controls at least one physical quantity such as the amount, temperature, pressure, flow rate, speed, and pH of an object in the process of the facility 10, and is an actuator such as a valve, a heater, a motor, a fan, and a switch, that is, it may be an operation end and executes a given operation according to the operation amount. Hereinafter, the case where the control target 15 is a valve provided between the reflux drum and the distillation column in the distillation apparatus will be described as an example. However, it is not limited thereto. The control target 15 may be a controller that controls the operation end. That is, the term "control" used in this specification may be interpreted in a broad sense to include not only directly controlling the operation end but also indirectly controlling the operation end via a controller.

[0025] In the facility 10 provided with the controlled object 15, one or more sensors capable of measuring various states (physical quantities) inside and outside the facility 10 may be provided. As an example, when the facility 10 is a distillation apparatus, the sensor may output measured values PV (Process Variable) obtained by measuring temperatures at various positions (for example, the top of the column, the center of the column, the bottom of the column, etc.) of the distillation apparatus and flow rates in various paths. Such measured values PV may be included in the state data indicating the state of the facility 10. Further, the state data may include an operation amount MV (Manipulated Variable) indicating the opening / closing degree of the valve which is the controlled object 15. In addition to the operation data indicating the operation state as a result of controlling the controlled object 15 in this way, the state data may include consumption amount data indicating the consumption amounts of energy and raw materials in the facility 10, disturbance environment data indicating physical quantities that can act as disturbances to the control of the controlled object 15, and the like.

[0026] The distillation apparatus is one of the apparatuses widely used in the petroleum and chemical processes, but has characteristics such as strong mutual interference between the top and bottom of the column, a long time constant, and a non-linear operation. When opening and closing the valve by PID (Proportional Integral Differential) or the like to control the reflux flow rate in such a distillation apparatus, it has been difficult to improve the controllability. Further, when such a valve is manually operated by an operator for the purpose of multiple items such as quality assurance, energy saving, GHG (GreenHouse Gas) reduction, and yield improvement, to what extent the valve is to be opened and closed largely depends on the experience and intuition of the operator.

[0027] Therefore, when opening and closing such a valve, it is conceivable to use an operation model generated by reinforcement learning. The model selection device 400 according to the present embodiment may, for example, target such an operation model for selection.

[0028] The simulator 100 simulates the operation in the facility 10. For example, the simulator 100 may be designed based on the design information in the facility 10 and execute the behavior simulating the operation in the facility 10. The simulator 100 acquires a signal simulating the operation amount for the control target 15, so that the environment changes, and outputs simulation data simulating the state (for example, the predicted value of the sensor) in the facility 10. As an example, the simulator 100 may be composed of a prediction model for predicting the state of the distillation apparatus and a plant control simulator. The prediction model may be able to predict the state change of the reactor from the accumulated process data using a time-series data modeling technique using deep learning. Also, the plant control simulator may be able to virtually simulate PID control for deriving the operation amount MV based on the difference between the target value SV and the control amount CV for the control target 15. That is, the simulator 100 may be able to simulate not only the state prediction value but also the behavior itself in the facility 10.

[0029] The evaluation model management device 200 manages an evaluation model that outputs an index for evaluating the state of the facility 10. For example, the evaluation model management device 200 may generate an evaluation model by machine learning and store the generated evaluation model in the device itself. Also, the evaluation model management device 200 may output the generated evaluation model to the operation model management device 300.

[0030] The operation model management device 300 manages a plurality of operation models that output actions according to the state in the facility 10. For example, the operation model management device 300 may generate a plurality of operation models by reinforcement learning using the output of the evaluation model managed by the evaluation model management device 200 as at least a part of the reward, and store the generated plurality of operation models in the device itself. Also, the operation model management device 300 may output the generated plurality of operation models to the model selection device 400.

[0031] When there are multiple operation models available for AI control, the model selection device 400 selects a model to be used for control from among a plurality of candidates. For example, the model selection device 400 may acquire a plurality of operation models managed by the operation model management device 300 as a plurality of candidate models, and select a target model for controlling the control target 15 from among the plurality of candidate models. Further, the model selection device 400 may output the selected target model to the control device 500.

[0032] The control device 500 controls the control target 15 using the target model. For example, the control device 500 may control the control target 15 in the facility 10 using the target model selected by the model selection device 400.

[0033] In this way, in the control system 1, the AI automatically detects bottlenecks (potential faults) in the operation and generates an evaluation model as an index for improvement. Then, the AI makes trial and error based on the given index and generates an operation model that instructs a better operation method. Thereby, according to the control system 1, an environment in which the facility 10 can be autonomously controlled using the AI technology is provided. The model selection device 400 according to the present embodiment selects a model to be used for control from among a plurality of candidates when there are multiple operation models available for AI control in such a control system 1. Details of each device will be described in order below.

[0034] FIG. 2 shows an example of a block diagram of the evaluation model management device 200. The evaluation model management device 200 may be a computer such as a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, or a general-purpose computer, or may be a computer system to which a plurality of computers are connected. Such a computer system is also a computer in a broad sense. Further, the evaluation model management device 200 may be implemented by one or more executable virtual computer environments in the computer. Instead of this, the evaluation model management device 200 may be a dedicated computer designed for managing the evaluation model, or may be dedicated hardware realized by a dedicated circuit. Also, when connectable to the Internet, the evaluation model management device 200 may be realized by cloud computing.

[0035] The evaluation model management device 200 includes an evaluation model generation unit 210, an evaluation model storage unit 220, and an evaluation model output unit 230.

[0036] The evaluation model generation unit 210 generates an evaluation model that outputs an index for evaluating the state of the facility 10. For example, the evaluation model generation unit 210 may acquire an operation target (such as a plant KPI (Key Performance Indicator)) in the facility 10, state data indicating the state of the facility 10, and a teacher label, and generate labeling data based on these. Then, the evaluation model generation unit 210 may use the generated labeling data as learning data to generate an evaluation model by a machine learning algorithm. Since the generation process of the evaluation model itself may be arbitrary, further details are omitted here. The evaluation model generation unit 210 supplies the generated evaluation model to the evaluation model storage unit 220.

[0037] The evaluation model storage unit 220 stores an evaluation model. For example, the evaluation model storage unit 220 may store the evaluation model generated by the evaluation model generation unit 210. In the above description, the case where the evaluation model storage unit 220 stores the evaluation model generated inside the evaluation model management device 200 is shown as an example, but it is not limited thereto. The evaluation model storage unit 220 may store the evaluation model generated outside the evaluation model management device 200. The evaluation model storage unit 220 duplicates the stored evaluation model and supplies it to the evaluation model output unit 230.

[0038] The evaluation model output unit 230 outputs an evaluation model. For example, the evaluation model output unit 230 may output the evaluation model duplicated by the evaluation model storage unit 220 to the operation model management device 300 via a network.

[0039] FIG. 3 shows an example of a block diagram of the operation model management device 300. Similar to the evaluation model management device 200, the operation model management device 300 may be a computer, or may be a computer system to which a plurality of computers are connected. Further, the operation model management device 300 may be implemented by one or more executable virtual computer environments in a computer. Instead of this, the operation model management device 300 may be a dedicated computer designed for managing an operation model, or may be dedicated hardware realized by a dedicated circuit. Also, when it can be connected to the Internet, the operation model management device 300 may be realized by cloud computing.

[0040] The operation model management device 300 includes an evaluation model acquisition unit 310, an operation model generation unit 320, an operation model storage unit 330, and an operation model output unit 340.

[0041] The evaluation model acquisition unit 310 acquires an evaluation model that outputs an index for evaluating the state of the facility 10. For example, the evaluation model acquisition unit 310 may acquire the evaluation model output from the evaluation model output unit 230 via a network. The evaluation model acquisition unit 310 supplies the acquired evaluation model to the operation model generation unit 320.

[0042] The operation model generation unit 320 generates a plurality of operation models capable of outputting actions according to the state of the facility 10 by reinforcement learning using at least a part of the output of the evaluation model as a reward. Such an operation model may have, as an example, a data table composed of a combination (S, A) of a set S of sampled state data and an action A taken under each state, and a weight W calculated by a reward. Note that at least a part of the reward for calculating such a weight W may use the output of the evaluation model.

[0043] In generating such an operation model, the operation model generation unit 320 may acquire learning environment data indicating the state of the learning environment. At this time, when a simulator 100 that simulates the operation in the facility 10 is used as the learning environment, the operation model generation unit 320 may acquire simulation data from the simulator 100 as the learning environment data. However, it is not limited to this. The actual facility 10 may be used as the learning environment. In this case, the operation model generation unit 320 may acquire state data indicating the state of the facility 10 as the learning environment data.

[0044] Next, the operation model generation unit 320 may randomly determine an action or use a known AI algorithm such as FKDPP described later, and give an operation amount based on the action to a control target in the learning environment. Accordingly, the state of the learning environment changes.

[0045] Then, the operation model generation unit 320 may acquire the learning environment data again. Thereby, the operation model generation unit 320 can acquire the state of the learning environment after the change according to the fact that the operation amount based on the determined action is given to the control target.

[0046] Then, the operation model generation unit 320 may calculate a reward value based at least partially on the output of the evaluation model. As an example, in response to inputting learning environment data indicating the state of the learning environment after the change into the evaluation model, an index output by the evaluation model may be directly calculated as the reward value.

[0047] After the operation model generation unit 320 repeatedly performs the state acquisition process corresponding to such a decision of action a plurality of times, in addition to overwriting the values of the weight column in the data table, new sample data that has not been saved so far may be added to a new row in the data table to update the operation model. The operation model generation unit 320 can generate an operation model by repeatedly performing such an update process a plurality of times. Since the generation of the operation model itself may be arbitrary, further details are omitted here.

[0048] The operation model generation unit 320 can generate a plurality of different operation models by, for example, executing such an operation model generation process under different learning environments or with different learning algorithms. The operation model generation unit 320 supplies the generated plurality of operation models to the operation model storage unit 330.

[0049] The operation model storage unit 330 stores a plurality of operation models. For example, the operation model storage unit 330 may store a plurality of operation models generated by the operation model generation unit 320. In the above description, the case where the operation model storage unit 330 stores a plurality of operation models generated inside the operation model management device 300 is shown as an example, but it is not limited thereto. The operation model storage unit 330 may store a plurality of operation models that are partially or entirely generated outside the operation model management device 300. The operation model storage unit 330 replicates the stored plurality of operation models and supplies them to the operation model output unit 340.

[0050] The operation model output unit 340 outputs a plurality of operation models. For example, the operation model output unit 340 may output a plurality of operation models replicated by the operation model storage unit 330 to the model selection device 400 via a network.

[0051] FIG. 4 shows an example of a block diagram of the model selection device 400 according to the present embodiment. Similar to the evaluation model management device 200, the model selection device 400 may be a computer, or may be a computer system to which a plurality of computers are connected. Further, the model selection device 400 may be implemented by one or a plurality of executable virtual computer environments in a computer. Alternatively, the model selection device 400 may be a dedicated computer designed for model selection, or may be dedicated hardware realized by a dedicated circuit. Further, when connectable to the Internet, the model selection device 400 may be realized by cloud computing.

[0052] The model selection device 400 includes a candidate model acquisition unit 410, a candidate model storage unit 420, a state data acquisition unit 430, an index acquisition unit 440, a model selection unit 450, a target model output unit 460, and an input unit 470.

[0053] The candidate model acquisition unit 410 acquires a plurality of candidate models. For example, the candidate model acquisition unit 410 may acquire a plurality of operation models output by the operation model output unit 340 as a plurality of candidate models. The candidate model acquisition unit 410 supplies the acquired plurality of candidate models to the candidate model storage unit 420.

[0054] The candidate model storage unit 420 stores a plurality of candidate models. For example, the candidate model storage unit 420 may store a plurality of candidate models acquired by the candidate model acquisition unit 410. In this way, the candidate model storage unit 420 can store a plurality of candidate models each generated by reinforcement learning using, as at least part of the reward, the output of an evaluation model that outputs an index for evaluating the state of the facility 10 and capable of outputting an action according to the state in the facility 10.

[0055] The state data acquisition unit 430 acquires a plurality of state data. For example, the state data acquisition unit 430 may acquire a plurality of state data indicating the state of the facility 10 when each operation amount based on the outputs of the plurality of candidate models stored in the candidate model storage unit 420 is given to the control target 15 in the facility 10. The state data acquisition unit 430 supplies the acquired plurality of state data to the index acquisition unit 440.

[0056] The index acquisition unit 440 acquires a plurality of indexes. For example, the index acquisition unit 440 may acquire a plurality of indexes output by the evaluation model in response to inputting each of the plurality of state data acquired by the state data acquisition unit 430. The index acquisition unit 440 supplies the acquired plurality of indexes to the model selection unit 450.

[0057] The model selection unit 450 selects a target model. For example, the model selection unit 450 may select a target model for controlling the control target 15 from among the plurality of candidate models stored in the candidate model storage unit 420 based on the plurality of indexes acquired by the index acquisition unit 440. The model selection unit 450 supplies information for identifying the selected target model to the target model output unit 460.

[0058] The target model output unit 460 outputs a target model. For example, the target model output unit 460 may replicate the target model from among the plurality of candidate models stored in the candidate model storage unit 420 according to the information for identifying the target model selected by the model selection unit 450. Then, the target model output unit 460 may output the target model to the control device 500 via the network.

[0059] The input unit 470 receives user input. For example, the input unit 470 may receive user input in response to the target model being output by the target model output unit 460. And when the input unit 470 reselects the target model, it may trigger the acquisition of a plurality of state data by the state data acquisition unit 430 and the acquisition of a plurality of candidate models by the candidate model acquisition unit 410.

[0060] FIG. 5 shows an example of a block diagram of the control device 500. The control device 500 may be, for example, a controller in a DCS (Distributed Control System) or an instrumentation system for medium scale, or may be a real-time OS controller or the like.

[0061] The control device 500 includes a target model acquisition unit 510, an actual environment data acquisition unit 520, and a control unit 530.

[0062] The target model acquisition unit 510 acquires a target model. For example, the target model acquisition unit 510 may acquire the target model output by the target model output unit 460 via a network. The target model acquisition unit 510 supplies the acquired target model to the control unit 530.

[0063] The actual environment data acquisition unit 520 acquires actual environment data indicating the actual environment, that is, the state of the facility 10. Such actual environment data may be the same data as the above-described state data. The actual environment data acquisition unit 520 supplies the acquired actual environment data to the control unit 530.

[0064] The control unit 530 controls the control target 15 using the target model. For example, the control unit 530 may determine an action by a known AI algorithm such as FKDPP described later. Then, the control unit 530 may give an operation amount obtained by adding the determined action to the value of the control target 15 to the control target 15 in the facility 10. For example, in this way, the control unit 530 can perform AI control on the control target 15 using the target model selected by the model selection device 400.

[0065] FIG. 6 shows an example of a flowchart of a model selection method that the model selection device 400 according to the present embodiment may execute.

[0066] In step S610, the model selection device 400 acquires a plurality of candidate models. For example, the candidate model acquisition unit 410 may acquire, as a plurality of candidate models, the plurality of operation models output by the operation model output unit 340 from the operation model management device 300 via a network. However, it is not limited thereto. The candidate model acquisition unit 410 may acquire a plurality of candidate models via other means different from the network (such as various memory devices and user input), or may acquire a plurality of candidate models from another device different from the operation model management device 300. The candidate model acquisition unit 410 supplies the acquired plurality of candidate models to the candidate model storage unit 420.

[0067] In step S620, the model selection device 400 stores a plurality of candidate models. For example, the candidate model storage unit 420 may store the plurality of candidate models acquired in step S610. In the above description, the case where the candidate model storage unit 420 stores a plurality of candidate models acquired from another device such as the operation model management device 300 is shown as an example, but it is not limited thereto. The candidate model storage unit 420 may store a plurality of candidate models in advance. The candidate model storage unit 420 can store, for example, a plurality of candidate models each generated by reinforcement learning in which the output of an evaluation model that outputs an index for evaluating the state of the facility 10 is used as at least part of the reward and that can output an action according to the state in the facility 10. In other words, the candidate model storage unit 420 can store a plurality of different candidate models generated in different learning environments or different learning algorithms with the output of a common evaluation model as at least part of the reward. Here, as an example, it is assumed that the candidate model storage unit 420 stores three candidate models: candidate model x, candidate model y, and candidate model z.

[0068] In step S630, the model selection device 400 acquires a plurality of state data. For example, the state data acquisition unit 430 may acquire various physical quantities measured by various sensors provided in the facility 10 from the facility 10 via a network as state data. However, it is not limited thereto. The state data acquisition unit 430 may acquire state data via other means different from the network, or may acquire state data from other devices different from the facility 10.

[0069] Next, the state data acquisition unit 430 may determine each of a plurality of actions by a known AI algorithm such as FKDPP using the plurality of candidate models stored in step S620. When using such a kernel method, the state data acquisition unit 430 may generate a vector of the state S from the sensor values obtained from the acquired state data. Next, the state data acquisition unit 430 may generate a combination of the state S and all possible actions A as an action decision table. Then, the state data acquisition unit 430 may input the action decision table to each of the plurality of candidate models stored in step S620. In response thereto, each of the plurality of candidate models may perform a kernel calculation between each row of the action decision table and each sample data excluding the weight column in the data table, and may calculate the distance between each sample data. Then, each of the plurality of candidate models may sequentially add up the products of the distances calculated for each sample data and the values of their respective weight columns, and calculate the expected reward value for each action. The state data acquisition unit 430 may determine a plurality of actions, for example, by selecting, for each of the plurality of candidate models, the action determined to have the highest expected reward value using the plurality of candidate models. In other words, the state data acquisition unit 430 may determine, for each candidate model, the action determined by each of the plurality of candidate models to maximize the expected reward value according to the state of the facility 10. Here, as an example, it is assumed that the state data acquisition unit 430 determines an action Ax using the candidate model x, determines an action Ay using the candidate model y, and determines an action Az using the candidate model z.

[0070] Then, the state data acquisition unit 430 may apply each operation amount obtained by adding the determined plurality of actions to the value of the control target 15 to the control target 15 via the control device 500. Accordingly, the state of the facility 10 changes. The state data acquisition unit 430 may further acquire state data indicating the state of the facility after the change. The state data acquisition unit 430 may acquire, for example, a plurality of state data indicating the state of the facility 10 when each operation amount based on the output of a plurality of candidate models is applied to the control target 15 in the facility 10 in this way. Here, it is assumed that the state data acquisition unit 430 acquires the state data Sx when the operation amount MVx based on the action Ax is applied to the control target 15, acquires the state data Sy when the operation amount MVy based on the action Ay is applied to the control target 15, and acquires the state data Sz when the operation amount MVz based on the action Az is applied to the control target 15. The state data acquisition unit 430 supplies the acquired plurality of state data to the index acquisition unit 440.

[0071] In step S640, the model selection device 400 acquires a plurality of indexes. For example, the index acquisition unit 440 may input each of the plurality of state data acquired in step S630 to the evaluation model stored in the evaluation model storage unit 220 and acquire each of the plurality of indexes output by the evaluation model. The index acquisition unit 440 may acquire, for example, a plurality of indexes output by the evaluation model in response to inputting each of the plurality of state data in this way. Here, it is assumed that the index acquisition unit 440 acquires the index Ix output by the evaluation model in response to inputting the state data Sx, acquires the index Iy output by the evaluation model in response to inputting the state data Sy, and acquires the index Iz output by the evaluation model in response to inputting the state data Sz. The index acquisition unit 440 supplies the acquired plurality of indexes to the model selection unit 450.

[0072] In step S650, the model selection device 400 selects a target model. For example, the model selection unit 450 may select a target model for controlling the control target 15 from among a plurality of candidate models stored in step S620 based on the plurality of indicators acquired in step S640.

[0073] At this time, the model selection unit 450 may select, as the target model, the candidate model that output the action that reached the highest indicator among the plurality of candidate models. As an example, when the plurality of indicators are Ix > Iy > Iz, the model selection unit 450 may select the candidate model x that output the action Ax as the target model.

[0074] Note that in the above description, the case where the model selection unit 450 selects a candidate model based on the indicators at one point in time is shown as an example, but it is not limited thereto. The model selection unit 450 may select a candidate model based on the statistical quantity of the indicators at a plurality of points in time. As an example, when the plurality of indicators are Iy_min > Iz_min > Ix_min (where min indicates the minimum value at a plurality of points in time), the model selection unit 450 may select the candidate model y that output the action Ay as the target model.

[0075] Also, when the plurality of indicators are Iz_ave > Ix_ave > Iy_ave (where ave indicates the average value at a plurality of points in time), the model selection unit 450 may select the candidate model z that output the action Az as the target model.

[0076] For example, in this way, the model selection unit 450 may select, as the target model, the candidate model that outputs the action that has led to the highest statistical value of the metrics at multiple time points among a plurality of candidate models. At this time, the statistical value may include at least either the average value or the minimum value. At this time, when selecting a candidate model based on a plurality of statistical values, the model selection unit 450 may select, as the target model, the candidate model that outputs the action that has led to the highest sum or weighted average obtained by weighted addition of each statistical value. The model selection unit 450 supplies information for identifying the selected target model to the target model output unit 460.

[0077] In step S660, the model selection device 400 outputs a target model. For example, the target model output unit 460 may replicate the target model from among the plurality of candidate models stored in step S620 according to the information for identifying the target model selected in step S650. Then, the target model output unit 460 may output the target model to the control device 500 via, for example, a network. In response to this, the control device 500 can start AI control using the target model.

[0078] In step S670, the model selection device 400 determines whether to reselect the target model. For example, the input unit 470 may receive user input in response to the target model being output in step S660. Then, when receiving an instruction from the user to reselect the target model, the input unit 470 may determine to reselect the target model.

[0079] If it is determined to reselect the target model (Yes), the model selection device 400 may return the process to step S630 and continue the flow. In this case, the input unit 470 may trigger the acquisition of a plurality of state data by the state data acquisition unit 430. Thereby, the model selection device 400 can re-acquire a plurality of state data and reselect the target model. In the above description, the case where the model selection device 400 returns the process to step S630 is shown as an example, but it is not limited thereto. The model selection device 400 may return the process to step S610 and continue the flow. In this case, the input unit 470 may trigger the acquisition of a plurality of candidate models by the candidate model acquisition unit 410. Thereby, the model selection device 400 may newly acquire a plurality of candidate models and reselect the target model from among the newly acquired plurality of candidate models.

[0080] If it is determined not to reselect the target model (No), the model selection device 400 ends the flow of the model selection method.

[0081] The model selection device 400 can also execute the flow of such a model selection method again according to various triggers (event triggers and time triggers). For example, the model selection device 400 may execute the model selection method again by using the update of the evaluation model as a trigger. Therefore, the model selection unit 450 may reselect the target model according to the update of the evaluation model.

[0082] Also, the model selection device 400 may execute the model selection method again by using the elapse of a predetermined time after the target model was previously selected as a trigger. Therefore, the model selection unit 450 may reselect the target model according to the elapse of the predetermined time.

[0083] Generally, the operation models generated by reinforcement learning are black-boxed, making it difficult to evaluate the operation models. Therefore, when multiple such operation models are available, it has been difficult to select which operation model to use for AI control. In contrast, the model selection device 400 according to the present embodiment evaluates the respective states of the facility 10 when each operation amount based on the outputs of a plurality of candidate models is given to the control target 15, using an evaluation model, and selects a target model based on each index output by the evaluation model. Thereby, according to the model selection device 400 according to the present embodiment, it is possible to select which candidate model to use for AI control based on the objective results obtained by evaluating each of the plurality of actions output by the plurality of candidate models using a common evaluation model.

[0084] Further, the model selection device 400 according to the present embodiment may select, as the target model, a candidate model that outputs an action for which the index output by the evaluation model is the highest among the plurality of candidate models. Thereby, according to the model selection device 400 according to the present embodiment, it is possible to select, as the target model, a candidate model that can maximize operation goals such as KPIs.

[0085] Further, the model selection device 400 according to the present embodiment may select, as a target model, a candidate model that outputs an action with the highest statistical value of the index at a plurality of time points among a plurality of candidate models. Thereby, according to the model selection device 400 according to the present embodiment, instead of a candidate model that temporarily outputs an action with the highest index, a candidate model that outputs an action with the highest index over a certain period can be selected as the target model. At this time, an average value may be used as the statistical value. Thereby, according to the model selection device 400 according to the present embodiment, a candidate model that stably outputs an action with a high index over a long period can be selected as the target model. Also, a minimum value may be used as the statistical value. Thereby, according to the model selection device 400 according to the present embodiment, even when mission-critical operations such as plant operations are required, an optimal candidate model can be selected as the target model.

[0086] Further, the model selection device 400 according to the present embodiment can also reselect the target model in response to the evaluation model being updated. Thereby, according to the model selection device 400 according to the present embodiment, even when the operation target is changed, an optimal candidate model can be reselected as the target model in light of the new operation target.

[0087] Further, the model selection device 400 according to the present embodiment can also reselect the target model in response to a predetermined time having elapsed. Thereby, according to the model selection device 400 according to the present embodiment, even when the facility 10 has changed over time since the target model was previously selected, an optimal candidate model can be reselected as the target model in light of the current state of the facility 10.

[0088] In addition, the model selection device 400 according to the present embodiment can also receive user input in response to the output of the target model. Thereby, according to the model selection device 400 according to the present embodiment, after the target model is output, the user can feedback the result of judging the validity of the target model. And, according to the model selection device 400 according to the present embodiment, when the target model is not valid, the target model can be reselected.

[0089] FIG. 7 shows an example of a block diagram of the model selection device 400 according to the first modification. In FIG. 7, members having the same functions and configurations as those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted except for the following differences. In the above-described embodiment, an example is shown in which the evaluation model management device 200, the operation model management device 300, the model selection device 400, and the control device 500 are provided as separate independent devices. However, these devices may be provided as a single integrated device, partially or entirely. In this modification, the model selection device 400 provides the functions of the control device 500 in addition to the functions of the model selection device 400 according to the above-described embodiment.

[0090] The model selection device 400 according to this modification may further include a control unit 530. That is, the model selection device 400 may further include a control unit 530 that controls the control target 15 using the target model.

[0091] In addition, in this modification, the target model output unit 460 may output the selected target model to the control unit 530 instead of the control device 500. And the control unit 530 may acquire the target model output by the target model output unit 460.

[0092] In addition, in this modification, the state data acquisition unit 430 may supply the state data acquired during AI control to the control unit 530. That is, in this modification, the state data acquisition unit 430 may also function as the real environment data acquisition unit 520.

[0093] Then, the control unit 530 may control the control target 15 using the target model. The model selection device 400 may also provide, for example, the functions of the control device 500 in this way.

[0094] In this way, the model selection device 400 according to this modification example can also control the control target 15 using the target model. Thus, according to the model selection device 400 according to this modification example, the function of selecting the target model and the function of controlling the control target 15 using the selected target model can be realized by one device. Further, according to the model selection device 400 according to this modification example, since there is no need to exchange the target model between the model selection device 400 and the control device 500, communication costs and time can be reduced.

[0095] FIG. 8 shows an example of a block diagram of the model selection device 400 according to the second modification example. In FIG. 8, members having the same functions and configurations as those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted except for the following differences. In this modification example, the model selection device 400 provides the functions of the operation model management device 300 in addition to the functions of the model selection device 400 according to the above-described embodiment.

[0096] The model selection device 400 according to this modification example may further include an evaluation model acquisition unit 310 and an operation model generation unit 320. That is, the model selection device 400 may further include an operation model generation unit that generates a plurality of operation models that become a plurality of candidate models by reinforcement learning.

[0097] Further, in this modification example, the operation model generation unit 320 may supply the generated plurality of operation models to the candidate model storage unit 420. Then, the candidate model storage unit 420 may store the plurality of operation models supplied from the operation model generation unit 320 as a plurality of candidate models.

[0098] Further, in this modification, when reselecting the target model, the input unit 470 may trigger the generation of a plurality of operation models by the operation model generation unit 320. Thereby, the model selection device 400 according to this modification may newly generate a plurality of operation models that become a plurality of candidate models, and reselect the target model from among the newly generated plurality of candidate models. The model selection device 400 may also provide, for example, the function as the operation model management device 300 in this way.

[0099] As described above, the model selection device 400 according to this modification can also generate a plurality of operation models that become a plurality of candidate models by itself through reinforcement learning. Thereby, according to the model selection device 400 according to this modification, the function of generating a plurality of operation models that are candidates for selecting the target model and the function of selecting the target model can be realized by one device. Further, according to the model selection device 400 according to this modification, since there is no need to exchange a plurality of operation models between the operation model management device 300 and the model selection device 400, communication costs and time can be reduced.

[0100] FIG. 9 shows an example of a block diagram of the model selection device 400 according to the third modification. In FIG. 9, members having the same functions and configurations as those in FIG. 1 are denoted by the same reference numerals, and the description thereof will be omitted except for the following differences. In this modification, the model selection device 400 provides the function of the evaluation model management device 200 in addition to the function of the model selection device 400 according to the above-described embodiment.

[0101] The model selection device 400 according to this modification further includes an evaluation model generation unit 210 and an evaluation model storage unit 220. That is, the model selection device 400 may further include an evaluation model storage unit 220 that stores an evaluation model. Further, the model selection device 400 may further include an evaluation model generation unit 210 that generates an evaluation model by machine learning.

[0102] Also, in this modified example, the index acquisition unit 440 may input a plurality of state data into the evaluation models stored in the evaluation model storage unit 220 respectively, and acquire the plurality of indexes output by the evaluation models respectively. The model selection device 400 may also provide, for example, the function as the evaluation model management device 200 in this way.

[0103] In this way, the model selection device 400 according to this modified example can also store the evaluation model. Thereby, according to the model selection device 400 according to this modified example, when acquiring a plurality of indexes, it is not necessary to exchange a plurality of state data and a plurality of indexes with the evaluation model management device 200, so that communication costs and time can be reduced. Further, the model selection device 400 according to this modified example can also generate the evaluation model by itself through machine learning. Thereby, according to the model selection device 400 according to this modified example, the function of generating the evaluation model and the function of selecting the target model can be realized by one device.

[0104] So far, the implementable forms have been illustrated and described. However, the above-described embodiments may be changed or applied in various forms. For example, in the above-described modified example, the case where the model selection device 400 further provides the functions of the control device 500, the operation model management device 300, and the evaluation model management device 200 is shown as separate modified examples. However, it is not limited thereto. The model selection device 400 may further provide two or more functions among the control device 500, the operation model management device 300, and the evaluation model management device 200, or may further provide all the functions. Thereby, according to the model selection device 400, all the functions related to the operation of controlling the control target 15 can also be realized by one device.

[0105] Also, in the above description, when acquiring a plurality of state data, as an example, the model selection device 400 gives each operation amount based on the outputs of a plurality of candidate models to the control target 15 in the actual facility 10, and acquires a plurality of state data from the actual facility 10, but the present invention is not limited thereto. The model selection device 400 may give each operation amount based on the outputs of a plurality of candidate models to the control target in the simulation environment, and acquire a plurality of state data from the simulator 100. Thereby, the model selection device 400 can also complete the flow until selecting the target model in the simulation environment without using the actual machine.

[0106] Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where the blocks may represent (1) a stage of a process in which an operation is performed or (2) a section of a device having a role of performing an operation. Specific stages and sections may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable medium. The dedicated circuit may include digital and / or analog hardware circuits, and may include an integrated circuit (IC) and / or discrete circuits. The programmable circuit may include a reconfigurable hardware circuit including memory elements such as logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.

[0107] A computer-readable medium may include any tangible device that can store instructions executable by an appropriate device. As a result, a computer-readable medium having instructions stored therein will comprise a product that includes instructions that can be executed to create means for performing the operations specified in a flowchart or block diagram. Examples of computer-readable media may include electronic memory media, magnetic memory media, optical memory media, electromagnetic memory media, semiconductor memory media, and the like. More specific examples of computer-readable media may include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated circuit card, and the like.

[0108] Computer-readable instructions may include any combination of one or more programming languages, including source code or object code written in any combination of assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages.

[0109] Computer-readable instructions may be provided to a processor or programmable circuitry of a general purpose computer, special purpose computer, or other programmable data processing apparatus locally or via a wide area network (WAN) such as a local area network (LAN), the Internet, etc., and executed to create means for performing the operations specified in a flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0110] FIG. 10 shows an example of a computer 9900 in which multiple aspects of the present invention may be embodied, in whole or in part. Programs installed on the computer 9900 may cause the computer 9900 to function as an operation associated with an apparatus according to an embodiment of the present invention or as one or more sections of the apparatus, or to execute the operation or the one or more sections, and / or may cause the computer 9900 to execute a process according to an embodiment of the present invention or a stage of the process. Such programs may be executed by the CPU 9912 to cause the computer 9900 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0111] The computer 9900 according to this embodiment includes a CPU 9912, a RAM 9914, a graphic controller 9916, and a display device 9918, which are mutually connected by a host controller 9910. The computer 9900 also includes an input / output unit such as a communication interface 9922, a hard disk drive 9924, a DVD drive 9926, and an IC card drive, which are connected to the host controller 9910 via an input / output controller 9920. The computer also includes legacy input / output units such as a ROM 9930 and a keyboard 9942, which are connected to the input / output controller 9920 via an input / output chip 9940.

[0112] The CPU 9912 operates according to programs stored in the ROM 9930 and the RAM 9914, thereby controlling each unit. The graphic controller 9916 acquires image data generated by the CPU 9912 in a frame buffer or the like provided in the RAM 9914 or in itself, and causes the image data to be displayed on the display device 9918.

[0113] The communication interface 9922 communicates with other electronic devices via a network. The hard disk drive 9924 stores programs and data used by the CPU 9912 in the computer 9900. The DVD drive 9926 reads a program or data from the DVD-ROM 9901 and provides the program or data to the hard disk drive 9924 via the RAM 9914. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0114] ROM 9930 stores therein a boot program or the like executed by computer 9900 at activation, and / or a program dependent on the hardware of computer 9900. Input / output chip 9940 may also be connected to input / output controller 9920 via various input / output units through a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0115] The program is provided by a computer-readable medium such as DVD-ROM 9901 or an IC card. The program is read from the computer-readable medium, installed in hard disk drive 9924, RAM 9914, or ROM 9930, which is also an example of a computer-readable medium, and executed by CPU 9912. The information processing described in these programs is read by computer 9900, resulting in cooperation between the programs and the various types of hardware resources described above. The apparatus or method may be configured by realizing the operation or processing of information according to the use of computer 9900.

[0116] For example, when communication is executed between computer 9900 and an external device, CPU 9912 may execute a communication program loaded in RAM 9914 and instruct communication interface 9922 to perform communication processing based on the processing described in the communication program. Communication interface 9922 reads the transmission data stored in the transmission buffer processing area provided in a recording medium such as RAM 9914, hard disk drive 9924, DVD-ROM 9901, or an IC card under the control of CPU 9912, transmits the read transmission data to the network, or writes the received data received from the network to the reception buffer processing area or the like provided on the recording medium.

[0117] Also, the CPU 9912 may cause all or necessary parts of files or databases stored in external recording media such as hard disk drives 9924, DVD drives 9926 (DVD-ROM 9901), and IC cards to be read into the RAM 9914, and may perform various types of processing on the data on the RAM 9914. The CPU 9912 then writes back the processed data to the external recording media.

[0118] Various types of information such as various types of programs, data, tables, and databases may be stored in the recording media and may undergo information processing. The CPU 9912 may perform various types of processing on the data read from the RAM 9914, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 9914. Also, the CPU 9912 may search for information in files, databases, etc. within the recording media. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording media, the CPU 9912 searches for an entry that matches the condition where the attribute value of the first attribute is specified from among the plurality of entries, reads the attribute value of the second attribute stored within the entry, and thereby may obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0119] The programs or software modules described above may be stored on a computer-readable medium on or near the computer 9900. Also, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the program to the computer 9900 via the network.

[0120] As described above, the present invention has been described using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that forms with such changes or improvements can also be included in the technical scope of the present invention.

[0121] It should be noted that the execution order of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the specification, and the drawings is not explicitly stated as "earlier" or "preceding", etc., and can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the specification, and the drawings, even if it is described using "first", "next", etc. for convenience, it does not mean that it is essential to implement in this order.

Explanation of Reference Numerals

[0122] 1 Control system 10 Facilities 15 Controlled object 100 Simulator 200 Evaluation model management device 210 Evaluation model generation unit 220 Evaluation model storage unit 230 Evaluation model output unit 300 Operation model management device 310 Evaluation model acquisition unit 320 Operation model generation unit 330 Operation model storage unit 340 Operation model output unit 400 Model selection device 410 Candidate model acquisition unit 420 Candidate model storage unit 430 State data acquisition unit 440 Index acquisition unit 450 Model selection unit 460 Target model output unit 470 Input unit 500 Control Device 510 Target Model Acquisition Unit 520 Actual Environment Data Acquisition Unit 530 Control Unit 9900 Computer 9901 DVD-ROM 9910 Host Controller 9912 CPU 9914 RAM 9916 Graphics Controller 9918 Display Device 9920 Input / Output Controller 9922 Communication Interface 9924 Hard Disk Drive 9926 DVD Drive 9930 ROM 9940 Input / Output Chip 9942 Keyboard

Claims

1. A candidate model storage unit that stores a plurality of candidate models, each of which is generated by reinforcement learning using, as at least part of the reward, the output of an evaluation model that outputs an index evaluating the state of the facility and is capable of outputting an action according to the state of the facility; A state data acquisition unit that acquires a plurality of state data indicating the state of the facility when each operation amount based on the output of the plurality of candidate models is applied to a control target in the facility; An index acquisition unit that acquires a plurality of indexes output by the evaluation model in response to each of the plurality of state data being input; A model selection unit that selects a target model for controlling the control target from among the plurality of candidate models based on the plurality of indexes; A target model output unit that outputs the target model; A model selection device comprising:

2. The model selection device according to claim 1, wherein the model selection unit selects, as the target model, a candidate model that outputs an action that has led to the highest index among the plurality of candidate models.

3. The model selection device according to claim 2, wherein the model selection unit selects, as the target model, a candidate model that outputs an action that has led to the highest statistical value of the indexes at a plurality of time points among the plurality of candidate models.

4. The model selection device according to claim 3, wherein the statistical value includes at least one of an average value or a minimum value.

5. The model selection device according to any one of claims 1 to 4, wherein the model selection unit reselects the target model in response to the evaluation model being updated.

6. The model selection device according to any one of claims 1 to 4, wherein the model selection unit reselects the target model in response to a predetermined time having elapsed.

7. The model selection device according to any one of claims 1 to 4, further comprising an input unit that receives user input in response to the target model being output.

8. The model selection device according to any one of claims 1 to 4, further comprising a control unit that controls the control target using the target model.

9. The model selection device according to any one of claims 1 to 4, further comprising an operation model generation unit that generates a plurality of operation models that become the plurality of candidate models by the reinforcement learning.

10. The model selection device according to any one of claims 1 to 4, further comprising an evaluation model storage unit that stores the evaluation model.

11. The model selection device according to any one of claims 1 to 4, further comprising an evaluation model generation unit that generates the evaluation model by machine learning.

12. Executed by a computer, the computer Stores a plurality of candidate models, each of which is generated by reinforcement learning in which the output of an evaluation model that outputs an index for evaluating the state of the facility is used as at least part of the reward, and can output an action according to the state in the facility; Acquires a plurality of state data indicating the state of the facility when each operation amount based on the output of the plurality of candidate models is given to a control target in the facility; Acquires a plurality of indices output by the evaluation model in response to each of the plurality of state data being input; Based on the plurality of indices, selects a target model for controlling the control target from among the plurality of candidate models; Outputs the target model; A model selection method comprising:

13. Executed by a computer, the computer A candidate model storage unit that stores a plurality of candidate models, each of which is generated by reinforcement learning in which the output of an evaluation model that outputs an index for evaluating the state of the facility is used as at least part of the reward, and can output an action according to the state in the facility; A state data acquisition unit that acquires a plurality of state data indicating the state of the facility when each operation amount based on the output of the plurality of candidate models is given to a control target in the facility; An index acquisition unit that acquires a plurality of indices output by the evaluation model in response to each of the plurality of state data being input; A model selection unit that selects a target model for controlling the control target from among the plurality of candidate models based on the plurality of indices; A target model output unit that outputs the target model; A model selection program that causes the computer to function as such.

Images