Model selection apparatus, model selection method, and computer program product

Abstract

The present application provides a model selection device, a model selection method, and a non-transitory computer readable medium, the model selection device including: a candidate model storage unit that stores a plurality of candidate models, the plurality of candidate models being generated by reinforcement learning that uses, as a reward, at least a part of an output of an evaluation model that evaluates an index of a state of a device, and being capable of outputting an action corresponding to the state in the device; a state data acquisition unit that acquires a plurality of state data representing the state of the device when each operation amount based on the output of the plurality of candidate models is provided to a control target in the device; an index acquisition unit that acquires a plurality of indexes output by the evaluation model in accordance with each of the plurality of state data being input; a model selection unit that selects, from the plurality of candidate models, an object model for controlling the control target based on the plurality of indexes; and an object model output unit that outputs the object model.

Description

TECHNICAL FIELD

[0001] The present application relates to a model selection device, a model selection method, and a non-transitory computer-readable medium. BACKGROUND

[0002] In Patent Literature 1, it is described that “the model 45 outputs a recommended control parameter indicating a first control content recommended in order to improve a return value in accordance with an input of measurement data”. Further, in Non-Patent Literature 1, a “FKDPP (Factorial Kernel Dynamic Policy Programming)” is described.

[0003] PRIOR ART DOCUMENTS

[0004] Patent Literature 1: Japanese Patent Publication No. 2021-086283

[0005] Non-Patent Literature 1: “Yokogawa and NAIST Reinforcement Learning for Chemical Plants”, Nikkei Robotics, March 2019 SUMMARY

[0006] A model selection device is provided in a first aspect of the present application. The model selection device includes a candidate model storage unit that stores a plurality of candidate models, the plurality of candidate models being generated by reinforcement learning in which an output of an evaluation model that evaluates an index of a state of a device is taken as a part of a return, and being capable of outputting an action corresponding to the state of the device, the evaluation model outputting the index evaluating the state of the device; a state data acquisition unit that acquires a plurality of state data indicating the state of the device when each operation amount based on outputs of the plurality of candidate models is provided to a control target in the device; an index acquisition unit that acquires a plurality of indexes output by the evaluation model in accordance with each of the plurality of state data being input; a model selection unit that selects, based on the plurality of indexes, a target model for controlling the control target from among the plurality of candidate models; and a target model output unit that outputs the target model.

[0007] In the model selection device, the model selection unit can select, as the target model, a candidate model from among the plurality of candidate models that outputs an action for which the index is the highest.

[0008] In any one of the model selection devices, the model selection unit can select, as the target model, a candidate model from among the plurality of candidate models that outputs an action for which a statistical quantity of the index at a plurality of time points is the highest.

[0009] In any of the model selection apparatuses, the statistical quantity can include at least any of a mean value and a minimum value.

[0010] In any of the model selection apparatuses, the model selection unit can reselect the object model in accordance with the evaluation model being updated.

[0011] In any of the model selection apparatuses, the model selection unit can reselect the object model in accordance with a predetermined time elapsing.

[0012] Any of the model selection apparatuses can further include an input unit that receives a user input in accordance with the object model being output.

[0013] Any of the model selection apparatuses can further include a control unit that controls the control target using the object model.

[0014] Any of the model selection apparatuses can further include a job model generation unit that generates a plurality of job models that become the plurality of candidate models through the reinforcement learning.

[0015] Any of the model selection apparatuses can further include an evaluation model storage unit that stores the evaluation model.

[0016] Any of the model selection apparatuses can further include an evaluation model generation unit that generates the evaluation model through machine learning.

[0017] A model selection method is provided in a second aspect of the present disclosure. The model selection method is executed by a computer, and includes: storing a plurality of candidate models, the plurality of candidate models each being generated through reinforcement learning that uses an output of an evaluation model as at least a part of a reward, and being capable of outputting an action corresponding to a state of a device, the evaluation model outputting an index that evaluates the state of the device; acquiring a plurality of state data that represent the state of the device when each operation amount based on outputs of the plurality of candidate models is provided to a control target in the device; acquiring a plurality of indexes that are output by the evaluation model in accordance with each of the plurality of state data being input; selecting an object model for controlling the control target from the plurality of candidate models based on the plurality of indexes; and outputting the object model.

[0018] In a third aspect of the present invention, a non-transitory computer readable medium having a model selection program recorded thereon is provided. A computer functions as a candidate model storage section, a state data acquisition section, an index acquisition section, a model selection section, and an object model output section by executing the model selection program, the candidate model storage section stores a plurality of candidate models, the plurality of candidate models are each generated by reinforcement learning that uses an output of an evaluation model as a reward of at least a part, and are capable of outputting an action corresponding to a state in a device, the evaluation model outputs an index that evaluates the state of the device, the state data acquisition section acquires a plurality of state data that indicates the state of the device when each operation amount based on an output of the plurality of candidate models is provided to a control target in the device, the index acquisition section acquires a plurality of indexes that are output by the evaluation model for each of the plurality of state data, the model selection section selects an object model for controlling the control target from the plurality of candidate models based on the plurality of indexes, and the object model output section outputs the object model.

[0019] In addition, the above summary of the invention does not list all the features of the invention. Furthermore, sub-combinations of these feature groups can also be inventions. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 An example of a block diagram representing the control system 1.

[0021] Figure 2 An example of a block diagram representing the evaluation model management apparatus 200.

[0022] Figure 3 An example of a block diagram representing the job model management apparatus 300.

[0023] Figure 4 An example of a block diagram representing the model selection apparatus 400 of the present embodiment.

[0024] Figure 5 An example of a block diagram representing the control apparatus 500.

[0025] Figure 6 An example of a flowchart of a model selection method executable by the model selection apparatus 400 of the present embodiment.

[0026] Figure 7 An example of a block diagram representing the model selection apparatus 400 of the first modified example.

[0027] Figure 8 An example of a block diagram representing the model selection apparatus 400 of the second modified example.

[0028] Figure 9 An example of a block diagram representing the model selection apparatus 400 of the third modified example.

[0029] Figure 10 An example of a computer 9900 that represents a variety of ways in which the present application can be implemented in whole or in part.

[0030] Explanation of Reference Signs

[0031] 1 control system, 10 device, 15 control object, 100 simulator, 200 evaluation model management device, 210 evaluation model generation section, 220 evaluation model storage section, 230 evaluation model output section, 300 job model management device, 310 evaluation model acquisition section, 320 job model generation section, 330 job model storage section, 340 job model output section, 400 model selection device, 410 candidate model acquisition section, 420 candidate model storage section, 430 state data acquisition section, 440 index acquisition section, 450 model selection section, 460 object model output section, 470 input section, 500 control device, 510 object model acquisition section, 520 actual environment data acquisition section, 530 control section, 9900 computer, 9901 DVD-ROM, 9910 main controller, 9912 CPU, 9914 RAM, 9916 graphics controller, 9918 display device, 9920 input / output controller, 9922 communication interface, 9924 hard disk drive, 9926 DVD drive, 9930 ROM, 9940 input / output chip, 9942 keyboard. DETAILED DESCRIPTION

[0032] Hereinafter, the present application will be described through embodiments of the application, but the following embodiments do not limit the application involved in the claims. Furthermore, not all of the combinations of the features described in the embodiments are necessary for the solution means of the application.

[0033] Figure 1 An example of a block diagram that represents the control system 1. In addition, these modules are functionally separate functional modules, respectively, and can not necessarily be consistent with the actual device structure. That is, in the present drawing, although represented as one module, it can not necessarily be constituted by one device. Furthermore, in the present drawing, although represented as different modules, they can not necessarily be constituted by different devices. The same applies to the block diagrams hereinafter.

[0034] In the control system 1, an evaluation model that evaluates an index of the state of the device 10 is generated by machine learning of an output, and a job model is generated by reinforcement learning in which an output of the evaluation model is at least a part of a reward. Also, in the control system 1, the control target 15 in the device 10 is controlled using the generated job model. The control using such a job model is also called AI (Artificial Intelligence) control. In such a control system 1, in a case where a plurality of job models that can be utilized in AI control exist, the model selection device 400 of the present embodiment selects a model for control from among the plurality of candidates.

[0035] The control system 1 can include the device 10, the simulator 100, the evaluation model management device 200, the job model management device 300, the model selection device 400, and the control device 500.

[0036] The device 10 is a facility or a device provided with a control target 15. For example, the device 10 can be a factory, or a composite device in which a plurality of machines are compounded. As a factory, in addition to a chemical or biological industrial plant, a factory that manages and controls a wellhead and its surroundings of a gas field or an oil field, a factory that manages and controls power generation of water power, thermal power, atomic power, and the like, a factory that manages and controls environmental power generation of sunlight or wind power, and a factory that manages and controls water or a dam, and the like can be exemplified.

[0037] Hereinafter, a case where the device 10 is a distillation device that is one of process devices will be described as an example. Generally, in a distillation device, a low-boiling component is evaporated in a distillation column and is drawn out from the top of the column, and the vapor of the drawn-out low-boiling component is condensed by a condenser and is stored in a reflux drum. Then, the distillation device causes a part of the low-boiling component stored in the reflux drum to reflux into the distillation column, and causes it to contact with the vapor in the distillation column, and distill into a low-boiling component and a high-boiling component. In such a distillation device, as an example, in order to control the reflux amount, a valve provided between the reflux drum and the distillation column is controlled to be opened and closed.

[0038] The control target 15 is a machine provided to the plant 10 and becomes a target of control. For example, the control target 15 can be an actuator, i.e., an operation end, such as a valve, a heater, a motor, a fan, and a switch, which controls at least one of physical quantities, such as a quantity, a temperature, a pressure, a flow rate, a speed, and a pH of an object in a process of the plant 10, and performs a given operation corresponding to an operation quantity. Hereinafter, a case where the control target 15 is a valve provided between a reflux drum and a distillation column in a distillation apparatus will be described as an example. However, the present embodiment is not limited thereto. The control target 15 can also be a controller which controls the operation end. That is, the term "control" used in the present specification can be interpreted broadly as including not only direct control of the operation end but also indirect control of the operation end via the controller.

[0039] One or more sensors which can measure various states (physical quantities) inside and outside the plant 10 can be provided in the plant 10 provided with the control target 15. As an example, in a case where the plant 10 is a distillation apparatus, the sensors can output measurement values PV (Process Variable) of temperatures measured at various positions (for example, a column top, a column center, a column bottom, and the like) of the distillation apparatus, flow rates in various paths, and the like. Such measurement values PV can be included in state data which represents states of the plant 10. In addition, an operation quantity MV (Manipulated Variable) which represents an opening degree of the valve as the control target 15 can be included in the state data. In the state data, in addition to operation data which represents an operation state as a result of controlling the control target 15 in the above-described manner, consumption amount data which represents a consumption amount of energy or raw materials in the plant 10, disturbance environment data which represents a physical quantity which can act as a disturbance on the control of the control target 15, and the like can be included.

[0040] The distillation apparatus is one of apparatuses which are used very often in petroleum and chemical processes, and has characteristics that mutual interference between a column top and a column bottom is strong, a time constant is long, and an operation is nonlinear. In such a distillation apparatus, in a case where a valve is controlled to be opened and closed by PID (Proportional Integral Differential) or the like in order to control a reflux amount, it is difficult to achieve improvement in controllability. In addition, in a case where an operator manually operates such a valve for the purpose of ensuring quality, saving energy, reducing GHG (Green House Gas), and improving a yield, to what extent the valve is controlled to be opened and closed depends largely on experience and intuition of the operator.

[0041] Therefore, in controlling such a valve to be opened and closed, a job model generated by reinforcement learning is considered to be used. The model selection apparatus 400 of the present embodiment can select, for example, such a job model as a target.

[0041]

[0042] The simulator 100 simulates the operation in the plant 10. For example, the simulator 100 can be designed based on the design information in the plant 10, and execute the operation that simulates the behavior in the plant 10. The simulator 100 changes the environment by taking a signal that simulates the manipulated variable for the control target 15, and thereby outputs simulation data that simulates the state (for example, the predicted value of the sensor) in the plant 10. As an example, the simulator 100 can be constituted by a prediction model that predicts the state of the distillation apparatus and a plant control simulator. The prediction model can utilize a modeling technique of time series data using deep learning, and thereby be able to predict the state change of the reactor from the stored process data. Further, the plant control simulator can virtually simulate the PID control that derives the manipulated variable MV from the difference between the target value SV and the control variable CV, with respect to the control target 15. That is, the simulator 100 can simulate the behavior itself in the plant 10 in addition to the state predicted value.

[0043] The evaluation model management device 200 manages an evaluation model that outputs an index that evaluates the state of the plant 10. For example, the evaluation model management device 200 can generate the evaluation model by machine learning, and store the generated evaluation model in the device itself. Further, the evaluation model management device 200 can output the generated evaluation model to the operation model management device 300.

[0044] The operation model management device 300 manages a plurality of operation models that output actions corresponding to the state of the plant 10. For example, the operation model management device 300 can generate a plurality of operation models by reinforcement learning that uses at least a part of the output of the evaluation model managed by the evaluation model management device 200 as a reward, and store the generated plurality of operation models in the device itself. Further, the operation model management device 300 can output the generated plurality of operation models to the model selection device 400.

[0045] In a case where a plurality of operation models that can be utilized in AI control exist, the model selection device 400 selects a model for control from among the plurality of candidates. For example, the model selection device 400 can take the plurality of operation models managed by the operation model management device 300 as a plurality of candidate models, and select an object model for controlling the control target 15 from among the plurality of candidate models. Further, the model selection device 400 can output the selected object model to the control device 500.

[0046] The control device 500 controls the control target 15 using the object model. For example, the control device 500 can control the control target 15 in the plant 10 using the object model selected by the model selection device 400.

[0047] Thus, in the control system 1, the AI automatically searches for a bottleneck (potential failure) in the work, and generates an index for improvement as an evaluation model. Also, the AI makes repeated attempts based on a given index, and generates a work model indicating a better work method. Thus, according to the control system 1, an environment is provided in which the equipment 10 can be autonomously controlled using AI technology. In such a control system 1, in a case where there are a plurality of work models that can be utilized in AI control, the model selection device 400 of the present embodiment selects a model for control from among the plurality of candidates. For this reason, the details of each device will be explained in turn.

[0048] Figure 2 An example of a block diagram representing the evaluation model management device 200 is shown. The evaluation model management device 200 can be a computer such as a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, or a general-purpose computer, or a computer system in which a plurality of computers are connected. Such a computer system is also a computer in a broad sense. Furthermore, the evaluation model management device 200 can also be implemented by executing one or more virtual computer environments within a computer. Alternatively, the evaluation model management device 200 can be a special-purpose computer designed for management of evaluation models, or a special-purpose hardware implemented by a special-purpose circuit. Furthermore, in a case where connection with the Internet is possible, the evaluation model management device 200 can also be implemented by cloud computing.

[0049] The evaluation model management device 200 includes an evaluation model generation section 210, an evaluation model storage section 220, and an evaluation model output section 230.

[0050] The evaluation model generation section 210 generates an evaluation model that outputs an index that evaluates the state of the equipment 10. For example, the evaluation model generation section 210 can acquire a work target (a factory KPI (Key Performance Indicator), etc.) in the equipment 10, state data representing the state of the equipment 10, and a training label, and generate label data based on these. Also, the evaluation model generation section 210 can generate an evaluation model by an algorithm of machine learning using the generated label data as learning data. Since the generation process of the evaluation model itself can be arbitrary, a more detailed explanation is omitted here. The evaluation model generation section 210 supplies the generated evaluation model to the evaluation model storage section 220.

[0051] The evaluation model storage section 220 stores evaluation models. For example, the evaluation model storage section 220 can store the evaluation models generated by the evaluation model generation section 210. In addition, in the above description, a case where the evaluation model storage section 220 stores the evaluation models generated inside the evaluation model management apparatus 200 is exemplified, but is not limited thereto. The evaluation model storage section 220 can also store the evaluation models generated outside the evaluation model management apparatus 200. The evaluation model storage section 220 duplicates the stored evaluation models and supplies them to the evaluation model output section 230.

[0052] The evaluation model output section 230 outputs evaluation models. For example, the evaluation model output section 230 can output the evaluation models duplicated by the evaluation model storage section 220 to the job model management apparatus 300 via a network.

[0053] Figure 3 An example of a block diagram of the job model management apparatus 300 is shown. As with the evaluation model management apparatus 200, the job model management apparatus 300 can be a computer, or a computer system in which a plurality of computers are connected. Further, the job model management apparatus 300 can be realized by one or a plurality of virtual computer environments executable in a computer. Alternatively, the job model management apparatus 300 can be a special-purpose computer designed for management of job models, or a special-purpose hardware realized by a special-purpose circuit. Further, in a case where connection with the Internet is possible, the job model management apparatus 300 can also be realized by cloud computing.

[0054] The job model management apparatus 300 includes an evaluation model acquisition section 310, a job model generation section 320, a job model storage section 330, and a job model output section 340.

[0055] The evaluation model acquisition section 310 acquires an evaluation model that outputs an index of the state of the device 10. For example, the evaluation model acquisition section 310 can acquire the evaluation model output from the evaluation model output section 230 via a network. The evaluation model acquisition section 310 supplies the acquired evaluation model to the job model generation section 320.

[0056] The job model generation section 320 generates a plurality of job models capable of outputting actions corresponding to the states of the device 10 by reinforcement learning in which the output of the evaluation model is used as at least a part of a reward. As an example, such a job model can have a data table constituted by a combination (S, A) of a set of sampled state data S and an action A taken in each state, and a weight W calculated from the reward. In addition, at least a part of the reward used for calculating such a weight W can use the output of the evaluation model.

[0057] When generating such a job model, the job model generation section 320 can acquire learning environment data indicating the state of the learning environment. At this time, in a case where the simulator 100 that simulates the job in the device 10 is used as the learning environment, the job model generation section 320 can acquire simulation data from the simulator 100 as the learning environment data. However, it is not limited thereto. The actual device 10 can also be used as the learning environment. In this case, the job model generation section 320 can acquire state data indicating the state of the device 10 as the learning environment data.

[0058] Next, the job model generation section 320 can determine an action randomly or using a known AI algorithm such as FKDPP described later, and provide an operation amount based on the action to the control object in the learning environment. The state of the learning environment changes in correspondence thereto.

[0059] Then, the job model generation section 320 can acquire the learning environment data again. Thereby, the job model generation section 320 can acquire the state of the learning environment that has changed in accordance with the operation amount provided to the control object, the operation amount being based on the determined action.

[0060] Further, the job model generation section 320 can calculate the reward value based at least in part on the output of the evaluation model. As an example, the index outputted from the evaluation model can be directly calculated as the reward value in accordance with inputting the learning environment data indicating the changed state of the learning environment to the evaluation model.

[0061] The job model generation section 320, after repeating the acquisition process of the state corresponding to the determination of such an action a plurality of times, can update the job model by additionally adding new sample data that has not been saved so far to a new row in the data table in addition to the values of the weight column in the data table. The job model generation section 320 can generate the job model by repeating such an update process a plurality of times. Since the generation of the job model itself can be arbitrary, a more detailed explanation is omitted here.

[0062] The job model generation section 320 can generate a plurality of job models different from each other, for example, by performing such a job model generation process under different learning environments or with different learning algorithms. The job model generation section 320 supplies the generated plurality of job models to the job model storage section 330.

[0063] The job model storage section 330 stores a plurality of job models. For example, the job model storage section 330 can store a plurality of job models generated by the job model generation section 320. In addition, in the above description, a case where the job model storage section 330 stores a plurality of job models generated inside the job model management apparatus 300 is described as an example, but is not limited thereto. The job model storage section 330 can also store a part or all of a plurality of job models generated outside the job model management apparatus 300. The job model storage section 330 duplicates a plurality of job models stored therein and supplies the job model output section 340.

[0064] The job model output section 340 outputs a plurality of job models. For example, the job model output section 340 can output a plurality of job models duplicated by the job model storage section 330 to the model selection apparatus 400 via a network.

[0065] Figure 4 An example of a block diagram of the model selection apparatus 400 according to the present embodiment is shown in FIG. 4. As with the evaluation model management apparatus 200, the model selection apparatus 400 can also be a computer, or a computer system in which a plurality of computers are connected. Further, the model selection apparatus 400 can also be implemented by executing one or a plurality of virtual computer environments in a computer. Instead, the model selection apparatus 400 can be designed as a dedicated computer for selection of a model, or can be a dedicated hardware realized by a dedicated circuit. Further, in a case where the model selection apparatus 400 can be connected to the Internet, the model selection apparatus 400 can also be implemented by cloud computing.

[0066] The model selection apparatus 400 includes a candidate model acquisition section 410, a candidate model storage section 420, a state data acquisition section 430, an index acquisition section 440, a model selection section 450, an object model output section 460, and an input section 470.

[0067] The candidate model acquisition section 410 acquires a plurality of candidate models. For example, the candidate model acquisition section 410 can acquire a plurality of job models output by the job model output section 340 as a plurality of candidate models. The candidate model acquisition section 410 supplies the acquired plurality of candidate models to the candidate model storage section 420.

[0068] The candidate model storage section 420 stores a plurality of candidate models. For example, the candidate model storage section 420 can store a plurality of candidate models acquired by the candidate model acquisition section 410. The candidate model storage section 420 can store a plurality of candidate models, each of which is generated by reinforcement learning of at least a part of a reward of outputting an evaluation model that evaluates an index of a state of the device 10, for example, in the above-described manner, and can output an action corresponding to the state in the device 10.

[0069] The state data acquisition unit 430 acquires a plurality of state data. For example, the state data acquisition unit 430 can acquire a plurality of state data indicating a state of the device 10 when each operation amount based on an output of a plurality of candidate models stored in the candidate model storage unit 420 is provided to the control target 15 in the device 10. The state data acquisition unit 430 supplies the acquired plurality of state data to the index acquisition unit 440.

[0070] The index acquisition unit 440 acquires a plurality of indexes. For example, the index acquisition unit 440 can acquire a plurality of indexes output by the evaluation model according to each of a plurality of state data acquired by the state data acquisition unit 430 being input. The index acquisition unit 440 supplies the acquired plurality of indexes to the model selection unit 450.

[0071] The model selection unit 450 selects a target model. For example, the model selection unit 450 can select a target model for controlling the control target 15 from a plurality of candidate models stored in the candidate model storage unit 420 based on a plurality of indexes acquired by the index acquisition unit 440. The model selection unit 450 supplies information identifying the selected target model to the target model output unit 460.

[0072] The target model output unit 460 outputs a target model. For example, the target model output unit 460 can copy a target model from a plurality of candidate models stored in the candidate model storage unit 420 according to information identifying the target model selected by the model selection unit 450. Also, the target model output unit 460 can output the target model to the control device 500 via a network.

[0073] The input unit 470 receives a user input. For example, the input unit 470 can receive a user input according to a target model being output by the target model output unit 460. Also, in a case where the target model is reselected, the input unit 470 can trigger a plurality of state data being acquired by the state data acquisition unit 430 or a plurality of candidate models being acquired by the candidate model acquisition unit 410.

[0074] Figure 5 An example of a block diagram indicating the control device 500. The control device 500 can be, for example, a controller in a DCS (Distributed Control System), a controller in a medium-scale instrument system, or a real-time OS controller.

[0075] The control device 500 includes a target model acquisition unit 510, an actual environment data acquisition unit 520, and a control unit 530.

[0076] The object model acquisition section 510 acquires an object model. For example, the object model acquisition section 510 can acquire the object model output by the object model output section 460 via a network. The object model acquisition section 510 supplies the acquired object model to the control section 530.

[0077] The actual environment data acquisition section 520 acquires actual environment data representing the actual environment, that is, the state of the device 10. Such actual environment data can be the same data as the above-described state data. The actual environment data acquisition section 520 supplies the acquired actual environment data to the control section 530.

[0078] The control section 530 controls the control target 15 using the object model. For example, the control section 530 can determine an action by a known AI algorithm such as the FKDPP described later. Also, the control section 530 can supply the control target 15 in the device 10 with an operation amount after adding the determined action to the value of the control target 15. The control section 530 can perform AI control of the control target 15 using the object model selected by the model selection apparatus 400 in the above-described manner, for example.

[0079] Figure 6 An example of a flowchart representing a model selection method executable by the model selection apparatus 400 of the present embodiment.

[0080] In step S610, the model selection apparatus 400 acquires a plurality of candidate models. For example, the candidate model acquisition section 410 can acquire the plurality of job models output by the job model output section 340 of the job model management apparatus 300 via a network as the plurality of candidate models. However, it is not limited thereto. The candidate model acquisition section 410 can acquire the plurality of candidate models via other means (various storage devices or user input, etc.) other than the network, or can acquire the plurality of candidate models from another apparatus other than the job model management apparatus 300. The candidate model acquisition section 410 supplies the acquired plurality of candidate models to the candidate model storage section 420.

[0081] In step S620, the model selection device 400 stores a plurality of candidate models. For example, the candidate model storage 420 can store the plurality of candidate models acquired in step S610. In addition, in the above description, as an example, a case where the candidate model storage 420 stores a plurality of candidate models acquired from another device such as the job model management device 300 is described, but the present embodiment is not limited thereto. The candidate model storage 420 can also store a plurality of candidate models in advance. The candidate model storage 420 can store, for example, a plurality of candidate models each of which is generated by reinforcement learning that uses, as a reward, at least a part of an output of an evaluation model that evaluates an index of a state of the device 10, and can output an action corresponding to the state of the device 10, in the above-described manner. In other words, the candidate model storage 420 can store a plurality of candidate models that are different from each other, which are generated in different learning environments or by different learning algorithms, using at least a part of an output of a common evaluation model as a reward. Here, as an example, the candidate model storage 420 stores three candidate models of a candidate model x, a candidate model y, and a candidate model z.

[0082] In step S630, the model selection device 400 acquires a plurality of state data. For example, the state data acquisition unit 430 can acquire, as the state data, various physical quantities measured by various sensors provided to the device 10 from the device 10 via a network. However, the present embodiment is not limited thereto. The state data acquisition unit 430 can acquire the state data via other means different from the network, or can acquire the state data from another device different from the device 10.

[0083] Next, the state data acquisition section 430 can determine a plurality of actions respectively by using a plurality of candidate models stored in step S620 through an AI algorithm known as FKDPP or the like. In a case where such a core method is used, the state data acquisition section 430 can generate a vector of the state S from sensor values obtained by the acquired state data. Next, the state data acquisition section 430 can generate a combination of the state S and all adoptable actions A as an action determination table. Then, the state data acquisition section 430 can input the action determination table to each of the plurality of candidate models stored in step S620. In correspondence thereto, each of the plurality of candidate models can perform core calculation between each row of the action determination table and each sample data in the data table except for the weight column, and calculate a distance from each sample data respectively. Also, each of the plurality of candidate models can add values obtained by multiplying the distance calculated for each sample data by values of each weight column in order, and calculate a return expectation value in each action. The state data acquisition section 430 can select each action judged to have the highest return expectation value by using the plurality of candidate models in the above-described manner, for example, and thereby determine a plurality of actions. In other words, the state data acquisition section 430 can determine an action judged to have the highest return expectation value according to the state of the device 10 by each of the plurality of candidate models. Here, as an example, the state data acquisition section 430 determines an action Ax using a candidate model x, determines an action Ay using a candidate model y, and determines an action Az using a candidate model z.

[0084] Then, the state data acquisition section 430 can provide each operation amount after adding the determined plurality of actions to the value of the control target 15 to the control target 15 via the control device 500. In correspondence thereto, the state of the device 10 changes. The state data acquisition section 430 can further acquire state data indicating the state of the changed device. The state data acquisition section 430 can acquire a plurality of state data indicating the state of the device 10 when each operation amount based on the output of the plurality of candidate models is provided to the control target 15 in the device 10 in the above-described manner, for example. Here, the state data acquisition section 430 acquires state data Sx when an operation amount MVx based on the action Ax is provided to the control target 15, acquires state data Sy when an operation amount MVy based on the action Ay is provided to the control target 15, and acquires state data Sz when an operation amount MVz based on the action Az is provided to the control target 15. The state data acquisition section 430 supplies the acquired plurality of state data to the index acquisition section 440.

[0085] In step S640, the model selection device 400 acquires a plurality of indexes. For example, the index acquisition section 440 can input the plurality of state data acquired in step S630 to the evaluation models stored in the evaluation model storage section 220 respectively, and acquire a plurality of indexes output from the evaluation models respectively. The index acquisition section 440 can acquire a plurality of indexes output from the evaluation models according to each of the plurality of state data input thereto in the above-described manner, for example. Here, the index acquisition section 440 acquires an index Ix output from the evaluation model according to the input of the state data Sx, acquires an index Iy output from the evaluation model according to the input of the state data Sy, and acquires an index Iz output from the evaluation model according to the input of the state data Sz. The index acquisition section 440 supplies the acquired plurality of indexes to the model selection section 450.

[0086] In step S650, the model selection device 400 selects an object model. For example, the model selection section 450 can select an object model for controlling the control object 15 from the plurality of candidate models stored in step S620, on the basis of the plurality of indexes acquired in step S640.

[0087] At this time, the model selection section 450 can select a candidate model outputting an action for which the index among the plurality of candidate models is the highest, as the object model. As an example, in a case where the plurality of indexes are Ix > Iy > Iz, the model selection section 450 can select the candidate model x outputting the action Ax, as the object model.

[0088] In addition, in the above-described description, a case where the model selection section 450 selects a candidate model on the basis of indexes at one time point is exemplified, but is not limited thereto. The model selection section 450 can select a candidate model on the basis of a statistical quantity of indexes at a plurality of time points. As an example, in a case where the plurality of indexes are Iy_min > Iz_min > Ix_min (where min indicates a minimum value at a plurality of time points), the model selection section 450 can select the candidate model y outputting the action Ay, as the object model.

[0089] Further, in a case where the plurality of indexes are Iz_ave > Ix_ave > Iy_ave (where ave indicates an average value at a plurality of time points), the model selection section 450 can select the candidate model z outputting the action Az, as the object model.

[0090] The model selection unit 450 can also select, for example, a candidate model that outputs an action for which a statistical quantity of an index at a plurality of time points is the highest among the plurality of candidate models as the target model in the manner described above. At this time, the statistical quantity can include at least any one of a mean value or a minimum value. At this time, in a case where the candidate models are selected based on a plurality of statistical quantities, the model selection unit 450 can select a candidate model that outputs a sum or a weighted average after weighting addition of each statistical quantity that is the highest as the target model. The model selection unit 450 supplies information that identifies the selected target model to the target model output unit 460.

[0091] In step S660, the model selection device 400 outputs the target model. For example, the target model output unit 460 can copy the target model from the plurality of candidate models stored in step S620 in accordance with information that identifies the target model selected in step S650. Also, the target model output unit 460 can output the target model to the control device 500, for example, via a network. In correspondence thereto, the control device 500 can start AI control using the target model.

[0092] In step S670, the model selection device 400 determines whether to reselect the target model. For example, the input unit 470 can receive a user input in accordance with output of the target model in step S660. Also, in a case where an instruction to reselect the target model is received from the user, the input unit 470 can determine to reselect the target model.

[0093] In a case where it is determined to reselect the target model (Yes), the model selection device 400 can return the process to step S630 and continue the flow. In this case, the input unit 470 can trigger acquisition of the plurality of state data by the state data acquisition unit 430. Thereby, the model selection device 400 can reacquire the plurality of state data and reselect the target model. Note that, in the above description, a case where the model selection device 400 returns the process to step S630 is exemplified, but the present technology is not limited thereto. The model selection device 400 can return the process to step S610 and continue the flow. In this case, the input unit 470 can trigger acquisition of the plurality of candidate models by the candidate model acquisition unit 410. Thereby, the model selection device 400 can newly acquire the plurality of candidate models and reselect the target model from the newly acquired plurality of candidate models.

[0094] In a case where it is determined not to reselect the target model (No), the model selection device 400 ends the flow of the model selection method.

[0095] The model selection device 400 can also execute the flow of the model selection method again according to various triggers (event-triggered or time-triggered). For example, the model selection device 400 can execute the model selection method again taking the update of the evaluation model as a trigger. Thus, the model selection unit 450 can also reselect the target model according to the update of the evaluation model.

[0096] Further, the model selection device 400 can execute the model selection method again taking the elapse of a predetermined time from the selection of the target model as a trigger. Thus, the model selection unit 450 can also reselect the target model according to the elapse of the predetermined time.

[0097] Generally, a job model generated by reinforcement learning is black-boxed and difficult to evaluate. Thus, in a case where a plurality of such job models are available, it is difficult to select which job model to use for AI control. In contrast, the model selection device 400 of the present embodiment evaluates the respective states of the device 10 when each operation amount output based on a plurality of candidate models is provided to the control target 15 using an evaluation model, and selects the target model based on each index output by the evaluation model. Thus, according to the model selection device 400 of the present embodiment, which candidate model to use for AI control can be selected based on an objective result of evaluating a plurality of actions output by a plurality of candidate models using a common evaluation model.

[0098] Further, the model selection device 400 of the present embodiment can also select, as the target model, a candidate model that outputs an action with the highest index output by the evaluation model among the plurality of candidate models. Thus, according to the model selection device 400 of the present embodiment, a candidate model that most improves a job target such as a KPI can be selected as the target model.

[0099] Further, the model selection device 400 of the present embodiment can also select, as the target model, a candidate model that outputs an action with the highest statistical quantity of indexes at a plurality of time points among the plurality of candidate models. Thus, according to the model selection device 400 of the present embodiment, a candidate model that outputs an action with the highest index for a long period of time within a certain period of time can be selected as the target model, rather than a candidate model that outputs an action with the temporarily highest index. At this time, the average value can also be used as the statistical quantity. Thus, according to the model selection device 400 of the present embodiment, a candidate model that outputs an action with a higher index for a long period of time and stably can be selected as the target model. Further, the minimum value can also be used as the statistical quantity. Thus, according to the model selection device 400 of the present embodiment, even in a case of a job that requires a critical task such as a job in a factory, the best candidate model can be selected as the target model.

[0100] Further, the model selection device 400 of the present embodiment can reselect the object model in accordance with the evaluation model being updated. Thus, the model selection device 400 of the present embodiment can reselect the optimal candidate model as the object model with reference to a new job target even in a case where the job target is changed.

[0101] Further, the model selection device 400 of the present embodiment can also reselect the object model in accordance with a predetermined time elapsing. Thus, the model selection device 400 of the present embodiment can reselect the optimal candidate model as the object model with reference to the current state of the device 10 even in a case where the device 10 has undergone a temporal change from the point in time at which the object model was previously selected.

[0102] Further, the model selection device 400 of the present embodiment can receive a user input in accordance with the object model being output. Thus, the model selection device 400 of the present embodiment can feedback the result of the user judging the appropriateness of the object model after the object model is output. Also, the model selection device 400 of the present embodiment can reselect the object model in a case where the object model is inappropriate.

[0103] Figure 7 An example of a block diagram of the model selection device 400 of the first modified example is shown in FIG. 10. In Figure 7 the same functions and configurations as Figure 1 the same reference numerals are affixed to the members having the same functions and configurations, and the description is omitted except for the following differences. In the above-described embodiment, the evaluation model management device 200, the job model management device 300, the model selection device 400, and the control device 500 were provided as independent different devices as an example. However, these devices can be provided as one device as a whole or a part. In the present modified example, the model selection device 400 further provides the functions of the control device 500 in addition to the functions of the model selection device 400 of the above-described embodiment.

[0104] The model selection device 400 of the present modified example can further include a control section 530. That is, the model selection device 400 can further include the control section 530 that controls the control target 15 using the object model.

[0105] Further, in the present modified example, the object model output section 460 can output the selected object model to the control section 530 instead of the control device 500. Also, the control section 530 can acquire the object model output by the object model output section 460.

[0106] Furthermore, in this modified example, the status data acquisition unit 430 can supply the status data acquired in the AI ​​control to the control unit 530. That is, in this modified example, the status data acquisition unit 430 can also function as the actual environment data acquisition unit 520.

[0107] Furthermore, the control unit 530 can control the controlled object 15 using the object model. The model selection device 400 can, for example, provide the function of the control device 500 in the manner described above.

[0108] Therefore, the model selection device 400 of this modification can also control the controlled object 15 using the object model. Thus, according to the model selection device 400 of this modification, the functions of selecting an object model and controlling the controlled object 15 using the selected object model can be achieved through a single device. Furthermore, according to the model selection device 400 of this modification, there is no need to exchange object models between the model selection device 400 and the control device 500, thereby reducing communication costs and time.

[0109] Figure 8 An example of a block diagram showing the model selection device 400 of the second variation. Figure 8 In the middle, for those with Figure 1 The same reference numerals are used for components with the same functions and structures, and descriptions are omitted except for the following differences. In this variation, the model selection device 400, in addition to the functions of the model selection device 400 of the above embodiment, further provides the functions of the work model management device 300.

[0110] The model selection device 400 in this variant may further include an evaluation model acquisition unit 310 and a job model generation unit 320. That is, the model selection device 400 may further include a job model generation unit that generates multiple job models as multiple candidate models through reinforcement learning.

[0111] Furthermore, in this modified example, the job model generation unit 320 can supply the generated multiple job models to the candidate model storage unit 420. The candidate model storage unit 420 can also store the multiple job models supplied from the job model generation unit 320 as multiple candidate models.

[0112] Furthermore, in this modified example, when a new object model is selected, the input unit 470 can trigger the job model generation unit 320 to generate multiple job models. Therefore, the model selection device 400 of this modified example can also generate multiple new job models as multiple candidate models, and reselect the object model from the newly generated multiple candidate models. The model selection device 400 can also, for example, further provide the function of a job model management device 300 in the manner described above.

[0113] Thus, the model selection device 400 of the present modified example can also generate a plurality of job models that become candidates for the selection target model by itself through reinforcement learning. Thus, according to the model selection device 400 of the present modified example, the function of generating a plurality of job models that become candidates for the selection target model and the function of selecting the selection target model can be implemented by one device. Further, according to the model selection device 400 of the present modified example, a plurality of job models do not need to be exchanged between the job model management device 300 and the model selection device 400, and thus communication cost and time can be reduced.

[0114] Figure 9 An example of a block diagram of the model selection device 400 of the third modified example is shown in FIG. 12. In Figure 9 the same functions and configurations as those of the model selection device 400 of the first embodiment are denoted by the same reference numerals, and the description thereof is omitted except for the following differences. In the present modified example, the model selection device 400 further provides the function of the evaluation model management device 200 in addition to the functions of the model selection device 400 of the above-described embodiment. Figure 1

[0115] The model selection device 400 of the present modified example further includes the evaluation model generation section 210 and the evaluation model storage section 220. That is, the model selection device 400 can further include the evaluation model storage section 220 that stores the evaluation model. Further, the model selection device 400 can further include the evaluation model generation section 210 that generates the evaluation model through machine learning.

[0116] Further, in the present modified example, the index acquisition section 440 can input a plurality of state data to the evaluation models stored in the evaluation model storage section 220, respectively, and acquire a plurality of indexes output from the evaluation models, respectively. The model selection device 400 can further provide the functions of the evaluation model management device 200 in the above-described manner, for example.

[0117] Thus, the model selection device 400 of the present modified example can store the evaluation model. Thus, according to the model selection device 400 of the present modified example, a plurality of state data and a plurality of indexes do not need to be exchanged between the evaluation model management device 200 when a plurality of indexes are acquired, and thus communication cost and time can be reduced. Further, the model selection device 400 of the present modified example can generate the evaluation model by itself through machine learning. Thus, according to the model selection device 400 of the present modified example, the function of generating the evaluation model and the function of selecting the selection target model can be implemented by one device.

[0118] ​Here, illustrative embodiments have been described. However, the above embodiments can be modified or applied in various ways. For example, in the above variations, as different variations, the model selection device 400 is shown to further provide the functions of the control device 500, the job model management device 300, and the evaluation model management device 200. However, it is not limited to this. The model selection device 400 may further provide two or more of the functions of the control device 500, the job model management device 300, and the evaluation model management device 200, or it may further provide all of the functions. Thus, according to the model selection device 400, all functions related to the control of the controlled object 15 can be realized by a single device.

[0119] Furthermore, as an example in the above description, when acquiring multiple state data, the model selection device 400 provides each operational quantity based on the output of multiple candidate models to the controlled object 15 in the actual device 10 and acquires multiple state data from the actual device 10, but is not limited to this. The model selection device 400 may also provide each operational quantity based on the output of multiple candidate models to the controlled object in the simulation environment and acquire multiple state data from the simulator 100. Thus, the model selection device 400 can also complete the process up to selecting the target model in a simulation environment without using a physical machine.

[0120] Various embodiments of the present invention can be described with reference to flowcharts and block diagrams, where a module may represent (1) a stage of the process of performing an operation or (2) a part of a device that performs the operation. Specific stages and parts may be implemented by dedicated circuitry, programmable circuitry supplied together with computer-readable instructions stored on a computer-readable medium, and / or a processor supplied together with computer-readable instructions stored on a computer-readable medium. Dedicated circuitry may include digital and / or analog hardware circuitry, and may also include integrated circuits (ICs) and / or discrete circuitry. Programmable circuitry may include reconfigurable hardware circuitry, including logic AND, logic OR, logic XOR, logic NAND, logic NOR and other logic operations, flip-flops, registers, field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and other memory elements.

[0121] The computer readable medium can include any tangible device that can store instructions for execution by an appropriate device, and the computer readable medium that stores such instructions, as its storage thereof, includes a product that includes an instruction capable of being executed to implement a means for performing operations specified by a flowchart or a block diagram. Examples of the computer readable medium can include an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, or the like. More specific examples of the computer readable medium can include a floppy disk, a magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or a flash memory), an electrically erasable programmable read-only memory (EEPROM), a static random access memory (SRAM), a compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a Blu-ray (RTM) disk, a memory stick, an integrated circuit card, or the like.

[0122] The computer readable instructions include any one of source code and object code described by one or more programming languages including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or any combination of one or more programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, and the like, and existing procedural programming languages such as a "C" programming language or the like.

[0123] The computer readable instructions can be provided to a processor or a programmable circuit of a general purpose computer, a special purpose computer, or other programmable data processing apparatus via a local or local area network (LAN), the Internet, or the like, and the computer readable instructions are executed to implement a means for performing operations specified by a flowchart or a block diagram. Examples of the processor include a computer processor, a processing unit, a microprocessor, a digital signal processor, a controller, a microcontroller, or the like.

[0124] Figure 10 The computer 9900 represents an example of a variety of means that can implement the present application in whole or in part. The computer 9900 can function as a function associated with or perform one or more parts of an apparatus of an embodiment of the present application by a program installed in the computer 9900, or perform the operation or the one or more parts, and / or the computer 9900 can perform a process or a stage of an embodiment of the present application. In order for the computer 9900 to perform a specific operation associated with several or all of the modules of the flowcharts and block diagrams described in the present specification, such a program can be executed by the CPU 9912.

[0125] The computer 9900 of the present embodiment includes a CPU 9912, a RAM 9914, a graphics controller 9916, and a display device 9918, which are connected to each other through a host controller 9910. The computer 9900 further includes a communication interface 9922, a hard disk drive 9924, a DVD drive 9926, and an input / output unit such as an IC card drive, which are connected to the host controller 9910 via an input / output controller 9920. The computer further includes a ROM 9930 and a conventional input / output unit such as a keyboard 9942, which are connected to the input / output controller 9920 via an input / output chip 9940.

[0126] The CPU 9912 acts in accordance with programs stored in the ROM 9930 and the RAM 9914, thereby controlling the units. The graphics controller 9916 acquires image data generated by the CPU 9912 in a frame buffer or the like provided in the RAM 9914 or in itself, and displays the image data on the display device 9918.

[0127] The communication interface 9922 is capable of communicating with other electronic devices via a network. The hard disk drive 9924 stores programs and data used by the CPU 9912 in the computer 9900. The DVD drive 9926 reads programs or data from the DVD-ROM 9901, and provides the programs or data to the hard disk drive 9924 via the RAM 9914. The IC card drive reads programs and data from an IC card and / or writes programs and data to the IC card.

[0128] The ROM 9930 stores therein a boot program or the like to be executed by the computer 9900 at the time of activation and / or a program depending on the hardware of the computer 9900. The input / output chip 9940 can also connect various input / output units to the input / output controller 9920 via a parallel port, a serial port, a keyboard port, a mouse port, or the like.

[0129] A program is provided by a computer readable medium such as the DVD-ROM 9901 or an IC card. The program is read from the computer readable medium, and installed in the hard disk drive 9924, the RAM 9914, or the ROM 9930, which are also examples of computer readable media, and executed by the CPU 9912. Information processing described in these programs is read to the computer 9900, thereby bringing cooperation between the programs and the various types of hardware resources described above. An apparatus or a method can be constituted by the operation or processing of information accompanying the use of the computer 9900.

[0130] For example, in a case where communication is performed between the computer 9900 and an external device, the CPU 9912 can execute a communication program loaded on the RAM 9914, and instruct the communication interface 9922 to the communication processing based on the processing described in the communication program. The communication interface 9922 reads transmission data stored in a transmission buffer processing area provided in the RAM 9914, the hard disk drive 9924, the DVD-ROM 9901, or a recording medium such as an IC card, transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer processing area provided in the recording medium, and the like, under the control of the CPU 9912.

[0131] Further, the CPU 9912 can read all or a necessary part of a file or a database stored in the hard disk drive 9924, the DVD drive 9926 (DVD-ROM 9901), an external recording medium such as an IC card, to the RAM 9914, and execute various types of processing on the data on the RAM 9914. Then, the CPU 9912 writes the processed data back to the external recording medium.

[0132] Various types of information such as various types of programs, data, tables, and databases can be stored in the recording medium and subjected to information processing. The CPU 9912 executes various types of processing described throughout the present disclosure on the data read from the RAM 9914 and writes the results back to the RAM 9914, the various types of processing including various types of operations specified by an instruction sequence of a program, information processing, conditional judgment, conditional branching, unconditional branching, search / replacement of information, and the like. Further, the CPU 9912 can search for information in a file, a database, and the like in the recording medium. For example, in a case where a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 9912 can search for an entry that coincides with a condition that specifies an attribute value of the first attribute from the plurality of entries, and read an attribute value of the second attribute stored in the entry, thereby acquiring an attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0133] The program or the software module described above can be stored in a computer readable medium on or near the computer 9900. Further, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as the computer readable medium, whereby the program is provided to the computer 9900 via the network.

[0134] The above describes the present application using embodiments, but the technical scope of the present application is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or modifications can be made to the above embodiments. It is therefore intended that the technical scope of the present application includes any and all such changes or modifications.

[0135] The order of execution of the processes of the various processing shown in the devices, systems, programs, and methods of the claims, specifications, and drawings is not particularly shown as "earlier", "before", etc., and in addition, it should be noted that as long as the output of the previous processing is not used in the subsequent processing, it can be implemented in any order. Regarding the flow of actions in the claims, specifications, and drawings, even if it is described using "first", "next", etc. for convenience of explanation, it does not mean that it must be implemented in that order.

Claims

1. A model selection apparatus characterized by comprising: including: a candidate model storage section that stores a plurality of candidate models for controlling a control object in a device, the plurality of candidate models each being generated by reinforcement learning that uses an output of an evaluation model as at least a part of a reward, and being able to output an action corresponding to a state in the device, the evaluation model outputting an index that evaluates the state of the device; a state data acquisition section that acquires a plurality of state data that represent the state of the device when each operation amount based on an output of the plurality of candidate models is provided to the control object; an index acquisition section that acquires a plurality of indexes that are output by the evaluation model according to each of the plurality of state data being input; a model selection section that selects, based on the plurality of indexes, an object model for controlling the control object from among the plurality of candidate models; and an object model output section that outputs the object model. The model selection section selects, as the object model, a candidate model that outputs an action for which the index is highest among the plurality of candidate models.

2. The model selection apparatus according to claim 1, characterized by The model selection section selects, as the object model, a candidate model that outputs an action for which a statistical quantity of the index at a plurality of time points is highest among the plurality of candidate models.

3. The model selection apparatus according to claim 2, wherein The statistical quantity includes at least any one of an average value and a minimum value.

4. The model selection apparatus according to claim 3, wherein The model selection section reselects the object model according to the evaluation model being updated.

5. The model selection apparatus according to any one of claims 1 to 4, characterized by, The model selection section reselects the object model according to a predetermined time elapsing.

6. The model selection apparatus according to any one of claims 1 to 4, characterized by, The model selection apparatus further includes an input section that receives a user input according to the object model being output.

7. The model selection apparatus according to any one of claims 1 to 4, characterized by, The model selection apparatus further includes a control section that controls the control object using the object model.

8. The model selection apparatus according to any one of claims 1 to 4, characterized by, The model selection apparatus further includes a job model generation section that generates a plurality of job models that become the plurality of candidate models through the reinforcement learning.

9. The model selection apparatus according to any one of claims 1 to 4, characterized by, The model selection apparatus further includes an evaluation model storage section that stores the evaluation model.

10. The model selection apparatus according to any one of claims 1 to 4, characterized by, The model selection apparatus further includes an evaluation model generation section that generates the evaluation model through machine learning.

11. The model selection apparatus according to any one of claims 1 to 4, characterized by, 12. A model selection method characterized by being executed by a computer, the model selection method including: storing a plurality of candidate models for controlling a control object in a device, the plurality of candidate models each being generated by reinforcement learning that uses an output of an evaluation model as at least a part of a reward, and being able to output an action corresponding to a state in the device, the evaluation model outputting an index that evaluates the state of the device; acquiring a plurality of state data that represent the state of the device when each operation amount based on an output of the plurality of candidate models is provided to the control object; acquiring a plurality of indexes that are output by the evaluation model according to each of the plurality of state data being input; selecting, based on the plurality of indexes, an object model for controlling the control object from among the plurality of candidate models; and outputting the object model.

13. A computer program product in which a model selection program is recorded, the model selection program being characterized by ​ ​ The computer functions as a candidate model storage section, a state data acquisition section, an index acquisition section, a model selection section, and an object model output section by executing the model selection program, The candidate model storage section stores a plurality of candidate models for controlling a control object in a device, the plurality of candidate models being respectively generated by reinforcement learning that uses an output of an evaluation model as at least a part of a reward, the evaluation model outputting an index that evaluates a state of the device, and being able to output an action corresponding to the state in the device, The state data acquisition section acquires a plurality of state data indicating the state of the device when each operation amount based on an output of the plurality of candidate models is provided to the control object, The index acquisition section acquires a plurality of indexes output by the evaluation model according to each of the plurality of state data being input, The model selection section selects an object model for controlling the control object from the plurality of candidate models based on the plurality of indexes, The object model output section outputs the object model.

Images