Learning device, learning method, learning program, and control device

JP7593262B2Active Publication Date: 2024-12-03YOKOGAWA ELECTRIC CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2021129016
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-08-05
Publication Date
2024-12-03
Estimated Expiration
2041-08-05

Smart Images

  • Figure 0007593262000001
    Figure 0007593262000001
  • Figure 0007593262000002
    Figure 0007593262000002
  • Figure 0007593262000003
    Figure 0007593262000003
Patent Text Reader

Abstract

To realize a control method that realizes better control performance with a small number of times of learning.SOLUTION: A learning device comprises: a data acquisition unit that acquires initial setting data including state data for indicating a state of a facility and action data for indicating an action with respect to a controlled target prior to control of the controlled target provided in the facility by a machine learning model which outputs an action according to the state of the facility; and a pre-learning unit that initializes the machine learning model by pre-learning based on the initial setting data prior to a commencement of reinforcement learning according to control of the controlled target by the machine learning model.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a learning device, a learning method, a learning program, and a control device. [Background technology]

[0002] Patent document 1 states that "the system observes the current state of the environment in which the learning subject exists, executes a prescribed action in the current state, and gives some kind of reward for that action. This cycle is repeated through trial and error, and the system learns a strategy that maximizes the total reward as the optimal solution." [Prior art document] [Patent documents] [Patent Document 1] JP 2018-202564 Summary of the Invention

[0003] In a first aspect of the present invention, a learning device is provided. The learning device may include a data acquisition unit that acquires initial setting data including state data indicating a state of the equipment and behavior data indicating behavior toward the control target, prior to control of a control target provided in the equipment by a machine learning model that outputs behavior according to the state of the equipment. The learning device may include a pre-learning unit that initializes the machine learning model by pre-learning based on the initial setting data, prior to starting reinforcement learning of the machine learning model.

[0004] The learning device may further include an extraction unit that extracts sample data used for initial setting of the machine learning model from the initial setting data.

[0005] The extraction section may include a selection section that selects the initial setting data. The extraction section may extract the sample data from the selected initial setting data.

[0006] The extraction unit may have a definition unit that defines options for the machine learning model to select the action, and the extraction unit may extract, as the sample data, a combination of the state data included in the initial setting data and the action included in the options.

[0007] The machine learning model may output the action corresponding to the state of the equipment based on respective weights for combinations of the state data included in the initial setting data and each action included in the options.

[0008] The definition section may define the options based on a distribution of behaviors indicated by the behavior data included in the initial setting data.

[0009] The definition section may define the options that are common regardless of the state of the equipment.

[0010] The definition unit may define a plurality of options according to a state of the equipment.

[0011] The data acquisition unit may acquire the state data in response to the control of the control target by the machine learning model. The learning device may further include a reinforcement learning unit that updates the machine learning model by performing reinforcement learning on the state data and the action data acquired from the machine learning model in response to inputting the state data to the machine learning model as learning data.

[0012] The pre-learning unit may initialize the machine learning model based on the initial setting data in response to the input of the state data so as to select an action that is closest to the action data corresponding to the state data. The reinforcement learning unit may update the machine learning model so as to increase a reward obtained by a series of actions.

[0013] In a second aspect of the present invention, there is provided a control device. The control device may include the learning device. The control device may include a control unit that controls the control target using the machine learning model.

[0014] In a third aspect of the present invention, there is provided a learning method. The learning method may include acquiring initial setting data including state data indicating a state of the equipment and behavior data indicating behavior with respect to the control target, prior to control of a control target provided in the equipment by a machine learning model that outputs behavior according to the state of the equipment. The learning method may include initializing the machine learning model by pre-learning based on the initial setting data, prior to starting reinforcement learning of the machine learning model.

[0015] In a fourth aspect of the present invention, a learning program is provided. The learning program may be executed by a computer. The learning program may cause the computer to function as a data acquisition unit that acquires initial setting data including status data indicating a status of the equipment and behavior data indicating behavior toward the control target, prior to control of a control target provided in the equipment by a machine learning model that outputs behavior according to the status of the equipment. The learning program may cause the computer to function as a pre-learning unit that initializes the machine learning model by pre-learning based on the initial setting data, prior to the start of reinforcement learning of the machine learning model.

[0016] The above summary of the invention does not list all of the features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0017] [Figure 1] An example of a block diagram of a learning device 100 according to this embodiment is shown together with a facility 10 in which a control target 20 is provided. [Diagram 2]4 shows an example of a measurement value PV and a manipulated variable MV that may be acquired as state data by the learning device 100 according to this embodiment. [Diagram 3] 13 shows an example of a distribution of an operation change amount ΔMV that may be acquired as behavior data by the learning device 100 according to the present embodiment. [Figure 4] 1 shows an example of a flow of pre-learning by the learning device 100 according to this embodiment. [Diagram 5] 13 shows an example of a table of initially set machine learning models that are initially set by the learning device 100 according to this embodiment through pre-learning. [Figure 6] An example of a block diagram of a learning device 100 according to a modified example of this embodiment is shown together with a facility 10 in which a control target 20 is provided. [Figure 7] 11 shows an example of a calculation result when the learning device 100 according to the modified example of the present embodiment outputs an action according to a state using a machine learning model. [Figure 8] 13 shows an example of a table of machine learning models updated through reinforcement learning by the learning device 100 according to a modified example of the present embodiment. [Figure 9] An example of a block diagram of a control device 900 according to this embodiment is shown together with a facility 10 in which a controlled object 20 is provided. [Figure 10] An example of a computer 9900 in which aspects of the present invention may be embodied, in whole or in part, is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0019] 1 shows an example of a block diagram of a learning device 100 according to this embodiment, together with a facility 10 in which a control target 20 is provided. Before reinforcement learning of a machine learning model used for controlling the control target 20 is started, the learning device 100 according to this embodiment initializes the machine learning model by pre-learning.

[0020] The equipment 10 is a facility or device equipped with a control target 20. For example, the equipment 10 may be a plant, or a composite device in which a plurality of devices are combined. Examples of the plant include industrial plants such as chemical and bio plants, plants that manage and control wellheads and surrounding areas of gas and oil fields, plants that manage and control power generation such as hydroelectric, thermal and nuclear power, plants that manage and control environmental power generation such as solar and wind power, and plants that manage and control water supply and sewage, dams, etc. As an example, the equipment 10 may be a three-stage water tank or a heat treatment furnace, which is one type of process equipment.

[0021] The facility 10 is provided with a control target 20. In the present figure, a case where only one control target 20 is provided in the facility 10 is shown as an example, but the present invention is not limited to this. A plurality of control targets 20 may be provided in the facility 10.

[0022] The facility 10 may also be provided with one or more sensors (not shown) that measure various conditions (physical quantities) inside and outside the facility 10. The sensors output condition data indicating the measured conditions. Such condition data may include, for example, operation data, consumption data, and external environment data.

[0023] Here, the operating data indicates the operating state resulting from controlling the controlled object 20. For example, the operating data may include a measured value PV (Process Variable) called a process value. As an example, if the equipment 10 is a three-stage water tank, the operating data may include data indicating the water level of the water tank. Also, if the equipment 10 is a heat treatment furnace, the operating data may include data indicating the temperature inside the furnace (furnace temperature).

[0024] The operation data may also include data indicating a manipulated variable (MV) given to the control object 20. As an example, when the equipment 10 is a three-stage water tank, the operation data may include data indicating the opening degree of a valve, which is the control object 20. When the equipment 10 is a heat treatment furnace, the operation data may include data indicating a current to a heating wire of a heater, which is the control object 20.

[0025] The consumption data indicates the consumption of at least one of energy and raw materials in the facility 10. For example, the consumption data may include the consumption of electricity, fuel, and the like.

[0026] The external environment data indicates physical quantities that may act as disturbances to the control of the control target 20. For example, the external environment data may include the temperature, humidity, sunlight, wind direction, wind volume, and precipitation outside the facility 10, as well as various physical quantities that change with the control of other devices provided in the facility 10.

[0027] The controlled object 20 is a device or apparatus to be controlled. For example, the controlled object 20 may be an actuator such as a valve, heater, motor, fan, or switch that controls at least one physical quantity such as the amount, temperature, pressure, flow rate, speed, and pH of an object in the process of the equipment 10, and executes a required operation according to the manipulated variable MV. As an example, if the equipment 10 is a three-stage water tank, the controlled object 20 may be a valve that controls the water level of the water tank. Also, if the equipment 10 is a heat treatment furnace, the controlled object 20 may be a heater that controls the furnace temperature.

[0028] Such a control target 20 may be switchable between, for example, FB control based on a manipulated variable MV(FB) provided by a feedback (FB) controller and AI control based on a manipulated variable MV(AI) provided by a machine learning model (also called an AI: Artificial Intelligence model). In addition, such FB control may be, for example, control using at least one of proportional control (P control), integral control (I control), and differential control (D control), and may be, for example, PID control.

[0029] The learning device 100 according to the present embodiment initializes the machine learning model by pre-learning before the start of reinforcement learning of the machine learning model used for AI control of the control target 20. That is, the learning device 100 according to the present embodiment initializes the machine learning model so that the reinforcement learning of the machine learning model is started not from a clean state but from a state in which prior knowledge has been introduced by pre-learning.

[0030] The learning device 100 may be a computer such as a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, or a general-purpose computer, or may be a computer system to which multiple computers are connected. Such a computer system is also a computer in a broad sense. The learning device 100 may be implemented by one or more virtual computer environments executable within a computer. Alternatively, the learning device 100 may be a dedicated computer designed for pre-learning of a machine learning model, or may be dedicated hardware realized by a dedicated circuit. In addition, when the learning device 100 can be connected to the Internet, the learning device 100 may be realized by cloud computing.

[0031] The learning device 100 includes a data acquisition unit 110, an extraction unit 120, a pre-learning unit 130, and a model storage unit 140. These blocks are functionally separated functional blocks and may not necessarily match the actual device configuration. That is, even if one block is shown in this figure, it does not necessarily have to be configured by one device. Also, even if separate blocks are shown in this figure, they do not necessarily have to be configured by separate devices.

[0032] Prior to controlling the control target 20 provided in the facility 10 using a machine learning model that outputs an action according to the state of the facility 10, the data acquiring unit 110 acquires initial setting data including state data indicating the state of the facility 10 and action data indicating an action toward the control target 20. The data acquiring unit 110 supplies the acquired initial setting data to the extracting unit 120.

[0033] The extraction unit 120 extracts sample data used for the initial setting of the machine learning model from the initial setting data. More specifically, the extraction unit 120 includes a selection unit 122 and a definition unit .

[0034] The selection unit 122 selects the initial setting data acquired by the data acquisition unit 110. As a result, the extraction unit 120 extracts sample data from the selected initial setting data. The selection unit 122 supplies the selected initial setting data to the definition unit 124.

[0035] The definition unit 124 defines options for the machine learning model to select an action, based on the initial setting data selected by the selection unit 122. As a result, the extraction unit 120 extracts combinations of state data included in the initial setting data and actions included in the options as sample data. The extraction unit 120 supplies the extracted sample data to the pre-learning unit 130.

[0036] Prior to the start of reinforcement learning of the machine learning model, the pre-learning unit 130 initializes the machine learning model by pre-learning based on the initial setting data. More specifically, the pre-learning unit 130 initializes the machine learning model by pre-learning using the sample data extracted by the extraction unit 120 from the initial setting data acquired by the data acquisition unit 110.

[0037] The model storage unit 140 stores the machine learning model. When the pre-learning unit 130 performs pre-learning based on initial setting data, the model storage unit 140 stores the initialized machine learning model that is initialized by the pre-learning unit 130. In this way, the learning device 100 initializes the machine learning model by pre-learning before reinforcement learning of the machine learning model used for AI control of the control target 20 is started. This will be described in detail using an example in which the equipment 10 is a three-stage water tank.

[0038] FIG 2 shows an example of a measurement value PV and a manipulated variable MV that may be acquired as state data by the learning device 100 according to this embodiment. In this figure, the horizontal axis represents time T. In addition, in the upper part of this figure, the vertical axis represents the measurement value PV. Here, the measurement value PV represents the water level of the water tank. In addition, in the lower part of this figure, the vertical axis represents the manipulated variable MV. Here, the manipulated variable MV represents the valve opening.

[0039] In this figure, at time TA, the measured value PV was 30 and the manipulated variable MV was 10. Then, at time TB following time TA, the state changed to the manipulated variable MV = 5.1. The learning device 100 according to this embodiment may acquire at least such measured value PV and manipulated variable MV as state data.

[0040] FIG. 3 shows an example of the distribution of the operation change amount ΔMV that the learning device 100 according to the present embodiment may acquire as behavioral data. In this figure, the horizontal axis indicates the operation change amount ΔMV. Here, the operation change amount ΔMV indicates the change amount in the operation amount MV, that is, the value obtained by subtracting the current value from the next value in the operation amount MV. As an example, the operation change amount ΔMV at the time TA is 5.1-10=-4.9. The learning device 100 according to the present embodiment may acquire such an operation change amount ΔMV as behavioral data. Also, in this figure, the vertical axis indicates the number of times the corresponding operation change amount ΔMV has appeared. In this way, the operation change amount ΔMV may be distributed so that there are several groups of operation change amounts ΔMV that are concentrated to a certain extent, rather than any operation change amount ΔMV being randomly distributed as shown in this figure.

[0041] FIG. 4 shows an example of a flow of pre-learning by the learning device 100 according to this embodiment.

[0042] In step S410, the learning device 100 acquires initial setting data. For example, the data acquiring unit 110 acquires initial setting data including state data indicating the state of the equipment 10 and behavior data indicating behavior with respect to the control target 20, prior to control of the control target 20 provided in the equipment 10 by a machine learning model that outputs behavior according to the state of the equipment 10.

[0043] The data acquisition unit 110 acquires the initial setting data prior to the control (AI control) of the control target 20 by the machine learning model. At this time, the data acquisition unit 110 may acquire the initial setting data from data acquired when the control target 20 is subjected to FB control (e.g., PID control), may acquire the initial setting data from data acquired when the control target 20 is manually controlled by an operator, or may acquire the initial setting data from data acquired from the step response of the control target 20. Note that, when there is no actual data or there is insufficient real data, the data acquisition unit 110 may acquire the initial setting data from simulation data simulated based on a physical model of the control target 20. At this time, the data acquisition unit 110 may acquire the initial setting data so that it includes not only limited data for stabilizing the control target 20 to a target value from one initial state, but also various data in various situations due to a large number of initial conditions and disturbances.

[0044] For example, the data acquiring unit 110 receives status data measured by a sensor provided in the facility 10 from the facility 10 in chronological order via a network. However, this is not limited to this. The data acquiring unit 110 may acquire such status data by receiving it from a device other than the facility 10, by user input, or by reading it from various memory devices.

[0045] 2, the data acquiring unit 110 may acquire state data including a measured value PV as state 1 and a manipulated variable MV as state 2. In this way, the data acquiring unit 110 acquires state data indicating that, for example, at time TA, the state (state 1, state 2) was (30, 10).

[0046] Furthermore, the data acquiring unit 110 acquires data indicating the operation change amount ΔMV by subtracting the current value from the next value in the operation amount MV. As an example, assume that the state of the operation amount MV is changed to 5.1 at time TB following time TA. In this case, the data acquiring unit 110 acquires data indicating that the operation change amount ΔMV at time TA was -4.9 by subtracting the operation amount MV = 10 at time TA from the operation amount MV = 5.1 at time TB. The data acquiring unit 110 may acquire such an operation change amount ΔMV as behavior data. As a result, the data acquiring unit 110 acquires behavior data indicating that the behavior was (-4.9) at time TA, for example.

[0047] That is, the data acquiring unit 110 may acquire a state (30, 10) as the state data and an action (-4.9) as the action data for the time TA. This means that at the time TA, when the water level in the aquarium is 30 and the valve opening is 10%, the valve as the control target 20 is rotated by -4.9% (for example, 4.9% clockwise, which is the direction to close the valve).

[0048] The data acquisition unit 110 may acquire the initial setting data in this manner, for example. In the above description, the data acquisition unit 110 receives status data via a network, and acquires behavioral data by performing calculations using the received status data. However, the present invention is not limited to this. The data acquisition unit 110 may receive behavioral data as well as status data via a network. The data acquisition unit 110 supplies the acquired initial setting data to the extraction unit 120.

[0049] In step S420, the learning device 100 selects initial setting data. For example, the selection unit 122 selects the initial setting data acquired in step S410. That is, the selection unit 122 selects data to be used for pre-learning from the acquired initial setting data. At this time, the selection unit 122 may automatically calculate, for example, an evaluation value of the control performance, such as an overshoot / undershoot or hunting width, an offset value, and the like, and select the initial setting data so that each evaluation value is within a predetermined range. In addition, the selection unit 122 may evaluate the similarity between data based on a kernel function, for example, and select the initial setting data so that a lot of data with low similarity is included. The selection unit 122 supplies the selected initial setting data to the definition unit 124.

[0050] In step S430, the learning device 100 defines options. For example, the definition unit 124 defines options for the machine learning model to select an action based on the initial setting data selected in step S420. As an example, the definition unit 124 defines the options by analyzing the operation change amount ΔMV included in the initial setting data selected in step S420. In this case, the definition unit 124 may classify the operation change amount ΔMV by an existing cluster analysis technique such as the x-means method, and define the operation change amount ΔMV that is representative of each class (for example, the median or average value of the operation change amount ΔMV belonging to the same class) as an option. As an example, it is assumed that the operation change amount ΔMV included in the selected initial setting data is distributed as shown in FIG. 3. In this case, the definition unit 124 may classify the operation change amount ΔMV into seven classes, and define a set of operation change amounts ΔMV consisting of the representative values ​​of each class, here, the operation change amount ΔMV=-10, -5, -3, 0, 3, 5, and 10, as an option. In this manner, the definition unit 124 may define options based on the distribution of behaviors indicated by the behavior data included in the initial setting data.

[0051] In step S440, the learning device 100 extracts sample data. For example, the extraction unit 120 extracts sample data from the initial setting data selected in step S420. At this time, the extraction unit 120 does not use the actual data of the operation change amount ΔMV as it is, but replaces it with the closest operation change amount ΔMV′ among the options defined in step S430. Then, the extraction unit 120 extracts a combination of the state data at the same time and the replaced operation change amount ΔMV′ as sample data. As an example, when an action (−4.9) is acquired as action data for the time TA, the extraction unit 120 replaces “−4.9” with the closest operation change amount ΔMV′, “−5” here, among the options defined in step S430. Then, the extraction unit 120 extracts a combination of the state (30, 10) and the action (−5) as sample data for the time TA. In this way, the extraction unit 120 extracts a combination of the state data included in the initial setting data (more specifically, the initial setting data selected in step S420) and the action included in the option as sample data. The extraction unit 120 supplies the extracted sample data to the pre-learning unit 130.

[0052] In step S450, the learning device 100 performs pre-learning. For example, the pre-learning unit 130 initializes the machine learning model by pre-learning based on initial setting data prior to the start of reinforcement learning of the machine learning model. More specifically, the pre-learning unit 130 initializes the machine learning model by pre-learning using sample data extracted in step S440 from the initial setting data acquired in step S410.

[0053] Here, the pre-learning unit 130 stores in the machine learning model a policy for determining an action for controlling the controlled object 20 according to the state of the equipment 10. As an example, the pre-learning unit 130 stores a plurality of sample data extracted in step S440 in a table of the machine learning model. Such a table is composed of combinations of states (state 1, state 2), i.e., the measured values ​​PV and the manipulated variables MV, and actions, i.e., the manipulated variable ΔMV', and weights representing evaluations of the combinations. The pre-learning unit 130 stores in the table each combination of state and action in the sample data extracted in step S440, and sets the weights for each combination to an initial value (for example, all 1).

[0054] In the above description, the pre-learning unit 130 temporarily sets the weight for each combination to a uniform value, but the present invention is not limited to this example. If the importance of each combination is different, the pre-learning unit 130 may set the weight for each combination to a value according to the importance.

[0055] In the above description, the pre-learning unit 130 stores the states and actions in the sample data in a table as they are, but this is not limiting. The pre-learning unit 130 may normalize at least one of the states and actions in the sample data to a predetermined range (for example, 0 to 1) and store the normalized values.

[0056] In this way, the pre-learning unit 130 initializes the machine learning model based on the initial setting data, in response to input of state data, so as to select an action that is closest to the action data corresponding to the state data.

[0057] In step S460, the learning device 100 stores the machine learning model. For example, the model storage unit 140 stores the initially set machine learning model that was initially set by pre-learning in step 450.

[0058] 5 shows an example of a table of initially set machine learning models that are initially set by the learning device 100 according to this embodiment through pre-learning. As described above, state 1 indicates the measured value PV, which here indicates the water level of the aquarium. State 2 indicates the manipulated variable MV, which here indicates the valve opening. Action indicates the manipulated variable ΔMV'.

[0059] In this diagram, for example, the first line stores sample data in which the water level in the tank is 0, the valve opening is 0, and the valve is rotated +10% (10% counterclockwise). Similarly, the second line stores sample data in which the water level in the tank is 3, the valve opening is 10, and the valve is rotated +5%. In this table, the weights for each combination of such states and actions are all set to the initial value of 1.

[0060] The machine learning model determines actions using this initialized table as a policy, and outputs actions that correspond to the condition of the equipment based on the weights assigned to each combination of the status data included in the initial setting data and each action included in the options.

[0061] Note that, it should be noted here that only any one of the values ​​of -10, -5, -3, 0, 3, 5, and 10 is stored as the action. That is, only the actions included in the options defined by the definition unit 124 are stored in the machine learning model table. As a result, the action output by the machine learning model is limited to any one of the actions included in the options, that is, the operation change amount ΔMV=-10, -5, -3, 0, 3, 5, and 10.

[0062] Traditionally, PID control has been used in process control such as temperature adjustment, liquid level adjustment, and flow rate adjustment. While PID control provides stable control, overshoot and undershoot can occur at start-up. In particular, if overshoot occurs in temperature adjustment control, the temperature of the target object does not drop, causing problems such as delays in the start of production. It is possible to adjust the PID gains to prevent overshoot. However, in that case, the settling time until the response stabilizes becomes longer. For this reason, the current situation is that a lot of time and effort is spent adjusting each PID coefficient to an optimal value to improve control performance.

[0063] Therefore, AI control using machine learning models has also been proposed. In AI control, if a machine learning model is generated by machine learning to stabilize the target value of a certain control object closer to the target value as quickly as possible while suppressing phenomena such as overshooting, the expected control can be achieved. Reinforcement learning is one of the methods for generating such machine learning models. Generally, in reinforcement learning algorithms, the machine learning model randomly changes the manipulated variable at the beginning of learning, and the machine learning model is updated by repeating a large number of trial and error processes. In this case, the current issue is that it takes a huge amount of learning time to create a model with good control performance. In addition, when applying reinforcement learning to an N-th order lag system with a long response time such as temperature control, there are problems such as the randomness of action selection at the beginning of learning and the inappropriate setting of the action range, which means that no matter how many times learning is repeated, it is not possible to converge to the target value, or a model with good control performance cannot be obtained.

[0064] Therefore, the learning device 100 according to the present embodiment initializes the machine learning model by pre-learning before the reinforcement learning of the machine learning model used for the AI ​​control of the control target 20 is started. That is, the learning device 100 according to the present embodiment initializes the machine learning model so that the reinforcement learning of the machine learning model is started from a state where prior knowledge is introduced by pre-learning, rather than starting from a completely new state. As a result, according to the learning device 100 according to the present embodiment, prior knowledge of control is introduced into the machine learning model, so that it is possible to shorten the learning time in the subsequent reinforcement learning and improve the accuracy of the model. That is, in the initial stage of learning of the reinforcement learning executed after the fact, the machine learning model does not randomly select an action that changes the manipulated variable, but selects an action based on the initial setting including know-how such as PID control and manual control, so that a model that realizes better control performance can be obtained with a small number of learning times.

[0065] Furthermore, the learning device 100 according to this embodiment selects initial setting data and extracts sample data to be used for pre-learning from the selected initial setting data. As a result, according to the learning device 100 according to this embodiment, in pre-learning, instead of using all of the acquired initial setting data, data when the control performance was good or data with low similarity is actively used, thereby making it possible to further shorten the learning time and improve the accuracy of the model.

[0066] Furthermore, the learning device 100 according to the present embodiment defines options for the machine learning model to select an action, and extracts a combination of state data included in the initial setting data and actions included in the options as sample data to be used in pre-learning. As a result, the learning device 100 according to the present embodiment can limit the action output by the machine learning model to any of the actions included in the options, thereby suppressing adverse effects due to randomness in action selection and inappropriate setting of action range in the initial learning of reinforcement learning.

[0067] In this case, the learning device 100 according to the present embodiment defines options based on the distribution of actions indicated by the action data included in the initial setting data. As a result, the learning device 100 according to the present embodiment can be initially set so that the machine learning model outputs actions that are frequently taken under PID control or manual control, for example.

[0068] Fig. 6 shows an example of a block diagram of a learning device 100 according to a modified example of this embodiment. In Fig. 6, the same reference numerals are used for components having the same functions and configurations as those in Fig. 1, and descriptions thereof will be omitted except for the following differences. The learning device 100 according to this modified example further has a function of updating the machine learning model by reinforcement learning, in addition to the function of initializing the machine learning model by pre-learning. The learning device 100 according to this modified example further includes a reinforcement learning unit 610 in addition to the functional units included in the learning device 100 according to the above-described embodiment.

[0069] In this modification, the data acquisition unit 110 acquires state data in response to the control target 20 being controlled by the machine learning model. That is, the data acquisition unit 110 acquires state data under AI control using an initially set machine learning model or an updated machine learning model obtained by updating the initially set machine learning model. The data acquisition unit 110 supplies the acquired state data to the reinforcement learning unit 610. In addition, the data acquisition unit 110 inputs the acquired state data to the machine learning model stored in the model storage unit 140.

[0070] The reinforcement learning unit 610 updates the machine learning model by performing reinforcement learning on the state data and the behavior data acquired from the machine learning model in response to inputting the state data to the machine learning model as learning data. For example, the reinforcement learning unit 610 acquires, as behavior data, the behavior output by the machine learning model in response to inputting the state data acquired by the data acquisition unit 110 to the machine learning model stored in the model storage unit 140 (an initially set machine learning model or an updated machine learning model obtained by updating the initially set machine learning model).

[0071] Here, the machine learning model outputs an action according to the state of the facility 10, for example, as follows. The machine learning model performs kernel calculations between each sample data stored in the table for combinations of the input state data and each action included in the options, and calculates the distance between each sample data. Then, the machine learning model sequentially adds up the distances calculated for each sample data multiplied by the respective weights, and calculates an evaluation value for each combination. Then, the machine learning model outputs the action in the combination with the highest evaluation value as the next action. The reinforcement learning unit 610 acquires, for example, the action output from the machine learning model in this way as action data. Then, the reinforcement learning unit 610 executes reinforcement learning using the state data and action data under AI control acquired in this way as learning data.

[0072] The reinforcement learning here may be the same as conventional reinforcement learning, except that the machine learning model is initially set. For example, the reinforcement learning unit 610 executes reinforcement learning by a known algorithm such as KDPP (Kernel Dynamic Policy Programming) based on each sample data in the learning data and the reward value for the sample data. At this time, the reinforcement learning unit 610 evaluates the action selected based on the next state data of the operated control target 20 and calculates the reward value. In this case, as an example, the reinforcement learning unit 610 may set a reward function so that the closer the measurement value PV is to the target value, the higher the reward value becomes. As a result, the reinforcement learning unit 610 overwrites the weight of each sample data in the initially set table, and adds new sample data that has not been saved to the table.

[0073] FIG. 7 shows an example of a calculation result when the learning device 100 according to the modified embodiment of the present invention outputs an action according to a state by a machine learning model. In this figure, a case where the learning device 100 acquires a state (state 1, state 2)=(0.3, 0.6) as state data under AI control is shown as an example. In addition, in this figure, a case where a set of operation change amounts ΔMV consisting of operation change amounts ΔMV=-10, -5, -3, 0, 3, 5, and 10 is defined as an option is shown as an example. Therefore, in this figure, each row shows a combination of the input state data and each action included in the option.

[0074] As an example, the first line means that in state (0.3, 0.6), action (10) is selected, which is one of the options. Similarly, the second line means that in state (0.3, 0.6), action (5) is selected, which is one of the options. The machine learning model calculates an evaluation value for each combination of such state data and each action included in the options.

[0075] For example, the machine learning model performs kernel calculations between the combination in the first row and each sample data stored in the table, and calculates the distance between each sample data. Then, the machine learning model sequentially adds up the distances calculated for each sample data multiplied by the respective weights to calculate the evaluation value S(10). The machine learning model repeatedly executes such calculations to calculate the evaluation value S(5) when action (5) is selected, the evaluation value S(3) when action (3) is selected, the evaluation value S(0) when action (0) is selected, the evaluation value S(-3) when action (-3) is selected, the evaluation value S(-5) when action (-5) is selected, and the evaluation value S(-10) when action (-10) is selected. Then, the machine learning model outputs the action in the combination with the highest evaluation value as the next action. As an example, when the evaluation value S(-5) is the highest, the machine learning model outputs the action (-5) as the next action.

[0076] FIG. 8 shows an example of a table of a machine learning model updated by the learning device 100 according to the modified example of the present embodiment through reinforcement learning. As shown in this figure, the weight of each sample data initially set in the pre-learning has been updated from the initial value. Also, as shown in this figure, new sample data not saved in the initial learning has been added to the table. The reinforcement learning unit 610 evaluates the action output by the machine learning model according to the evaluation result of FIG. 7, for example, based on the next state data in the facility 10, and calculates a reward value. Then, the reinforcement learning unit 610 updates the machine learning model so as to further increase the reward obtained by a series of actions. That is, the reinforcement learning unit 610 overwrites the weight of each sample data saved in the table so as to make it easier for the machine learning model to output an action that increases the reward. Also, the reinforcement learning unit 610 can add new sample data not saved so far to the table. For example, in this way, the reinforcement learning unit 610 updates the machine learning model so as to further increase the reward obtained by a series of actions.

[0077] In general reinforcement learning, in the early stages of learning, the machine learning model selects random actions, whereas in the learning device of this modified example, actions based on initial settings including know-how on PID control, manual control, etc. are selected, making it possible to explore a control method that can achieve better control performance with fewer learning iterations.

[0078] 9 shows an example of a block diagram of a control device 900 according to this embodiment, together with a facility 10 in which a control target 20 is provided. In FIG. 9, the same reference numerals are used for components having the same functions and configurations as those in FIG. 6, and descriptions thereof will be omitted except for the following differences. In addition to the functions of the learning device 100 described above, the control device 900 according to this embodiment further has a function of controlling the control target 20 using a machine learning model. In addition to the functional units included in the learning device 100 described above, the control device 900 further includes a control unit 910.

[0079] The control unit 910 controls the control target 20 by the machine learning model. For example, the control unit 910 provides the action output by the machine learning model to the control target 20, and controls the control target 20. That is, the control unit 910 may function as a so-called AI controller. In this manner, the control device 900 according to the present embodiment may include the above-mentioned learning device 100 and the control unit 910 that controls the control target by the machine learning model. In this case, the control unit 910 and other functional units may be configured integrally or separately (for example, the other functional units may be executed in the cloud, etc.).

[0080] Moreover, such a control device 900 may be combined with an existing FB controller, for example, a PID controller, and the control of the control target 20 may be switched depending on the situation. That is, the control device 900 may further include an FB controller, and control the control target 20 by switching between FB control by the FB controller and AI control by a machine learning model depending on various situations (for example, the progress of learning, control accuracy, etc.).

[0081] Up to this point, the above-mentioned embodiment has been described by exemplifying one possible embodiment. However, the above-mentioned embodiment may be modified or applied in various ways. For example, in the above description, a case where the definition unit 124 defines a common option regardless of the state of the equipment has been shown as an example. That is, a case where the definition unit 124 defines a set of operation change amounts ΔMV consisting of operation change amounts ΔMV=-10, -5, -3, 0, 3, 5, and 10 as the only option regardless of the state of the equipment 10 has been shown as an example. However, if an analysis is performed for each state of the equipment 10, the distribution of the operation change amounts ΔMV may also be different. For example, in a state where the water tank is close to empty (the measurement value PV is close to 0), it is considered that the number of occurrences of operation change amounts ΔMV with a large absolute value and a + sign is high. Conversely, in a state where the water level of the water tank is close to the target value, it is considered that the number of occurrences of operation change amounts ΔMV with a small absolute value and a + or - sign is high. In this way, when the state of the equipment 10 can affect the number of occurrences of the operation change amount ΔMV, the definition unit 124 may define a plurality of options according to the state of the equipment 10.

[0082] Various embodiments of the present invention may be described with reference to flow charts and block diagrams, where the blocks may represent (1) stages of a process in which operations are performed or (2) sections of an apparatus responsible for performing the operations. Particular stages and sections may be implemented by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable medium, and / or a processor provided with computer readable instructions stored on a computer readable medium. Dedicated circuitry may include digital and / or analog hardware circuitry and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, memory elements such as flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

[0083] A computer-readable medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that the computer-readable medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray (RTM) disks, memory sticks, integrated circuit cards, and the like.

[0084] The computer readable instructions may include either assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0085] Computer readable instructions may be provided to a processor or programmable circuitry of a general purpose computer, special purpose computer, or other programmable data processing apparatus, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to execute the computer readable instructions to create means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0086] 10 illustrates an example of a computer 9900 in which aspects of the present invention may be embodied in whole or in part. Programs installed on the computer 9900 may cause the computer 9900 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to an embodiment of the present invention, and / or to perform a process or steps of a process according to an embodiment of the present invention. Such programs may be executed by the CPU 9912 to cause the computer 9900 to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0087] The computer 9900 according to this embodiment includes a CPU 9912, a RAM 9914, a graphic controller 9916, and a display device 9918, which are connected to each other by a host controller 9910. The computer 9900 also includes input / output units such as a communication interface 9922, a hard disk drive 9924, a DVD drive 9926, and an IC card drive, which are connected to the host controller 9910 via an input / output controller 9920. The computer also includes legacy input / output units such as a ROM 9930 and a keyboard 9942, which are connected to the input / output controller 9920 via an input / output chip 9940.

[0088] The CPU 9912 operates according to programs stored in the ROM 9930 and the RAM 9914, thereby controlling each unit. The graphics controller 9916 retrieves image data generated by the CPU 9912 into a frame buffer or the like provided in the RAM 9914 or into itself, and causes the image data to be displayed on the display device 9918.

[0089] The communication interface 9922 communicates with other electronic devices over a network. The hard disk drive 9924 stores programs and data used by the CPU 9912 in the computer 9900. The DVD drive 9926 reads programs or data from the DVD-ROM 9901 and provides the programs or data to the hard disk drive 9924 via the RAM 9914. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0090] The ROM 9930 stores therein a boot program or the like executed by the computer 9900 upon activation, and / or a program that depends on the hardware of the computer 9900. The input / output chip 9940 may also connect various input / output units to the input / output controller 9920 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0091] The programs are provided by a computer-readable medium such as a DVD-ROM 9901 or an IC card. The programs are read from the computer-readable medium, installed in the hard disk drive 9924, the RAM 9914, or the ROM 9930, which are also examples of computer-readable media, and executed by the CPU 9912. Information processing described in these programs is read by the computer 9900, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing information manipulation or processing according to the use of the computer 9900.

[0092] For example, when communication is performed between the computer 9900 and an external device, the CPU 9912 may execute a communication program loaded in the RAM 9914 and instruct the communication interface 9922 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 9912, the communication interface 9922 reads transmission data stored in a transmission buffer processing area provided in the RAM 9914, the hard disk drive 9924, the DVD-ROM 9901, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0093] The CPU 9912 may also cause all or a necessary portion of a file or database stored on an external recording medium such as the hard disk drive 9924, the DVD drive 9926 (DVD-ROM 9901), an IC card, etc. to be read into the RAM 9914, and perform various types of processing on the data on the RAM 9914. The CPU 9912 then writes back the processed data to the external recording medium.

[0094] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 9912 may perform various types of processing on the data read from the RAM 9914, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 9914. The CPU 9912 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 9912 may search for an entry that matches a condition, in which an attribute value of the first attribute is specified, from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0095] The above-described programs or software modules may be stored in a computer-readable medium on the computer 9900 or in the vicinity of the computer 9900. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the programs to the computer 9900 via the network.

[0096] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0097] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order. [Explanation of symbols]

[0098] 10 Equipment 20 Control Object 100 Learning Device 110 Data Acquisition Section 120 Extraction part 122 Selection Department 124 Definition part 130 Pre-Learning Section 140 Model Memory Section 610 Reinforcement Learning Department 900 Control device 910 Control section 9900 Computer 9901 DVD-ROM 9910 Host Controller 9912 CPU 9914 RAM 9916 Graphics Controller 9918 Display Device 9920 Input / Output Controller 9922 Communication Interface 9924 Hard Disk Drive 9926 DVD Drive 9930 ROM 9940 Input / Output Chip 9942 Keyboard

Claims

1. a data acquisition unit that acquires initial setting data including status data indicating a status of the equipment and behavior data indicating a behavior with respect to the control target, prior to control of a control target provided in the equipment by a machine learning model that outputs a behavior according to a status of the equipment; An extraction unit that extracts sample data used for initial setup of the machine learning model from the initial setup data; a pre-learning unit that initializes the machine learning model by pre-learning based on the sample data prior to the start of reinforcement learning of the machine learning model; Equipped with The initial setting data is obtained by at least one of a feedback control of the controlled object, a manual control of the controlled object, a step response of the controlled object, or a simulation of the controlled object. Learning device.

2. The extraction unit has a selection unit that selects the initial setting data, The learning device according to claim 1 , wherein the extraction unit extracts the sample data from the selected initial setting data.

3. The extraction unit has a definition unit that defines options for the machine learning model to select the action, The learning device according to claim 1 , wherein the extraction unit extracts, as the sample data, a combination of the state data included in the initial setting data and an action included in the option.

4. The learning device according to claim 3 , wherein the machine learning model outputs the action corresponding to the state of the equipment based on respective weights for combinations of the state data included in the initial setting data and each action included in the options.

5. The learning device according to claim 3 , wherein the definition unit defines the options based on a distribution of behaviors indicated by the behavior data included in the initial setting data.

6. The learning device according to claim 3 , wherein the definition unit defines the options in common regardless of a state of the equipment.

7. The learning device according to claim 3 , wherein the definition unit defines a plurality of the options according to a state of the equipment.

8. The data acquisition unit acquires the state data in response to the control of the control target by the machine learning model; a reinforcement learning unit that updates the machine learning model by performing reinforcement learning using the state data and the action data acquired from the machine learning model in response to inputting the state data to the machine learning model as learning data; A learning device according to any one of claims 1 to 7.

9. the pre-learning unit initially sets a combination of the state data and the action data based on the sample data in a table serving as a policy for determining an action for controlling the control target; The reinforcement learning unit updates the table of the machine learning model so as to increase a reward obtained by a series of actions. The learning device according to claim 8.

10. a data acquisition unit that acquires initial setting data including status data indicating a status of the equipment and behavior data indicating a behavior with respect to the control target, prior to control of a control target provided in the equipment by a machine learning model that outputs a behavior according to a status of the equipment; a pre-learning unit that initializes the machine learning model by pre-learning based on the initial setting data prior to the start of reinforcement learning of the machine learning model; Equipped with the initial setting data is obtained by at least one of a feedback control of the controlled object, a manual control of the controlled object, a step response of the controlled object, or a simulation of the controlled object; The data acquisition unit acquires the state data in response to the control of the control target by the machine learning model; a reinforcement learning unit that updates the machine learning model by performing reinforcement learning using the state data and the action data acquired from the machine learning model in response to inputting the state data to the machine learning model as learning data; the advance learning unit initially sets a combination of the state data and the action data based on the initial setting data in a table serving as a policy for determining an action for controlling the control target; The reinforcement learning unit updates the table of the machine learning model so as to increase a reward obtained by a series of actions. Learning device.

11. A learning device according to any one of claims 1 to 10; a control unit that controls the control target using the machine learning model; A control device comprising:

12. Prior to controlling a control target provided in the equipment by a machine learning model that outputs an action according to the state of the equipment, initial setting data including state data indicating the state of the equipment and action data indicating an action with respect to the control target is acquired; Extracting sample data used for initial setup of the machine learning model from the initial setup data; Prior to the start of reinforcement learning of the machine learning model, the machine learning model is initially configured by pre-learning based on the sample data; Equipped with The initial setting data is obtained by at least one of a feedback control of the controlled object, a manual control of the controlled object, a step response of the controlled object, or a simulation of the controlled object. How to learn.

13. Prior to controlling a control target provided in the equipment by a machine learning model that outputs an action according to the state of the equipment, initial setting data including state data indicating the state of the equipment and action data indicating an action with respect to the control target is acquired; Prior to the start of reinforcement learning of the machine learning model, initializing the machine learning model by pre-learning based on the initial setting data; acquiring the state data in response to the control of the control target by the machine learning model; updating the machine learning model by performing reinforcement learning on the state data and the behavior data acquired from the machine learning model in response to inputting the state data into the machine learning model as learning data; the initial setting data is obtained by at least one of a feedback control of the controlled object, a manual control of the controlled object, a step response of the controlled object, or a simulation of the controlled object; In the pre-learning, a combination of the state data and the action data based on the initial setting data is initially set in a table serving as a policy for determining an action for controlling the control target; In updating the machine learning model, the table of the machine learning model is updated so as to increase the reward obtained by a series of actions. A learning method that provides:

14. When executed by a computer, the computer is a data acquisition unit that acquires initial setting data including status data indicating a status of the equipment and behavior data indicating a behavior with respect to the control target, prior to control of a control target provided in the equipment by a machine learning model that outputs a behavior according to a status of the equipment; An extraction unit that extracts sample data used for initial setup of the machine learning model from the initial setup data; a pre-learning unit that initializes the machine learning model by pre-learning based on the sample data prior to the start of reinforcement learning of the machine learning model; and make it work. The initial setting data is obtained by at least one of a feedback control of the controlled object, a manual control of the controlled object, a step response of the controlled object, or a simulation of the controlled object. Study program.

15. When executed by a computer, the computer is a data acquisition unit that acquires initial setting data including status data indicating a status of the equipment and behavior data indicating a behavior with respect to the control target, prior to control of a control target provided in the equipment by a machine learning model that outputs a behavior according to a status of the equipment; a pre-learning unit that initializes the machine learning model by pre-learning based on the initial setting data prior to the start of reinforcement learning of the machine learning model; and make it work. the initial setting data is obtained by at least one of a feedback control of the controlled object, a manual control of the controlled object, a step response of the controlled object, or a simulation of the controlled object; The data acquisition unit acquires the state data in response to the control of the control target by the machine learning model; causing the computer to further function as a reinforcement learning unit that updates the machine learning model by performing reinforcement learning on the state data and the action data acquired from the machine learning model in response to inputting the state data into the machine learning model as learning data; the advance learning unit initially sets a combination of the state data and the action data based on the initial setting data in a table serving as a policy for determining an action for controlling the control target; The reinforcement learning unit updates the table of the machine learning model so as to increase a reward obtained by a series of actions. Study program.

Citation Information

Patent Citations

  • Control device for plant

    JP2008305194A

  • Control device, control method and control program

    JP2020098538A