Inverse reinforcement learning method and inverse reinforcement learning device

Abstract

To provide an inverse enforcement learning method capable of generating reward functions and measure functions applicable to a plurality of environment conditions and achieving efficient learning.SOLUTION: An inverse enforcement learning method comprises a reward function generating step and a measure function generating step. The reward function generating step generates a first reward function common to a plurality of environment conditions by executing inverse enforcement learning based on expert data indicating a locus of behaviors of an expert. The measure function generating step generates a first measure function common to the plurality of environment conditions by executing enforcement learning based on agent data indicating a locus of behaviors of an agent in the plurality of environment conditions or an environment condition representing the plurality of environment conditions.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an inverse reinforcement learning method and an inverse reinforcement learning device. [Background technology]

[0002] There is a Reinforcement Learning (RL) technique that seeks a policy to maximize the reward obtained from the environment, and its application to the automation of plant operations, etc. is being considered. In addition, because it is difficult to design the reward function used in reinforcement learning, an inverse reinforcement learning method has been proposed that estimates the reward function based on expert data showing the operation of experts (experts).

[0003] The reward function obtained by inverse reinforcement learning and the policy function obtained by reinforcement learning using the inverse reinforcement learning have appropriate function specifications that change depending on environmental conditions. In this regard, inverse reinforcement learning described in Patent Document 1, the application environment is classified, and the reward function and policy function to be used are switched according to the application environment, so that a function appropriate to the environment is used. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 163538 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, in the method described in Patent Document 1, the application environment is classified, and the reward function and the policy function to be used are switched according to the application environment. For this reason, learning related to environment recognition (classification) is necessary, and since the result of environment recognition affects the quality of the obtained policy, it is difficult to efficiently proceed with learning.

[0006] In view of the above circumstances, an object of the present disclosure is to provide an inverse reinforcement learning method and an inverse reinforcement learning device that can generate reward functions and policy functions applicable to a plurality of environmental conditions to realize efficient learning. [Means for solving the problem]

[0007] In order to achieve the above object, an inverse reinforcement learning method according to at least one embodiment of the present disclosure includes: A reward function generating step of performing inverse reinforcement learning based on expert data indicating a trajectory of an expert's actions under a plurality of environmental conditions to generate a first reward function common to the plurality of environmental conditions; a policy function generating step of executing reinforcement learning based on the first reward function generated in the reward function generating step and agent data indicating a trajectory of an agent's behavior under the plurality of environmental conditions or an environmental condition representative of the plurality of environmental conditions, to generate a first policy function common to the plurality of environmental conditions; Equipped with.

[0008] In order to achieve the above object, an inverse reinforcement learning device according to at least one embodiment of the present disclosure includes: A reward function generating device configured to perform inverse reinforcement learning based on expert data indicating a trajectory of an expert's actions under a plurality of environmental conditions to generate a first reward function common to the plurality of environmental conditions; a policy function generating device configured to execute reinforcement learning based on the first reward function generated by the reward function generating device and agent data indicating a trajectory of an agent's behavior under the multiple environmental conditions or an environmental condition representative of the multiple environmental conditions, to generate a first policy function common to the multiple environmental conditions; Equipped with. Effect of the Invention

[0009] According to the present disclosure, there is provided an inverse reinforcement learning method and an inverse reinforcement learning device that can generate reward functions and policy functions applicable to a plurality of environmental conditions to realize efficient learning. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a schematic functional configuration of an inverse reinforcement learning system 2 according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an inverse reinforcement learning device 4. [Diagram 3] 1 is a flowchart showing an example of a learning flow in an inverse reinforcement learning system 2 (2A) according to an embodiment. [Figure 4] 2 is a schematic diagram showing a steam system 50 including a learning target of an inverse reinforcement learning device 4. FIG. [Diagram 5] 1 is a diagram showing the valve openings of valves V1 to V3, the temperature of a warm-up operation target portion 62, and the pressure of the warm-up operation target portion 62 in time series. [Figure 6] FIG. 13 is a diagram showing combinations of pressure P0 and temperature T0 of the steam in the steam header 52 for each of a plurality of environmental conditions j (j=1 to 4). [Figure 7A] 13 is a diagram showing the steam pressure of the warm-up operation target portion 62 in time series. FIG. [Figure 7B] 11 is a diagram showing the temperature of steam in a warm-up operation target section 62 in time series. FIG. [Figure 7C] 11 is a diagram showing the relationship between the temperature and pressure of steam in a warming operation target portion 62. FIG. [Figure 8A] FIG. 11 is a diagram showing the opening degree of the valve V3 over time. [Figure 8B] FIG. 4 is a diagram showing the opening degree of the valve V1 over time. [Figure 8C] FIG. 11 is a diagram showing the relationship between the opening degree of valve V3 and the opening degree of valve V1. [Figure 9] 13 is a flowchart showing another example of the learning flow in the inverse reinforcement learning system 2 (2B) according to an embodiment. [Figure 10] FIG. 13 is a diagram showing an example of a reward function R3 generated by a simple average of multiple reward functions R2. [Figure 11] FIG. 13 is a diagram showing a schematic diagram of the relationship between changes in environmental conditions and the optimality of the resulting policy function (Π1 or Π3) for the inverse reinforcement learning system 2 (2A) and the inverse reinforcement learning system 2 (2B). [Figure 12] 10 is a diagram showing the relationship between the absolute value of the steam temperature and the absolute value of the steam pressure in a warm-up operation target portion 62. FIG. [Figure 13] 13 is a diagram showing the relationship between the relative value of the steam temperature and the relative value of the steam pressure in a warm-up operation target portion 62. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, some embodiments of the present disclosure will be described with reference to the accompanying drawings. However, the dimensions, materials, shapes, relative arrangements, etc. of components described as the embodiments or shown in the drawings are merely illustrative examples and are not intended to limit the scope of the invention. For example, expressions expressing relative or absolute configuration, such as "in a certain direction," "along a certain direction," "parallel," "orthogonal," "center," "concentric," or "coaxial," not only strictly express such a configuration, but also express a state in which there is a relative displacement with a tolerance or an angle or distance to the extent that the same function is obtained. For example, expressions indicating that things are in an equal state, such as "identical," "equal," and "homogeneous," not only indicate a state of strict equality, but also indicate a state in which there is a tolerance or a difference to the extent that the same function is obtained. For example, expressions describing shapes such as a rectangular shape or a cylindrical shape do not only refer to rectangular shapes, cylindrical shapes, etc. in the strict geometric sense, but also refer to shapes that include uneven portions, chamfered portions, etc., to the extent that the same effect is obtained. On the other hand, the expressions "comprise," "include," "have," "includes," or "have" of one element are not exclusive expressions excluding the presence of other elements.

[0012] Fig. 1 is a diagram showing a schematic functional configuration of an inverse reinforcement learning system 2 according to an embodiment. Fig. 2 is a diagram showing an example of a hardware configuration of an inverse reinforcement learning device 4.

[0013] The inverse reinforcement learning system 2 may be used, for example, to learn operational control of a plant (not shown) and to assist in the operation of the plant, or may be used, for example, to automate plant operation or to assist operators with low skills.

[0014] As shown in Fig. 1, the inverse reinforcement learning system 2 includes an inverse reinforcement learning device 4 and an environment 5. The inverse reinforcement learning device 4 corresponds to an agent placed in the environment 5, and executes an action a with respect to the environment 5. The environment 5 receives the action a from the inverse reinforcement learning device 4 (agent), updates the state s of the environment, and provides a reward to the inverse reinforcement learning device 4.

[0015] The inverse reinforcement learning device 4 may be configured to perform Adversarial Inverse Reinforcement Learning (AIRL) using, for example, a Generative Adversarial Network (GAN). The environment 5 may be a real environment or a physical simulator that imitates a real environment.

[0016] In this specification, "action" means an action that the inverse reinforcement learning device 4 executes with respect to the environment 5, "state" means a state (observable) observed from the environment 5, "policy" means a rule for deciding an action, and "reward" means an evaluation value of whether an action is good or bad. Furthermore, "trajectory data" means time-series data indicating the trajectory of an action (trajectory of a combination of an action and a state) linked to a state (observable).

[0017] As shown in Fig. 1, an inverse reinforcement learning device 4 includes a reward function generator 6 (reward function learning unit), a policy function generator 8 (policy function learning unit), and a storage unit 10. Here, a schematic explanation of the configuration shown in Fig. 1 will be given, and a detailed explanation will be given later.

[0018] The reward function generating unit 6 generates a reward function R1 common to a plurality of environmental conditions by performing inverse reinforcement learning based on expert data indicating the trajectory of an expert's (skilled person's) action a. Here, the reward function generating unit 6 generates a reward function R1 common to a plurality of environmental conditions by performing inverse reinforcement learning based on the expert data stored in the storage unit 10 and the observation amount (state s) received from the environment 5. The expert data is trajectory data indicating the trajectory of the action a linked to the state s (the trajectory of the combination of the state s and the expert's action a), and the expert data for a plurality of environmental conditions is stored in the storage unit 10. The reward function generating unit 6 includes a neural network that receives the expert data and the observation amount as input and outputs the reward function R1.

[0019] The policy function generator 8 executes reinforcement learning based on the reward function R1 generated by the reward function generator 6 and the observation amount (state s) observed from the environment 5, and generates a policy function Π1 common to a plurality of environmental conditions. The policy function generator 8 includes a neural network that receives the reward function R1 and the observation amount observed from the environment 5 as input, and outputs the policy function Π1. The policy function generator 8 also determines an action a based on the observation amount observed from the environment 5 and the generated policy function Π1, and inputs the action a to the environment 5.

[0020] As shown in FIG. 2, the inverse reinforcement learning device 4 includes, for example, a processor 72, a RAM (Random Access Memory) 74, a ROM (Read Only Memory) 76, a HDD (Hard Disk Drive) 78, an input I / F 80, and an output I / F 82, and is configured using a computer that is connected to each other via a bus 84. The inverse reinforcement learning device 4 is configured by a computer executing a program that realizes each function of the inverse reinforcement learning device 4. The functions of each part in the inverse reinforcement learning device 4 described below are realized, for example, by loading a program held in a ROM 76 into the RAM 74 and executing it with the processor 72, and by reading and writing data in the RAM 74 and the ROM 76. When the environment 5 is a physical simulator, the physical simulator may also be configured using a computer that includes a processor, a RAM, a ROM, a HDD, an input I / F, and an output I / F, and is connected to each other via a bus.

[0021] Next, a specific example of the inverse reinforcement learning system 2 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of a learning flow in the inverse reinforcement learning system 2 (2A) according to an embodiment. Here, a case where the environment 5 is a physical simulator will be described as an example.

[0022] In S101, an environmental condition j, which is a condition (prerequisite) satisfied by the environment 5, is set or changed. j is a flag for the environmental condition, and here, j is an integer from 1 to jmax. Next, in S102, the state s of the environment 5 is initialized (reset). Then, in S103, trajectory data (agent data) of a combination of the action a and the state s of the environment 5 is calculated for one episode using the policy function Π1 (first policy function) generated by the policy function generation unit 8, and the trajectory data is temporarily stored in the storage unit 10 in S104.

[0023] In S105, it is determined whether sampling of the trajectory data is completed. In S105, for example, it may be determined that sampling of the trajectory data is completed when the number of episodes of the trajectory data sampled in each of a plurality of environmental conditions j (j=1 to jmax) reaches a specified number, or when the elapsed time of the calculation reaches a specified time, or it may be determined that sampling of the trajectory data is completed based on other criteria.

[0024] In S105, if it is determined that sampling of the trajectory data is not complete, the environmental condition j is updated in S106. In S106, if j≠jmax is satisfied, j is incremented by 1, and the process returns to S101. In S106, if j=jmax is satisfied, j is updated to 1, and the process returns to S101.

[0025] If it is determined in S105 that sampling of the trajectory data is complete, then in S107 it is determined whether the current number of learning steps N is equal to or greater than the maximum number of learning steps Nmax. If it is determined in S107 that the current number of learning steps N is equal to or greater than the maximum number of learning steps Nmax, then learning is completed. If it is determined in S107 that the current number of learning steps N is not equal to or greater than the maximum number of learning steps Nmax, then in S108 the trajectory data temporarily stored in S104 is read from the storage unit 10 to learn the reward function R1 and the policy function Π1.

[0026] In S108, the reward function generating unit 6 generates a reward function R1 (first reward function) by performing inverse reinforcement learning based on a comparison between the trajectory data stored in the storage unit 10 (the trajectory data temporarily stored in S104) and the expert data stored in the storage unit 10. Also, in S108, the policy function generating unit 8 generates a policy function Π1 by performing reinforcement learning based on the trajectory data stored in the storage unit 10 (the trajectory data temporarily stored in S104) and the reward function R1 generated by the reward function generating unit 6. Then, in S109, 1 is added to the current number of learning steps N, and the process returns to S106 to perform learning for the next number of learning steps. Note that the policy function Π1 generated in S108 for the current number of learning steps N is used when calculating trajectory data (agent data) for one episode of the combination of state s and action a in S103 for the next number of learning steps.

[0027] In the learning flow shown above, the environmental conditions under which inverse reinforcement learning is performed are switched in sequence at regular intervals (for each episode in the above example). For example, when there are four environmental conditions under which inverse reinforcement learning is performed (for example, when inverse reinforcement learning is performed for environmental conditions A, B, C, and D), trajectory data obtained by switching between environmental conditions A, B, C, D, A, B, C, and D is learned. In addition, the reward function R1 generated by the reward function generating unit 6 is a reward function common to the multiple environmental conditions under which inverse reinforcement learning is performed, and the policy function Π1 generated by the policy function generating unit 8 is a policy function common to the multiple environmental conditions.

[0028] Therefore, it is possible to generate a reward function R1 and a policy function Π1 that can be applied to a plurality of different environmental conditions, and it is not necessary to switch the reward function according to the environmental condition, and it is not necessary to switch the policy function according to the environmental condition. Therefore, it is possible to realize efficient learning without learning for each environmental condition. In addition, it is not necessary to identify the environment.

[0029] Next, an example in which the inverse reinforcement learning device 4 is applied to a warm-up operation of a steam pipe in a steam turbine will be described.

[0030] The steam system 50 includes a steam header 52, a steam turbine 54, a steam line 56, a bypass line 58, a branch line 60, a valve V1, a valve V2, a valve V3, and a turbine inlet gate valve V4.

[0031] A steam line 56 connects the steam header 52 and the steam turbine 54, and a valve V1 is provided in the steam line 56. A bypass line 58 branches off from the steam line 56 upstream of the valve V1 in the steam line 56, bypasses the valve V1, and joins the steam line 56 downstream of the valve V1 in the steam line 56. A valve V2 is provided in the bypass line 58. A branch line 60 branches off from the steam line 56 downstream of the position where the bypass line 58 joins the steam line 56, and a valve V3 is provided in the branch line 60. A turbine inlet gate valve V4 is provided between the position where the branch line 60 branches off from the steam line 56 and the steam turbine 54. The pressure of the steam in the steam header 52 is P0, and the temperature of the steam in the steam header 52 is T0.

[0032] In such a configuration, an example will be described in which the warm-up operation of the portion surrounded by the dashed line in Figure 4 (the portion between valve V1 in steam line 56 and turbine inlet gate valve V4; hereinafter referred to as warm-up operation target portion 62) is the learning target.

[0033] 5 shows the valve openings of the valves V1 to V3 in time series, and the lower part of FIG. 5 shows the temperature and pressure of the warm-up operation target portion 62 in time series.

[0034] In the example shown in Figure 5, the process includes a heating process in which the temperature of the warm-up operation target section 62 is raised from atmospheric temperature by gradually adjusting the opening of valves V1 to V3, a pressure increase process following the pressure increase process in which the pressure of the warm-up operation target section 62 is raised to the steam pressure P0 of the steam header 52, and a heating process following the pressure increase process in which the temperature of the warm-up operation target section 62 is raised to the steam temperature T0 of the steam header 52.

[0035] 7A to 7C and 8A to 8C show learning results by the inverse reinforcement learning device 4 in a case where the opening operation of the valves V1 and V3 as illustrated in Fig. 5 is the above-mentioned action a, the temperature and pressure of the steam in the warm-up operation target section 62 is the above-mentioned observation quantity (state s), and the combination of the steam pressure P0 and temperature T0 in the steam header 52 is the above-mentioned environmental condition j (see Fig. 6). Note that in the example shown in Fig. 6, the above-mentioned multiple environmental conditions j are composed of an environmental condition (j=1) indicating a combination of the allowable lower limit value of the pressure P0 and the allowable lower limit value of the temperature T0, an environmental condition (j=2) indicating a combination of the allowable upper limit value of the pressure P0 and the allowable lower limit value of the temperature T0, an environmental condition (j=3) indicating a combination of the allowable lower limit value of the pressure P0 and the allowable upper limit value of the temperature T0, and an environmental condition (j=4) indicating a combination of the allowable upper limit value of the pressure P0 and the allowable upper limit value of the temperature T0, as shown in Fig. 6.

[0036] Fig. 7A shows the steam pressure of the warm-air operation target section 62 in a time series, Fig. 7B shows the steam temperature of the warm-air operation target section 62 in a time series, and Fig. 7C shows the relationship between the steam temperature and pressure of the warm-air operation target section 62. Fig. 8A shows the opening degree of valve V3 in a time series, Fig. 8B shows the opening degree of valve V1 in a time series, and Fig. 8C shows the relationship between the opening degree of valve V3 and the opening degree of valve V1.

[0037] 7A to 7C and 8A to 8C, the thin lines indicate the expert history (the above-mentioned expert data), and the thick lines indicate the learning results by the inverse reinforcement learning device 4. As shown in each of Figs. 7A to 7C and 8A to 8C, it can be seen that the learning results by the inverse reinforcement learning device 4 (thick lines) are roughly similar to the expert history (thin lines).

[0038] Next, a specific example of the inverse reinforcement learning system 2 shown in FIGS. 1 and 2, other than the example shown in FIG. 3, will be described with reference to FIG. 9 is a flowchart showing another example of the learning flow in the inverse reinforcement learning system 2 (2B) according to an embodiment. Here, a case where the environment 5 is a physical simulator will be described as an example.

[0039] In S201, an environmental condition j, which is a condition (prerequisite) satisfied by the environment 5, is set or changed. j is a flag for the environmental condition, and here, j is an integer from 1 to jmax. Next, in S202, the state s of the environment 5 is initialized (reset). Then, in S203, using the policy function Π2 (policy function for each environmental condition j) generated by the policy function generation unit 8, trajectory data (time-series data) of a combination of state s and action a for environmental condition j is calculated for one episode, and in S204, the trajectory data is temporarily stored in the storage unit 10.

[0040] In S205, it is determined whether sampling of the trajectory data is completed. In S205, for example, it may be determined that sampling of the trajectory data is completed when the number of episodes of the trajectory data sampled under the environmental condition j reaches a specified number, or when the elapsed time of the calculation reaches a specified time, or it may be determined that sampling of the trajectory data is completed based on other criteria.

[0041] If it is determined in S205 that sampling of the trajectory data is not complete, the process returns to S202 while maintaining the environmental condition j, and S202 to S205 are repeated again.

[0042] If it is determined in S205 that sampling of the trajectory data is completed, it is determined in S206 whether the current number of learning steps N is equal to or greater than the maximum number of learning steps Nmax. If it is determined in S206 that the current number of learning steps N is not equal to or greater than the maximum number of learning steps Nmax, the trajectory data temporarily stored in S204 is read from the storage unit 10 and a reward function R2 and a policy function Π2 are learned in S207.

[0043] In S207, the reward function generating unit 6 generates a reward function R2 for each environmental condition j based on a comparison between the trajectory data stored in the storage unit 10 (the trajectory data for each environmental condition j temporarily stored in S204) and the expert data for each environmental condition j stored in the storage unit 10. Also, in S207, the policy function generating unit 8 generates a policy function Π2 for each environmental condition j based on the trajectory data stored in the storage unit 10 (the trajectory data for each environmental condition j temporarily stored in S204) and the reward function R2 (reward function R2 for each environmental condition j) generated by the reward function generating unit 6. Then, in S208, 1 is added to the current number of learning steps N, and the process returns to S202 to perform learning for the next number of learning steps. Note that the policy function Π2 (policy function Π2 for each environmental condition j) generated in S207 for the current number of learning steps N is used when calculating trajectory data (agent data) for one episode of the combination of state s and action a in S203 for the next number of learning steps.

[0044] If it is determined in S206 that the current number of learning steps N is equal to or greater than the maximum number of learning steps Nmax, then in S209 it is determined whether or not the environmental condition j is equal to or greater than jmax. If it is determined in S209 that the environmental condition j is not equal to or greater than jmax, then in S210 the environmental condition j is updated by adding 1 to it, and the process returns to S201. If it is determined in S209 that the environmental condition j is equal to or greater than jmax, then in S211 an environmental condition representative of the multiple environmental conditions j (j=1 to jmax) is set. Here, the environmental condition representative of the multiple environmental conditions j (j=1 to jmax) may be, for example, an average value, a median value, or a mode value of the multiple environmental conditions j (j=1 to jmax).

[0045] Next, in S212, the state s of the environment 5 is initialized (reset). Then, in S213, using the policy function Π3 (first policy function) generated by the policy function generation unit 8, trajectory data (agent data) of a combination of state s and action a for one episode regarding an environmental condition representative of multiple environmental conditions j (j=1 to jmax) is calculated, and in S214, the trajectory data is temporarily stored in the storage unit 10.

[0046] In S215, it is determined whether sampling of the trajectory data is completed. In S215, for example, it may be determined that sampling of the trajectory data is completed when the number of episodes of the trajectory data sampled under an environmental condition representing a plurality of environmental conditions j (j=1 to jmax) reaches a specified number, or when the elapsed time of the calculation reaches a specified time, or it may be determined that sampling of the trajectory data is completed based on other criteria.

[0047] If it is determined in S215 that the sampling of the trajectory data is not completed, the process returns to S212 while maintaining the environmental conditions, and S212 to S215 are repeated again. If it is determined in S215 that the sampling of the trajectory data is completed, it is determined in S216 whether the current number of learning steps N is equal to or greater than the maximum number of learning steps Nmax. If it is determined in S216 that the current number of learning steps N is not equal to or greater than the maximum number of learning steps Nmax, the trajectory data (agent data) temporarily stored in S214 is read from the storage unit 10 in S217 to learn a policy function Π3. In S17, a new reward function R3 (first reward function) is generated by combining the multiple reward functions R2 (reward functions R2 generated for each environmental condition for multiple environmental conditions j) generated in S207, and reinforcement learning based on the reward function R3 and the trajectory data (agent data) temporarily stored in S214 is performed to generate the policy function R3. In S217, the reward function R3 may be generated, for example, by linearly combining multiple reward functions R2. For example, as illustrated in FIG. 10, the reward function R3 may be generated by taking a simple average or a weighted average of multiple reward functions R2.

[0048] Then, in S218, 1 is added to the current learning step number N, and the process returns to S212 to perform learning for the next learning step number while maintaining the environmental condition j. Note that the policy function Π3 generated in S217 for the current learning step number N is used when calculating trajectory data (agent data) for the combination of state s and action a for one episode in S213 for the next learning step number.

[0049] If it is determined in S216 that the current number of learning steps N is equal to or greater than the maximum number of learning steps Nmax, the inverse reinforcement learning by the inverse reinforcement learning device 4 is completed.

[0050] 9, the reward function generator 6 generates multiple reward functions R2 by performing inverse reinforcement learning for each of multiple environmental conditions j (j=1 to jmax), and generates reward function R3 by combining the multiple reward functions R2. The policy function generator 8 performs reinforcement learning based on the reward function R3 generated by the reward function generator 6 and trajectory data (agent data) indicating the trajectory of the agent's action (trajectory of action a linked to state s) under environmental conditions representative of the multiple environmental conditions j (j=1 to jmax), to generate a policy function Π3 common to the multiple environmental conditions j (j=1 to jmax).

[0051] Therefore, it is possible to generate a reward function R3 and a policy function Π3 that can be applied to a plurality of different environmental conditions, and it is not necessary to switch the reward function according to the environmental conditions, and it is not necessary to switch the policy function according to the environmental conditions. Therefore, efficient learning can be realized.

[0052] In addition, by using the average, median, or mode of multiple environmental conditions j (j=1 to jmax) as the environmental condition that represents multiple cyclic conditions j, it is possible to generate a reward function R3 and a policy function Π3 that are suitable for multiple environmental conditions.

[0053] Here, the proper use of the inverse reinforcement learning system 2(2A) and the inverse reinforcement learning system 2(2B) will be explained with reference to Fig. 11. Fig. 11 is a diagram showing a schematic diagram of the relationship between the change in environmental conditions and the optimality of the resulting policy function (Π1 or Π3) for the inverse reinforcement learning system 2(2A) and the inverse reinforcement learning system 2(2B).

[0054] As shown in Fig. 11, the inverse reinforcement learning system 2 (2A) can obtain a policy function suitable for the environmental conditions stably over a wide range of environmental conditions. Also, the inverse reinforcement learning system 2 (2B) can obtain a policy function suitable for the environmental conditions more than the inverse reinforcement learning system 2 (2A) near the representative value of the environmental conditions set in S211. Also, in the inverse reinforcement learning system 2 (2B), the optimality gradually decreases as the environmental conditions move away from the representative value of the environmental conditions set in S211, so that the inverse reinforcement learning system 2 (2A) can obtain a policy function suitable for the environmental conditions more than the inverse reinforcement learning system 2 (2B) under environmental conditions that are somewhat away from the representative value.

[0055] For this reason, in cases where emphasis is placed on robustness even at the expense of lowering the priority of optimality for the resulting policy, the inverse reinforcement learning system 2 (2A) may be selected, whereas in cases where emphasis is placed on optimality near the representative value even at the expense of lowering the priority of robustness for the resulting policy, the inverse reinforcement learning system 2 (2B) may be selected.

[0056] The present disclosure is not limited to the above-described embodiments, and includes modifications to the above-described embodiments and appropriate combinations of these modifications.

[0057] For example, in some embodiments, in the above-mentioned inverse reinforcement learning systems 2 (2A) and 2 (2B), the observation quantity (state s) observed by the inverse reinforcement learning device 4 from the environment 5 may be a normalized relative value so as to be a common index among multiple environmental conditions j (j = 1 to jmax).

[0058] For example, in the example of the warm-up operation of the steam pipe in the steam turbine described with reference to Figures 4 to 8C, in the process of increasing the pressure of the warm-up operation target part 62, if the target pressure of the warm-up operation target part 62 (the end condition of the pressure of the warm-up operation target part 62, i.e., the steam pressure P0 of the steam header 52 in the above example) varies depending on the environmental condition j (j = 1 to jmax), the pressure during increase in pressure is not used as an absolute value of the observation amount as shown in Figure 12, but a normalized (relative) value obtained by dividing the operating pressure (pressure of the warm-up operation target part 62) by the target pressure as shown in Figure 13 may be used as the observation amount. This allows the observation amount to be treated as a common index for multiple environmental conditions j (j = 1 to jmax) when the operation content is the same even if the environments are different.

[0059] In this way, by using the target pressure (terminal condition of the observation amount) instead of the sensor range as the denominator used for normalizing the observation amount, the corrected observation amount can be learned to approach the terminal condition (i.e., 1) under any environmental condition, making learning under different environmental conditions (operating conditions) easier. In the example shown in Fig. 12, multiple environmental conditions j are learned as being significantly different when the absolute values ​​of the observation amount are used because the pressure levels are different, but by making them relative values ​​as shown in Fig. 13, multiple environmental conditions can be treated as being similar, making learning efficient.

[0060] In the example shown in FIG. 13, the operating pressure (pressure of the warm-up operation target section 62) is normalized (converted into a relative value) by dividing it by the target pressure and used as the observed quantity. However, in other embodiments, for example, the difference value obtained by subtracting the operating pressure (pressure of the warm-up operation target section 62) from the target pressure may be used as the observed quantity.

[0061] The contents described in each of the above embodiments can be understood, for example, as follows.

[0062] (1) An inverse reinforcement learning method according to at least one embodiment of the present disclosure, A reward function generation step of performing inverse reinforcement learning based on expert data showing the trajectory of the expert's actions to generate a first reward function (e.g., the above-mentioned reward function R1 or R3) common to a plurality of environmental conditions (e.g., the above-mentioned plurality of environmental conditions j (j=1 to jmax)); a policy function generating step of executing reinforcement learning based on the first reward function generated in the reward function generating step and agent data indicating a trajectory of an agent's behavior under the plurality of environmental conditions or an environmental condition representative of the plurality of environmental conditions, to generate a first policy function (e.g., the above-mentioned policy function Π1 or Π3) common to the plurality of environmental conditions; Equipped with.

[0063] According to the inverse reinforcement learning method described in (1) above, it is possible to generate a reward function and a policy function that are applicable to a plurality of environmental conditions, and it is not necessary to switch the reward function according to the environmental conditions, and it is not necessary to switch the policy function according to the environmental conditions. Therefore, it is possible to realize efficient learning.

[0064] (2) In some embodiments, in the inverse reinforcement learning method described in (1) above, The agent data is data indicating a trajectory of the agent's behavior under the multiple environmental conditions.

[0065] According to the inverse reinforcement learning method described in (2) above, agent data indicating the trajectory of an agent's actions under a plurality of environmental conditions can be efficiently learned without identifying the environmental conditions.

[0066] (3) In some embodiments, in the inverse reinforcement learning method described in (2) above, an agent data generating step of generating the agent data by switching the environmental conditions at a constant cycle; In the policy function generating step, reinforcement learning is performed based on the first reward function generated in the reward function generating step and the agent data generated in the agent data generating step to generate the first policy function common to the multiple environmental conditions.

[0067] According to the inverse reinforcement learning method described in (3) above, agent data showing the trajectory of an agent's actions under a plurality of environmental conditions can be efficiently learned without identifying the environmental conditions. Also, a policy function suitable for the environmental conditions can be obtained stably over a wide range of environmental conditions.

[0068] (4) In some embodiments, in the inverse reinforcement learning method described in (3) above, The certain period is a period of n times the episode (where n is a natural number).

[0069] According to the inverse reinforcement learning method described in (4) above, agent data is generated by switching environmental conditions every n-th episode, so that the first policy function common to a plurality of environmental conditions can be efficiently learned.

[0070] (5) In some embodiments, in the inverse reinforcement learning method described in (1) above, In the reward function generation step, a plurality of second reward functions (e.g., the above-mentioned reward function R2) are generated by performing inverse reinforcement learning for each of the plurality of environmental conditions, and the first reward function (e.g., the above-mentioned reward function R3) is generated by combining the plurality of second reward functions.

[0071] According to the inverse reinforcement learning method described in (5) above, it is possible to efficiently learn a first reward function that is applicable to a plurality of environmental conditions.

[0072] (6) In some embodiments, in the inverse reinforcement learning method described in (5) above, The agent data is data indicating a trajectory of an agent's behavior under environmental conditions representative of the plurality of environmental conditions.

[0073] According to the inverse reinforcement learning method described in (6) above, it is possible to obtain a policy function that is more suitable for environmental conditions near representative values ​​of the environmental conditions than the inverse reinforcement learning methods described in (2) to (4) above.

[0074] (7) In some embodiments, in the inverse reinforcement learning method described in (6) above, The environmental condition representative of the plurality of environmental conditions is the average value, median value, or mode value of the plurality of environmental conditions.

[0075] The inverse reinforcement learning method described in (7) above can be generally applied to a wide range of environmental conditions.

[0076] (8) An inverse reinforcement learning device according to at least one embodiment of the present disclosure (for example, the above-described inverse reinforcement learning device 4) A reward function generating device (e.g., the above-mentioned reward function generating unit 6) configured to perform inverse reinforcement learning based on expert data indicating the trajectory of the expert's actions to generate a first reward function (e.g., the above-mentioned reward function R1 or R3) common to a plurality of environmental conditions (e.g., the above-mentioned environmental condition j (j=1 to jmax)); a policy function generating device (e.g., the above-mentioned policy function generating unit 8) configured to execute reinforcement learning based on the first reward function generated by the reward function generating device and agent data indicating a trajectory of an agent's behavior under the multiple environmental conditions or an environmental condition representative of the multiple environmental conditions, and generate a first policy function (e.g., the above-mentioned policy function Π1 or Π3) common to the multiple environmental conditions; Equipped with.

[0077] According to the inverse reinforcement learning device described in (8) above, a common reward function and a policy function applicable to a plurality of environmental conditions can be generated, and there is no need to switch the reward function according to the environmental conditions, and there is no need to switch the policy function according to the environmental conditions, which makes it possible to realize efficient learning. [Explanation of symbols]

[0078] 2. Inverse Reinforcement Learning System 4. Inverse Reinforcement Learning Device 5 Environment 6 Reward function generation 8. Policy Function Generator 10 Storage section 50 Steam System 52 Steam Header 54 Steam Turbine 56 Steam Line 58 Bypass Line 60 Branch Line 62 Warm-up operation target part 72 processors 74 RAM 76 ROM 78 HDD 80 Input I / F 82 Output I / F 84 Bus

Claims

1. A reward function generation step of performing inverse reinforcement learning based on expert data indicating a trajectory of an expert's action to generate a first reward function common to a plurality of environmental conditions; a policy function generating step of executing reinforcement learning based on the first reward function generated in the reward function generating step and agent data indicating a trajectory of an agent's behavior under the plurality of environmental conditions or an environmental condition representative of the plurality of environmental conditions, to generate a first policy function common to the plurality of environmental conditions; The inverse reinforcement learning method includes:

2. The inverse reinforcement learning method according to claim 1 , wherein the agent data is data indicating a trajectory of the agent's behavior under the plurality of environmental conditions.

3. an agent data generating step of generating the agent data by switching the environmental conditions at a constant cycle; 3. The inverse reinforcement learning method according to claim 2, wherein in the policy function generating step, reinforcement learning is performed based on the first reward function generated in the reward function generating step and the agent data generated in the agent data generating step to generate the first policy function common to the plurality of environmental conditions.

4. The inverse reinforcement learning method according to claim 3 , wherein the certain period is a period n times the length of an episode (n is a natural number).

5. 2. The inverse reinforcement learning method according to claim 1, wherein in the reward function generating step, a plurality of second reward functions are generated by performing inverse reinforcement learning for each of the plurality of environmental conditions, and the first reward function is generated by combining the plurality of second reward functions.

6. The inverse reinforcement learning method according to claim 5 , wherein the agent data is data indicating a trajectory of an agent's behavior under environmental conditions representative of the plurality of environmental conditions.

7. The inverse reinforcement learning method according to claim 6 , wherein the environmental condition representative of the plurality of environmental conditions is an average value, a median value, or a mode value of the plurality of environmental conditions.

8. A reward function generating device configured to perform inverse reinforcement learning based on expert data indicating a trajectory of an expert's action to generate a first reward function common to a plurality of environmental conditions; a policy function generating device configured to execute reinforcement learning based on the first reward function generated by the reward function generating device and agent data indicating a trajectory of an agent's behavior under the plurality of environmental conditions or an environmental condition representative of the plurality of environmental conditions, to generate a first policy function common to the plurality of environmental conditions; An inverse reinforcement learning device comprising:

Images