Information processing device, information processing method, and computer program

Abstract

Reinforcement learning continues even when the only available action options for the agent are those that the agent cannot choose. [Solution] The information processing device comprises a memory for storing a learning program and a processor for executing the learning program, wherein the processor includes a first determination unit that determines whether the state S(p,i+1) of the environment model is a specific state based on the action A(p,i,k) selected in episode p, a duplication unit that duplicates the history H(p2) of episode p2 in which it was determined that the state S(p2,i+1) is not the specific state, and an overwrite unit that overwrites the history H(p1) of episode p1 in which it was determined that the state S(p1,i+1) is the specific state with a duplicate of the history H(p2) of episode p2.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a computer program.

Background Art

[0002] In reinforcement learning, an agent observes the state of the environment and selects an action to be executed from the action options based on the observed state. The environment is updated according to the selected action, and the agent selects the next action based on the state of the updated environment. In such a reinforcement learning process, there may only be action options that the agent cannot select, and the agent may fall into a state where it cannot solve the problem.

[0003] Patent Document 1 discloses a reinforcement learning method in which when a state (attention state) in which the environment is reset occurs during learning of environmental data, the features of the first environmental data in the first state in which the attention state occurs are compared with the features of the second environmental data in the second state traced back from the first state, and the difference between the respective features is learned by a learner.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the reinforcement learning method disclosed in Patent Document 1, every time the state of resetting the environment is entered, reinforcement learning needs to be stopped and the above complex learning needs to be executed.

Means for Solving the Problems

[0006] An information processing device according to one aspect of the present disclosure is an information processing device comprising a memory for storing a learning program and a processor for executing the learning program, wherein the learning program includes a reinforcement learning program that rewards the agent for an action A(p,i,k) selected by the agent in step i of episode p, and transitions the state S(p,i) of the environment model to the state S(p,i+1) of the next step i+1, and the processor, based on the selected action A(p,i,k) from among the actions A(p,i,j) in step i of episode p, transitions the state S of the environment model in the next step i+1 The system includes: a first determination unit that determines whether (p,i+1) is in a specific state in which all actions A(p,i+1,j) are unavailable; a duplication unit that duplicates the history H(p2) of episode p2 when the first determination unit determines that the state S(p1,i+1) of the environment model in episode p1 is in the specific state and the state S(p2,i+1) of the environment model in episode p2 is not in the specific state; and an overwrite unit that overwrites the history H(p1) of episode p1, in which the state S(p1,i+1) of the environment model was determined to be in the specific state, with a copy of the history H(p2) of episode p2. however, i: A variable representing the number of steps in the state of the environmental model. p1, p2: Episode identifiers p: A variable representing an episode (p ∋ p1, p2) S(p,i): A set of variables representing the state of the environmental model at step i of episode p. j: Identifier of the action (1 ≤ j ≤ N) A(p,i,j): A set of variables representing the action of identifier j at step i of episode p. k: Identifier of the action currently selected by the agent (1 ≤ k ≤ N) A(p,i,k): A set of variables representing the action of identifier k, which is actually selected in step i of episode p.

[0007] This disclosure can be implemented not only as an information processing device having the characteristic configuration described above, an information processing method using the characteristic processing as a step, and a computer program for causing the information processing device to execute the characteristic processing, but also as an information processing system including the information processing device, or as part or all of the information processing device being implemented as a semiconductor integrated circuit. [Effects of the Invention]

[0008] According to this disclosure, reinforcement learning can continue even when, during the reinforcement learning process, the only available action options for the agent are those that it cannot choose. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of the configuration of a machine learning system according to an embodiment. [Figure 2] Figure 2 is a block diagram showing an example of the hardware configuration of a learning device according to this embodiment. [Figure 3] Figure 3 shows the relationship between the agent and the environment. [Figure 4] Figure 4 is a diagram illustrating the flow of one episode. [Figure 5] Figure 5 shows an example of a probability distribution of behavior. [Figure 6] Figure 6 shows an example of the configuration of an agent (learner) in a neural network. [Figure 7] Figure 7 is a functional block diagram showing an example of the functions of the learning device according to this embodiment. [Figure 8] Figure 8 is a diagram illustrating an example of a problem involving the transport and delivery of petroleum. [Figure 9] Figure 9 shows an example of initial environment information for initial environment setup. [Figure 10] Figure 10 shows an example of a set loading and unloading plan. [Figure 11]FIG. 11 is a diagram for explaining the action mask. [Figure 12] FIG. 12 is a diagram for explaining a dead end. [Figure 13] FIG. 13 is a diagram for explaining the determination of a dead end by the first determination unit. [Figure 14] FIG. 14 is a diagram for explaining an example of replication and overwrite of the episode history. [Figure 15] FIG. 15 is a diagram for explaining an example of the action mask when it is determined that the state of the environment has transitioned to a dead end. [Figure 16] FIG. 16 is a diagram for explaining an example of step backtracking. [Figure 17] FIG. 17 is a flowchart showing an example of reinforcement learning processing by the learning device according to the embodiment. [Figure 18] FIG. 18 is a diagram for explaining the first modification example of step backtracking. [Figure 19] FIG. 19 is a diagram for explaining the action mask and the second modification example of step backtracking when it is determined that the state of the environment has transitioned to a dead end. [Figure 20] FIG. 20 is a diagram for explaining the action mask and the third modification example of step backtracking when it is determined that the state of the environment has transitioned to a dead end. [Figure 21] FIG. 21 is a flowchart showing an example of inference processing by the inference device according to the fourth modification example. [Figure 22] FIG. 22 is a flowchart showing an example of machine learning processing by the learning device according to the fifth modification example.

Embodiments for Carrying Out the Invention

[0010] <Summary of Embodiments of the Present Disclosure> Hereinafter, the summary of the embodiments of the present disclosure will be listed and explained.

[0011] (1) The information processing device according to this embodiment is an information processing device comprising a memory for storing a learning program and a processor for executing the learning program, wherein the learning program includes a reinforcement learning program which rewards the agent for an action A(p,i,k) selected by the agent in step i of episode p, and transitions the state S(p,i) of the environment model to the state S(p,i+1) of the next step i+1, and the processor determines the state S(p,i,k) of the environment model in the next step i+1 based on the selected action A(p,i,k) from among the actions A(p,i,j) in step i of episode p The system includes: a first determination unit that determines whether p,i+1) is in a specific state in which all actions A(p,i+1,j) are unavailable; a duplication unit that duplicates the history H(p2) of episode p2 when the first determination unit determines that the state S(p1,i+1) of the environment model in episode p1 is in the specific state and that the state S(p2,i+1) of the environment model in episode p2 is not in the specific state; and an overwrite unit that overwrites the history H(p1) of episode p1, in which the state S(p1,i+1) of the environment model was determined to be in the specific state, with a copy of the history H(p2) of episode p2. however, i: A variable representing the number of steps in the state of the environmental model. p1, p2: Episode identifiers p: A variable representing an episode (p ∋ p1, p2) S(p,i): A set of variables representing the state of the environmental model at step i of episode p. j: Identifier of the action (1 ≤ j ≤ N) A(p,i,j): A set of variables representing the action of identifier j at step i of episode p. k: Identifier of the action currently selected by the agent (1 ≤ k ≤ N) A(p,i,k): A set of variables representing the action of identifier k, which is actually selected in step i of episode p.

[0012] As a result, the history H(p1) of episode p1 that leads to a specific state is replaced with the history H(p2) of episode p2 that does not lead to that specific state, allowing episode p1 to proceed after the replaced history H(p2). Therefore, even if the reinforcement learning process encounters a situation where only action options that the agent cannot choose exist, reinforcement learning can still proceed.

[0013] (2) In (1) above, the specific state may be a state in which all of the actions A(p,i+1,j) in step i+1 violate the constraints defined in the environment model. This allows reinforcement learning to proceed even when the system falls into a specific state due to constraints in an environment where constraints are imposed.

[0014] (3) In (1) or (2) above, the history H(p2) may include all actions A_p2_i={A(p2,n,k)|1≦n≦i} selected from step 1 to step i in episode p2. This makes it possible to reproduce the state of the environment in episode p2 S_p2_i={S(p2,n)|1≦n≦i} from action A_p2_i, and to proceed with episode p1 immediately after the history H(p2).

[0015] (4) In (3) above, the history H(p2) may include the state of the environment model S_p2_i={S(p2,n)|1≦n≦i} in all of step 1 to step i in episode p2. This allows episode p1 to proceed immediately after history H(p2) without having to recalculate the state of the environment in episode p2 S_p2_i={S(p2,n)|1≦n≦i}.

[0016] (5) In any one of (1) to (4) above, the processor may further include an exclusion unit that excludes the action A(p3,in,k) actually selected in step in from the agent's selection target when the first determination unit determines that the state S(p3,i+1) of the environment model in episode p3 is the specific state, and a rewind unit that returns step i of episode p3 to the previous step im when the first determination unit determines that the state S(p3,i+1) of the environment model in episode p3 is the specific state. however, p3: Episode identifier (p3∈p) n: A setting value of 0 or greater that represents the number of steps i to return from. m: A setting value representing the number of returns for step i (m>n) As a result, after going back to step im, the agent will not select action A(p3,in,k), which was the cause of falling into a specific state, again. Therefore, reinforcement learning can proceed without overwriting the episode history. Note that episode p3 refers to the episode in which the action is excluded by the exclusion unit and the step is returned by the regress unit, and does not mean that it is a different episode from episodes p1 and p2. In other words, episode p3 here may be an episode in which the history has been overwritten by the overwrite unit, an episode in which the history has been duplicated by the duplication unit, or an episode in which the history has never been overwritten or duplicated.

[0017] (6) In (5) above, the processor further includes a second determination unit that determines whether or not there exists an episode p in which the state S(p,i+1) of the environment model in step i+1 is not the specific state, and if the second determination unit determines that there exists an episode p in which the state S(p,i+1) of the environment model in step i+1 is not the specific state, the duplication unit duplicates the history H(p2) of the episode p2, and the overwrite unit overwrites the history H(p1) of the episode p1 with the duplication of the history H(p2) of the episode p2, and if the second determination unit determines that there does not exist an episode p in which the state S(p,i+1) of the environment model in step i+1 is not the specific state, the exclusion unit excludes the action A(p3,in,k) from the selection target of the agent, and the rewind unit may return step i of the episode p3 to the previous step im. This allows reinforcement learning to proceed by overwriting the episode history as long as there are episodes p that do not reach a specific state, and by excluding actions and reversing steps as long as there are no episodes p that do not reach a specific state.

[0018] (7) The information processing method according to this embodiment is an information processing method by an information processing device that rewards the agent for the action A(p,i,k) selected by the agent in step i of episode p, and transitions the state S(p,i) of the environment model to the state S(p,i+1) of the next step i+1, wherein in step i of episode p1, the state S(p1,i+1) of the environment model in the next step i+1 is determined by the selected action A(p1,i,k) from among the actions A(p1,i,j), and all actions A(p1,i+1,j) are not selected. The process includes the steps of determining whether or not a specific state is possible, duplicating the history H(p2) of episode p2 if the state S(p1,i+1) of the environment model in episode p1 is determined to be the specific state and the state S(p2,i+1) of the environment model in episode p2 is determined not to be the specific state, and overwriting the history H(p1) of episode p1 in which the state S(p1,i+1) of the environment model is determined to be the specific state with a copy of the history H(p2) of episode p2. As a result, the history H(p1) of episode p1 that leads to the specific state is replaced with the history H(p2) of episode p2 that does not lead to the specific state, so that episode p1 can continue after the replaced history H(p2). Therefore, even if the agent encounters a situation in the reinforcement learning process where only action options that it cannot select exist, reinforcement learning can still proceed.

[0019] (8) The computer program according to this embodiment is a computer program that rewards the agent for the action A(p,i,k) selected by the agent in step i of episode p, and causes the state S(p,i) of the environment model to transition to the state S(p,i+1) of the next step i+1, wherein the computer causes the state S(p1,i+1) of the environment model in the next step i+1 to be due to the selected action A(p1,i,k) from among the actions A(p1,i,j) in step i of episode p1, and all of the actions A(p1,i+1,j) The process involves the steps of: determining whether or not a specific state is unavailable; if the state S(p1,i+1) of the environment model in episode p1 is determined to be the specific state, and the state S(p2,i+1) of the environment model in episode p2 is determined not to be the specific state, then duplicating the history H(p2) of episode p2; and overwriting the history H(p1) of episode p1, in which the state S(p1,i+1) of the environment model is determined to be the specific state, with the duplicated history H(p2) of episode p2. As a result, the history H(p1) of episode p1 that leads to the specific state is replaced with the history H(p2) of episode p2 that does not lead to the specific state, allowing episode p1 to continue after the replaced history H(p2). Therefore, even if the agent encounters a situation in the reinforcement learning process where only action options that it cannot select exist, reinforcement learning can still proceed.

[0020] <Details of the embodiments of this disclosure> The embodiments of this disclosure will be described in detail below with reference to the drawings. At least some of the embodiments described below may be combined in any way.

[0021] [1. Machine Learning Systems] Figure 1 shows an example of the configuration of a machine learning system according to an embodiment. The machine learning system 1 according to the embodiment includes a learning device 10 and a terminal device 20. The learning device 10 is an example of an "information processing device".

[0022] The learning device 10 and the terminal device 20 are connected, for example, by a communication line 30, and can communicate data with each other. Figure 1 shows an example where the learning device 10 and the terminal device 20 are connected one-to-one, but the learning device 10 may be connected to multiple terminal devices 20 via a network.

[0023] The learning device 10 performs reinforcement learning. The terminal device 20 includes, for example, an input device and a display device. The user can, for example, operate the terminal device 20 to instruct the start of learning. Furthermore, the user can input setting information for the learning model into the terminal device 20, or input input data for machine learning into the terminal device 20. The terminal device 20 can send the setting information or input data to the learning device 10 to set up the learning model or input the input data into the learning device 10. When the learning device 10 finishes machine learning, the learning device 10 sends the results of machine learning to the terminal device 20, and the terminal device 20 can display the learning results on the display device.

[0024] For example, the learning device 10 is comprised of a computer. In a specific example, the learning device 10 is a computer with large-scale and high-speed computing capabilities, such as a supercomputer. The learning device 10 may be a computer dedicated to machine learning, or it may be a general-purpose computer. The learning device 10 may be a server, and the terminal device 20 may be a client.

[0025] For example, the learning device 10 may have the functions of a terminal device 20. In this case, the machine learning system 1 is composed of one learning device 10.

[0026] [2. Hardware configuration of the learning device] Figure 2 is a block diagram showing an example of the hardware configuration of a learning device according to this embodiment.

[0027] The learning device 10 includes a processor 101, a non-volatile memory 102, a volatile memory 103, and an interface (hereinafter also referred to as "IF") 104. The processor 101, the non-volatile memory 102, the volatile memory 103, and the IF 104 are each connected to one another by a bus (data bus). The processor 101, the non-volatile memory 102, the volatile memory 103, and the IF 104 can each transmit data to one another via the bus.

[0028] The volatile memory 103 is a semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The non-volatile memory 102 is a rewritable storage device such as flash memory or a hard disk. The learning program 200 is stored in the non-volatile memory 102. The learning function of the learning device 10 is realized when the learning program 200 is executed by the processor 101. The learning program 200 is an example of a "computer program".

[0029] The processor 101 is, for example, a CPU (Central Processing Unit). However, the processor 101 is not limited to a CPU. The processor 101 may also be a GPU (Graphics Processing Unit). In a specific example, the processor 101 is a multi-core processor. The processor 101 may also be a single-core processor. The processor 101 may include multiple processors or cores and be capable of performing parallel processing. The processor 101 is configured to execute computer programs. The processor 101 may include, for example, an ASIC (Application Specific Integrated Circuit) as part, or programmable hardware such as an FPGA (Field Programmable Gate Array) or a CPLD (Complex Programmable Logic Device) as part.

[0030] The learning program 200 includes an agent 210 and an environment model 220. The agent 210 is composed of a learner (learning model). The environment model 220 provides the environment in which the agent 210 acts. The environment model 220 is, for example, a simulator that simulates a real environment. Both the agent 210 and the environment model 220 are computer programs. Hereinafter, the environment model will also be simply referred to as the "environment".

[0031] IF104 is a communication interface for communication with the terminal device 20. For example, IF104 is an Ethernet interface ("Ethernet" is a registered trademark).

[0032] [3. Reinforcement Learning] The following describes reinforcement learning that can be performed using learning program 200.

[0033] Figure 3 shows the relationship between the agent and the environment. In reinforcement learning, agent 210 and environment 220 interact according to a Markov decision process, and learning progresses. Agent 210 is given the state of environment 220, and in the given state, agent 210 acts according to the policy. In response to agent 210's actions, the state of environment 220 changes, and agent 210 is given the updated new state of environment 220 and a reward corresponding to the action. The goal of reinforcement learning is to learn a policy that maximizes the accumulated reward at the end of such interaction cycles (episodes).

[0034] Figure 4 is a diagram illustrating the flow of one episode. An episode consists of one or more steps. In the example in Figure 4, one episode 1 consists of three steps. Hereinafter, the state of environment 220 in step i of episode p will be denoted as S(p,i), and the action of agent 210 will be denoted as A(p,i,j) (j: action identifier (1≦j≦N)).

[0035] In the diagram, the white circle represents the state S of environment 220. In environment 220 in the initial state S(1,1), agent 210 can choose from three actions A(1,1,1), A(1,1,2), and A(1,1,3). For example, in step 1, agent 210 selects action A(1,1,2) from the three actions A(1,1,1), A(1,1,2), and A(1,1,3), and when the selected action A(1,1,2) is executed, the state of environment 220 transitions from state S(1,1) to state S(1,2).

[0036] Agent 210 observes the state S(1,2) of the environment 220. In the environment 220 in state S(1,2), Agent 210 can choose from three actions A(1,2,1), A(1,2,2), and A(1,2,3). For example, in step 2, when Agent 210 selects action A(1,2,2) from the three actions A(1,2,1), A(1,2,2), and A(1,2,3) and performs the selected action A(1,2,2), the state of the environment 220 transitions from state S(1,2) to state S(1,3).

[0037] Agent 210 observes the state S(1,3) of the environment 220. In the environment 220 in state S(1,3), Agent 210 can choose from three actions A(1,3,1), A(1,3,2), and A(1,3,3). For example, in step 3, Agent 210 selects action A(1,3,2) from the three actions A(1,3,1), A(1,3,2), and A(1,3,3), and when the selected action A(1,3,2) is executed, the state of the environment 220 transitions from state S(1,2) to the terminal state ST, which satisfies the termination conditions of episode 1.

[0038] Agent 210 calculates a probability distribution for selecting each action according to the policy at each step. Figure 5 shows an example of the action probability distribution. The probability distribution includes the selection probability for each action A1, A2, A3, A4, A5, ... The policy is a function that defines the relationship between the observed state of the environment 220 and the action probability distribution. Agent 210 is composed of a function approximator that learns the policy, and the function approximator includes one or more parameters.

[0039] For example, a function approximator is a neural network, and more specifically, a deep neural network (DNN).

[0040] Figure 6 shows an example of the configuration of an agent (learner) in a neural network. A neural network includes an input layer, a hidden layer, and an output layer.

[0041] The input layer contains multiple nodes (shown as circles in the diagram). Each node in the input layer receives parameters s1, s2, s3, ... of the state S of environment 220 as input.

[0042] The intermediate layer consists of one or more processing layers. In the example shown in Figure 4, the intermediate layer has a multiphase structure. The intermediate layer contains multiple nodes.

[0043] The output layer includes multiple nodes.

[0044] In the input, hidden, and output layers, nodes in adjacent layers are connected by edges (synaptic connections, shown as line segments in the diagram). Each edge is assigned a weight (connection weight). The policy is represented as a combination of these weights.

[0045] Each node in the output layer corresponds to an action A1, A2, A3, ... The selection probability of the corresponding action A1, A2, A3, ... is output from each node in the output layer.

[0046] [4. Functions of the learning device] Figure 7 is a functional block diagram showing an example of the functions of the learning device according to this embodiment.

[0047] The learning device 10 includes the following functions: setting unit 301, start unit 302, observation unit 303, probability generation unit 304, exclusion unit 305, selection unit 306, first determination unit 307, execution unit 308, environment update unit 309, termination determination unit 310, model update unit 311, second determination unit 312, duplication unit 313, overwrite unit 314, and retrospective unit 315. The setting unit 301, the start unit 302, the observation unit 303, the probability generation unit 304, the exclusion unit 305, the selection unit 306, the first determination unit 307, the execution unit 308, the environment update unit 309, the termination determination unit 310, the model update unit 311, the second determination unit 312, the duplication unit 313, the overwrite unit 314, and the rewind unit 315 are all implemented by the processor 101.

[0048] The configuration unit 301 initializes the agent 210 and the environment 220. For example, the configuration unit 301 initializes the parameters (weights) of the agent 210. The configuration unit 301 initializes the environment 220 according to the configuration information provided by the terminal device 20.

[0049] As an example, let's consider reinforcement learning that deals with the problem of loading and unloading petroleum (crude oil). Figure 8 is a diagram illustrating an example of the problem of loading and unloading petroleum. In the example in Figure 8, petroleum is loaded into and unloaded from five petroleum tanks A, B, C, D, and E. In the example in Figure 8, the petroleum occupancy rate in petroleum tank A is 30% (i.e., 30% of the maximum capacity of petroleum tank A is contained), the petroleum occupancy rate in petroleum tank B is 50%, the petroleum occupancy rate in petroleum tank C is 20%, the petroleum occupancy rate in petroleum tank D is 35%, and the petroleum occupancy rate in petroleum tank E is 60%.

[0050] Furthermore, in the example in Figure 8, the API specific gravity of oil in oil tank A is SGa, the API specific gravity of oil in oil tank B is SGb, the API specific gravity of oil in oil tank C is SGc, the API specific gravity of oil in oil tank D is SGd, and the API specific gravity of oil in oil tank E is SGe. Here, API specific gravity refers to the specific gravity of crude oil as defined by the American Petroleum Institute. API specific gravity is a value that can be obtained, for example, in accordance with ASTM D1289.

[0051] In this problem, the following operations are possible: selecting the receiving tank, determining the amount to be received, selecting the source tank, and determining the amount to be removed. Selecting the receiving tank is the operation of choosing the destination tank for the oil from oil tanks A, B, C, D, and E. Determining the amount to be received is the operation of determining the amount of oil to be received. Selecting the source tank is the operation of choosing the source tank for the oil from oil tanks A, B, C, D, and E. Determining the amount to be removed is the operation of determining the amount of oil to be removed.

[0052] For example, if oil tank A is selected as the receiving tank and the amount to be received is determined to be 5kL, then 5kL of oil will be delivered from the oil tanker (ship) to oil tank A. For example, if oil tank B is selected as the outgoing tank and the amount to be received is determined to be 10kL, then 10kL of oil will be delivered from oil tank B to the oil tanker (ship).

[0053] In this problem, it is also possible to perform the operation of transferring oil between oil tanks (hereinafter also referred to as "shifting"). For example, if oil tank C is selected as the receiving tank and the amount to be received is determined to be 7kL, and oil tank D is selected as the source tank and the amount to be delivered is determined to be 7kL, then 7kL of oil will be transferred from oil tank D to oil tank C.

[0054] For example, when oil is brought into an oil tank (let's call it oil tank E), the API specific gravity of the oil may change. For instance, if the API specific gravity of the oil stored in oil tank E is SGe, and 10kL of oil with an API specific gravity of SG0 is brought in from an oil tanker (ship), the API specific gravity of the oil stored in oil tank E will change from SGe to SGe1.

[0055] In this problem, environment 220 consists of the oil storage capacity (stock) of each oil tank A, B, C, D, and E, the amount of oil brought in from the ship, and the amount of oil shipped out to the ship. The actions that agent 210 can perform are selecting the oil tank to receive the oil from, determining the amount to be received, selecting the oil tank from which the oil is shipped out, and determining the amount to be shipped out.

[0056] Figure 9 shows an example of initial environment information for the initial setup of environment 220. Initial environment information is included in the setup information. The initial setup of environment 220 includes inventory lower limit, inventory upper limit, API lower limit, API upper limit, initial inventory, and initial API for each oil tank (referred to as "tank" in the figure) A, B, C, D, E. In the example in Figure 9, the initial setup of oil tank A has an inventory lower limit of "STL_A", an inventory upper limit of "STU_A", an API lower limit of "SGL_A", an API upper limit of "SGU_A", an inventory of "ST_A0", and an API specific gravity of "SG_A0". The initial setup of oil tank B has an inventory lower limit of "STL_B", an inventory upper limit of "STU_B", an API lower limit of "SGL_B", an API upper limit of "SGU_B", an inventory of "ST_B0", and an API specific gravity of "SG_B0". In the initial settings for oil tank C, the lower limit of inventory is "STL_C", the upper limit of inventory is "STU_C", the lower limit of API is "SGL_C", the upper limit of API is "SGU_C", the inventory is "ST_C0", and the API specific gravity is "SG_C0". In the initial settings for oil tank D, the lower limit of inventory is "STL_D", the upper limit of inventory is "STU_D", the lower limit of API is "SGL_D", the upper limit of API is "SGU_D", the inventory is "ST_D0", and the API specific gravity is "SG_D0". In the initial settings for oil tank E, the lower limit of inventory is "STL_E", the upper limit of inventory is "STU_E", the lower limit of API is "SGL_E", the upper limit of API is "SGU_E", the inventory is "ST_E0", and the API specific gravity is "SG_E0".

[0057] Returning to Figure 7, the setting unit 301 initializes the environment 220 according to the initial environment information described above.

[0058] The configuration unit 301 sets an import / export plan as the target for importing and exporting petroleum. The agent 210 selects an action according to the policy, with the set import / export plan as the target. The import / export plan is included, for example, in the configuration information.

[0059] Figure 10 shows an example of a set loading / unloading plan. The loading / unloading plan specifies, for example, the plan number, the type of loading / unloading, the date of loading or unloading, the ID of the vessel to be loaded or unloaded, the error limit for the API specific gravity of the oil to be unloaded, the API specific gravity of the oil to be loaded or unloaded, and the amount of oil to be loaded or unloaded. In the example in Figure 10, plan number "1" is of type "loading", the date is "November 21, 2024", the vessel ID is "EN01", the API specific gravity is "SG_I1", and the amount of oil is "Am1". Plan number "2" is of type "loading", the date is "November 23, 2024", the vessel ID is "GA02", the API specific gravity is "SG_I2", and the amount of oil is "Am2". In the incoming plans, plan numbers "1" and "2", the error limit for API specific gravity is not specified. Plan number "3" is of type "Outbound", the date is "November 25, 2024", the ship ID is "KA03", the error limit for API specific gravity is "Tol3_U-Tol3_L" (Tol3_U is the upper limit, Tol3_L is the lower limit), the API specific gravity is "SG_O3", and the amount of oil is "Am3". Plan number "4" is of type "Outbound", the date is "November 27, 2024", the ship ID is "SA04", the error limit for API specific gravity is "Tol4_U-Tol4_L" (Tol4_U is the upper limit, Tol4_L is the lower limit), the API specific gravity is "SG_O4", and the amount of oil is "Am4".

[0060] Returning to Figure 7, the setting unit 301 further sets constraints for agent 210. The constraints are defined, for example, as follows: (1) The amount of oil tank inventory does not exceed the upper limit. (2) The amount of oil tank inventory does not fall below the minimum limit. (3) The number of oil tanks used during loading and unloading must not exceed the maximum limit. (4) The amount of oil brought in does not fall below the minimum limit. (5) The amount of oil transported must not fall below the minimum limit. (6) The API specific gravity of petroleum in stock in oil tanks shall not exceed the upper limit. (7) The specific gravity of the API of petroleum in stock in the oil tanks must not fall below the lower limit.

[0061] For example, the upper and lower limits of inventory in constraints (1) and (2) are determined by the initial environmental information. The number of oil tanks used in constraint (3) is defined, for example, in the constraint information that defines the constraints. The lower limits of the amount of oil brought in and taken out in constraints (4) and (5) are defined in the constraint information. The upper and lower limits of the API specific gravity of oil in constraints (6) and (7) are determined by the initial environmental information of environment 220. The constraint information is included, for example, in the setting information.

[0062] The starter unit 302 initiates an episode. The starter unit 302 can initiate multiple episodes using multiple agents 210. For example, multiple agents 210 are configured with a common policy.

[0063] The observation unit 303 observes the state of the environment 220. The state S of the environment 220 includes multiple parameters s1, s2, s3, s4, ... For example, parameter s1 is the inventory amount in oil tank A, parameter s2 is the API specific gravity of the oil in oil tank A, parameter s3 is the inventory amount in oil tank B, and parameter s4 is the API specific gravity of the oil in oil tank B. The environment 220 outputs the state S, and the observation unit 303 observes the state S by receiving the state S.

[0064] The probability generation unit 304 generates probability distributions for actions A1, A2, A3, ... for each agent 210 according to the policy. For example, the state S=(s1,s2,s3,s4, ...) of the environment 220 is input to the DNN described above, and the probability distributions for actions A1, A2, A3, ... are output from the DNN. Specifically, the probability generation unit 304 is a function approximator (e.g., DNN) of the policy that constitutes agent 210.

[0065] The exclusion unit 305 excludes from the selection any actions A1, A2, A3, ... that violate the constraints. Hereafter, excluding an action from the selection will also be referred to as "masking an action."

[0066] Figure 11 is a diagram illustrating action masking. In the diagram, actions selectable by agent 210 are shown as black-filled circles, and masked actions are shown as dashed circles. In the state S(p1,i) of environment 220 at step i (where i is a natural number) of episode p1, if action A(p1,i,1) violates a constraint, then action A(p1,i,1) is masked. For example, in the state S(p1,i) of environment 220, if the inventory in oil tank A is 9kL less than the inventory limit, and action A(p1,i,1) is to bring 10kL of oil into oil tank A, then executing action A(p1,i,1) will cause the inventory in oil tank A to exceed the inventory limit. In this case, the exclusion unit 305 masks action A(p1,i,1).

[0067] Here, in step i, if the state S(p1,i) of environment 220 is such that all of the actions A(p1,i,1), A(p1,i,2), and A(p1,i,3) are unavailable, then state S(p1,i) is said to be a dead end. A dead end is an example of a "specific state".

[0068] Figure 12 illustrates a dead end. In step i of episode p1, in state S(p1,i) of environment 220, if all actions A(p1,i,1), A(p1,i,2), and A(p1,i,3) violate the constraints, then agent 210 cannot select any of actions A(p1,i,1), A(p1,i,2), and A(p1,i,3). In this case, state S(p1,i) of environment 220 is a dead end. In a dead end, agent 210 cannot select an action, and therefore episode p1 cannot proceed.

[0069] Returning to Figure 7, the selection unit 306 selects an action based on the probability distribution generated by the probability generation unit 304. Actions with a high probability are selected with a high probability, and actions with a low probability are selected with a low probability. However, the selection unit 306 cannot select masked actions.

[0070] The first determination unit 307 determines whether, as a result of the first action selected by agent 210 in the first step of an episode, the state of environment 220 in the second step, which is later than the first step, will transition to a dead end.

[0071] Figure 13 is a diagram illustrating the dead end determination by the first determination unit. In the figure, unselectable actions are shown by solid circles. In step i of episode p1, in state S(p1,i) of environment 220, actions A(p1,i,1), A(p1,i,2), and A(p1,i,3) are selectable. If agent 210 selects action A(p1,i,2), the state of environment 220 transitions to S(p1,i+1) in the next step i+1. The first determination unit 307 determines whether all of actions A(p1,i+1,1), A(p1,i+1,2), and A(p1,i+1,3) in step i+1 violate the constraints. That is, the first determination unit 307 determines whether the state S(p1,i+1) of environment 220 in step i+1 is a dead end. In the example in Figure 13, all actions A(p1,i+1,1), A(p1,i+1,2), and A(p1,i+1,3) violate the constraints. In this case, the first determination unit 307 determines that the state S(p1,i+1) of the environment 220 at step i+1 is a dead end.

[0072] The first determination unit 307 determines whether the state of environment 220 in a future step is a dead end. That is, the target of the first determination unit 307's determination is not the state S(p1,i) of environment 220 in the current step i (the step i to be executed), but the state S(p1,i+1) of environment 220 in the next step i+1. Therefore, the actions determined by the first determination unit 307 are not masked by the exclusion unit 305. In Figure 13, unselectable actions are shown as solid circles to illustrate the difference from masked actions (dashed circles).

[0073] Returning to Figure 7, if the first determination unit 307 determines that the state S of the environment 220 does not transition to a dead end, the execution unit 308 executes the action selected by the selection unit 306.

[0074] The environment update unit 309 updates the state S of the environment 220 as a result of the action being performed. That is, the environment update unit 309 estimates the changes in the environment 220 caused by the action being performed through simulation and determines the state of the environment 220 after the change. For example, if in step i of episode p1 the action is performed to deliver QkL of oil with an API specific gravity P to oil tank A, where the inventory is STa0% and the API specific gravity is SGa0, the environment update unit 309 determines the API specific gravity and inventory of oil tank A in step i+1 after the delivery and updates the state of the environment 220 from S(p1,i) to S(p1,i+1).

[0075] The termination determination unit 310 determines whether or not the termination condition of the episode has been met. The termination condition is, for example, that the number of steps has reached a predetermined value N.

[0076] If the termination determination unit 310 determines that the termination condition has not been met, the step number i is incremented, and the observation unit 303 observes the updated state of the environment 220.

[0077] If the termination determination unit 310 determines that the termination condition has been met, the model update unit 311 updates the agent 210. That is, the model update unit 311 updates the DNN weights as a policy.

[0078] For example, the model update unit 311 generates an oil loading and unloading plan as a result of the action for each of several episodes p. The model update unit 311 compares the oil loading and unloading plan generated from each episode p with the oil loading and unloading plan set as the target, and determines the oil loading and unloading plan that best approximates the target. The model update unit 311 determines, for example, the cumulative reward in the episode p in which the oil loading and unloading plan that best approximates the target was generated, and determines the parameters (weights) of agent 210 according to the cumulative reward. The model update unit 311 updates agent 210 by setting the determined parameters to agent 210.

[0079] The second determination unit 312 determines whether there exists an episode p in step i+1 where the state S(p,i+1) of the environment 220 is not a dead end. For example, the second determination unit 312 can use the determination result from the first determination unit 307 to determine whether there exists an episode p in step i+1 where the state S(p,i+1) of the environment 220 is not a dead end. If the first determination unit 307 determines that the state of the environment 220 is not a dead end in a certain episode p, it determines that there exists an episode p in which the state S(p,i+1) is not a dead end. If the first determination unit 307 determines that the state of the environment 220 is a dead end in all episode p, the second determination unit 312 determines that there is no episode p in step i+1 where the state S(p,i+1) of the environment 220 is not a dead end.

[0080] The replication unit 313 replicates the history H(p2) of episode p2 if the first determination unit 307 determines that the state S(p1,i+1) of the environment 220 in episode p1 is a dead end, and the state S(p2,i+1) of the environment 220 in episode p2 is not a dead end. More specifically, if the second determination unit 312 determines that there is an episode p in which the state S(p,i+1) of the environment 220 in step i+1 is not a dead end, the replication unit 313 replicates the history H(p2) of episode p2 which is not a dead end.

[0081] Figure 14 illustrates an example of episode history duplication and overwriting. In episode p1, all actions A(p1,i+1,1), A(p1,i+1,2), and A(p1,i+1,3) in environment state S(p1,i+1) are unselectable. That is, environment state S(p1,i+1) in episode p1 is a dead end. On the other hand, in episode p2, actions A(p2,i+1,1) and A(p2,i+1,2) in environment state S(p2,i+1) are selectable. That is, environment state S(p2,i+1) in episode p2 is not a dead end.

[0082] In the example in Figure 14, the duplication unit 313 duplicates the history H(p2) of episode p2. The history H(p2) includes at least all actions A_p2_i={A(p2,n,k)|1≦n≦i} selected from step 1 to step i in episode p2. That is, the history H(p2) includes the action A(p2,1,k) selected in step 1, the action A(p2,2,k) selected in step 2, ..., and the action A(p2,i,k) selected in step i.

[0083] In this embodiment, the history H(p2) further includes the states of environment 220 S_p_i={S(p,n)|1≦n≦i} in all steps from step 1 to step i in episode p2. That is, the history H(p2) includes the state of environment 220 S(p2,1) in step 1, the state of environment 220 S(p2,2) in step 2, ..., and the state of environment 220 (p2,i) in step i. As another example, the history H(p2) may omit the state of environment 220 S_p2_i.

[0084] For example, the history H(p) of each episode p is stored in the non-volatile memory 102. The replication unit 313 replicates the history H(p2) of episode p2 stored in the non-volatile memory 102. The history H(p) may also be stored in the volatile memory 103 instead of the non-volatile memory 102.

[0085] The replication by the replication unit 313 is performed by storing the same data as the history H(p2) to be replicated in the area of ​​the volatile memory 103.

[0086] The overwrite unit 314 overwrites the history H(p1) of episode p1, in which the state S(p1,i+1) of environment 220 is determined to be a dead end, with a copy of the history H(p2) of episode p2.

[0087] The overwrite unit 314 stores a copy of the history H(p2) of episode p2 in the storage area for the history H(p1) of episode p1 in the non-volatile memory 102. As a result, the history H(p1) is deleted, and both the original history H(p2) and the copy of history H(p2) are stored in the non-volatile memory 102. However, it is sufficient that the non-volatile memory 102 after the overwrite is substantially the same as the above state, and the storage location of the copy of the history H(p2) of episode p2 does not have to be the same area where the history H(p1) of episode p1 was stored. The overwrite unit 314 may also delete the history H(p1) of episode p1 from the non-volatile memory 102 and store a copy of the history H(p2) of episode p2 in the non-volatile memory 102.

[0088] Note that the actual data of history H(p1) of episode p1 may remain in the non-volatile memory 102 after overwriting. In normal memory management, the actual data of history H(p1) is not deleted from the non-volatile memory 102; instead, information indicating that history H(p1) has been deleted is recorded in the non-volatile memory 102, or the metadata of history H(p1) is deleted from the non-volatile memory 102. Such memory operations are also included in the deletion of history H(p1) of episode p1.

[0089] Furthermore, it is not necessary for the non-volatile memory 102, after overwriting, to store two actual data (identical data) of the history H(p2) of episode p2. For example, since the original history H(p2) is stored in the non-volatile memory 102, if information indicating a copy of history H(p2) is stored in the non-volatile memory 102, it is not necessary for the non-volatile memory 102 to store identical copy data to the original data. Such memory operations are also included in overwriting a copy of the history H(p2) of episode p2.

[0090] For example, if there are multiple episodes in which the first determination unit 307 determines that the state of the environment 220 at step i+1 is not a dead end, the replication unit 313 selects the episodes to be replicated from among the multiple episodes in which the state of the environment 220 at step i+1 is determined not to be a dead end. In one specific example, if the states S(p2,i+1) and S(p4,i+1) of the environment 220 at step i+1 of episodes p2 and p4 are both determined not to be dead ends, the replication unit 313 can randomly select the replication target from the histories H(p2) and H(p4) of episodes p2 and p4. In another example, the replication unit 313 can select the replication target from the histories H(p2) and H(p4) of episodes p2 and p4 according to certain rules.

[0091] For example, if there are multiple episodes in which the first determination unit 307 determines that the state of the environment 220 in step i+1 is a dead end, the overwrite unit 314 overwrites the history of all episodes in which the state of the environment 220 in step i+1 was determined to be a dead end with copies of the history of multiple episodes in which the state of the environment 220 in step i+1 was determined not to be a dead end. For example, if the states S(p1,i+1) and S(p5,i+1) of the environment 220 at step i+1 of episodes p1 and p5 are determined to be dead ends, and the states S(p2,i+1) and S(p4,i+1) of the environment 220 at step i+1 of episodes p2 and p4 are determined not to be dead ends, the overwrite unit 314 may overwrite the histories H(p1) and H(p5) of episodes p1 and p5, respectively, with a copy of the same history H(p2) of episode p2, or it may overwrite the history H(p1) of episode p1 with a copy of the history H(p2) of episode p2, and overwrite the history H(p5) of episode p5 with a copy of the history H(p4) of episode p4.

[0092] In the example shown in Figure 14, the observation unit 303 observes the state S(p1,i) of the environment 220 at step i for episode p1, in which history H(p1) is overwritten by a copy of history H(p2). Subsequently, at step i, the learning device 10 executes the process described above. As a result, the state S(p1,i) of the environment 220 that leads to a dead end is replaced with the state S(p2,i) of the environment 220 that does not lead to a dead end, allowing reinforcement learning (episode) to proceed.

[0093] Returning to Figure 7, the exclusion unit 305 excludes (masks) the action A(p3,in,k) that was actually selected in step in from the selection targets of agent 210 if the first determination unit 307 determines that the state S(p3,i+1) of the environment 220 in episode p3 is a dead end. More specifically, if the second determination unit 312 determines that there is no episode p in which the state S(p,i+1) of the environment 220 in step i+1 is not a dead end, the exclusion unit 305 excludes action A(p3,in,k) from the selection targets of agent 210. Note that episode p3 refers to the episode in which the exclusion unit excludes actions and the rewind unit returns to a previous step, and does not mean that it is a different episode from episodes p1 and p2.

[0094] The rewind unit 315 returns step i of episode p3 to the previous step im if the first determination unit determines that the state S(p3,i+1) of the environment 220 in episode p3 is a dead end, provided that m > n.

[0095] Figure 15 illustrates an example of action masking when the environmental state is determined to be a dead end. Figure 15 uses the example from Figure 13 to illustrate action masking, i.e., p3 = p1. In the example shown in Figure 15, if agent 210 selects action A(p1,i,2) in step i of episode p1, then in the next step i+1, the state S(p1,i+1) of the environment 220 is a dead end. The exclusion unit 305 masks action A(p1,i,2) selected in the currently executed step i of episode p1. In other words, the exclusion unit 305 masks action A(p1,i,2) selected in the step i immediately preceding step i+1, where the state S(p1,i+1) of the environment 220 is determined to be a dead end. That is, in the example of Figure 15, n = 1.

[0096] Figure 16 is a diagram illustrating an example of step reversal. In the example in Figure 16, the reversal unit 315 reverses from the current execution target step i to step i-1, which is one step prior. That is, the reversal unit 315 reverses from step i, which is masked for action A(p1,i,2), to step i-1, which is one step prior. In the example in Figure 16, m=2.

[0097] In the example shown in Figure 16, the observation unit 303 observes the state S(p1,i-1) of the environment 220 at step i-1, the step to which the episode p1 has been revisited. Subsequently, in step i-1, the learning device 10 executes the process described above. As a result, the action A(p1,i,2) that causes a dead end is masked, so action A(p1,i,2) is not selected again, and reinforcement learning (episode) can proceed.

[0098] [5. Operation of the learning device] Figure 17 is a flowchart showing an example of reinforcement learning processing by a learning device according to an embodiment.

[0099] For example, the user operates the terminal device 20 to input configuration information. The configuration information includes environment initial information that defines the initial state of the environment 220, constraint information that defines the constraint conditions, and target information. In learning about the oil loading and unloading problem, the target information is, for example, the oil loading and unloading plan described above. The input configuration information is transmitted from the terminal device 20 to the learning device 10.

[0100] The processor 101 receives the configuration information and initializes the learning device 10 according to the configuration information (step S101). That is, the processor 101 initializes the environment 220 according to the environment initial information and sets the constraints according to the constraint information. Furthermore, the processor 101 sets the goal according to the goal information.

[0101] The processor 101 sets the parameters (weights) of agent 210 and reads out agent 210 (step S102). The agent 210 read out may be a pre-trained model that has been pre-processed using machine learning (e.g., supervised learning), or it may be an untrained model.

[0102] Processor 101 sets the variable i to its initial value "1" and starts the episode (step S103). That is, the first step 1 of the episode is set as the target for execution. Specifically, processor 101 starts multiple episodes. The number of episodes is included in the configuration information. The following processes proceed in parallel across multiple episodes.

[0103] The processor 101 observes the state S(p,i) of the environment 220 for each episode p(p∋p1,p2,…) (step S104).

[0104] The processor 101 identifies actions that violate the constraints based on the observed state S(p,i) of the environment 220, and masks the identified actions (step S105).

[0105] Based on the observed state S(p,i) of the environment 220, the processor 101 calculates the selection probability for each of the multiple actions A(p,i,j) for each episode p and generates a probability distribution. Based on the generated probability distribution, the processor 101 selects action A(p,i,k) for each episode p (step S106).

[0106] The processor 101 determines for each episode p whether the state Si+1 of the environment 220 in the next step i+1 transitions to a dead end as a result of the execution of the selected action A(p,i,k) (step S107). That is, for each episode p, the processor 101 determines whether all of the actions A(i+1,j) in step i+1 violate the constraint conditions.

[0107] If the state S(p,i+1) of environment 220 in episode p does not transition to a dead end (NO in step S107), the processor 101 executes the selected action A(p,i,k) and updates the state S(p,i) of environment 220 to S(p,i+1) (step S108). In other words, the processor 101 determines the next state S(p,i+1) of environment 220 by simulation.

[0108] The processor 101 determines whether the termination condition (i=N) for episode p has been met (step S109). If the termination condition for episode p has not been met (NO in step S109), the processor 101 increments the variable i (step S110) and returns to step S104.

[0109] If the state S(p,i+1) of environment 220 in episode p transitions to a dead end (YES in step S107), the processor 101 determines whether there is an episode p in which the state S(p,i+1) of environment 220 does not transition to a dead end (step S111).

[0110] If there is an episode p in which the state S(p,i+1) of environment 220 does not transition to a dead end (YES in step S111), the processor 101 replicates the history H(p2) of episode p2 in which the state S(p1,i+1) of environment 220 does not transition to a dead end (step S112).

[0111] Processor 101 overwrites the history H(p2) of episode p1, in which the state S(p1,i+1) of environment 220 transitions to a dead end, with the history H(p2) of the duplicated episode p2 (step S113). As a result, the state S(p1,i+1) of environment 220 in episode p1 no longer transitions to a dead end. Processor 101 then returns to step S104.

[0112] If there is no episode p in which the state S(p,i+1) of environment 220 does not transition to a dead end (NO in step S111), the processor 101 masks the action A(p,i,k) selected in step i for which the state S(p,i+1) of environment 220 transitions to a dead end (step S114).

[0113] Furthermore, processor 101 decrements the variable i (step S115) and returns to step S104. This causes the execution of the step to be reversed to the previous step.

[0114] If the termination condition for episode p is met (YES in step S109), the processor 101 updates agent 210 (step S116). That is, the processor 101 determines the cumulative reward in episode p in which the oil loading and unloading plan that most closely approximates the target was generated, and determines the parameters of agent 210 according to the cumulative reward. The processor 101 updates agent 210 by setting the determined parameters to agent 210.

[0115] The processor 101 determines whether the termination condition for reinforcement learning has been met (step S117). For example, the termination condition for reinforcement learning is that the number of episode executions has reached a predetermined value. In another example, the termination condition for reinforcement learning may be that the performance of agent 210 has reached a predetermined value. Here, the performance of agent 210 may be the degree of agreement between the outputted oil loading and unloading plan and the target.

[0116] If the termination condition for reinforcement learning is not met (NO in step S117), the processor 101 returns the environment 220 to its initial state and returns to step S103. If the termination condition for reinforcement learning is met (YES in step S117), the reinforcement learning process ends.

[0117] [6-1. First variation] In the embodiment described above, when it was determined that the state S(p1,i+1) of the environment 220 in episode p1 transitioned to a dead end, the execution was reversed from the current execution target step i to step i-1, which was one step prior (see Figure 16), but it is not limited to this.

[0118] Figure 18 is a diagram illustrating a first modified example of step reversal. In the modified example shown in Figure 18, the reversal unit 315 goes back two steps past the current execution target step i to step i-2. That is, the reversal unit 315 goes back two steps past the step i in which action A(p1,i,2) was masked to step i-2. Furthermore, the number of steps that the reversal unit 315 goes back to may be three or more. In this way, even if the reversal goes back two or more steps past the step of the masked action A(p1,i,2), the masked action A(p1,i,2) will not be selected again, and the state S of the environment 220 will not reach a dead end via the same path as last time.

[0119] [6-2. Second variation] In the embodiment described above, when it was determined that the state S(p1,i+1) of the environment 220 in episode p1 transitioned to a dead end, the action A(p1,i,2) selected in the currently executed step i was masked (see Figure 15), but the embodiment is not limited to this.

[0120] Figure 19 illustrates a second modified example of action masking and step reversal when it is determined that the environmental state is transitioning to a dead end. In the modified example shown in Figure 19, the exclusion unit 305 masks action A(p1,i-1,2) selected in step i-1, one step prior to the current execution step i, for episode p1, rather than action A(p1,i,2) selected in step i-1, which is the current execution step. In other words, the exclusion unit 305 masks action A(p1,i-1,2) selected in step i-1, two steps prior to step i+1, where it is determined that the state S(p1,i+1) of the environment 220 is a dead end. Furthermore, the action that the exclusion unit 305 masks may be an action selected in a step two or more steps prior to the current execution step i.

[0121] Furthermore, in this case, the retracing unit 315 retraces to a step at least one step prior to the step of the masked action. In the modified example shown in Figure 19, the retracing unit 315 retraces to step i-2, one step prior to step i-1, where action A(p1,i-1,2) was masked. However, the retracing unit 315 may also retrace to a step at least two steps prior to step i-1, where action A(p1,i-1,2) was masked. As a result, since one of the actions in the path leading to the dead end is masked, the masked action is not selected again after the steps have been retraced, and the state S of the environment 220 does not reach the dead end via the same path as before.

[0122] Furthermore, it is possible to mask multiple previously selected actions together. For example, in the example shown in Figure 19, the exclusion unit 305 may mask actions A(p1,i,2) and actions A(p1,i-1,2). This allows multiple actions that cause a dead end to be masked together.

[0123] [6-3. Third Variation] For example, the masking of the action and step reversal when the state S of environment 220 transitions to a dead end may be repeated multiple times. Figure 20 is a diagram illustrating a third modified example of the masking of the action and step reversal when it is determined that the state of the environment transitions to a dead end. In the modified example shown in Figure 20, as a result of the selection unit 306 selecting action A(p1,i,2) in step i of episode p1, the state S(p1,i+1) of environment 220 in the next step i+1 becomes a dead end. The exclusion unit 305 masks the action A(p1,i,2) selected in step i, and the reversal unit 315 reverses to step i-1, one step prior to step i.

[0124] In step i-1, the selection unit 306 selects action A(p1,i-1,2) again, which it had previously selected. If all of actions A(p1,i,1), A(p1,i,2), and A(p1,i,3) in step i are masked, then the state S(p1,i) of environment 220 in episode p1 is a dead end, the exclusion unit 305 masks action A(p1,i-1,2) selected in step i-1, and the regress unit 315 goes back to step i-2, which is one step prior to step i-1.

[0125] In step i-2, the selection unit 306 selects action A(p1,i-2,2) again, which it had previously selected. If all of actions A(p1,i-1,1), A(p1,i-1,2), and A(p1,i-1,3) in step i-1 are masked, then the state S(p1,i-1) of environment 220 in episode p1 is a dead end, the exclusion unit 305 masks action A(p1,i-2,2) selected in step i-2, and the regress unit 315 goes back to step i-3, one step prior to step i-2. Repeating the above, the regress unit 315 goes back to step q.

[0126] This allows us to identify the action A(p1,q,2) that causes state S of environment 220 to lead to a dead end, and to mask action A(p1,p,2). Therefore, we can search for a path that avoids the dead end.

[0127] As another variation, if the ratio of masked actions to the total number of actions in the steps being traced back exceeds a threshold, the exclusion unit 305 may mask all actions in that step, even if unmasked actions remain, and the trace-back unit 315 may trace back further steps. This reduces the computational complexity in the process of tracing back multiple steps and allows for the efficient search of a path that avoids dead ends.

[0128] [6-4. Fourth variation] In the embodiments described above, the reinforcement learning process of the learning device 10 in the learning phase in which the agent 210 is trained on machine learning has been explained, but the invention is not limited thereto. In the inference process in the inference phase in which the reinforcement-trained agent 210 is trained on inference, it may be determined for each episode whether or not the state S of the environment 220 transitions to a dead end, and the history of episodes in which the state S of the environment 220 transitions to a dead end may be overwritten by a copy of the history of episodes in which the state S of the environment 220 does not transition to a dead end. A fourth modified example of the embodiment will be described in detail below.

[0129] In the fourth modified example, the learning device 10 functions as an inference device. However, the agent 210, which has been trained by reinforcement learning using the learning device 10, may be installed in an inference device different from the learning device 10, and the inference process may be performed by the inference device.

[0130] Referring to Figure 7, in the learning device 10 which functions as an inference device, the function of the model update unit 311 is disabled (not used). That is, since there is no need to adjust the parameters of the trained agent 210, the model update unit 311 is unnecessary. The other functions of the inference device are the same as those of the learning device 10.

[0131] Figure 21 is a flowchart showing an example of inference processing by the inference device according to the fourth modified example.

[0132] For example, the user operates the terminal device 20 to input configuration information. The configuration information includes initial environmental information that shows the actual inventory status of the oil tanks, and target information that is the actual oil loading and unloading plan. Furthermore, the configuration information includes the same constraints as those in the reinforcement learning process.

[0133] The processor 101 receives the configuration information and initializes the learning device 10 according to the configuration information (step S101). That is, the processor 101 initializes the environment 220 according to the environment initial information and sets the constraints according to the constraint information. Furthermore, the processor 101 sets the goal according to the goal information.

[0134] The processor 101 sets the parameters (weights) of agent 210 and reads out agent 210 (step S202). The agent 210 read out is a trained model obtained through reinforcement learning.

[0135] Processor 101 sets the variable i to its initial value "1" and starts the episode (step S103). That is, the first step 1 of the episode is set as the target for execution. Specifically, processor 101 starts multiple episodes. The number of episodes is included in the configuration information. The following processes proceed in parallel across multiple episodes.

[0136] The processor 101 observes the state S(p,i) of the environment 220 for each episode p(p∋p1,p2,…) (step S104).

[0137] The processor 101 identifies actions that violate the constraints based on the observed state S(p,i) of the environment 220, and masks the identified actions (step S105).

[0138] Based on the observed state S(p,i) of the environment 220, the processor 101 calculates the selection probability for each of the multiple actions A(p,i,j) for each episode p and generates a probability distribution. Based on the generated probability distribution, the processor 101 selects action A(p,i,k) for each episode p (step S106).

[0139] The processor 101 determines for each episode p whether the state Si+1 of the environment 220 in the next step i+1 transitions to a dead end as a result of the execution of the selected action A(p,i,k) (step S107). That is, for each episode p, the processor 101 determines whether all of the actions A(i+1,j) in step i+1 violate the constraint conditions.

[0140] If the state S(p,i+1) of environment 220 in episode p does not transition to a dead end (NO in step S107), the processor 101 executes the selected action A(p,i,k) and updates the state S(p,i) of environment 220 to S(p,i+1) (step S108). In other words, the processor 101 determines the next state S(p,i+1) of environment 220 by simulation.

[0141] The processor 101 determines whether the termination condition (i=N) for episode p has been met (step S109). If the termination condition for episode p has not been met (NO in step S109), the processor 101 increments the variable i (step S110) and returns to step S104.

[0142] If the state S(p,i+1) of environment 220 in episode p transitions to a dead end (YES in step S107), the processor 101 determines whether there is an episode p in which the state S(p,i+1) of environment 220 does not transition to a dead end (step S111).

[0143] If there is an episode p in which the state S(p,i+1) of environment 220 transitions to a dead end (YES in step S111), the processor 101 replicates the history H(p2) of episode p2 in which the state S(p1,i+1) of environment 220 does not transition to a dead end (step S112).

[0144] Processor 101 overwrites the history H(p2) of episode p1, in which the state S(p1,i+1) of environment 220 transitions to a dead end, with the history H(p2) of the duplicated episode p2 (step S113). As a result, the state S(p1,i+1) of environment 220 in episode p1 no longer transitions to a dead end. Processor 101 then returns to step S104.

[0145] If there is no episode p in which the state S(p,i+1) of environment 220 transitions to a dead end (NO in step S111), the processor 101 masks the action A(p,i,k) selected in step i for the episode p in which the state S(p,i+1) of environment 220 transitions to a dead end (step S114).

[0146] Furthermore, processor 101 decrements the variable i (step S115) and returns to step S104. This causes the execution of the step to be reversed to the previous step.

[0147] If the termination condition for episode p is met (YES in step S109), the processor 101 determines whether the termination condition for the inference process has been met (step S216). For example, the termination condition for the inference process is that the number of times the episode has been executed has reached a predetermined value. In another example, the termination condition for the inference process is that the degree of agreement between the outputted oil transport plan and the target is greater than a predetermined value.

[0148] If the termination condition for the inference process is not met (NO in step S216), the processor 101 returns the environment 220 to its initial state and returns to step S103. If the termination condition for the inference process is met (YES in step S216), the processor 101 outputs the inference result, i.e., the oil loading and unloading plan with the highest degree of agreement with the target (step S217). This completes the inference process.

[0149] [6-5. Fifth variation] In the embodiment described above, the learning device 10 performed both the process of overwriting the history of episodes in which the state S of the environment 220 transitions to a dead end with a copy of the history of episodes in which the state S of the environment 220 does not transition to a dead end, and the process of masking the actions of episodes in which the state S of the environment 220 transitions to a dead end and reverting to a previous step, but is not limited to this. In the fifth modified embodiment, the process of masking the actions of episodes in which the state S of the environment 220 transitions to a dead end and reverting to a previous step is omitted. The fifth modified embodiment will be described in detail below.

[0150] Referring to Figure 7, in the learning device 10 according to the fifth modified example, the function of the retracing unit 315 is disabled (not used). Furthermore, the exclusion unit 305 retains the function of masking actions that violate the constraints, but the function of masking the currently selected action for episodes in which the state S of the environment 220 transitions to a dead end is disabled (not used). The other functions of the learning device 10 according to the fifth modified example are the same as those of the learning device 10 according to the embodiment described above.

[0151] Figure 22 is a flowchart showing an example of machine learning processing by a learning device according to the fifth modified example.

[0152] For example, the user operates the terminal device 20 to input configuration information into the terminal device 20.

[0153] The processor 101 receives the configuration information and initializes the learning device 10 according to the configuration information (step S101). That is, the processor 101 initializes the environment 220 according to the environment initial information and sets the constraints according to the constraint information. Furthermore, the processor 101 sets the goal according to the goal information.

[0154] The processor 101 sets the parameters (weights) of agent 210 and reads out agent 210 (step S102). The agent 210 read out may be a pre-trained model that has been pre-processed using machine learning (e.g., supervised learning), or it may be an untrained model.

[0155] Processor 101 sets the variable i to its initial value "1" and starts the episode (step S103). That is, the first step 1 of the episode is set as the target for execution. Specifically, processor 101 starts multiple episodes. The number of episodes is included in the configuration information. The following processes proceed in parallel across multiple episodes.

[0156] The processor 101 observes the state S(p,i) of the environment 220 for each episode p(p∋p1,p2,…) (step S104).

[0157] The processor 101 identifies actions that violate the constraints based on the observed state S(p,i) of the environment 220, and masks the identified actions (step S105).

[0158] Based on the observed state S(p,i) of the environment 220, the processor 101 calculates the selection probability for each of the multiple actions A(p,i,j) for each episode p and generates a probability distribution. Based on the generated probability distribution, the processor 101 selects action A(p,i,k) for each episode p (step S106).

[0159] The processor 101 determines for each episode p whether the state Si+1 of the environment 220 in the next step i+1 transitions to a dead end as a result of the execution of the selected action A(p,i,k) (step S107). That is, for each episode p, the processor 101 determines whether all of the actions A(i+1,j) in step i+1 violate the constraint conditions.

[0160] If the state S(p,i+1) of environment 220 in episode p does not transition to a dead end (NO in step S107), the processor 101 executes the selected action A(p,i,k) and updates the state S(p,i) of environment 220 to S(p,i+1) (step S108). In other words, the processor 101 determines the next state S(p,i+1) of environment 220 by simulation.

[0161] The processor 101 determines whether the termination condition (i=N) for episode p has been met (step S109). If the termination condition for episode p has not been met (NO in step S109), the processor 101 increments the variable i (step S110) and returns to step S104.

[0162] If the state S(p,i+1) of environment 220 in episode p transitions to a dead end (YES in step S107), the processor 101 determines whether there is an episode p in which the state S(p,i+1) of environment 220 does not transition to a dead end (step S111).

[0163] If there is an episode p in which the state S(p,i+1) of environment 220 does not transition to a dead end (YES in step S111), the processor 101 replicates the history H(p2) of episode p2 in which the state S(p1,i+1) of environment 220 does not transition to a dead end (step S112).

[0164] Processor 101 overwrites the history H(p2) of episode p1, in which the state S(p1,i+1) of environment 220 transitions to a dead end, with the history H(p2) of the duplicated episode p2 (step S113). As a result, the state S(p1,i+1) of environment 220 in episode p1 no longer transitions to a dead end. Processor 101 then returns to step S104.

[0165] If there is no episode p in which the state S(p,i+1) of environment 220 does not transition to a dead end (NO in step S111), the processor 101 proceeds to step S108.

[0166] If the termination condition for episode p is met (YES in step S109), the processor 101 updates agent 210 (step S116). That is, the processor 101 determines the cumulative reward in episode p in which the oil loading and unloading plan that most closely approximates the target was generated, and determines the parameters of agent 210 according to the cumulative reward. The processor 101 updates agent 210 by setting the determined parameters to agent 210.

[0167] The processor 101 determines whether the termination condition for reinforcement learning has been met (step S117). For example, the termination condition for reinforcement learning is that the number of episode executions has reached a predetermined value, or that the state S(p,i,k) of the environment 220 is a dead end in all episodes p. In another example, the termination condition for reinforcement learning may be that the performance of agent 210 has reached a predetermined value. Here, the performance of agent 210 may be the degree of agreement between the outputted oil loading and unloading plan and the target.

[0168] If the termination condition for reinforcement learning is not met (NO in step S117), the processor 101 returns the environment 220 to its initial state and returns to step S103. If the termination condition for reinforcement learning is met (YES in step S117), the reinforcement learning process ends.

[0169] [6-6. Other variations] The above-described embodiment illustrates a learning device 10 that deals with the problem of loading and unloading petroleum, but this is illustrative and not limiting. For example, the learning device 10 can deal with problems similar to the problem of loading and unloading petroleum. A similar problem is, for example, the problem of receiving and shipping goods in a warehouse. In this problem, multiple petroleum tanks are replaced with multiple warehouses or multiple areas in one warehouse, and petroleum is replaced with cargo or items to be stored. Furthermore, the learning device 10 may deal with problems different from the problem of loading and unloading petroleum and the problem of receiving and shipping goods in a warehouse.

[0170] [7. Addendum] (Note 1) For each of multiple episodes, a first determination unit determines whether, as a result of the learning agent performing the first action selected in the first step, the state of the environment in which the learning agent acts in the second step, which is later than the first step, transitions to a specific state in which it is impossible for the learning agent to select one or more second actions. If the first determination unit determines that the first state of the environment in the first episode transitions to the specific state, and determines that the second state of the environment in the second episode, which is different from the first episode, does not transition to the specific state, the replication unit replicates the second history, which is the history of the second episode. An overwrite unit that overwrites the first history, which is the history of the first episode in which the first state of the environmental model was determined to be the specific state, by copying the second history of the second episode, Equipped with, Information processing device.

[0171] (Note 2) The aforementioned specific state is a state in which all of one or more second actions violate the constraints defined in the environment. The information processing device described in Appendix 1.

[0172] (Note 3) The aforementioned history includes all actions selected in the aforementioned episode. The information processing device described in Appendix 1 or Appendix 2.

[0173] (Note 4) The aforementioned history includes all states of the environment model from the first step to the current step in the aforementioned episode. An information processing device as described in any one of the appendices 1 through 3.

[0174] (Note 5) When it is determined that the state of the environment in the third episode transitions to the specific state, an exclusion unit excludes the specific action, which is at least one of the actions selected by the learning agent in the third episode, from the learning agent's selection targets. If it is determined that the state of the environment in the third episode transitions to the specific state, the system includes a retracing unit that retraces the steps to be executed by the learning agent in the third episode to a fourth step prior to the third step in which the specific action was selected, Furthermore, An information processing device as described in any one of the appendices 1 through 4.

[0175] (Note 6) A second determination unit determines whether or not there are episodes in which the state of the environmental model in the second step is not the specific state, Furthermore, If the second determination unit determines that there is an episode in which the state of the environmental model in the second step is not the specific state, The duplication unit duplicates the second history of the second episode, The overwrite unit overwrites the first history of the first episode by copying the second history of the second episode. If the second determination unit determines that there are no episodes in which the state of the environment model in the second step is not the specific state, The exclusion unit excludes the third action from the selection target of the agent, The aforementioned retracing unit retraces the steps to be executed in the third episode, from the third step to the fourth step. The information processing device described in Appendix 5.

[0176] (Note 7) For each of the multiple episodes, the step of determining whether, as a result of the learning agent performing the first action selected in the first step, the state of the environment in which the learning agent acts in the second step, which is later than the first step, transitions to a specific state in which it is impossible for the learning agent to select one or more second actions; If it is determined that the first state of the environment in the first episode transitions to the specific state, and it is determined that the second state of the environment in a second episode, which is different from the first episode, does not transition to the specific state, the steps include: replicating the second history, which is the history of the second episode; The steps include overwriting the first history, which is the history of the first episode in which the first state of the environmental model was determined to be the specific state, by copying the second history of the second episode, including, Information processing methods.

[0177] (Note 8) On the computer, For each of the multiple episodes, the step of determining whether, as a result of the learning agent performing the first action selected in the first step, the state of the environment in which the learning agent acts in the second step, which is later than the first step, transitions to a specific state in which it is impossible for the learning agent to select one or more second actions; If it is determined that the first state of the environment in the first episode transitions to the specific state, and it is determined that the second state of the environment in a second episode, which is different from the first episode, does not transition to the specific state, the steps include: replicating the second history, which is the history of the second episode; The steps include overwriting the first history, which is the history of the first episode in which the first state of the environmental model was determined to be the specific state, by copying the second history of the second episode, To execute Computer program.

[0178] [8. Supplementary Notes] The embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of rights in this disclosure is defined by the claims rather than by the embodiments described above, and includes the meaning of equivalents to the claims and all modifications within that scope. [Explanation of symbols]

[0179] 1. Machine Learning Systems 10. Learning device (information processing device) 20 Terminal devices 30 communication lines 101 Processors 102 Non-volatile memory 103 Volatile memory 104 Interface (IF) 200 Learning Programs 210 Agents (Learning Agents) 220 Environmental Models (Environment) 301 Settings Section 302 Start part 303 Observation Unit 304 Probability Generation Unit 305 Exclusion part 306 Selection Section 307 1st Judgment Department 308 Execution Department 309 Environment Update Department 310 Termination Determination Unit 311 Model Update Section 312 Second Judgment Department 313 Reproduction Department 314 Overwrite section 315 Upstream section

Claims

1. Memory for storing the learning program, An information processing apparatus comprising a processor that executes the aforementioned learning program, The aforementioned learning program includes: The program includes a reinforcement learning program that rewards the agent for the action A(p,i,k) selected by the agent in step i of episode p, and transitions the state S(p,i) of the environment model to the state S(p,i+1) of the next step i+1. The aforementioned processor, A first determination unit determines, based on the selected action A(p,i,k) in step i of episode p, whether the state S(p,i+1) of the environment model in the next step i+1 is a specific state in which all actions A(p,i+1,j) are unavailable. If the first determination unit determines that the state S(p1, i+1) of the environment model in episode p1 is the specified state, and determines that the state S(p2, i+1) of the environment model in episode p2 is not the specified state, the replication unit replicates the history H(p2) of episode p2, An overwrite unit overwrites the history H(p1) of episode p1, which is determined to be the state S(p1, i+1) of the environmental model, by copying the history H(p2) of episode p2. including, Information processing device. however, i: A variable representing the number of steps in the state of the environment model. p1, p2: Episode identifiers p: A variable representing an episode (p ∋ p1, p2) S(p,i): A set of variables representing the state of the environmental model at step i of episode p. j: Identifier of the action (1 ≤ j ≤ N) A(p, i, j): A set of variables representing the action of identifier j at step i of episode p. k: Identifier of the action currently selected by the agent (1 ≤ k ≤ N) A(p, i, k): A set of variables representing the action of identifier k, which is actually selected in step i of episode p.

2. The aforementioned specific state is a state in which all of the actions A(p, i+1, j) in step i+1 violate the constraints defined in the environmental model. The information processing apparatus according to claim 1.

3. The history H(p2) includes all actions A_p2_i = {A(p2, n, k) | 1 ≤ n ≤ i} selected from step 1 to step i in episode p2. The information processing apparatus according to claim 1.

4. The history H(p2) includes the state of the environment model S_p2_i = {S(p2, n) | 1 ≤ n ≤ i} in all of step 1 to step i in episode p2. The information processing apparatus according to claim 3.

5. The aforementioned processor, If the first determination unit determines that the state S(p3, i+1) of the environment model in episode p3 is the specific state, the exclusion unit excludes the action A(p3, i-n, k) that was actually selected in step i-n from the selection target of the agent, If the first determination unit determines that the state S(p3, i+1) of the environment model in episode p3 is the specific state, the reverse unit returns step i of episode p3 to the previous step i-m, Further including, The information processing apparatus according to any one of claims 1 to 4. however, p3: Episode identifier (p3 ∈ p) n: A setting value of 0 or greater that represents the number of returns for step i. m: A setting value representing the number of returns for step i (m > n)

6. The aforementioned processor, A second determination unit determines whether or not there exists an episode p in step i+1 where the state S(p, i+1) of the environment model is not the specific state. It further includes, If the second determination unit determines that there exists an episode p in which the state S(p, i+1) of the environment model in step i+1 is not the specific state, The duplication unit duplicates the history H(p2) of episode p2, The overwrite unit overwrites the history H(p1) of episode p1 by copying the history H(p2) of episode p2. If the second determination unit determines that there is no episode p in which the state S(p, i+1) of the environment model in step i+1 is not the specific state, The exclusion unit excludes the action A (p3, i-n, k) from the selection target of the agent. The aforementioned rewind unit returns step i of episode p3 to the previous step i-m. The information processing apparatus according to claim 5.

7. An information processing method by an information processing device that rewards the agent for the action A(p,i,k) selected by the agent in step i of episode p, and transitions the state S(p,i) of the environment model to the state S(p,i+1) of the next step i+1, wherein In step i of episode p1, the selected action A(p1,i,k) among the actions A(p1,i,j) determines whether the state S(p1,i+1) of the environment model in the next step i+1 is a specific state in which all actions A(p1,i+1,j) are unavailable. If it is determined that the state S(p1, i+1) of the environment model in episode p1 is the specified state, and it is determined that the state S(p2, i+1) of the environment model in episode p2 is not the specified state, the steps include: replicating the history H(p2) of episode p2; The steps include overwriting the history H(p1) of episode p1, in which the state S(p1, i+1) of the environmental model is determined to be the specific state, by copying the history H(p2) of episode p2, including, Information processing methods. however, i: An incrementable variable representing the number of steps in the state of the environment model. p1, p2: Episode identifiers p: A variable representing an episode (p ∋ p1, p2) S(p,i): A set of variables representing the state of the environmental model at step i of episode p. j: Identifier of the action (1 ≤ j ≤ N) A(p, i, j): A set of variables representing the action of identifier j at step i of episode p. k: Identifier of the action currently selected by the agent (1 ≤ k ≤ N) A(p, i, k): A set of variables representing the action of identifier k, which is actually selected in step i of episode p.

8. A computer program that rewards the agent for the action A(p,i,k) selected in step i of episode p, and transitions the state S(p,i) of the environment model to the state S(p,i+1) of the next step i+1, On the computer, In step i of episode p1, the selected action A(p1,i,k) among the actions A(p1,i,j) determines whether the state S(p1,i+1) of the environment model in the next step i+1 is a specific state in which all actions A(p1,i+1,j) are unavailable. If it is determined that the state S(p1, i+1) of the environment model in episode p1 is the specified state, and it is determined that the state S(p2, i+1) of the environment model in episode p2 is not the specified state, the steps include: replicating the history H(p2) of episode p2; The steps include overwriting the history H(p1) of episode p1, in which the state S(p1, i+1) of the environmental model is determined to be the specific state, by copying the history H(p2) of episode p2, To execute Computer program. however, i: An incrementable variable representing the number of steps in the state of the environment model. p1, p2: Episode identifiers p: A variable representing an episode (p ∋ p1, p2) S(p,i): A set of variables representing the state of the environmental model at step i of episode p. j: Identifier of the action (1 ≤ j ≤ N) A(p, i, j): A set of variables representing the action of identifier j at step i of episode p. k: Identifier of the action currently selected by the agent (1 ≤ k ≤ N) A(p, i, k): A set of variables representing the action of identifier k, which is actually selected in step i of episode p.

Images