Machine learning device, inference device, machine learning method, machine learning program, and method for generating trained models.

Abstract

To improve the efficiency and accuracy of learning. [Solution] The machine learning device is a machine learning device that performs reinforcement learning on a learning model that outputs about the actions of agent A1 from the state of environment E1, and comprises: a first acquisition unit that acquires the current state in environment E1 as the first state; a decision unit that determines the action of agent A1 based on a first probability distribution which is the probability distribution of multiple actions of agent A1 for the first state obtained by the learning model; an estimation unit that estimates the change state of a predetermined evaluation index in response to a change in the action; a calculation unit that calculates a loss based on the first probability distribution and the change state of the evaluation index; and an update unit that updates the learning model based on the loss.

Description

Technical Field

[0001] The present disclosure relates to a machine learning device, an inference device, a machine learning method, a machine learning program, and a method for generating a learned model.

Background Art

[0002] Various methods have been proposed to solve complex problems. For example, it has been devised to infer the optimal solution of a problem using a model obtained by machine learning.

[0003] For example, in Patent Document 1, it is described that reinforcement learning is applied and a neural network is trained by a policy gradient algorithm.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in reinforcement learning using a learning model such as a neural network, there is room for further improving the efficiency and accuracy of learning by performing learning while appropriately considering more preferable assumed actions.

[0006] In view of the above problems, an object of the present disclosure is to provide a machine learning device, an inference device, a machine learning method, a machine learning program, and a method for generating a learned model that can improve the efficiency and accuracy of learning.

Means for Solving the Problems

[0007] To solve the above problems, a machine learning device according to one aspect of the present disclosure is a machine learning device that performs reinforcement learning on a learning model that outputs an agent's behavior from the state of the environment, and comprises: a first acquisition unit that acquires the current state of the environment as a first state; a decision unit that determines the agent's behavior based on a first probability distribution which is a probability distribution of a plurality of the agent's behaviors for the first state obtained by the learning model; an estimation unit that estimates the state of change of a predetermined evaluation index in response to a change in the behavior; a calculation unit that calculates a loss based on the first probability distribution and the state of change of the evaluation index; and an update unit that updates the learning model based on the loss.

[0008] A machine learning method according to another aspect of the present disclosure is a machine learning method that performs reinforcement learning on a learning model that outputs an agent's behavior from the state of an environment, comprising the steps of: obtaining the current state of the environment as a first state; determining the agent's behavior based on a first probability distribution which is a probability distribution of a plurality of the agent's behaviors for the first state obtained by the learning model; estimating the state of change of a predetermined evaluation index in response to a change in the behavior; calculating a loss based on the first probability distribution and the state of change of the evaluation index; and updating the learning model based on the loss.

[0009] A machine learning program according to another aspect of this disclosure is a machine learning program that performs reinforcement learning on a learning model that outputs an agent's actions from the state of the environment, wherein the computer functions as a first acquisition unit that acquires the current state of the environment as a first state, a decision unit that determines the agent's actions based on a first probability distribution which is a probability distribution of a plurality of the agent's actions for the first state obtained by the learning model, an estimation unit that estimates the state of change of a predetermined evaluation index in response to a change in the action, a calculation unit that calculates a loss based on the first probability distribution and the state of change of the evaluation index, and an update unit that updates the learning model based on the loss.

[0010] A method for generating a trained model according to another aspect of the present disclosure is a method for generating a trained model by performing reinforcement learning on a training model that outputs an agent's actions from the state of the environment, comprising the steps of: obtaining the current state of the environment as a first state; determining the agent's actions based on a first probability distribution which is a probability distribution of a plurality of the agent's actions for the first state obtained by the training model; estimating the state of change of a predetermined evaluation index in response to a change in the action; calculating a loss based on the first probability distribution and the state of change of the evaluation index; and updating the training model based on the loss.

[0011] Furthermore, this disclosure may be implemented as a semiconductor integrated circuit that implements part or all of the program, as an information processing device, or as a system including an information processing device. [Effects of the Invention]

[0012] The machine learning apparatus, inference apparatus, machine learning method, machine learning program, and method for generating trained models described herein can improve the efficiency and accuracy of learning. [Brief explanation of the drawing]

[0013] [Figure 1] This diagram schematically shows an example of the configuration of a planning system according to the embodiment of this disclosure. [Figure 2] This figure shows an example of the loading and unloading of raw materials to and from a tanking base. [Figure 3] This diagram shows the relationship between the environment and the agent involved in machine learning. [Figure 4] This diagram schematically shows an example of a server device hardware configuration. [Figure 5] This block diagram shows an example of various functions in a server device. [Figure 6] This figure shows an example of loading and unloading plan information. [Figure 7] It is a diagram showing an example of the probability distribution of actions. [Figure 8] It is a diagram showing an example of the constraint conditions in the restriction part. [Figure 9] It is a diagram showing an example of the raw material matching rate for carrying out. [Figure 10] It is a diagram showing an example of the change in the raw material matching rate when the actions related to carrying in and out are changed. [Figure 11] It is a diagram showing an example of the correction of the first probability distribution. [Figure 12] It is a diagram showing an overview of the input and output by the learned model. [Figure 13] It is a flowchart showing an example of the flow of the learning process. [Figure 14] It is a flowchart showing an example of the flow of the inference process.

Embodiments for Carrying out the Invention

[0014] The following description shows some aspects in the present disclosure.

[0015] The machine learning device according to the first aspect of the present disclosure is a machine learning device that performs reinforcement learning on a learning model that outputs an action of an agent from the state of the environment, and includes a first acquisition unit that acquires the current state in the environment as a first state, a determination unit that determines the action of the agent based on a first probability distribution that is a probability distribution of a plurality of actions of the agent with respect to the first state obtained by the learning model, an estimation unit that estimates a change state of a predetermined evaluation index corresponding to the case where the action is changed, a calculation unit that calculates a loss based on the first probability distribution and the change state of the evaluation index, and an update unit that updates the learning model based on the loss.

[0016] In the machine learning device according to the second aspect of the present disclosure, based on the first aspect, it further includes a correction unit that corrects the first probability distribution into a second probability distribution based on the change state of the evaluation index, and the correction unit corrects the first probability distribution into the second probability distribution so that the evaluation index is improved.

[0017] A machine learning apparatus according to a third aspect of this disclosure further comprises a correction unit that, according to the first aspect, corrects the first probability distribution to a second probability distribution based on the change state of the evaluation index, wherein the estimation unit estimates the gradient of the change in the evaluation index when the behavior is changed, and the correction unit corrects the first probability distribution to the second probability distribution based on the gradient.

[0018] A machine learning apparatus according to a fourth aspect of this disclosure further comprises a correction unit that corrects the first probability distribution to a second probability distribution based on the change state of the evaluation index, and the calculation unit calculates the loss based on the difference between the first probability distribution and the second probability distribution.

[0019] In the machine learning apparatus according to the fifth aspect of this disclosure, the calculation unit calculates the difference using KL divergence, as per the fourth aspect.

[0020] In the machine learning apparatus according to the sixth aspect of this disclosure, relating to the first aspect, the update unit updates the learning model so that the loss is reduced.

[0021] In the machine learning device according to the seventh aspect of this disclosure, in the first aspect, the learning model takes data relating to at least one of the plans for loading and unloading of multiple types of raw materials to and from the tank base as input, and outputs data relating to at least one of the plans for loading and unloading of each of the multiple tanks owned by the tank base.

[0022] In the machine learning apparatus according to the eighth aspect of this disclosure, the evaluation index, according to the seventh aspect, is an index that indicates the degree of agreement between the raw materials to be discharged from each of the plurality of tanks and the raw materials required for such discharge.

[0023] In the machine learning apparatus according to the ninth aspect of this disclosure, as per the first aspect, the update unit updates the learning model by backpropagation using the loss.

[0024] A machine learning apparatus according to a tenth aspect of this disclosure further comprises a restriction unit that sets a restriction on each of the multiple actions in the first probability distribution, according to the first or second aspect, and the decision unit selects one of the multiple actions from among the actions for which no restriction has been set.

[0025] An inference device according to the eleventh aspect of this disclosure uses the learned model, which has been trained by the above-described machine learning device, as a trained model to infer output data corresponding to input data.

[0026] A machine learning method according to a twelfth aspect of the present disclosure is a machine learning method that performs reinforcement learning on a learning model that outputs an agent's behavior from the state of the environment, comprising the steps of: acquiring the current state of the environment as a first state; determining the agent's behavior based on a first probability distribution which is a probability distribution of a plurality of the agent's behaviors for the first state obtained by the learning model; estimating the state of change of a predetermined evaluation index in response to a change in the behavior; calculating a loss based on the first probability distribution and the state of change of the evaluation index; and updating the learning model based on the loss.

[0027] A machine learning program according to a 13th aspect of this disclosure is a machine learning program that performs reinforcement learning on a learning model that outputs an agent's actions from the state of the environment, wherein the computer functions as a first acquisition unit that acquires the current state of the environment as a first state, a decision unit that determines the agent's actions based on a first probability distribution which is a probability distribution of a plurality of the agent's actions for the first state obtained by the learning model, an estimation unit that estimates the state of change of a predetermined evaluation index in response to a change in the action, a calculation unit that calculates a loss based on the first probability distribution and the state of change of the evaluation index, and an update unit that updates the learning model based on the loss.

[0028] A method for generating a trained model according to a 14th aspect of this disclosure is a method for generating a trained model by performing reinforcement learning on a training model that outputs an agent's actions from the state of the environment, comprising the steps of: obtaining the current state of the environment as a first state; determining the agent's actions based on a first probability distribution which is a probability distribution of a plurality of the agent's actions for the first state obtained by the training model; estimating the state of change of a predetermined evaluation index in response to a change in the action; calculating a loss based on the first probability distribution and the state of change of the evaluation index; and updating the training model based on the loss.

[0029] The following description illustrates embodiments of the present disclosure. To facilitate understanding of the description, the same components and steps are denoted by the same reference numerals in each drawing whenever possible, and redundant descriptions are omitted.

[0030] <Overall Structure> Figure 1 is a schematic diagram showing an example of the configuration of a planning system 1 according to one embodiment. The planning system 1 is a system that creates a plan corresponding to a pre-defined problem.

[0031] As shown in Figure 1, the planning system 1 consists of a server device 2 and a user terminal 3. The server device 2 and the user terminal 3 can communicate with each other via the network NT.

[0032] Server device 2 is an information processing device (computer) that creates a plan to address a problem using information entered by user terminal 3. In this embodiment, one example of a problem is a problem related to the loading (inbound) and unloading (outbound) of raw materials to and from tank base 21. In this embodiment, a problem related to the planning of loading and unloading raw materials to and from tank base 21 is referred to as the "loading and unloading problem".

[0033] Figure 2 shows an example of the loading and unloading of raw materials to and from the tank base 21. The tank base 21 is a base for temporarily storing raw materials. For example, raw materials are loaded into the tank base 21 using a ship F1, and raw materials are unloaded from the tank base 21 using a ship F2, etc. The means of loading and unloading are not limited to ships F1 and F2. The tank base 21 is equipped with multiple tanks 24. In the example shown in Figure 2, the tank base 21 has tanks 24a, 24b, and 24c. The number of tanks 24 that the tank base 21 has is not limited. The tank base 21 can store the loaded raw materials in each of the tanks 24. The tank base 21 can also unload the raw materials stored in each of the tanks 24. Multiple types of raw materials may be loaded into the tank base 21 and stored in each of the tanks 24. Furthermore, the tank base 21 may have multiple tanks 24 into which different types of raw materials are delivered, or multiple types of raw materials may be delivered to a single tank 24. Each of the multiple types of raw materials stored in each tank 24 can be removed from each tank 24. Although this embodiment uses the problem of both the delivery and removal of raw materials to and from the tank base 21 as an example, it may also use the problem of either the delivery or removal of raw materials to and from the tank base 21 as an example.

[0034] In this embodiment, the server device 2 creates a general plan (overall plan) for the loading and unloading of multiple types of raw materials to and from the tank base 21, and then creates a specific plan for the loading and unloading of each raw material to and from each tank 24 in the tank base 21.

[0035] Specifically, server device 2 is a device that uses machine learning to infer a plan. Therefore, server device 2 corresponds to a machine learning device when performing training and to an inference device when performing inference.

[0036] Figure 3 is a diagram illustrating an overview of machine learning. In this embodiment, the server device 2 performs reinforcement learning. For example, the server device 2 performs deep reinforcement learning. Reinforcement learning is a method in which agent A1 learns what actions will result in a greater reward (evaluation) by completing tasks (solutions to problems) while interacting with the environment E1. Agent A1 is the subject of action. Action is the behavior of agent A1. Environment E1 is both the target of agent A1's actions and a prerequisite for agent A1. That is, the state of environment E1 is the state in which agent A1 is placed within environment E1. The state of environment E1 changes according to the actions agent A1 takes with respect to environment E1. Agent A1 is presented with a reward (evaluation for the action) corresponding to that action. If agent A1 performs a favorable action in environment E1, it is presented with a greater reward compared to if it performs an unfavorable action in environment E1. The reward evaluation method can be set as appropriate according to the learning objectives, etc.

[0037] As shown in action S1, agent A1 takes action on environment E1 in the given state of environment E1. As a result, the state of environment E1 changes to a new state according to the action taken by agent A1. Furthermore, as shown in action S2, agent A1 is presented with the new state of environment E1 and a reward corresponding to the action taken by agent A1. In reinforcement learning, learning is performed based on the interaction between environment E1 and agent A1 that occurs by repeating actions S1 and S2. For example, when learning is performed using episodes with multiple steps, interactions between environment E1 and agent A1 such as actions S1 and S2 are performed in accordance with each step. Here, an episode is the flow (period) from the start to the end of the task to be solved in reinforcement learning.

[0038] Furthermore, the action that agent A1 selects in operation S1 is determined based on policy W1. Policy W1 is a rule (policy) that serves as an indicator for determining agent A1's action based on the state of environment E1 before the action. For example, policy W1 is information that associates multiple actions that agent A1 can take in response to the state of environment E1 with the probability (selection probability) that each action will be performed. For example, reinforcement learning aims to adjust policy W1 so that agent A1 can perform a more favorable action in the state of environment E1. Alternatively, policy W1 may be information that associates multiple actions that agent A1 can take in response to the state of environment E1 with the Q-value (action evaluation) for each action, by applying Q-learning. Alternatively, the Q-value may be converted into a probability corresponding to each action, for example, by using a softmax function.

[0039] In deep reinforcement learning, a neural network is used to derive a policy W1 from the state of environment E1. That is, the neural network outputs information about the actions of agent A1 based on the state of environment E1. The neural network may output a policy W1 that includes multiple actions, or it may output a policy W1 that includes only one action. The neural network is then updated (learned) to maximize the reward. In other words, the neural network is an example of a "learning model." Note that the learning model is not limited to a neural network and can be changed as appropriate depending on the reinforcement learning method and type of problem chosen by the user.

[0040] Returning to Figure 1, User Terminal 3 is a terminal device, an information processing device (computer) used by the user. User Terminal 3 is, for example, a personal computer. The user can input various information through User Terminal 3 and instruct Server Device 2 to create a plan.

[0041] <Hardware Configuration> Figure 4 is a schematic diagram showing an example of the hardware configuration of server device 2.

[0042] As shown in Figure 4, the server device 2 comprises a control device 40, a communication device 41, and a storage device 42. The control device 40 is mainly composed of a CPU (Central Processing Unit) 43 and memory 44.

[0043] In the control device 40, the CPU 43 executes a predetermined program stored in the memory 44 or storage device 42, thereby functioning in various functional configurations described later. The memory 44 is a computer-readable storage medium and may consist of at least one of the following: RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), etc. The memory 44 can store various types of data, such as programs necessary for executing processing in the server device 2.

[0044] The communication device 41 consists of a communication interface and the like for communicating with an external device. For example, the communication device 41 can communicate with the user terminal 3.

[0045] The storage device 42 is a computer-readable instruction recording medium, and is composed of, for example, a hard disk or a solid-state drive. The storage device 42 stores various programs and information necessary for executing processing in the control device 40, as well as information on the processing results. Other examples of non-temporary computer-readable instruction recording media include portable recording media such as magnetic tape, flexible disks, optical disks, digital versatile disks, Blu-ray discs, magneto-optical disks, memory cards, and USB memory.

[0046] The server device 2 may consist of a single information processing device or multiple information processing devices. Furthermore, Figure 4 only shows a portion of the main hardware configuration of the server device 2, and the server device 2 may have other configurations. For example, the server device 2 may further include an input device (not shown) and a display device (not shown). The input device is an input device that receives input from an external source (e.g., a keyboard, mouse, etc.). The input device receives user operations and inputs those operations to the server device 2. The display device is a display device that performs output to an external source (e.g., a display, etc.). The display device outputs characters and images. The server device 2 may have an integrated input device and output device (e.g., a touch panel). The user terminal 3, like the server device 2, includes a control device (CPU and memory), a communication device, a storage device, an input device, and a display device.

[0047] <Functional configuration> Figure 5 is a block diagram illustrating an example of various functions in server device 2. Various processes are executed according to the functions in each block. Computer programs implementing the functions of at least some of the function blocks shown in Figure 5 may be installed in the storage of one or more computers. The CPUs of one or more computers may perform the functions of multiple function blocks shown in Figure 5 by reading the computer programs installed on their own machines into main memory and executing them.

[0048] Furthermore, the functions of each functional block shown in Figure 5 may be executed by a single computer, or they may be executed in a distributed manner across multiple computers. When the functions of each functional block shown in Figure 5 are executed in a distributed manner across multiple computers, these multiple computers may send and receive data via a communication network including a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet.

[0049] As shown in Figure 5, the server device 2 has a functional configuration that mainly consists of a learning unit 51 and an inference unit 52. In other words, in the server device 2, the learning unit 51 functions as a machine learning device, and the inference unit 52 functions as an inference device.

[0050] The learning unit 51 performs learning (deep reinforcement learning) on ​​the learning model. The learning unit 51 mainly comprises a simulation unit 60, a first acquisition unit 61, a decision unit 62, a restriction unit 63, a second acquisition unit 64, an estimation unit 65, a correction unit 66, a calculation unit 67, and an update unit 68.

[0051] The simulation unit 60 executes a simulation related to the loading and unloading problem. For example, with respect to the loading and unloading problem, environment E1 corresponds to a simulation model of the tank base 21. Agent A1 is a virtual entity that performs actions such as loading and unloading at the tank base 21. Agent A1 executes an action determined according to policy W1 based on the state of the tank base 21. An episode is the entire loading and unloading problem (the whole process), and the loading and unloading of raw materials to and from the tank base 21 each corresponds to a step (stage). That is, loading in the loading and unloading problem corresponds to one step, unloading in the loading and unloading problem corresponds to another step, and unloading corresponds to a step different from loading. Then, policy W1 is learned so that the best action (or the action closest to the best) can be determined for the loading and unloading problem. That is, learning is performed on a neural network that outputs policy W1 in response to the loading and unloading problem.

[0052] Specifically, the neural network can receive input data related to the planning of loading and unloading multiple types of raw materials to and from the tank base 21. For example, the input data may include at least one of the following: initial inventory information, loading and unloading plan information, and constraint information.

[0053] The initial inventory information shows the inventory status of each tank 24 in the initial state of the loading and unloading problem. The inventory status is the type and quantity of raw materials stored in each tank 24. In other words, the initial inventory information is information that associates tank 24 (name, identification information, etc.) with the type of raw materials stored and the quantity (weight or percentage) stored.

[0054] The loading and unloading plan information is the overall plan for loading and unloading included in the loading and unloading problem. Figure 6 shows an example of loading and unloading plan information. The loading and unloading plan information associates attributes, dates (or sequences), and loading or unloading quantities for each type of raw material. Attributes indicate loading or unloading. Loading or unloading quantities for each type of raw material indicate the amount of raw material loaded into or unloaded from the tank base 21. In other words, the loading and unloading plan information does not limit individual loading or unloading to tank 24, but shows the loading and unloading plan for the tank base 21. In Figure 6, loading or unloading quantities for each type of raw material are shown by the quantity of each type of raw material corresponding to the attribute (loading or unloading). The example in Figure 6 shows a case where six types of raw materials, raw material G1, raw material G2, raw material G3, raw material G4, raw material G5, and raw material G6, are handled in the loading and unloading problem. Furthermore, the loading / unloading plan information may be associated with the identification information (name, etc.) of the vessel involved in the loading or unloading.

[0055] The constraint information indicates the constraints in the loading and unloading problem. For example, the constraint information includes at least one of the following: upper and lower limit constraints on the inventory in tank 24, constraints on the number of tanks 24 used during loading and unloading, minimum quantities for loading and unloading, concentration constraints, and total tank separation constraints. The concentration constraint is a constraint on the combination and proportion (concentration) of multiple types of raw materials stored in one tank 24. The total tank separation constraint is information that restricts the combination and order in which multiple types of raw materials are stored to prevent separation of multiple types of raw materials within tank 24. The constraint information may also include upper and lower limit constraints on the API (American Petroleum Institute) specific gravity of tank 24. Here, API specific gravity refers to the specific gravity of crude oil as defined by the American Petroleum Institute. API specific gravity is a value that can be measured, for example, in accordance with ASTM D1298. In this disclosure, API specific gravity may be simply referred to as "API".

[0056] Furthermore, the input to the neural network may be preprocessed to ensure that it can be input to the neural network in accordance with the information described above.

[0057] Furthermore, the neural network can output data related to the loading and unloading plans for each of the multiple tanks 24 located at the tank base 21.

[0058] For example, the output data includes information that associates tank selection information, raw material quantity information, and type information. The tank selection information specifies the tank 24 to be operated on regarding loading or unloading. That is, the tank 24 to be loaded into or unloaded is selected in the tank selection information. The raw material quantity information is information that indicates the amount of raw material to be loaded into or unloaded from the selected tank 24, corresponding to the selected tank 24. The type information is information that indicates the type of raw material to be loaded into or unloaded from the selected tank 24, corresponding to the selected tank 24. For example, the output data is shown as loading 100 kl of raw material G3 into tank 24a. If the loading / unloading problem involves multiple steps (loading or unloading), the tank selection information, raw material quantity information, and type information are output corresponding to each step.

[0059] Furthermore, the output data may include evaluation indicators corresponding to the output tank selection information, raw material quantity information, and type information. Examples of evaluation indicators include raw material agreement rate, raw material group agreement rate, and API error. The raw material agreement rate is the ratio of the planned quantity (actual quantity) of raw material to the required quantity (ideal quantity) for each type. For example, in the shipment of raw materials, the raw material agreement rate is the ratio of the amount of raw material to be shipped to the amount of raw material required for shipment. When calculating the raw material agreement rate for multiple types, for example, the average of the raw material agreement rates for each type of raw material is used. The raw material group agreement rate is the ratio of the planned quantity (actual quantity) of raw material to the required quantity (ideal quantity) for each group. A group is, for example, a group of multiple types of raw materials with similar properties. For example, in the shipment of raw materials, the raw material group agreement rate is the ratio of the total amount of raw material from the same group to be shipped to the total amount of raw material from that group required for shipment. When calculating the raw material group agreement rate for multiple groups, for example, the average of the raw material agreement rates for each group is used. API error is the difference between the API of the required raw materials (e.g., the average API of various raw materials) and the API of the planned raw materials (e.g., the average API of various raw materials). For example, in the shipment of raw materials, the API error represents the difference between the API of the raw materials required for shipment and the API of the raw materials planned for shipment.

[0060] Furthermore, the output of the neural network may be subjected to post-processing.

[0061] Thus, the simulation unit 60 is capable of performing simulations related to the loading and unloading problem.

[0062] Returning to Figure 5, the first acquisition unit 61 acquires the current state in environment E1 as the "first state". That is, the first acquisition unit 61 acquires the first state in which agent A1 is located before agent A1 takes action. Specifically, the first acquisition unit 61 observes the state of the tank base 21 as environment E1 and sets it as the first state.

[0063] In the loading and unloading problem, the first state includes, for example, the state of the tanks 24 in the tank base 21 as environment E1. The state of the tanks 24 is information indicating the amount of raw materials stored in each of the tanks 24 and the type of raw materials being stored.

[0064] When learning is performed through an episode, the first acquisition unit 61 acquires a first state corresponding to each of the multiple steps included in the episode.

[0065] The decision unit 62 determines the action of agent A1 based on policy W1. Specifically, the decision unit 62 determines the action of agent A1 based on policy W1 obtained using a neural network in response to the first state. Policy W1 may be restricted by the restriction unit 63, which will be described later.

[0066] Policy W1 is information about a probability distribution that associates multiple actions that agent A1 can perform in response to the first state with their probabilities (selection probabilities). In this embodiment, the probability distribution related to policy W1 obtained from the neural network is referred to as the "first probability distribution P1". The decision unit 62 selects one action from among the multiple actions based on the first probability distribution P1. For example, the decision unit 62 probabilistically selects one action based on the first probability distribution P1. In this way, the decision unit 62 determines the action of agent A1 corresponding to the first state.

[0067] When learning is performed through episodes, the decision unit 62 determines the action of agent A1 for each of the first states of each step acquired by the first acquisition unit 61.

[0068] The restriction unit 63 sets restrictions on the action selection based on policy W1 in the decision unit 62. Specifically, the restriction unit 63 sets restrictions on each of the multiple actions in the first probability distribution P1 of policy W1. The restrictions, for example, prohibit the decision unit 62 from making a selection. That is, the restriction unit 63 sets a mask for the multiple actions included in policy W1. Figure 7 is a diagram showing an example of when restrictions are set for each action of policy W1. Figure 7 shows an example of the first probability distribution P1. The example in Figure 7 shows a case where probabilities are set corresponding to each of the actions AC1, AC2, AC3, AC4, AC5, AC6, AC7, AC8, AC9, and AC10. Based on predetermined constraints, the restriction unit 63 sets restrictions so that the decision unit 62 does not decide on an action as an action for agent A1 that is judged to be an inappropriate action for agent A1 corresponding to the first state. In the example shown in Figure 7, the restrictions set by the restriction unit 63 are shown as the mask MS. Figure 7 shows an example where the decision unit 62 is prohibited from selecting actions AC1, AC2, AC3, AC4, AC9, and AC10 as actions for agent A1. In this case, the decision unit 62 selects one action (for example, action AC6) from among the multiple actions included in policy W1 that are not restricted (actions AC5, AC6, AC7, and AC8), and decides it to be the action for agent A1.

[0069] Figure 8 shows an example of constraints used in the limiting unit 63. For example, the constraints are upper and lower limits on the inventory of tank 24, upper and lower limits on the API of the inventory of tank 24, upper limit limit on the number of tanks 24 used during unloading, upper limit limit on the number of tanks 24 used during loading, lower limit limit on the amount loaded per tank, lower limit limit on the amount unloaded per tank, upper limit limit on the amount transported during shifts between tanks, lower limit limit on the amount transported during shifts between tanks, upper limit limit on the number of shifts between tanks per predetermined period (e.g., 1 month), and upper limit limit on the number of shifts between tanks per predetermined period (e.g., 1 day). A shift is an operation to move raw materials from one tank 24 to another tank 24. The limiting unit 63 may use at least one of the above constraints. In addition, each constraint is assigned a type of action to which it applies. The types of actions are the selection of tank 24 during unloading, the amount of raw materials during unloading, the selection of tank 24 during loading, and the amount of raw materials during loading. Furthermore, the types of actions correspond to the shifts and include whether or not to use the tank, the selection of the tank 24 to unload from, the selection of the tank 24 to load into, and the amount of raw material. Figure 8 is a diagram illustrating an example of the correspondence between constraints and the types of actions to which those constraints can be applied. In Figure 8, "○" indicates that a particular constraint is applicable to a particular action. In addition, each constraint may or may not be assigned a priority. In the example in Figure 8, an example is shown where three levels of priority (high, medium, low) are set. In the example in Figure 8, "high" is the highest priority, "medium" is the next highest, and "low" is the next highest. More specifically, if there are multiple constraints imposed on the same action, the constraint with high priority is applied preferentially over the constraints with medium priority and the constraints with low priority. Also, if there are multiple constraints imposed on the same action, the constraint with medium priority is applied preferentially over the constraint with low priority.Furthermore, "preferential application" may mean, for example, when multiple constraints are imposed on the same action, adjusting the degree of influence of each constraint on the action using a weighting according to the priority order of those constraints; or it may mean that only the constraint with the highest priority among those multiple constraints is applied to the action; or it may mean that the constraint is applied to the action in such a way that the constraint with the highest priority is always satisfied, while the constraints of other priorities are satisfied as much as possible. Furthermore, the number of priority levels is not limited to three, and can be changed as appropriate by the user, for example, to two levels. Furthermore, the priority of each constraint can be changed as appropriate by the user.

[0070] In this way, the restriction unit 63 sets restrictions on multiple actions before the action decision is made. In this embodiment, the case in which the restriction unit 63 is provided in the learning unit 51 is given as an example, but the restriction unit 63 may be omitted. Furthermore, the constraints used by the restriction unit 63 are not limited to the above example in the loading and unloading problem and may be changed by the user as appropriate. In addition, when applying a problem other than the loading and unloading problem to the planning creation system 1, the constraints used by the restriction unit 63 may be set as appropriate constraints that are effective for that problem. Furthermore, in this case, when multiple constraints are set, each of the multiple constraints may be set as an appropriate priority.

[0071] Returning to Figure 5, the second acquisition unit 64 acquires the state of environment E1, which has changed as a result of the action determined by the decision unit 62, as the "second state." The second acquisition unit 64 also acquires the reward for the action determined by the decision unit 62 as the "first reward."

[0072] Specifically, the second acquisition unit 64 observes the state of the tank base 21 as environment E1 after the action and sets it to the second state. The second acquisition unit 64 also acquires an evaluation of the action that changed the state of the tank base 21 from the first state to the second state as the first reward.

[0073] In the loading and unloading problem, the second state includes, for example, the state of the tank 24 in the tank base 21 as environment E1.

[0074] When learning occurs through episodes, the second acquisition unit 64 acquires a first reward and a second state corresponding to each step's action determined by the decision unit 62.

[0075] The estimation unit 65 estimates the state of change of a predetermined evaluation index in response to a change in behavior. In this embodiment, the predetermined evaluation index is the raw material match rate. Specifically, the raw material match rate is the ratio of the amount of raw material to be shipped out to the amount of raw material required for shipment. That is, the raw material match rate is an index that shows the degree of match between the raw material to be shipped out from each of the multiple tanks 24 and the raw material required for said shipment. Note that the predetermined evaluation index is not limited to the raw material match rate in the loading and unloading problem and may be changed by the user as appropriate. Also, when applying a problem other than the loading and unloading problem to the planning system 1, the user may appropriately set an index that is effective for that problem.

[0076] The raw material matching rate is based on the amount of raw material to be shipped out, and is therefore related to the content of the action (e.g., action related to shipping). For this reason, the estimation unit 65 estimates the change in the raw material matching rate in response to a small change in the action. The action to be changed is the action of agent A1, for example, the action determined in the decision unit 62. Note that the action to be changed may be an action other than the action determined in the decision unit 62. Specifically, the estimation unit 65 estimates a slope indicating the change in the raw material matching rate. The slope indicates the inclination of the change in the raw material matching rate when the action is changed. The raw material matching rate is higher the more the requirements related to shipping are met. For this reason, if the slope when the action is changed slightly is positive, it indicates that the change in the action is a favorable change for the raw material matching rate. Conversely, if the slope when the action is changed slightly is negative, it indicates that the change in the action is an unfavorable change for the raw material matching rate. Therefore, the slope of the raw material matching rate indicates the favorable direction of action changes. Furthermore, if an index other than the raw material matching rate is set as a predetermined evaluation index estimated by the estimation unit 65, the favorable change (gradient) for that index may vary depending on the type of index.

[0077] Figure 9 shows an example of the raw material matching rate for shipment. In Figure 9, the required quantities of each raw material for shipment are 30 for raw material G1, 20 for raw material G2, and 30 for raw material G3. The quantities of each raw material to be shipped out by ship F2, for example, are 18 for raw material G1, 15 for raw material G2, 17 for raw material G3, and 30 for raw material G4. In this case, the raw material matching rate is (18+15+17) / (30+20+30) ≈ 0.63 (63%). When a small change in action causes the raw material matching rate to decrease, the slope of that change in action will be negative.

[0078] Figure 10 shows an example of how the raw material matching rate changes when the actions related to loading and unloading are changed. In Figure 10, numbers, operations, raw material types, classifications, and tanks 24 are associated with each other. The numbers indicate the order in which loading and unloading are performed. In the example in Figure 10, the case where the order is set from N1 to N4 is shown as an example. In this embodiment, loading or unloading is performed sequentially from N1, with unloading (operation) N4 being performed last. The operation is loading or unloading. The raw material type is the type of raw material to be operated on. For example, in loading with order N1, it indicates that raw material G1 is the target of loading. The classification is the amount of change and the gradient. For example, in unloading with order N2, it indicates that if the amount unloaded from tank 24a is increased by 30, the gradient of the raw material matching rate becomes -0.2. Figure 10 shows that when the amount of material loaded into tank 24a is increased by 20 in sequence N1, the gradient becomes -0.1, and this gradient indicates the effect on the raw material matching rate related to the unloading in N2. In this way, the estimation unit 65 estimates the gradient related to the change in the raw material matching rate in response to changes in the loading and unloading actions. The estimation unit 65 estimates the gradient by changing the actions in response to multiple patterns, for example, as shown in Figure 10.

[0079] The correction unit 66 corrects the first probability distribution P1 based on the change in a predetermined evaluation index. Specifically, the correction unit 66 corrects the first probability distribution P1 so that the predetermined evaluation index is improved. In this embodiment, the predetermined evaluation index is the raw material matching rate, so the first probability distribution P1 is corrected so that the raw material matching rate is improved. The correction unit 66 corrects the first probability distribution P1 based on the slope of the predetermined evaluation index (raw material matching rate). The correction unit 66 corrects the first probability distribution P1 to obtain the "second probability distribution P2".

[0080] Specifically, the correction unit 66 corrects the probability of each action in the first probability distribution P1 using a gradient. The correction unit 66 corrects the probability of the action in the first probability distribution P1 corresponding to the action content changed in the estimation unit 65 using a gradient corresponding to that change. That is, the probability of the action in the first probability distribution P1 that is equal to or close to the changed action content changed in the estimation unit 65 is corrected by the gradient. For example, if the estimation unit 65 changes action AC6 and estimates the gradient, and the direction of the change corresponds to action AC3, the probability of action AC3 is corrected by that gradient. For example, if the gradient is positive, action AC3 corresponds to an action that is favorable for the raw material matching rate (evaluation index), so the probability is corrected to increase (in the direction of increase). On the other hand, if the gradient is negative, action AC3 corresponds to an action that is unfavorable for the raw material matching rate (evaluation index), so the probability is corrected to decrease (in the direction of decrease). That is, the probability of actions that are favorable for the raw material matching rate increases, and the probability of actions that are unfavorable for the raw material matching rate decreases. This corrects the first probability distribution P1 so that actions favorable to the raw material matching rate have a higher probability, and actions unfavorable to the raw material matching rate have a lower probability. As a result, the probability distribution as a whole is corrected to be more favorable to the raw material matching rate (evaluation metric), i.e., the raw material matching rate (evaluation metric) is improved. In other words, the first probability distribution P1 is improved from the viewpoint of the raw material matching rate (evaluation metric) and becomes the second probability distribution P2. That is, correcting the first probability distribution P1 to become the second probability distribution P2 so that the evaluation metric is improved means that the probability of agent A1 performing an action favorable to the evaluation metric in the second probability distribution P2 is greater than the probability of agent A1 performing an action favorable to the evaluation metric in the first probability distribution P1. Here, correcting the first probability distribution to become the second probability distribution so that the evaluation metric is improved does not mean that the probabilities of all actions favorable to the evaluation metric are greater in the second probability distribution P2 than in the first probability distribution P1. For example, it is sufficient if the probability of a desirable behavior for the evaluation index among the behaviors not restricted by the restriction unit 63 in the first probability distribution P1 is greater in the second probability distribution P2 than in the first probability distribution P1.For example, it is sufficient that the probability of an action favorable to the evaluation metric among the actions pre-specified by the user is greater in the second probability distribution P2 than in the first probability distribution P1. Furthermore, it is preferable that the larger the absolute value of the gradient, the larger the increase or decrease in probability. For example, the correction unit 66 may add the gradient to the probability, or it may multiply the gradient by a constant and add it to the probability. The correction method is not limited as long as the probability is corrected by the sign (positive or negative) and absolute value of the gradient.

[0081] Figure 11 shows an example of the correction of the first probability distribution P1. The first probability distribution P1 in Figure 11 corresponds to the first probability distribution P1 in Figure 7. By correcting the first probability distribution P1 using the gradient, the probability of each action is updated. For example, for actions AC1, AC2, AC6, and AC8, the corresponding gradients become positive, and the probability increases. On the other hand, for actions AC3, AC4, AC5, AC7, AC9, and AC10, the corresponding gradients become negative, and the probability decreases. In particular, action AC6 is the most favorable for the raw material matching rate, with the largest absolute value of the gradient and the largest increase in probability. In this way, the first probability distribution P1 is corrected using the gradient, and the second probability distribution P2 is derived. As a result, the second probability distribution P2 is a more favorable probability distribution for the raw material matching rate than the first probability distribution P1.

[0082] The calculation unit 67 calculates the loss related to the gradient of the evaluation index based on the difference between the first probability distribution P1 and the second probability distribution P2. In this embodiment, the "loss" calculated by the calculation unit 67 is referred to as the "evaluation gradient loss LS". The evaluation gradient loss LS can also be said to represent the difference between the policy W1 obtained from the neural network and a more preferable policy W1 that improves the evaluation index.

[0083] Specifically, the calculation unit 67 calculates the KL divergence (Kullback-Leibler divergence) as the difference between the first probability distribution P1 and the second probability distribution P2. The KL divergence is an indicator that shows the difference between two probability distributions. The calculation unit 67 calculates the KL divergence using the first probability distribution P1 and the second probability distribution P2 and uses it as the evaluation gradient loss LS. The calculation unit 67 may use the value of the KL divergence as the evaluation gradient loss LS, or it may convert the value of the KL divergence in a predetermined way (for example, by a constant multiplier) and use it as the evaluation gradient loss LS.

[0084] The update unit 68 updates the neural network based on the evaluation gradient loss LS. Specifically, the update unit 68 updates the neural network so that the evaluation gradient loss LS becomes smaller (in the direction of decreasing it). More specifically, the update unit 68 updates the weights and other components included in the neural network so that the value of the evaluation gradient loss LS of the updated neural network is smaller than the value of the evaluation gradient loss LS of the neural network before the update. In other words, the update unit 68 updates the neural network so that a probability distribution that is the same as the second probability distribution P2, or closer to the second probability distribution P2 than the first probability distribution P1, is obtained. It is preferable that the update unit 68 updates the neural network so that the evaluation gradient loss LS is minimized. The update unit 68 updates the neural network by backpropagation using the evaluation gradient loss LS. Note that the method of updating the neural network is not limited to backpropagation, and other methods may be used. As a result, the neural network is updated so that it can output a policy W1 that makes the evaluation gradient loss LS smaller.

[0085] Furthermore, if the evaluation gradient loss LS is calculated for each step, the update unit 68 may use the evaluation gradient loss LS for each step, or it may use the evaluation gradient loss LS for some steps. Also, if the evaluation gradient loss LS for multiple steps is used, the update unit 68 may use a loss obtained by combining the multiple evaluation gradient losses LS. A combined loss is, for example, the sum of the multiple evaluation gradient losses LS, or the average loss obtained by combining the multiple evaluation gradient losses LS. In addition, if a loss (different from the evaluation gradient loss LS) is calculated using a loss function or the like, the update unit 68 may perform the update using that loss and the evaluation gradient loss LS.

[0086] In this way, deep reinforcement learning is performed in the learning unit 51.

[0087] The inference unit 52 performs inference using the updated neural network as a trained model. Inference is the process of obtaining output data corresponding to input data using the trained model. Figure 12 shows an overview of the input and output by the trained model. The inference unit 52 receives input data from the trained model regarding the plans for loading and unloading multiple types of raw materials to and from the tank base 21. The input data includes, for example, initial inventory information, loading and unloading plan information, and constraint information. For example, each input data is preprocessed so that it can be input into the neural network. The inference unit 52 then obtains output data as an inference result regarding the plans for loading and unloading each raw material corresponding to each of the multiple tanks 24 owned by the tank base 21. The output data includes, for example, information that associates tank selection information, raw material quantity information, and type information. For example, each output data is postprocessed so that it can be output from the neural network.

[0088] <Processing flow> Figure 13 is a flowchart illustrating an example of the learning process flow according to this embodiment. Each of the following processes is initiated, for example, by a user's instruction to start learning. In the learning process, each episode consists of M steps, and each step is associated with a number i (an integer from 1 to M). The order and content of each of the following steps can be changed as appropriate.

[0089] (Step SP10) The simulation unit 60 initializes the model related to the loading and unloading problem. For example, the state of the environment E1 and the weights of the neural network are initialized. In other words, each parameter related to the loading and unloading problem is set to its initial state. Alternatively, each hyperparameter related to learning may be set. Then, the process moves to step SP11.

[0090] (Step SP11) The simulation unit 60 sets the step number i included in the episode to 1 (i=1). Then, the process moves to step SP12.

[0091] (Step SP12) The simulation unit 60 starts the episode and executes step number i. Then, the process moves on to step SP13.

[0092] (Step SP13) The first acquisition unit 61 acquires the current state in environment E1 as the first state. That is, the first acquisition unit 61 acquires the state of environment E1 before agent A1's action in step number i as the first state. Then, the process moves on to step SP14.

[0093] (Step SP14) The decision unit 62 determines the action of agent A1 according to the first probability distribution P1, which is the policy W1 obtained from the neural network in response to the first state. That is, the decision unit 62 determines the action of agent A1 in step number i. Then, the process moves on to step SP15.

[0094] (Step SP15) The simulation unit 60 reflects the determined actions of agent A1 into environment E1. Then, the process moves on to step SP16.

[0095] (Step SP16) The second acquisition unit 64 acquires the state that has changed due to agent A1's actions in environment E1 as the second state, and also acquires the reward for the action as the first reward. In other words, the second acquisition unit 64 acquires the second state and the first reward corresponding to environment E1 after agent A1's actions in step number i. Then the process moves on to step SP17.

[0096] (Step SP17) The estimation unit 65 estimates the gradient of the change in the evaluation index when the behavior is changed. That is, the estimation unit 65 estimates the gradient of the evaluation index when the behavior of agent A1 in step number i is changed. Then the process moves on to step SP18.

[0097] (Step SP18) The correction unit 66 corrects the first probability distribution P1 using the gradient of the evaluation index to obtain the second probability distribution P2. That is, the correction unit 66 derives a more favorable second probability distribution P2 that corresponds to the first probability distribution P1 as policy W1 in step number i. Then the process moves on to step SP19.

[0098] (Step SP19) The calculation unit 67 calculates the evaluation gradient loss LS based on the difference between the first probability distribution P1 and the second probability distribution P2. That is, the calculation unit 67 calculates the evaluation gradient loss LS at step number i. Then, the process moves on to step SP20.

[0099] (Step SP20) The simulation unit 60 determines whether the episode has finished. Specifically, the simulation unit 60 determines that the episode has finished if the step number i is the final number (i=M). If the episode has not finished, the program proceeds to step SP21. If the episode has finished, the program proceeds to step SP22.

[0100] (Step SP21) The simulation unit 60 adds 1 to the number i. Then, the process moves to step SP12 and is executed again. That is, steps SP12 to SP19 are repeatedly executed until the episode ends.

[0101] (Step SP22) The update unit 68 updates the neural network based on the evaluation gradient loss LS. This updates the policy W1 for determining the action of agent A1.

[0102] In this way, deep reinforcement learning is performed. That is, a trained model is generated by training (updating) a neural network, which is an example of a learning model. The neural network updated in step SP23 may be evaluated for its learning status using test episodes, etc. If the learning objective is achieved, training is terminated, or if the learning objective is not achieved, training may be restarted from step SP11.

[0103] Figure 14 is a flowchart showing an example of the inference process flow according to this embodiment. Each of the following processes is started, for example, in accordance with a user's instruction to start inference. Note that the order and content of each of the following steps can be changed as appropriate.

[0104] (Step SP30) The inference unit 52 acquires the trained neural network as a trained model. Then, the process moves to step SP31.

[0105] (Step SP31) The inference unit 52 inputs input data to the neural network. For example, the input data is preprocessed before being input to the neural network. Then the process moves on to step SP32.

[0106] (Step SP32) The inference unit 52 obtains output data from the neural network. For example, post-processing may be performed on the output data output from the neural network. Then the processing is completed.

[0107] <Effects and Effects> As described above, the server device 2 according to this embodiment is a machine learning device that performs reinforcement learning on a learning model that outputs about the actions of agent A1 from the state of environment E1, and comprises: a first acquisition unit 61 that acquires the current state in environment E1 as the first state; a decision unit 62 that determines the action of agent A1 based on a first probability distribution P1 which is the probability distribution of multiple actions of agent A1 for the first state obtained by the learning model; an estimation unit 65 that estimates the change state of a predetermined evaluation index in response to a change in the action; a calculation unit 67 that calculates the evaluation gradient loss LS based on the first probability distribution P1 and the change state of the evaluation index; and an update unit 68 that updates the learning model based on the evaluation gradient loss LS.

[0108] This configuration calculates the evaluation gradient loss LS based on the first probability distribution P1 used to determine the action and the changes in the evaluation metric when the action is changed. Furthermore, the learning model is updated based on the calculated evaluation gradient loss LS. Therefore, the learning model can be updated considering not only the first probability distribution P1 but also the changes in the evaluation metric. Consequently, the learning model is trained while considering actions that are assumed to be more favorable. This makes it possible to improve the efficiency and accuracy of learning.

[0109] Furthermore, the server device 2 is further equipped with a correction unit 66 that corrects the first probability distribution P1 to create a second probability distribution P2 based on the change in the evaluation index, and the correction unit 66 corrects the first probability distribution P1 to create the second probability distribution P2 so that the evaluation index is improved.

[0110] With this configuration, the first probability distribution P1 can be corrected in a direction favorable to the evaluation index to become the second probability distribution P2.

[0111] Furthermore, the server device 2 includes a correction unit 66 that corrects the first probability distribution P1 to a second probability distribution P2 based on the change in the evaluation index, the estimation unit 65 estimates the gradient of the change in the evaluation index when the behavior is changed, and the correction unit 66 corrects the first probability distribution P1 to a second probability distribution P2 based on the gradient.

[0112] This configuration allows for the appropriate indication of changes in the evaluation index by using gradients, enabling efficient correction processing.

[0113] Furthermore, the server device 2 includes a correction unit 66 that corrects the first probability distribution P1 to a second probability distribution P2 based on the change in the evaluation index, and the calculation unit 67 calculates the evaluation gradient loss LS based on the difference between the first probability distribution P1 and the second probability distribution P2.

[0114] With this configuration, the difference between the current probability distribution, the first probability distribution P1, and the second probability distribution P2, which represents a more favorable probability distribution, can be expressed as the evaluation gradient loss LS and applied to learning.

[0115] Furthermore, in server device 2, the calculation unit 67 calculates the difference using KL divergence.

[0116] This configuration makes it possible to accurately calculate the difference in probability distributions.

[0117] Furthermore, in the server device 2, the update unit 68 updates the learning model so that the evaluation gradient loss LS becomes smaller.

[0118] This configuration makes it possible to perform learning so that the first probability distribution P1 obtained from the learning model approaches the second probability distribution P2.

[0119] Furthermore, in the server device 2, the learning model takes data relating to at least one of the plans for loading and unloading multiple types of raw materials to and from the tank base 21 as input, and outputs data relating to at least one of the plans for loading and unloading each of the multiple tanks 24 that the tank base 21 has.

[0120] This configuration allows for the training of a learning model that can handle the problem of raw material loading and unloading.

[0121] Furthermore, in the server device 2, the evaluation index is an index that shows the degree of agreement between the raw materials that are scheduled to be discharged from each of the multiple tanks 24 and the raw materials required for said discharge.

[0122] With this configuration, it becomes possible to derive the second probability distribution P2 from the first probability distribution P1 in terms of raw material consistency.

[0123] Furthermore, in the server device 2, the update unit 68 updates the learning model by backpropagation using the evaluation gradient loss LS.

[0124] This configuration allows the learning model to be updated to minimize the evaluation gradient loss LS.

[0125] Furthermore, the server device 2 is further equipped with a restriction unit 63 that sets restrictions on each of the multiple actions in the first probability distribution P1, and the decision unit 62 selects one action from among the multiple actions for which no restrictions have been set.

[0126] This configuration suppresses the selection of inappropriate actions included in the probability distribution by the decision unit 62.

[0127] Furthermore, server device 2 uses the trained model as a trained model to infer output data corresponding to the input data.

[0128] This configuration allows the trained model to perform inferences, for example, to address loading and unloading problems.

[0129] <Variation> This disclosure is not limited to the embodiments described above. That is, any modifications made to the embodiments described above by a person skilled in the art are also included in the scope of this disclosure, as long as they retain the features of this disclosure. Furthermore, the elements of the embodiments described above and the modifications described later can be combined to the extent that it is technically possible, and any combination thereof is also included in the scope of this disclosure, as long as it retains the features of this disclosure.

[0130] In the above embodiment, one example is that each function is provided by the server device 2, but each function may also be provided by the user terminal 3. Alternatively, each function may be distributed between the server device 2 and the user terminal 3. For example, the user terminal 3 may function as both a machine learning device and an inference device. Alternatively, one of the server device 2 and the user terminal 3 may function as a machine learning device, and the other of the server device 2 and the user terminal 3 may function as an inference device. Alternatively, two server devices 2 that can communicate with each other may be installed, with one server device 2 functioning as a machine learning device and the other server device 2 functioning as an inference device. Alternatively, multiple server devices 2 that can communicate with each other may be installed, with each function constituting the machine learning device and each function constituting the inference device distributed among multiple server devices 2, and these may function together as a machine learning device or an inference device.

[0131] Furthermore, while the above embodiment described the application of the problem of loading and unloading raw materials at the tank base 21 to the planning system 1 as an example, the problems that can be applied to the planning system 1 are not limited to those described above. For example, the problem of vehicle entry and exit at a parking facility may be applied to the planning system 1. Also, the problem of ship entry and departure at a port may be applied to the planning system 1. Also, the problem of receiving and shipping goods and merchandise at a warehouse or store may be applied to the planning system 1. It should be noted that the problems that can be applied to the planning system 1 are not limited to those described above, and a variety of problems can be applied.

[0132] Furthermore, while the above embodiment shows an example where the evaluation gradient loss LS is calculated using KL divergence, the embodiment is not limited to this. For example, the evaluation gradient loss LS may be calculated using an index other than KL divergence, as long as it can show the difference between the first probability distribution P1 and the second probability distribution P2.

[0133] Furthermore, in the above embodiment, the difference between the first probability distribution P1 and the second probability distribution P2 (e.g., KL divergence) was used as an example of the evaluation gradient loss LS. However, the evaluation gradient loss LS may also be a value obtained by performing a predetermined transformation on the difference between the first probability distribution P1 and the second probability distribution P2. For example, the transformation may be performed using a function that multiplies the difference by a constant.

[0134] The various types of information described in this disclosure (e.g., status, reward, etc.) may be expressed using absolute values, relative values ​​from a given value, or corresponding other information.

[0135] In this disclosure, expressions such as "based on," "using," and "by" (including equivalent expressions) do not mean "based solely on," "using only," or "by" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "at least on," and the same applies to equivalent expressions such as "using" and "by."

[0136] The term “decision” in this disclosure may encompass a wide variety of actions. “Decision” may include, for example, judgment, calculation, calculation, processing, derivation, investigation, exploration, and confirmation. Furthermore, “decision” may include, for example, considering something to have been “decided,” such as resolving, selecting, choosing, establishing, or comparing. In short, “decision” may include considering any action to have been “decided.”

[0137] In this disclosure, where expressions such as "obtain / set / as input / using / based on" (including similar expressions) are used, unless otherwise specified, this includes cases where the information itself is used, or where the information has been processed in some way (e.g., noise-added, normalized, features extracted from the information, intermediate representation of the information, etc.) is used. Furthermore, where it is stated that some result is obtained by "obtaining / setting / as input / using / based on" (including similar expressions), unless otherwise specified, this includes cases where the result is obtained solely based on the information in question, or where the result is influenced by other information, factors, conditions, and / or states other than the information in question. Furthermore, where it is stated that "outputs" (including similar expressions), unless otherwise specified, this includes cases where the information itself is used as output, or where the information has been processed in some way (e.g., noise-added, normalized, features extracted from the information, intermediate representation of various types of information, etc.) is used as output. [Explanation of Symbols]

[0138] 2: Server device (machine learning device) 61:First Acquisition Department 62: Decision Section 65: Estimation part 66: Correction section 67: Calculation section 68: Update section A1: Agent AC1~AC10: Actions E1 :Environment LS: Evaluation Gradient Loss (Loss) P1: First probability distribution P2: Second probability distribution

Claims

1. A machine learning device that performs reinforcement learning on a learning model that outputs information about an agent's actions based on the state of the environment, A first acquisition unit acquires the current state in the aforementioned environment as the first state, A decision unit that determines the agent's action based on a first probability distribution, which is a probability distribution of multiple actions of the agent for the first state obtained by the learning model, An estimation unit that estimates the state of change of a predetermined evaluation index in response to a change in the aforementioned behavior, A calculation unit that calculates the loss based on the first probability distribution and the change state of the evaluation index, An update unit that updates the learning model based on the loss, A machine learning device equipped with the following features.

2. A correction unit that corrects the first probability distribution to obtain a second probability distribution based on the change state of the evaluation index. Furthermore, The correction unit corrects the first probability distribution to obtain the second probability distribution so that the evaluation index is improved. The machine learning apparatus according to claim 1.

3. A correction unit that corrects the first probability distribution to obtain a second probability distribution based on the change state of the evaluation index. Furthermore, The estimation unit estimates the gradient of the change in the evaluation index when the behavior is changed. The correction unit corrects the first probability distribution using the gradient to obtain the second probability distribution. The machine learning apparatus according to claim 1.

4. A correction unit that corrects the first probability distribution to obtain a second probability distribution based on the change state of the evaluation index. Furthermore, The calculation unit calculates the loss based on the difference between the first probability distribution and the second probability distribution. The machine learning apparatus according to claim 1.

5. The calculation unit calculates the difference using KL divergence. The machine learning apparatus according to claim 4.

6. The update unit updates the learning model so that the loss is reduced. The machine learning apparatus according to claim 1.

7. The learning model takes data relating to at least one of the plans for the loading and unloading of multiple types of raw materials to and from the tank base as input, and outputs data relating to at least one of the plans for the loading and unloading of each of the multiple tanks owned by the tank base. The machine learning apparatus according to claim 1.

8. The aforementioned evaluation index is an index that indicates the degree of agreement between the raw materials to be discharged from each of the multiple tanks and the raw materials required for such discharge. The machine learning apparatus according to claim 7.

9. The update unit updates the learning model by backpropagation using the loss. The machine learning apparatus according to claim 1.

10. A limiting unit that sets a restriction on each of the multiple actions in the first probability distribution, Furthermore, The decision unit selects one of the actions from among the multiple actions for which no restrictions are set. The machine learning apparatus according to claim 1.

11. Using the machine learning device described in any one of claims 1 to 10 as the trained model, output data corresponding to input data is inferred. Reasoning device.

12. A machine learning method that performs reinforcement learning on a learning model that outputs information about an agent's actions based on the state of the environment, A step of acquiring the current state in the aforementioned environment as the first state, A step of determining the agent's action based on a first probability distribution, which is a probability distribution of multiple actions of the agent for the first state obtained by the learning model, A step of estimating the state of change of a predetermined evaluation index in response to a change in the aforementioned behavior, A step of calculating the loss based on the first probability distribution and the change state of the evaluation index, A step of updating the learning model based on the loss, A machine learning method that uses [a specific feature].

13. A machine learning program that performs reinforcement learning on a learning model that outputs information about an agent's actions based on the state of the environment, Computers, A first acquisition unit acquires the current state in the aforementioned environment as the first state. A decision unit determines the agent's action based on a first probability distribution, which is the probability distribution of multiple actions of the agent for the first state obtained by the learning model. An estimation unit that estimates the state of change of a predetermined evaluation index in response to a change in the aforementioned behavior. A calculation unit calculates the loss based on the first probability distribution and the change state of the evaluation index. An update unit that updates the learning model based on the loss, A machine learning program that functions as such.

14. A method for generating a trained model by performing reinforcement learning on a learning model that outputs information about an agent's actions based on the state of the environment, A step of acquiring the current state in the aforementioned environment as the first state, A step of determining the agent's action based on a first probability distribution, which is a probability distribution of multiple actions of the agent for the first state obtained by the learning model, A step of estimating the state of change of a predetermined evaluation index in response to a change in the aforementioned behavior, A step of calculating the loss based on the first probability distribution and the change state of the evaluation index, A step of updating the learning model based on the loss, A method for generating a trained model having [a certain characteristic].

Images