Machine learning device, inference device, machine learning method, machine learning program, and method for generating trained models.

Abstract

To improve the efficiency and accuracy of learning. [Solution] A machine learning device that learns a problem involving multiple stages, comprising: a plurality of individual models MP that output intermediate results of the problem corresponding to each stage; and an overall model MS that takes the output of each individual model MP as input and outputs the final result of the problem, comprising: a first acquisition unit that acquires the information output by each individual model MP as first information; a first calculation unit that calculates the loss corresponding to each piece of first information as the first loss based on the loss function corresponding to each individual model MP; and a first update unit that updates the overall model MS based on the first loss.

Description

Technical Field

[0001] The present disclosure relates to a machine learning device, an inference device, a machine learning method, a machine learning program, and a method for generating a learned model.

Background Art

[0002] Various methods have been proposed to solve complex problems. For example, it has been devised to infer an optimal solution to a problem using a model trained by machine learning.

[0003] For example, in Patent Document 1, it is described that a new deep reinforcement learning model is learned using a plurality of deep reinforcement learning models in different environments.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when a problem includes a plurality of stages, for example, it has been difficult to learn a more preferable state as a whole while taking a balance between an initial stage and a later stage. That is, there is room for further improving the efficiency and accuracy of learning when a problem includes a plurality of stages.

[0006] In view of the above problems, an object of the present disclosure is to provide a machine learning device, an inference device, a machine learning method, a machine learning program, and a method for generating a learned model that can improve the efficiency and accuracy of learning.

Means for Solving the Problems

[0007] To solve the above problems, a machine learning device according to one aspect of the present disclosure is a machine learning device that learns a plurality of individual models that output intermediate results of a problem corresponding to each of the stages, and an overall model that takes the output of each individual model as input and outputs the final result of the problem, and comprises: a first acquisition unit that acquires information output by each individual model as first information; a first calculation unit that calculates a loss corresponding to each of the first pieces of information as a first loss based on a loss function corresponding to each of the individual models; and a first update unit that updates the overall model based on the first loss.

[0008] A machine learning method according to another aspect of the present disclosure is a machine learning method that learns a plurality of individual models that output intermediate results of a problem corresponding to each of the stages, and an overall model that takes the output of each of the individual models as input and outputs the final result of the problem, comprising the steps of: acquiring information output by each of the individual models as first information; calculating a loss corresponding to each of the first pieces of information as a first loss based on a loss function corresponding to each of the individual models; and updating the overall model based on the first loss.

[0009] A machine learning program according to another aspect of the present disclosure is a machine learning program that learns a plurality of individual models that output intermediate results of a problem corresponding to each of the stages, and an overall model that takes the output of each of the individual models as input and outputs the final result of the problem, wherein the computer functions as a first acquisition unit that acquires information output by each of the individual models as first information, a first calculation unit that calculates a loss corresponding to each of the first pieces of information as a first loss based on a loss function corresponding to each of the individual models, and a first update unit that updates the overall model based on the first loss.

[0010] A method for generating a trained model according to another aspect of the present disclosure is a method for generating a trained model that trains a plurality of individual models that output intermediate results of a problem corresponding to each of the stages, and an overall model that takes the outputs of each of the individual models as input and outputs the final result of the problem, comprising the steps of: acquiring information output by each of the individual models as first information; calculating a loss corresponding to each of the first pieces of information as a first loss based on a loss function corresponding to each of the individual models; and updating the overall model based on the first loss.

[0011] Furthermore, this disclosure may be implemented as a semiconductor integrated circuit that implements part or all of the program, as an information processing device, or as a system including an information processing device. [Effects of the Invention]

[0012] The machine learning apparatus, inference apparatus, machine learning method, machine learning program, and method for generating trained models described herein can improve the efficiency and accuracy of learning. [Brief explanation of the drawing]

[0013] [Figure 1] This diagram schematically shows an example of the configuration of a planning system according to the embodiment of this disclosure. [Figure 2] This figure shows an example of the loading and unloading of raw materials to and from a tanking base. [Figure 3] This diagram shows the relationship between the environment and the agent involved in machine learning. [Figure 4] This diagram schematically shows an example of a server device hardware configuration. [Figure 5] This block diagram shows an example of various functions in a server device. [Figure 6] This figure shows an example of a learning model configuration. [Figure 7] This figure shows an example of loading and unloading plan information. [Figure 8]It is a diagram showing an example of the probability distribution of actions. [Figure 9] It is a diagram showing an example of the constraint conditions in the restriction part. [Figure 10] It is a diagram showing an overview of input and output by the learned model. [Figure 11] It is a flowchart showing an example of the flow of the learning process. [Figure 12] It is a flowchart showing an example of the flow of the inference process.

Mode for Carrying Out the Invention

[0014] The following description shows some aspects in the present disclosure.

[0015] The machine learning device according to the first aspect of the present disclosure is a machine learning device that performs learning on a plurality of individual models that output intermediate results of the problem corresponding to each of the plurality of stages in a problem including a plurality of stages, and an overall model that outputs the final result of the problem using the output of each of the individual models as an input. The machine learning device includes a first acquisition unit that acquires the information output from each of the individual models as first information, a first calculation unit that calculates the loss corresponding to each of the first information as a first loss based on the loss function corresponding to each of the individual models, and a first update unit that updates the overall model based on the first loss.

[0016] In the machine learning device according to the second aspect of the present disclosure, based on the first aspect, a second acquisition unit that acquires the information output from the overall model as second information using the information obtained from each of the individual models as an input, a second calculation unit that calculates the loss corresponding to the second information as a second loss based on the loss function corresponding to the overall model, and a third calculation unit that calculates a combined loss obtained by combining each of the first losses and the second loss are further provided. The first update unit updates each of the individual models and the overall model based on the combined loss.

[0017] In the machine learning device according to the third aspect of the present disclosure, in relation to the first aspect or the second aspect, each of the individual models is a model corresponding to each of the different stages from the initial stage of the problem to after the initial stage.

[0018] In the machine learning device according to the fourth aspect of the present disclosure, in relation to the first aspect or the second aspect, the information input to the overall model includes information obtained from an intermediate layer in the neural network constituting the individual model and information obtained from a final layer in the neural network.

[0019] In the machine learning device according to the fifth aspect of the present disclosure, in relation to the second aspect, the first update unit updates each of the individual models and the overall model so that the combined loss becomes smaller.

[0020] The machine learning device according to the sixth aspect of the present disclosure further includes a second update unit that updates each of the individual models based on each of the first losses, in relation to the second aspect.

[0021] The machine learning device according to the seventh aspect of the present disclosure further includes a third update unit that updates the overall model based on the second loss, in relation to the sixth aspect.

[0022] In the machine learning device according to the eighth aspect of the present disclosure, in relation to the seventh aspect, the second update unit updates each of the individual models so that the first loss becomes smaller, and the third update unit updates the overall model so that the second loss becomes smaller.

[0023] In the machine learning device according to the ninth aspect of the present disclosure, in relation to the first aspect or the second aspect, the learning model including the individual models and the overall model takes, as input, data related to at least one of the incoming and outgoing plans of a plurality of types of raw materials for a tank base, and outputs data related to at least one of the incoming and outgoing plans of each of the raw materials corresponding to each of the plurality of tanks possessed by the tank base.

[0024] In the machine learning apparatus according to the tenth aspect of this disclosure, according to the second aspect, the first update unit updates each of the individual models and the overall model by backpropagation using the combined loss.

[0025] An inference device according to the 11th aspect of this disclosure uses each of the individual models learned by the machine learning device described above and the overall model to infer output data corresponding to input data.

[0026] A machine learning method according to a twelfth aspect of this disclosure is a machine learning method that learns a plurality of individual models that output intermediate results of a problem corresponding to each of the stages, and an overall model that takes the output of each individual model as input and outputs the final result of the problem, comprising the steps of: acquiring information output by each individual model as first information; calculating a loss corresponding to each of the first pieces of information as a first loss based on a loss function corresponding to each of the individual models; and updating the overall model based on the first loss.

[0027] A machine learning program according to a 13th aspect of this disclosure is a machine learning program that learns a plurality of individual models that output intermediate results of a problem corresponding to each of the stages, and an overall model that takes the output of each of the individual models as input and outputs the final result of the problem, wherein the computer functions as a first acquisition unit that acquires information output by each of the individual models as first information, a first calculation unit that calculates a loss corresponding to each of the first pieces of information as a first loss based on a loss function corresponding to each of the individual models, and a first update unit that updates the overall model based on the first loss.

[0028] A method for generating a trained model according to a 14th aspect of this disclosure is a method for generating a trained model that trains a plurality of individual models that output intermediate results of a problem corresponding to each of the stages, and an overall model that takes the output of each individual model as input and outputs the final result of the problem, comprising the steps of: acquiring information output by each individual model as first information; calculating a loss corresponding to each of the first pieces of information as a first loss based on a loss function corresponding to each of the individual models; and updating the overall model based on the first loss.

[0029] The following description illustrates embodiments of the present disclosure. To facilitate understanding of the description, the same components and steps are denoted by the same reference numerals in each drawing whenever possible, and redundant descriptions are omitted.

[0030] <Overall Structure> Figure 1 is a schematic diagram showing an example of the configuration of a planning system 1 according to one embodiment. The planning system 1 is a system that creates a plan corresponding to a pre-defined problem.

[0031] As shown in Figure 1, the planning system 1 consists of a server device 2 and a user terminal 3. The server device 2 and the user terminal 3 can communicate with each other via the network NT.

[0032] Server device 2 is an information processing device (computer) that creates a plan to address a problem using information entered by user terminal 3. In this embodiment, one example of a problem is a problem related to the loading (inbound) and unloading (outbound) of raw materials to and from tank base 21. In this embodiment, a problem related to the planning of loading and unloading raw materials to and from tank base 21 is referred to as the "loading and unloading problem".

[0033] Figure 2 shows an example of the loading and unloading of raw materials to and from the tank base 21. The tank base 21 is a base for temporarily storing raw materials. For example, raw materials are loaded into the tank base 21 using a ship F1, and raw materials are unloaded from the tank base 21 using a ship F2, etc. The means of loading and unloading are not limited to ships F1 and F2. The tank base 21 is equipped with multiple tanks 24. In the example shown in Figure 2, the tank base 21 has tanks 24a, 24b, and 24c. The number of tanks 24 that the tank base 21 has is not limited. The tank base 21 can store the loaded raw materials in each of the tanks 24. The tank base 21 can also unload the raw materials stored in each of the tanks 24. Multiple types of raw materials may be loaded into the tank base 21 and stored in each of the tanks 24. Furthermore, the tank base 21 may have multiple tanks 24 into which different types of raw materials are delivered, or multiple types of raw materials may be delivered to a single tank 24. Each of the multiple types of raw materials stored in each tank 24 can be removed from each tank 24. Although this embodiment uses the problem of both the delivery and removal of raw materials to and from the tank base 21 as an example, it may also use the problem of either the delivery or removal of raw materials to and from the tank base 21 as an example.

[0034] In this embodiment, the server device 2 creates a general plan (overall plan) for the loading and unloading of multiple types of raw materials to and from the tank base 21, and then creates a specific plan for the loading and unloading of each raw material to and from each tank 24 in the tank base 21.

[0035] Specifically, server device 2 is a device that uses machine learning to infer a plan. Therefore, server device 2 corresponds to a machine learning device when performing training and to an inference device when performing inference.

[0036] Figure 3 is a diagram illustrating an overview of machine learning. In this embodiment, the server device 2 performs reinforcement learning. For example, the server device 2 performs deep reinforcement learning. Reinforcement learning is a method in which agent A1 learns what actions will result in a greater reward (evaluation) by completing tasks (solutions to problems) while interacting with the environment E1. Agent A1 is the subject of action. Action is the behavior of agent A1. Environment E1 is both the target of agent A1's actions and a prerequisite for agent A1. That is, the state of environment E1 is the state in which agent A1 is placed within environment E1. The state of environment E1 changes according to the actions agent A1 takes with respect to environment E1. Agent A1 is presented with a reward (evaluation for the action) corresponding to that action. If agent A1 performs a favorable action in environment E1, it is presented with a greater reward compared to if it performs an unfavorable action in environment E1. The reward evaluation method can be set as appropriate according to the learning objectives, etc.

[0037] As shown in action S1, agent A1 takes action on environment E1 in the given state of environment E1. As a result, the state of environment E1 changes to a new state according to the action taken by agent A1. Furthermore, as shown in action S2, agent A1 is presented with the new state of environment E1 and a reward corresponding to the action taken by agent A1. In reinforcement learning, learning is performed based on the interaction between environment E1 and agent A1 that occurs by repeating actions S1 and S2. For example, when learning is performed using episodes with multiple steps, interactions between environment E1 and agent A1 such as actions S1 and S2 are performed in accordance with each step. Here, an episode is the flow (period) from the start to the end of the task to be solved in reinforcement learning.

[0038] Furthermore, the action that agent A1 selects in operation S1 is determined based on policy W1. Policy W1 is a rule (policy) that serves as an indicator for determining agent A1's action based on the state of environment E1 before the action. For example, policy W1 is information that associates multiple actions that agent A1 can take in response to the state of environment E1 with the probability (selection probability) that each action will be performed. For example, reinforcement learning aims to adjust policy W1 so that agent A1 can perform a more favorable action in the state of environment E1. Alternatively, policy W1 may be information that associates multiple actions that agent A1 can take in response to the state of environment E1 with the Q-value (action evaluation) for each action, by applying Q-learning. Alternatively, the Q-value may be converted into a probability corresponding to each action, for example, by using a softmax function.

[0039] In deep reinforcement learning, a neural network is used to derive a policy W1 from the state of environment E1. That is, the neural network outputs information about the actions of agent A1 based on the state of environment E1. The neural network may output a policy W1 that includes multiple actions, or it may output a policy W1 that includes only one action. The neural network is then updated (learned) to maximize the reward. In other words, the neural network is an example of a "learning model." Note that the learning model is not limited to a neural network and can be changed as appropriate depending on the reinforcement learning method and type of problem chosen by the user.

[0040] Returning to Figure 1, User Terminal 3 is a terminal device, an information processing device (computer) used by the user. User Terminal 3 is, for example, a personal computer. The user can input various information through User Terminal 3 and instruct Server Device 2 to create a plan.

[0041] <Hardware Configuration> Figure 4 is a schematic diagram showing an example of the hardware configuration of server device 2.

[0042] As shown in Figure 4, the server device 2 comprises a control device 40, a communication device 41, and a storage device 42. The control device 40 is mainly composed of a CPU (Central Processing Unit) 43 and memory 44.

[0043] In the control device 40, the CPU 43 executes a predetermined program stored in the memory 44 or storage device 42, thereby functioning in various functional configurations described later. The memory 44 is a computer-readable storage medium and may consist of at least one of the following: RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), etc. The memory 44 can store various types of data, such as programs necessary for executing processing in the server device 2.

[0044] The communication device 41 consists of a communication interface and the like for communicating with an external device. For example, the communication device 41 can communicate with the user terminal 3.

[0045] The storage device 42 is a computer-readable instruction recording medium, and is composed of, for example, a hard disk or a solid-state drive. The storage device 42 stores various programs and information necessary for executing processing in the control device 40, as well as information on the processing results. Other examples of non-temporary computer-readable instruction recording media include portable recording media such as magnetic tape, flexible disks, optical disks, digital versatile disks, Blu-ray discs, magneto-optical disks, memory cards, and USB memory.

[0046] The server device 2 may consist of a single information processing device or multiple information processing devices. Furthermore, Figure 4 only shows a portion of the main hardware configuration of the server device 2, and the server device 2 may have other configurations. For example, the server device 2 may further include an input device (not shown) and a display device (not shown). The input device is an input device that receives input from an external source (e.g., a keyboard, mouse, etc.). The input device receives user operations and inputs those operations to the server device 2. The display device is a display device that performs output to an external source (e.g., a display, etc.). The display device outputs characters and images. The server device 2 may have an integrated input device and output device (e.g., a touch panel). The user terminal 3, like the server device 2, includes a control device (CPU and memory), a communication device, a storage device, an input device, and a display device.

[0047] <Functional configuration> Figure 5 is a block diagram illustrating an example of various functions in server device 2. Various processes are executed according to the functions in each block. Computer programs implementing the functions of at least some of the function blocks shown in Figure 5 may be installed in the storage of one or more computers. The CPUs of one or more computers may perform the functions of multiple function blocks shown in Figure 5 by reading the computer programs installed on their own machines into main memory and executing them.

[0048] Furthermore, the functions of each functional block shown in Figure 5 may be executed by a single computer, or they may be executed in a distributed manner across multiple computers. When the functions of each functional block shown in Figure 5 are executed in a distributed manner across multiple computers, these multiple computers may send and receive data via a communication network including a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet.

[0049] As shown in Figure 5, the server device 2 has a functional configuration that mainly consists of a learning unit 51 and an inference unit 52. In other words, in the server device 2, the learning unit 51 functions as a machine learning device, and the inference unit 52 functions as an inference device.

[0050] The learning unit 51 performs learning (deep reinforcement learning) on ​​the learning model. The learning unit 51 comprises a simulation unit 60, a first observation unit 61, a decision unit 62, a restriction unit 63, a second observation unit 64, a first acquisition unit 65, a first calculation unit 66, a second acquisition unit 67, a second calculation unit 68, a third calculation unit 69, a first update unit 71, a second update unit 72, and a third update unit 73.

[0051] The simulation unit 60 executes a simulation related to the loading and unloading problem. For example, with respect to the loading and unloading problem, environment E1 corresponds to a simulation model of the tank base 21. Agent A1 is a virtual entity that performs actions such as loading and unloading at the tank base 21. Agent A1 executes an action determined according to policy W1 based on the state of the tank base 21. An episode is the entire loading and unloading problem (the whole process), and the loading and unloading of raw materials to and from the tank base 21 each corresponds to a step (stage). That is, loading included in the loading and unloading problem corresponds to one step, unloading included in the loading and unloading problem corresponds to another step, and unloading corresponds to a step different from loading. Then, policy W1 is learned so that the best action (or the action closest to the best) can be determined for the loading and unloading problem. That is, learning is performed on a learning model that outputs policy W1 in response to the loading and unloading problem.

[0052] Figure 6 shows an example of the configuration of a learning model according to this embodiment. As shown in Figure 6, the learning model that outputs policy W1 is composed of a combination of multiple models. Specifically, the learning model is composed of an individual model MP and an overall model MS.

[0053] Individual model MPs are models constructed using neural networks. Specifically, individual model MPs correspond to input / output problems involving multiple steps, outputting intermediate results corresponding to each stage of the input / output problem. For example, if an input / output problem has multiple steps, from the 1st step to the Nth step (where N is an integer greater than or equal to 1), individual model MPs produce outputs corresponding to each step. Therefore, each individual model MP corresponds to a different step from the 1st step (initial stage) onward. For example, individual model MPs could include individual model MP1 corresponding to the 1st step, individual model MP2 corresponding to the 1st to the 2nd step, and individual model MPN corresponding to the 1st to the Nth step. In other words, if an input / output problem has steps from the 1st to the Nth step, individual model MPs can be divided into N models. The model structures of multiple individual model MPs may be identical or different. Furthermore, the division into stages (step divisions) in the input / output problem can be designed as appropriate. For example, the loading and unloading problem can be divided into steps for each stage of unloading, or it can be divided into steps for loading and unloading separately.

[0054] The overall model MS is a model constructed using a neural network. The overall model MS can receive the output of each individual model MP as input. Specifically, it is preferable that the information input to the overall model MS includes information obtained from the hidden layers of the neural network that constitutes the individual model MP, and information obtained from the final layer of the neural network. The overall model MS then outputs the final result corresponding to the input / output problem. That is, the overall model MS outputs the final plan for the input / output problem, including steps 1 through N.

[0055] Specifically, the learning models, including the individual model MP and the overall model MS, can accept input data related to the planning of loading and unloading multiple types of raw materials to and from the tank base 21. For example, the input data may include at least one of the following: initial inventory information, loading and unloading plan information, and constraint information.

[0056] The initial inventory information shows the inventory status of each tank 24 in the initial state of the loading and unloading problem. The inventory status is the type and quantity of raw materials stored in each tank 24. In other words, the initial inventory information is information that associates tank 24 (name, identification information, etc.) with the type of raw materials stored and the quantity (weight or percentage) stored.

[0057] The loading and unloading plan information is the overall plan for loading and unloading included in the loading and unloading problem. Figure 7 shows an example of loading and unloading plan information. The loading and unloading plan information associates attributes, dates (or sequences), and loading or unloading quantities for each type of raw material. Attributes indicate loading or unloading. Loading or unloading quantities for each type of raw material indicate the amount of raw material loaded into or unloaded from the tank base 21. In other words, the loading and unloading plan information does not limit individual loading or unloading to tank 24, but shows the loading and unloading plan for the tank base 21. In Figure 7, loading or unloading quantities for each type of raw material are shown by the quantity of each type of raw material corresponding to the attribute (loading or unloading). The example in Figure 7 shows a case where six types of raw materials, raw material G1, raw material G2, raw material G3, raw material G4, raw material G5, and raw material G6, are handled in the loading and unloading problem. Furthermore, the loading / unloading plan information may be associated with the identification information (name, etc.) of the vessel involved in the loading or unloading.

[0058] The constraint information indicates the constraints in the loading and unloading problem. For example, the constraint information includes at least one of the following: upper and lower limit constraints on the inventory in tank 24, constraints on the number of tanks 24 used during loading and unloading, minimum quantities for loading and unloading, concentration constraints, and total tank separation constraints. The concentration constraint is a constraint on the combination and proportion (concentration) of multiple types of raw materials stored in one tank 24. The total tank separation constraint is information that restricts the combination and order in which multiple types of raw materials are stored to prevent separation of multiple types of raw materials within tank 24. The constraint information may also include upper and lower limit constraints on the API (American Petroleum Institute) specific gravity of tank 24. Here, API specific gravity refers to the specific gravity of crude oil as defined by the American Petroleum Institute. API specific gravity is a value that can be measured, for example, in accordance with ASTM D1298. In this disclosure, API specific gravity may be simply referred to as "API".

[0059] Furthermore, the input to the learning model may be preprocessed so that it can be input to the learning model (neural network) in accordance with the information described above.

[0060] Furthermore, the learning model can output output data related to the loading and unloading plans for each of the multiple tanks 24 located at the tank base 21.

[0061] For example, the output data includes information that associates tank selection information, raw material quantity information, and type information. The tank selection information specifies the tank 24 to be operated on regarding loading or unloading. That is, the tank 24 to be loaded into or unloaded is selected in the tank selection information. The raw material quantity information is information that indicates the amount of raw material to be loaded into or unloaded from the selected tank 24, corresponding to the selected tank 24. The type information is information that indicates the type of raw material to be loaded into or unloaded from the selected tank 24, corresponding to the selected tank 24. For example, the output data is shown as loading 100 kl of raw material G3 into tank 24a. If the loading / unloading problem involves multiple steps (loading or unloading), the tank selection information, raw material quantity information, and type information are output corresponding to each step.

[0062] Furthermore, the output data may include evaluation indicators corresponding to the output tank selection information, raw material quantity information, and type information. Examples of evaluation indicators include raw material agreement rate, raw material group agreement rate, and API error. The raw material agreement rate is the ratio of the planned quantity (actual quantity) of raw material to the required quantity (ideal quantity) for each type. For example, in the shipment of raw materials, the raw material agreement rate is the ratio of the amount of raw material to be shipped to the amount of raw material required for shipment. When calculating the raw material agreement rate for multiple types, for example, the average of the raw material agreement rates for each type of raw material is used. The raw material group agreement rate is the ratio of the planned quantity (actual quantity) of raw material to the required quantity (ideal quantity) for each group. A group is, for example, a group of multiple types of raw materials with similar properties. For example, in the shipment of raw materials, the raw material group agreement rate is the ratio of the total amount of raw material from the same group to be shipped to the total amount of raw material from that group required for shipment. When calculating the raw material group agreement rate for multiple groups, for example, the average of the raw material agreement rates for each group is used. API error is the difference between the API of the required raw materials (e.g., the average API of various raw materials) and the API of the planned raw materials (e.g., the average API of various raw materials). For example, in the shipment of raw materials, the API error represents the difference between the API of the raw materials required for shipment and the API of the raw materials planned for shipment.

[0063] Furthermore, the output of the trained model may be subjected to post-processing.

[0064] Thus, the simulation unit 60 is capable of performing simulations related to the loading and unloading problem using the environment E1, agent A1, etc.

[0065] Returning to Figure 5, the first observation unit 61 observes the current state in environment E1 as the first state. That is, before agent A1 takes action, the first observation unit 61 observes the state of the tank base 21 as environment E1 as the first state. For example, the first state includes, for example, the state of the tanks 24 in the tank base 21 as environment E1. The state of the tanks 24 is information indicating the amount of raw materials stored in each of the tanks 24 and the types of raw materials stored.

[0066] The decision unit 62 determines the action of agent A1 based on policy W1. Specifically, the decision unit 62 determines the action of agent A1 based on policy W1 corresponding to the first state. Policy W1 is information of a probability distribution P1 that associates multiple actions that agent A1 can perform corresponding to the first state with their probabilities (selection probabilities). The decision unit 62 selects one action from the multiple actions according to the probability distribution P1. For example, the decision unit 62 probabilistically selects one action according to the probability distribution P1.

[0067] The restriction unit 63 sets restrictions on the action selection based on policy W1 in the decision unit 62. Specifically, the restriction unit 63 sets restrictions on each of the multiple actions in the probability distribution P1 of policy W1. The restrictions, for example, prohibit the decision unit 62 from selecting some of the actions. Figure 8 shows an example of when restrictions are set for each action of policy W1. Figure 8 shows an example of the probability distribution P1. The example in Figure 8 shows a case where probabilities are set for each of the actions AC1, AC2, AC3, AC4, AC5, AC6, AC7, AC8, AC9, and AC10. Based on predetermined constraints, the restriction unit 63 sets a mask as a restriction so that the decision unit 62 does not decide on an action for agent A1 that is judged to be an inappropriate action for agent A1 corresponding to the first state. Figure 8 shows an example where the decision unit 62 is prohibited from selecting actions AC1, AC2, AC3, AC4, AC9, and AC10 as actions for agent A1. In this case, the decision unit 62 selects one action (for example, action AC6) from among the multiple actions included in policy W1 that are not restricted (actions AC5, AC6, AC7, and AC8), and decides it to be the action for agent A1.

[0068] Figure 9 shows an example of constraints used in the limiting unit 63. For example, the constraints are upper and lower limits on the inventory of tank 24, upper and lower limits on the API of the inventory of tank 24, upper limit limit on the number of tanks 24 used during unloading, upper limit limit on the number of tanks 24 used during loading, lower limit limit on the amount loaded per tank, lower limit limit on the amount unloaded per tank, upper limit limit on the amount transported during shifts between tanks, lower limit limit on the amount transported during shifts between tanks, upper limit limit on the number of shifts between tanks per predetermined period (e.g., 1 month), and upper limit limit on the number of shifts between tanks per predetermined period (e.g., 1 day). A shift is an operation to move raw materials from one tank 24 to another tank 24. The limiting unit 63 may use at least one of the above constraints. In addition, each constraint is assigned a type of action to which it applies. The types of actions are the selection of tank 24 during unloading, the amount of raw materials during unloading, the selection of tank 24 during loading, and the amount of raw materials during loading. Furthermore, the types of actions correspond to the shifts and include whether or not to use the tank, the selection of the tank 24 to unload from, the selection of the tank 24 to load into, and the amount of raw material. Figure 9 is a diagram illustrating an example of the correspondence between constraints and the types of actions to which those constraints can be applied. In Figure 9, "○" indicates that a particular constraint is applicable to a particular action. In addition, each constraint may or may not be assigned a priority. In the example in Figure 9, an example is shown where three levels of priority (high, medium, low) are set. In the example in Figure 9, "high" is the highest priority, "medium" is the next highest, and "low" is the next highest. More specifically, if there are multiple constraints imposed on the same action, the constraint with high priority is applied preferentially over the constraints with medium priority and the constraints with low priority. Also, if there are multiple constraints imposed on the same action, the constraint with medium priority is applied preferentially over the constraints with low priority.Furthermore, "preferential application" may mean, for example, when multiple constraints are imposed on the same action, adjusting the degree of influence of each constraint on the action using a weighting according to the priority order of those constraints; or it may mean that only the constraint with the highest priority among those multiple constraints is applied to the action; or it may mean that the constraint is applied to the action in such a way that the constraint with the highest priority is always satisfied, while the constraints of other priorities are satisfied as much as possible. Furthermore, the number of priority levels is not limited to three, and can be changed as appropriate by the user, for example, to two levels. Furthermore, the priority of each constraint can be changed as appropriate by the user.

[0069] In this way, the restriction unit 63 sets restrictions on multiple actions before the action decision is made. In this embodiment, the case in which the restriction unit 63 is provided in the learning unit 51 is given as an example, but the restriction unit 63 may be omitted. Furthermore, the constraints used by the restriction unit 63 are not limited to the above example in the loading and unloading problem and may be changed by the user as appropriate. In addition, when applying a problem other than the loading and unloading problem to the planning creation system 1, the constraints used by the restriction unit 63 may be set as appropriate constraints that are effective for that problem. Furthermore, in this case, when multiple constraints are set, each of the multiple constraints may be set as an appropriate priority.

[0070] Returning to Figure 5, the second observation unit 64 observes the state of environment E1 as the second state, which has changed as a result of the action determined by the decision unit 62. The second observation unit 64 also observes the reward for the action determined by the decision unit 62 as the first reward.

[0071] The first acquisition unit 65 acquires the information output by each individual model MP. The information output by each individual model MP is referred to as "first information".

[0072] The first calculation unit 66 calculates the loss corresponding to each individual model MP. The loss corresponding to each individual model MP is referred to as the "first loss." Specifically, the first calculation unit 66 calculates the first loss corresponding to the first information based on the loss function corresponding to each individual model MP. The loss function corresponding to each individual model MP can be a variety of methods, such as the mean squared difference, cross-entropy error, mean absolute error, mean squared logarithmic error, hinge loss, etc. The specific configuration of the loss function is not limited.

[0073] The second acquisition unit 67 acquires information output from the overall model MS. The information output from the overall model MS is referred to as "second information." In other words, the output of the individual model MP is input to the overall model MS, and the information output from the overall model MS becomes the second information.

[0074] The second calculation unit 68 calculates a loss corresponding to the overall model MS. This loss corresponding to the overall model MS is referred to as the "second loss." Specifically, the second calculation unit 68 calculates a second loss corresponding to the second information based on the loss function corresponding to the overall model MS. The loss function corresponding to the overall model MS can be configured using various methods, such as the mean squared difference, cross-entropy error, mean absolute error, mean squared logarithmic error, hinge loss, etc. The specific configuration of the loss function is not limited.

[0075] The third calculation unit 69 calculates a combined loss by combining the first loss corresponding to each individual model MP and the second loss corresponding to the overall model MS. The combined loss of the first and second losses is called the "combined loss". For example, the third calculation unit 69 calculates the combined loss by weighting multiple first losses and second losses and adding them together. The method for calculating the combined loss is not limited. For example, the first and second losses may be converted into a combined loss using a predetermined function. Alternatively, the combined loss may be the average value of the first and second losses.

[0076] The first update unit 71 updates each individual model MP and the overall model MS based on the combined loss. Since the combined loss includes the first loss, which is the loss of the individual model MP, it can also be said that the first update unit 71 updates the overall model MS based on the first loss of the individual model MP. The first update unit 71 updates each individual model MP and the overall model MS by backpropagation using the combined loss. Note that the method of updating each model is not limited to backpropagation, and other methods may be used. Specifically, the first update unit 71 updates each individual model MP and the overall model MS so that the combined loss becomes smaller (in the direction of decreasing it). More specifically, the first update unit 71 updates each individual model MP and the overall model MS so that the value of the combined loss after the update is smaller than the value of the combined loss before the update. For example, the first update unit 71 updates each individual model MP and the overall model MS so that the combined loss is minimized. As a result, the individual model MP and the overall model MS are updated so that a policy W1 that reduces the combined loss can be output.

[0077] The second update unit 72 updates each individual model MP based on each of the first losses. The second update unit 72 updates each individual model MP by backpropagation using the first losses. Note that the method for updating each individual model MP is not limited to backpropagation, and other methods may be used. Specifically, the second update unit 72 updates each individual model MP so that the first loss becomes smaller (in the direction of decreasing it). More specifically, the second update unit 72 updates each individual model MP so that the value of the first loss after the update is smaller than the value of the first loss before the update. For example, the second update unit 72 updates each individual model MP so that the first loss is minimized.

[0078] The third update unit 73 updates the overall model MS based on the second loss. The third update unit 73 updates the overall model MS by backpropagation using the second loss. Note that the method for updating the overall model MS is not limited to backpropagation, and other methods may be used. Specifically, the third update unit 73 updates the overall model MS so that the second loss becomes smaller (in the direction of decreasing it). More specifically, the third update unit 73 updates the overall model MS so that the value of the second loss after the update is smaller than the value of the second loss before the update. For example, the third update unit 73 updates the overall model MS so that the second loss is minimized.

[0079] In this manner, the individual model MP and the overall model MS are updated by the first update unit 71 using the combined loss, and the second update unit 72 and the third update unit 73 are updated using the losses of their respective models (first loss and second loss).

[0080] In this way, deep reinforcement learning is performed in the learning unit 51.

[0081] The inference unit 52 performs inference using a trained model that includes the trained individual model MP and the overall model MS. Inference is the process of obtaining output data corresponding to the input data using the trained model. Figure 10 shows an overview of the input and output by the trained model. The inference unit 52 receives input data from the trained model regarding the plans for loading and unloading multiple types of raw materials to and from the tank base 21. The input data includes, for example, initial inventory information, loading and unloading plan information, and constraint information. For example, each input data is preprocessed so that it can be input into the trained model. The inference unit 52 then obtains output data as an inference result regarding the plans for loading and unloading each raw material corresponding to each of the multiple tanks 24 that the tank base 21 has. The output data includes, for example, information that associates tank selection information, raw material quantity information, and type information. For example, each output data is postprocessed so that it can be output by the trained model.

[0082] <Processing flow> Figure 11 is a flowchart showing an example of the learning process flow according to this embodiment. Each of the following processes is started, for example, according to a user's instruction to start learning. It is assumed that N individual model MPs are set. The order and content of each of the following steps can be changed as appropriate.

[0083] (Step SP10) The simulation unit 60 initializes the model related to the loading and unloading problem. For example, the state of the environment E1, the weights of the N individual models MP and the neural networks of the overall model MS are initialized. In other words, each parameter related to the loading and unloading problem is set to its initial state. Alternatively, each hyperparameter related to learning may be set. Then, the process moves to step SP11.

[0084] (Step SP11) The first acquisition unit 65 acquires first information as the output result (inference result) of each individual model MP. That is, first information corresponding to each of the N items is acquired. Then, the process moves on to step SP12.

[0085] (Step SP12) The second acquisition unit 67 acquires the second information as the output result (inference result) of the overall model MS. The input to the overall model MS includes information obtained from the intermediate layers of each individual model MP and information obtained from the final layer. Then, the process moves to step SP13.

[0086] (Step SP13) The first arithmetic unit 66 calculates the first loss corresponding to each of the N first pieces of information. That is, the first loss corresponding to each of the N individual model MPs is calculated. Then the process moves on to step SP14.

[0087] (Step SP14) The second calculation unit 68 calculates a second loss corresponding to the second information. That is, a second loss corresponding to the overall model MS is calculated. Then, the process moves to step SP15.

[0088] (Step SP15) The simulation unit 60 determines whether the final plan output from the overall model MS satisfies predetermined conditions. These predetermined conditions include, for example, that the second loss is within a predetermined value. The learning termination conditions can be set as appropriate. If the final plan output from the overall model MS satisfies the predetermined conditions, the process terminates. If the final plan output from the overall model MS does not satisfy the predetermined conditions, the process proceeds to step SP16.

[0089] (Step SP16) The third arithmetic unit 69 calculates a combined loss by combining the first loss corresponding to each individual model MP and the second loss corresponding to the overall model MS. Then, the process moves to step SP17.

[0090] (Step SP17) The first update unit 71 updates each individual model MP and the overall model MS using the combined loss. Then, the process moves on to step SP18.

[0091] (Step SP18) The second update unit 72 updates each individual model MP using the first loss. Then, the process moves to step SP19.

[0092] (Step SP19) The third update unit 73 updates the overall model MS using the second loss. Then, the process returns to step SP11, and each process is executed again.

[0093] In this way, deep reinforcement learning is performed. That is, a trained model is generated by training (updating) a neural network, which is an example of a learning model. In particular, the model is updated using individual losses (first loss and second loss), as well as a combined loss (combined loss) which is a combination of the individual losses. Note that the order of each process in the above learning process may be changed, or the processes may be performed in parallel. For example, the process by the first update unit 71 in step SP17, the process by the second update unit 72 in step SP18, and the process by the third update unit 73 in step SP19 may be processed in parallel. Specifically, the training of individual models MP using the combined loss and the training of individual models MP using the first loss may be performed sequentially or in parallel. Parallel processing can shorten the processing time. Also, the training of the overall model MS using the combined loss and the training of the overall model MS using the second loss may be performed sequentially or in parallel. Parallel processing can shorten the processing time. Furthermore, the training of individual models MP using the combined loss, the training of the overall model MS using the combined loss, the training of individual models MP using the first loss, and the training of the overall model MS using the second loss can be performed sequentially or in parallel in any combination. Thus, the order and combination of training using each loss are not limited.

[0094] Figure 12 is a flowchart showing an example of the inference process flow according to this embodiment. Each of the following processes is started, for example, in accordance with a user's instruction to start inference. Note that the order and content of each of the following steps can be changed as appropriate.

[0095] (Step SP30) The inference unit 52 acquires the trained model, which includes the trained individual models MP and the overall model MS, as the trained model. Then, the process moves on to step SP31.

[0096] (Step SP31) The inference unit 52 inputs input data to the trained model. For example, the input data is preprocessed before being input to the trained model. Then, the process moves on to step SP32.

[0097] (Step SP32) The inference unit 52 obtains output data from the trained model. For example, post-processing may be performed on the output data output from the trained model. Then the processing is completed.

[0098] <Effects and Effects> As described above, the server device 2 according to this embodiment is a machine learning device that performs learning on a plurality of individual model MPs that output intermediate results of a problem corresponding to each stage in a problem involving multiple stages, and an overall model MS that takes the output of each individual model MP as input and outputs the final result of the problem, and comprises a first acquisition unit 65 that acquires the information output by each individual model MP as first information, a first calculation unit 66 that calculates the loss corresponding to each piece of first information as the first loss based on the loss function corresponding to each individual model MP, and a first update unit 71 that updates the overall model MS based on the first loss.

[0099] In this configuration, the output of individual model MP is input to the overall model MS, and the overall model MS outputs the final result of the problem. The overall model MS is updated by the first loss of the individual model MP. This allows the overall model MS to learn while considering the output and first loss of the individual model MP corresponding to each stage that makes up the problem. In other words, the overall model MS can learn to output the final result of the problem while balancing each stage that makes up the problem. The overall model MS can learn the state of each stage step by step as it learns. As a result, the overall model MS can appropriately advance curriculum learning corresponding to the problem and efficiently learn the final result for the entire problem. In other words, it is possible to improve the efficiency and accuracy of learning for problems that include multiple stages.

[0100] Furthermore, the server device 2 includes a second acquisition unit 67 that takes information obtained from each individual model MP as input and acquires information output from the overall model MS as second information, a second calculation unit 68 that calculates the loss corresponding to the second information as the second loss based on the loss function corresponding to the overall model MS, and a third calculation unit 69 that calculates a combined loss by combining each first loss and the second loss, and the first update unit 71 updates each individual model MP and the overall model MS based on the combined loss.

[0101] With this configuration, the individual model MP and the overall model MS are updated using the combined loss, allowing the overall model MS to learn while considering the loss of the individual model MP, and the individual model MP to learn while considering the loss of the overall model MS.

[0102] Furthermore, in server device 2, each individual model MP is a model that corresponds to different stages of the problem, from the initial stage to subsequent stages.

[0103] With this configuration, since each individual model MP corresponds to a specific stage, the overall model MS can be updated while taking into account the losses at each stage.

[0104] Furthermore, in server device 2, the information input to the overall model MS includes information obtained from the intermediate layers of the neural network constituting the individual model MP, and information obtained from the final layer of the neural network.

[0105] With this configuration, the overall model MS can update the state of individual model MPs in more detail by using information from the intermediate layers as well as the final layer information of the individual model MPs.

[0106] Furthermore, in the server device 2, the first update unit 71 updates each individual model MP and the overall model MS in such a way that the combined loss is reduced.

[0107] This configuration allows for updating both the individual model MP and the overall model MS to minimize losses incurred in both models.

[0108] Furthermore, the server device 2 also includes a second update unit 72 that updates each individual model MP based on each of the first losses.

[0109] This configuration allows learning to proceed using the losses generated in individual MP models. In other words, by performing learning using both the combined loss and the first loss, it becomes possible to perform learning while considering both the overall loss (combined loss) and the individual losses (first loss). This makes it possible to improve the learning accuracy for problems involving multiple stages.

[0110] Furthermore, the server device 2 also includes a third update unit 73 that updates the overall model MS based on the second loss.

[0111] This configuration allows learning to proceed using the loss generated in the overall model MS. In other words, by performing learning using the combined loss and learning using the second loss, it becomes possible to perform learning while considering both the overall loss (combined loss) and the individual losses (second loss). This makes it possible to improve the learning accuracy for problems involving multiple stages.

[0112] Furthermore, in the server device 2, the second update unit 72 updates each individual model MP to reduce the first loss, and the third update unit 73 updates the overall model MS to reduce the second loss.

[0113] With this configuration, both the individual MP model and the overall MS model can be trained to minimize their own losses.

[0114] Furthermore, in the server device 2, the learning model, which includes the individual model MP and the overall model MS, takes data relating to at least one of the plans for the loading and unloading of multiple types of raw materials to and from the tank base 21 as input, and outputs data relating to at least one of the plans for the loading and unloading of each raw material corresponding to each of the multiple tanks 24 that the tank base 21 has.

[0115] This configuration allows for the training of a learning model that can handle the problem of raw material loading and unloading.

[0116] Furthermore, in the server device 2, the first update unit 71 updates each individual model MP and the overall model MS by backpropagation using the combined loss.

[0117] This configuration allows for updating both the individual MP model and the overall MS model to minimize the combined loss.

[0118] Furthermore, server device 2 uses the learned individual models MP and the overall model MS to infer output data corresponding to the input data.

[0119] This configuration allows the trained model to perform inferences, for example, to address loading and unloading problems.

[0120] <Variation> This disclosure is not limited to the embodiments described above. That is, any modifications made to the embodiments described above by a person skilled in the art are also included in the scope of this disclosure, as long as they retain the features of this disclosure. Furthermore, the elements of the embodiments described above and the modifications described later can be combined to the extent that it is technically possible, and any combination thereof is also included in the scope of this disclosure, as long as it retains the features of this disclosure.

[0121] In the above embodiment, one example is that each function is provided by the server device 2, but each function may also be provided by the user terminal 3. Alternatively, each function may be distributed between the server device 2 and the user terminal 3. For example, the user terminal 3 may function as both a machine learning device and an inference device. Alternatively, one of the server device 2 and the user terminal 3 may function as a machine learning device, and the other of the server device 2 and the user terminal 3 may function as an inference device. Alternatively, two server devices 2 that can communicate with each other may be installed, with one server device 2 functioning as a machine learning device and the other server device 2 functioning as an inference device. Alternatively, multiple server devices 2 that can communicate with each other may be installed, with each function constituting the machine learning device and each function constituting the inference device distributed among multiple server devices 2, and these may function together as a machine learning device or an inference device.

[0122] Furthermore, while the above embodiment described the application of the problem of loading and unloading raw materials at the tank base 21 to the planning system 1 as an example, the problems that can be applied to the planning system 1 are not limited to those described above. For example, the problem of vehicle entry and exit at a parking facility may be applied to the planning system 1. Also, the problem of ship entry and departure at a port may be applied to the planning system 1. Also, the problem of receiving and shipping goods and merchandise at a warehouse or store may be applied to the planning system 1. It should be noted that the problems that can be applied to the planning system 1 are not limited to those described above, and a variety of problems can be applied.

[0123] The various types of information described in this disclosure (e.g., status, reward, etc.) may be expressed using absolute values, relative values ​​from a given value, or corresponding other information.

[0124] In addition, in the above-described embodiment, when the problem has steps from the first step to the Nth step, the case where N individual models MP are provided is taken as an example, but it is not limited to the above. For example, some of the steps from the first step to the Nth step may be grouped together (regarded as steps from the first step to the Mth step), and M (<N) individual models MP may be provided.

[0125] In addition, in the above-described embodiment, the case where both learning using the combined loss and learning using the losses of the respective models (the first loss and the second loss) are performed is taken as an example, but the learning method is not limited to the above. For example, learning using the combined loss may be performed, and learning using the losses of the respective models (the first loss and the second loss) may be omitted.

[0126] In addition, in the above-described embodiment, the case where the overall model MS is updated by the combined loss is taken as an example, but the overall model MS may be updated by the first loss of the individual model MP. For example, the overall model MS may be updated by the loss obtained by combining the first losses of the individual models MP or by a part of the first losses of the individual models MP.

[0127] In addition, in the above-described embodiment, the first update unit 71 updates the model so that the combined loss is minimized as an example, but it is not limited to the case of minimization. For example, the update is not limited to the case of minimization as long as it is performed so that the combined loss becomes smaller, and it may be performed so that the combined loss becomes less than or equal to a predetermined criterion. The same applies to the second update unit 72 and the third update unit 73, and the update process is not limited to the case where the loss is minimized.

[0128] Expressions such as "based on", "using", "by", etc. (including equivalent expressions) in the present disclosure do not mean "only based on", "only using", "only by", etc. unless otherwise specified. In other words, the description "based on" means both "only based on" and "at least based on", and the same applies to equivalent expressions such as "using" and "by".

[0129] The term “decision” in this disclosure may encompass a wide variety of actions. “Decision” may include, for example, judgment, calculation, calculation, processing, derivation, investigation, exploration, and confirmation. Furthermore, “decision” may include, for example, considering something to have been “decided,” such as resolving, selecting, choosing, establishing, or comparing. In short, “decision” may include considering any action to have been “decided.”

[0130] In this disclosure, where expressions such as "obtain / set / as input / using / based on" (including similar expressions) are used, unless otherwise specified, this includes cases where the information itself is used, or where the information has been processed in some way (e.g., noise-added, normalized, features extracted from the information, intermediate representation of the information, etc.) is used. Furthermore, where it is stated that some result is obtained by "obtaining / setting / as input / using / based on" (including similar expressions), unless otherwise specified, this includes cases where the result is obtained solely based on the information in question, or where the result is influenced by other information, factors, conditions, and / or states other than the information in question. Furthermore, where it is stated that "outputs" (including similar expressions), unless otherwise specified, this includes cases where the information itself is used as output, or where the information has been processed in some way (e.g., noise-added, normalized, features extracted from the information, intermediate representation of various types of information, etc.) is used as output. [Explanation of Symbols]

[0131] 2: Server device (machine learning device) 65:First Acquisition Department 66:First calculation section 71:First update part MP: Individual Model MS: Overall Model

Claims

1. A machine learning device that learns a problem involving multiple stages, comprising a plurality of individual models that output intermediate results for each stage of the problem, and an overall model that takes the output of each individual model as input and outputs the final result of the problem, A first acquisition unit acquires the information output in each of the aforementioned individual models as primary information, A first calculation unit calculates the loss corresponding to each of the first pieces of information as the first loss based on the loss function corresponding to each of the individual models, A first update unit updates the overall model based on the first loss, A machine learning device equipped with the following features.

2. A second acquisition unit takes information obtained from each of the individual models as input and acquires information output from the overall model as second information, A second calculation unit calculates the loss corresponding to the second information as the second loss based on the loss function corresponding to the overall model, A third calculation unit calculates a combined loss obtained by combining each of the first losses and the second losses, Furthermore, The first update unit updates each of the individual models and the overall model based on the combined loss. The machine learning apparatus according to claim 1.

3. Each of the aforementioned individual models is a model that corresponds to the initial stage of the problem and to different stages thereafter. The machine learning apparatus according to claim 1.

4. The information input to the overall model includes information obtained from the intermediate layers of the neural networks constituting the individual models and information obtained from the final layer of the neural networks. The machine learning apparatus according to claim 1.

5. The first update unit updates each of the individual models and the overall model so that the combined loss is reduced. The machine learning apparatus according to claim 2.

6. A second update unit updates each of the individual models based on each of the first losses. The machine learning apparatus according to claim 2, further comprising:

7. A third update unit updates the overall model based on the second loss. The machine learning apparatus according to claim 6, further comprising:

8. The second update unit updates each of the individual models so that the first loss is reduced. The third update unit updates the overall model so that the second loss is reduced. The machine learning apparatus according to claim 7.

9. The learning model, which includes the individual model and the overall model, takes data relating to at least one of the plans for loading and unloading multiple types of raw materials to and from the tank base as input, and outputs data relating to at least one of the plans for loading and unloading each of the multiple tanks owned by the tank base. The machine learning apparatus according to claim 1.

10. The first update unit updates each of the individual models and the overall model by backpropagation using the combined loss. The machine learning apparatus according to claim 2.

11. Using each of the individual models trained by the machine learning device described in any one of claims 1 to 10, and the overall model, output data corresponding to input data is inferred. Reasoning device.

12. A machine learning method for a problem involving multiple stages, which involves training multiple individual models that output intermediate results corresponding to each stage of the problem, and an overall model that takes the output of each individual model as input and outputs the final result of the problem, A step of acquiring the information output in each of the individual models as primary information, A step of calculating the loss corresponding to each of the first pieces of information as the first loss, based on the loss function corresponding to each of the individual models, A step of updating the overall model based on the first loss, A machine learning method that uses [a specific feature].

13. A machine learning program that learns a problem involving multiple stages, comprising a plurality of individual models that output intermediate results corresponding to each stage of the problem, and an overall model that takes the output of each individual model as input and outputs the final result of the problem, Computers, A first acquisition unit acquires the information output in each of the aforementioned individual models as primary information. A first calculation unit calculates the loss corresponding to each of the first pieces of information as the first loss, based on the loss function corresponding to each of the individual models. Based on the first loss, the first update unit updates the overall model. A machine learning program that functions as such.

14. A method for generating a trained model in a problem involving multiple stages, comprising training a plurality of individual models that output intermediate results for each stage of the problem, and an overall model that takes the output of each individual model as input and outputs the final result of the problem, A step of acquiring the information output in each of the individual models as primary information, A step of calculating the loss corresponding to each of the first pieces of information as the first loss, based on the loss function corresponding to each of the individual models, A step of updating the overall model based on the first loss, A method for generating a trained model having [a certain characteristic].

Images