Delivery plan creation device and delivery plan creation system

Abstract

To provide a delivery plan generation apparatus configured to generate an efficient delivery route so as to suppress reduction in sales of commodities in a vending machine due to sold out, while suppressing increase in delivery time.SOLUTION: A delivery plan generation apparatus 101 includes a delivery plan generation unit 23 and an output unit 4. The delivery plan generation apparatus 101 generates a delivery plan indicating a delivery route of commodities to multiple vending machines V using a trained model of the delivery plan generation unit 23, on the basis of forecasted demand which is a forecasted commodity sales value and inventory information of the commodities. The delivery plan generation apparatus 101 updates the trained model of the delivery plan generation unit by machine learning using the trained model of the delivery plan generation unit, on the basis of delivery actual result information obtained when the delivery plan is actually implemented. The output unit 4 outputs the delivery plan generated using the updated trained model of the delivery plan generation unit 23 to a portable terminal 102 over a network 200.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a delivery plan creation device and a delivery plan creation system, and more particularly to a delivery plan creation device and a delivery plan creation system that create a delivery plan indicating product delivery routes to a plurality of vending machines. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a vending machine management system is known (see, for example, Patent Document 1).

[0003] The above-mentioned Patent Document 1 discloses a vending machine management system for vending machines that sell products in response to electronic money. In this vending machine management system, electronic money data is collected from the vending machines by a terminal carried by a route man who visits multiple vending machines. In addition, in conventional vending machines such as those described in the above-mentioned Patent Document 1, an operator (route man) performs tasks such as replenishing products and collecting money. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-301402 A Summary of the Invention [Problem to be solved by the invention]

[0005] Here, in vending machines, there are cases where products are sold out, and sales are lost due to lost sales opportunities. For this reason, a worker (route man) who delivers (replenishes) products to vending machines by visiting multiple vending machines as needed or traveling along a predetermined delivery route performs work such as replenishing products to vending machines. However, there is a problem in that the patrol route and the predetermined delivery route are not necessarily efficient delivery routes that can suppress an increase in delivery time during actual delivery while suppressing a decrease in sales of vending machine products caused by sold-out products.

[0006] The present invention has been made to solve the problems described above, and one object of the present invention is to provide a delivery plan creation device and a delivery plan creation system that are capable of creating efficient delivery routes that suppress increases in delivery time while suppressing declines in sales of products in vending machines due to products being sold out. [Means for solving the problem]

[0007] In order to achieve the above object, a delivery plan creation device according to a first aspect of the present invention includes a delivery plan creation unit that creates a delivery plan indicating a delivery route for products to a plurality of vending machines by machine learning, and an output unit that outputs the delivery plan created by the delivery plan creation unit via a network to a mobile terminal carried by an operator who performs the task of replenishing products in the vending machines, and outputs a forecast demand, which is a forecast value of sales of products in the plurality of vending machines, and inventory information of each product in the plurality of vending machines, to the delivery plan creation unit. Represents the value gained by forecasting demand and inventory information as states and by planning delivery as actions. The system is configured to create a delivery plan using a trained model, and when the delivery plan is actually implemented, the sales of products from multiple vending machines and the total travel time required to deliver the products are recorded. Inventory information for each product in multiple vending machines Delivery record information including Using Delivery Planning Department Regarding the value of the trained model's delivery plan, Pre-trained model reinforcementThe delivery plan creation unit is configured to update the trained model of the delivery plan creation unit by performing learning, and the output unit is configured to output the delivery plan created by the updated trained model of the delivery plan creation unit to a mobile terminal via a network.

[0008] In the delivery plan creation device according to the first aspect, as described above, a delivery plan is created by the trained model of the delivery plan creation unit based on the predicted demand, which is a predicted value of sales of products from a plurality of vending machines, and inventory information of each product from a plurality of vending machines. This makes it possible to create a delivery plan that can suppress a decrease in sales of products caused by selling out. In addition, the trained model of the delivery plan creation unit is updated by performing machine learning using the trained model of the delivery plan creation unit based on delivery performance information when the delivery plan is actually implemented. This makes it possible to update the trained model of the delivery plan creation unit to a trained model that reflects delivery performance information including sales of products from a plurality of vending machines and the total travel time required to deliver the products when the delivery plan is actually implemented. As a result, the delivery plan creation unit can create an efficient delivery plan that reflects delivery performance information including sales of products from a plurality of vending machines and the total travel time required to deliver the products when the delivery plan is actually implemented, at the time of delivery after the trained model is updated. As a result, it is possible to create an efficient delivery route that suppresses an increase in delivery time while suppressing a decrease in sales of products from vending machines caused by selling out. In addition, since the output unit outputs the delivery plan created by the updated trained model of the delivery plan creation unit to the mobile device via the network, the worker can check, on the mobile device, an efficient delivery plan that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of vending machine products caused by sold-outs. As a result, the worker can perform efficient product delivery that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of vending machine products caused by sold-outs.

[0009] In the delivery plan creation device according to the first aspect, preferably, the trained model of the delivery plan creation unit includes a state action value function that represents a value obtained by a delivery plan as an action for predicted demand and inventory information as a state, and the delivery plan creation unit is configured to create a delivery plan based on the state action value function, and is configured to update the state action value function of the trained model by reinforcement learning using the trained model of the delivery plan creation unit based on delivery performance information. With this configuration, a delivery plan that reflects the delivery performance information, has a short delivery time, and has high sales of products from vending machines, can be created by the state action value function updated based on the delivery performance information. As a result, the delivery plan creation unit can create an efficient delivery route that can suppress an increase in delivery time when delivering products to multiple vending machines and suppress a decrease in sales of products from vending machines due to sold-outs.

[0010] In the delivery plan creation device according to the first aspect, the device is preferably configured to update the trained model of the delivery plan creation unit by machine learning using the trained model of the delivery plan creation unit based on delivery record information after delivery of products to the multiple vending machines is completed. With this configuration, the trained model of the delivery plan creation unit can be updated to reflect the delivery record information from the start of delivery of products to the multiple vending machines to the completion of delivery. As a result, the delivery plan creation unit after updating the trained model can create a more efficient delivery plan (delivery route) that can further suppress an increase in delivery time when delivering products to the multiple vending machines and further suppress a decrease in sales of products in the vending machines due to sold-outs.

[0011] In the delivery plan creation device according to the first aspect, preferably, delivery record information including the location information of the worker when the worker delivers the product is acquired, and the trained model of the delivery plan creation unit is configured to be updated by machine learning using the trained model based on the delivery record information including the acquired location information. With this configuration, the travel time between the multiple vending machines when the worker delivers the product can be acquired from the delivery record information including the location information of the worker. As a result, if the travel time between the multiple vending machines in the actual delivery of the product is longer than the travel time assumed when the delivery plan was created, the trained model of the delivery plan creation unit can be updated based on the travel time between the multiple vending machines acquired from the delivery record information including the location information of the worker. This makes it possible to create a more optimal delivery route by changing the route between the multiple vending machines or the order of the vending machines to be delivered, and therefore to create a more efficient delivery plan.

[0012] In the delivery plan creation device according to the first aspect, the delivery plan creation unit is preferably configured to use predicted demand, which is a predicted value of sales of products from multiple vending machines over multiple days, in creating the delivery plan. With this configuration, it is possible to create a delivery route that has a short total travel time or a short total travel distance required for delivering products over multiple days. As a result, the delivery plan creation unit can create an efficient delivery route for products to multiple vending machines when delivering products over multiple days.

[0013] In the delivery plan creation device according to the first aspect, preferably, when the worker delivers the goods, delivery record information when the delivery plan is actually implemented is acquired, and the delivery plan creation unit is configured to output a delivery plan using delivery record information including inventory information acquired when the worker delivers the goods and forecast demand, which is a forecast value of sales of goods of the multiple vending machines, as inputs, and is configured to correct the delivery plan based on the acquired delivery record information without updating the trained model of the delivery plan creation unit. With this configuration, the delivery plan creation unit can correct the delivery plan based on actual inventory information before updating the trained model. As a result, the delivery plan can be corrected to a more efficient delivery plan by reflecting actual inventory information before updating the trained model. In addition, by correcting the delivery plan without updating the trained model of the delivery plan creation unit, the delivery plan can be easily corrected compared to the case where the trained model of the delivery plan creation unit is updated from time to time.

[0014] In the delivery plan creation device according to the first aspect, preferably, when a worker delivers a product, delivery record information when the delivery plan is actually executed is acquired via a network, and the delivery plan is modified during the delivery of the product by the worker based on the acquired delivery record information without updating the trained model of the delivery plan creation unit, and the modified delivery plan is transmitted via a network to each mobile terminal of the multiple workers. With this configuration, while the worker is delivering the product, the delivery plan modified based on the acquired delivery record information is transmitted to each mobile terminal of the multiple workers, so that the worker can efficiently deliver the product while the product is being delivered, reflecting the delivery situation in real time. In addition, by modifying the delivery plan without updating the trained model of the delivery plan creation unit, the delivery plan can be modified in real time during the delivery of the product more quickly than when the trained model of the delivery plan creation unit is updated from time to time.

[0015] In the delivery plan creation device according to the first aspect, preferably, when a plurality of workers deliver goods, a product transport volume, which is the number of goods that each of the plurality of workers can deliver, is acquired via a network, and the delivery plan is modified during the delivery of goods by the workers based on the acquired product transport volume of each of the plurality of workers without updating the trained model of the delivery plan creation unit, and the modified delivery plan is transmitted via a network to each mobile terminal of the plurality of workers. With this configuration, the delivery plan can be modified based on the acquired product transport volume of each of the plurality of workers while the workers are delivering goods, so that even if a certain worker's product transport volume is insufficient, the delivery route can be modified so that another worker who has a surplus in product transport volume replenishes the goods instead. Then, the modified delivery plan is transmitted to each mobile terminal of the plurality of workers, so that the workers can deliver goods efficiently reflecting the product transport volume of each worker. In addition, by modifying the delivery plan without updating the trained model of the delivery plan creation unit, the delivery plan based on the acquired product transport volume of each of the plurality of workers can be modified during the delivery of goods, compared to the case where the trained model of the delivery plan creation unit is updated from time to time.

[0016] In the delivery plan creation device according to the first aspect, preferably, when creating the trained model of the delivery plan creation unit, the delivery plan creation unit creates a tentative delivery plan based on at least one of the predicted value or past actual value of sales of products of the multiple vending machines and at least one of the predicted value or past actual value of the inventory quantity of each product of the multiple vending machines, obtains predicted delivery performance information including the sales of products in the simulation result of product delivery to the multiple vending machines according to the tentative delivery plan and the total travel time required for product delivery in the simulation result of product delivery to the multiple vending machines, and performs reinforcement learning based on the predicted delivery performance information as machine learning in the delivery plan creation unit to create the trained model. With this configuration, learning by reinforcement learning can be performed before actually delivering products using the predicted delivery performance information obtained by the simulation result based on the tentative delivery plan. As a result, a trained model of the delivery plan creation unit capable of creating a more efficient delivery plan before actually delivering products can be created compared to a case where reinforcement learning is not performed using the predicted delivery performance information obtained by the simulation result based on the tentative delivery plan.

[0017] In order to achieve the above object, a delivery plan creation system according to a second aspect of the present invention includes a mobile terminal carried by an operator who replenishes products in vending machines, and a delivery plan creation device that creates a delivery plan indicating a delivery route for products to a plurality of vending machines, the delivery plan creation device including a delivery plan creation unit that creates a delivery plan by machine learning, and an output unit that outputs the delivery plan created by the delivery plan creation unit to the mobile terminal via a network, and based on a predicted demand, which is a predicted value of sales of products in the plurality of vending machines, and inventory information of each product in the plurality of vending machines, the delivery plan creation unit Represents the value gained by forecasting demand and inventory information as states and by planning delivery as actions. The system is configured to create a delivery plan using a trained model, and when the delivery plan is actually implemented, the sales of products from multiple vending machines and the total travel time required to deliver the products are recorded. Inventory information for each product in multiple vending machines Delivery record information including Using Delivery Planning Department Regarding the value of the trained model's delivery plan, Trained Model reinforcementThe delivery plan creation unit is configured to update the trained model of the delivery plan creation unit by performing learning, and the output unit is configured to output the delivery plan created by the updated trained model of the delivery plan creation unit to a mobile terminal via a network.

[0018] In the delivery plan creation system according to the second aspect, as described above, the delivery plan creation device creates a delivery plan using the trained model of the delivery plan creation unit based on the predicted demand, which is a predicted value of sales of products from multiple vending machines, and inventory information of each product from multiple vending machines. This makes it possible to create a delivery plan that can suppress a decrease in sales of products caused by selling out. In addition, the trained model of the delivery plan creation unit is updated by performing machine learning using the trained model of the delivery plan creation unit based on delivery performance information when the delivery plan is actually implemented. This makes it possible to update the trained model of the delivery plan creation unit to a trained model that reflects delivery performance information including sales of products from multiple vending machines and the total travel time required to deliver the products when the delivery plan is actually implemented. As a result, the delivery plan creation unit can create an efficient delivery plan that reflects delivery performance information including sales of products from multiple vending machines and the total travel time required to deliver the products when the delivery plan is actually implemented, at the time of delivery after the trained model is updated. As a result, it is possible to create an efficient delivery route that suppresses an increase in delivery time while suppressing a decrease in sales of products from vending machines caused by selling out. In addition, since the output unit outputs the delivery plan created by the updated trained model of the delivery plan creation unit to the mobile device via the network, the worker can check, on the mobile device, an efficient delivery plan that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of vending machine products caused by sold-outs. As a result, the worker can perform efficient product delivery that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of vending machine products caused by sold-outs. Effect of the Invention

[0019] According to the present invention, as described above, it is possible to create an efficient delivery route that suppresses an increase in delivery time while suppressing a decrease in sales of products in a vending machine due to sold-out items. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram illustrating an example of a delivery plan creation system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram of a delivery plan creation device according to an embodiment of the present invention. [Diagram 3] FIG. 13 is a diagram showing data input and output when creating a trained model using simulation results. [Figure 4] FIG. 13 is a diagram showing an example of creating a tentative delivery plan. [Diagram 5] FIG. 1 illustrates the generation of a Q function. [Figure 6] FIG. 13 is a diagram showing an example of a simulation result. [Figure 7] FIG. 11 is a diagram showing another example of a simulation result. [Figure 8] FIG. 13 shows an example of simulation results of a tentative distribution plan created by a trained model after reinforcement learning using the simulation results. [Figure 9] FIG. 1 is a first diagram showing an example of a simulation result based on demand information for multiple days. [Figure 10] FIG. 2 is a second diagram showing an example of a simulation result based on demand information for multiple days. [Figure 11] FIG. 13 is a diagram showing an example of creating a delivery plan. [Figure 12] FIG. 1 is a first diagram showing an example of a correction to a delivery plan during delivery of goods. [Figure 13] FIG. 2 is a second diagram showing an example of a correction to a delivery plan during delivery of goods. [Figure 14] FIG. 1 is a first diagram showing another example of correction of a delivery plan during delivery of goods. [Figure 15]FIG. 2 is a second diagram showing another example of correction of a delivery plan during delivery of goods. [Figure 16] This figure shows the input and output of data when updating a trained model after delivery is completed. [Figure 17] FIG. 13 is a diagram showing an update of the Q function after delivery is completed. [Figure 18] Figure 1 shows an example of changes to a delivery plan due to an update to a trained model after delivery is completed. [Figure 19] Figure 2 shows an example of a change in the delivery plan due to an update of the trained model after delivery is completed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0022] (Overall configuration of the delivery plan creation system) The overall configuration of a delivery plan creation system 100 according to this embodiment will be described with reference to FIG.

[0023] As shown in FIG. 1, the delivery plan creation system 100 includes a delivery plan creation device 101 that creates a delivery plan indicating product delivery routes to multiple vending machines V, and a mobile terminal 102 carried by a worker M (route man) who performs the work of replenishing products at the vending machines V.

[0024] The delivery plan creation device 101 notifies (displays) the delivery route on a mobile terminal 102 carried (held) by an operator M (routeman) via a network 200. In addition, delivery record information, which will be described later, is periodically transmitted to the delivery plan creation device 101. The delivery record information may be transmitted by an operation (input) by the operator M, or may be transmitted automatically by the mobile terminal 102. Then, the delivery plan creation device 101 modifies the delivery plan to reflect the received delivery record information, and notifies the mobile terminal 102 of the operator M.

[0025] The delivery plan creation device 101 is a server communicably connected to the mobile terminal 102 and the vending machine V via a network 200. The delivery plan creation device 101 transmits and receives data to and from the mobile terminal 102 via the network 200. The delivery plan creation device 101 acquires delivery record information, which will be described later, via the network 200. Then, the delivery plan creation device 101 transmits the created delivery plan to the mobile terminal 102 via the network 200.

[0026] The delivery plan creation device 101 also transmits and receives data to and from the vending machine V via the network 200. For example, the delivery plan creation device 101 acquires data such as sales (number of items sold and sales amount) and inventory of products of the vending machine V via the network 200. The delivery plan creation system 100 may also be configured to notify the delivery plan creation device 101 via the network 200 that refilling or maintenance work is being performed on the vending machine V, and provide the delivery plan creation device 101 with location information of the worker M based on the state of the vending machine V. FIG. 1 shows an example in which data from the vending machine V is transmitted to the delivery plan creation device 101 via the network 200, but the data from the vending machine V may be transmitted to the delivery plan creation device 101 via a management server (vending machine management server) for the vending machine V. The data from the vending machine V may also be transmitted to the delivery plan creation device 101 via the mobile terminal 102.

[0027] The mobile terminal 102 is a terminal that can be connected to the network 200. For example, the mobile terminal 102 is a terminal such as a smartphone or a tablet PC. An application for performing data communication with the delivery plan creation device 101 and notifying (displaying) a delivery plan is installed in the mobile terminal 102. The mobile terminal 102 is also configured to be able to communicate with the vending machine V, and is capable of transmitting and receiving data to and from the vending machine V.

[0028] Then, worker M delivers and replenishes products to vending machine V according to a delivery plan (delivery route) created by delivery plan creation device 101 and notified to mobile terminal 102. The delivery plan created by delivery plan creation device 101 (delivery plan creation system 100) includes the delivery route of each of multiple workers M in charge of delivery (replenishment) work for multiple vending machines V managed by a user who uses delivery plan creation device 101. In addition, the delivery plan created by delivery plan creation device 101 (delivery plan creation system 100) may include information on the type and quantity of products that each of multiple workers M delivers (replenishes) to vending machine V (type and quantity of products to be carried out from a delivery base).

[0029] (Configuration of the delivery plan creation device) As shown in FIG. 2, the delivery plan creation device 101 includes a database unit 1, a processing unit 2, an input unit 3, and an output unit 4.

[0030] The database unit 1 stores data (such as inter-base movement information, initial inventory information, delivery plan, forecast demand, location information, demand information, inventory information, total travel time, sales, and product transportation volume) used in creating a delivery plan by a delivery plan creation unit 23 described later. The database unit 1 includes a storage device (internal storage) such as an HDD (Hard Disk Drive) or SSD (Solid State Drive).

[0031] The processing unit 2 includes a calculation unit 21, a simulator 22, a delivery plan creation unit 23, and a prediction unit 24. In this embodiment, the calculation unit 21, the simulator 22, the delivery plan creation unit 23, and the prediction unit 24 are configured as functional blocks in the delivery plan creation device 101, but each may be an independent device. The processing unit 2 (calculation unit 21) includes a processor such as a CPU (Central Processing Unit), a memory, and the like.

[0032] The calculation unit 21 is configured to process functions and data of the database unit 1, the simulator 22, the delivery plan creation unit 23, and the prediction unit 24. The simulator 22 is configured to perform a simulation of the delivery of products to multiple vending machines V according to a tentative delivery plan described below.

[0033] In addition, the delivery plan creation unit 23 is configured to create a delivery plan indicating product delivery routes to multiple vending machines V by machine learning.

[0034] In addition, in this embodiment, the delivery plan creation unit 23 performs reinforcement learning (offline learning) using the simulation results, and reinforcement learning (online learning) using delivery performance information (total travel time, sales, product transportation volume, location information, and inventory information) obtained when actually carrying out delivery.

[0035] In addition, the trained model of the delivery plan creation unit 23 includes a state-action value function (Q function) that represents a value obtained by a delivery plan as an action with respect to forecast demand and inventory information as a state, as described later. The delivery plan creation unit 23 is configured to create a delivery plan based on the state-action value function (Q function).

[0036] The prediction unit 24 is configured to predict the demand for the products sold by the vending machine V (prediction of sales volume). The prediction unit 24 is also configured to predict the future inventory volume of the products in the vending machine V (changes in the inventory volume of the products in the vending machine V). The prediction unit 24 predicts the sales volume and inventory volume using existing prediction techniques such as neural networks.

[0037] In addition, the delivery plan creation device 101 is configured to create a delivery plan using the learned model of the delivery plan creation unit 23 based on predicted demand, which is a predicted value of sales of products of multiple vending machines V, and inventory information of each product of the multiple vending machines V.

[0038] In addition, the delivery plan creation device 101 is configured to update the trained model of the delivery plan creation unit 23 by performing machine learning using the trained model of the delivery plan creation unit 23 based on delivery record information when the delivery plan is actually implemented, as described later. The delivery record information includes the sales of products of multiple vending machines V and the total travel time required for the delivery of the products. In this embodiment, the delivery record information includes the total travel time required for the delivery of the products, the sales of products of multiple vending machines V (number of sales), the transported amount of products, the location information of the worker M, and the inventory information of the products of multiple vending machines V (see FIG. 16). The delivery record information may include the sales amount as well as the number of sales as the sales of products of multiple vending machines V. The delivery record information is information (data) obtained from the mobile terminal 102 and the vending machine V.

[0039] The input unit 3 is configured to input delivery record information (receive delivery record information) via the network 200.

[0040] The output unit 4 is configured so that the delivery plan creation device 101 outputs the delivery plan created by the delivery plan creation unit 23 to the mobile terminal 102 via the network 200.

[0041] In addition, the output unit 4 is configured to output the delivery plan created by the trained model of the delivery plan creation unit 23 after the update (machine learning using delivery performance information has been performed) to the mobile terminal 102 via the network 200.

[0042] (Machine learning using simulation results) Next, machine learning using simulation results for creating a trained model used in creating a delivery plan (delivery plan calculation) will be described with reference to Figures 3 to 10. Machine learning using simulation results is performed before actual delivery is performed by worker M (before the delivery plan creation system 100 is put into actual operation).

[0043] When creating a trained model for the delivery plan creation unit 23, the delivery plan creation device 101 creates a tentative delivery plan based on the predicted sales values ​​of products in multiple vending machines V and the predicted inventory values ​​of each product in the multiple vending machines V (see Figure 3) by the delivery plan creation unit 23.

[0044] In this embodiment, the delivery plan creation unit 23 creates a provisional delivery plan using the predicted values ​​of sales of products of the multiple vending machines V over multiple days predicted by the prediction unit 24 and the predicted values ​​of the stock quantity of products of each of the multiple vending machines V over multiple days predicted by the prediction unit 24. The period (range of multiple days) used when creating the provisional delivery plan is the same for the predicted values ​​of sales of products of the multiple vending machines V and the predicted values ​​of the stock quantity of products of each of the multiple vending machines V.

[0045] 3, a simulation based on the tentative delivery plan is performed in the delivery plan creation device 101. In addition, in machine learning using the simulation results, a delivery plan (tenant delivery plan) used in the simulation is created (calculated) from demand information and inventory information.

[0046] The delivery plan creation device 101 acquires predicted delivery performance information including product sales in a simulation result of product delivery to multiple vending machines V according to a tentative delivery plan, and a total travel time required for product delivery in a simulation result of product delivery to multiple vending machines V. In this embodiment, the predicted delivery performance information includes the total travel time required for product delivery, product sales (number of units sold) of the multiple vending machines V, product transport volume, location information of the worker M, and product inventory information of the multiple vending machines V (see FIG. 3). Furthermore, the predicted delivery performance information may include sales amounts in addition to the number of units sold as sales of products from the multiple vending machines V.

[0047] Then, through a simulation by the simulator 22, inventory information, total travel time, sales volume of products, the transport volume of products (product transport volume) transported by the worker M (routeman), and the position information of the worker M are calculated for each time.

[0048] In machine learning using the simulation results, learning is performed based on demand information, inventory information (initial inventory information), a tentative delivery plan, and predicted delivery performance information (total travel time required for product delivery, sales of products from multiple vending machines V, product transport volume, location information of worker M, and product inventory information from multiple vending machines V). In machine learning using the simulation results, machine learning is repeated multiple times for a continuous fixed period of time using simulator 22. Note that the demand information can use past demand performance and predicted demand calculated using existing prediction technology such as a neural network.

[0049] In addition, the delivery plan creation device 101 creates a trained model by reinforcement learning. The delivery plan creation device 101 creates a trained model by performing reinforcement learning based on predicted delivery performance information as machine learning in the delivery plan creation unit 23.

[0050] 4, the calculation of the delivery plan (provisional delivery plan) uses demand information, inventory information, product transportation volume, and location information of the worker M. As for inventory information, a preset inventory quantity (initial inventory information) is used at the beginning of a continuous simulation for a certain period of time, and in subsequent simulations, inventory information output by the simulator 22 is used.

[0051] In addition, the demand information uses predicted values ​​or past actual values, and demand information for future periods is used. The delivery destination uses the learned results. Depending on the calculation results, there are some delivery destinations (vending machines V) to which delivery is not made, and there are also cases where delivery is not made at all. If the delivery plan calculated by the delivery plan creation unit 23 is A, the delivery plan (A) is planned according to the following formula (1). A = π(S) (1) π is a function (policy function) that calculates a delivery plan from the state (S) and represents calculations using a neural network or optimization calculations. The state (S) represents demand information, inventory information, location information of each worker M, and the amount of goods transported.

[0052] In a simulation by simulator 22, for example, the number of products at the delivery destinations indicated in the delivery plan is replenished to a maximum value. Then, based on the demand information, inventory information and sales for the next time are calculated. If the demand for products is greater than the inventory quantity (number of products) in vending machine V, the existing inventory quantity (number of products) becomes sales, and inventory becomes 0. Also, if the inventory quantity (number of products) is greater than demand, the inventory quantity (number of products) minus the demand quantity becomes inventory for the next time, and sales become equal to the demand quantity.

[0053] The total travel time is the sum of the travel times of each worker M (multiple workers M). The travel time of each worker M is calculated based on the inter-base travel time included in the inter-base travel information. The inter-base travel time is set (input) as the travel time between each delivery destination (between each vending machine V) and the travel time between the delivery base and the delivery destination (vending machine V).

[0054] As shown in Fig. 5, the delivery plan creation unit 23 of the delivery plan creation device 101 performs learning using the simulation results. Learning using the simulation results is performed by an existing reinforcement learning method. The reward r required for reinforcement learning is calculated by, for example, the sum of the total travel time (R1) and the sales (R2) weighted by (α), as shown in the following formula (2). r = α × R1 + R2 (2) The reward r may be learned by taking into account the total delivery completion time (the maximum total travel time of each routeman), the sales amount of the product, and the profit from the sales (profit margin), etc.

[0055] In this embodiment, a Deep Q Network is used as a reinforcement learning method. A state-action value function (Q function: Q(S,A)) for a delivery plan (A), which is an action according to a state (S) of inventory information and demand information, is learned by a neural network.

[0056] Note that other existing reinforcement learning methods may be used for the machine learning, such as DoubleDQN, A3C, Actor-Critic, policy gradient method, Rainbow, Q-learning, SARSA, Ape-X, R2D2, NGU, Agent57, SAC, TD3, DDPG, TRPO, and PPO.

[0057] As shown in Figure 5, when the Q function is learned, the calculation of the delivery plan (A) is performed by, for example, calculating the delivery plan (A) that optimizes the Q function (the Q function becomes a minimum value). * ) is calculated using the following formula (3). A * =π(S)=argmin A Q(S,A) (3) The delivery plan created by the trained model using the simulation results is the optimal delivery plan (A * ), with probability λ, a non-optimal routing plan (e.g., a random routing plan) is generated and a search is performed for a better routing plan (smaller local minimum).

[0058] In addition, the optimal delivery plan (A * ) can also be calculated using a neural network as shown in the following equation (4). A * =π(S)=NSθ(S) (4) In addition, θ is a parameter of the neural network. In this case, learning of NSθ(S), which calculates the delivery plan, is also performed in machine learning using the simulation results and machine learning using delivery record information.

[0059] An example of learning using simulation results will be described with reference to Figs. 6 to 8. In the example shown in Figs. 6 to 8, worker M loads products from a delivery base and replenishes them in vending machine V, which is the delivery destination. When delivery is made, all products at the delivery destination are replenished to their maximum. It is assumed that there are three types of products to be replenished (products A1 to A3), and that the maximum number of products that can be replenished in each vending machine V is 10. It is further assumed that the worker M who makes the delivery is one worker M1, and that there are three delivery bases (vending machines V1 to V3).

[0060] FIG. 6 shows the results of a simulation in which travel time is prioritized and products are delivered only to vending machines V1 and V2. As shown on the left side of FIG. 6, the demand information indicates that the demand for each of products A to C in vending machine V3 is eight bottles, whereas the inventory information indicates that the inventory for each of products A to C in vending machine V3 is one bottle. In this case, the inventory (number of products) of vending machine V3 is insufficient compared to the demand, and when products are delivered only to vending machines V1 and V2, the sales of each of products A to C in vending machine V3 are seven bottles less than the demand. FIG. 7 also shows the results of a simulation in which sales (number of bottles sold) are taken into consideration and all vending machines (vending machines V1 to V3) are delivered. In this case, by delivering to vending machine V3 as well, products are replenished to meet the demand, and sales (number of bottles sold) are higher than when products are not delivered to vending machine V3 (see FIG. 6). Specifically, when no products are delivered to vending machine V3 (see Figure 6), the sales (number of bottles sold) of vending machine V3 increases from 3 bottles (product A: 1 bottle, product B: 1 bottle, product C: 1 bottle) to 21 bottles (product A: 7 bottles, product B: 7 bottles, product C: 7 bottles) (see Figure 7). On the other hand, the total travel time also increases from 3.5 hours (see the bottom right of Figure 6) to 4 hours (see the bottom right of Figure 7). In this way, the creation of a tentative delivery plan and reinforcement learning using the simulation results based on the tentative delivery plan are repeated.

[0061] FIG. 8 shows the results of a simulation using the trained model of the delivery plan creation unit 23 in a state where it has been learned that vending machine V2 does not need to be delivered because demand for it is low. By setting the delivery destinations to vending machines V1 and V3, a delivery plan can be created that ensures a sufficient number of products to meet demand and has a short total travel time. Specifically, the sales (number of products sold) of vending machine V3 are increased to 21 (product A: 7, product B: 7, and product C: 7) (see right of FIG. 8) without reducing the sales (number of products sold) of vending machines V1 and V2, and the total travel time is also kept to 3.5 hours (see bottom of FIG. 8).

[0062] 9 and 10 show an example of a simulation result based on demand information for multiple days. In the simulation shown in Fig. 9 and 10, demand information for two days is used. This simulation shows that by using future (two days) demand information, a delivery plan can be made so that the travel time over two days (total travel time) is short even if the travel time for one day is long.

[0063] The example shown in FIG. 9 is a case where the demand for the product on the second day is not taken into consideration, and on the first day, delivery is made to vending machines V1 and V3, and on the second day, delivery is made only to vending machine V2, which is inefficient. In this case, the total travel time over two days is 6.5 hours (see the bottom of FIG. 9). On the other hand, in the example shown in FIG. 10, the demand for the product on the second day is taken into consideration, and delivery is made to vending machine V2 in advance on the first day in anticipation of the need for replenishment on the second day. In this case, the total travel time over two days is 4 hours (see the bottom of FIG. 10). By delivering in this way, it is no longer necessary to deliver only to vending machine V2 on the second day, and the total travel time over two days is reduced from 6.5 hours (see the bottom of FIG. 9) to 4 hours (see the bottom of FIG. 10), thereby shortening the total travel time over two days.

[0064] 9 and 10 show an example in which demand information for two days is used, but demand information for more than two days may be used. For example, demand information for a long period such as one week, one month, or one year may be used.

[0065] Machine learning using the simulation results is performed again when there is a change in the number of vending machines V, the installation locations of the vending machines V, the number of workers M, road conditions, etc. Machine learning using the simulation results may also be performed again when there is a change in the products sold by the vending machines V, or when the seasons change.

[0066] Furthermore, when worker M actually delivers products (during actual operation of delivery plan creation system 100), a delivery plan is first created using the results of machine learning using the simulation results (trained model). Then, worker M actually carries out delivery according to the created delivery plan (delivery route). Then, delivery plan creation system 100 (delivery plan creation device 101) performs reinforcement learning using delivery record information (total travel time required to deliver products, sales of products from multiple vending machines V, product transport volume, location information of worker M, and product inventory information from multiple vending machines V), which is the actual delivery record.

[0067] (Creating delivery plans and modifying delivery plans during delivery) The delivery plan creation unit 23 is configured to use predicted demand (see FIG. 11), which is a predicted value of sales of products of multiple vending machines V over multiple days, when creating a delivery plan.

[0068] Furthermore, the delivery plan creation device 101 acquires delivery performance information when the delivery plan is actually executed when the worker M delivers the product. Specifically, the delivery plan creation device 101 acquires delivery performance information when the delivery plan is actually executed when the worker M delivers the product via the network 200.

[0069] The delivery plan creation unit 23 is configured to output a delivery plan using as input delivery performance information, including inventory information acquired at the time of delivery of the products by the worker M, and predicted demand, which is the predicted sales value of the products of multiple vending machines V (see FIG. 11).

[0070] Furthermore, the delivery plan creation device 101 is configured to correct the delivery plan based on the acquired delivery record information without updating the trained model of the delivery plan creation unit 23. Specifically, the delivery plan creation device 101 corrects the delivery plan while the worker M is delivering the goods based on the acquired delivery record information without updating the trained model of the delivery plan creation unit 23. Then, the delivery plan creation device 101 is configured to transmit the corrected delivery plan to the mobile terminal 102 of each of the multiple workers M via the network 200.

[0071] More specifically, the delivery plan creation device 101 acquires, via the network 200, a product transport volume, which is the number of products that each of the multiple workers M can deliver, when the multiple workers M delivers the products. Then, the delivery plan creation device 101 modifies the delivery plan while the worker M is delivering the products, based on the acquired product transport volume of each of the multiple workers M, without updating the trained model of the delivery plan creation unit 23. Furthermore, the delivery plan creation device 101 is configured to transmit, via the network 200, the delivery plan after modification, which is modified based on the acquired product transport volume of each of the multiple workers M, to the mobile terminal 102 of each of the multiple workers M.

[0072] In this embodiment, as described above, the delivery plan creation device 101 modifies the delivery plan based on the product transport volume transported by the worker M with respect to the delivery plan at the start of delivery (see FIG. 12). For example, if the sales (number of sales) of the vending machine V4 in charge of the worker M2 are greater than expected, and the amount of products to be replenished to the vending machine V4 assumed (predicted) in the delivery plan creation (predicted replenishment amount) was three (see FIG. 12), but six bottles actually needed to be replenished (see FIG. 13), the number of products to be delivered (product transport volume) will be insufficient, and the worker M2 will not be able to replenish the products to the vending machine V7. Therefore, the delivery plan is modified (see FIG. 13) so that the worker M1, who has a surplus in the product transport volume, replenishes the products of the vending machine V7 instead of the worker M2 because the sales (number of sales) of the vending machine V1 are less than expected, and the predicted replenishment amount of the vending machine V1 was five (see FIG. 12), but only one bottle actually needed to be replenished. Also, another worker M, who was set in advance to have a larger amount of product to be transported in the delivery plan, may replenish products in vending machine V7. As a result, even if the amount of product to be replenished in a specific vending machine (vending machine V4) becomes greater than the replenishment amount for the predicted number of sales (expected replenishment amount) and the amount of product to be transported by worker M (worker M2) is insufficient, it is possible to prevent a case in which the amount of product to be replenished in vending machine V is insufficient by cooperating with another worker M (worker M1) to revise the delivery plan (see FIG. 13).

[0073] The delivery plan creation device 101 is also configured to acquire delivery record information including location information of the worker M at the time of delivery of the goods by the worker M. Then, the delivery plan creation device 101 modifies the delivery plan while the worker M is delivering the goods based on the acquired location information of the worker M without updating the trained model of the delivery plan creation unit 23. Furthermore, the delivery plan creation device 101 is configured to transmit the delivery plan modified based on the acquired location information of the worker M to the mobile terminal 102 of each of the multiple workers M via the network 200.

[0074] The location information of worker M may be transmitted (notified) from mobile terminal 102 to delivery plan creation device 101 by input operation by worker M or by using a Global Positioning System (GPS), or may be transmitted (notified) from vending machine V to delivery plan creation device 101 when communication between mobile terminal 102 and vending machine V and opening and closing of vending machine V are triggered when vending machine V is replenished with products (during maintenance).

[0075] In addition, in a delivery plan (delivery route) such as that shown in Fig. 14, when a specific worker M (worker M2) is late in delivery due to traffic congestion or the like, the delivery plan creation device 101 modifies the delivery plan (see Fig. 15) so that another worker M (worker M1) delivers the product to the vending machine (vending machine V7) in charge of the worker M2 who is late in delivery, instead of the worker M2 whose delivery is delayed. For example, when delivering products according to the delivery plan (delivery route) such as that shown in Fig. 14, if the delivery by worker M2 is delayed due to traffic congestion or the like, and the delivery by worker M1 has progressed to vending machine V3, but the delivery by worker M2 has only progressed to vending machine V4 (see Fig. 15), the delivery plan is modified. In such a case, the delivery plan is revised from a delivery plan (see FIG. 14) in which worker M1 delivers vending machines V1, V2, V3 in that order, and worker M2 delivers vending machines V4, V5, V6, V7 in that order, to a delivery plan (see FIG. 15) in which worker M1 delivers vending machines V1, V2, V3, V7 in that order, and worker M2 delivers vending machines V4, V5, V6 in that order. This makes it possible to speed up the time when all workers M complete their deliveries by cooperating among multiple workers M, even if deliveries are delayed.

[0076] (Machine learning using delivery performance information) As shown in Fig. 16, the delivery plan creation device 101 is configured to update the trained model of the delivery plan creation unit 23 by machine learning using the trained model of the delivery plan creation unit 23 based on delivery performance information (total travel time, sales, product transportation volume, location information, and inventory information) after the delivery of products to multiple vending machines V is completed. Note that the trained model of the delivery plan creation unit 23 may be updated daily or every few days. Furthermore, the trained model of the delivery plan creation unit 23 may be updated at any period or timing of the user's choice.

[0077] The delivery plan creation device 101 updates a state-action value function (Q function) of the learned model by reinforcement learning using the learned model of the delivery plan creation unit 23 based on delivery record information. Specifically, the delivery plan creation device 101 is configured to update the state-action value function (Q function) of the learned model by reinforcement learning using the learned model of the delivery plan creation unit 23 based on delivery record information after delivery of products to multiple vending machines V is completed.

[0078] Fig. 17 shows an example of learning using delivery record information, and shows an example of using a reinforcement learning method using a state-action value function (Q function). In learning using delivery record information, as in the case of learning using simulation results, reward r is calculated from the actual total travel time and actual sales record when worker M (routeman) actually makes a delivery.

[0079] Then, using the delivery plan in which the delivery was actually performed, delivery performance information (total travel time, sales, product transportation volume, location information, and inventory information), and predicted demand, the state-action value function (Q function) learned using the simulation results is updated (modified) before the actual delivery. In this embodiment, by re-learning using performance (delivery performance information) associated with the actual delivery, it becomes possible to calculate a more optimal delivery route (delivery plan). Note that, like learning using the delivery performance information, existing reinforcement learning techniques can be used for learning using the simulation results. In this embodiment, the learned model (Q function) is updated by reinforcement learning similar to learning using the simulation results.

[0080] As described above, the delivery plan creation device 101 acquires delivery record information including the location information of the worker M at the time of delivery of the goods by the worker M. Then, the trained model of the delivery plan creation unit 23 is configured to be updated by machine learning using the trained model based on the delivery record information including the acquired location information.

[0081] 18 and 19 show the results of reinforcement learning using delivery record information including the location information of worker M. In reinforcement learning using delivery record information including the location information of worker M, it is possible to learn a delivery plan that avoids traffic jams, for example. In learning using delivery record information, the delivery route (see FIG. 18) that delivered vending machine V1, vending machine V3 in that order is changed to vending machine V3, vending machine V1 in that order (see FIG. 19) in learning that takes traffic jams into account (by learning the past travel time results). As a result, the total travel time is reduced from 4.5 hours (see the bottom right of FIG. 18) to 3.5 hours (see the bottom right of FIG. 19), making it possible to create a delivery plan in which the total travel time is shorter (shortened by 1 hour).

[0082] (Effects of this embodiment) In this embodiment, the following effects can be obtained.

[0083] In the delivery plan creation device 101 (delivery plan creation system 100) according to this embodiment, a delivery plan is created by the trained model of the delivery plan creation unit 23 based on the predicted demand, which is a predicted value of the sales of products of multiple vending machines V, and the inventory information of each product of multiple vending machines V. This makes it possible to create a delivery plan that can suppress a decrease in sales of products due to sell-out. In addition, the trained model of the delivery plan creation unit 23 is updated by performing machine learning using the trained model of the delivery plan creation unit 23 based on the delivery performance information when the delivery plan is actually implemented. This makes it possible to update the trained model of the delivery plan creation unit 23 to a trained model that reflects delivery performance information including the sales of products of multiple vending machines V and the total travel time required for delivery of the products when the delivery plan is actually implemented. As a result, the delivery plan creation unit 23 can create an efficient delivery plan that reflects delivery performance information including the sales of products of multiple vending machines V and the total travel time required for delivery of the products when the delivery plan is actually implemented, at the time of delivery after the trained model is updated. As a result, it is possible to create a delivery route that suppresses an increase in delivery time while suppressing a decrease in sales of products from the vending machine V due to selling out. In addition, the output unit 4 outputs the delivery plan created by the trained model of the updated delivery plan creation unit 23 to the mobile terminal 102 via the network 200, so that the worker M can check, by the mobile terminal 102, an efficient delivery plan that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of products from the vending machine V due to selling out. As a result, the worker M can perform efficient product delivery that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of products from the vending machine V due to selling out.

[0084] In the present embodiment, as described above, the trained model of the delivery plan creation unit 23 includes a state action value function (Q function) that represents value, and the delivery plan creation unit 23 is configured to create a delivery plan based on the state action value function. The delivery plan creation device 101 then updates the state action value function of the trained model by reinforcement learning using the trained model of the delivery plan creation unit 23 based on delivery performance information. This allows the state action value function updated based on the delivery performance information to create a delivery plan that reflects the delivery performance information, has a short delivery time, and has a large sales of products from the vending machine V. As a result, the delivery plan creation unit 23 can create an efficient delivery route that can suppress an increase in delivery time when delivering products to multiple vending machines V, while suppressing a decrease in sales of products from the vending machine V due to selling out.

[0085] Furthermore, in this embodiment, as described above, after the delivery of products to multiple vending machines V is completed, the trained model of the delivery plan creation unit 23 is updated by machine learning using the trained model of the delivery plan creation unit 23 based on the delivery record information. This makes it possible to update the trained model of the delivery plan creation unit 23 so as to reflect the above-mentioned delivery record information from the start of delivery of products to multiple vending machines V to the completion of delivery. As a result, the delivery plan creation unit 23 after updating the trained model can create a more efficient delivery plan (delivery route) that can further suppress an increase in delivery time when delivering products to multiple vending machines V and further suppress a decrease in sales of products from the vending machines V due to sold-outs.

[0086] In the present embodiment, as described above, the delivery plan creation device 101 acquires delivery record information including the position information of the worker M at the time of delivery of the worker M's goods. Then, the trained model of the delivery plan creation unit 23 is updated by machine learning using the trained model based on the delivery record information including the acquired position information. As a result, the travel time between the multiple vending machines V at the time of delivery of the worker M's goods can be acquired from the delivery record information including the position information of the worker M. As a result, when the travel time between the multiple vending machines V in the actual delivery of goods is longer than the travel time assumed when the delivery plan was created, the trained model of the delivery plan creation unit 23 can be updated based on the travel time between the multiple vending machines V acquired from the delivery record information including the position information of the worker M. As a result, a more optimal delivery route can be created by changing the route between the multiple vending machines V or the order of the vending machines V to be delivered, and therefore a more efficient delivery plan can be created.

[0087] In this embodiment, as described above, the delivery plan creation unit 23 uses the predicted demand, which is a predicted value of sales of products from multiple vending machines V over multiple days, when creating a delivery plan. This makes it possible to create a delivery route with a short total travel time or total travel distance required for delivery of products over multiple days. As a result, the delivery plan creation unit 23 can create an efficient delivery route for products to multiple vending machines V when delivering products over multiple days.

[0088] In addition, in this embodiment, as described above, the delivery plan creation device 101 acquires delivery record information when the delivery plan is actually implemented when the worker M delivers the goods. Then, the delivery plan creation unit 23 is configured to output a delivery plan using delivery record information including inventory information acquired when the worker M delivers the goods and forecast demand, which is a forecast value of sales of goods of multiple vending machines V, as inputs. Furthermore, the delivery plan creation device 101 corrects the delivery plan based on the acquired delivery record information without updating the trained model of the delivery plan creation unit 23. This allows the delivery plan creation unit 23 to correct the delivery plan before updating the trained model based on actual inventory information. As a result, the delivery plan can be corrected to a more efficient delivery plan by reflecting actual inventory information before updating the trained model. In addition, by correcting the delivery plan without updating the trained model of the delivery plan creation unit 23, the delivery plan can be easily corrected compared to the case where the trained model of the delivery plan creation unit 23 is updated from time to time.

[0089] In the present embodiment, as described above, the delivery plan creation device 101 acquires delivery record information when the delivery plan is actually executed when the worker M delivers the goods via the network 200. Then, the delivery plan creation device 101 modifies the delivery plan during the delivery of the goods by the worker M based on the acquired delivery record information without updating the trained model of the delivery plan creation unit 23, and transmits the modified delivery plan to each of the mobile terminals 102 of the multiple workers M via the network 200. As a result, during the delivery of the goods by the worker M, the delivery plan modified based on the acquired delivery record information is transmitted to each of the mobile terminals 102 of the multiple workers M, so that the worker M can efficiently deliver the goods while the goods are being delivered, reflecting the situation during delivery in real time. In addition, by modifying the delivery plan without updating the trained model of the delivery plan creation unit 23, the delivery plan can be modified in real time during the delivery of the goods more quickly than when the trained model of the delivery plan creation unit 23 is updated from time to time.

[0090] In the present embodiment, as described above, the delivery plan creation device 101 acquires, via the network 200, the product transport volume, which is the number of products that each of the multiple workers M can deliver, when the multiple workers M delivers the products. The delivery plan creation device 101 then modifies the delivery plan during the delivery of the products by the workers M based on the acquired product transport volume of each of the multiple workers M without updating the trained model of the delivery plan creation unit 23, and transmits the modified delivery plan to each of the mobile terminals 102 of the multiple workers M via the network 200. As a result, the delivery plan can be modified based on the acquired product transport volume of each of the multiple workers M during the delivery of the products by the workers M, so that even if the product transport volume of a certain worker M is insufficient, the delivery route can be modified so that another worker M who has a surplus in product transport volume replenishes the products instead. Then, the modified delivery plan is transmitted to the mobile terminals 102 of each of the multiple workers M, so that the workers M can perform efficient product delivery that reflects the product transport volume of each worker M. Furthermore, by revising the delivery plan without updating the trained model of the delivery plan creation unit 23, the delivery plan can be revised quickly while the goods are being delivered, based on the acquired product transportation volume of each of the multiple workers M, compared to the case where the trained model of the delivery plan creation unit 23 is updated from time to time.

[0091] In the present embodiment, as described above, when creating a trained model for the delivery plan creation unit 23, the delivery plan creation device 101 creates a tentative delivery plan based on the predicted sales value of the products of the multiple vending machines V and the predicted inventory value of each product of the multiple vending machines V by the delivery plan creation unit 23. Then, the delivery plan creation device 101 acquires predicted delivery performance information including the sales value of the products in the simulation result of the delivery of the products to the multiple vending machines V by the tentative delivery plan and the total travel time required for the delivery of the products in the simulation result of the delivery of the products to the multiple vending machines V. Then, the delivery plan creation device 101 creates a trained model by performing reinforcement learning based on the predicted delivery performance information as machine learning in the delivery plan creation unit 23. This allows learning by reinforcement learning to be performed before actually delivering the products by the predicted delivery performance information obtained by the simulation result based on the tentative delivery plan. As a result, it is possible to create a trained model for the delivery plan creation unit 23 that can create a more efficient delivery plan before actually delivering the products, compared to the case where reinforcement learning is not performed by the predicted delivery performance information obtained by the simulation result based on the tentative delivery plan.

[0092] [Variations] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims rather than the description of the above embodiments, and further includes all modifications (variations) within the meaning and scope of the claims.

[0093] For example, in the above embodiment, an example was shown in which the delivery performance information includes the total travel time required to deliver the products, the sales (number of units sold) of the products of the multiple vending machines V, the product transport volume, the location information of the worker M, and the inventory information of the products of the multiple vending machines V, but the present invention is not limited to this. In the present invention, the delivery performance information may include the total travel distance required to deliver the products, and may include the sales amount and profit (profit margin) of the products sold by the vending machine V.

[0094] In the above embodiment, the predicted delivery performance information includes the total travel time required to deliver the products, the sales (number of units sold) of the products of the multiple vending machines V, the product transport volume, the location information of the worker M, and the inventory information of the products of the multiple vending machines V, but the present invention is not limited to this. In the present invention, the predicted delivery performance information may include the total travel distance required to deliver the products, and may include the sales amount and profit (profit margin) of the products sold by the vending machine V.

[0095] In the above embodiment, the delivery plan creation device 101 updates the state-action value function of the learned model by reinforcement learning using the learned model of the delivery plan creation unit 23 based on delivery record information, but the present invention is not limited to this. In the present invention, the delivery plan creation device 101 may be configured to update the learned model by machine learning other than reinforcement learning, such as learning using a neural network, unsupervised learning, and supervised learning.

[0096] In the above embodiment, after the delivery of products to multiple vending machines V is completed, the trained model of the delivery plan creation unit 23 is updated by machine learning using the trained model of the delivery plan creation unit 23 based on delivery record information, but the present invention is not limited to this. In the present invention, the delivery plan creation device 101 may update the trained model of the delivery plan creation unit 23 while the worker M is delivering the products.

[0097] In the above embodiment, an example was shown in which the trained model of the delivery plan creation unit 23 is updated by machine learning using the trained model based on the delivery performance information including the acquired location information, but the present invention is not limited to this. In the present invention, the trained model of the delivery plan creation unit may be updated by machine learning using the trained model based on the acquired product transportation volume of each of the multiple workers.

[0098] In the above embodiment, an example is shown in which the delivery plan is revised while the worker M is delivering the goods based on the acquired product transport volume of each of the multiple workers M without updating the trained model of the delivery plan creation unit 23, and the delivery plan is revised while the worker M is delivering the goods based on the acquired location information without updating the trained model of the delivery plan creation unit 23, but the present invention is not limited to this. In the present invention, a configuration may be used in which the delivery plan is revised while the worker is delivering the goods based on either the acquired product transport volume or location information of each of the multiple workers without updating the trained model of the delivery plan creation unit.

[0099] In the above embodiment, the delivery plan creation unit 23 uses the predicted demand, which is a predicted value of sales of products from multiple vending machines V over multiple days, when creating the delivery plan, but the present invention is not limited to this. In the present invention, the delivery plan creation unit 23 may use the predicted demand, which is a predicted value of sales of products from multiple vending machines V for one day, when creating the delivery plan and the tentative delivery plan.

[0100] In the above embodiment, the delivery plan creation device 101 corrects the delivery plan while the worker M is delivering the goods based on the acquired delivery performance information without updating the trained model of the delivery plan creation unit 23, but the present invention is not limited to this. In the present invention, it is also possible to only update the trained model after the delivery is completed, and not to correct the delivery plan while the worker M is delivering the goods.

[0101] In the above embodiment, the delivery plan creation unit 23 creates a provisional delivery plan using a predicted value of sales of products of multiple vending machines V and a predicted value of the stock quantity of each product of multiple vending machines V, but the present invention is not limited to this. In the present invention, in creating a provisional delivery plan, past actual values ​​of sales and stock quantities of products of multiple vending machines may be used, or both predicted values ​​and past actual values ​​of sales and stock quantities of products of multiple vending machines may be used. In the present invention, in creating a provisional delivery plan, both predicted values ​​and past actual values ​​of sales and stock quantities of products of multiple vending machines may be used for one, and either predicted values ​​or past actual values ​​may be used for the other. [Explanation of symbols]

[0102] 4 Output section 23 Delivery Planning Department 100 Delivery planning system 101 Delivery plan creation device 102 Mobile Devices 200 Network M, M1, M2 workers V, V1, V2, V3, V4, V5, V6, V7 vending machine

Claims

1. a delivery plan creation unit that creates a delivery plan showing product delivery routes to multiple vending machines using machine learning; an output unit for outputting the delivery plan created by the delivery plan creation unit via a network to a mobile terminal carried by an operator who replenishes products in the vending machines; The delivery plan is created based on a predicted demand, which is a predicted value of sales of products of the plurality of vending machines, and inventory information of each of the products of the plurality of vending machines, using a trained model that represents a value obtained by the delivery plan as an action with respect to the predicted demand and the inventory information as a state of the delivery plan creation unit, The trained model of the delivery plan creation unit is updated by performing reinforcement learning of the trained model regarding the value of the delivery plan represented by the trained model of the delivery plan creation unit using delivery record information including sales of products of the multiple vending machines, a total travel time required to deliver the products, and inventory information of the products of each of the multiple vending machines when the delivery plan is actually implemented, The output unit is configured to output the delivery plan created by the trained model of the delivery plan creation unit after updating to the mobile terminal via a network.

2. The trained model of the delivery plan creation unit includes a state-action value function that represents a value obtained by the delivery plan as an action with respect to the forecasted demand and the inventory information as a state, The delivery plan creation unit is configured to create the delivery plan based on the state-action value function, The delivery plan creation device according to claim 1, configured to update the state-action value function of the trained model by reinforcement learning using the trained model of the delivery plan creation unit based on the delivery performance information.

3. 3. The delivery plan creation device according to claim 1 or 2, wherein after delivery of products to the plurality of vending machines is completed, the trained model of the delivery plan creation unit is updated by machine learning using the trained model of the delivery plan creation unit based on the delivery performance information.

4. Acquire the delivery record information including location information of the worker at the time of delivery of the goods by the worker; The delivery plan creation device according to claim 3, wherein the trained model of the delivery plan creation unit is configured to be updated by machine learning using the trained model based on the delivery performance information including the acquired location information.

5. The delivery plan creation device according to any one of claims 1 to 4, wherein the delivery plan creation unit is configured to use the predicted demand, which is a predicted value of sales of products of the plurality of vending machines over a plurality of days, in creating the delivery plan.

6. When the worker delivers the goods, the delivery performance information is acquired when the delivery plan is actually carried out, the delivery plan creation unit is configured to output the delivery plan using as input the delivery record information including the inventory information acquired when the worker delivers the products and the predicted demand, which is a predicted value of sales of the products of the multiple vending machines; The delivery plan creation device according to any one of claims 1 to 5, configured to correct the delivery plan based on the acquired delivery record information without updating the trained model of the delivery plan creation unit.

7. When the worker delivers the goods, the delivery performance information when the delivery plan is actually executed is acquired via a network; The delivery plan is modified during delivery of the goods by the worker based on the acquired delivery performance information without updating the trained model of the delivery plan creation unit; The delivery plan creation device according to any one of claims 1 to 6, configured to transmit the modified delivery plan to the mobile terminals of each of the plurality of workers via a network.

8. When the plurality of workers delivers the goods, a product transportation volume, which is the number of goods that each of the plurality of workers can deliver, is acquired via a network; The delivery plan is modified during delivery of the goods by the workers based on the acquired product transportation volume of each of the multiple workers without updating the trained model of the delivery plan creation unit; The delivery plan creation device according to any one of claims 1 to 7, configured to transmit the revised delivery plan to the mobile terminals of each of the multiple workers via a network.

9. When creating the trained model of the delivery plan creation unit, creating a provisional distribution plan based on at least one of a predicted value or a past actual value of sales of the products of the plurality of vending machines and at least one of a predicted value or a past actual value of inventory quantity of the products of each of the plurality of vending machines by the distribution plan creation unit; Acquire predicted delivery performance information including sales of products in a simulation result of product delivery to the multiple vending machines according to the tentative delivery plan and a total travel time required for product delivery in a simulation result of product delivery to the multiple vending machines; The delivery plan creation device according to any one of claims 1 to 8, wherein the learned model is created by performing reinforcement learning based on the predicted delivery performance information as machine learning in the delivery plan creation unit.

10. A mobile terminal carried by an operator who replenishes products in a vending machine; a delivery plan creation device that creates a delivery plan indicating product delivery routes to a plurality of vending machines; The delivery plan creation device includes a delivery plan creation unit that creates the delivery plan by machine learning, and an output unit that outputs the delivery plan created by the delivery plan creation unit to the mobile terminal via a network, The delivery plan is created based on a predicted demand, which is a predicted value of sales of products of the plurality of vending machines, and inventory information of each of the products of the plurality of vending machines, using a trained model that represents a value obtained by the delivery plan as an action with respect to the predicted demand and the inventory information as a state of the delivery plan creation unit, The trained model of the delivery plan creation unit is updated by performing reinforcement learning of the trained model regarding the value of the delivery plan represented by the trained model of the delivery plan creation unit using delivery record information including sales of products of the multiple vending machines, a total travel time required for delivering the products, and inventory information of the products of each of the multiple vending machines when the delivery plan is actually implemented, The output unit is configured to output the delivery plan created by the trained model of the delivery plan creation unit after updating to the mobile terminal via a network.

Images