Training device, control device, training method, and control method
A reinforcement learning model is employed to prioritize interrupt tasks over normal tasks in automated warehouses, addressing the disruption caused by interrupt tasks and enhancing operational efficiency.
Patent Information
- Application Number
- JP2025077806
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2026-11-19
AI Technical Summary
Automated warehouses face significant disruptions due to interrupt tasks, leading to prolonged total task completion times when these tasks are simply added to the normal task queue, necessitating a method to reduce the impact of such interruptions.
A reinforcement learning model is trained to prioritize interrupt tasks over normal tasks based on parameter information, using a training device and management device that manage and control transport devices in automated warehouses, with specific memory areas for normal and interrupt tasks, and processes to adjust task execution based on model output.
The impact of interrupt tasks on automated warehouses is reduced by optimizing task execution, thereby minimizing the overall completion time and improving operational efficiency.
Smart Images

Figure 2026181959000001_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to a training device, a management device, a training method, and a management method. BACKGROUND ART
[0002] Japanese Unexamined Patent Application Publication No. 6-214998 (Patent Document 1) provides a production process progress management system that enables grasping the impact on other processes when the plan of a certain process is modified. PRIOR ART DOCUMENTS PATENT DOCUMENTS
[0003] PATENT DOCUMENT 1 Japanese Unexamined Patent Application Publication No. 6-214998 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION
[0004] An automated warehouse includes, for example, a plurality of transport devices each processing a task of transporting a load from a plurality of storage spaces to a plurality of shipping ports. The automated warehouse is configured to, as normal processing, assign the next normal task to a transport device that has become free after processing a certain normal task among the plurality of transport devices. On the other hand, in the automated warehouse, an interrupt task may occur in addition to normal tasks. When an interrupt task occurs, reorganization of task management may be required.
[0005] For example, a method is conceivable in which an interrupt task is added to the top of the queue of normal tasks to be processed first. However, with such a simple method, the interrupt processing may significantly affect the processing of the entire automated warehouse, for example, the total task completion time (the time from the occurrence of the interrupt task to the completion of all tasks) is greatly prolonged. Therefore, it is desirable to reduce the impact of the interrupt task on the automated warehouse.
[0006] This disclosure was made to solve the above-mentioned problems, and one of the purposes of this disclosure is to reduce the impact of interrupt tasks on automated warehouses. [Means for solving the problem]
[0007] A first aspect of this disclosure is a training device for training a reinforcement learning model used to control an automated warehouse. The automated warehouse includes a transport device that performs tasks of transporting goods. The tasks include normal tasks and interrupt tasks that occur after the normal tasks have occurred. The reinforcement learning model receives parameter information including information about normal tasks, information about interrupt tasks, and information about the future location of the transport device, and outputs a result indicating whether or not to prioritize interrupt tasks over normal tasks. The training device comprises an arithmetic unit and memory. The memory includes a first area for storing information about normal tasks and a second area for storing information about interrupt tasks. The arithmetic unit repeatedly executes a first predetermined process. The first predetermined process includes the process of obtaining the output result of a reinforcement learning model by inputting parameter information into the reinforcement learning model, the process of adding information of interrupt tasks in the second domain to the information of normal tasks in the first domain and deleting the information of interrupt tasks in the second domain so that the transport device executes interrupt tasks with priority over normal tasks if the output result indicates that interrupt tasks should be prioritized, and the process of retaining the information of interrupt tasks in the second domain if the output result indicates that interrupt tasks should not be prioritized. When all normal tasks and all interrupt tasks in the first domain have been processed by repeatedly executing the first predetermined process, the reward is determined based on speed parameters relating to the processing speed of all normal tasks and the processing speed of all interrupt tasks, and the reinforcement learning model is trained to increase the reward.
[0008] A second aspect of this disclosure is a management device for an automated warehouse. The automated warehouse includes a transport device that performs tasks of transporting goods. The tasks include normal tasks and interrupt tasks that occur after the occurrence of normal tasks. The management device comprises a control device that controls the transport device and a memory that stores a reinforcement learning model. The reinforcement learning model receives parameter information including information on normal tasks, information on interrupt tasks, and information on the future location of the transport device, and outputs a result indicating whether or not to prioritize interrupt tasks over normal tasks. The memory further includes a third area for storing information on normal tasks and a fourth area for storing information on interrupt tasks. The control device repeatedly executes a second predetermined process. The second predetermined process includes, when an interrupt task occurs, a process to store information of the interrupt task in the fourth area; a process to obtain the output result of the reinforcement learning model by inputting parameter information into the reinforcement learning model; if the output result indicates that the interrupt task should take priority, a process to add the interrupt task information in the fourth area to the normal task information in the third area and erase the interrupt task information in the fourth area so that the transport device executes the interrupt task with priority over normal tasks; if the output result indicates that the interrupt task should not take priority, a process to retain the interrupt task information in the fourth area; and a process to operate the transport device to complete all normal tasks and all interrupt tasks in the third area.
[0009] A third aspect of this disclosure is a training method for training a reinforcement learning model used for controlling an automated warehouse. The automated warehouse includes a transport device that performs tasks of transporting goods. The tasks include normal tasks and interrupt tasks that occur after the normal tasks have occurred. The reinforcement learning model receives parameter information, which includes information on normal tasks stored in a first area of memory, information on interrupt tasks stored in a second area of memory, and information on the future location of the transport device, and outputs a result indicating whether or not to prioritize interrupt tasks over normal tasks. The training method comprises repeatedly executing a first predetermined process. The first predetermined process includes obtaining the output result of the reinforcement learning model by inputting parameter information to the reinforcement learning model, adding the interrupt task information from the second area to the normal task information in the first area and erasing the interrupt task information in the second area if the output result indicates that the interrupt tasks should be prioritized, and leaving the interrupt task information in the second area if the output result indicates that the interrupt tasks should not be prioritized. The training method further comprises determining a reward based on speed parameters relating to the processing speed of all normal tasks and all interrupt tasks when all normal tasks and all interrupt tasks in the first domain have been processed by repeatedly executing a first predetermined process, and training a reinforcement learning model to maximize the reward.
[0010] A fourth aspect of this disclosure is a management method for managing an automated warehouse using a reinforcement learning model. The automated warehouse includes a transport device that performs tasks of transporting goods. The tasks include normal tasks and interrupt tasks that occur after the occurrence of normal tasks. The reinforcement learning model receives parameter information, including information on normal tasks stored in a third area of memory, information on interrupt tasks stored in a fourth area of memory, and information on the future location of the transport device, and outputs a result indicating whether or not to prioritize interrupt tasks over normal tasks. The management method comprises repeatedly executing a second predetermined process. The second predetermined process includes, when an interrupt task occurs, storing information about the interrupt task in the fourth area, obtaining the output result of the reinforcement learning model by inputting parameter information into the reinforcement learning model, adding the interrupt task information in the fourth area to the normal task information in the third area and deleting the interrupt task information in the fourth area if the output result indicates that the interrupt task should take priority over normal tasks, and leaving the interrupt task information in the fourth area if the output result indicates that the interrupt task should not take priority, and operating the transport device to complete all normal tasks and all interrupt tasks in the third area. [Effects of the Invention]
[0011] According to this disclosure, the impact of interrupt tasks on the automated warehouse can be reduced. [Brief explanation of the drawing]
[0012] [Figure 1] This figure shows an example configuration of the management system of this embodiment. [Figure 2] This is a block diagram showing the overall configuration of the warehouse system, including the task management device. [Figure 3] This is a block diagram showing the overall configuration of a training system, including training equipment. [Figure 4] This is a schematic diagram of an automated warehouse. [Figure 5] This diagram illustrates the normal task information and interrupt task information during the utilization phase. [Figure 6] This is a diagram for explaining the refresh processing in the utilization phase. [Figure 7] This is a functional block diagram of the task management device. [Figure 8] This is a flowchart showing the main processing of the task management device. [Figure 9] This is a diagram for explaining the interrupt task list. [Figure 10] This is a diagram for explaining the normal task information and interrupt task information in the learning phase. [Figure 11] This is a diagram for explaining the refresh processing in the learning phase. [Figure 12] This is a functional block diagram of the training device. [Figure 13] This is a flowchart showing the main processing of the training device. [Figure 14] This is a flowchart showing the main processing of the training device. [Figure 15] This is a diagram for explaining the reward calculation formula. [Figure 16] This is a graph of the first value when a is set to 1.5. [Figure 17] This is a graph of the first value when a is set to 0.5. [Figure 18] This is a diagram for explaining the average of V1 and V2. [Mode for Carrying Out the Invention]
[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the same reference numerals are assigned to the same or corresponding parts in the drawings, and the description thereof will not be repeated.
[0014] <Explanation of Terms> Carrying out packages from the automatic warehouse to the outside is referred to as "shipping", and carrying packages into the automatic warehouse from the outside is referred to as "receiving". In addition, the term "package" may be replaced with "article".
[0015] <Management System> Figure 1 shows an example configuration of the management system 300 according to this embodiment. In the example shown in Figure 1, the management system 300 comprises a task management device 2 and a training device 200. The task management device 2 and the training device 200 are able to communicate with each other via a network or the like.
[0016] The training device 200 holds the reinforcement learning model 600 and trains the reinforcement learning model 600. The training device 200 transmits the trained reinforcement learning model 600 to the task management device 2.
[0017] The task management device 2 stores the reinforcement learning model 600 transmitted from the training device 200 as reinforcement learning model 500. The task management device 2 uses the reinforcement learning model 500 to execute the tasks described later.
[0018] In the example shown in Figure 1, the task management device 2 and the training device 200 are disclosed as separate devices. However, the task management device 2 and the training device 200 may be the same device.
[0019] Furthermore, the reinforcement learning models 500 and 600 include, for example, the following network structure and parameters used by the network structure. The network structure is, for example, a neural network. Other examples of reinforcement learning models 500 and 600 include Bayesian models. In this embodiment, a neural network is adopted as the structure of the reinforcement learning models 500 and 600.
[0020] <Warehouse System> Figure 2 is a block diagram showing the overall configuration of the warehouse system, including the task management device 2. In this example, the warehouse system 100 includes an automated warehouse 1, a task management device 2, and an integrated management device 3.
[0021] Automated warehouse 1 is configured to store multiple items and to automatically receive and ship those items. Automated warehouse 1 is, for example, a shuttle-type automated warehouse. The configuration of automated warehouse 1 is explained in Figure 4.
[0022] Task management device 2 performs information processing for management of automated warehouse 1. Task management device 2 is typically a server or workstation, but may also be a general-purpose computer such as a PC (Personal Computer) or a smart device. Task management device 2 includes a processor 21, memory 23, input device 24, display 25, network controller 26, and bus 27.
[0023] The processor 21 includes processing circuitry such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The memory 23 includes volatile storage devices such as DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory). The memory 23 also includes non-volatile storage devices such as ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), and flash memory. The processor 21 corresponds to the "control device" of the task management device 2. The processor 21 may also be referred to as the "control circuit."
[0024] Memory 23 stores a system program (not shown) including the OS (Operating System) and a program 231 for task management of the automated warehouse 1. The processor 21 reads the system program and program 231, loads them into memory 23, and executes them.
[0025] Memory 23 is further provided with a queue 233 and a stack 234. Furthermore, the reinforcement learning model 500 is stored in memory 23. Queue 233 is an area for storing information on normal tasks (normal task information 243 described later) and corresponds to the "third area" of this disclosure. Stack 234 is an area for storing information on interrupt tasks (interrupt task information 244 described later) and corresponds to the "fourth area" of this disclosure.
[0026] Queue 233 and Queue 333 (described later) employ a First In, First Out (FIFO) configuration. Stack 234 and Stack 334 (described later) employ a Last In, First Out (LIFO) configuration.
[0027] Although only one processor is shown in Figure 2, the task management device 2 may include multiple processors. That is, the task management device 2 includes one or more processors. The same applies to the memory 23. In this specification, "processor" is not limited to a processor in the narrow sense that executes processing using a stored program method, but may also include hardwired circuits such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field-Programmable Gate Arrays). Therefore, the term "processor" can also be interpreted as a processing circuit (circuitry or processing circuitry) whose processing is predefined by computer-readable code and / or hardwired circuits. The task management device 2 may be divided into multiple units according to function.
[0028] The input device 24 is a keyboard, touch panel, etc., and accepts operation from the user (such as the person in charge of task management for the automated warehouse 1). The display 25 displays the task management status (such as normal task information 243) by the task management device 2. The network controller 26 is configured to communicate with the outside of the task management device 2 (integrated management device 3) according to the control of the processor 21. The bus 27 connects the components of the task management device 2 in a way that allows communication between them.
[0029] The integrated management device 3 operates at a higher level than the task management device 2 and manages the automated warehouse 1 (which may include other automated warehouses not shown). The integrated management device 3 may have a similar hardware configuration to the task management device 2. Therefore, a detailed explanation of the configuration of the integrated management device 3 will not be repeated.
[0030] The integrated management device 3 outputs multiple routine tasks to the task management device 2 for shipping goods from the automated warehouse 1 according to a prior plan. When the task management device 2 receives multiple routine tasks from the integrated management device 3, it generates routine task information 243 for processing the multiple routine tasks and stores it in the queue 233.
[0031] Furthermore, the integrated management device 3 outputs one or more interrupt tasks to the task management device 2 for shipping goods from the automated warehouse 1 in accordance with prior plans (for example, requests that arise suddenly). Thus, interrupt tasks are tasks that occur after the occurrence of normal tasks. When the task management device 2 receives one or more interrupt tasks from the integrated management device 3, it generates interrupt task information 244 and temporarily stores it in the stack 234.
[0032] Then, the task management device 2 updates the normal task information 243 stored in the queue 233 by adding the interrupt task information 244 to the normal task information 243, according to the output result of the reinforcement learning model 500. In this way, the task management device 2 stores the interrupt task information 244 in the queue 233. In this embodiment, in the utilization phase and the learning phase described later, the interrupt task information 244 stored in the queue 233 is used as normal task information 243. Also, in the utilization phase and the learning phase described later, "normal task" and "interrupt task" are collectively referred to as "task".
[0033] Alternatively, the task management device 2 may acquire tasks (normal tasks or interrupt tasks) through user operations on the input device 24, instead of receiving them from the integrated management device 3.
[0034] The task management device 2 outputs a cargo transport command to the shuttle (described later) of the automated warehouse 1 based on the generated or updated normal task information 243. In addition, the task management device 2 receives reports on the cargo transport status from the automated warehouse 1.
[0035] <Training System> Figure 3 is a block diagram showing the overall configuration of the training system, including the training device 200. In this example, the training system 150 includes the training device 200 and the simulator 203. The simulator 203 corresponds to the “external device” in this disclosure.
[0036] The training device 200 performs information processing for training the reinforcement learning model 600. The training device 200 is typically a server or workstation, but may be a general-purpose computer such as a PC, or a smart device. The training device 200 includes a processor 221, memory 223, input device 224, display 225, network controller 226, and bus 227.
[0037] The processor 221 includes processing circuits such as a CPU and an MPU. The memory 223 includes volatile memory devices such as DRAM and SRAM. The memory 223 also includes non-volatile memory devices such as ROM, HDD, SSD, and flash memory. The processor 221 corresponds to the "arithmetic unit" of the training device 200. The processor 221 may also be referred to as the "arithmetic circuit".
[0038] Memory 223 stores a system program (not shown) including the OS, and a program 251 for training the reinforcement learning model 600. The processor 21 reads the system program and program 251, loads them into memory 23, and executes them.
[0039] Memory 223 is further provided with a queue 333 and a stack 334. Furthermore, the reinforcement learning model 600 is stored in memory 23. Queue 333 is an area for storing information about normal tasks (normal task information 343 described later) and corresponds to the "first area" of this disclosure. Stack 334 is an area for storing information about interrupt tasks (interrupt task information 344 described later) and corresponds to the "second area" of this disclosure.
[0040] Although only one processor is shown in Figure 3, the training device 200 may include multiple processors. That is, the training device 200 includes one or more processors. The same applies to the memory 223. In this specification, "processor" is not limited to a processor in the narrow sense that executes processing in a stored-program manner, but may include hardwired circuits such as ASICs and FPGAs. Therefore, the term "processor" can also be interpreted as a processing circuit whose processing is predefined by computer-readable code and / or hardwired circuits. The training device 200 may be divided into multiple units according to function.
[0041] The input device 224 is a keyboard, touch panel, etc., and accepts operation by the user (such as the person in charge of managing the training device 200). The display 225 displays the training status by the training device 200. The network controller 226 is configured to communicate with the outside of the training device 200 (simulator 203) according to the control of the processor 221. The bus 227 connects the components of the training device 200 so that they can communicate with each other.
[0042] The simulator 203 transmits a normal task list (not shown) and an interrupt task list to the training device 200. The normal task list and the interrupt task list are used to train the reinforcement learning model 600. The normal task list is a list that defines multiple virtual normal tasks, not actual normal tasks. Similarly, the interrupt task list is a list that defines virtual interrupt tasks, not actual interrupt tasks.
[0043] Alternatively, the training device 200 may acquire tasks (normal tasks or interrupt tasks) through user operations on the input device 24, instead of receiving tasks (normal tasks or interrupt tasks) from the simulator 203.
[0044] <Warehouse Configuration> Figure 4 is a schematic diagram of automated warehouse 1. Figure 4 shows a plan view of automated warehouse 1 from above. In the example shown in Figure 4, automated warehouse 1 includes five vertical aisles 411-415, five storage units 421-425, one horizontal aisle 43, five shipping ports 441-445, five buffer units 451-455, and three shuttles 51-53. However, the number of components other than the horizontal aisle 43 is not limited to these numbers and can be any number of two or more. Also, shuttles 51-53 are collectively referred to as the "shuttle group 50".
[0045] The vertical passages 411 to 415 are passages that extend in the vertical direction (direction V in Figure 4). Each of the vertical passages 411 to 415 is restricted to carrying only one shuttle at a time. The vertical passages 411 to 415 correspond to the "multiple first passages" as described in this disclosure.
[0046] Storage units 421 to 425 are located on either side of the vertical aisles 411 to 415, respectively. Each of the storage units 421 to 425 contains multiple storage spaces 46. Each of the multiple storage spaces 46 can either store only one package L or no packages L at all. In other words, two or more packages L are never stored in the same storage space 46.
[0047] The lateral passage 43 is a passage that extends in the lateral direction (direction H in Figure 4). The lateral passage 43 connects two adjacent shipping ports from among the shipping ports 441 to 445 on the upper side of Figure 4, and connects two adjacent vertical passages from among the vertical passages 411 to 415 on the lower side. Two or more shuttles may travel in the lateral passage 43 as long as they do not collide with each other. The lateral passage 43 corresponds to the "second passage" in this disclosure.
[0048] Shipping ports 441-445 are the locations (shipping locations) from which packages L are shipped. Shipping ports 441-445 may also serve as receiving ports.
[0049] Buffer units 451 to 455 are each located near the shipping ports 441 to 445. Buffer units 451 to 455 temporarily store the goods L that have been transported to the corresponding shipping port among the shipping ports 441 to 445 until they are shipped. Buffer units 451 to 455 do not necessarily have to be provided in the automated warehouse 1.
[0050] Shuttles 51-53 are transport devices (shuttle carts) that transport cargo L. The shuttles may also be called agents. Shuttles 51-53 process the task of transporting cargo L according to transport commands from the task management device 2. Specifically, according to transport commands from the task management device 2, shuttles 51-53 retrieve cargo L from the designated storage space 46 (collection location) of the designated storage unit 421-425 among the storage units 421-425, and transport the retrieved cargo L to the designated shipping port (shipping location) among the shipping ports 441-445.
[0051] In the situation shown in Figure 4, shuttle 51 is transporting cargo L in the horizontal passage 43. Shuttle 52 has completed its task of transporting cargo L to the shipping port 444 and is in a "free" state with no tasks to perform. Shuttle 53 is transporting cargo L in the vertical passage 415.
[0052] The time required for shuttles 51-53 to move from one node to an adjacent node (i.e., to move one square in the diagram) is referred to as "1 time step." A 1 time step is also called a "unit of time." The processing cycle of the task management device 2 and training device 200 in this embodiment is 1 time step.
[0053] In the following, the area occupied by a single shuttle will also be referred to as a "square." There are 15 squares in the horizontal aisle 43. Also, since there is a constraint that only one shuttle may travel through each of the vertical aisles 411-415 at a time, each of the vertical aisles 411-415 is considered to be "1 square." Furthermore, each of the shipping ports 441-445 is also "1 square." Therefore, in Figure 4, the area in which shuttles 51-53 can move is 25 squares. These 25 squares are also represented by coordinates.
[0054] The following will explain the process in two phases: the learning phase and the application phase. The learning phase is the phase in which the training device 200 trains the reinforcement learning model 500. The application phase is the phase in which the task management device 2 controls the shuttle group 50 using the trained reinforcement learning model 500 (reinforcement learning model 600). First, the application phase will be explained.
[0055] <Normal task information and interrupt task information during the utilization phase> Figure 5 is a diagram illustrating the normal task information and interrupt task information in the utilization phase. In the example in Figure 5, the normal task information 243 is stored in queue 233, and the interrupt task information 244 is stored in stack 234.
[0056] First, let's explain the normal task information 243. The queue 233 where the normal task information 243 is stored has the form of a parallel task queue, and in this example, it contains five task queues. Task queues TQ1 to TQ5 are provided corresponding to the shipping ports 441 to 445, respectively. Each task queue TQ1 to TQ5 may be said to correspond to a "waiting line".
[0057] For each task queue TQ1 to TQ5, tasks are processed in the order shown at the bottom of Figure 5. Tasks are normally removed from task information 243 when they are assigned to any shuttle. When a shuttle is assigned a task, it processes that task and becomes free once processing is complete. Below, we assume that each of task queues TQ1 to TQ5 initially has 10 normal tasks.
[0058] Each normal task and each interrupt task includes four pieces of information, for example, the pickup location of package L, the shipping location of package L, the pickup time of package L, and the task priority. These four pieces of information are represented as a tuple (pickup location, shipping location, pickup time, priority) in Figure 5. The four pieces of information for normal tasks and the four pieces of information for interrupt tasks are also referred to as the features of normal tasks and the features of interrupt tasks, respectively.
[0059] The collection location is indicated by a number from 0 to 4, representing the vertical passages 411-415 where the storage space 46 for collecting (where package L is stored) is located. In other words, the collection location is 5-dimensional information.
[0060] The shipping location refers to the shipping dock from which package L is shipped, indicated by numbers 1 to 5. Since there is a one-to-one correspondence between task queues and shipping docks, the shipping locations within the same task queue are the same. In other words, the shipping location is also 5-dimensional information. Furthermore, the shipping docks indicated by numbers 1 to 5 correspond to the "target destination location" in this disclosure. For example, the target destination location for task queue TQ1 at (1,0,7,2) is shipping dock 441.
[0061] Collection time is the time the shuttle occupies the vertical aisle. More specifically, collection time is the sum of the time it takes for the shuttle to reach the designated storage space 46 via the vertical aisle from the horizontal aisle 43, the time it takes for the shuttle to collect the cargo L from the designated storage space 46, and the time it takes for the shuttle to return to the horizontal aisle 43 via the vertical aisle from the storage space 46. The location of the cargo L storage space 46 in each of the vertical aisles 411-415 can be determined from the collection time. Collection time is one-dimensional information.
[0062] Priority refers to the order in which each task is executed. In the example in Figure 5, a smaller absolute value of the priority indicates a higher priority. Tasks with higher priority are processed earlier than tasks with lower priority (they are placed at the front of each task queue, at the bottom of the figure, to ensure faster processing). Priority is one-dimensional information.
[0063] Therefore, the features of one normal task (normal task information) and the features of one interrupted task (interrupted task information) represent 12 dimensions (= 5 + 5 + 1 + 1) of information.
[0064] Even if an interrupt task occurs, the task management device 2 does not immediately add the interrupt task to the normal task information 243. In other words, the task management device 2 does not immediately have the shuttle group 50 execute the interrupt task. The task management device 2 temporarily stores the interrupt task as interrupt task information 244 in the stack 234. Then, the task management device 2 postpones the execution of adding the interrupt task information 244 to the normal task information 243 (the refresh process described later) until an appropriate time.
[0065] In the following, at least a portion of the normal task information 243, at least a portion of the interrupt task information 244, and the location information 245 are collectively referred to as "parameter information." At least a portion of the normal task information 243 is, for example, at least one normal task that can be executed in the immediate future (high priority). In the example in Figure 5, at least a portion of the normal task information 243 is the feature quantities of the 10 tasks enclosed by the thick lines. Also, as mentioned above, the feature quantities of one task are 12-dimensional information. Therefore, at least a portion of the normal task information 243 is 120-dimensional (12 × 10) information.
[0066] At least a portion of the interrupt task information 244 is, for example, at least one interrupt task that can be executed most recently (high priority). In the example in Figure 5, at least a portion of the interrupt task information 244 is the feature quantities of the two tasks enclosed by the thick lines. Also, as mentioned above, the feature quantities of one task are 12-dimensional information. Therefore, at least a portion of the interrupt task information 244 is 24-dimensional (12 × 2) information.
[0067] Furthermore, in the example shown in Figure 5, the priorities of interrupt tasks P1 to P3, as indicated by interrupt task information 244, are -1, -1, and -2, respectively. Therefore, the priorities of interrupt tasks P1 and P2 are higher than the priority of interrupt task P3.
[0068] Note that interrupt tasks P1 and P2 are given the same priority. As shown in Figure 6 below, the interrupt task information 244 is added to the normal task information 243 at the same target location as the interrupt task. In other words, each interrupt task is stored in the task queue at the same target location as the interrupt task. Therefore, since interrupt tasks P1 and P2 are stored in separate normal tasks, their priorities are given the same priority.
[0069] The position information 245 indicates the future position of the shuttle group 50 for executing tasks in each task queue. In other words, the position information 245 indicates the position at each of L (where L is an integer greater than or equal to 1) time steps in the future from a predetermined timing (for example, the present moment). Here, L = 50.
[0070] The task management device 2 generates and updates position information 245 using a predetermined algorithm, for example, by using the normal task information 243. The predetermined algorithm may be, for example, a greedy algorithm. Alternatively, the predetermined algorithm may be one that assigns the highest priority task from the task queue with the most remaining tasks at that time to a freed shuttle.
[0071] Furthermore, as explained in Figure 4, the area in which shuttles 51-53 can move is 25 squares. Therefore, the position information for one time step is 25-dimensional information. Consequently, the position information for 50 time steps is 1250-dimensional information.
[0072] In other words, the parameter information is 1394-dimensional (=1250+120+24) information. The task management device 2 inputs the 1394-dimensional parameter information into the reinforcement learning model 500. The reinforcement learning model 500 outputs a result (output result) indicating whether or not to prioritize the interrupt task over the normal task at each time step. That is, the task management device 2 decides whether or not to prioritize the interrupt task over the normal task based on the parameter information. This output result is 2-dimensional information because it indicates whether or not to prioritize the interrupt task over the normal task.
[0073] If the output indicates that an interrupt task should take priority, the task management device 2 performs the following refresh process. Figure 6 is a diagram illustrating the state after the refresh process has been performed. The refresh process will be explained using Figures 5 and 6.
[0074] The refresh process includes the following addition and deletion processes. The addition process adds the interrupt task information 244 from stack 234 to the normal task information 243 in queue 233 (stores the interrupt task in the task queue). The interrupt task information 244 is added to the normal task information 243 and the normal task information 243 is updated. Furthermore, the addition process adds the interrupt task corresponding to the interrupt task information 244 so that it is executed with higher priority than the normal task corresponding to the normal task information 243. The deletion process deletes the interrupt task information 244 from stack 234.
[0075] As described above, the task management device 2 adds the interrupt task information 244 to the normal task information 243 with the same target location as the interrupt tasks P1 to P3 (stores it in the task queue). Therefore, for example, since the target location of interrupt tasks P2 and P3 is "0", it is stored in the task queue TQ1.
[0076] As described above, stack 234 employs a last-in, first-out (LIFO) configuration. Interrupt tasks are assigned higher priority and are stored in stack 234 earlier the earlier they occur. In other words, interrupt tasks are stored in the task queue in order of priority, starting with those that occur later, and interrupt tasks that occur earlier are stored in the task queue after those that occur later have been stored, with the highest priority given to them. Therefore, due to the LIFO configuration of stack 234, interrupt tasks that occur earlier (higher priority) are executed by the shuttle group 50 with higher priority than interrupt tasks that occur later (lower priority).
[0077] Furthermore, if the output of the reinforcement learning model 500 indicates that the interrupt task should not be prioritized, the task management device 2 does not perform a refresh process. In other words, in this case, the task management device 2 performs a process that retains the interrupt task information 244 in the stack 234.
[0078] Task management device 2 operates shuttles 51-53 to complete all tasks in queue 233. However, task management device 2 does not use the interrupt task information 244 of stack 234 in the operation of shuttles 51-53. Therefore, for assigning tasks to each shuttle, task management device 2 only needs to use the normal task information 243 of queue 233.
[0079] <Functional block diagram of the task management system> Figure 7 is a functional block diagram of the task management device 2. In the example shown in Figure 7, the task management device 2 mainly comprises a queue 233, a stack 234, a memory area 235, a control unit 240, a shuttle control unit 241, and a reinforcement learning model 500. The memory area 235 stores position information 245. The control unit 240 performs overall control of the task management device 2. The shuttle control unit 241 controls the shuttle group 50. Note that the control unit 240 and the shuttle control unit 241 may be integrated.
[0080] As described above, the integrated management device 3 outputs multiple normal tasks to the task management device 2. The task management device 2 stores the multiple normal tasks as normal task information 243 in the queue 233. In addition, when an interrupt task occurs, the integrated management device 3 outputs the interrupt task to the task management device 2. Each time the task management device 2 receives an interrupt task, it stores the interrupt task as interrupt task information 244 in the stack 234.
[0081] The control unit 240 inputs the normal task information 243, the interrupt task information 244, and the position information 245 to the reinforcement learning model 500. If the output of the reinforcement learning model 500 indicates that the interrupt task should take priority, the task management device 2 executes the refresh process described above (see dashed line in Figure 7). The shuttle control unit 241 also operates the shuttle group based on the normal task information 243. The position information 245 is updated along with the update of the normal task information 243.
[0082] <Flowchart of the utilization phase> Figure 8 is a flowchart showing the main processes of the task management device 2. For example, when the task management device 2 receives a start signal from the integrated management device 3 indicating the start of a task, the task management device 2 starts the processes shown in Figure 8. Note that this start signal includes multiple normal tasks.
[0083] First, in step S2, the task management device 2 performs initial processing. Here, initial processing is, for example, the process of storing the multiple normal tasks included in the start signal as normal task information 243 in the queue 233. Furthermore, initial processing includes the process of generating position information 245. Specifically, the task management device 2 estimates the future positions of the shuttle group 50 when processing the multiple normal tasks received by the task management device 2 based on a predetermined algorithm. Based on this estimation result, the task management device 2 generates position information 245 for the L (for example, 50) time steps in the future.
[0084] Next, in step S4, the task management device 2 determines whether or not an interrupt task has occurred. In other words, in step S4, the task management device 2 determines whether or not it has received an interrupt task from the integrated management device 3.
[0085] If an interrupt task occurs (YES in step S4), the task management device 2 stores the interrupt task information 244 of the interrupt task in the stack 234. Then, the process proceeds to step S8. If no interrupt task occurs (NO in step S4), the process proceeds to step S8.
[0086] Next, in step S8, the task management device 2 inputs parameter information (normal task information 243, interrupt task information 244, and position information 245) to the reinforcement learning model 500. The task management device 2 obtains the output result of the reinforcement learning model 500 based on this input.
[0087] In step S10, the task management device 2 determines whether or not to perform a refresh process based on the output of the reinforcement learning model 500. If the output indicates that the reinforcement learning model 500 prioritizes the interrupt task, the task management device 2 decides to perform a refresh process. On the other hand, if the output indicates that the reinforcement learning model 500 does not prioritize the interrupt task, the task management device 2 decides not to perform a refresh process.
[0088] If a refresh process is to be performed (YES in step S10), the task management device 2 performs the refresh process described above in step S12. On the other hand, if a refresh process is not to be performed (NO in step S10), the task management device 2 leaves the interrupt task information 244 in the stack 234 in step S14. When the processing in step S12 or step S14 is completed, the process proceeds to step S16.
[0089] In step S16, the task management device 2 operates the shuttle group 50 for one time step to complete all normal tasks and all interrupt tasks stored in queue 233. The task management device 2 updates the position information 245 and normal task information 243 according to the completion of one time step of operation for each shuttle 51 to 53. For example, the task management device 2 updates the normal task information 243 by deleting tasks that have been processed (assigned to shuttles). Furthermore, the task management device 2 updates the position information 245 based on the updated normal task information 243.
[0090] Next, in step S18, the task management device 2 determines whether all tasks have been completed. If not all tasks have been completed (NO in step S18), the process returns to step S4. If all tasks have been completed (YES in step S18), the shuttle control process ends.
[0091] Steps S4 to S16 correspond to the "second predetermined processing" in this disclosure. The task management device 2 repeatedly executes steps S4 to S16 in a cycle of one time step.
[0092] <Normal tasks and interrupt tasks in the learning phase> Next, the learning phase will be described. In the learning phase, the training device 200 virtually generates multiple normal tasks and multiple interrupt tasks, and virtually operates the shuttle group 50 of the automated warehouse 1. The training device 200 trains the reinforcement learning model 600 based on these virtual normal tasks and virtual interrupt tasks. Therefore, the training device 200 does not need to be connected to the automated warehouse 1. Furthermore, the training device 200 is shown the virtual normal tasks and interrupt tasks by the interrupt task list and normal task list from the simulator 203.
[0093] Furthermore, one episode is composed of multiple virtual normal tasks and multiple virtual interrupt tasks. The training device 200 then trains the reinforcement learning model 600 by processing P (where P is an integer greater than or equal to 2) episodes. P is a very large number, for example, 100,000.
[0094] In this disclosure, examples are shown where the normal tasks and interrupt tasks in the utilization phase are identical to the normal tasks and interrupt tasks in the learning phase, respectively. However, in reality, the normal tasks and interrupt tasks in the utilization phase are different from those in the learning phase, respectively. Furthermore, the normal tasks and interrupt tasks in the learning phase are represented as tuples, similar to the normal tasks and interrupt tasks in the utilization phase.
[0095] Furthermore, during the learning phase, the virtual automated warehouse 1 is also referred to as the "virtual automated warehouse." When describing the virtual automated warehouse, the reference numbers for automated warehouse 1 shown in Figure 4 are used when referring to the components of automated warehouse 1.
[0096] Furthermore, the virtual shuttle may also be referred to as a "virtual shuttle" or "virtual transport device," and the virtual cargo L may also be referred to as a "virtual cargo." Additionally, the virtual normal task and virtual interrupt task may be referred to as "virtual normal task" and "virtual interrupt task," respectively. In the following, "virtual normal task" and "virtual interrupt task" may simply be referred to as "normal task" and "interrupt task."
[0097] Figure 9 is a diagram illustrating the interrupt task list 390 described above. The interrupt task list 390 includes information indicating interrupt tasks P1 to P3. Furthermore, the interrupt task list 390 includes timing information 391. The timing information 391 indicates the timing of occurrence for each of the multiple interrupt tasks (interrupt tasks P1 to P3).
[0098] In the example in Figure 9, interrupt tasks P1 to P3 are specified to occur at future timings t3, t5, and t7, respectively. These future timings are, for example, the time elapsed since the start of processing for the episode being processed.
[0099] Then, during the learning process, when the training device 200 reaches the timing for an interrupt task to occur (step S104 described later), it stores the interrupt task corresponding to that timing in the stack 334.
[0100] Figure 10 is a diagram illustrating the normal task information and interrupt task information during the learning phase. In the example in Figure 10, the normal task information 343 is stored in queue 333, and the interrupt task information 344 is stored in stack 334. The example in Figure 10 shows the case where all interrupt tasks P1 to P3 are stored.
[0101] First, let's explain the normal task information 343. The queue 333 where the normal task information 343 is stored has the form of a parallel task queue, and in this example, it contains five task queues. Task queues TQ1 to TQ5 are provided corresponding to the shipping ports 441 to 445, respectively. Each task queue TQ1 to TQ5 may be considered equivalent to a "waiting line".
[0102] For each task queue TQ1 to TQ5, tasks are processed in the order shown at the bottom of Figure 10. Normal tasks are removed from the normal task information 343 once they are assigned to any virtual shuttle.
[0103] Each normal task and each interrupt task includes four pieces of information, for example, the virtual package's pickup location, virtual package's shipping location, virtual package's pickup time, and the virtual task's priority. These four pieces of information are represented as a tuple (pickup location, shipping location, pickup time, priority) in Figure 10. The four pieces of information for normal tasks and the four pieces of information for interrupt tasks are also referred to as the normal task features and the interrupt task features, respectively. The simulator 203 randomly determines the normal task features and the interrupt task features.
[0104] The collection location is a representation of the vertical aisle (corresponding to vertical aisles 411-415 mentioned above) of the virtual automated warehouse where the virtual packages are collected (where they are stored), indicated by a number from 0 to 4. In other words, the collection location is 5-dimensional information.
[0105] The shipping location is indicated by numbers 1 through 5 as the shipping dock of the virtual automated warehouse from which the virtual goods are shipped. Since there is a one-to-one correspondence between the task queue and the shipping dock, the shipping locations for the same task queue are the same. In other words, the shipping location is also 5-dimensional information. Furthermore, the shipping docks indicated by numbers 1 through 5 correspond to the "target location" in this disclosure. For example, the target location of (1,0,7,2) in task queue TQ1 is shipping dock 441 in the virtual automated warehouse.
[0106] Collection time is the time that the virtual shuttle occupies the vertical aisle. More specifically, collection time is the sum of the time it takes for the shuttle to reach the designated storage space 46 via the horizontal aisle and vertical aisle, the time it takes for the shuttle to collect the virtual cargo from the designated storage space 46, and the time it takes for the shuttle to return to the horizontal aisle 43 via the vertical aisle from the storage space 46. The location of the virtual cargo storage space 46 in each vertical aisle 411-415 can be determined from the collection time. Collection time is one-dimensional information.
[0107] Priority refers to the priority given to the execution of each task. In the example in Figure 10, a smaller absolute value of the priority indicates a higher priority. Priority is one-dimensional information.
[0108] Therefore, the features of one normal task and the features of one interrupt task represent 12 dimensions (= 5 + 5 + 1 + 1) of information.
[0109] Even when the timing for an interrupt task is reached, the training device 200 does not immediately add the interrupt task to the normal task information 343. In other words, the training device 200 does not immediately have the virtual shuttle execute the interrupt task. The training device 200 temporarily stores the interrupt task as interrupt task information 344 in the stack 334. In this way, the training device 200, like the task management device 2, postpones adding the interrupt task information 344 to the normal task information 343 (the refresh process described later) until an appropriate time.
[0110] In the following, at least a portion of the normal task information 343, at least a portion of the interrupt task information 344, and the location information 345 are collectively referred to as "parameter information." At least a portion of the normal task information 343 is, for example, at least one normal task that can be executed in the immediate vicinity. In the example in Figure 10, at least a portion of the normal task information 343 consists of the feature quantities of the 10 tasks enclosed by the thick lines. Also, as mentioned above, the feature quantity of one task is 12-dimensional information. Therefore, at least a portion of the normal task information 343 consists of 120-dimensional (12 × 10) pieces of information.
[0111] At least a portion of the interrupt task information 344 is, for example, at least one interrupt task that may be executed in the immediate future. In the example in Figure 10, at least a portion of the interrupt task information 344 is the feature quantities of the two tasks enclosed by the thick lines. Also, as mentioned above, the feature quantities of one task are 12-dimensional information. Therefore, at least a portion of the interrupt task information 344 is 24-dimensional (12 × 2) information.
[0112] Furthermore, in the example shown in Figure 10, the priorities of interrupt tasks P1 to P3, as indicated by the interrupt task information 344, are -1, -1, and -2, respectively. Therefore, the priorities of interrupt tasks P1 and P2 are higher than the priority of interrupt task P3.
[0113] Note that interrupt tasks P1 and P2 have the same priority. As shown in Figure 11 below, each interrupt task is stored in the task queue of the same target location as the interrupt task. Therefore, since interrupt tasks P1 and P2 are stored in different task queues, their priorities are considered to be the same.
[0114] Location information 345 indicates the future position of the virtual shuttle for virtually executing tasks in each task queue. In other words, location information 345 indicates the position at each of L (where L is an integer greater than or equal to 1) time steps in the future from a predetermined timing (for example, the present moment). Here, L = 50.
[0115] In this embodiment, the postponement of a virtual interrupt task is modeled by a Markov decision process, similar to the utilization phase. The Markov decision process is a mathematical model that assumes the Markov property of state transitions due to the actions of the player (in this case, the training device 200), and is represented by M=(S, A, P, γ, R).
[0116] Here, S is, for example, the set of states of a virtual automated warehouse. In the following, states s and s' are used, where s∈S and s'∈S. A is the set of actions, which determine whether or not to perform the refresh process described later. In the following, action a is used, where a∈A.
[0117] Furthermore, P can also be expressed as P(s'|s, a). P(s'|s, a) is the probability that the training device 200 will transition to the next state s' when it selects action a in state s. The discount rate γ is the discount rate used in Q-learning. The reward R is a function that returns the reward for state s. For example, reward R is a function that decreases the reward, as in step S116 described later. The determined reward is stored, for example, in a predetermined area of memory 223 (hereinafter also referred to as the "reward area"). The training device 200 then updates (increases or decreases) the reward stored in the reward area.
[0118] The training device 200 updates the position information 345 using, for example, the same predetermined algorithm as the task management device 2, using the normal task information 343.
[0119] Furthermore, as explained in Figure 4, the area in which the virtual shuttle can virtually move is 25 squares. Therefore, the position information for one time step is 25-dimensional information. Consequently, the position information for 50 time steps is 1250-dimensional information.
[0120] In other words, the parameter information is 1394-dimensional (=1250+120+24) information. The training device 200 inputs this 1394-dimensional parameter information into the reinforcement learning model 600. The reinforcement learning model 600 outputs a result (output result) indicating whether or not to prioritize the interrupt task over the normal task. Since this output result indicates whether or not to prioritize the interrupt task over the normal task, it is 2-dimensional information.
[0121] If the output indicates that the interrupt task should take priority, the training device 200 performs the following refresh process. Figure 11 is a diagram illustrating the state after the refresh process has been performed. The refresh process will be explained using Figures 10 and 11.
[0122] The refresh process includes the following addition and deletion processes. The addition process adds the interrupt task information 344 from stack 334 to the normal task information 343 in queue 333 (stores it in the task queue). The interrupt task information 344 is added to the normal task information 343 and the normal task information 343 is updated. Furthermore, the addition process adds the interrupt task corresponding to the interrupt task information 344 so that it is executed with higher priority than the normal task corresponding to the normal task information 343. The deletion process deletes the interrupt task information 344 from stack 334.
[0123] As described above, the training device 200 adds the interrupt task information 344 to the normal task information 343 for the same target location as the interrupt tasks P1 to P3 (stores it in the task queue). Therefore, for example, since the target location for interrupt tasks P2 and P3 is "0", it is stored in the task queue TQ1.
[0124] Furthermore, stack 334 employs a Last-In, First-Out (LIFO) configuration. Interrupt tasks are assigned higher priority and are stored in stack 334 first the earlier they occur. In other words, "later occurring interrupt tasks" are stored in the task queue in order of highest priority, and "earlier occurring interrupt tasks" are stored in the task queue after "later occurring interrupt tasks" have been stored, in order of highest priority. Therefore, due to the Last-In, First-Out configuration of stack 334, earlier occurring (higher priority) interrupt tasks are executed in the virtual shuttle with higher priority than later occurring (lower priority) interrupt tasks.
[0125] Furthermore, if the output of the reinforcement learning model 600 indicates that the interrupt task should not be prioritized, the training device 200 does not perform a refresh process. In other words, in this case, the training device 200 performs a process that retains the interrupt task information 344 in the stack 334.
[0126] The training device 200 operates the virtual shuttle to complete all tasks in queue 333. However, the training device 200 does not use the interrupt task information 344 of stack 334 in the operation of the virtual shuttle. Therefore, for assigning tasks to each virtual shuttle, the training device 200 only needs to use the normal task information 343 of queue 333.
[0127] <Functional block diagram of the task management system> Figure 12 is a functional block diagram of the training device 200. In the example shown in Figure 12, the training device 200 mainly consists of a queue 333, a stack 334, a memory area 335, an arithmetic unit 250, and a reinforcement learning model 600. The memory area 335 stores position information 345.
[0128] As described above, the simulator 203 outputs a normal task list to the training device 200. The training device 200 stores the multiple normal tasks shown in the normal task list as normal task information 343 in the queue 333. The simulator 203 also outputs an interrupt task list 390 to the training device 200. Based on the timing information 391 shown in the interrupt task list 390, the training device 200 stores the interrupt tasks shown in the interrupt task list 390 as interrupt task information 344 in the queue 333.
[0129] The arithmetic unit 250 inputs the normal task information 343, the interrupt task information 344, and the position information 345 to the reinforcement learning model 600. If the output of the reinforcement learning model 600 indicates that the interrupt task should take priority, the training device 200 executes the refresh process described above (see dashed line in Figure 12). The arithmetic unit 250 also performs actions such as virtual shuttle operation based on the normal task information 343. The position information 345 is updated along with the update of the normal task information 343.
[0130] <Flowchart of the Learning Phase> Figures 13 and 14 are flowcharts showing the main processes of the training device 200. For example, when the training device 200 receives a start signal from the simulator 203 indicating the start of training, the training device 200 starts the process shown in Figure 13.
[0131] First, in step S102, the training device 200 obtains the normal task list for each of the episodes (the P episodes mentioned above) and the interrupt task list 390 for each of the episodes. Then, it stores the multiple normal tasks indicated by the normal task list of one episode as normal task information 343 in the queue 333. Furthermore, the training device 200 generates position information 345 using the predetermined algorithm described above.
[0132] Next, in step S104, the training device 200 determines whether the timing for executing the decision process in step S104 is the occurrence timing indicated by the timing information 391.
[0133] If the decision timing is the occurrence timing indicated by the timing information 391 (YES in step S104), then in step S106, the training device 200 stores the interrupt task in the stack 334.
[0134] Furthermore, the processing in steps S104 to S122 corresponds to the "first predetermined processing" in this disclosure. The training device 200 repeatedly executes steps S104 to S122 in a period of one time step as described above.
[0135] For example, when the first predetermined process is executed in the third cycle, that is, when the decision timing reaches timing t3 (see Figure 9), the interrupt task P1 corresponding to timing t3 is stored in the stack 334. Similarly, when the first predetermined process is executed in the fifth cycle, that is, when the decision timing reaches timing t5 (see Figure 9), the interrupt task P2 corresponding to timing t5 is stored in the stack 334. Furthermore, when the first predetermined process is executed in the seventh cycle, that is, when the decision timing reaches timing t7 (see Figure 9), the interrupt task P3 corresponding to timing t7 is stored in the stack 334.
[0136] If it is determined that the timing of the decision is not the timing of the event (NO in step S104), or after the completion of the processing in step S106, the process proceeds to step S108.
[0137] Next, in step S108, the training device 200 inputs parameter information (normal task information 343, interrupt task information 344, and position information 345) to the reinforcement learning model 600. The training device 200 then obtains the output result of the reinforcement learning model 600 based on this input.
[0138] In step S110, the training device 200 determines whether or not to perform a refresh process based on the output of the reinforcement learning model 600. If the output indicates that the reinforcement learning model 600 prioritizes the interrupt task, the training device 200 decides to perform a refresh process. On the other hand, if the output indicates that the reinforcement learning model 600 does not prioritize the interrupt task, the training device 200 decides not to perform a refresh process.
[0139] If the refresh process is not performed (NO in step S110), the training device 200 leaves the interrupt task information 344 in the stack 334 in step S118.
[0140] If a refresh process is to be performed (YES in step S110), the training device 200 determines whether or not an interrupt task (interrupt task information 344) is stored in the stack 334.
[0141] Here, there are cases where the output indicates that the interrupt task takes priority even though the interrupt task information 344 is not stored in the stack 334 (when it is determined to be YES in step S110 and NO in step S112). However, the output in this case is an incorrect result and therefore undesirable.
[0142] Therefore, if the output result of the reinforcement learning model 600 is the undesirable output result described above (if it is determined to be YES in step S110 and NO in step S112), in step S116, the training device 200 reduces the reward described below. In this embodiment, the task management device 2 receives a negative reward. The negative reward is a predetermined negative value.
[0143] In step S114, the training device 200 performs a refresh process. Once the processes in steps S114, S116, and S118 are completed, the process proceeds to step S120.
[0144] In step S120, the training device 200 operates the virtual shuttle to complete all normal tasks and all interrupt tasks stored in queue 333. The training device 200 performs one time step of operations on the virtual shuttle. The training device 200 updates the position information 345 and normal task information 343 in response to the completion of one time step of operations on the virtual shuttle. For example, the training device 200 updates the normal task information 343 by deleting tasks that have been processed, and updates the position information 345 based on the updated normal task information 243.
[0145] Next, in step S122, the training device 200 determines whether the current episode has ended. If the current episode has not ended (NO in step S122), the process returns to step S104. If the current episode has ended (YES in step S122), the process proceeds to step S124.
[0146] When an episode (the current episode) is completed, in step S124, the training device 200 calculates (determines) the total reward. The training device 200 then updates the total reward in the reward area to match the calculated reward. The training device 200 calculates the total reward using the negative reward from step 116 and the reward calculated by formula (A) described below.
[0147] Next, in step S126, the training device 200 trains the reinforcement learning model 600 to maximize the total reward. For example, from step S104 to step S126, the training device 200 trains the reinforcement learning model 600 using Q-learning.
[0148] Training the reinforcement learning model 600 includes, for example, the process of updating the parameters of the reinforcement learning model 600. Specifically, the training device 200 updates the parameters of the reinforcement learning model 600 to increase the reward. Specifically, the training device 200 updates the parameters of the reinforcement learning model 600 to maximize the value (Q value) of the virtual shuttle's action at each time step.
[0149] Next, in step S128, the training device 200 determines whether all episodes have been completed. If not all episodes have been completed (NO in step S128), the process returns to step S104. On the other hand, if all episodes have been completed (YES in step S128), the learning phase ends.
[0150] <Method for calculating total compensation> Next, the formula for calculating the total reward in step S124 will be explained. In this embodiment, the total reward is the sum of the negative reward in step S116 within one episode and the positive reward based on the velocity parameters (the reward calculated by formula (A) described later). The velocity parameters are, for example, parameters relating to the processing speed of all normal tasks and all interrupt tasks in one episode (hereinafter also referred to as the "target episode") that was determined to have ended in step S122.
[0151] The speed parameter includes a primary value and a secondary value. The primary value is based on the processing speed of all normal tasks and all interrupt tasks in the target episode. The secondary value is based on the processing speed of all interrupt tasks in the target episode.
[0152] Furthermore, the training device 200 calculates a reward based on the velocity parameter using the harmonic mean of the first and second values. The reason for using the harmonic mean will be explained later. Figure 15 is a diagram illustrating the formula for calculating the reward in step S124. In the example in Figure 15, the following formula is shown.
[0153]
number
[0154] Furthermore, in formula (A), V1 represents the processing speed of all normal tasks and all interrupt tasks in the target episode. V1 is calculated, for example, by dividing the sum of the number of normal tasks and the number of interrupt tasks in the target episode by the number of time steps required for the processing of all normal tasks and all interrupt tasks to be completed.
[0155] Furthermore, in formula (A), V2 represents the processing speed of all interrupt tasks in the target episode. V2 is calculated, for example, by dividing the total number of interrupt tasks in the target episode by the number of time steps required for the processing of all interrupt tasks to be completed.
[0156] Furthermore, Va, a, Vb, and b on the right-hand side of the above calculation formula are hyperparameters set by the user, etc. In other words, Va, a, Vb, and b are predetermined values. Note that Va and Vb are reference values used to measure the quality of the evaluation of V1 and V2, and are empirically standard processing speed values. Also, a and b are parameters that adjust the degree to which V1 and V2 influence the reward. For example, if V1 is 10 3 Order, V2 is 10 1 In the case of order values, setting a > b allows V1 and V2 to be evaluated with approximately the same degree of influence.
[0157] Furthermore, Figure 15 shows the first value M1 and the second value M2. The first value M1 and the second value M2 are values to which the sigmoid function has been applied. In Figure 15, the graphs of the first value M1 and the second value M2 are shown as examples.
[0158] As an example, Figure 16 shows the graph of the first value M1 when Va=3 and a=1.5. Also, as an example, Figure 17 shows the graph of the first value M1 when Va=3 and a=0.5. Designers and others should adjust the slope of the sigmoid function by adjusting a and b to equalize the influence of V1 and V2 on the reward. Figure 16 shows the graph of the first value M1 when a is set to 1.5. Figure 17 shows the graph of the first value M1 when a is set to 0.5.
[0159] Next, we will explain why the harmonic mean of the first value M1 and the second value is used in the above calculation formula (A). Figure 18 is a diagram illustrating the various means of V1 and V2. The thick line in Figure 18 represents the harmonic mean of V1 and V2. The thin line in Figure 18 represents the geometric mean of V1 and V2. The dashed line in Figure 18 represents the arithmetic mean of V1 and V2. However, the sum of V1 and V2 is assumed to be constant.
[0160] The reason for using the harmonic mean in the reward calculation formula is to make the reward calculated to be the highest when V1 and V2 are balanced (when the difference between V1 and V2 is small).
[0161] As indicated by the dashed line in Figure 18, the arithmetic mean value has the relationship shown in the following formula (B). Arithmetic mean value when V1 and V2 are not balanced = Arithmetic mean value when V1 and V2 are balanced (B) Further, as indicated by the thin line in Figure 18, the geometric mean value has the relationship shown in the following formula (C). Geometric mean value when V1 and V2 are not balanced < Geometric mean value when V1 and V2 are balanced (C) Further, as indicated by the thick line in Figure 18, the harmonic mean value has the relationship shown in the following formula (D). Harmonic mean value when V1 and V2 are not balanced ≪ Harmonic mean value when V1 and V2 are balanced (D) The management device of the comparative example to be compared with the task management device 2 of the present embodiment is as follows. The management device of the comparative example adds the interrupt task to the head of the queue of normal tasks at the time step next to the time step in which the interrupt task occurs.
[0162] That is, when an interrupt task occurs, the management device of the comparative example causes the shuttle to process the interrupt task immediately. In this case, V2 is highly likely to be faster, but the total task completion time (the time from the occurrence of the interrupt task to the completion of all tasks) is significantly longer (that is, V1 becomes slower), so the interrupt processing may greatly affect the processing of the entire automated warehouse.
[0163] That is, when V1 becomes extremely fast, V2 becomes slow, and conversely, when V1 becomes extremely slow, V2 may become fast. Therefore, V1 and V2 are in a trade-off relationship. In the present embodiment, it is preferable that the output result from the reinforcement learning model 500 is a result that balances V1 and V2.
[0164] Therefore, in training the reinforcement learning model 600, it is preferable that a large reward is given when V1 and V2 are balanced, and the smallest possible reward is given when V1 and V2 are not balanced. From this perspective, in this embodiment, a formula that uses the harmonic mean of V1 and V2 is adopted as the reward calculation formula (see formula (D) above).
[0165] <Summary> (1) In the management device of the comparative example described above, interrupt tasks are added to the front of the queue of normal tasks and processed first. However, with such a simple method, the interrupt processing may have a significant impact on the overall processing of the automated warehouse, such as by significantly increasing the total task completion time (the time from when an interrupt task occurs until all tasks are completed).
[0166] In contrast, the training device 200 of this embodiment determines the reward based on speed parameters related to the processing speed of all normal tasks and all interrupt tasks, and trains the reinforcement learning model 600 to maximize the reward. The training device 200 can train the reinforcement learning model 600 so that the timing of the refresh process is optimal. In other words, the training device 200 can train the reinforcement learning model 600 so that the processing speed of all normal tasks and all interrupt tasks is increased. Therefore, if the task management device 2 controls the shuttle of the automated warehouse 1 using the reinforcement learning model 600 (reinforcement learning model 500), the impact of interrupt tasks on the warehouse system 100 can be reduced.
[0167] (2) The training device 200 also determines the reward based on a first value M1, which is based on the processing speed of all normal tasks and all interrupt tasks, and a second value M2, which is based on the processing speed of all interrupt tasks. Thus, the training device 200 can train the reinforcement learning model 600 based on the first value M1 and the second value M2.
[0168] (3) As explained in Figure 18, the harmonic mean is large when there is a good balance between V1 and V2 (for example, when the difference between V1 and V2 is small). On the other hand, the harmonic mean is small when there is a poor balance between V1 and V2 (for example, when the difference between V1 and V2 is large). Taking this into consideration, the training device 200 determines the reward using the harmonic mean of V1 and V2. Therefore, the training device 200 can train the reinforcement learning model 500 so that there is a good balance between V1 and V2. Thus, if the task management device 2 controls the shuttle group 50 of the automated warehouse 1 using the reinforcement learning model 500, it can process tasks by increasing the processing speed of all normal tasks and all interrupt tasks in a balanced manner.
[0169] (4) The training device 200 can also calculate the reward using the above formula (A). (5) Also, as shown in step S116, the training device 200 may indicate that the output result of the reinforcement learning model 600 prioritizes the interrupt task over the normal task, even though the interrupt task information 344 is not stored in the stack 334 (NO in step S112 of Figure 13). In this case, the output result of the reinforcement learning model 600 can be said to be an incorrect result. Therefore, in this case, the training device 200 reduces the reward (adds a negative reward to the reward based on the speed parameter described above). Thus, the training device 200 can train the reinforcement learning model 600 so that it can suppress the output result of the reinforcement learning model 600 from being an incorrect result.
[0170] (6) The training device 200 also processes multiple episodes to train the reinforcement learning model 600 (see step S122 in Figure 14, etc.). Thus, the training device 200 can train the reinforcement learning model 600 more robustly.
[0171] (7) The training device 200 also obtains an interrupt task list 390 (see Figure 9) when starting multiple episodes. The interrupt task list 390 also includes the occurrence timing (timing information 391) for each of the multiple interrupt tasks. When the occurrence timing is reached, the training device 200 stores the interrupt task information corresponding to that occurrence timing in the stack 334. Therefore, the training device 200 can train the reinforcement learning model 600 by assuming virtual interrupt tasks without using the actual automated warehouse 1.
[0172] (8) As shown in Figure 5, the normal task information 343 includes the target location of the normal task. Also, as shown in Figure 10, the interrupt task information 344 includes the target location of the interrupt task and the priority of the interrupt task. Also, as shown in the interrupt tasks P2 and P3 in Figure 10, for two or more interrupt tasks with the same target location, the earlier the interrupt task occurs, the higher the priority is set. The training device 200 then adds the interrupt task information 344 to the normal task information 343 for the same target location as the interrupt task (see Figure 11). Thus, the training device 200 can train the reinforcement learning model 600 so that the virtual shuttle executes an interrupt task that occurs at a certain timing with higher priority than an interrupt task that occurs at a later timing.
[0173] <Variation> In the above-described embodiment, the task management device 2 is configured to control the shuttle group 50 using a reinforcement learning model 600 trained by the training device 200. However, the task management device 2 may also control the shuttle group 50 using a reinforcement learning model 600 trained by a method different from that of the above-described embodiment.
[0174] <Note> (Note 1) A training device for training a reinforcement learning model used to control an automated warehouse, The automated warehouse includes a conveying device that performs the task of transporting goods, The aforementioned task includes a normal task and an interrupt task that occurs after the occurrence of the normal task. The reinforcement learning model receives parameter information including information about the normal task, information about the interrupt task, and information about the future position of the transport device, and outputs a result indicating whether or not to prioritize the interrupt task over the normal task. The training device comprises a computing device and a memory, The aforementioned memory is A first area for storing information about the aforementioned normal task, It includes a second area for storing information about the interrupt task, The aforementioned computing device repeatedly executes the first predetermined process, The first predetermined process is, The process involves inputting the parameter information into the reinforcement learning model to obtain the output result of the reinforcement learning model, If the output result indicates that the interrupt task should be prioritized, the process involves adding the information of the interrupt task in the second area to the information of the normal task in the first area, and deleting the information of the interrupt task in the second area, so that the transport device executes the interrupt task with priority over the normal task. If the output result indicates that the interrupt task is not prioritized, the process includes retaining the information of the interrupt task in the second region, The aforementioned computing device is When all of the normal tasks and interrupt tasks in the first region have been processed by repeatedly executing the first predetermined process, a reward is determined based on speed parameters relating to the processing speed of all the normal tasks and the processing speed of all the interrupt tasks. A training device for training the reinforcement learning model so that the aforementioned rewards increase.
[0175] With this configuration, the computing unit determines the reward based on the speed parameter of the processing speed of all normal tasks and all interrupt tasks, and trains the reinforcement learning model to maximize the reward. Therefore, the reinforcement learning model can be trained to optimize the timing of the process of adding the information of the interrupt tasks in the second domain to the information of the normal tasks in the first domain. Consequently, if the task management device uses this reinforcement learning model to control the transport device of the automated warehouse, the impact of interrupt tasks on the automated warehouse can be reduced.
[0176] (Note 2) The aforementioned speed parameter is, A first value based on the processing speed of all the normal tasks and all the interrupt tasks, The training apparatus described in Appendix 1, including a second value based on the processing speed of all the interrupt tasks mentioned above.
[0177] With this configuration, the training device determines the reward based on a first value, which is based on the processing speed of all normal tasks and all interrupt tasks, and a second value, which is based on the processing speed of all interrupt tasks. Therefore, the reinforcement learning model can be trained based on the first and second values.
[0178] (Note 3) The training apparatus described in Appendix 2, wherein the calculation device calculates the reward using the harmonic mean of the first value and the second value.
[0179] Generally, the harmonic mean is large when there is a good balance between the first and second values (for example, when the difference between the first and second values is small). Conversely, the harmonic mean is small when there is a poor balance between the first and second values (for example, when the difference between the first and second values is large). Taking this into consideration, the computing unit uses the harmonic mean of the first and second values to determine the reward. Therefore, the computing unit can train the reinforcement learning model to achieve a good balance between the first and second values. Thus, if the task management device uses this reinforcement learning model to control the transport device of the automated warehouse, all normal tasks and all interrupt tasks can be processed in a balanced manner. Therefore, the training device can improve the processing capacity of the automated warehouse by the task management device.
[0180] (Note 4) The aforementioned computing device calculates the reward using the following formula:
[0181]
number
[0182] V1 represents the processing speed of all normal tasks and all interrupt tasks. V2 indicates the processing speed of all interrupt tasks mentioned above. Va, Vb, a, and b are predetermined values; the training device is described in Appendix 3.
[0183] The computing unit can determine the reward using the above formula. (Note 5) The training apparatus according to any one of the appendices 1 to 4, wherein the arithmetic unit reduces the reward if the output result indicates that the interrupt task takes priority even though the information of the interrupt task is not stored in the second area.
[0184] With this configuration, the reinforcement learning model can be trained so that it suppresses the tendency for the model's output to prioritize the interrupt task, even though the interrupt task information is not stored in the second domain.
[0185] (Note 6) An episode consists of all the normal tasks and all the interrupt tasks mentioned above. The training device according to any one of Appendix 1 to Appendix 5, wherein the computing device trains the reinforcement learning model by processing a plurality of the episodes.
[0186] With this configuration, the computing unit can train the reinforcement learning model more robustly by processing multiple episodes.
[0187] (Note 7) When the training device starts the episode, it obtains a list of multiple interrupt tasks from an external device. The aforementioned list includes the timing of occurrence for each of the multiple interrupt tasks, The training device as described in Appendix 6, wherein the calculation unit stores information of the interrupt task corresponding to the occurrence timing in the second area when the occurrence timing is reached.
[0188] With this configuration, the training device obtains a list containing the specified timing for each of the multiple interrupt tasks, allowing it to train a reinforcement learning model on a hypothetical interrupt task without using an actual automated warehouse.
[0189] (Note 8) The transport device performs the task by transporting the cargo to a target location among a plurality of target locations. The information of the normal task includes the target location of the normal task, The information of the interrupt task includes the target location of the interrupt task and the priority of the interrupt task. For two or more interrupt tasks with the same target location, the earlier the interrupt task occurs, the higher the priority is set. The training device as described in Appendix 7, wherein the arithmetic unit adds the information of the interrupt task to the information of the normal task at the same target location as the target location of the interrupt task.
[0190] With this configuration, a reinforcement learning model can be trained so that the transport device prioritizes the execution of interrupt tasks that occur at a certain time over interrupt tasks that occur at a later time.
[0191] (Note 9) An automated warehouse management device, The automated warehouse includes a conveying device that performs the task of transporting goods, The aforementioned task includes a normal task and an interrupt task that occurs after the occurrence of the normal task. The aforementioned control device is A control device for controlling the transport device, It has memory for storing reinforcement learning models, The reinforcement learning model receives parameter information including information about the normal task, information about the interrupt task, and information about the future position of the transport device, and outputs a result indicating whether or not to prioritize the interrupt task over the normal task. The aforementioned memory further, A third area for storing information about the aforementioned normal task, It includes a fourth area for storing information about the interrupt task, The control device repeatedly executes the second predetermined process, The second predetermined process is: When the aforementioned interrupt task occurs, the process involves storing information about the interrupt task in the fourth area. The process involves inputting the parameter information into the reinforcement learning model to obtain the output result of the reinforcement learning model, If the output result indicates that the interrupt task should be prioritized, the process involves adding the information of the interrupt task in the fourth region to the information of the normal task in the third region, and deleting the information of the interrupt task in the fourth region, so that the transport device executes the interrupt task with priority over the normal task. If the output result indicates that the interrupt task is not prioritized, the process of retaining the information of the interrupt task in the fourth region is performed, A management device including a process for operating the transport device to complete all of the normal tasks and all of the interrupt tasks in the third region.
[0192] (Note 10) The aforementioned reinforcement learning model is trained by the training device described in any one of the appendices 1 to 8, and the management device described in appendice 9.
[0193] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of this disclosure is indicated by the claims rather than by the description of the embodiments above, and all modifications within the meaning and scope equivalent to the claims are intended to be included. [Explanation of symbols]
[0194] 1 Automated warehouse, 2 Task management device, 3 Integrated management device, 21,221 Processor, 23,223 Memory, 24,224 Input device, 25,225 Display, 26,226 Network controller, 27,227 Bus, 43 Transverse aisle, 46 Storage space, 50 Shuttle group, 51,52,53 Shuttle, 100 Warehouse system, 150 Training system, 200 Training device, 203 Simulator, 231,251 Program, 233,333 Queue, 234,334 Stack, 235,335 Memory area, 240 Control unit, 241 Shuttle control unit, 243,343 Normal task information, 244,344 Interrupt task information, 245,345 Position information, 250 Arithmetic unit, 300 Management system, 390 Interrupt task list, 391 Timing information, 500,600 reinforcement learning models.
Claims
1. A training device for training a reinforcement learning model used in the control of an automated warehouse, The automated warehouse includes a conveying device that performs the task of transporting goods, The aforementioned task includes a normal task and an interrupt task that occurs after the occurrence of the normal task. The reinforcement learning model receives parameter information including information about the normal task, information about the interrupt task, and information about the future position of the transport device, and outputs a result indicating whether or not to prioritize the interrupt task over the normal task. The training device comprises a computing device and a memory, The aforementioned memory is A first area for storing information of the aforementioned normal task, It includes a second area for storing information about the interrupt task, The aforementioned computing device repeatedly executes the first predetermined process, The first predetermined process is, The process involves inputting the parameter information into the reinforcement learning model to obtain the output result of the reinforcement learning model, If the output result indicates that the interrupt task should be prioritized, the process involves adding the information of the interrupt task in the second region to the information of the normal task in the first region, and deleting the information of the interrupt task in the second region, so that the transport device executes the interrupt task with priority over the normal task. If the output result indicates that the interrupt task is not prioritized, the process includes retaining the information of the interrupt task in the second region, The aforementioned computing device is When all of the normal tasks and interrupt tasks in the first region have been processed by repeatedly executing the first predetermined process, a reward is determined based on speed parameters relating to the processing speed of all the normal tasks and the processing speed of all the interrupt tasks. A training device for training the reinforcement learning model so that the aforementioned rewards increase.
2. The aforementioned speed parameter is, A first value based on the processing speed of all the normal tasks and all the interrupt tasks, The training apparatus according to claim 1, further comprising a second value based on the processing speed of all interrupt tasks.
3. The training apparatus according to claim 2, wherein the calculation device calculates the reward using the harmonic mean of the first value and the second value.
4. The aforementioned computing device calculates the reward using the following formula: [Math 1] V1 represents the processing speed of all normal tasks and all interrupt tasks. V2 represents the processing speed of all interrupt tasks mentioned above. The training device according to claim 3, wherein Va, Vb, a, and b are predetermined values.
5. The training apparatus according to any one of claims 1 to 4, wherein the arithmetic unit reduces the reward if the output result indicates that the interrupt task takes priority even though the interrupt task information is not stored in the second area.
6. An episode consists of all the normal tasks and all the interrupt tasks mentioned above. The training apparatus according to any one of claims 1 to 4, wherein the computing device trains the reinforcement learning model by processing a plurality of episodes.
7. When the training device starts the episode, it obtains a list of multiple interrupt tasks from an external device. The aforementioned list includes the timing of occurrence for each of the multiple interrupt tasks, The training apparatus according to claim 6, wherein the calculation unit stores information of the interrupt task corresponding to the occurrence timing in the second area when the occurrence timing is reached.
8. The transport device performs the task by transporting the cargo to a target location among a plurality of target locations. The information of the normal task includes the target location of the normal task, The information of the interrupt task includes the target location of the interrupt task and the priority of the interrupt task. For two or more interrupt tasks with the same target location, the earlier the interrupt task occurs, the higher the priority is set. The training apparatus according to claim 7, wherein the arithmetic unit adds the information of the interrupt task to the information of the normal task at the same target location as the target location of the interrupt task.
9. An automated warehouse management device, The automated warehouse includes a conveying device that performs the task of transporting goods, The aforementioned task includes a normal task and an interrupt task that occurs after the occurrence of the normal task. The aforementioned control device is A control device for controlling the transport device, It has memory for storing reinforcement learning models, The reinforcement learning model receives parameter information including information about the normal task, information about the interrupt task, and information about the future position of the transport device, and outputs a result indicating whether or not to prioritize the interrupt task over the normal task. The aforementioned memory further, A third area for storing information about the aforementioned normal task, It includes a fourth area for storing information about the interrupt task, The control device repeatedly executes the second predetermined process, The second predetermined process is, When the aforementioned interrupt task occurs, the process involves storing information about the interrupt task in the fourth area. The process involves inputting the parameter information into the reinforcement learning model to obtain the output result of the reinforcement learning model, If the output result indicates that the interrupt task should be prioritized, the process involves adding the information of the interrupt task in the fourth region to the information of the normal task in the third region, and deleting the information of the interrupt task in the fourth region, so that the transport device executes the interrupt task with priority over the normal task. If the output result indicates that the interrupt task should not be prioritized, the process of retaining the information of the interrupt task in the fourth region is performed. A management device including a process for operating the transport device to complete all of the normal tasks and all of the interrupt tasks in the third region.
10. The management device according to claim 9, wherein the reinforcement learning model is trained by the training device according to any one of claims 1 to 4.
11. A training method for training a reinforcement learning model used for controlling an automated warehouse, The automated warehouse includes a conveying device that performs the task of transporting goods, The aforementioned task includes a normal task and an interrupt task that occurs after the occurrence of the normal task. The reinforcement learning model receives parameter information, including information about the normal task stored in a first memory region, information about the interrupt task stored in a second memory region, and information about the future position of the transport device, and outputs a result indicating whether or not to prioritize the interrupt task over the normal task. The training method comprises repeatedly executing a first predetermined process, The first predetermined process is, The output result of the reinforcement learning model is obtained by inputting the parameter information into the reinforcement learning model, If the output result indicates that the interrupt task should be prioritized, the information of the interrupt task in the second region is added to the information of the normal task in the first region, and the information of the interrupt task in the second region is deleted, so that the transport device executes the interrupt task with priority over the normal task. If the output result indicates that the interrupt task is not prioritized, the information of the interrupt task in the second region is retained, The aforementioned training method further includes, When all of the normal tasks and interrupt tasks in the first region have been processed by repeatedly executing the first predetermined process, the reward is determined based on the rate parameters relating to the processing speed of all the normal tasks and the processing speed of all the interrupt tasks. A training method comprising training the reinforcement learning model so that the aforementioned reward increases.
12. A management method for managing an automated warehouse using a reinforcement learning model, The automated warehouse includes a conveying device that performs the task of transporting goods, The aforementioned task includes a normal task and an interrupt task that occurs after the occurrence of the normal task. The reinforcement learning model receives parameter information including information about the normal task stored in the third memory region, information about the interrupt task stored in the fourth memory region, and information about the future position of the transport device, and outputs a result indicating whether or not to prioritize the interrupt task over the normal task. The management method comprises repeatedly executing a second predetermined process, The second predetermined process is, When the aforementioned interrupt task occurs, the information of the interrupt task is stored in the fourth area. The output result of the reinforcement learning model is obtained by inputting the parameter information into the reinforcement learning model, If the output result indicates that the interrupt task should be prioritized, the information of the interrupt task in the fourth region is added to the information of the normal task in the third region, and the information of the interrupt task in the fourth region is deleted, so that the transport device executes the interrupt task with priority over the normal task. If the output result indicates that the interrupt task is not prioritized, the information of the interrupt task in the fourth region is retained, A management method comprising operating the transport device to complete all of the normal tasks and all of the interrupt tasks in the third region.
Citation Information
Patent Citations
System for managing production process progress
JP1994214998A
