A multi-agent reinforcement learning framework for dynamic dispatching in material handling systems.

Abstract

This invention provides a system and method for implementing a multi-agent reinforcement learning-based decision system for material handling systems. [Solution] The method includes initializing a simulation environment that includes decision points and attributes for material dispatch of a material handling system, and which requests a multi-agent reinforcement learning-based decision system to make decisions for material dispatch at the decision points; initializing a reinforcement learning agent that represents the decision points; initializing domain expert heuristics for the decision points; and iteratively training the multi-agent reinforcement learning-based decision system using the domain expert heuristics initialized in the simulation environment.

Description

Technical Field

[0001] The present disclosure generally relates to material handling systems, and more specifically to a multi-agent reinforcement learning framework for dynamic dispatching.

Background Art

[0002] Material handling systems are essential for warehouse management and logistics operations in many diverse industries and play a crucial role in ensuring an efficient material flow. Achieving optimal performance metrics within these systems, such as maximizing throughput and minimizing congestion, has potential cascading effects on downstream applications and leads to business rationalization and cost reduction.

[0003] Dynamic dispatching, an important facet of material handling systems, involves real-time task allocation and resource management. Reinforcement learning (RL) offers a promising means to enhance dynamic dispatching, enabling algorithms to adapt to real-time scenarios and optimize decisions. Conventional dynamic dispatching algorithms are often suboptimal when introduced into complex conveyor systems due to uncertainties inherent in both the processing and demand sides in many systems, interconnections between different sub-processes leading to complex interactions between sub-processes, and limited resources shared among many sub-processes.

[0004] These issues motivate the development of reinforcement learning algorithms that may overcome the above issues. However, training RL algorithms is often costly and infeasible to train in actual systems, and thus a simulator that mimics the complexity of the real world is required to enable the development and testing of algorithms.

[0005] In related technologies, existing methods primarily rely on domain or subject matter experts to manually develop rule-based heuristics to understand the system and optimize dynamic dispatching problems. Furthermore, there are several implementation examples of related technologies for optimizing dynamic dispatching using non-RL-based algorithms, as well as the general idea of ​​using RL for dynamic dispatching in job scheduling of computer systems.

[0006] For example, an example of an implementation of related technologies for dynamic dispatching that is not related to RL is a related technology that uses mixed integer programming for lean manufacturing systems to perform dynamic dispatching and dynamic track routing between automated equipment. An example of an implementation of related technologies that uses RL for dynamic dispatching is an implementation of related technologies for job shop scheduling, which is closely related but is a different problem from dynamic dispatching for material handling systems. [Overview of the Initiative]

[0007] The exemplary implementations described herein are aimed at developing a framework for training event-based multi-agent RL (MARL) algorithms to improve key performance indicators (KPIs) of material handling systems, such as system throughput, and at developing a Python-based simulator for material handling systems with integrated general-purpose processes as a platform for developing RL / heuristic algorithms for dynamic dispatching, which can be easily customized to specific layouts and operations.

[0008] The primary problem with related technologies is that manually created dynamic dispatching logic often results in suboptimal throughput because material handling systems are frequently exposed to inherent uncertainties in upstream and downstream processes. Furthermore, even when manually developed logic is optimal, hardcoded logic lacks the ability to generalize and is often fine-tuned according to the characteristics of the material handling system, requiring extensive redevelopment when the characteristics of the material handling system change.

[0009] Another problem with related technologies is that, due to the complex interrelationships between all subprocesses in a material handling system, manually developing the logic in an actual system often requires experts in the subject area to have a deep understanding of the system, and therefore takes a lot of time.

[0010] Another problem with related technologies is that conventional RL algorithms are difficult to extend to systems with arbitrary decision points and to explore large decision spaces in order to learn the optimal logic.

[0011] Another problem with related technologies is that developing RL / heuristic algorithms on off-the-shelf, black-box simulators is difficult, and developing them on actual systems is expensive and inefficient.

[0012] Aspects of the present disclosure may include a method for implementing a multi-agent reinforcement learning-based decision system for a material handling system, the method comprising: initializing a simulation environment including decision points and attributes for dispatching materials for a material handling system, the simulation environment being configured to request a multi-agent reinforcement learning-based decision system to make decisions for dispatching materials at the decision points; initializing a reinforcement learning agent representing the decision points for the multi-agent reinforcement learning-based decision system; initializing domain expert heuristics for the decision points for the multi-agent reinforcement learning-based decision system; and iteratively training the multi-agent reinforcement learning-based decision system using the domain expert heuristics initialized in the simulation environment.

[0013] Aspects of the present disclosure may include a computer program having instructions for implementing a multi-agent reinforcement learning-based decision system for a material handling system, the instructions comprising: initializing a simulation environment including decision points for dispatching materials and attributes of a material handling system, the simulation environment configured to request the multi-agent reinforcement learning-based decision system to make decisions for material dispatching at the decision points; initializing a reinforcement learning agent representing the decision points for the multi-agent reinforcement learning-based decision system; initializing domain expert heuristics for the decision points for the multi-agent reinforcement learning-based decision system; and iteratively training the multi-agent reinforcement learning-based decision system using the domain expert heuristics initialized on the simulation environment. The computer program and instructions may be stored in a non-transient computer-readable medium and executed by one or more processors.

[0014] Aspects of the present disclosure may include a system for implementing a multi-agent reinforcement learning-based decision system for a material handling system, the system including means for initializing a simulation environment which includes decision points and attributes for dispatching materials for a material handling system and is configured to require a multi-agent reinforcement learning-based decision system to make decisions regarding the dispatch of materials at the decision points; means for initializing reinforcement learning agents representing the decision points for the multi-agent reinforcement learning-based decision system; means for initializing domain expert heuristics for the decision points for the multi-agent reinforcement learning-based decision system; and means for iteratively training the multi-agent reinforcement learning-based decision system using the domain expert heuristics initialized on the simulation environment.

[0015] Aspects of the present disclosure may include an apparatus for implementing a multi-agent reinforcement learning-based decision system for a material handling system, the apparatus including a processor, which initializes a simulation environment including decision points for dispatching materials and attributes of a material handling system, the simulation environment configured to request a multi-agent reinforcement learning-based decision system to make decisions for material dispatching at the decision points, initializes reinforcement learning agents representing the decision points for the multi-agent reinforcement learning-based decision system, initializes domain expert heuristics for the decision points for the multi-agent reinforcement learning-based decision system, and iteratively trains the multi-agent reinforcement learning-based decision system using the domain expert heuristics initialized on the simulation environment.

[0016] One aspect involves a component that includes training an event-based multi-agent RL system incorporating domain knowledge. An example includes (i) initializing an RL-based agent at a decision point of a conveyor system with an appropriate state space and decision space, and an RL-based critic agent; (ii) initializing existing logic (random logic may also be used, if any); (iii) having the RL agent make decisions by alternating between existing heuristic logic and its own logic as needed; (iv) having the RL agent store the state, decisions, and associated rewards returned from the environment; (v) updating the weights of the RL agent based on rewards and the output of the RL-based critic, with the RL-critic weights updated based on rewards; and (vi) repeating steps (iii) through (v) until the performance of the RL agent converges.

[0017] In one aspect, there is a component that involves incorporating domain knowledge through incremental knowledge improvement. An example includes (i) running training in parallel with a Python-based simulator for the first iteration, (ii) training a new set of multiple RL agents for the next iteration by replacing the existing logic in the training component with the RL agents trained from the previous iteration, and (iii) repeating steps (i) and (ii) until convergence occurs.

[0018] In one aspect, there is a component that includes a Python-based simulator. An example implementation includes (i) initializing the simulator environment with user settings and layout, (ii) generating an initial state and exposing the state to the RL agent, (iii) receiving decisions from the RL agent and transitioning to the next state based on the system's user-defined logic, (iv) rewarding the RL agent based on the decisions made, and (v) repeating steps (ii) through (iv) until the RL agent terminates. [Brief explanation of the drawing]

[0019] [Figure 1] Figure 1 shows a multi-agent reinforcement learning system as an implementation example. [Figure 2] Figure 2 shows an example of the learning process flow for a multi-agent reinforcement learning policy based on an implementation example. [Figure 3] Figure 3 shows an example flow for applying multiple RL agents after training for dynamic dispatching in a material handling system related to an implementation example. [Figure 4] Figure 4 shows an example of incorporating domain knowledge through incremental knowledge improvement in an implementation example. [Figure 5] Figure 5 shows an example of a simulator architecture with multiple abstraction layers that enable versatility and reusability in implementation examples. [Figure 6] Figure 6 shows an example of the core logic flow of a simulator for a material handling system in an implementation example. [Figure 7] Figure 7 shows an example of a material handling system to which the implementation example can be applied. [Figure 8] Figure 8 shows an example of a computing environment with examples of computer devices suitable for use in implementation examples. [Modes for carrying out the invention]

[0020] In the following detailed description, the figures and examples of the present application will be described. Reference numerals between figures and descriptions of redundant elements are omitted for clarity. The terms used throughout this specification are provided as examples and are not intended to be limiting. For example, the use of the term "automatic" may include fully automatic or semi-automatic embodiments with user or administrator control over specific aspects of the embodiment, depending on the desired embodiment of those skilled in the art implementing the embodiments of the present invention. The selection may be performed by the user via a user interface or other input means, or may be performed via a desired algorithm. The exemplary implementation examples described herein can be used alone or in combination, and the functions of the exemplary implementation examples can be implemented by any means according to the desired implementation example.

[0021] FIG. 1 is a diagram showing a multi-agent reinforcement learning system according to an embodiment. In the first component 100, event-based multi-agent RL training incorporating domain knowledge is performed. Training of the multi-agent RL policy for dynamic dispatching is initiated in the first component 100 by initializing n policies representing the number of decision points in the conveyor system. These policies are typically parameterized and represented using a neural network. Additionally, additional critics, also represented using a neural network, are parameterized.

[0022] Furthermore, the available existing heuristic-based logic should be initialized with their respective hyperparameters. For example, an expert may decide that items may be dispatched to the nearest subprocess, subject to the constraint that no more than 10 items should be dispatched to any subprocess.

[0023] Next, the environment simulator 103 is initialized with each parameter so as to reflect the actual system as faithfully as possible. Examples of parameters include, but are not limited to, the number of subprocesses, the positions of the subprocesses, the processing time, etc., and can be set based on the desired user configuration 102 according to the desired implementation example.

[0024] When the environment simulator 103 and the multi-agent policy are ready, the training of the multi-agent RL policy can be started. The training process includes the environment simulator 103 publishing the current state of the simulated system. This state information is used as input by the multi-agent RL 100. Within the multi-agent RL block, the critic 104 takes the state as input and outputs a prediction of the value of the current state. Further, n policies also receive the state information as input and output dispatching decisions as required. The simulator 103 receives the dispatching decisions by the n policies, transitions to the next state, and returns the reward value defined by the user. The process of interaction between the policy and the environment simulator 103 continues until the simulation ends. When one simulation ends, the weights of the critic and the weights of the policy are accumulated based on the estimated value and the rewards accumulated by the policy. The described process represents one training episode. Here, the state, action, and reward are arbitrary values defined by the user according to the context of the application. Examples of state information can be those according to the desired implementation example, such as the number of items in the queue in each subprocess, the type of each item, and / or the number of items in each section of the system. Examples of actions can be the destination or direction of the item according to the desired implementation example. Examples of rewards can be, according to the desired implementation example, the total throughput of the system, the length of the queue in each subprocess, etc.

[0025] Figure 2 shows an example flow of the training process for a multi-agent reinforcement learning policy in an implementation example. To facilitate the RL policy training process, dispatching decisions made by heuristic-based logic are also occasionally used to replace the policy's dispatching decisions, and the corresponding rewards made by heuristic logic are then used to train and update the policy weights.

[0026] At step 200, the flow begins with the initialization of all necessary components, including user-defined heuristics, the simulator, and RL agents, including actors and critics. The RL agents are representatives of decision points in the MARL-based decision system. Once these components are initialized, a Multi-Agent Reinforcement Learning (MARL) system is created. At step 201, the formed MARL system receives the current state and reward from the simulator. In the exemplary implementation described herein, the current state and reward from the simulator are provided in an event-based manner rather than a time-based manner (e.g., when a dispatching decision is needed for an incoming package, or for other decisions on the exemplary factory floor). Thus, not every time step of the simulator requires a decision. In one example, the simulator iterates to the next step until a decision is required (e.g., until the package arrives at the receiving point). Thus, at step 202, a determination is made as to whether a decision is needed. If not (No), the flow proceeds to step 211, where the simulator iterates to the next time step to obtain the next state or reward based on any outstanding actions (e.g., the package continues moving along the conveyor belt).

[0027] If a decision is needed (Yes), there are three possible processes. In the first process 204, the MARL network is used to generate a decision on the next action. In the second process 205, a user-defined heuristic is executed to generate an action. Furthermore, in the third process 203, the critic calculates a value based on the state and notifies MARL of the current state along with the estimated throughput or state reward.

[0028] Next, at step 206, a decision is made as to whether to use a heuristic action or a MARL action. The decision on heuristic actions can be set according to any desired implementation example (for example, a flag or condition indicating that the heuristic should be used every other action or every three actions according to a schedule). If the condition is met (Yes), the heuristic action is executed at step 207. Otherwise (No), the action from MARL is executed at step 208.

[0029] Once the selected action is executed, the simulator will perform the action at 209 and continue the simulation based on the execution of the action.

[0030] At step 210, a determination is made as to whether it is time to update the neural network of the MARL system. The update time can be set according to the desired implementation (e.g., after 100 different decisions). If it is time to update (Yes), the flow proceeds to 212, where the MARL agent is updated using the data collected by the simulator along with the critic value, reward, and value. The MARL update can be performed in any way according to the desired implementation. Rewards are assigned based on the successful material dispatch of each reinforcement learning agent from 211 to 201. Each time step of the simulator 211 in the simulation environment is repeated until convergence or a predetermined goal is reached. Convergence can be set according to a desired threshold, or the predetermined goal can be user-defined according to the desired implementation (e.g., after 2000 iterations).

[0031] In the implementation examples described herein, the critic is used to estimate the value of the agent's decision and what the next state will be once the agent's decision is executed. For example, when a MARL agent decides to dispatch an object to a specific location, it may take time to reach that location, and the time when the package will be received there is not necessarily immediately known. In such an example, a delayed reward signal is received and provided to the critic for updating, allowing the MARL agent to learn to be notified early if necessary.

[0032] Through this flow in Figure 2, the MARL system can be trained for deployment and learn whether to maintain heuristics or rely purely on the MARL system agent to select actions.

[0033] In the implementation example, the reinforcement learning agent may be heterogeneous in each class of decisions being trained in parallel, even though they are heterogeneous themselves. For example, different devices (e.g., conveyor belts, robots, etc.) require different types of decisions to be made. For instance, a picking robot has decisions to pick up, dispatch, and place, while a conveyor belt has decisions to start and stop. Since such decisions and the information about those decisions are heterogeneous, the simulator data associated with these decisions and the agent may also be heterogeneous and therefore of different sizes. Thus, during training, such information may be truncated so that the data provided to the reinforcement learning-based decision system in each iteration is uniform or of the same size, so that the data received by the reinforcement learning-based decision system is of the same size. In this way, even heterogeneous MARL systems can be trained using the implementation example described herein.

[0034] Figure 3 shows an example flow for applying multiple RL agents after training for dynamic dispatching in a material handling system related to an implementation example. After the training process in Figure 2, the critic is omitted, and the agents do not need to be updated unless the user requests an update. In this second component, domain knowledge is incorporated through incremental knowledge improvement.

[0035] At step 300, the material handling system is initialized, and the heuristics and trained RL agent are initialized. At step 301, the MARL system receives and observes the current state of the system and at step 302 determines whether a decision is needed. If not (No), the flow proceeds to step 303, allowing the system to continue. If not (Yes), the flow generates an action at step 304 via the MARL system and at step 305 via a user-defined heuristic.

[0036] In 306, a decision is made whether to use the action provided by the MARL system in 304 or the action based on heuristics. In the example implementation, to facilitate the desired implementation, user-defined heuristics may be maintained, or the decision may depend solely on the judgment of the MARL agent, or the user may choose the decision based on performance or the desired implementation. If heuristics are used (Yes), the process proceeds to 307, where a heuristic action is used; otherwise (No), a MARL agent action is used in 308. In 309, the action is selected and executed, and the flow proceeds to 303, where the system executes the action / decision.

[0037] Furthermore, the simulator can be configured at a high or low level (e.g., layout, cycle time, position and speed of each dispatching location) depending on the desired implementation example.

[0038] In the implementation example, to improve the performance of the RL policy, the first iteration of the RL policy is trained using heuristic-based logic. In the second iteration, the heuristics are replaced with the RL policy trained in the first iteration and used for training the second iteration. This process is repeated any number of times until performance improvement stops or converges. Figure 4 shows an example of incorporation of domain knowledge through incremental knowledge improvement in the implementation example. Specifically, Figure 4 shows an example of repeating the training of multiple RL agents multiple times. In the first iteration, domain knowledge is incorporated in the form of heuristics. In subsequent iterations, the heuristics are replaced with the RL agents from the previous iteration, and the improved version of the knowledge is used to guide the multiple RL agents. After the RL policy achieves satisfactory performance, the RL policy can be applied to dynamic dispatching decisions in a real material handling system.

[0039] The third component is a Python-based material handling simulator that can easily interface with external Python packages, is transparent, and can be generalized for simulating material handling in other domains, making it easily reusable. The implementation examples described herein are also intended to serve as a platform for developing optimization algorithms, which often require many iterations and therefore need to be not only accurate but also scalable.

[0040] To achieve the requirements of a simple interface and transparency, the implementation examples described herein fully implement the simulator in native Python without relying on task-specific open-source / commercial packages. This isolates the simulator from dependency issues, allowing users to easily interface with external Python packages.

[0041] To achieve the goal of versatility, the implementation examples described herein include multiple layers of abstraction in the simulator design, thereby allowing users to easily customize and reuse the simulator with any amount of modification. Figure 5 shows an example of a simulator architecture with multiple layers of abstraction that enables the versatility and reusability of the implementation example.

[0042] In an example of abstraction implementation, from the user application's perspective, the first level of abstraction restricts the simulator to the main code that runs the simulator, a configuration file that allows the user to quickly change high-level parameters of the material handling system against fixed logic (e.g., layout, processing time, number of processes, etc.), and an asset file containing the core logic of the material handling system.

[0043] If users require advanced modifications beyond high-level fixes, they can modify the logic within the asset files. Asset files define several commonly used software classes, including planning logic, materials, sensors, junctions, merges, timers, processes, paths, and job queues. Designing the simulator at this level of abstraction allows users to directly modify the properties of each component of the simulator or define new components as needed.

[0044] Finally, the next level of simulator abstraction is the simulator's core logic, which simulates the transition of the entire material handling system from one point in time to the next. The implementation examples described herein include a scheme to update the simulator state by assigning a layout-independent unique identifier (ID) to each material in the simulation. During the simulation, the program iterates through all materials in reverse order of the material flow direction, checking for collisions with other materials and ensuring the material arrives at all defined processes. If a material is found to be in a defined subprocess, the subprocess's logic is executed, enabling an accurate simulation of the entire system.

[0045] Figure 6 shows an example of the core logic flow of a simulator for a material handling system according to an embodiment. In the example in Figure 6, the position of each material is simulated and updated in reverse order of the material flow, and each material is checked for collisions with other materials or intersections with other processes at each step.

[0046] In version 600, the simulator is initialized with all user-defined parameters, as well as parameters related to the material handling system environment. Such parameters may include decision points for dispatching materials, and attributes of the material handling system (e.g., the location of the material's pickup and drop-off points, the distance and path between the pickup and drop-off points, the weight of the material being picked up, etc.).

[0047] In step 601, the simulator repeatedly runs for each material in the system in the reverse order of the material flow. In step 602, it is determined whether or not there is a material collision. If there is a collision (Yes), the material's position is maintained in step 603. Otherwise (No), the flow proceeds to step 604, where it is determined whether or not the material is in a process. If so (Yes), the flow proceeds to step 605, where the process's logic is executed. Otherwise (No), the material's position is updated in step 606.

[0048] Through the embodiments described herein, multi-agent systems can be used to discover dynamic dispatching strategies that are far more effective than existing heuristics, thereby improving the overall throughput of the system.

[0049] Furthermore, the implementation examples can promote a learning-based dispatch logic that is more adaptable to uncertainties within the system compared to static, manually designed dispatch logic.

[0050] Furthermore, by combining existing hand-design logic with learning-based logic as explained in the implementation example, domain knowledge can be incorporated, while simultaneously improving and stabilizing the learning process of multi-agent logic.

[0051] Finally, a modular, configurable Python-based simulator, such as the one used in the implementation examples described herein, allows the simulator to be easily adapted to other use cases and enables a smoother interface with existing Python-based optimizers.

[0052] Figure 7 shows an exemplary material handling system to which an implementation example can be applied. One or more material dispatching devices 721 include physical machinery (e.g., pickup robots, conveyor belts, vehicles, etc.) that are communicably coupled to a network 720 (e.g., a local area network (LAN), a wide area network (WAN)) via corresponding network interfaces of sensor systems installed on the material dispatching devices 721, and are connected to a control device 722 configured to facilitate functions for making decisions on the material dispatching devices 721. One or more material dispatching devices 721 may or may not be associated with sensors or other data acquisition mechanisms, depending on the desired implementation example. The control device 722 manages a database 723, which stores historical data collected from each of the material dispatching devices 721 from sensor systems or data acquisition mechanisms. In alternative implementation examples, data from the sensor system of the material dispatching apparatus 721 can be stored in a dedicated database that takes in data from the material dispatching apparatus 721, or in a central repository or database such as a corporate resource planning system, and the management device 722 can access or retrieve the data from the central repository or database. The sensor system of the material dispatching apparatus 721 may include, but is not limited to, any type of sensor to facilitate a desired implementation and provide internal state mechanical data, such as a gyroscope, accelerometer, Global Positioning Satellite System (GPS), thermometer, hygrometer, or any other sensor. As described herein, the management device 722 may also be connected to one or more cameras (not shown) that monitor the external state of the material dispatching apparatus 721.

[0053] Figure 8 shows an exemplary computing environment having exemplary computer devices suitable for use in several exemplary implementations, such as a management device 722 for facilitating the functionality of a material handling system. The computer device 805 of the computing environment 800 may include one or more processing units, cores, or processors 810, memory 815 (e.g., RAM, ROM, and / or similar), internal storage 820 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 825, any of which may be coupled on a communication mechanism or bus 830 for communicating information or incorporated into the computer device 805. The I / O interface 825 may also be configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation example.

[0054] Computer device 805 may be communicatively coupled to an input / user interface 835 and an output device / interface 840. Either or both of the input / user interface 835 and the output device / interface 840 may be wired or wireless interfaces and may be detachable. The input / user interface 835 may include any physical or virtual device, component, sensor, or interface that can be used to provide input (e.g., buttons, touchscreen interfaces, keyboards, pointing / cursor controls, microphones, cameras, Braille, motion sensors, optical readers, and / or similar). The output device / interface 840 may include displays, televisions, monitors, printers, speakers, Braille, etc. In some implementations, the input / user interface 835 and the output device / interface 840 may be built into or physically coupled to computer device 805. In other implementations, other computer devices may function as or provide the functions of the input / user interface 835 and the output device / interface 840 of computer device 805.

[0055] Examples of computer devices 805 include, but are not limited to, advanced mobile devices (e.g., smartphones, devices mounted in vehicles and other machines, devices carried by humans and animals), portable devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for portability (e.g., desktop computers, other computers, information kiosks, televisions, radios, etc., having one or more processors embedded therein and / or coupled thereto).

[0056] Computer device 805 may be communicatively coupled (for example, via I / O interface 825) to external storage 845 and network 850 for communication with any number of network-connected components, devices, and systems, including one or more computer devices of the same or different configurations. Computer device 805 or any connected computer device may function, provide services, or be referred to as a server, client, thin server, general-purpose machine, special-purpose machine, or other label.

[0057] The I / O interface 825 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMAX, modem, cellular network protocol, etc.) for communicating information with at least all connected components, devices, and networks within the computing environment 800. The network 850 may be any network or combination of networks (e.g., the Internet, local area network, wide area network, telephone network, cellular network, satellite network, etc.).

[0058] Computer device 805 may use and / or communicate using computer-usable or computer-readable media, including transient and non-transient media. Transient media include transmission media (e.g., metal cables, optical fibers), signals, carrier waves, etc. Non-transient media include magnetic media (disks, tapes, etc.), optical media (CD-ROMs, digital video discs, Blu-ray discs, etc.), solid-state media (RAM, ROMs, flash memory, solid-state storage, etc.), and other non-volatile storage or memory.

[0059] Computer device 805 can be used to implement techniques, methods, applications, processes, or computer executable instructions in several exemplary computing environments. Computer executable instructions can be obtained from transient media, stored in non-transient media, and retrieved from non-transient media. Executable instructions can originate from one or more programming languages, scripting languages, and machine languages ​​(e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0060] The processor(s) 810 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 860, an application programming interface (API) unit 865, an input unit 870, an output unit 875, and an inter-unit communication mechanism 895 for different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements may be modified in design, function, configuration, or implementation, and are not limited to the description provided. The processor(s) 810 may take the form of a hardware processor such as a central processing unit (CPU), or a combination of hardware and software units.

[0061] In some implementations, when information or execution instructions are received by the API unit 865, they may be transmitted to one or more other units (e.g., logic unit 860, input unit 870, output unit 875). In some examples, the logic unit 860 may be configured to control the flow of information between units and to direct the services provided by the API unit 865, input unit 870, and output unit 875 in some of the implementations described above. For example, one or more processes or implementation flows may be controlled by the logic unit 860 alone or in conjunction with the API unit 865. The input unit 870 may be configured to receive input for the computations described in the implementations, and the output unit 875 may be configured to provide outputs based on the computations described in the exemplary implementation.

[0062] The processor(s) 810 may be configured to execute a method or instruction for implementing a multi-agent reinforcement learning (MARL) based decision system for the material handling system of Figure 7. This method or instruction includes: initializing a simulation environment containing decision points for dispatching attributes and materials of the material handling system; configuring the simulation environment to request a multi-agent reinforcement learning based decision system (e.g., as shown in 600 in Figure 6) to make decisions on material dispatching at the decision points; initializing a reinforcement learning agent representing the decision points of the multi-agent reinforcement learning based decision system; initializing a domain expert heuristic for the decision points for the multi-agent reinforcement learning based decision system (e.g., as shown in 200 in Figure 2); and iteratively training the reinforcement learning based decision system using the domain expert heuristic initialized on the simulation environment, as shown in Figure 2. As described herein, the initialization of the simulation environment includes decision points and material handling system attributes for dispatching material and material handling system attributes, and the simulation environment is configured to request a MARL-based decision system to make material dispatching decisions at the decision points, thereby allowing the MARL system to be trained much faster than the implementation of related technologies by avoiding the need for training at each time step.

[0063] Processor(s) 810 is configured to perform the methods and instructions described herein, and iteratively training a reinforcement learning-based decision system in a simulation environment using initialized domain expert heuristics includes, for each time step in the simulation environment, receiving a reward from the simulator based on the current state and each successful material dispatching for each of the reinforcement learning agents (e.g., as shown in 201 of Figure 2), and selecting a decision point for a decision point among the decision points reached from decisions from the domain expert heuristics or decisions generated by the multi-agent reinforcement learning-based decision system (e.g., as shown in 202-209 of Figure 2). Each time step in the simulation environment is iteratively performed as shown in Figure 2 until convergence or a specified goal is reached.

[0064] Processor(s) 810 is configured to perform the methods and instructions described herein, and the reinforcement learning-based decision system is configured to select a decision from either an initialized domain expert heuristic (e.g., 207 shown in Figure 3) or a trained decision heuristic (e.g., 208 shown in Figure 3).

[0065] The processor(s) 810 are configured to execute the methods and instructions described herein, and the simulation environment can be configured via adjustable parameters at multiple levels of the material handling system, as shown across different levels in Figure 5.

[0066] A processor(s) 810 is configured to perform the methods and instructions described herein, wherein the reinforcement learning agents are heterogeneous for each class of decisions to be made, and the reinforcement learning agents are trained in parallel as shown in Figure 3.

[0067] The processor(s) 810 is configured to perform the methods and instructions described herein, and further includes truncating the data provided by the reinforcement learning-based decision system at each iteration, so that the data received by the reinforcement learning-based decision system is the same size during training, as illustrated in Figure 2.

[0068] Some parts of the detailed explanation are presented in terms of symbolic representations of algorithms and computer operations. These algorithmic descriptions and symbolic representations are means used by those skilled in the field of data processing technology to convey the essence of the innovation. An algorithm is a set of defined steps that lead to a desired final state or result. In the implementation examples, the steps performed require a specific amount of physical operation to achieve the specific result.

[0069] Unless otherwise stated, as will be evident from the discussions, discussions throughout this specification using terms such as “processing,” “computing,” “calculating,” “determining,” and “displaying” may include the operations and processes of a computer system or other information processing device that manipulate and convert data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system or other information storage, transmission, or display devices.

[0070] The implementation examples also relate to apparatus for performing the operations described herein. This apparatus may be configured specifically for a particular purpose, or it may include one or more general-purpose computers that are selectively started or reconfigured by one or more computer programs. Such computer programs may be stored on computer-readable media, such as computer-readable storage media or computer-readable signal media. Computer-readable storage media include, but are not limited to, tangible media such as optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices, and drives, as well as other types of tangible or non-temporary media suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and representations presented herein are not inherently related to any particular computer or other apparatus. Computer programs may include purely software implementations containing instructions that perform the operations of a desired implementation.

[0071] Various general-purpose systems may be used with the programs and modules according to the embodiments herein, or it may be convenient to construct more specialized devices for performing desired method steps. Furthermore, these embodiments are not described with reference to any particular programming language. It will be understood that various programming languages ​​may be used to implement the techniques of the implementation examples described herein. Instructions in a programming language may be executed by one or more processing units, such as a central processing unit (CPU), a processor, or a controller.

[0072] As is known in the art, the operations described above can be performed by hardware, software, or any combination of software and hardware. Various embodiments of the implementation examples may be implemented using circuits and logic devices (hardware), while other embodiments, when performed by a processor, may be implemented using instructions stored on a machine-readable medium (software) that causes the processor to perform the method of performing the implementation of the application. Furthermore, some implementation examples of the application may be performed by hardware alone, while other exemplary implementations may be performed by software alone. Moreover, the various functions described may be performed by a single unit or may span a number of components in any number of ways. When performed by software, the method may be performed by a processor such as a general-purpose computer based on instructions stored on a computer-readable medium. If necessary, the instructions may be stored on the medium in a compressed and / or encrypted format.

[0073] Furthermore, other implementations of the present application will be apparent to those skilled in the art from the considerations in this specification and from the practice of the present technology. Various aspects and / or components of the exemplary implementations described herein can be used individually or in any combination. This specification and exemplary embodiments are intended to be considered illustrative only, and the true scope and nature of the present application are given by the following claims.

Claims

1. A method for implementing a decision system using multi-agent reinforcement learning for a material handling system on a computer, The aforementioned computer, Initializing a simulation environment that includes decision points for material dispatching and attributes of the material handling system, configured to input a state to the decision system at the decision points and receive a corresponding decision; This involves initializing a reinforcement learning agent that is provided in conjunction with the aforementioned decision points, outputs a decision for material dispatching based on the input state, and updates its policy based on the reward, Rules for making decisions for material dispatch at the aforementioned decision point, which involve initializing predefined heuristics for material dispatch based on domain knowledge, The decision system is repeatedly trained in the simulation environment using the aforementioned heuristics. Execute method.

2. The method according to claim 1, Iteratively training the decision system on the simulation environment using the initialized heuristics is For each time step in the aforementioned simulation environment, The simulation environment receives the current state and the reward based on each successful material dispatch of each reinforcement learning agent, At the aforementioned decision point, a decision is selected from among the decisions generated by the heuristic and the decisions generated by the reinforcement learning agent. Includes, Each time step in the aforementioned simulation environment is executed repeatedly until convergence or a specified goal is reached. method.

3. In the method according to claim 1, The decision system is configured to select a decision from either the heuristic or the trained reinforcement learning agent. method.

4. In the method according to claim 1, The simulation environment can be configured via multiple levels of adjustable parameters of the material handling system. method.

5. In the method according to claim 1, The reinforcement learning agents are heterogeneous for each class of decisions to be made, and the reinforcement learning agents are trained in parallel. method.

6. In the method according to claim 5, The computer further performs the task of truncating the data provided to the decision system in each iteration so that the data received by the decision system remains the same size during training. method.

7. A non-temporary computer-readable medium that stores instructions for implementing a decision system using multi-agent reinforcement learning for a material handling system, The aforementioned instruction is, Initializing a simulation environment that includes decision points for material dispatching and attributes of the material handling system, configured to input a state to the decision system at the decision points and receive a corresponding decision; This involves initializing a reinforcement learning agent that is provided in conjunction with the aforementioned decision points, outputs a decision for material dispatching based on the input state, and updates its policy based on the reward, Rules for making decisions for material dispatch at the aforementioned decision point, which involve initializing predefined heuristics for material dispatch based on domain knowledge, The decision system is repeatedly trained in the simulation environment using the aforementioned heuristics. including, A non-temporary computer-readable medium.

8. A non-temporary computer-readable medium according to claim 7, Using the initialized heuristics, iteratively training the decision system in the simulation environment is: For each time step in the aforementioned simulation environment, The simulation environment receives the current state and the reward based on each successful material dispatch of each reinforcement learning agent, At the aforementioned decision point, a decision is selected from among the decisions generated by the heuristic and the decisions generated by the reinforcement learning agent. Includes, Each time step in the aforementioned simulation environment is executed repeatedly until convergence or a specified goal is reached. A non-temporary computer-readable medium.

9. A non-temporary computer-readable medium according to claim 7, The decision system is configured to select a decision from either the heuristic or the trained reinforcement learning agent. A non-temporary computer-readable medium.

10. A non-temporary computer-readable medium according to claim 7, The simulation environment can be configured via multiple levels of adjustable parameters of the material handling system. A non-temporary computer-readable medium.

11. A non-temporary computer-readable medium according to claim 7, The reinforcement learning agents are heterogeneous for each class of decisions to be made, and the reinforcement learning agents are trained in parallel. A non-temporary computer-readable medium.

12. A non-temporary computer-readable medium according to claim 11, The instruction further includes truncating the data provided to the decision system in each iteration so that the data received by the decision system is of the same size during training. A non-temporary computer-readable medium.

13. A device for implementing a decision system using multi-agent reinforcement learning for a material handling system, Including the processor, The aforementioned processor, Initialize a simulation environment that includes decision points for material dispatching and attributes of the material handling system, configured to input a state to the decision system at the decision points and receive a corresponding decision; initialize a reinforcement learning agent provided corresponding to the decision points, which outputs a decision for material dispatching based on the input state and updates its policy based on a reward; initialize a heuristic for material dispatching that is a rule for making a decision for material dispatching at the decision points and is predefined based on domain knowledge; and iteratively train the decision system in the simulation environment using the heuristic. It is configured in such a way. Device.

14. In the apparatus according to claim 13, The aforementioned processor, For each time step in the aforementioned simulation environment, The simulation environment receives the current state and the reward based on each successful material dispatch of each reinforcement learning agent, At the aforementioned decision point, a decision is selected from among the decisions generated by the heuristic and the decisions generated by the reinforcement learning agent. The decision system is configured to be iteratively trained in the simulation environment using the heuristics described above. Each time step in the aforementioned simulation environment is executed repeatedly until convergence or a specified goal is reached. Device.

15. In the apparatus according to claim 13, The decision system is configured to select a decision from either the heuristic or the trained reinforcement learning agent. Device.

16. In the apparatus according to claim 13, The simulation environment can be configured via multiple levels of adjustable parameters of the material handling system. Device.

17. In the apparatus according to claim 13, The reinforcement learning agents are heterogeneous for each class of decisions to be made, and the reinforcement learning agents are trained in parallel. Device.

18. In the apparatus according to claim 17, The further includes truncating the data provided to the decision system in each iteration so that the data received by the decision system remains the same size during training, Device.

Images