Artificial intelligence control and optimization of agent tasks in warehouse

Abstract

A control system for a warehouse (100) includes a controller (202) for communicating commands executed by an article carrier vehicle (106), a robotic picker (104), and a human operator (102). A warehouse simulation (210) performs a simulated operation of order picking and replenishment activities. Simulation results and experience data are recorded and stored in a memory (212). The stored data includes operation data, and the operation data includes real-time results and experience data recorded when the operator executes according to the executable command from the controller (202). The training module (210) receives simulation results, simulated experience data, and recorded operational data from the memory (212). A training module (210) trains an algorithm using the simulated data and the operational data. The training module (210) generates an updated algorithm for the controller (202). Using the updated algorithm, the controller (202) communicates executable commands to the operator (102, 104, 106).

Description

Technical Field

[0001] The present invention relates to the control of order picking systems in a warehouse environment, and in particular to the use of artificial intelligence (AI) algorithms to control agents (human or robotic) to perform order picking processes including replenishment, storage allocation and batch building. Background Art

[0002] The control of order picking and replenishment systems in warehouses that include various workers or agents (e.g., human pickers, robotic pickers, item transporters, conveyors, and other components of order picking and replenishment systems) is a complex task. Traditional algorithms are used to pursue various objectives in the ever-increasing complexity of order fulfillment, which is characterized by a large variety of stock keeping units (SKUs), order compositions ranging from single SKUs to multiple SKUs, wide variations in order demand in quantity and time scales, and very strict delivery deadlines. Other changing priorities include minimizing delivery time, order processing schedules, or scheduling of orders with the highest priority. Minimizing energy consumption, minimizing travel distance, and reducing labor costs are also important factors. Additional factors such as pallet stability, traffic congestion, and avoidance measures are also considered in the control of warehouse operations. The optimality of these strategies depends on different factors, including warehouse size, warehouse geometry, number of orders, order profiles, slot allocation, and the number of agents (workers) in the system. Heuristic-based algorithms written by experts are mainly used in practice to solve such challenges. However, good executable heuristics require a lot of effort to design, test, implement, optimize, program, and validate, and such algorithms are often very specific to customer requirements and use cases (i.e., not easily transferable to another warehouse and / or customer). Furthermore, these heuristics do not adapt well to changing warehouse operations and / or order conditions. Summary of the invention

[0003] Embodiments of the present invention provide methods and systems for highly flexible order picking and replenishment solutions that can dynamically respond to changing warehouse operations and order conditions. The flexible solution can be applied to a variety of warehouses with ever-changing customer conditions.

[0004] Warehouse order fulfillment systems and operations employing the exemplary adaptable / trainable algorithmic solution are well suited to unique customer conditions and can continually adapt and account for changes in operating conditions. The exemplary algorithm is also able to optimize the operation by taking into account all of the different factors discussed in the background of the invention (e.g., energy consumption, labor costs, driving distance, etc.).

[0005] By means of strategic decision-making and controlling the operations of storing items, retrieving items, building batches, allocating resources to perform item replenishment, assigning orders to pickers and vehicles, selecting resources for specific tasks, and allocating and coordinating the movement of vehicles and pickers to perform the picking and replenishment processes, exemplary embodiments of the present invention achieve overall optimization of both human-to-goods (or human-to-robot) and goods-to-human (or goods-to-robot) order picking processes.

[0006] The warehouse order picking and replenishment control system includes: an exemplary controller for communicating commands executed by operators or agents (e.g., human pickers, robotic pickers, item carriers, conveyors, and other components of the order picking and replenishment system). Such a control system may include, for example: one or more computers or servers (e.g., computers or servers running in a network), which include hardware and software, including one or more programs, such as cooperative interaction programs.

[0007] The exemplary architecture consists of a digital twin warehouse simulation, which includes: one or more programs running on one or more computers / servers, such as programs executed on one or more processors, which continuously perform simulated runs of order picking and replenishment activities in the simulated warehouse. The simulated run can also use data from the real warehouse order fulfillment operating system. The simulated result data and experience data are recorded in a storage module (e.g., a database). The storage module includes operational data, which includes: real-time result data and experience data recorded when the agent performs tasks according to executable commands conveyed by the controller. The operational data may also include data recorded from other warehouses to broaden the knowledge of the learning model. The training module (or training cluster) 210 receives the simulation results, simulated experience data, and recorded operational data from the storage module. The training module 210 may include one or more programs (e.g., cooperative interaction programs) and is configured to train the algorithm using simulated data and operational data. The training module 210 generates new / updated neural network weight results for the algorithm (to update the algorithm) and forwards it to the controller. Using the updated algorithm, the controller 202 communicates executable commands to a worker (e.g., a human worker, a robotic picker, an automated guided vehicle (AGV), an autonomous mobile robot (AMR), etc.) As described herein, AGVs and AMRs may be used interchangeably, but it is understood that they play the same role in the order fulfillment system, but with different levels of autonomy.

[0008] An exemplary method of the present invention includes: recording operational data related to order picking and replenishment activities performed in a warehouse. The operational data includes real system results and experience data. The operational data is collected and stored in a storage module. The algorithm is retrained using updated operational data and continuously generated simulation data. The updated neural network weight results from the retraining of the algorithm are used to generate updated executable commands for AGVs and robotic pickers.

[0009] The present invention provides an exemplary architecture and method of an AI-based self-learning approach to comprehensively control and optimize the problem.

[0010] First Exemplary Use Case: Human-to-Goods / Robot-to-Goods:

[0011] In this exemplary scenario, an operator (e.g., AGV, human picker, robot, etc.) is responsible for picking empty packages or order media (in the form of pallets, totes, empty cartons, bags, etc.) at a designated starting point within the warehouse and delivering the completed order at a designated destination in the warehouse. An order contains multiple items that are located in different storage locations throughout the warehouse. During the picking process, a "picking" operator (human or robotic vehicle) moves through the warehouse to various storage locations containing items from a specified order. The picker then picks the items from a specific storage location (e.g., box, carton, individual item, etc.) and places them on itself or a separate vehicle (e.g., AGV, robot, cart, pallet jack, pallet truck, etc.). Once all items in the order have been picked, the picker and / or vehicle moves to the designated destination to drop off the completed order.

[0012] Second Exemplary Use Case: Goods-to-Person / Goods-to-Robot:

[0013] Based on a given order containing multiple items distributed in the warehouse, a vehicle (e.g. AGV, cart, ...) is responsible for transporting the items from their storage location to a picker (human or robot) located at a fixed picking location. The vehicle picks up a handling unit (e.g. in the form of a shelf, pallet, tote, etc.) in order to deliver the order-related items contained in the handling unit to the picking location. The picker (e.g. robot / human) then picks the items from the handling unit and places them in a completed order medium (e.g. tote, pallet, bag, etc.) or a buffer location (e.g. placement wall) for subsequent processing. After the picking is completed, the vehicle can deposit the handling unit back to the warehouse.

[0014] These and other objects, advantages, purposes and features of the present invention will become more apparent from the following description read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1is a diagram of an exemplary warehouse with various workers performing activities in the warehouse in accordance with the present invention;

[0016] Figure 1A is a warehouse layout diagram divided into locations and sections according to the present invention;

[0017] Figure 2 is an exemplary system architecture diagram of an order picking control system according to the present invention, illustrating data exchange between warehouse subsystems;

[0018] Figure 3 is a block diagram showing the steps of a method for controlling an order picking system of a warehouse in accordance with the present invention by continuously training an algorithm and combining it with real experience data;

[0019] Figure 4 is a block diagram illustrating an exemplary reinforcement learning algorithm;

[0020] Figure 4A is a diagram showing an exemplary central reinforcement learning controller using a neural network for training an algorithm to control an order picking system for a warehouse in accordance with the present invention;

[0021] Figure 4B is a diagram showing an exemplary multi-agent reinforcement learning controller using a neural network for training an algorithm to control an order picking system for a warehouse in accordance with the present invention;

[0022] Figure 4C is a diagram showing an exemplary hierarchical multi-agent reinforcement learning controller using a neural network for training an algorithm to control an order picking system for a warehouse in accordance with the present invention;

[0023] Figure 5 is a perspective view of an exemplary digital twin warehouse simulation according to the present invention;

[0024] Fig. 6A , Figure 6B and Figure 6C is a graph showing exemplary results and improvements of key KPIs during a learning process of a case study for evaluating an algorithm for managing automated guided vehicles (AGVs) and pickers (robots or humans) in an order picking system for a warehouse according to the present invention. DETAILED DESCRIPTION

[0025] The present invention will now be described with reference to the drawings, wherein referenced elements in the following written description correspond to like referenced elements in the drawings.

[0026] Artificial intelligence provides concepts in natural language understanding and computer vision that are widely used in commercial products, however, large-scale robotic control and automation remains challenging and is mostly solved using traditional fixed strategies.

[0027] Exemplary embodiments of machine learning solutions discussed in this article utilize deep reinforcement learning (DRL), multi-agent deep reinforcement learning (MARL), and hierarchical reinforcement learning (HRL) to improve the efficiency and flexibility of order picking systems in real warehouse systems.

[0028] The exemplary reinforcement learning solution has the potential to improve the actual performance of such order picking systems (e.g., by reducing order delivery times in any warehouse configuration).

[0029] Another benefit of these learning systems in controlling agents (compared to traditional fixed strategies) is their flexibility, which can be effectively applied to any warehouse because they can adapt to new situations. The adaptability enables the exemplary system to continuously improve and learn, taking into account changes in warehouse size, layout, operating mode, item storage strategy, and changes in the number and type of workers (robots or humans). The adaptability also allows the system to relatively easily incorporate more constraints in its optimization (e.g., pallet stability, energy use, labor costs), which is particularly difficult and cumbersome in traditional fixed algorithmic approaches.

[0030] Exemplary embodiments of the present invention provide an AI-based program for controlling an agent in a warehouse based on deep reinforcement learning to solve a set of described problems. Such embodiments may be implemented with various hardware and software constituting one or more computer systems or servers, such as a computer system or server running in a network, which includes hardware and software, and the hardware and software include one or more programs, such as cooperative interaction programs. For example, exemplary embodiments may include hardware, such as one or more processors configured to read and execute software programs. Such programs (and any associated data) may be stored and / or retrieved from one or more storage devices. The hardware may also include power supplies, network devices, communication devices, and input / output devices, such devices being used to communicate with local and remote resources and / or other computer systems. Such embodiments may include one or more computer systems, and optionally, be communicatively coupled with one or more additional computer systems accessed locally or remotely. Certain computer components of exemplary embodiments may be implemented with local resources and systems, remote or "cloud"-based systems, or a combination of local and remote resources and systems. The software executed by the computer system of the exemplary embodiment may include or access one or more algorithms to guide or control the execution of a computer-implemented process, such as execution within an exemplary warehouse order fulfillment system. As discussed herein, such algorithms define the order and coordination of process steps performed by the exemplary embodiments.As also discussed herein, improvements and / or refinements to the algorithms will improve the operation of the process steps performed by the exemplary embodiments according to the updated algorithms.

[0031] Exemplary embodiments include: a learning system in which a trained neural network is used as a decision maker for the control of these agents in the warehouse. Such an exemplary system would include the following: 1) pre-training based on the environment / digital twin and specific resource characteristics and processes to learn a general policy (encoded in the neural network) based on a reward function; 2) synchronizing the real environment with the digital twin; 3) training with and incorporating real execution data; 4) controlling warehouse processes within or via a warehouse execution system (WES); and 5) continuous improvement over multiple cycles of data collection and additional training. Using a trained neural network as a decision maker allows the system to control the operation in a variety of situations that may be encountered.

[0032] Figure 4The high-level concepts of the reinforcement learning algorithm are shown. The environment is defined as a model of a simulated or real warehouse environment, where items are stored in warehouse locations. "Workers" are physical entities, such as human workers or robots, who move around in the environment and perform physical tasks to complete orders. The state of the environment is any information available from the information system of the simulated or real facility, such as the location of workers in the system, the target (or future) location of workers in the system, the busy status of workers, and order data (e.g., the number of picks remaining and the location of items).

[0033] The agent is defined as a decision-making system that maps the state of the environment to a set of actions for each worker and internally tries to predict and maximize the expected cumulative reward. The data that the agent communicates to the environment are the actions of each worker in the system, which can include, for example, the target warehouse location or area that the worker should go to and what the worker should do at the destination.

[0034] A "reward" is a numerical value (or another way of indicating value) that conveys the effectiveness of a chosen action in the environment. Reward functions can be derived from many different events in the environment. Positive rewards are given for good actions (i.e., actions taken in the environment that lead to good outcomes), while negative rewards are given for bad actions (i.e., actions taken in the environment that lead to bad outcomes). Some examples include: completing an order (positive reward), picking the next item in an order (positive reward), and not moving for a long time (negative reward). It should be noted that rewards are not limited to these scenarios, and a variety of reward distributions can be chosen for different environmental events.

[0035] Figure 1 An exemplary warehouse environment 100 is shown with various different workers 102, 104, 106. The reinforcement learning agent can control a single worker or multiple workers by category (e.g., 1 agent controls all AGVs). Each class of workers has different goals and capabilities. Figure 1 1 and 2, and three different worker categories are shown in FIG. 1 , which include human pickers 102, robotic pickers 104, and automated guided vehicles (AGVs) 106. The overall logistics of the warehouse 100 will be allocated to each worker category.

[0036] An exemplary controller of the warehouse 100 is configured to provide artificial intelligence (AI) control and optimization of agent tasks in the warehouse 100. An exemplary AI controller using deep reinforcement learning (DRL, also known as RL) is configured to control different types of operators in the warehouse 100 (via an RL agent) and optimize various goals of the warehouse 100. These goals may include, for example, order completion time / order delivery time, traffic and congestion, reduction in the number of operators (e.g., pickers, vehicles, and robots), energy use, driving distance, labor costs, and pallet stability and picking patterns. Unlike traditional methods, these goals can be very simply added to the AI ​​method (by adjusting the reward function), in which professional programmers try to add these constraints as best they can, using past experience and manual trial and error on simulators.

[0037] Due to the continuous learning nature of AI algorithms and the ease of adding additional constraints, the following advantages can be enjoyed compared to traditional approaches (where professional programmers must specifically customize their implementation to enjoy these advantages):

[0038] 1. Continuously consider changing operating conditions, such as warehouse reconfiguration, new employees, new equipment, new vehicles, etc.

[0039] 2. Continuously consider changing order conditions, such as changes in the number of items per order or order profile, seasonal changes, etc.

[0040] 3. Improved efficiency.

[0041] 4. Flexibility in methods that work for each warehouse.

[0042] 5. Easy to set up in customer warehouse, simplifying deployment and reducing debugging time.

[0043] 6. Scalable to large and complex warehouse systems.

[0044] 7. Warehouse layout optimization.

[0045] 8. Slotting optimization.

[0046] 9. Order forecast.

[0047] 10. Optimize batch building.

[0048] For large systems, warehouses are divided into "segments" (or "segments"), and these segments are further divided into clusters of locations within these segments, e.g. Figure 1AAs shown. The main motivation is to divide the action space to reduce the complexity of each agent and improve exploration efficiency. This is achieved by introducing a "manager" agent, which provides the operator agent with a "goal", that is, the target section to which the operator must go. The manager is also an RL agent, albeit a logical (non-physical) agent, and the decisions it makes (i.e., the goals it provides to the operator) are also learned via reinforcement learning methods. The decisions made by the manager agent occur longer than those of the operator agent, requiring a method that uses a larger time window for learning. This is solved by using hierarchical reinforcement learning, which involves making decisions over a longer time frame. The spatial division of the warehouse can be further abstracted and other manager agents (i.e., multiple managers, or managers of managers) can be introduced, but for the purposes of this example, only a single manager agent is described. By decomposing the action space in this way, effective strategies can be learned that will scale with the size of the warehouse and the number of operators in the system.

[0049] Figure 2 An exemplary system architecture is shown in which commands and data are communicated between components of the system architecture. With the exception of the operator HMI, these systems can be physically separated (i.e., running on their own computers as shown in the figure), or as virtual modules on a single computer, or as virtual modules on the cloud. Exemplary commands and data between systems include:

[0050]

[0051] The order data originates from the order management system 206, which is generated based on the customer's order fulfillment requirements (e.g., the customer orders multiple items stored in the warehouse 100 online). The order data is communicated to the AI ​​controller 202, and the AI ​​controller 202 uses this information and other information available to it to generate commands. The AI ​​controller 202 transmits them to the vehicle management and execution system 204, which then uses the information to control and command the movement of the exemplary vehicles (e.g., robotic pickers and robotic vehicles) 104, 106. Vehicle status data is communicated by the vehicles 104, 106 of the fleet to the vehicle management and execution system 204, which passes the vehicle status data to the AI ​​controller 202.

[0052] Once the vehicles 104, 106 have performed their tasks for the relevant orders, order completion and status information will also be transmitted by the vehicle management and execution system 204 to the AI ​​controller 202 and the order management system 206 to convey the completion status of the order and other operational information regarding the status.

[0053] Finally, for human operators operating within the system and performing order tasks (e.g., picking items from shelves), operator HMI 208 (human-machine interface, e.g., user interaction screen) sends and receives order data from order management system 206 and sends order completion and status data to AI controller 202. Operator status data is communicated by the corresponding operator HMI 208 to AI controller 202. Operator commands are communicated by AI controller 202 to operator HMI 208 (and executed by the operator of the corresponding operator HMI 208).

[0054] Experience data (which includes simulation results, AI neural network weights, buffered data (state actions, state data, etc.), order data, vehicle data, vehicle commands, operator states, and operator commands) is transmitted back and forth between the training cluster 210 and the experience storage 212, between the training cluster 210 and the AI ​​controller 202, and between the experience storage 212 and the AI ​​controller 202. This ensures that the AI ​​controller 202 can use the experience when retraining in the future and enhance the digital twin simulation (in the training cluster 210) with real operational data. Note that each server or database component can be implemented as cloud-based or on-site. Note that Figure 2 The dashed lines in the diagram represent wireless transmission of data (e.g., via Wi-Fi, 5G, etc.).

[0055] Figure 3 Operational flow steps of an exemplary process provided by an AI controller for training and updating algorithms executed by vehicles of a fleet and / or by operators via their operator HMIs are shown.

[0056] In step 302, the training cluster trains the AI ​​algorithm based on the digital twin system simulation; in step 304, the simulation results and simulation experience are stored in the experience storage.

[0057] In one embodiment, the experience store is configured to be seeded with experiences from other repositories. In step 306, the updated neural network weights are copied from the training cluster to the AI ​​controller, which is responsible for running the real ("live") system. These updated neural network weights are used to update the algorithm, changing its execution characteristics and resulting in improved performance.

[0058] In step 308, the AI ​​controller uses the updated algorithm to run the real or live system by communicating commands to the downstream execution system and operator HMI. In step 310, the AI ​​controller records its own operations and collects data from the order management system, operator system, and vehicle management and execution system. In step 312, the AI ​​controller collects real system results and experience data and stores it in the experience memory. In step 314, the training cluster retrieves the experience memory and retrains the AI ​​algorithm in the training cluster using the real data and results and the continuously generated simulation data. The operational flow then continues back to step 306, where the updated neural network weights are again copied from the training cluster to the AI ​​controller (to update the algorithm).

[0059] Figure 4A , Figure 4B and Figure 4C Three exemplary neural network architectures are shown, representing Figure 4 The “Agent” portion of the diagram in Figure 1 shows the neural network used to command two exemplary types of workers (hereafter referred to as AGVs and pickers):

[0060] · Figure 4A -Central reinforcement learning controller.

[0061] · Figure 4B -Multi-agent reinforcement learning controller.

[0062] · Figure 4C -Hierarchical multi-agent reinforcement learning controller.

[0063] Figure 4A , Figure 4B and Figure 4C A reinforcement learning controller based on the Advantage Actor-Critic algorithm (A2C) is presented, which has a trainable internal neural network where the input is applied to a first set of nodes (input layer), and the output of the first set of nodes is applied to a second set of nodes (hidden layer) and then to a final set of nodes (output layer). Although the A2C algorithm is chosen in this example, other reinforcement learning algorithm architectures can be used, such as Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG), Deep Q Network (DQN), or other value optimization algorithms. In addition, in Figure 4BIn the A2C example, the neural network weights are learned in these networks via the standard A2C reinforcement learning loss function. The final output layer in the policy network with multiple nodes (e.g., four (4) nodes) provides the action probabilities that will be statistically sampled and executed, representing the improved algorithm executed by the AI ​​controller. The value network typically outputs a single value that is an estimate of the favorableness of the current state of the system and is used in the loss calculation to assist in learning.

[0064] Central Reinforcement Learning Controller:

[0065] exist Figure 4A In , the inputs to the reinforcement learning controller include: current worker position (AGV and picker), target worker position (AGV and picker), and remaining picking positions (for AGV). Additional inputs may include: work arrival time (AGV and picker), worker-to-target (AGV and picker) distance, and number of completed orders. Figure 4A As shown, the output of the above statistical sampling method may include: an example; a sex operator destination location, which is handed over to the operator for execution.

[0066] Multi-agent reinforcement learning controller:

[0067] exist Figure 4B In the multi-agent reinforcement learning controller, the input consists of Figure 4A The same information is provided as in the exemplary configuration shown in . Subsets of this information can be provided to each individual controller as needed to reduce network input complexity. There can be multiple controllers, each with advantages and disadvantages for different configurations. One exemplary configuration is one controller per operator (e.g., 3 pickers, 3 AGVs, 6 controllers - 1 controller per operator), which has the advantage of making the operators' behavior more personalized, allowing them to further specialize, but reduces the ability of operators to share experiences, making training more complex. Another exemplary configuration is one controller per operator category (e.g., 3 pickers, 3 AGVs, 2 controllers - 1 picker controller, 1 AGV controller), which has the advantage of making training more efficient because they can share experiences, but reduces the ability of operators to exhibit personalized behaviors that may be beneficial in some circumstances. As Figure 4B As shown, the output of the statistical sampling method described above may include: an exemplary operator destination location, which is delivered to the operator for execution.

[0068] Hierarchical Multi-Agent Reinforcement Learning Controller:

[0069] exist Figure 4C In the multi-agent reinforcement learning controller, the input consists of Figure 4A and Figure 4B The example configuration shown in . A subset of this information can be provided to each individual controller as needed to reduce network input complexity. The key difference in this configuration is the introduction of a manager agent, which communicates goals to other reinforcement learning controllers. An example goal issued by the manager agent to the workers includes: the target warehouse section that the manager wants all individual workers to go to. Figure 4B As shown in the example configuration shown in , there can be multiple controllers, with the same advantages and disadvantages for different configurations. One example configuration is: each worker is equipped with a controller (for example, 3 pickers, 3 AGVs, 1 manager; resulting in 7 controllers - 1 controller for each worker and 1 controller for each manager), with the associated advantages and disadvantages as shown in Figure 4B Another exemplary configuration is one in which each worker category is equipped with one controller (e.g., 3 pickers, 3 AGVs, 1 supervisor; resulting in 3 controllers - 1 picker controller, 1 AGV controller, 1 supervisor controller), with the associated advantages and disadvantages as shown in Figure 4B The configuration shown in Figure 1 is outlined in the figure. Figure 4B As shown, the output of the statistical sampling method described above may include: exemplary operator destination locations (which are handed over to operators for execution) and operator target segments (which are generated by the supervisor agent).

[0070] like Figure 5 As shown, an exemplary machine learning solution includes a warehouse simulator. Figure 5 The exemplary warehouse simulation shown is a high performance 3D simulator running on a processor that can represent any warehouse, manage order generation and distribution, and an AGV control system for guiding workers through the simulated warehouse. According to the present invention, any controlled entity is represented as an "operator".

[0071] The warehouse simulation includes: AGVs, which are configured to collect and deliver ordered items; and pickers, which are responsible for collecting items and placing them on the AGVs. The complexity of the tasks performed by the warehouse control system mainly depends on the number of AGVs, the number of pickers, and the number of item locations in the warehouse.

[0072] As described herein, an exemplary warehouse includes two types of workers that are configured to perform different tasks and each has specific capabilities. AGV stands for Robotic Automated Guided Vehicle (AGV), which distributes orders in sequence. For each order, the AGV collects a given number of specific items. Once all ordered items are collected, the AGV moves to a specific location to deliver and complete the order. Upon completion, the AGV is assigned a new order (as long as there are uncompleted, unassigned orders).

[0073] The exemplary picker is configured to move through the same location as the AGV and is required to pick items and load any required items onto the AGV. For the picker to load an item onto the AGV, both operators must be located at the location of that particular item. As discussed herein, the picker can be a robotic picker or a human picker.

[0074] Warehouse Simulator is also compatible with real customer data to create simulations of real-world warehouse systems.

[0075] Fig. 6A and Figure 6B Shows the use of Figure 5 High-performance simulator and Figure 4C Results of an exemplary case study of an exemplary neural network architecture shown in , where there is 1 supervisor agent, 1 AGV agent (controlling thirty (30) AGV operators), and 1 picker agent (controlling fifteen (15) human picker agents) in a warehouse with over 700 different locations. Three key performance indicators were defined, including order picking delivery time, number of picks per hour, and expected cumulative reward. The AI-based approach reduced the delivery time by 22% compared to a classic heuristic (baseline) algorithm coded by professional programmers to perform order fulfillment tasks. Figure 6C The algorithm is shown to be Fig. 6A and Figure 6B The improvement in the expected cumulative reward that can be obtained during the learning process is related to the KPI improvement shown in .

[0076] Thus, embodiments of the exemplary neural network are configured to provide a highly flexible solution that dynamically responds to changing warehouse operating and order conditions. This flexible solution can be applied to every warehouse where customer conditions are highly variable. It was found that the exemplary algorithmic solution is well suited to customer conditions and continuously takes into account changes in operating conditions.

[0077] Changes and modifications may be made to the specifically described embodiments without departing from the principles of the invention, which is limited only by the scope of the appended claims, which are to be interpreted in accordance with the principles of patent law including the doctrine of equivalents.

Claims

1. An order fulfillment control system for a warehouse, the order fulfillment control system comprising: Warehouse simulation, which is configured to continuously perform warehouse simulations, including simulated runs of order fulfillment activities; a storage module configured to retain and store operational data including outcome data and experience data, and wherein the warehouse simulation is configured to output the simulated operational data to the storage module; a controller configured to control order fulfillment activities of a plurality of agents, and wherein the controller is configured to: record real-time operational data as the agents perform their order fulfillment activities, and wherein the controller is configured to output the real-time operational data to the storage module; a training module configured to retrieve the real-time operating data and the simulated operating data stored in the storage module, wherein the training module is configured to train an algorithm using the real-time operating data and the simulated operating data, and wherein the training module is configured to generate neural network weight results for the algorithm and forward them to the controller; and Wherein, the controller is configured to update the algorithm using the received neural network weight results, and use the updated algorithm to control the order fulfillment activities of the multiple agents.

2. The order picking control system according to claim 1, wherein: The training module includes a neural network configured to iteratively perform training runs, wherein the training runs replay the simulated operational data and the real-time operational data to attempt to find optimal neural network weight results for an optimal algorithm, wherein the optimal algorithm is defined by a desired priority of order fulfillment activities in the warehouse.

3. The order fulfillment control system according to any one of claims 1 or 2, wherein: The intelligent agents include: multiple human pickers, robot pickers and automated guided vehicles (AGVs).

4. The order fulfillment control system according to claim 3, wherein: The controller is configured to control the AGV and the robotic picker via executable commands communicated by the controller.

5. The order fulfillment control system according to claim 3, wherein: The AGV includes a transport vehicle configured to collect and deliver ordered items within the warehouse.

6. The order fulfillment control system according to claim 5, wherein: The robotic picker is configured to collect the ordered items and place them onto the transport vehicle.

7. The order fulfillment control system according to claim 5, wherein: The controller is configured to control the human pickers via executable commands communicated by the controller to a human machine interface (HMI), and wherein each HMI is configured to direct a corresponding human picker in order fulfillment activities.

8. The order fulfillment control system according to claim 7, wherein: The directed fulfillment activities include collecting the ordered items and placing them on the shipping vehicle.

9. The order fulfillment control system according to any one of claims 1 or 2, wherein: The warehouse simulation is a digital twin simulation of the warehouse, and wherein the warehouse simulation is configured to execute one of: a single warehouse simulation instance or multiple warehouse simulation instances.

10. A method for controlling order fulfillment activities of a plurality of agents in a warehouse, the method comprising: continuously executing a warehouse simulation, the warehouse simulation including a simulated run of order fulfillment activities; Retaining and storing operation data in a storage module, wherein the operation data includes result data and experience data; outputting simulated operational data from the simulated run of the order fulfillment activity to the storage module; controlling order fulfillment activities of the plurality of agents; recording real-time operational data as the agents perform their order fulfillment activities; Outputting the real-time operation data to the storage module; Retrieving the real-time operation data and the simulated operation data stored in the storage module; Train algorithms using retrieved real-time operational data and simulated operational data; generating neural network weight results for the algorithm; The algorithm is updated using the received neural network weight results, and the updated algorithm is used to control the order fulfillment activities of the plurality of agents.

11. The method according to claim 10, wherein: Training of the algorithm includes iteratively performing training runs that replay the simulated operational data and the real-time operational data to attempt to find optimal neural network weight results for an optimal algorithm, and wherein the optimal algorithm is defined by a desired priority of order fulfillment activities in the warehouse.

12. The method according to any one of claims 10 or 11, wherein: The intelligent agents include: multiple human pickers, robot pickers and automated guided vehicles (AGVs).

13. The method according to claim 12, wherein: Controlling the order fulfillment activities of the plurality of agents includes communicating executable commands to the AGVs and robotic pickers.

14. The method according to claim 12, wherein: The AGV includes a transport vehicle configured to collect and deliver ordered items within the warehouse.

15. The method according to claim 14, wherein: The robotic picker is configured to collect the ordered items and place them onto the transport vehicle.

16. The method according to claim 14, further comprising: The human pickers are controlled via executable commands communicated to a human machine interface (HMI), and wherein each HMI directs a corresponding human picker in order fulfillment activities.

17. The method according to claim 16, wherein: The directed fulfillment activities include collecting the ordered items and placing them on the shipping vehicle.

18. The method according to any one of claims 10 or 11, wherein: Executing a warehouse simulation continuously includes executing one of the following: a single warehouse simulation instance or multiple warehouse simulation instances.

Images