An order batch processing method and system applied to intelligent warehousing

Abstract

This invention discloses an order batch processing method and system for intelligent warehousing, comprising: a modeling step: constructing a semi-Markov decision process model for the dynamic order batch processing decision problem; a state awareness step: collecting warehouse operation data in real time and encoding it into a structured state vector containing order queue characteristics, equipment load characteristics, and global monitoring characteristics; an action decision step: based on a trained deep reinforcement learning agent, adaptively selecting a target heuristic rule from multiple preset underlying order batch processing heuristic rules according to the current state vector; and a task execution step: calling the selected heuristic rule to batch process the current order pool, generating picking tasks and driving the physical system to execute. This invention, through an adaptive decision-making mechanism based on deep reinforcement learning, enables the warehousing system to perceive the environment in real time and dynamically select the optimal batch processing strategy from multiple rules, thereby intelligently balancing efficiency and timeliness.

Description

Technical Field

[0001] This invention relates to the field of warehousing and logistics automation and artificial intelligence, and in particular to an order batch processing method and system applied to intelligent warehousing. Background Technology

[0002] With the rapid development of e-commerce, warehousing and logistics systems face multiple challenges, including fragmented orders, high timeliness requirements, and large operational fluctuations. The "goods-to-person" robotic picking system significantly improves warehousing efficiency by using mobile robots to transport shelves to fixed picking stations. In this system, order batch processing (merging multiple orders into a single picking batch) is a key optimization step that reduces robot travel distance, minimizes shelf handling, and improves equipment utilization.

[0003] Currently, the order batch processing methods commonly used in the industry mainly rely on fixed heuristic rules or static optimization algorithms, which have the following prominent problems:

[0004] 1. Rigid strategies and lack of dynamic adaptability: Existing methods mostly adopt single static rules such as "prioritizing the largest order overlap" or "prioritizing the earliest delivery date". These rules cannot perceive and respond to changes in the real-time operating status of the warehouse (such as fluctuations in order inflow rate, differences in order structure, uneven equipment load, etc.), resulting in a significant decrease in decision-making performance during peak business periods or complex operating conditions, and poor system robustness.

[0005] 2. Single Optimization Objective, Difficult to Balance Conflicting Multiple Objectives: Traditional methods typically focus on only a single optimization objective. For example, excessively pursuing shelf sharing to reduce movement may lead to the prolonged delay of "isolated" orders with urgent delivery deadlines, causing order delays; conversely, focusing solely on order delivery urgency may sacrifice the economies of scale brought by batch processing, resulting in inefficiency. Existing technologies lack an effective mechanism for dynamically and intelligently balancing "operational efficiency" (minimizing shelf movement) and "service level" (minimizing order delays).

[0006] 3. Lack of self-learning and continuous optimization capabilities: The decision-making logic of existing methods remains fixed after deployment, lacking the ability to learn from historical operational data and discover potential optimization patterns. The system cannot self-adjust for specific warehouse layouts, order characteristics, or equipment configurations, nor can it continuously evolve with the changing business model, resulting in limited intelligence.

[0007] Therefore, there is an urgent need in this field for an order batch processing solution that can perceive the environment in real time, dynamically adapt, intelligently balance multiple objectives, and has continuous learning capabilities. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology. To achieve the above objective, an order batch processing method and system applied to intelligent warehousing is adopted to solve the problems mentioned in the background technology.

[0009] A batch order processing method for smart warehousing includes the following steps: S1. Modeling steps: The dynamic order batch processing decision problem is constructed into a semi-Markov decision process model, which is defined by state space, action space, state transition, reward function and variable decision time interval; S2, State Awareness Step: Collect warehouse operation data in real time and encode it into a structured state vector containing order queue characteristics, equipment load characteristics and global monitoring characteristics; S3, Action Decision Steps: Based on the trained deep reinforcement learning agent, according to the current state vector, the agent adaptively selects a low-level heuristic rule from a number of preset low-level order batch processing heuristic rules. The agent first selects the low-level heuristic, and then the orders are added sequentially by the heuristic to form a batch until the stopping condition and / or the batch limit is met. The agent is trained in a high-fidelity simulation environment with the goal of maximizing long-term cumulative rewards. S4. Task execution steps: Call the selected heuristic rule to batch process the current order pool, generate picking tasks and drive the physical system to execute.

[0010] As a further aspect of the present invention: the structured state vector in step S2 is constructed by encoding and concatenating the following three types of information, specifically including: Order queue characteristics are numerical vectors obtained by classifying and statistically analyzing orders based on the required shelf combinations and delivery urgency. Equipment load characteristics, a numerical vector reflecting the availability of picking station and mobile robot resources and the load of tasks in transit; and Global monitoring features include a macro performance vector comprising the cumulative number of delayed orders, the cumulative number of completed orders, and system uptime.

[0011] As a further aspect of the present invention: the multiple underlying order batch processing heuristic rules preset in step S3 include at least two of the following categories: The first type of shelf-sharing priority rule aims to maximize the shelf overlap of orders within a batch. The second type of order delivery deadline priority rule prioritizes ensuring on-time order delivery and combines it with the principle of shelf sharing for batch processing.

[0012] As a further aspect of the present invention: the reward function used for training the agent is a composite instantaneous reward function, the formula of which is:

[0013] in, This represents the immediate reward value obtained by the agent after performing an action at decision time t. This represents the total number of shelf moves required if all orders within the batch generated by this action are picked as independent tasks. This represents the total number of shelf moves that actually occurred when these orders were combined into a single batch for picking. The difference is the efficiency gain brought by this batch of operations, which serves as a positive reward. This represents the total delay time of all orders in the batch when the picking task is completed; the delay time of a single order is defined as the difference between its actual completion time and its promised delivery time, or 0 if there is no delay. This is a preset penalty weight coefficient greater than 0, used to adjust the intensity of the delay penalty. Its value can be adjusted according to the business's emphasis on service level. This is a negative punishment.

[0014] As a further aspect of the present invention: the deep reinforcement learning agent is trained using a deep Q-network algorithm, and at least one optimization mechanism is integrated during the training process, which may be a dual deep Q-network mechanism, a priority experience replay mechanism, or a dual deep Q-network architecture.

[0015] As a further aspect of the present invention: the high-fidelity simulation environment is a configurable software module used to simulate the order flow generation function of randomly arriving orders; the physical process simulation function of simulating warehouse layout, equipment movement and operation time, queuing logic; and the standard interface function for interacting with the intelligent agent in terms of status, action, and reward.

[0016] As a further aspect of the present invention, it also includes a closed loop of post-deployment strategy evaluation and iterative optimization, with the following specific steps: Collect real operational data to calibrate parameters in the simulation environment and / or fine-tune and retrain the agent model, thereby achieving continuous optimization of the model.

[0017] As a further aspect of the present invention: the decision-making action in step S3 is event-triggered, and the triggering conditions include: New order arrival event, or picking task completion event.

[0018] As a further aspect of the present invention: the deep reinforcement learning agent is implemented using a fully connected feedforward neural network, the dimension of its input layer is the same as the dimension of the state vector, the dimension of its output layer is the same as the number of preset heuristic rules, and the output value is the Q-value estimate of each rule in the current state.

[0019] The second aspect of the technical solution: an intelligent warehousing system, including shelves, mobile robots, picking workstations and a central control system, wherein the central control system is configured to execute the order batch processing method as described in any one of the above.

[0020] Compared with the prior art, the present invention has the following technical advantages: The above technical solution constructs a semi-Markov decision process model for the complex and dynamic order batch processing problem. The system then perceives the warehouse status (orders, equipment, global information) in real time and encodes it as a feature vector. Next, a pre-trained deep reinforcement learning agent acts as the decision-maker, adaptively selecting the optimal rule from a set of preset underlying batch processing heuristics based on the current state vector. Finally, the selected rule is executed to complete the batch processing task. The beneficial effect of this solution is that, through end-to-end learning, the system can perceive environmental changes in real time and intelligently and dynamically balance and optimize between the conflicting goals of "improving picking efficiency" (reducing shelf movement) and "ensuring service levels" (reducing order delays). This overcomes the rigidity and singular objectives of traditional static rule strategies, significantly improving the overall efficiency, adaptability, and robustness of the warehousing system. Attached Figure Description

[0021] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings: Figure 1 This is a schematic diagram illustrating the steps of an order batch processing method according to an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please refer to Figure 1 In this embodiment of the invention, an order batch processing method applied to intelligent warehousing includes the following steps: S1. Modeling steps: The dynamic order batch processing decision problem is constructed into a semi-Markov decision process model, which is defined by state space, action space, state transition, reward function and variable decision time interval; Specifically, the order batch processing problem, which involves random order arrivals and dynamic changes in equipment resources, is constructed as a semi-Markov decision process model. This model is jointly defined by the state space, action space, state transition probabilities, reward function, and variable decision time interval. S2, State Awareness Step: Collect warehouse operation data in real time and encode it into a structured state vector containing order queue characteristics, equipment load characteristics and global monitoring characteristics; In this embodiment, the structured state vector in step S2 is constructed by encoding and concatenating the following three types of information, specifically including: Order queue characteristics are numerical vectors obtained by classifying and statistically analyzing orders based on the required shelf combinations and delivery urgency. Equipment load characteristics, a numerical vector reflecting the availability of picking station and mobile robot resources and the load of tasks in transit; and Global monitoring features include a macro performance vector comprising the cumulative number of delayed orders, the cumulative number of completed orders, and system uptime.

[0024] The specific steps are as follows: The order queue information to be processed is encoded as follows: First, the orders are classified according to the combination of picking shelves required; second, within each category, they are further subdivided according to the urgency of delivery; finally, each subcategory is counted, and the count values ​​are used to form a numerical vector. Load information encoding of core picking equipment: Quantify key indicators describing the availability of physical resources to form a numerical vector, which includes at least: the number of idle picking stations, the number of idle mobile robots, and the total number of orders in progress; Global monitoring information encoding: Introduce global variables for macro-level monitoring to form a supplementary vector, which includes at least: the cumulative number of overdue orders, the cumulative total number of completed orders, and the current running time.

[0025] Specifically, real-time warehouse operation data is collected and defined as a structured state vector, which includes at least: information on the order queue to be processed, load information of the core picking equipment, and global monitoring information; S3, Action Decision Steps: Based on the trained deep reinforcement learning agent, according to the current state vector, the agent adaptively selects a low-level heuristic rule from a number of preset low-level order batch processing heuristic rules. The agent first selects the low-level heuristic, and then the orders are added sequentially by the heuristic to form a batch until the stopping condition and / or the batch limit is met. The agent is trained in a high-fidelity simulation environment with the goal of maximizing long-term cumulative rewards. In this embodiment, the multiple underlying order batch processing heuristic rules preset in step S3 include at least two of the following categories: The first type of shelf-sharing priority rule aims to maximize the shelf overlap of orders within a batch. Among them, the shelf-sharing priority heuristic aims to maximize shelf sharing and includes the following: High-demand driven variant: Select the order with the largest required number of shelves (the size of the shelf set corresponding to the order) as the starting order; Low-demand driven variant: Select the order with the smallest required shelf space as the starting order; The second type of order delivery deadline priority rule prioritizes ensuring on-time delivery of orders and combines batch processing with the shelf-sharing principle; The order delivery deadline priority heuristic aims to ensure on-time order delivery. Its execution logic is as follows: First, normalize the delivery deadlines of all orders and sort them in ascending order of the normalized value. Second, set three thresholds and only consider orders with a delivery deadline normalized value not greater than the threshold as active orders. If the number of active orders is less than the batch capacity, add active orders to the batch and fill the gap using the shelf-sharing priority rule. If the number of active orders is greater than the batch capacity, filter active orders using the shelf-sharing priority rule to meet the batch capacity limit.

[0026] In this embodiment, the decision action in step S3 is event-triggered, and the triggering conditions include: A new order arrival event, or a picking task completion event, which causes a change in the availability of picking equipment.

[0027] Specifically, a set of underlying order batch processing heuristic rules with different optimization objectives is defined as the action space, where the agent selects the most suitable heuristic rule as the action to be executed at each decision moment; S4. Task execution steps: Call the selected heuristic rule to batch process the current order pool, generate picking tasks and drive the physical system to execute.

[0028] In this embodiment, the reward function used for agent training is a composite instantaneous reward function, with the following formula:

[0029] in, This represents the immediate reward value obtained by the agent after performing an action at decision time t. This represents the total number of shelf moves required if all orders within the batch generated by this action are picked as independent tasks. This represents the total number of shelf moves that actually occurred when these orders were combined into a single batch for picking. The difference is the efficiency gain brought by this batch of operations, which serves as a positive reward. This represents the total delay time of all orders in the batch when the picking task is completed; the delay time of a single order is defined as the difference between its actual completion time and its promised delivery time, or 0 if there is no delay. This is a preset penalty weight coefficient greater than 0, used to adjust the intensity of the delay penalty. Its value can be adjusted according to the business's emphasis on service level. This is a negative punishment.

[0030] In this embodiment, the deep reinforcement learning agent is trained using a deep Q-network algorithm, and at least one optimization mechanism is integrated during the training process. The optimization mechanism is a dual deep Q-network mechanism, a priority experience replay mechanism, or a dual deep Q-network architecture.

[0031] Among them, the dual deep Q-network mechanism decouples the action selection and target Q-value estimation in the process of calculating the target Q-value; the current online network is used to select the optimal action for the next state, but the relatively stable target network is used to evaluate the Q-value of the action, thereby effectively alleviating the training instability problem caused by overestimation of Q-value in traditional deep Q-networks. Priority Experience Replay Mechanism: Instead of the traditional practice of uniformly and randomly sampling from the experience replay pool, each stored experience sample is assigned a priority. This priority is positively correlated with the absolute value of the temporal difference error (TD-error) of the sample. That is, the more inaccurate the agent's predictions are and the more information is contained in the sample, the more likely it is to be selected for training, thereby significantly improving data utilization and learning speed. Dual Deep Q-Network Architecture: The output layer of the neural network is structurally modified by dividing it into two branches: one branch is used to estimate the value function of the state. Another branch is used to estimate the advantage function for each action. The final action value function The architecture is composed of the outputs of these two branches; it allows the agent to learn more effectively the relative good and bad of different actions.

[0032] In this embodiment, the high-fidelity simulation environment is a configurable software module used to simulate the order flow generation function of randomly arriving orders; the physical process simulation function of simulating warehouse layout, equipment movement and operation time, queuing logic; and the standard interface function for interacting with the intelligent agent in terms of status, action, and reward.

[0033] It has the following functions: Order flow generation function: It can simulate the random arrival of new orders based on preset statistical distributions (such as Poisson distribution), and can configure parameters such as the average arrival rate of orders, the distribution of the number of goods in an order, and the mapping relationship between goods and shelves to simulate workloads of different intensities and types. Physical process simulation function: accurately simulates the physical layout within the warehouse, including shelf locations and picking station locations; simulates the workflow and time consumption of mobile robots and picking stations, including robot movement time (calculated based on Manhattan distance), time for picking and placing items, and equipment queuing time, etc. Interactive interface function: It has a standardized interface for interacting with deep reinforcement learning agents; when it receives the action selected by the agent, the simulation environment is responsible for executing the batch processing logic corresponding to the action, driving the internal time flow, updating the state of the entire system, calculating the corresponding reward value according to the formula defined above, and then returning the new state and reward value to the agent.

[0034] This embodiment also includes a post-deployment strategy evaluation and iterative optimization closed loop, with the following specific steps: Collect real operational data to calibrate parameters in the simulation environment and / or fine-tune and retrain the agent model, thereby achieving continuous optimization of the model.

[0035] Specifically, the deployment and application process also includes a closed loop of strategy evaluation and iterative optimization. In practical applications, real business data is continuously recorded, and this data is used to calibrate the simulation environment or to fine-tune and retrain the deployed model, so as to achieve continuous self-evolution of the model.

[0036] In this embodiment, the deep reinforcement learning agent is implemented using a fully connected feedforward neural network. Its input layer dimension is the same as the state vector dimension, and its output layer dimension is the same as the number of preset heuristic rules. The output value is the Q-value estimate of each rule in the current state.

[0037] Example 2 Step S1, Problem Modeling: The sequential decision-making process of order batch processing is formalized as a semi-Markov decision process. This model is chosen because in actual warehousing, the time interval between two decisions is variable, depending on the execution time of the batch task, which is consistent with the characteristics of a semi-Markov decision process.

[0038] Step S2, Definition of High-Dimensional State Space: To enable the decision-making agent to fully perceive the warehouse environment, this invention defines a high-dimensional state space. At any decision moment, the state is represented by a feature vector, which is composed of the following three modules: Order Queue Profile Module: Describes the composition of the pool of orders to be processed, including the number of orders in different shelf combinations, the number of orders with different urgency levels, etc., to gain insight into the similarity distribution of orders and time pressure.

[0039] Physical resource load module: Reflects the real-time availability of core picking resources, such as the number of idle picking stations, the number of idle robots, and the amount of tasks in transit, to help the agent make judgments.

[0040] Global performance monitoring module: Provides macro-level historical performance metrics, such as the cumulative number of delayed orders and the total number of completed orders, as an auxiliary reference for decision-making.

[0041] Step S3, Heuristic Action Space Construction: A core innovation of this invention lies in the design of the action space. Traditional reinforcement learning directly determines order combinations, leading to an excessively large action space. This invention employs a hyperheuristic approach, defining an "action" as a selection from a pre-defined set of "underlying heuristic rules." For example, action 1 might be "execute the 'shelf sharing priority' rule," and action 2 might be "execute the 'most urgent order priority' rule." The agent's task is to learn which rule should be invoked to guide the specific batch processing operation in the current state.

[0042] Step S4: Design of a multi-objective balanced reward function: To guide the agent to find a balance between efficiency and service, this invention designs a composite reward function. After the agent performs an action, the environment provides a reward signal, which consists of two parts: Efficiency gain (positive reward): Equal to the number of shelf moves saved by this batch of operations. The more you save, the greater the reward.

[0043] Delay Penalty (Negative Reward): If the processing of this batch causes order delays, the system will calculate a penalty value based on the total delay time. The more severe the delay, the greater the penalty.

[0044] Agent Training Based on Deep Q-Networks: A neural network is trained using a deep Q-network algorithm to serve as the "brain" of the agent. Training takes place in a highly simulated environment, where the agent learns through massive amounts of trial and error. To ensure efficient and stable training, this invention integrates advanced technologies such as dual deep Q-networks and priority experience replay.

[0045] Online Deployment and Application: After training, the agent model is integrated into the warehouse's order management system. During system runtime, state information is input into the model in real time, and the model outputs the current optimal heuristic rule within milliseconds. By invoking this rule, a high-quality, dynamically adaptive order batch processing decision can be made.

[0046] Example 3 Example This embodiment verifies the method steps of the present invention in a large-scale simulated e-commerce warehouse environment, specifically as follows: The formal modeling steps for the environment in step S1 are as follows: The entire decision-making process is modeled as a semi-Markov decision process, and its core elements are defined as follows: Decision Timing: The moment when an agent makes a decision is not a fixed point in time, but is triggered when the following two types of events occur: (1) All picking tasks from the previous batch have been assigned to the robot, and there are still empty picking stations and orders to be processed; (2) A new order arrives, making the previously empty order pool non-empty.

[0047] state space It contains all possible warehouse states. .

[0048] Action space It contains all the underlying heuristics that an agent can choose from. .

[0049] reward function : Defines the state Next action The instant reward received afterward.

[0050] The specific steps for defining and encoding the high-dimensional system state space in step S2 are as follows: At each decision-making moment, the state vector is constructed in the following way. : Order queue profiling module: For each pending order, it is classified and statistically analyzed according to its required shelf set (multi-hot code) and urgency level (single hot code).

[0051] The quantity statistics of various orders are flattened into a one-dimensional numerical vector. To ensure that the dimension is fixed, only the K order types with the highest frequency can be counted.

[0052] Physical resource load module: Create a normalized vector of length 3, with the following elements: [Current number of available picking stations / Total number of picking stations] [Current number of idle robots / Total number of robots] [Total number of orders currently in progress / Capacity value] Global performance monitoring module: Create a vector of length 3, with the following elements: [Cumulative number of delayed orders] [Total number of completed orders] [Minutes since the start of operations today] Finally, the vectors from these three modules are concatenated to form a complete state vector s, which serves as the input to the deep Q-network.

[0053] The specific steps for constructing the heuristic action space in step S3 are as follows: In this embodiment, the action space A consists of 8 specific underlying heuristic rules: Actions 1-2 (Efficiency-First): Action 1 (High Demand Driven - Shelf Sharing Priority): Invoke the "High Demand Driven Variant" rule to maximize shelf sharing.

[0054] Action 2 (Low Demand Driven - Shelf Sharing Priority): Invoke the "Low Demand Driven Variant" rule to prioritize processing isolated orders.

[0055] Actions 3-8 (Time-sensitive, with efficiency considerations): Actions 3-4 (Emergency Queue Processing): Only for orders with a remaining delivery time less than 0.3 of the threshold, invoke the "High Demand Driven" (Action 3) and "Low Demand Driven" (Action 4) rules respectively.

[0056] Actions 5-6 (Secondary Emergency Queue Processing): Only for orders with remaining delivery time less than 0.6 seconds of the threshold, invoke the "High Demand Driven" (Action 5) and "Low Demand Driven" (Action 6) rules respectively.

[0057] Actions 7-8 (Regular Queue Processing): Only for orders with remaining delivery time less than the threshold of 0.9, invoke the "High Demand Driven" (Action 7) and "Low Demand Driven" (Action 8) rules respectively.

[0058] The specific calculation of the composite reward function in step S4 is as follows: At decision time t, the agent selects an action. , generated a containing Batch of orders After the simulation environment executes this batch, the reward will be given. The calculation is as follows:

[0059] in: It represents the number of shelf moves required to pick the i-th order in a batch.

[0060] The entire batch Combine the actual number of shelf moves that occurred during picking. w is the preset delay penalty weight, for example, w = 0.5.

[0061] It is the actual completion time of the i-th order.

[0062] It is the promised delivery time for the i-th order.

[0063] In this embodiment, the specific steps for training the deep reinforcement learning agent are as follows: Neural Network Architecture: A fully connected feedforward neural network is used. The number of nodes in the input layer is equal to the dimension of the state vector *s*. Two hidden layers are set (256 neurons each, ReLU activation function). The output layer has 8 neurons, directly outputting the Q-values ​​corresponding to the 8 actions.

[0064] Training process: Initialization: Randomly initialize the weights of the online network and the target network, and establish a priority experience replay pool with a capacity of 50,000.

[0065] Exploration and Utilization: An ε-greedy strategy is employed for action selection. The value of ε decreases linearly from an initial 1.0 to 0.01 as the number of training steps increases, to achieve a smooth transition from full exploration to full utilization.

[0066] Learning and updating: Each step will generate an experience tuple. Store the data in the replay pool. Every 4 steps, sample a small batch (e.g., 32 samples) of data from the replay pool according to priority. Calculate the target Q-value using a dual-depth Q-network approach, then calculate the mean squared error loss, and update the weights of the online network through backpropagation using the Adam optimizer.

[0067] Target network synchronization: Every 1000 training steps, the weights of the online network are completely copied to the target network to maintain the stability of the target Q value.

[0068] Training terminates when training has been completed for a total of 1 million time steps, or when the agent’s performance no longer improves significantly in multiple consecutive evaluation rounds.

[0069] In this embodiment, the steps for online decision-making and application are as follows: After training, the weights of the best-performing online network model are saved. In a real-world warehouse management system, a decision service is deployed. This service updates the status of orders and equipment scheduling in real time via message queues. Once a decision is triggered, a state vector is immediately constructed as described above, and the loaded model is called for forward propagation calculations to obtain the Q-values ​​of eight actions. The action index with the largest Q-value is selected, and the corresponding batch processing module is called to complete a decision. The entire decision-making process can be completed in milliseconds to meet real-time decision-making requirements; the specific time consumption depends on hardware configuration, model size, and concurrent load.

[0070] After training, the weights of the best-performing model are saved. In a real-world warehouse management system, a decision service is deployed. This service updates its state in real time; when a decision is triggered, it immediately constructs a state vector, calls the model to calculate the Q-value of each action, and selects the action with the highest Q-value to execute the corresponding batch processing function. The entire decision-making process meets real-time requirements.

[0071] The second aspect of the technical solution: an intelligent warehousing system, including shelves, mobile robots, picking workstations and a central control system, wherein the central control system is configured to execute the order batch processing method as described in any one of the above.

[0072] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention. The scope of the invention is defined by the appended claims and their equivalents, all of which should be included within the scope of protection of the invention.

Claims

1. A batch order processing method applied to intelligent warehousing, characterized in that, Includes the following steps: S1. Modeling steps: The dynamic order batch processing decision problem is constructed into a semi-Markov decision process model, which is defined by state space, action space, state transition, reward function and variable decision time interval; S2, State Awareness Step: Collect warehouse operation data in real time and encode it into a structured state vector containing order queue characteristics, equipment load characteristics and global monitoring characteristics; S3, Action Decision Steps: Based on the trained deep reinforcement learning agent, according to the current state vector, the agent adaptively selects a low-level heuristic rule from a number of preset low-level order batch processing heuristic rules. The agent first selects the low-level heuristic, and then the orders are added sequentially by the heuristic to form a batch until the stopping condition and / or the batch limit is met. The agent is trained in a high-fidelity simulation environment with the goal of maximizing long-term cumulative rewards. S4. Task execution steps: Call the selected heuristic rule to batch process the current order pool, generate picking tasks and drive the physical system to execute.

2. The order batch processing method for intelligent warehousing according to claim 1, characterized in that, The structured state vector in step S2 is constructed by encoding and concatenating the following three types of information, specifically including: Order queue characteristics are numerical vectors obtained by classifying and statistically analyzing orders based on the required shelf combinations and delivery urgency. Equipment load characteristics, a numerical vector reflecting the availability of picking station and mobile robot resources and the load of tasks in transit; and Global monitoring features include a macro performance vector comprising the cumulative number of delayed orders, the cumulative number of completed orders, and system uptime.

3. The order batch processing method for intelligent warehousing according to claim 1, characterized in that, The multiple underlying order batch processing heuristic rules preset in step S3 include at least two of the following categories: The first type of shelf-sharing priority rule aims to maximize the shelf overlap of orders within a batch. The second type of order delivery deadline priority rule prioritizes ensuring on-time order delivery and combines it with the principle of shelf sharing for batch processing.

4. The order batch processing method for intelligent warehousing according to claim 1, characterized in that, The reward function used for training the agent is a composite instantaneous reward function, with the following formula: in, This represents the immediate reward value obtained by the agent after performing an action at decision time t. This represents the total number of shelf moves required if all orders within the batch generated by this action are picked as independent tasks. This represents the total number of shelf moves that actually occurred when these orders were combined into a single batch for picking. The difference is the efficiency gain brought by this batch of operations, which serves as a positive reward. This represents the total delay time of all orders in the batch when the picking task is completed; the delay time of a single order is defined as the difference between its actual completion time and its promised delivery time, or 0 if there is no delay. This is a preset penalty weight coefficient greater than 0, used to adjust the intensity of the delay penalty. Its value can be adjusted according to the business's emphasis on service level. This is a negative punishment.

5. The order batch processing method for intelligent warehousing according to claim 1, characterized in that, The deep reinforcement learning agent is trained using a deep Q-network algorithm, and at least one optimization mechanism is integrated during the training process. The optimization mechanism is either a dual deep Q-network mechanism, a priority experience replay mechanism, or a dual deep Q-network architecture.

6. The order batch processing method applied to intelligent warehousing according to claim 1, characterized in that, The high-fidelity simulation environment is a configurable software module used to simulate the order flow generation function of randomly arriving orders; the physical process simulation function of simulating warehouse layout, equipment movement and operation time, queuing logic; and the standard interface function for interacting with intelligent agents in terms of status, action, and reward.

7. The order batch processing method for intelligent warehousing according to claim 1, characterized in that, It also includes a closed loop of post-deployment strategy evaluation and iterative optimization, with the following specific steps: Collect real operational data to calibrate parameters in the simulation environment and / or fine-tune and retrain the agent model, thereby achieving continuous optimization of the model.

8. The order batch processing method for intelligent warehousing according to claim 1, characterized in that, The decision-making action in step S3 is event-triggered, and the triggering conditions include: New order arrival event, or picking task completion event.

9. The order batch processing method for intelligent warehousing according to claim 1, characterized in that, The deep reinforcement learning agent is implemented using a fully connected feedforward neural network. Its input layer dimension is the same as the state vector dimension, and its output layer dimension is the same as the number of preset heuristic rules. The output value is the Q-value estimate of each rule in the current state.

10. An intelligent warehousing system, comprising shelves, mobile robots, picking workstations, and a central control system, characterized in that, The central control system is configured to perform the order batch processing method as described in any one of claims 1-9.

Images