Warehouse robot joint scheduling method and system based on soft order allocation and reinforcement learning

Abstract

This invention discloses a joint scheduling method and system for warehouse robots based on soft order allocation and reinforcement learning, belonging to the field of intelligent warehousing and logistics scheduling technology. The method includes: calculating the matching degree between the order and the shelf / workstation when an order arrives, filtering candidate shelves, and updating the soft order set and popularity; when a robot scheduling event is triggered, acquiring the system state containing the entity state and soft allocation state, encoding it as a heterogeneous graph, updating the node embedding through a graph neural network, generating an action probability distribution, and selecting a scheduling action; during the execution of the scheduling action, the soft allocation state is transformed into a deterministic allocation: when picking up goods, determining the order that can be fulfilled based on the soft order set and inventory; when delivering goods, placing them in the picking queue and triggering supplementary scheduling. This invention achieves joint optimization of order allocation and robot scheduling through a soft allocation mechanism, overcoming the information barrier of phased methods and avoiding the computational bottleneck of joint optimization methods, significantly reducing completion time and average order completion time.

Description

Technical Field

[0001] This invention relates to the field of intelligent warehousing and logistics scheduling technology, and more specifically to a joint scheduling method and system for warehousing robots based on soft order allocation and reinforcement learning. Background Technology

[0002] Robotic mobile picking systems are a mainstream solution for modern smart warehousing. In this system, a large number of autonomous mobile robots move freely on the warehouse floor, transporting shelves loaded with goods from the storage area to fixed picking workstations, where pickers retrieve the goods needed for an order from the shelves. After picking is completed, the robots then return the shelves to the storage area.

[0003] The core scheduling problem of this system involves two tightly coupled decision-making levels:

[0004] (a) Order Allocation Layer: When a customer order arrives, the system needs to decide which workstation to assign the order to and from which shelves to pick the required goods. This decision directly affects the handling distance of the shelves and the load balance of the workstations.

[0005] (ii) Robot Scheduling Layer: After the order is assigned, the system needs to plan a specific handling task sequence for each robot. Each handling task includes three stages: picking up the goods, that is, the robot goes to the storage location to retrieve the shelf, delivering the goods, that is, the robot moves the shelf to the target workstation, and returning the goods, that is, after picking, the robot returns the shelf to the empty storage location.

[0006] The system's primary optimization objective is to minimize completion time, which is the maximum time required for all robots to complete all handling tasks. Simultaneously, the system also focuses on average order completion time, a metric that measures the average waiting time from order placement to completion of picking, directly impacting customer experience.

[0007] To address the aforementioned scheduling problem, existing technologies mainly fall into three categories: (a) Phased approach: This type of approach breaks down order allocation and robot scheduling into two independent sub-problems, which are solved in sequence.

[0008] Typical practices include: (1) During the order allocation phase, heuristic rules such as shortest queue priority and workload balancing are used to allocate orders to workstations and shelves; (2) During the robot scheduling phase, greedy strategies such as nearest neighbor and earliest arrival are used to plan the transport route for the robot.

[0009] In the prior art, CN111680893B belongs to this type of method. This patent is for a robotic mobile picking system, namely the Amazon Kiva robot system, which realizes scheduling and rescheduling by constructing a digital twin of the physical system and the virtual system. It detects anomalies and triggers rescheduling by comparing the real-time data of the physical system with the simulation results of the virtual system, but order allocation and robot scheduling are still independent decisions, and true joint optimization is not achieved.

[0010] In the prior art, CN119175706A also belongs to this type of method. This patent optimizes logistics efficiency by dividing the shelf area and assigning a handling robot to each area. It belongs to a static allocation strategy based on area division and cannot adapt to scenarios where orders arrive dynamically and robot status changes in real time.

[0011] The drawback of the phased approach is that, because the two phases are optimized independently, there is a significant information barrier. Specifically, the order allocation phase does not consider the robot's real-time location and status, and may allocate orders to shelves that are far from idle robots; while the robot scheduling phase cannot correct the upstream allocation decision and can only passively execute it.

[0012] (ii) Joint optimization methods: These methods attempt to optimize order allocation and robot scheduling simultaneously. Typical approaches include: modeling the problem as a mixed-integer programming model and using metaheuristic algorithms, such as a combination of simulated annealing and beam search or a mathematical programming solver; or using a rolling time-domain mechanism to divide the dynamic problem into several static time windows and solve the optimization model once in each window.

[0013] In the prior art, CN111409997A belongs to this type of method. This patent aims to minimize the picking completion time of all orders and constructs an integer programming model for picking task allocation and robot handling sequence planning. Its method belongs to the joint optimization method, but it relies on traditional mathematical programming solvers, has high computational complexity, and is difficult to respond in real time in large-scale dynamic scenarios.

[0014] The drawback of the joint optimization method is that: The computational complexity is too high: solving the joint model requires a significant amount of computation time, and the combination of variables involved grows exponentially. For medium-sized problems, such as solving a problem involving 20 robots and 500 orders, a single solution typically takes several seconds to several minutes, which is far from meeting the requirements for real-time response.

[0015] The inherent limitations of static modeling: The rolling time-domain mechanism artificially divides a continuously changing dynamic system into a series of discrete static time windows. Within each window, the system assumes that the environmental state remains unchanged, but in reality, information such as robot position, shelf occupancy status, and order arrival are constantly changing. This leads to decision gaps at the window boundaries—the optimization scheme of the previous window may be outdated at the start of the next window and needs to be recalculated.

[0016] (iii) Path planning / local rule method. This type of method does not focus on the joint decision of order allocation and robot task scheduling, but instead narrows its focus to the robot's underlying path collision avoidance or local picking rule optimization.

[0017] Among the existing technologies, CN109543872B belongs to this type of method. This patent uses a combination of Monte Carlo tree search and neural network prediction to perform conflict-free path planning for multiple robots. It mainly focuses on conflict resolution at the path level and does not involve joint decision-making on order allocation and robot task scheduling.

[0018] US11148882B2, a prior art patent, also falls into this category. This patent describes an automated picking system based on a mobile robot that merges order goods through a multi-level merging process, supporting an early recall mechanism. However, its scheduling logic is based on preset rules and does not use a learning-based method for adaptive optimization.

[0019] In addition, the existing technology CN112925307A uses deep Q-networks in deep reinforcement learning for distributed path planning, but its focus is limited to the single path collision avoidance problem and does not involve the upper-level decision-making of order allocation and transportation task scheduling.

[0020] In summary, there is a fundamental contradiction in existing technologies: although the phased approach is fast, it cannot achieve global optimization due to information fragmentation; although the joint optimization approach considers global information, its computational complexity is too high and it cannot respond in real time; while path planning or local rule-based approaches are limited to low-level control or static rules and do not address the joint optimization problem of order allocation and robot scheduling.

[0021] Therefore, how to achieve global joint optimization of order allocation and robot scheduling while maintaining real-time response capability is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0022] To address the technical contradictions in existing technologies, where phased methods cannot achieve global optimization and joint optimization methods cannot meet real-time requirements, this invention proposes a joint scheduling method for warehouse robots based on soft order allocation and reinforcement learning, aiming to achieve the following objectives: 1. Joint optimization: Integrate order allocation and robot scheduling into a unified decision-making process, eliminating information barriers between the two stages, enabling robot scheduling to perceive order demand information, while order allocation can refer to the robot's real-time status; 2. Real-time response: Through event-driven decision-making mechanisms and forward inference of neural networks, real-time response to asynchronous system events is achieved, with single-step decision latency controlled at the millisecond level, meeting industrial-grade real-time scheduling requirements; 3. Performance Improvement: While ensuring real-time performance, significantly reduce system completion time and average order completion time, thereby improving the overall operational efficiency of the warehouse.

[0023] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a joint scheduling method for warehouse robots based on soft order allocation and reinforcement learning, comprising the following steps: S100. When a new order arrives, calculate the matching degree between the order and each shelf and each workstation, filter candidate shelves according to the matching degree, and update the soft order set, shelf popularity, and workstation popularity of each shelf; the soft order set, shelf popularity, and workstation popularity together constitute the soft allocation state. S200. When a robot scheduling event is triggered, the current system state is obtained, including the entity state and the soft allocation state; the system state is encoded into a heterogeneous graph, and a graph neural network is used to perform message passing on the heterogeneous graph to update the node embedding; according to the current event type, candidate actions are scored and an action probability distribution is generated, and a scheduling action is selected as the current robot scheduling decision. S300. During the execution of the scheduling action, the soft allocation state is transformed into a deterministic allocation: when the robot takes away the shelf, the currently available orders are determined based on the soft order set and inventory status of the shelf, and the remaining order information is updated; when the robot delivers the shelf to the workstation, the available orders are sent to the picking queue, and supplementary scheduling is triggered for the remaining orders.

[0024] Furthermore, in step S100, the formula for calculating the matching degree between the order and the shelf / workstation is:

[0025] in, For orders For the The quantity of goods required; For shelves Upper The current inventory quantity of this type of goods; This represents the total number of different types of goods in the warehouse. For shelves Current storage location to workstation The distance; It is a very small positive number.

[0026] Furthermore, in step S100, the specific method for filtering candidate shelves based on matching degree is as follows: for each workstation Only the top matching scores are retained. One shelf as a candidate set:

[0027] in, for At the workstation The corresponding positions in the column, arranged in descending order. The hyperparameter for the candidate set size is... For hypothetical orders Assigned to workstation The collection of candidate shelves at that time.

[0028] Furthermore, in step S100, the specific method for updating the soft order set, shelf popularity, and workstation popularity of each shelf is as follows: Add the current order to the soft order set of each shelf in its candidate shelf set; Update the shelf popularity to the original popularity plus the sum of the shelf's matching degree at each workstation; Update the workstation popularity to the original popularity plus the matching degree of the corresponding candidate shelf.

[0029] Furthermore, in step S200, the robot scheduling events include: an idle event triggered when the robot becomes idle, a pickup completion event triggered when the robot completes pickup, and a delivery completion event triggered after the robot completes delivery and picking. Among them, the candidate action corresponding to the idle event is to select the next shelf to be moved, the candidate action corresponding to the pickup completion event is to select the delivery workstation, and the candidate action corresponding to the delivery completion event is to select the next workstation or select an empty storage location to return the shelf.

[0030] Furthermore, in step S200, The entity status includes: the coordinates and operating stages of all robots, the coordinates and occupancy status of all storage locations, and the coordinates and workload of all workstations. The soft allocation status includes the shelf heat vector, the workstation heat vector, and the soft order set for each shelf.

[0031] Furthermore, in step S200, the specific method of encoding the system state into a heterogeneous graph includes: using robot nodes, storage location nodes, workstation nodes, and a stage node as nodes; using distance-weighted bidirectional edges between different types of nodes, and directed edges from stage nodes to other nodes as edges; the initial embedding of the robot and storage location is represented as the mapping of their original continuous features by the multilayer perceptron plus the learnable discrete state embedding, and the initial embedding of the workstation is represented as the mapping of its original continuous features by the multilayer perceptron.

[0032] Furthermore, in step S200, the graph neural network is a multi-layer heterogeneous graph Transformer, where each layer applies a graph attention operator to each relation type and sums the results for all relation types; wherein the attention coefficient of the graph attention operator is calculated based on the source node, the target node, and the distance between nodes.

[0033] Furthermore, in step S300, when the robot takes away the shelf, a greedy strategy is used to traverse the soft order set of the shelf, prioritize satisfying orders with high matching degree, and pre-deduct the demand of the satisfied orders from the shelf inventory; when all the demand of an order is fully satisfied, the matching degree value contributed by the order is deducted from the heat of the relevant shelf and workstation.

[0034] In a second aspect, embodiments of the present invention provide a joint scheduling system for warehouse robots based on soft order allocation and reinforcement learning, characterized in that it applies the method described in the first aspect, and the system includes: The soft order allocation module is used to calculate the matching degree between the new order and each shelf and each workstation when the new order arrives, filter candidate shelves according to the matching degree, and update the soft order set, shelf popularity, and workstation popularity of each shelf; the soft order set, shelf popularity, and workstation popularity together constitute the soft allocation status; The scheduling decision module is used to obtain the current system state when a robot scheduling event is triggered. The system state includes the entity state and the soft allocation state. The system state is encoded into a heterogeneous graph. The graph neural network is used to perform message passing on the heterogeneous graph to update the node embedding. Based on the current event type, the candidate actions are scored and an action probability distribution is generated. A scheduling action is selected as the current robot scheduling decision. The final allocation execution module is used to transform the soft allocation state into a deterministic allocation during the execution of the scheduling action: when the robot takes away the shelf, it determines the currently available orders and updates the remaining order information based on the soft order set and inventory status of the shelf; when the robot delivers the shelf to the workstation, it sends the available orders into the picking queue and triggers supplementary scheduling for the remaining orders.

[0035] Compared with the prior art, the present invention has the following beneficial effects: This invention integrates order allocation and robot scheduling into a unified optimization process through a soft allocation mechanism: when a new order arrives, instead of hard allocation, it calculates the matching degree, filters candidate shelves, and updates the soft order set, shelf popularity, and workstation popularity, forming a soft allocation state. This soft allocation state is directly referenced by the robot scheduling steps as global context information, enabling the robot to perceive the full picture of goods demand at the order level. Simultaneously, the scheduling actions selected by the robot scheduling steps, during execution, transform the soft allocation state into a deterministic allocation through the final allocation step, and feed back the execution results to update the soft allocation information, forming a two-way information closed loop. The above technical means overcome the information barrier problem caused by the separation of order allocation and robot scheduling in existing phased methods, and avoid the computational bottleneck caused by existing joint optimization methods coupling the two problems into a single large-scale mathematical programming model.

[0036] This invention employs an event-driven Markov decision process, initiating decisions only when asynchronous events such as robot idleness, pickup completion, and delivery completion occur, avoiding the reaction delays and computational waste caused by fixed-time-window scheduling. Simultaneously, by encoding the system state as a heterogeneous graph and utilizing a graph neural network for message passing, it achieves rapid aggregation and reasoning of global warehouse information. These technical means stabilize single-step decision latency at the millisecond level, enabling real-time responses to continuously changing information such as robot positions, shelf status, and order arrivals in the warehouse. This solves the problem of existing joint optimization methods having decision latency at the second or even minute level, and the decision gaps caused by assuming unchanged states within the window.

[0037] While ensuring real-time performance, this invention guides the robot to prioritize handling high-demand shelves through soft allocation, achieves global coordination through event-driven graph neural network encoding, guides the robot to avoid congested workstations and select the shortest return path through stage-specific biases, and maximizes the order fulfillment rate of a single handling operation through a greedy strategy and a heat deduction mechanism in the final allocation step. In real-world warehouse deployment verification, this invention significantly reduces completion time and average order completion time, while improving workstation throughput, shelf hit rate, and robot driving efficiency, achieving superior performance on several key operational metrics compared to the most advanced existing methods.

[0038] This invention employs a data-driven approach based on deep reinforcement learning. Its graph neural network encoder and action decoder can learn the optimal scheduling strategy for different warehouse layouts and order arrival patterns through training, overcoming the problems of existing path planning or local rule-based methods that rely on static preset rules and cannot adapt to changes in order patterns and warehouse scale expansion. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 This is a flowchart of the joint scheduling method for warehouse robots based on soft order allocation and reinforcement learning provided in this embodiment of the invention.

[0041] Figure 2 This is a schematic diagram of order allocation and robot scheduling in the joint scheduling method for warehouse robots based on soft order allocation and reinforcement learning provided in this embodiment of the invention.

[0042] Figure 3 This is a robot state transition diagram for the joint scheduling method of warehouse robots based on soft order allocation and reinforcement learning provided in this embodiment of the invention.

[0043] Figure 4 This is an overall framework diagram of the warehouse robot joint scheduling system based on soft order allocation and reinforcement learning provided in this embodiment of the invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Example 1: like Figure 1 As shown in the figure, this invention discloses a joint scheduling method for warehouse robots based on soft order allocation and reinforcement learning, including the following steps: S100. When a new order arrives, calculate the matching degree between the order and each shelf and each workstation. Based on the matching degree, filter candidate shelves and update the soft order set, shelf popularity, and workstation popularity of each shelf. The soft order set, shelf popularity, and workstation popularity together constitute the soft allocation state. When a new order arrives, do not immediately lock the specific shelf and workstation. Instead, calculate the matching degree and filter candidate shelves, update the soft order set and popularity information, and form a soft allocation state. This state retains the decision-making flexibility in subsequent scheduling.

[0046] S200. When a robot scheduling event is triggered, the current system state is obtained, including the entity state and the soft allocation state; the system state is encoded into a heterogeneous graph, and a graph neural network is used to perform message passing on the heterogeneous graph to update the node embedding; according to the current event type, candidate actions are scored and an action probability distribution is generated, and a scheduling action is selected as the current robot scheduling decision; scheduling is triggered by events such as robot idleness, pickup completion, or delivery completion, the current entity state and soft allocation state are obtained, global information fusion is achieved through heterogeneous graph encoding and graph neural network, and an action probability distribution is generated based on the event type to select the optimal scheduling action.

[0047] S300. During the execution of the scheduling action, the soft allocation state is transformed into a deterministic allocation: when the robot picks up the shelf, the currently fulfillable orders are determined based on the soft order set and inventory status of the shelf, and the remaining order information is updated; when the robot delivers the shelf to the workstation, the fulfillable orders are sent to the picking queue, and supplementary scheduling is triggered for the remaining orders. During the execution of the scheduling action, the soft allocation state is transformed into a deterministic allocation: when picking up goods, the fulfillable orders are determined based on the soft order set and inventory, and the remaining demand is updated; when delivering goods, the fulfillable orders are sent to the picking queue, and subsequent supplementary scheduling is triggered for unfulfilled orders, thus achieving delayed decision-making.

[0048] This invention integrates order allocation and robot scheduling through a soft allocation mechanism, overcoming the information barriers of phased methods and avoiding the computational bottleneck of joint optimization methods. It adopts event-driven and graph neural network inference, achieving millisecond-level single-step decision latency to meet real-time response requirements. It significantly reduces completion time and average order completion time, improving throughput and shelf hit rate. Furthermore, it has strong generalization ability from simulation to real warehouses, making it highly practical for engineering applications.

[0049] The following is a detailed description of the joint scheduling method for warehouse robots based on soft order allocation and reinforcement learning of the present invention: like Figure 2 As shown, order allocation and robot scheduling are jointly optimized through a soft allocation mechanism. Whenever a new order is generated... Upon arrival, the soft allocation process executes the following three steps in sequence: (a) Constructing the matching degree matrix: For arriving orders Construct a matching degree matrix Its dimensions are ( For the number of shelves, (Number of workstations). Each element in the matrix Quantify the shelf Move to workstation To meet orders The suitability of this plan can be calculated using the following formula:

[0050] in, For orders For the The quantity of goods required; For shelves Upper The current inventory quantity of this type of goods; This represents the total number of different types of goods in the warehouse. For shelves Current storage location to workstation The distance; For a very small positive number, such as This is used to prevent division by zero errors.

[0051] In this formula, the numerator represents how many orders require the goods on the shelf. For each type of goods, the smaller of the demand and the inventory is taken, since the actual number that can be picked cannot exceed the demand or the inventory. Then, the values ​​are summed. The denominator represents how far the shelf needs to be moved.

[0052] Dividing the two gives the quantity of goods that can be transported per unit distance. Therefore, The larger the value, the better the handling route: the more goods can be accommodated on the shelf and the shorter the handling distance.

[0053] (II) Candidate Shelf Screening: In real-world warehouses, the number of shelves is typically enormous. Processing all shelves post-processed would be computationally prohibitive. Therefore, the matching matrix needs to be pruned: for each workstation... Only the top matching scores are retained. One shelf as a candidate set:

[0054] in, for At the workstation The corresponding positions in the column, arranged in descending order. The hyperparameter for the candidate set size is... For hypothetical orders Assigned to workstation Which shelves are candidate shelves to meet the order requirements?

[0055] The purpose of this step is to increase the number of candidate solutions from [previous number] with almost no loss of decision quality. Compress to This significantly reduces the overhead of subsequent calculations.

[0056] (III) Soft allocation information update: This step maintains two types of global information structures: a soft order set and a heat vector, which serve as an information bridge connecting the order allocation layer and the robot scheduling layer.

[0057] (a) Soft order set update: make For each shelf in the union, find the union of all candidate workstation sets. , will the current order Add it to its soft order collection:

[0058] Soft order collection Record which orders might require shelving The goods on the shelf. This tells the robot which orders might benefit from moving this shelf during scheduling.

[0059] (b) Shelf heat vector update: For each shelf in the candidate set The matching degree of each workstation is accumulated, and the shelf popularity is updated:

[0060] in, This is an indicator function that takes the value 1 if the condition is true and 0 otherwise. Shelf heat. This is a scalar value; the higher the value, the more unfulfilled orders require that shelf, and the greater the profit from moving that shelf. When selecting pickup targets, robots tend to prioritize moving shelves with higher demand, thus achieving efficient operation.

[0061] (c) Workstation heat vector update: For each workstation Accumulate the matching degree of its candidate shelves and update the workstation popularity:

[0062] Workstation popularity This reflects the sufficiency of available goods supply near the workstation, which helps guide the robot to make more balanced decisions when selecting delivery targets.

[0063] The heat vector output in soft allocation , And soft order collection Together, these constitute the global context information connecting order allocation and robot scheduling, enabling robot scheduling to see the full picture of demand at the order level. In existing phased methods, robot scheduling cannot see order information at all and can only blindly execute tasks allocated upstream.

[0064] To achieve real-time response to asynchronous events, this invention models the scheduling process as an event-driven Markov decision process and uses deep reinforcement learning to train the scheduling strategy.

[0065] Markov Decision Processes (MDPs) are the standard mathematical framework for reinforcement learning, used to describe the cyclical process of an agent taking an action in a given state, receiving a reward, and transitioning to the next state. Traditional MDPs assume that decision steps occur at fixed time intervals, but in a real warehouse, the timing of different events is irregular—for example, one robot might complete picking up an item in 3 seconds, while another might take 30 seconds.

[0066] This invention proposes an event-driven Markov decision process, where decision steps are triggered by discrete asynchronous events, and the time interval between decision steps is not fixed. The scheduling cycle is driven by the following three types of events, each corresponding to a specific optional action: Idle Event: Triggered when any robot completes its previous task and becomes idle. At this time, the robot needs to select the next shelf to move – that is, select a storage location with a shelf to retrieve the item. The selectable target is any storage location occupied by shelves.

[0067] Pickup Complete Event: Triggered when the robot arrives at the target storage location and successfully retrieves the shelf. At this point, the robot needs to select which workstation to deliver the shelf to. The target can be any workstation.

[0068] Delivery Complete Event: Triggered when the robot delivers the shelf to the workstation and the picker completes the pickup. At this point, the robot needs to decide its next action—whether to send the shelf to another workstation to continue serving other orders, or to return the shelf to an empty storage location. Possible objectives are all workstations and all empty storage locations.

[0069] After the robot returns the shelf, its status changes to idle, completing a full scheduling cycle and triggering a new idle event, such as... Figure 3 As shown, this forms a continuous decision-making cycle.

[0070] In this decision-making loop, the system transforms warehouse status information into a mathematical representation that can be processed by a neural network. The specific process is as follows.

[0071] (a) Initial state composition: In each decision step triggered by an event The initial state of the system consists of two parts: Entity status The physical state of the warehouse, including the coordinates and operating stages of all robots, the coordinates and occupancy status of all storage locations, and the coordinates and workload of all workstations; soft allocation state Guidance information from the soft order allocation module, including shelf popularity vectors. Workstation heat vector and the collection of soft orders for each shelf .

[0072] (b) Entity feature extraction: The system models key objects in the warehouse as three types of entities: robots, storage locations, and workstations. Shelves are considered as auxiliary features of robots or storage locations—that is, shelves carried by robots or shelves placed on storage locations—rather than independent entities, which effectively controls the scale of the graph.

[0073] The original feature vectors of each entity contain the following information: Robot characteristics: coordinate position, current number of tasks, soft allocation heat of the shelf it carries, and number of soft orders; Storage location characteristics: coordinates, distance to the current decision-making robot, number of tasks, shelf soft allocation popularity, number of soft orders; The characteristics of a workstation include: coordinate location, distance to the current decision-making robot, number of tasks, soft allocation heat, workload (i.e., the number of unprocessed goods) and queuing costs.

[0074] (c) Initial embedding generation: Because robots and storage locations have highly dynamic discrete states, such as robots being idle, picking up goods, or delivering goods, and storage locations being in a state with or without shelves, this invention adds the learnable discrete state embedding vector to the vector of continuous features mapped by a multilayer perceptron to obtain the initial embedding representation of each entity:

[0075]

[0076]

[0077] in It is a multilayer perceptron, a basic neural network structure used to transform raw numerical features into vector representations of fixed dimensions; and The learnable state is embedded in a lookup table, and the corresponding vector is output according to the entity's current discrete state, such as "idle" or "picking up goods". The representation vectors for different entities, such as robots, storage locations, and workstations, are at layer 0, i.e., the initial state. The workstation state is relatively stable, so no additional state embedding is used.

[0078] (d) Heterogeneous graph construction and graph neural network encoding: Complex interactions exist between different types of entities in a warehouse—robots need to travel to storage locations to retrieve goods and then deliver them to workstations. To enable neural networks to understand these interactions, this invention constructs all entities as a heterogeneous graph.

[0079] A heterogeneous graph is a graph that contains multiple types of nodes and multiple types of edges. In this invention: The nodes include all robots, storage locations, workstations, and a special stage node—this node uses a learnable embedding specific to the current event type. Initialization is used to inject event type information, such as idle / pickup completed / delivery completed, into the global code.

[0080] Edges include bidirectional edges between different types of entities, weighted by distance, such as six relationship types including robot-workstation, robot-storage location, and workstation-storage location, as well as directed edges from stage nodes to all other nodes.

[0081] A layer-L heterogeneous graph Transformer is used to complete global message passing. The core logic of message passing is to allow each node in the graph to aggregate information from its neighbors, thereby updating its own feature representation. Similar to a warehouse scenario, each entity can perceive the operational status and related information of other entities in its vicinity. Each layer of the model introduces graph attention operators for different relation types, and finally integrates and sums the outputs of all relation types.

[0082] in There are six types of relation types in total. This is the distance matrix between nodes.

[0083] The core idea of ​​graph attention operators is that when nodes aggregate neighbor information, they do not treat all neighbors equally, but rather distinguish the importance of different neighbors through learned attention weights. For example, when a robot makes a decision, nearby workstations and frequently used shelves should receive higher attention.

[0084] Specifically, the target node The update formula is:

[0085] Attention coefficient The calculation method is as follows:

[0086] in , , The weight matrix is ​​a learnable matrix, used for the source node, target node, and edge features, respectively, which is a linear transformation of the distance. This is a learnable attention parameter vector; For nodes and The distance between them.

[0087] To stabilize training and prevent gradient degradation in deep networks, gradient degradation is performed on each node type in each layer. Apply residual connectivity and layer normalization:

[0088] go through After layer message passing, the embedding of each entity It integrates global contextual information from different types of entities—each node not only knows its own state, but also understands the distribution of surrounding robots, the demand for shelves, and the load of workstations.

[0089] After the state encoding is completed, the stage knowledge-guided decoder uses the encoded entity representation to generate the optimal action.

[0090] This decoder learns decision-making capabilities from a large amount of training data through neural networks on the one hand; on the other hand, it injects the warehouse scheduling experience of human experts into the decision-making process in the form of mathematical biases, thereby accelerating learning and improving decision quality.

[0091] For currently active robots and any candidate target entity This can be a storage location, a workstation, or an empty storage location, depending on the current event type. First, calculate the raw score:

[0092] in This represents vector concatenation. and These are the embedding representations of the candidate target and the currently active robot in the last layer of the graph neural network encoder, respectively.

[0093] For stage-specific biases, inject human domain knowledge as prior guidance based on the current event type: During the pickup phase, i.e. when the idle event is triggered: ,in The bias is based on the popularity of candidate shelves. This bias means guiding the robot to prioritize shelves with higher popularity—those needed by more orders—thereby maximizing the number of orders that can be served in a single handling operation.

[0094] The delivery phase, i.e., when the pickup completion event is triggered: ,in For workstations The current workload. The negative sign gives a lower bias to workstations with heavier workloads, thus guiding the robot to avoid congested workstations and achieving global load balancing.

[0095] The return phase occurs when the delivery completion event is triggered: ,in This represents the distance from the current robot to the candidate empty storage location. This bias guides the robot to select the nearest empty storage location to return the shelf, minimizing empty travel distance and accelerating robot turnaround.

[0096] Then, based on the action space corresponding to the current event type, a validity mask is applied. By setting the score of invalid candidate targets to negative infinity, making them impossible to select, the policy probability distribution is obtained:

[0097] Finally, the action is sampled from this probability distribution. In the actual deployment phase, a greedy strategy can also be chosen, directly selecting the action with the highest probability to obtain a deterministic output.

[0098] When the robot makes a scheduling decision At the same time, the system transforms soft order allocation into deterministic final allocation. This is a key step in "delayed decision-making"—final allocation occurs when the robot actually picks up and delivers the goods, rather than when the order initially arrives, thus utilizing the latest system state information.

[0099] (a) Picking stage – Order allocation to shelves: When the robot chooses to take the shelf At that time, it is necessary to determine the soft order set. Which orders can be fulfilled by the current shelf inventory? Process them using the following greedy algorithm: 1. Initialization satisfies the set and the remainder set ; 2. Traversal Each order : –If the shelf Current inventory can meet orders All the needs, that is, all goods All Then join in And pre-deduct order demand from shelf inventory: ; , The vectors represent shelf inventory and order demand, respectively, and the number at the i-th position represents the shelf inventory and order demand of the i-th product.

[0100] Otherwise, Add to the remaining set We await replenishment through other shelves to meet our needs.

[0101] 3. Update global soft allocation information: Due to Orders may be soft-assigned to multiple candidate shelves, meaning the same order may appear on multiple shelves. It's necessary to clear the impact of these orders on other shelves and workstations.

[0102]

[0103]

[0104] (b) Delivery Stage – Order Allocation to Workstations and Handling of Remaining Orders: When the robot puts the shelf Send to workstation hour: 1. Sets that can be satisfied The order has officially entered the workstation. The picking queue waits for the pickers to complete the picking of goods; 2. For the remainder set For each order in the system, i.e., an order that cannot be fully fulfilled by a single shelf, the system processes it according to the workstation. The matching degree is used to sort other shelves, and additional shelves are greedily selected to supplement the remaining demand until all goods demand of the order is covered.

[0105] The training paradigm describes how to use reinforcement learning algorithms to train the above policy network so that it learns to make high-quality scheduling decisions.

[0106] The core idea of ​​reinforcement learning is that an agent receives reward signals by interacting with the environment and gradually learns a strategy that maximizes cumulative rewards. However, directly using completion time as the optimization objective presents two training difficulties: The sparse feedback problem: the completion time can only be calculated after all tasks have been completed, and there is no immediate feedback for each decision made during the intermediate process. Technically, this is known as the "delayed credit allocation" problem.

[0107] Zero gradient problem: The completion time is defined as the maximum completion time of all robots, i.e. Since the gradient of the maximum function is only passed to the robot that reaches the maximum value, which is the system bottleneck, all other robots do not receive any training signals at all—they do not know whether their behavior affects the final result, and therefore cannot learn to improve their strategies.

[0108] To address these two issues, this invention proposes a reward shaping strategy based on the p-norm. The core idea is to approximate the non-differentiable operation of maximizing the value using a differentiable smooth function.

[0109] make For decision-making steps All The cumulative active time vector of the robot is defined by the state potential function:

[0110] The key property of the norm is: when... When a relatively large finite value is chosen, it can both approximate the completion time, i.e. the maximum value, and allow all robots to obtain non-zero gradient signals.

[0111] Construct a dense reward signal for each step based on the potential function:

[0112] in This is the discount rate. The intuitive meaning of this reward is: to encourage the agent to reduce [the discount rate] at each step. The increase in the norm prevents the cumulative time of any single robot from growing too rapidly, thus achieving balanced scheduling. This allows the agent to receive immediate feedback at each decision step, without having to wait for all tasks to be completed.

[0113] This invention employs a proximal policy optimization algorithm to train the policy network. A unique challenge of event-driven Markov decision processes is the non-fixed time interval between decision steps. For example, a pickup completion event may occur after several seconds, while waiting for the workstation to complete picking may take tens of seconds. Standard policy optimization algorithms assume equally spaced decision steps and cannot properly handle this irregular time structure.

[0114] Therefore, this invention incorporates the duration of physical time into a discount mechanism. For the first The physical timestamps of each decision event, and the time interval between adjacent decision steps are... Define the dynamic discount factor:

[0115] in This represents the standard discount rate per unit of time. The longer the time interval, the larger the discount, which aligns with intuition—future rewards further away from the present have less guiding significance for current decisions.

[0116] The time-aware timing difference error is calculated as follows:

[0117] The generalized advantage estimation of time perception is calculated as follows:

[0118] in For smoothing parameters, Indicates the time from the current step to the 1st step. The cumulative physical time of each step.

[0119] The final training objective function is:

[0120] The three items are: pruning the agent objective function to prevent training collapse by limiting the policy update magnitude, loss of the value function used to train the evaluation network, and the entropy regularization term of the policy to encourage exploration and avoid premature convergence to a local optimum. , These are the corresponding weighting coefficients.

[0121] In real-world deployments, large-scale warehouses may contain hundreds of robots and thousands of storage locations. Encoding all entities into a heterogeneous graph would result in an excessively large graph, causing inference latency that exceeds real-time requirements. Therefore, this invention introduces an entity pruning mechanism, encoding only the most relevant subset of entities during inference: 1. Robotic Pruning: For each decision, only the robot that is closest to the current decision-maker is encoded. One robot, because a robot at a distance has little impact on the current decision; 2. Storage location pruning: Only coded shelf popularity. Top-ranked The storage location corresponding to each shelf, as well as the currently empty storage location, are used for the return phase, because shelves with low activity are unlikely to be selected for transport.

[0122] This optimization keeps the number of entities encoded in each decision within a manageable range, ensuring that inference latency is stably controlled at the millisecond level, meeting the requirements of industrial-grade real-time scheduling.

[0123] The following section uses a warehouse of moderate size as an example to fully demonstrate the entire process of the system of the present invention, from order arrival, soft allocation, robot scheduling to final order allocation.

[0124] I. Initial state of the warehouse: Warehouse layout, warehouse is A grid map with coordinate origin. Located in the lower left corner. The warehouse contains 4 categories of goods, labeled A, B, C, and D, 8 shelves S1~S8, 4 workstations W1~W4, and 3 robots R1~R3. The specific locations of the workstations are shown in Table 1, the initial inventory of the shelves is shown in Table 2, and the initial status of the robots is shown in Table 3.

[0125] Table 1: Specific Locations of Workstations

[0126] Table 2 Initial Inventory on Shelves

[0127] Table 3: Initial State of the Robot

[0128] All shelf and workstation heat values ​​are initially set to 0.

[0129] II. Order Arrival and Soft Allocation Process: Order 1 arrived (time) Second) Step 1: Construct the matching degree matrix: For each shelf and workstation Calculate the matching degree :

[0130] Taking shelf S1 (coordinates (1,1), inventory A=5, B=3) as an example: molecule = (Can fully meet all the requirements of Order 1) Distance to W1(0,0) = ,but

[0131] Distance to W2(6,0) = ,but

[0132] Distance to W3(0,6) = ,but

[0133] Distance to W4(6,6) = ,but

[0134] Taking shelf S2 (coordinates (2,2), inventory A=0, B=4) as an example: molecule = (S2 can only satisfy product B, but cannot provide product A) The distance to W1 is 4, then

[0135] Taking shelf S6 (coordinates (4,4), inventory A=4, B=0) as an example: molecule = (S6 can provide product A, but not product B) Distance to W4 = ,but

[0136] Table 4: Complete Matching Degree Matrix

[0137] Key observation: S1 has the highest match rate (2.50) with W1 because S1 fully meets the order requirements and is the closest. S7 has a match rate of 0 because it does not contain products A and B.

[0138] Step 2: Candidate Shelf Screening (Top-K, K=3) As shown in Table 5, for each workstation, the top 3 shelves with the highest matching degree are retained: Table 5 Specific Shelves for Workstations

[0139] Step 3: Update soft allocation information (a) Soft order set update: Add order 1 to the soft order set for each selected shelf:

[0140] The S7 was not selected as a candidate by any workstation, therefore .

[0141] (b) Shelf popularity update: Accumulate the matching degree of each shelf on its selected workstation: (Selected by workstations W1, W2, and W3) (Selected only by W2) (Selected by W1, W2, and W4) (Selected only by W4) (Selected by W1 and W3) (Selected only by W3) (Selected only by W4) Updated shelf popularity vector:

[0142] Key information: S1 has the highest popularity (4.16), indicating that it is the most valuable shelf to satisfy order 1.

[0143] (c) Workstation popularity update:

[0144]

[0145]

[0146]

[0147] Order 2 has arrived (time t = 5 seconds). Order details: Requires 4 items C and 2 items D.

[0148] The same three-step process is executed: matching degree matrix construction, Top-K filtering, and soft assignment update.

[0149] Taking shelf S2 (coordinates (2,2), inventory C=6, D=0) as an example: molecule =

[0150] The distance to W1 is 4, then

[0151] Taking shelf S7 (coordinates (1,5), inventory C=5, D=3) as an example: molecule = (S7 can fully meet order 2) Distance to W3 = ,but

[0152] Result: S7 achieved the highest match score because it fully satisfied order 2 and was close to W3. The popularity of S7 increased significantly from 0, and the popularity of S2 also increased further because it included product C.

[0153] After soft allocation of order 2, the heat vector is updated as follows (example value):

[0154] The S7's popularity jumped from 0 to 3.50, making it the second most popular item on the shelf after the S1.

[0155] III. Robot Scheduling Process: Idle Event: R1 selects a pickup target (time) Second) All three robots are in an idle state. The system processes them sequentially, first making a decision for R1.

[0156] State Encoding: The system encodes all entities—3 robots, 8 storage bits, and 4 workstations—into a heterogeneous graph. Each entity's features include its coordinates, heat level, number of tasks, etc. Stage nodes are initialized with embeddings of idle event types.

[0157] Motion space: All storage locations with shelves .

[0158] Phase bias calculation: Shelf heat is used as the bias during the picking phase.

[0159] Table 6: Evaluation data of candidate shelves when robot R1 selects a pickup target after an idle event is triggered.

[0160] The neural network comprehensively considers shelf popularity, namely bias guidance and the distance from the robot to the shelf, which is achieved through graph attention encoding. Ultimately, R1 selected S1—although it is not the closest, it has the highest popularity, and S1 can handle the most unfinished orders.

[0161] Decision result: R1 goes to storage location (1,1) to retrieve shelf S1. The estimated journey involves 4 steps, with an estimated arrival time of approximately [time missing]. Second.

[0162] Idle Event: R2 selects pickup target (time) Second) Next, a decision is made for R2. At this point, S1 has already been reserved by R1, and its storage space has been marked as an existing task. The heat of S1 will be consumed when R1 retrieves the task.

[0163] Of the popularity ratings observed by R2, S7 (popularity 3.50) had the highest remaining shelf space. After comprehensive evaluation by the neural network, R2 selected S7.

[0164] Decision result: R2 goes to storage location (1,5) to retrieve shelf S7.

[0165] Idle Event: R3 selects pickup target (time) Second) R3 sees that S1 and S7 have already been reserved, and the remaining most popular options are S2 (2.50) and S8 (1.87). S2 and S8 are both two steps away from R3, but S2 has higher popularity, so R3 chooses S2.

[0166] Decision result: R3 goes to storage location (2,2) to retrieve shelf S2.

[0167] Pickup completion event: R3 selects delivery workstation (time) Second) R3 is closest to S2 (2 steps) and completes pickup first. The system triggers a pickup completion event.

[0168] Final order allocation – pickup stage: After R3 removes shelf S2, check the soft order set of S2. : Order (Requires A×3, B×2): S2 inventory A=0, unable to meet the demand for A → add to the remaining set

[0169] Order (Requires C×4, D×2): S2 inventory C=6≥4, but D=0<2, which cannot be fully satisfied → add to the remaining set.

[0170] result: , S2 cannot fully satisfy either order on its own, but the goods C it carries still have value for order 2.

[0171] System cleanup , The soft allocation contribution to S2 is used to update the global heat vector.

[0172] Action space: All workstations .

[0173] Stage bias calculation (the negative logarithm of the workstation load is used as the bias during the delivery stage):

[0174] At this point, all workstations have a load of 0 and the bias is the same, so the neural network mainly makes decisions based on distance and graph encoding information. R3 is currently located at (2,2), with a distance of 4 to W1(0,0) and a distance of 6 to W2(6,0).

[0175] Decision result: R3 chooses to send S2 to W1 (the nearest workstation).

[0176] Pickup completion event: R1 selects delivery workstation (time) Second) R1 reaches the storage location (1,1) where S1 is located and takes the shelf S1.

[0177] Final order allocation – pickup stage: Check the soft order set of S1 : Order (Requires A×3, B×2): S1 inventory A=5≥3, B=3≥2, fully satisfied, add.

[0178] result: The system pre-deducts from the S1 inventory: A decreases from 5 to 2, and B decreases from 3 to 1.

[0179] Simultaneously clear The soft allocation contribution to all other shelves—S3, S4, S5, S6, and S8—decreases accordingly because... It has been assigned to S1.

[0180] Phase bias: W1 has a load of 1 (R3 is delivering or has been delivered), while W2 / W3 / W4 have a load of 0.

[0181]

[0182] W1's bias is significantly lower than other workstations, guiding R1 to avoid W1. R1 is currently at (1,1), a distance of 2 from W1 (very close), but the load bias causes W1's overall score to decrease. After weighing distance and load, the neural network chooses W2 (distance 6, but load 0).

[0183] Decision result: R1 will send S1 to W2.

[0184] Effect: This decision reflects load balancing – although W1 is closer, the system guides the robot to distribute the shelves to different workstations, avoiding congestion on W1.

[0185] Delivery completed event: R3 decides the next step (moment) Second); R3 delivers S2 to W1, and the picker takes the goods C×4 required for order 2 from S2.

[0186] Final order allocation – delivery stage: Orders in the W1 queue enter the picking queue. for For order 2 (which is missing 2 items D), the system greedily matches it with additional shelves.

[0187] Action space: The union of all workstations and all empty memory locations.

[0188] At this point, S2's soft order set had no other pending orders, meaning its activity level had dropped to zero, and no other workstations urgently needed the shelf. R3 decided to return S2.

[0189] Phase bias: The negative logarithm of the distance is used as the bias during the return phase.

[0190] R3 is currently in W1(0,0), and the original storage location of S2 (2,2) is 4 units away. If there is a closer empty storage location in the warehouse, R3 will be selected first. Assume that (2,2) is empty (it automatically becomes an empty storage location after S2 is taken), and the distance is 4 units.

[0191] Decision result: R3 chooses to return S2 to the nearest empty storage location (2,2) (i.e., the original location of S2).

[0192] Return completed, a new round of idle events begins (time). Second) R3 returns S2 to (2,2), the state becomes idle, and a new idle event is triggered. At this time, new orders may have arrived in the system, the soft allocation information has been updated, and R3 will select the next shelf to be moved based on the latest heat vector.

[0193] The cycle continues until all orders are completed.

[0194] In this embodiment, order 1 is in Upon arrival, traditional hard allocation methods immediately lock the system into the S1-W1 scheme. However, the soft allocation in this system only calculates the matching degree and popularity, without making a final decision.

[0195] arrive At this point, the final allocation only occurs after R1 actually takes S1. Since W1 already has R3 delivering goods and its load is 1, the system reassigns S1 to W2, achieving load balancing. If in If it is forcibly assigned to W1, this information cannot be used.

[0196] Efficient content delivery guided by trending topics: S1's popularity (4.16) is much higher than other shelves because it can fully meet all the needs of order 1. R1 therefore prioritizes handling S1.

[0197] In contrast, while S2 (popularity 2.50) was also moved by R3, it could only partially meet the order demand. The system uses popularity as a guide, ensuring that the most valuable shelves are moved first.

[0198] Event-driven real-time response: In this embodiment, R3 is The item picked up first is the closest because the system responds immediately and makes a delivery decision. R1 in When the pickup is completed in seconds, the system uses the latest information on the increased load on W1 after R3 delivers the goods to make a better decision.

[0199] This event-driven mechanism ensures that every decision is based on the latest system state, rather than an outdated static snapshot.

[0200] In this embodiment, different bias strategies are used for the three types of events: Picking phase: From heat bias to prioritizing the handling of shelves needed by the most orders; Delivery phase: From load bias to avoiding congested workstations, achieving load balancing; Return phase: From distance offset to nearby return, reducing empty runs.

[0201] These biases incorporate the scheduling experience of human experts into the decision-making of neural networks, which both accelerates learning and improves the quality of decision-making.

[0202] Example 2: like Figure 4 As shown, based on the same inventive concept, this embodiment of the invention also provides a joint scheduling system for warehouse robots based on soft order allocation and reinforcement learning, the system comprising: The soft order allocation module is used to calculate the matching degree between the new order and each shelf and each workstation when the new order arrives, filter candidate shelves according to the matching degree, and update the soft order set, shelf popularity, and workstation popularity of each shelf; the soft order set, shelf popularity, and workstation popularity together constitute the soft allocation status; The scheduling decision module is used to obtain the current system state when a robot scheduling event is triggered. The system state includes the entity state and the soft allocation state. The system state is encoded into a heterogeneous graph. The graph neural network is used to perform message passing on the heterogeneous graph to update the node embedding. Based on the current event type, the candidate actions are scored and an action probability distribution is generated. A scheduling action is selected as the current robot scheduling decision. The final allocation execution module is used to transform the soft allocation state into a deterministic allocation during the execution of the scheduling action: when the robot takes away the shelf, it determines the currently available orders and updates the remaining order information based on the soft order set and inventory status of the shelf; when the robot delivers the shelf to the workstation, it sends the available orders into the picking queue and triggers supplementary scheduling for the remaining orders.

[0203] Since these systems and the principles they address are similar to the aforementioned joint scheduling method for warehouse robots based on soft order allocation and reinforcement learning, the implementation of this system can refer to the implementation of the aforementioned method, and the repetitions will not be repeated.

[0204] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0205] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this invention may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A warehouse robot joint scheduling method based on soft order allocation and reinforcement learning, characterized in that, Includes the following steps: S100. When a new order arrives, calculate the matching degree between the order and each shelf and each workstation, filter candidate shelves according to the matching degree, and update the soft order set, shelf popularity, and workstation popularity of each shelf; the soft order set, shelf popularity, and workstation popularity together constitute the soft allocation state. S200. When a robot scheduling event is triggered, the current system state is obtained, including the entity state and the soft allocation state; the system state is encoded into a heterogeneous graph, and a graph neural network is used to perform message passing on the heterogeneous graph to update the node embedding; according to the current event type, candidate actions are scored and an action probability distribution is generated, and a scheduling action is selected as the current robot scheduling decision. S300. During the execution of the scheduling action, the soft allocation state is transformed into a deterministic allocation: when the robot takes away the shelf, the currently available orders are determined based on the soft order set and inventory status of the shelf, and the remaining order information is updated. When the robot delivers the shelf to the workstation, the orders that can be fulfilled will be sent to the picking queue, and supplementary scheduling will be triggered for the remaining orders.

2. The method as described in claim 1, characterized in that, In step S100, the formula for calculating the matching degree between the order and the shelf / workstation is: in, For orders For the first The quantity of goods required; For shelves Upper The current inventory quantity of this type of goods; This represents the total number of different types of goods in the warehouse. For shelves Current storage location to workstation The distance; It is a very small positive number.

3. The method as described in claim 1, characterized in that, In step S100, the specific method for filtering candidate shelves based on matching degree is as follows: for each workstation Only the top matching scores are retained. One shelf as a candidate set: in, for At the workstation The corresponding positions in the column, arranged in descending order. The hyperparameter for the candidate set size is... For hypothetical orders Assigned to workstation The collection of candidate shelves at that time.

4. The method as described in claim 1, characterized in that, In step S100, the specific method for updating the soft order set, shelf popularity, and workstation popularity of each shelf is as follows: Add the current order to the soft order set of each shelf in its candidate shelf set; Update the shelf popularity to the original popularity plus the sum of the shelf's matching degree at each workstation; Update the workstation popularity to the original popularity plus the matching degree of the corresponding candidate shelf.

5. The method as described in claim 1, characterized in that, In step S200, the robot scheduling events include: an idle event triggered when the robot becomes idle, a pickup completion event triggered when the robot completes pickup, and a delivery completion event triggered after the robot completes delivery and picking. Among them, the candidate action corresponding to the idle event is to select the next shelf to be moved, the candidate action corresponding to the pickup completion event is to select the delivery workstation, and the candidate action corresponding to the delivery completion event is to select the next workstation or select an empty storage location to return the shelf.

6. The method as described in claim 1, characterized in that, In step S200 The entity status includes: the coordinates and operating stages of all robots, the coordinates and occupancy status of all storage locations, and the coordinates and workload of all workstations. The soft allocation status includes the shelf heat vector, the workstation heat vector, and the soft order set for each shelf.

7. The method as described in claim 1, characterized in that, In step S200, the specific method of encoding the system state into a heterogeneous graph includes: using robot nodes, storage location nodes, workstation nodes, and a stage node as nodes; using distance-weighted bidirectional edges between different types of nodes, and directed edges from stage nodes to other nodes as edges; the initial embedding of the robot and storage location is represented as the mapping of their original continuous features by the multilayer perceptron plus the learnable discrete state embedding, and the initial embedding of the workstation is represented as the mapping of its original continuous features by the multilayer perceptron.

8. The method as described in claim 1, characterized in that, In step S200, the graph neural network is a multi-layer heterogeneous graph Transformer. Each layer applies a graph attention operator to each type of relation and sums the results for all relation types. The attention coefficient of the graph attention operator is calculated based on the source node, the target node, and the distance between nodes.

9. The method as described in claim 1, characterized in that, In step S300, when the robot takes away the shelf, a greedy strategy is used to traverse the soft order set of the shelf, prioritize the fulfillment of orders with high matching degree, and pre-deduct the fulfilled order requirements from the shelf inventory; when all the requirements of an order are fully met, the matching degree value contributed by the order is deducted from the heat of the relevant shelf and workstation.

10. A joint scheduling system for warehouse robots based on soft order allocation and reinforcement learning, characterized in that, The system comprises: The soft order allocation module is used to calculate the matching degree between the new order and each shelf and each workstation when the new order arrives, filter candidate shelves according to the matching degree, and update the soft order set, shelf popularity, and workstation popularity of each shelf; the soft order set, shelf popularity, and workstation popularity together constitute the soft allocation status; The scheduling decision module is used to obtain the current system state when a robot scheduling event is triggered. The system state includes the entity state and the soft allocation state. The system state is encoded into a heterogeneous graph. The graph neural network is used to perform message passing on the heterogeneous graph to update the node embedding. Based on the current event type, the candidate actions are scored and an action probability distribution is generated. A scheduling action is selected as the current robot scheduling decision. The final allocation execution module is used to transform the soft allocation state into a deterministic allocation during the execution of the scheduling action: when the robot takes away the shelf, it determines the currently available orders and updates the remaining order information based on the soft order set and inventory status of the shelf; when the robot delivers the shelf to the workstation, it sends the available orders into the picking queue and triggers supplementary scheduling for the remaining orders.

Images