Order scheduling decision-making methods, apparatus, computer equipment, and readable storage media
By acquiring the cumulative return prediction of the delivery object wave status and order sequence, the inefficiency caused by the independence of order backlog decision-making is solved, and the global optimization and intelligence improvement of the order scheduling system are realized.
Patent Information
- Application Number
- CN202611207412.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-10
- Publication Date
- 2026-11-17
AI Technical Summary
In the intelligent dispatch system for instant delivery, the decision to hold orders is independent of each other, and the interaction between orders and transportation resources within a wave cannot be effectively coordinated. As a result, the potential for system-level optimization is not fully released, and local optimal decisions affect overall efficiency, resulting in poor intelligence.
By acquiring the wave status of delivery objects, initializing the sequence of decided actions, combining it with the order sequence to predict cumulative returns, and continuously updating the sequence of decided actions to form the target sequence of decided actions, so as to take into account the collaborative and competitive relationships between orders and transform it into a serialized decision-making process.
It improves the intelligence of order scheduling decisions and the overall scheduling efficiency, avoids the limitation of local optima that can easily occur when optimizing a single order independently, and achieves global optimization.
Smart Images

Figure CN123117604A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of Internet technology, and in particular to an order scheduling decision-making method, apparatus, computer equipment, and readable storage medium. Background Technology
[0002] In intelligent dispatch systems for on-demand delivery, the order management module, as a key component in balancing rider capacity and order timeliness, has become an important application scenario for reinforcement learning technology in recent years. The order management module mainly focuses on the abstract modeling of order management scenarios. By treating orders as independent decision-making entities, it optimizes time resources throughout the order lifecycle, aiming to uncover the potential time value of waiting time and the risk of delays, and avoid time waste caused by resource mismatch.
[0003] In related technologies, an order-level single-agent framework is typically used for order placement decisions. This framework treats each order as an independent agent, collecting samples of each round of order placement decisions throughout the entire process from order creation to rider acceptance. The decision space consists of two actions: assignment or order placement. For each assignment sample, the decision action, state features, action reward, and next state features are extracted, where the action reward is defined as the time value of the order itself. Based on this, an offline reinforcement learning method is used to construct a decision model, with the optimization objective of maximizing the single-dimensional time value, to achieve independent control over the timing of order placement for each order.
[0004] However, the applicant recognizes that the relevant technology has at least the following technical problems in its implementation: Under a unified scheduling platform, the order consolidation decisions for each order are independent of each other, making it impossible to effectively coordinate the interaction between orders within a wave in terms of time and delivery resources. This results in the failure to fully unleash the potential for system-level optimization. Moreover, the local optimal order consolidation decisions at the order level are not entirely positively correlated with the global optimal decisions at the rider wave level. The independent decision-making mechanism is prone to getting stuck in local optima, affecting the overall order delivery efficiency at the wave level, resulting in low efficiency and poor intelligence in order scheduling decisions. Summary of the Invention
[0005] In view of this, this application provides an order scheduling decision-making method, apparatus, computer equipment, and readable storage medium. The main purpose is to solve the problem that the current independent decision-making mechanism is prone to getting stuck in local optima, affecting the overall order delivery efficiency in the wave dimension, resulting in low efficiency and poor intelligence in order scheduling decision-making.
[0006] According to a first aspect of this application, an order scheduling decision method is provided, the method comprising: Obtain the wave status of the delivery objects corresponding to the current scheduling round, and determine the order sequence in the current scheduling round that needs to be scheduled. The decided action sequence is initialized as an empty sequence. Combining the wave status of the delivery object and the decided action sequence, the cumulative return of each order in the order sequence in executing the assignment action and the order-pressing action is predicted in turn. Based on the prediction results, the decided action sequence is continuously updated until each order in the order sequence is traversed to obtain the target decided action sequence. Specifically, for the first order in the order sequence, the decided action sequence with an empty sequence is used to predict the cumulative return of the first order. For non-first orders in the order sequence, the decided action sequence updated based on the prediction results of the preceding orders adjacent to the non-first orders is used to predict the cumulative return of the non-first orders. Referring to the target's already decided action sequence, the orders in the order sequence are assigned or pushed back to complete the scheduling decision for the orders in the order sequence.
[0007] According to a second aspect of this application, an order scheduling decision-making apparatus is provided, the apparatus comprising: The acquisition module is used to acquire the wave status of the delivery objects corresponding to the current scheduling round, and to determine the order sequence to be scheduled in the current scheduling round. The prediction module is used to initialize the decided action sequence as an empty sequence, combine the wave status of the delivery object and the decided action sequence, and sequentially predict the cumulative return of each order in the order sequence in terms of executing the assignment action and the order-pressing action. Based on the prediction results, the decided action sequence is continuously updated until each order in the order sequence is traversed to obtain the target decided action sequence. Specifically, for the first order in the order sequence, the decided action sequence with an empty sequence is used to predict the cumulative return of the first order. For non-first orders in the order sequence, the decided action sequence updated based on the prediction results of the preceding orders adjacent to the non-first orders is used to predict the cumulative return of the non-first orders. The scheduling module is used to assign or suppress orders in the order sequence by referring to the target's already decided action sequence, so as to complete the scheduling decision for the orders in the order sequence.
[0008] According to a third aspect of this application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.
[0009] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the first aspects above.
[0010] By utilizing the above technical solutions, this application provides an order scheduling decision-making method, apparatus, computer equipment, and readable storage medium. This application combines the wave status of delivery objects with a continuously updated sequence of decided actions, and sequentially predicts the cumulative return of assigning or holding back orders for each order in the order sequence. This allows the decision-making of subsequent orders to fully rely on the decision-making results of previous orders, thereby transforming the originally independent order-level decisions into a sequential decision-making process with temporal dependencies at the wave dimension. This solves the problem in traditional solutions where the decisions to hold back orders are independent and cannot coordinate the interaction between orders within a wave. The decision-making model can comprehensively consider the collaborative and competitive relationships between orders at the wave level, optimizing the overall delivery efficiency from a global perspective. Moreover, since the decision-making process is always dynamically updated based on the sequence of decided actions, it can avoid the limitation of local optima that can easily occur when optimizing a single order independently, thus improving the intelligence of order scheduling decisions and the overall scheduling efficiency.
[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0012] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This paper illustrates a flowchart of an order scheduling decision-making method provided in an embodiment of this application. Figure 2 This illustration shows a schematic diagram of a prediction model processing flow provided in an embodiment of this application; Figure 3 This illustration shows a structural schematic diagram of an order scheduling decision-making device provided in an embodiment of this application; Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation
[0013] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.
[0014] This application provides an order scheduling decision-making method, such as... Figure 1 As shown, the method includes: S10: Obtain the wave status of the delivery object corresponding to the current scheduling round, and determine the order sequence to be scheduled in the current scheduling round.
[0015] The technical solution in this application embodiment can be applied to an order scheduling system, which can be used in the field of instant delivery, serving scenarios such as food delivery platforms, fresh food e-commerce, and same-city express delivery that require dynamic allocation of delivery capacity. The order scheduling system operates on a rider wave basis, transforming each round of scheduling decisions into an autoregressive processing procedure for the order sequence. It can perceive rider status, wave load, environmental changes, and route planning characteristics in real time, predicting the long-term benefits of assigning or holding back orders for each newly assigned order, and dynamically adjusting subsequent strategies based on previous decisions, ultimately outputting a globally coordinated joint decision result. Through this approach, the order scheduling system can achieve collaborative optimization among orders within the same wave, effectively balancing rider capacity and order timeliness during peak hours, in complex weather conditions, or when merchants are experiencing order cancellations, thereby improving overall delivery efficiency and scheduling intelligence.
[0016] In this embodiment, in each scheduling round, the order scheduling system first constructs the wave state for each individual delivery object. The delivery object refers to the rider in the instant delivery scenario. The wave state is used to characterize the global context information of the delivery task undertaken by the rider at the current moment. Specifically, it can include the rider's own delivery route, location, speed, and other rider status; the total number of orders in the current wave, the number of scheduling rounds, the number of meals picked up, and other wave status; as well as environmental status such as day of the week, time of day, and weather. The order scheduling system simultaneously obtains the set of newly assigned orders to be decided in this round and sorts each order according to the order of pickup points output by the route planning, forming an order sequence with a clear order of priority. This ensures that the subsequent decision-making process remains consistent with the actual physical order of meal pickup by the rider, laying the foundation for autoregressive decision-making.
[0017] In this way, through the above process, the originally scattered order information and rider status information are integrated into a structured wave status, and the orders are sequentially organized according to the physical execution order. This allows subsequent decisions to be made in a complete and orderly context, avoiding the fragmentation of decision information caused by ignoring the spatial relationship between orders within a wave in traditional solutions. For example, during the lunch rush, the order scheduling system obtains rider A's current wave status, including that he has 3 orders remaining, his current location is a certain business district, and there are 2 orders left uncollected in the current wave. At the same time, it obtains 3 new orders and arranges them in order of proximity of the pickup point as order A, order B, and order C, forming an order sequence.
[0018] S20: Initialize the decided action sequence as an empty sequence. Combining the wave status of the delivery object and the decided action sequence, predict the cumulative return of each order in the order sequence in terms of executing the assigned action and the order-pressing action. Based on the prediction results, continuously update the decided action sequence until each order in the order sequence is traversed to obtain the target decided action sequence.
[0019] In this embodiment, after obtaining the order sequence, the order scheduling system initializes an empty sequence of decided actions to record the decisions made in the current round. Subsequently, the order scheduling system performs cumulative return prediction for each order sequentially according to the order sequence. When making predictions, for the first order in the order sequence, since there are no decision records yet, the order scheduling system directly combines the wave state of the delivery object with the empty sequence to predict the cumulative return for performing the assignment action and the order-holding action for that order, respectively. For orders that are not the first order after the first order, the order scheduling system uses the sequence of decided actions, which has been updated based on the prediction results of previous orders, combined with the wave state of the delivery object, to predict the cumulative return for that order. Cumulative return refers to the estimated long-term benefit that can be obtained after taking a certain action, given the current state and some decisions already made. In practical applications, this estimate can be output through a prediction model. For each order prediction completed, the order scheduling system selects the action with the highest cumulative return as the decision result for that order based on the prediction results, and adds the action to the sequence of decided actions for use as the prediction input for subsequent orders. This process is repeated until all orders in the order sequence have completed their decisions, ultimately resulting in the target sequence of decided actions containing the decision results of all orders.
[0020] In this way, through the above process, the sequence of decisions is continuously updated, so that the decision of each order can be aware of the decision results of the preceding orders in the same wave. This transforms the independent order-level decisions in the traditional solution into a sequential decision process with temporal dependencies, and can take into account the collaborative and competitive relationships between orders at the wave level, providing a decision basis for global optimization. Continuing with the example above, the order scheduling system initializes an empty sequence as the decided action sequence. First, it predicts order A. Under the condition of an empty sequence, the cumulative return of the assigned action is 8.5, and the order-holding action is 6.2. It selects assignment, and the decided action sequence is updated to "Assign". Next, it predicts order B. Under the condition of the decided sequence being "Assign", the cumulative return of the assigned action is 7.8, and the order-holding action is 9.1. It selects order-holding, and the decided action sequence is updated to "Assign, Hold". Finally, it predicts order C. Under the condition of the decided sequence being "Assign, Hold", the cumulative return of the assigned action is 9.3, and the order-holding action is 5.4. It selects assignment, and finally obtains the target decided action sequence "Assign, Hold, Assign".
[0021] S30: Referring to the target's already decided action sequence, assign or process orders in the order sequence to complete the scheduling decision for orders in the order sequence.
[0022] In this embodiment, after obtaining the target decided action sequence containing all order decision results, the order scheduling system uses this sequence as the joint decision result of the current scheduling round and executes it one-to-one with the orders in the order sequence. Specifically, for orders marked as assignment actions in the sequence, the order scheduling system immediately dispatches the order to the corresponding delivery object, allowing it to enter the food pickup and delivery process; for orders marked as pending actions in the sequence, the order scheduling system temporarily holds the order in the pending order pool, not dispatching it immediately, waiting for subsequent scheduling rounds to merge it with other orders before dispatching it. After execution, the order scheduling system completes the scheduling decision for all orders in the order sequence for the current scheduling round. Through this process, the serialized decision results are directly converted into executable scheduling instructions, ensuring consistency between decision and execution. Furthermore, since the target decided action sequence is generated based on the wave global context and the dependencies between orders, its execution results can better coordinate the allocation of time and transportation resources among orders within the same wave, thereby improving the overall scheduling effect.
[0023] Continuing with the previous example, the order scheduling system refers to the target's already decided action sequence [assignment, order consolidation, assignment], and immediately assigns orders A and C to rider A so that he can start picking up and delivering the food; it consolidates order B and does not assign it for the time being, waiting for it to be merged with more orders along the same route in subsequent rounds, so as to optimize the overall delivery route and efficiency.
[0024] Optionally, in this embodiment, obtaining the wave status of the delivery object corresponding to the current scheduling round and determining the order sequence to be scheduled in the current scheduling round includes: obtaining the delivery object status, wave status, environmental status, and path planning features of the delivery object corresponding to the current scheduling round; concatenating the delivery object status, wave status, environmental status, and path planning features to obtain the delivery object wave status; obtaining the set of orders to be scheduled in the current scheduling round; extracting the order of pickup points for each order in the order set from the path planning features; and sorting all orders in the order set according to the order of pickup points to obtain the order sequence.
[0025] In this embodiment, the order scheduling system models the entire scheduling process as a Markov decision process. Specifically, each rider wave is considered a complete episode, and each episode contains... There are 10 decision steps, each corresponding to a scheduling round, meaning that a decision is made once in each scheduling round. This episode can be represented by the expression shown in Formula 1 below: Formula 1:
[0026] in, Indicates the first The status of each scheduling round Indicates the first Actions to be taken in each scheduling round Indicates the execution of an action The immediate reward obtained afterward. Within this MDP (Markov Decision Process) framework, rider waves are used as agents. The agents of each rider wave within each scheduling round are independent of each other; that is, the order allocation decision within one rider wave does not affect the decision-making process of another rider wave. In each scheduling round, the order scheduling system observes new orders and the current state, outputting the order allocation decision for this round—that is, deciding whether to allocate an order for each new order. This decision strategy is shared across all waves and all rounds, meaning all riders use the same decision model at all scheduling times.
[0027] Based on this, the order dispatch system collects multi-dimensional information required for the current dispatch round from the dispatch module. This multi-dimensional information specifically includes the delivery object status of the delivery object corresponding to the current dispatch round, the wave status of the delivery object in the current wave, the environmental status, and route planning characteristics. Specifically, the delivery object status refers to the rider's real-time operational data, including remaining available orders, current location coordinates, and movement speed; the wave status describes the rider's accumulated tasks within the current wave, such as the total number of orders in the current wave, the number of dispatch rounds completed, and the number of orders already picked up; the environmental status reflects external factors, including the current day of the week, the time of day (morning peak, noon peak, off-peak), and weather conditions; and the route planning characteristics can be pre-calculated by the route planning module, covering the planned pickup and delivery order of all orders within the wave and the estimated delivery time for each order. The order dispatch system concatenates these characteristics into vectors to form a unified delivery object wave status vector. This vector, as input to the subsequent decision-making model, fully preserves the global context information of riders, waves, environment, and route planning, so as to integrate the originally scattered capacity data, task data, and environmental data into a structured state representation, providing an information basis for collaborative decision-making in the wave dimension.
[0028] When determining the order sequence to be scheduled, the order scheduling system obtains all orders newly assigned to the rider in the current scheduling round, forming a set of orders to be decided. Subsequently, the order scheduling system extracts the order information of the pickup points corresponding to each order from the aforementioned path planning features. This order is generated by the path planning module based on the geographical location of each order's pickup point, the rider's current location, and the pickup and delivery time sequence, reflecting the order in which the rider will visit the pickup points during the actual pickup process. The order scheduling system will arrange the orders in ascending order according to the pickup point order, forming an order sequence with a clear priority, so that the decision order of the order sequence is consistent with the physical order in which the rider actually picks up the food, providing a reasonable dependency structure for the subsequent autoregressive decision model. For example, during the lunch rush hour on a weekday, rider B has two orders left uncollected in the current wave. The order dispatch system observes that he has two remaining orders, is located near an office building, and is moving at a constant speed. The wave status record shows 2 orders, 3 rounds, and 0 collected meals. The environmental status shows Wednesday, lunch rush hour, and sunny. The route planning features provide the pickup and delivery order of existing orders within the current wave and the estimated pickup location for newly assigned orders. The order dispatch system concatenates the above information into a wave status vector. Simultaneously, three new orders are arranged in order of proximity to pickup location as Order X, Order Y, and Order Z, forming an order sequence.
[0029] Optionally, in this embodiment, the decided action sequence is initialized as an empty sequence. Combining the wave state of the delivery object and the decided action sequence, the cumulative reward for each order in the order sequence in executing the assigned action and the order-placing action is predicted sequentially. Based on the prediction results, the decided action sequence is continuously updated until each order in the order sequence is traversed to obtain the target decided action sequence. This includes: creating an empty sequence as the decided action sequence, which stores the actions to be executed indicated by the prediction results in the current round; extracting the first order in the order sequence; inputting the wave state of the delivery object and the decided action sequence into a pre-trained prediction model to call the prediction model's state encoding layer and action embedding layer to generate an input sequence; and processing the input sequence through the position encoding and causal self-attention mechanism of the prediction model's sequence modeling layer to obtain... The process begins by extracting the hidden state sequence and then extracting the hidden state corresponding to the last action in the decided action sequence. This hidden state is then fed into the prediction layer of the prediction model to predict the cumulative returns of assigned and delayed actions, yielding prediction results. An action selection layer based on the prediction model is then used to determine the target action, which can be either an assigned action or a delayed action. The target action is added to the end of the decided action sequence, completing the update. The process continues by extracting subsequent orders adjacent to the first order from the order sequence, combining the wave state of the delivery object with the updated decided action sequence, predicting the cumulative returns of subsequent orders in executing assigned and delayed actions, and continuing to update the updated decided action sequence until every order in the order sequence has been traversed. The final updated decided action sequence is then taken as the target decided action sequence.
[0030] In this embodiment, the order scheduling system first creates an empty sequence as a sequence of decided actions. This sequence records the actions that have been determined to be executed in the current scheduling round, i.e., assignment or order posting, in the order of decision-making, thereby providing a dependency context for subsequent order decisions. In this embodiment, the action is defined as the joint decision result for all new orders in the current scheduling round. Specifically, let the... During the round-robin scheduling cycle, the order scheduling system observed a new number of orders. Then the action Belongs to the joint action space That is, a length of The system uses a binary vector, where each element corresponds to the decision result of an order. 0 indicates that the order will be assigned immediately to a rider; 1 indicates that the order will be held back and will be merged with other orders in a subsequent scheduling round. This representation of the joint action space allows the system to output the decision results of all new orders simultaneously in a single scheduling round, providing a foundation for subsequent autoregressive decomposition and sequential decision-making.
[0031] For the first order in the order sequence, the order scheduling system treats it as the current processing target and inputs the wave state of the delivery target and the current sequence of decided actions into a pre-trained prediction model. The prediction model internally includes a state encoding layer, an action embedding layer, a sequence modeling layer, a prediction layer, and an action selection layer: The state encoding layer maps the wave state of the delivery target to a high-dimensional state embedding vector; the action embedding layer converts each action in the decided action sequence into a continuous action embedding vector; the state embedding vector and each action embedding vector are concatenated sequentially and input into the sequence modeling layer, which generates a hidden state sequence through position encoding and causal self-attention mechanisms. Causal self-attention ensures that the output at each position depends only on the previous state and action, thus satisfying the autoregressive modeling requirements; subsequently, the hidden state corresponding to the last action in the decided action sequence is extracted from the hidden state sequence. For the first order, this means extracting the initial hidden state corresponding to the empty sequence, and this is input into the prediction layer to obtain the cumulative reward prediction results for executing the assignment action and the order placement action for the current order; the action selection layer selects the action with the higher cumulative reward as the target action based on the prediction results. After making a decision for the first order, the order scheduling system adds the target action to the end of the sequence of decided actions, completing an update.
[0032] Next, the order scheduling system continues to extract subsequent orders adjacent to the first order from the order sequence. It then inputs the wave status of the delivery object and the updated sequence of decided actions back into the prediction model, repeating the above processes of encoding, sequence modeling, prediction, and action selection. At this point, the sequence of decided actions already contains the decision results of previous orders. Therefore, during sequence modeling, the causal self-attention mechanism ensures that the prediction of the current order can fully utilize the decision information of previous orders, achieving the propagation of decision dependencies. After each order decision is completed, the order scheduling system appends the selected target action to the sequence of decided actions until all orders in the sequence have been processed. Ultimately, this sequence becomes the target sequence of decided actions containing the decision results of all orders, ready for subsequent execution.
[0033] For example, in rider C's wave, the order sequence is [Order X, Order Y, Order Z]. The order scheduling system first initializes an empty sequence, inputs the state and the empty sequence into the model, and finds that the cumulative reward of assigning order X is higher than that of holding back orders, so the order is selected and the sequence is updated to [Assignment]. Next, order Y is processed, and the state and [Assignment] are input into the model. At this time, the model already contains information about the previous assignment action when generating the hidden state. The prediction result shows that the cumulative reward of holding back orders is higher, so the order is selected and the sequence is updated to [Assignment, Holding Back Orders]. Finally, order Z is processed, and the state and [Assignment, Holding Back Orders] are input into the model. It is found that the cumulative reward of assigning orders is higher, so the order is selected and the final decision action sequence is [Assignment, Holding Back Orders, Assignment].
[0034] Optionally, in this embodiment, the method further includes: acquiring a training sample set, wherein each sample in the training sample set includes the historical wave state of the delivery object in the historical scheduling rounds, the historical action sequence, the immediate reward, and the next historical wave state of the delivery object; constructing an untrained prediction model, inputting the historical wave state of the delivery object into the state encoding layer of the prediction model to obtain a sample state embedding vector, inputting each sample action in the historical action sequence into the action embedding layer of the prediction model to obtain each sample action embedding vector, concatenating the sample state embedding vector and each sample action embedding vector in chronological order and inputting them into the sequence modeling layer of the prediction model, so that the sequence modeling layer outputs a sample hidden state sequence through position encoding and causal self-attention mechanism; and matching the sample hidden state sequence with the historical... The hidden states corresponding to the action positions of each sample in the action sequence are input into the prediction layer of the prediction model to obtain the conditional prediction values of each sample action position. A total loss function is constructed based on temporal difference error loss and conservative regularization loss. The temporal difference error loss is calculated based on the conditional prediction value of the last sample action in the historical action sequence, the immediate reward, and the maximum conditional prediction value of the next historical delivery object wave state. The conservative regularization loss is used to constrain the prediction model to estimate the prediction values of out-of-distribution actions. The total loss function is used to calculate the loss value of the prediction model during the training process, and the model parameters of the prediction model during the training process are updated according to the calculation results until the convergence condition is reached, thus obtaining the pre-trained prediction model.
[0035] In this embodiment, the training process of the prediction model is implemented through offline reinforcement learning. Its core lies in constructing a joint loss function that includes temporal difference error and conservative regularization to ensure the model's decision-making accuracy in the wave dimension and its ability to suppress out-of-distribution actions. Specifically, the order scheduling system first acquires a training sample set, where each sample comes from historical scheduling data and contains four key components: the wave state of the historical delivery object in the historical scheduling round, the actual historical action sequence executed in that round, the immediate reward obtained after executing the action sequence, and the next historical delivery object wave state reached after the state transition. The historical action sequence is a sequence of several binary actions arranged in decision order, and its length is equal to the number of orders to be decided in that round; 0 indicates assignment, and 1 indicates order confinement. The immediate reward is designed based on the effect before and after the state transition, and its calculation formula is shown in Formula 2 below: Formula 2:
[0036] in, This indicates the decrease in average waiting time compared to the previous wave. This represents the decrease in average actual delivery time compared to the previous wave. This is the meal card risk factor, which is positively correlated with the merchant's meal card risk. When the merchant's meal card risk is high... A larger value makes the model focus more on optimizing the waiting time for meals, thereby avoiding excessive waiting caused by order backlog.
[0037] In constructing the prediction model, the order scheduling system employs a network structure comprising a state encoding layer, an action embedding layer, a sequence modeling layer, a prediction layer, and an action selection layer. During training, the system inputs the historical wave states of delivery objects into the state encoding layer to obtain sample state embedding vectors; it then inputs each sample action from the historical action sequence into the action embedding layer to obtain sample action embedding vectors. Subsequently, the sample state embedding vectors and the sample action embedding vectors are concatenated chronologically to form a complete input sequence, which is then fed into the sequence modeling layer. The sequence modeling layer adds temporal information to each position through location encoding and processes the input sequence using a causal self-attention mechanism to ensure that the output of each position depends only on the state and actions preceding that position, thus outputting a sample hidden state sequence. The order scheduling system extracts the sample hidden states corresponding to each action position in the historical action sequence from the sample hidden state sequence and inputs them into the prediction layer to obtain the sample conditional prediction values for each action position, i.e., the cumulative reward estimate for the assigned action and the order placement action under the conditions of that action position. The core of training lies in constructing the total loss function, which is a weighted sum of the temporal difference error loss and the conservative regularization loss, as shown in Equation 3 below: Formula 3:
[0038] in, Represents the total loss function; The temporal difference error loss is calculated based on the sample condition prediction value corresponding to the last action in the historical action sequence, the immediate reward, and the maximum sample condition prediction value under the next historical delivery object wave state. To conserve regularization loss, its purpose is to encourage the actual actions collected in the samples. The value (i.e., the predicted value) is higher than that of other actions that were not sampled. This value helps to suppress the model's overestimation of out-of-distribution actions and prevents the policy from drifting into an unreliable action space outside the data distribution during the learning process. This is the regularization coefficient, used to balance the weights between the time-series difference error loss and the conservative regularization loss.
[0039] Furthermore, in formula 3 above The specific form is shown in Formula 4 below: Formula 4:
[0040] in, Indicates state, Represents the sample distribution. Indicates behavioral strategies, Indicates the current state Take action below of Value estimation, For the Bellman better operator to act on the current The target value after the value function is used to drive the model to learn accurate temporal difference relationships.
[0041] Furthermore, in formula 3 above The specific form is shown in Formula 5 below: Formula 5:
[0042] Similarly, as in formula 4, Indicates state, Represents the sample distribution. Indicates behavioral strategies, Indicates the current state Take action below of Value estimation. Let be the expectation operator, representing the expectation of the joint distribution of state ss and action aa. Where, Representing state From offline datasets Mid-sampling, It consists of samples collected from historical scheduling rounds; Indicates action From behavioral strategies Mid-sampling, The strategy used to generate historical data typically corresponds to the actual decision-making strategy of the historical scheduling system. In Formula 5, 0 represents the objective value, and the conservative regularization loss is achieved by... The value estimate approaches 0, suppressing the model's estimation of all actions. Value estimation, thereby indirectly affecting the actual sampling actions in the dataset. The value is relatively higher than that of unsampled actions, thus achieving the effect of suppressing overestimation of out-of-distribution actions.
[0043] In addition, in this embodiment, The update rules for the function are shown in Formula 6 below: Formula 6:
[0044] in, Indicates the first There are 1 scheduling rounds, and the state is... At that time, it had already been decided that... The decision to hold an order, each decision 0 indicates assignment, 1 indicates order placement, currently targeting the first... Action taken for each order Conditional Marginal Value. When That is, before all orders in this round have been processed, the Value updated to , indicating that in front Based on the established decision, the second decision... The corresponding optimal action for each order Values, thereby enabling the backward propagation of dependencies; when That is, the last order, Value updated to ,in To execute the complete action sequence The instant reward obtained afterward As a discount factor, For the next state Next, the better action Value estimated, this part consists only of the last order. The value is then linked to the next stage of value estimation. Through this recursive update method, the originally exponential joint action space is decomposed into... This approach employs a binary classification system while preserving the temporal dependencies between different order decisions.
[0045] Optionally, in this embodiment, the wave state of the delivery object and the sequence of decided actions are input into a pre-trained prediction model to generate an input sequence by calling the state encoding layer and the action embedding layer of the prediction model. This includes: calling the state encoding layer to map the wave state of the delivery object into a state embedding vector, and calling the action embedding layer to map each action in the sequence of decided actions into an action embedding vector; and concatenating the state embedding vector and the action embedding vector corresponding to each action in sequence to form an input sequence.
[0046] In this embodiment, during the online inference phase, the order scheduling system will update the current wave status. The decision-making action sequence, along with the inputs, is fed into the trained prediction model. Specifically, the state encoding layer within the prediction model first processes the wave states... The mapping is done as a high-dimensional state embedding vector; simultaneously, the action embedding layer converts each action in the decided action sequence into a continuous action embedding vector, where 0 represents assignment and 1 represents order placement. Subsequently, the order scheduling system concatenates the state embedding vector with each action embedding vector in the decision order to form a complete input sequence, which serves as the input for subsequent sequence modeling layers. In this way, the input sequence contains both global information of the current wave and the decision results of previous orders, laying the foundation for autoregressive prediction.
[0047] Optionally, in this embodiment, the input sequence is processed by positional encoding and causal self-attention mechanism of the sequence modeling layer of the prediction model to obtain the hidden state sequence, including: adding positional encoding to each element in the input sequence to obtain an encoded input sequence; processing the encoded input sequence with reference to the causal self-attention mechanism to obtain a processed input sequence, wherein the causal self-attention mechanism makes the output at a specified position in the encoded input sequence depend on the elements between the first position and the specified position in the encoded input sequence; and inputting the processed input sequence into the feedforward neural network layer of the prediction model to obtain the hidden state sequence.
[0048] In this embodiment, after the input sequence is assembled, the sequence modeling layer first adds a positional code to each element in the input sequence. The positional code is used to assign the element at each position in the sequence its order information in the sequence, so that the model can distinguish the input at different positions. This is because the self-attention mechanism itself does not have the ability to perceive the order of elements, while the order of order decisions directly determines the physical execution path. Therefore, the introduction of positional code enables the model to accurately identify which order is being processed and the historical position of the previous decision.
[0049] Subsequently, the sequence modeling layer processes the encoded input sequence using a causal self-attention mechanism. Causal self-attention is a special type of attention masking mechanism. Its core constraint is that when processing a sequence, the output at a specified position can only depend on the elements in the encoded input sequence from the first position to that specified position, and cannot "see" any elements after that position. Mathematically, this mechanism is implemented through a masking of the attention matrix, that is, resetting the attention weights of future positions to zero, thereby ensuring that the model can predict the... When making decisions for an order, only status information and previous data can be used. The decision-making results for a single order are determined without obtaining any information about subsequent orders, in order to follow the autoregressive generative logic and ensure that the temporal causal relationship of the decision-making process is not disrupted.
[0050] After processing by the causal self-attention mechanism, the sequence modeling layer further inputs the intermediate representations into the feedforward neural network layer. This feedforward neural network layer, composed of several fully connected layers, further extracts and fuses the features output by the attention mechanism through nonlinear transformations, enhancing the model's expressive power. Finally, the sequence modeling layer outputs a complete hidden state sequence. The hidden state at each position in the hidden state sequence corresponds to the encoding result at the corresponding position in the input sequence. The hidden state corresponding to the last action in the decided action sequence will be used by the subsequent prediction layer to generate a predicted value estimate for the current order.
[0051] For example, in rider A's wave, the order sequence is [Order X, Order Y, Order Z], and the decided action sequence is [Assign, Place Order], meaning order Z is being processed. The input sequence is formed by concatenating the state embedding vector, the action embedding vector for order X, and the action embedding vector for order Y in sequence. The sequence modeling layer first adds positional encoding to these three elements, clarifying which corresponds to the state, the first decision, and the second decision. Then, during processing, the causal self-attention mechanism, for the position representing order Z (the third position), only allows the fusion of the state, the decision for order X, and the decision for order Y in its output, without acquiring any future information. The processed result is further transformed by a feedforward neural network layer, ultimately outputting a hidden state sequence. Extracting the hidden state at the third position from this sequence allows it to be used for order Z. Value (i.e. predicted value) prediction.
[0052] Optionally, in this embodiment, the hidden state is input into the prediction layer of the prediction model to predict the cumulative returns of the assigned action and the order placement action, and a prediction result is obtained. Then, an action selection layer based on the prediction model is used to determine the target action based on the prediction result. This includes: inputting the hidden state into the prediction layer, calling the prediction layer to generate a first predicted value and a second predicted value, and using the first predicted value and the second predicted value as the prediction result. The first predicted value is used to characterize the predicted cumulative return of performing the assigned action on the first order, and the second predicted value is used to characterize the predicted cumulative return of performing the order placement action on the first order. The action selection layer compares the numerical values of the first predicted value and the second predicted value, and selects the target action from the assigned action and the order placement action based on the comparison result.
[0053] In this embodiment, after the sequence modeling layer outputs the hidden state sequence, the order scheduling system inputs the hidden state corresponding to the current order position into the prediction layer. The prediction layer consists of a fully connected network with an output dimension of 2, corresponding to the predicted cumulative rewards for performing the assignment action and the order-placing action for the order in the current state, respectively. The prediction layer generates a first prediction value and a second prediction value. The first prediction value represents the expected cumulative reward for performing the assignment action, and the second prediction value represents the expected cumulative reward for performing the order-placing action. Together, they constitute the prediction result for the current order.
[0054] After receiving the prediction result, the action selection layer performs a numerical comparison operation and selects the target action from the assigned action and the order-placing action based on the comparison result. This ensures that at each decision point, the order scheduling system can select the action with the highest expected cumulative return under the current conditions, effectively improving the coordination and intelligence of scheduling decisions.
[0055] In this embodiment of the application, optionally, the target action is selected from the assigned action and the order-placing action based on the obtained comparison result, including: if the first predicted value is greater than the second predicted value, the assigned action is selected as the target action; if the second predicted value is greater than or equal to the first predicted value, the order-placing action is selected as the target action.
[0056] In this embodiment, as described above, the first predicted value represents the expected cumulative return of assigning the action to the current order under the current state and the preceding actions already decided; the second predicted value corresponds to the expected cumulative return of executing the order-holding action. The action selection layer selects the action with the higher cumulative return as the target action through numerical comparison. Specifically, if the first predicted value is greater than the second predicted value, it indicates that assigning the action can bring higher long-term benefits under the current situation, so the assigned action is selected; if the second predicted value is greater than or equal to the first predicted value, it indicates that the long-term benefits of the order-holding action are not lower than those of the assigned action, so the order-holding action is selected to ensure that the better action under the current information is selected at each decision point.
[0057] By combining this selection mechanism with an autoregressive decision-making framework, sequential decision optimization at the wave dimension can be achieved. Since the predicted value of each order is generated based on the preceding actions, the comparison results of the action selection layer incorporate the dependencies between orders, enabling subsequent decisions to respond to the results of preceding decisions. For example, in rider B's wave, the order sequence is [Order X, Order Y, Order Z]. When processing order X, the prediction layer outputs a first predicted value of 8.3 (assignment) and a second predicted value of 6.1 (order suppression). After comparison, the action selection layer selects assignment because the assignment value is higher. When processing order Y, the already decided sequence is [assignment]. The prediction layer outputs a first predicted value of 7.2 and a second predicted value of 8.9. After comparison, order suppression is selected because the order suppression value is higher. When processing order Z, the already decided sequence is [assignment, order suppression]. The prediction layer outputs a first predicted value of 9.1 and a second predicted value of 5.4. After comparison, assignment is selected. Through this successive comparison, the joint decision [assignment, order suppression, assignment] is finally obtained.
[0058] In this embodiment of the application, optionally, the orders in the order sequence are assigned or suppressed with reference to the target decided action sequence, including: establishing a correspondence between the orders in the order sequence and the actions in the target decided action sequence according to the order order in the order sequence and the order order of the actions in the target decided action sequence; for at least one first order in the order sequence whose corresponding action value is 0, assigning at least one first order to the delivery object of the currently assignable order; for at least one second order in the order sequence whose corresponding action value is 1, suppressing at least one second order, controlling at least one second order to enter the suppression pool to wait for subsequent rounds of merging.
[0059] In this embodiment, the order scheduling system first establishes a one-to-one correspondence between the order sequence and the action sequence in the target decided action sequence. Since the order sequence is ordered according to the order of pick-up points in the path planning, and the target decided action sequence is generated autoregressively in the same order, their orders are naturally aligned. The order scheduling system only needs to associate each order with its corresponding action value (0 or 1) by index to ensure that the decision results can be accurately mapped to specific orders in the subsequent execution stage.
[0060] After the correspondence is established, the order scheduling system categorizes the orders in the order sequence. For at least one first order with an action value of 0, the system classifies these orders as "assigned" orders and immediately assigns them to the currently assignable delivery recipient, i.e., the rider. This assignment means the system adds the order to the rider's current wave task list, allowing the rider to pick up and deliver the food according to the pickup / delivery order output by the route planning module, thus entering the actual delivery process. For at least one second order with an action value of 1, the system classifies these orders as "pending" orders and performs a pending order operation. This means these orders are not immediately dispatched but instead placed in a pending order pool to await subsequent scheduling rounds. The pending order pool is a buffer area used by the system to temporarily store undispatched orders. In subsequent rounds, the system combines newly assigned orders with orders in the pending order pool to make merging decisions, seeking optimal route integration opportunities. Through the above process, not only can the precise mapping between decision results and execution actions be achieved, but the single-pool mechanism also retains flexibility for subsequent merging and optimization. This ensures the accurate execution of decisions in the current round while providing a foundation for global collaboration across rounds. The execution process fully inherits the joint action sequence of the autoregressive decision output, enabling wave-dimensional collaborative decisions to be translated into executable scheduling instructions, effectively improving the overall coordination and resource utilization efficiency of order scheduling.
[0061] For example, in rider C's wave, the order sequence is [Order X, Order Y, Order Z], and the corresponding target decision action sequence is [Assign, Deferred, Assign]. After establishing the correspondence, the order scheduling system classifies Order X and Order Z as the first order and Order Y as the second order. Subsequently, the order scheduling system immediately assigns Order X and Order Z to rider C, who can then begin picking up and delivering the food according to the planned pickup and delivery order. At the same time, the order scheduling system deferred Order Y, placing it in the deferred order pool, waiting to be merged with possible new orders in the next scheduling round before unified dispatch.
[0062] In practical applications, the prediction model can be a Q-transformer autoregressive model. The processing flow for this model can be found in [link to relevant documentation]. Figure 2 At any time step Within a single decision, the Q-Transformer autoregressive model first receives the input, which is the state. and action token sequence Subsequently, the state is mapped to a state embedding vector through a state embedding layer, and each action token is converted into a continuous action embedding vector through an action embedding layer. The action embedding vectors are concatenated sequentially and then fed into the Transformer module, which is the sequence modeling layer in this embodiment. The Transformer module combines positional encoding with a self-attention mechanism with causal masking to ensure that the first... Only the status can be seen at each action position. and before One action to Output the hidden state sequence ,in Corresponding state, Corresponding to the Each action position. Next, The value head module, also known as the prediction layer in this application embodiment, utilizes the conditions generated by each hidden state. Value, of which, From , From And so on, until... From .final, The value module outputs each position. The logits are used by the action selector and output as a combined value. value Used for calculating temporal difference loss (TD loss); at the same time, the previously predicted action dimension is fed back to the next time step to achieve autoregressive generation.
[0063] In summary, the technical solution of this application elevates the decision-making entity from the order dimension to the rider wave level, constructing a wave-level multi-agent collaborative decision-making framework. This effectively models the spatiotemporal coupling relationship between orders in the same wave, solving the problem of traditional frameworks neglecting the dynamic interaction effect of the group, and achieving a leap from local optimization to global optimization. Simultaneously, an autoregressive Q-Transformer model is used to decompose the exponential joint action space (2^N) into N binary classification problems, effectively avoiding the explosion of action space dimensions. Orders are assigned sequentially according to the path planning point selection order, preserving the combined effect and path coupling between orders, and avoiding the loss of combined effects caused by independent scoring and subsequent thresholding. Furthermore, by designing a reward function based on state transition effects, effective feedback signals are provided in each round of order dispatch, solving the problem of sparsity in traditional reinforcement learning reward design and accelerating model training convergence. Finally, a conservative regularization mechanism is introduced to prevent overestimation of out-of-distribution actions, improving the stability of offline reinforcement learning and solving the policy drift problem that easily occurs in traditional offline reinforcement learning. The above-mentioned components collectively enable effective modeling of the spatiotemporal coupling relationship of orders in the same wave, allowing the order scheduling system to leap from local optimization to global optimization, significantly reducing the average actual delivery time per order, improving scheduling efficiency, and enhancing the stability and training efficiency of offline reinforcement learning.
[0064] The method provided in this application combines the wave status of delivery objects with a continuously updated sequence of decided actions, and sequentially predicts the cumulative return of assigning or holding back orders for each order in the order sequence. This allows the decision-making of subsequent orders to fully rely on the decision results of previous orders, thereby transforming the originally independent order-level decisions into a sequential decision-making process with temporal dependencies at the wave dimension. This solves the problem in traditional solutions where the decisions to hold back orders are independent and cannot coordinate the interaction between orders within a wave. The decision model can comprehensively consider the collaborative and competitive relationships between orders at the wave level, optimize the overall delivery efficiency from a global perspective, and since the decision-making process is always dynamically updated based on the sequence of decided actions, it can avoid the limitation of local optima that can easily occur when optimizing a single order independently, thus improving the intelligence of order scheduling decisions and the overall scheduling efficiency.
[0065] Furthermore, as Figure 1 To specifically implement the method, this application provides an order scheduling decision-making device, such as... Figure 3 As shown, the device includes: an acquisition module 301, a prediction module 302, and a scheduling module 303.
[0066] The acquisition module 301 is used to acquire the wave status of the delivery object corresponding to the current scheduling round, and to determine the order sequence to be scheduled in the current scheduling round. The prediction module 302 is used to initialize the decided action sequence as an empty sequence, combine the wave status of the delivery object and the decided action sequence, and sequentially predict the cumulative return of each order in the order sequence in terms of executing the assignment action and the order-pressing action. Based on the prediction results, the decided action sequence is continuously updated until each order in the order sequence is traversed to obtain the target decided action sequence. Specifically, for the first order in the order sequence, the decided action sequence with an empty sequence is used to predict the cumulative return of the first order. For non-first orders in the order sequence, the decided action sequence updated based on the prediction results of the preceding orders adjacent to the non-first orders is used to predict the cumulative return of the non-first orders. The scheduling module 303 is used to assign or suppress orders in the order sequence by referring to the target's already decided action sequence, so as to complete the scheduling decision for the orders in the order sequence.
[0067] In a specific application scenario, the acquisition module 301 is used to acquire the delivery object status, wave status, environmental status, and path planning features of the delivery object corresponding to the current scheduling round; concatenate the delivery object status, wave status, environmental status, and path planning features to obtain the delivery object wave status; acquire the order set of orders to be scheduled in the current scheduling round; extract the order of pickup points for each order in the order set from the path planning features; and sort all orders in the order set according to the order of pickup points to obtain the order sequence.
[0068] In a specific application scenario, the prediction module 302 is used to create an empty sequence as the decided action sequence, which stores the actions to be executed indicated by the prediction results in the current round; extract the first order at the top of the order sequence, input the wave state of the delivery object and the decided action sequence into a pre-trained prediction model, and call the state encoding layer and action embedding layer of the prediction model to generate an input sequence; process the input sequence through the position encoding and causal self-attention mechanism of the sequence modeling layer of the prediction model to obtain a hidden state sequence, and extract the hidden state corresponding to the last action of the decided action sequence from the hidden state sequence; input the hidden state into the prediction layer of the prediction model to assign actions. The prediction results are obtained by predicting the cumulative returns of the order placement action and the order placement action. Based on the prediction model, an action selection layer is formed to determine the target action, which is either the assigned action or the order placement action. The target action is added to the end of the decided action sequence to update the decided action sequence. The subsequent orders adjacent to the first order are extracted from the order sequence. Combining the wave status of the delivery object and the updated decided action sequence, the cumulative returns of the subsequent orders in executing the assigned action and the order placement action are predicted, and the updated decided action sequence is updated again until every order in the order sequence is traversed. The final updated decided action sequence is taken as the target decided action sequence.
[0069] In specific application scenarios, the device further includes: The training module is used to acquire a training sample set. Each sample in the training sample set includes the historical wave state of the delivery object in the historical scheduling rounds, the historical action sequence, the immediate reward, and the state of the next historical delivery object wave. An untrained prediction model is constructed by inputting the historical delivery object wave states into the state encoding layer of the prediction model to obtain sample state embedding vectors. Each sample action in the historical action sequence is input into the action embedding layer of the prediction model to obtain sample action embedding vectors. The sample state embedding vectors and each sample action embedding vector are concatenated in chronological order and then input into the sequence modeling layer of the prediction model. This allows the sequence modeling layer to output a sample hidden state sequence through positional encoding and causal self-attention mechanisms. The hidden state sequence is then compared with each action in the historical action sequence. The hidden states corresponding to the sample action positions are input into the prediction layer of the prediction model to obtain the sample condition prediction values for each sample action position. A total loss function is constructed based on temporal difference error loss and conservative regularization loss. The temporal difference error loss is calculated based on the sample condition prediction value corresponding to the last sample action in the historical action sequence, the instant reward, and the maximum sample condition prediction value under the next historical delivery object wave state. The conservative regularization loss is used to constrain the prediction model to estimate the prediction values of out-of-distribution actions. The total loss function is used to calculate the loss value of the prediction model during the training process, and the model parameters of the prediction model during the training process are updated according to the calculated results until the convergence condition is reached, thus obtaining the pre-trained prediction model.
[0070] In a specific application scenario, the prediction module 302 is used to call the state encoding layer to map the wave state of the delivery object into a state embedding vector, and call the action embedding layer to map each action in the decided action sequence into an action embedding vector; the state embedding vector and the action embedding vector corresponding to each action are concatenated in sequence to form the input sequence.
[0071] In a specific application scenario, the prediction module 302 is used to add position encoding to each element in the input sequence to obtain the encoded input sequence; to process the encoded input sequence with reference to the causal self-attention mechanism to obtain the processed input sequence, wherein the causal self-attention mechanism makes the output at a specified position in the encoded input sequence depend on the elements between the first position and the specified position in the encoded input sequence; and to input the processed input sequence into the feedforward neural network layer of the prediction model to obtain the hidden state sequence.
[0072] In a specific application scenario, the prediction module 302 is used to input the hidden state into the prediction layer, call the prediction layer to generate a first prediction value and a second prediction value, and use the first prediction value and the second prediction value as the prediction result. The first prediction value is used to represent the predicted cumulative return of performing the assigned action on the first order, and the second prediction value is used to represent the predicted cumulative return of performing the order-placing action on the first order. The first prediction value and the second prediction value are compared based on the action selection layer, and the target action is selected from the assigned action and the order-placing action according to the comparison result.
[0073] In a specific application scenario, the prediction module 302 is used to select the assigned action as the target action if the first predicted value is greater than the second predicted value; and to select the order placement action as the target action if the second predicted value is greater than or equal to the first predicted value.
[0074] In a specific application scenario, the scheduling module 303 is used to establish a correspondence between orders in the order sequence and actions in the target decided action sequence based on the order order in the order sequence and the order of actions in the target decided action sequence; for at least one first order in the order sequence whose corresponding action value is 0, the at least one first order is assigned to the delivery object of the currently assignable order; for at least one second order in the order sequence whose corresponding action value is 1, the at least one second order is put into a waiting pool to wait for subsequent rounds of merging.
[0075] The apparatus provided in this application combines the wave status of delivery objects with a continuously updated sequence of decided actions to sequentially predict the cumulative return of each order in the order sequence for assignment or order consolidation. This allows the decision-making of subsequent orders to fully rely on the decision results of previous orders, thereby transforming the originally independent order-level decisions into a sequential decision-making process with temporal dependencies at the wave dimension. This solves the problem in traditional solutions where the order consolidation decisions are independent and cannot coordinate the interaction between orders within a wave. The decision model can comprehensively consider the collaborative and competitive relationships between orders at the wave level, optimizing overall delivery efficiency from a global perspective. Moreover, since the decision-making process is always dynamically updated based on the sequence of decided actions, it can avoid the limitation of local optima that can easily occur when optimizing a single order independently, thus improving the intelligence of order scheduling decisions and overall scheduling efficiency.
[0076] It should be noted that other corresponding descriptions of the functional units involved in the order scheduling decision-making device provided in this application embodiment can be found by referring to... Figure 1 and Figure 2The corresponding descriptions in [the document] will not be repeated here.
[0077] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0078] The above embodiments and the technical features in the embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
[0080] In an exemplary embodiment, see Figure 4 The invention also provides a computer device including a bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device, wherein the various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the order scheduling decision method described in the above embodiments.
[0081] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the order scheduling decision method.
[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0083] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.
[0084] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.
[0085] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.
[0086] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.
Claims
1. An order dispatch decision method characterized by, include: Obtain the wave status of the delivery objects corresponding to the current scheduling round, and determine the order sequence in the current scheduling round that needs to be scheduled. The decided action sequence is initialized as an empty sequence. Combining the wave status of the delivery object and the decided action sequence, the cumulative return of each order in the order sequence in executing the assignment action and the order-pressing action is predicted in turn. Based on the prediction results, the decided action sequence is continuously updated until each order in the order sequence is traversed to obtain the target decided action sequence. Specifically, for the first order in the order sequence, the decided action sequence with an empty sequence is used to predict the cumulative return of the first order. For non-first orders in the order sequence, the decided action sequence updated based on the prediction results of the preceding orders adjacent to the non-first orders is used to predict the cumulative return of the non-first orders. Referring to the target's already decided action sequence, the orders in the order sequence are assigned or pushed back to complete the scheduling decision for the orders in the order sequence.
2. The method of claim 1, wherein, The steps of obtaining the wave status of the delivery objects corresponding to the current scheduling round and determining the order sequence to be scheduled in the current scheduling round include: Obtain the delivery object status, wave status, environmental status, and path planning characteristics of the delivery object corresponding to the current scheduling round; and concatenate the delivery object status, wave status, environmental status, and path planning characteristics to obtain the wave status of the delivery object. Obtain the set of orders to be scheduled in the current scheduling round, extract the order of pickup points for each order in the order set from the path planning features, and sort all orders in the order set according to the order of pickup points to obtain the order sequence.
3. The method of claim 1, wherein, The initialized decided action sequence is an empty sequence. Combining the wave status of the delivery object and the decided action sequence, the cumulative return for each order in the order sequence in executing the assigned action and the order-holding action is predicted sequentially. Based on the prediction results, the decided action sequence is continuously updated until every order in the order sequence is traversed to obtain the target decided action sequence, including: Create an empty sequence as the decided action sequence, which is used to store the action to be executed indicated by the prediction result in the current round; Extract the first order that is at the top of the order sequence, input the wave state of the delivery object and the decided action sequence into the pre-trained prediction model, and call the state encoding layer and action embedding layer of the prediction model to generate the input sequence; The input sequence is processed by the position encoding and causal self-attention mechanism of the sequence modeling layer of the prediction model to obtain the hidden state sequence, and the hidden state corresponding to the last action of the decided action sequence is extracted from the hidden state sequence. The hidden state is input into the prediction layer of the prediction model to predict the cumulative returns on the assigned action and the order placement action, and the prediction result is obtained. Based on the prediction model, the action selection layer determines the target action according to the prediction result. The target action is the assigned action or the order placement action. The target action is added to the end of the sequence of decided actions to complete the update of the sequence of decided actions; Continue extracting subsequent orders adjacent to the first order from the order sequence, combining the wave status of the delivery object and the updated decided action sequence, predicting the cumulative returns of the subsequent orders in executing assigned actions and order-pressing actions, and continuing to update the updated decided action sequence until every order in the order sequence has been traversed, and finally taking the updated decided action sequence as the target decided action sequence.
4. The method of claim 3, wherein, The method further includes: Obtain a training sample set, wherein each sample in the training sample set includes the historical wave state of the delivery object in the historical scheduling round, the historical action sequence, the immediate reward, and the next historical wave state of the delivery object; An untrained prediction model is constructed. The historical delivery object wave states are input into the state encoding layer of the prediction model to obtain sample state embedding vectors. Each sample action in the historical action sequence is input into the action embedding layer of the prediction model to obtain each sample action embedding vector. The sample state embedding vectors and each sample action embedding vector are concatenated in chronological order and then input into the sequence modeling layer of the prediction model so that the sequence modeling layer outputs the sample hidden state sequence through position encoding and causal self-attention mechanism. The hidden states of the samples in the hidden state sequence that correspond to the positions of each sample action in the historical action sequence are respectively input into the prediction layer of the prediction model to obtain the sample condition prediction values corresponding to each sample action position. A total loss function is constructed based on temporal difference error loss and conservative regularization loss. The temporal difference error loss is calculated based on the sample condition prediction value corresponding to the last sample action in the historical action sequence, the instant reward, and the maximum sample condition prediction value under the next historical delivery object wave state. The conservative regularization loss is used to constrain the prediction model to estimate the prediction value of out-of-distribution actions. The total loss function is used to calculate the loss value of the prediction model during the training process, and the model parameters of the prediction model during the training process are updated according to the calculation results until the convergence condition is reached, so as to obtain the pre-trained prediction model.
5. The method of claim 3, wherein, The step of inputting the wave state of the delivery object and the decided action sequence into a pre-trained prediction model, and then calling the state encoding layer and action embedding layer of the prediction model to generate the input sequence, includes: The state encoding layer is invoked to map the wave state of the delivery object to a state embedding vector, and the action embedding layer is invoked to map each action in the decided action sequence to an action embedding vector; The state embedding vector and the action embedding vector corresponding to each action are concatenated sequentially to form the input sequence.
6. The method of claim 3, wherein, The process of processing the input sequence through positional encoding and causal self-attention mechanism of the sequence modeling layer of the prediction model to obtain the hidden state sequence includes: Add a positional code to each element of the input sequence to obtain the encoded input sequence; The encoded input sequence is processed according to the causal self-attention mechanism to obtain the processed input sequence, wherein the causal self-attention mechanism makes the output at a specified position in the encoded input sequence depend on the elements between the first position and the specified position in the encoded input sequence; The processed input sequence is fed into the feedforward neural network layer of the prediction model to obtain the hidden state sequence.
7. The method of claim 3, wherein, The process of inputting the hidden state into the prediction layer of the prediction model to predict the cumulative reward on the assigned action and the single-action prediction, obtaining the prediction result, and the action selection layer based on the prediction model to determine the target action according to the prediction result, includes: The hidden state is input to the prediction layer, the prediction layer is called to generate a first prediction value and a second prediction value, and the first prediction value and the second prediction value are used as the prediction result, wherein the first prediction value is used to characterize the predicted cumulative return of performing the assignment action on the first order, and the second prediction value is used to characterize the predicted cumulative return of performing the order-pressing action on the first order. Based on the action selection layer, the numerical values of the first predicted value and the second predicted value are compared, and the target action is selected from the assigned action and the order placement action according to the comparison result.
8. An order scheduling decision apparatus characterized by comprising: include: The acquisition module is used to acquire the wave status of the delivery objects corresponding to the current scheduling round, and to determine the order sequence to be scheduled in the current scheduling round. The prediction module is used to initialize the decided action sequence as an empty sequence, combine the wave status of the delivery object and the decided action sequence, and sequentially predict the cumulative return of each order in the order sequence in terms of executing the assignment action and the order-pressing action. Based on the prediction results, the decided action sequence is continuously updated until each order in the order sequence is traversed to obtain the target decided action sequence. Specifically, for the first order in the order sequence, the decided action sequence with an empty sequence is used to predict the cumulative return of the first order. For non-first orders in the order sequence, the decided action sequence updated based on the prediction results of the preceding orders adjacent to the non-first orders is used to predict the cumulative return of the non-first orders. The scheduling module is used to assign or suppress orders in the order sequence by referring to the target's already decided action sequence, so as to complete the scheduling decision for the orders in the order sequence. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.
