Real-time cooperative scheduling system for instant delivery based on hierarchical reinforcement learning

Abstract

This invention discloses a real-time cooperative scheduling system for instant delivery based on hierarchical reinforcement learning. The system includes a simulator module, a decision module, and a time constraint module. The simulator module is used to simulate the routine processes of delivery personnel in instant delivery and to build a training and evaluation environment for the algorithm to allocate orders and schedule delivery personnel. The decision module: Based on a cooperative delivery scheduling model based on hierarchical reinforcement learning, it extracts features through an actor-critic network, makes delivery personnel cooperative delivery decisions based on encounter interactions, and feeds these decisions into the simulator loop. The time constraint module comprehensively considers the real-time remaining delivery time of an order, the number of historical order transfers, and the real-time scheduling requirements of instant delivery, and schedules and guides the decision-making solutions in the decision module. This system recommends delivery personnel to interact with each other through scheduling to achieve the overall goals of increasing the number of orders that arrive during the delivery process, improving delivery efficiency while meeting delivery time constraints, and reducing order timeout rates.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer intelligent computing and application, and mainly relates to an instant delivery real-time cooperative scheduling system based on hierarchical reinforcement learning. Background Art

[0002] In real-time delivery services, users place orders online through the delivery platform at home or work. Stores receive the orders online from the platform and begin preparing the goods. The platform then dispatches the orders to appropriate delivery drivers. Delivery drivers accept the orders, pick up the goods at the store, and then deliver them to the user. In urban areas, the platform dispatches a single delivery driver to handle multiple orders simultaneously, each subject to strict delivery time constraints. Delivery driver and order status constantly change during the delivery process, and the optimal delivery driver-order matching relationship also changes dynamically. This dynamic nature poses significant challenges to order scheduling in real-time delivery scenarios. Even when orders are grouped and assigned to delivery drivers based on factors such as similarity, the order groups a delivery driver is assigned to in real time may not always be on their route. When a delivery driver accumulates multiple orders, this can lead to inefficient and timed deliveries. Therefore, given the high order volume, limited delivery driver availability, and the dynamic changes in delivery driver and order status in urban areas, it is essential to design an effective scheduling strategy that dynamically adjusts the matching relationship between delivery drivers and orders during the delivery process, increases the number of on-route orders for delivery drivers, and improves platform delivery efficiency while meeting delivery time constraints.

[0003] Existing research on independent delivery scheduling by couriers ignores the cooperation between couriers and the dynamic changes in the delivery status of couriers and orders during the delivery process, and does not fully utilize the couriers' transportation capacity. When multiple orders accumulate, it is easy to cause inefficiency in delivery by couriers. Existing cooperative scheduling solutions are mainly based on the deployment of fixed transfer point infrastructure and the scenario where the delivery route is relatively fixed, to achieve order dispatch and transfer. Because (1) the deployment of fixed additional infrastructure requires a lot of cost; (2) the routes of instant delivery couriers change dynamically with the orders dispatched in real time, if they go to fixed transfer points, it will bring a long additional detour distance, which is difficult to meet the strict time constraints of instant delivery. Therefore, these solutions are not suitable for solving the cooperative delivery scheduling problem under the instant delivery model. Secondly, the existing prediction-driven real-time scheduling research uses historical data to predict relevant information and guide real-time scheduling decisions. Since the prediction-based method is difficult to consider future uncontrollable changes, it is not applicable. Existing detection-driven real-time scheduling related research is only applied to emergency scheduling scenarios that support independent delivery scheduling, and is not suitable for solving the real-time cooperative delivery scheduling problem under the instant delivery model.

[0004] In recent years, with the intelligent upgrade of real-time delivery systems, virtual beacon services have been embedded in delivery drivers' smartphones to support reliable encounter detection. The virtual beacon service allows smartphones to simultaneously broadcast and scan Bluetooth signals (including information such as device IDs), while also uploading the scanned information in real time. By matching scanned device IDs with delivery driver IDs, the platform can detect encounters between delivery drivers in real time. The large number of delivery driver encounters provides ample opportunities for dynamically adjusting the matching relationship between delivery drivers and orders during the delivery process (i.e., delivery driver cooperative delivery scheduling). Summary of the Invention

[0005] The present invention addresses the problems existing in the prior art and provides a real-time cooperative scheduling system for instant delivery based on hierarchical reinforcement learning. The system comprises a simulator module, a decision module, and a time constraint module. The simulator module includes at least environmental information, delivery driver information, order information, and encounter information. All of this information is manipulated within the simulator module to simulate the routine processes of delivery drivers in instant delivery and to create a training and evaluation environment for the algorithm, thereby allocating orders and scheduling delivery drivers. The decision module utilizes a cooperative delivery scheduling model based on hierarchical reinforcement learning. Based on the delivery driver information, order information, and encounter information collected by the simulator module, it extracts features through an actor-critic network to make delivery decisions based on encounter interactions among delivery drivers, and feeds these decisions back into the simulator loop. The time constraint module extracts the time constraint features of delivery driver encounter interactions in encounter scenarios based on the encounter information. It then schedules and guides the decision-making schemes in the decision module, taking into account the real-time remaining delivery time of an order, the number of historical order transfers, and the real-time scheduling requirements of instant delivery. This system recommends delivery driver encounter interactions for cooperative delivery by scheduling, thereby achieving the overall goals of increasing the number of drop-off orders during the delivery process, improving delivery efficiency while meeting delivery time constraints, and reducing order overdue rates.

[0006] In order to achieve the above objectives, the technical solution adopted by the present invention is: a real-time cooperative scheduling system for instant delivery based on hierarchical reinforcement learning, including a simulator module, a decision module and a time constraint module.

[0007] The simulator module includes at least environmental information, delivery driver information, order information, and encounter information. All of this information is manipulated in the simulator module to simulate the delivery driver's routine processes in instant delivery and to create a training and evaluation environment for the algorithm, thereby allocating orders and dispatching delivery drivers.

[0008] The decision module uses a cooperative delivery scheduling model based on hierarchical reinforcement learning. Based on the delivery person information, order information, and encounter information collected by the simulator module, it extracts features through an actor-critic network, makes delivery cooperative delivery decisions based on encounter interactions, and feeds these decisions back to the simulator loop.

[0009] The time constraint module extracts the time constraint characteristics of the delivery person's encounter interaction in the encounter scenario based on the encounter information, comprehensively considers the real-time remaining delivery time of the order, the historical number of order transfers, and the real-time scheduling requirements of instant delivery, and schedules and guides the decision-making plan in the decision-making module.

[0010] As an improvement of the present invention, in the simulator module,

[0011] The data operation of the environmental information includes at least daily order record data loading, daily courier delivery record and trajectory data loading, delivery driver status update at each moment, real-time delivery driver set acquisition and initial order dispatch decision.

[0012] The data operations of the delivery personnel information include at least delivery personnel delivery route planning, order allocation and order transfer receipt, order transfer discarding, and delivery personnel acceptance or abandonment status update;

[0013] Data operation of the order information: initializing and updating the status of the order information one by one according to the environment information and delivery person information;

[0014] The encounter information: as a trigger event class, when an encounter between delivery persons is detected, the encounter status is obtained to support order transfer decisions.

[0015] As an improvement of the present invention, the cooperative delivery scheduling model based on hierarchical reinforcement learning in the decision module includes an encounter time selector and an order transfer scheduler.

[0016] The encounter event selector selects the encounter event and observes the high-level state And give an action To decide whether to meet Transfer order when If the encounter time selector chooses not to transfer the order, it will continue to process the next encounter event; otherwise, it will call the order transfer scheduler and receive the delayed reward after executing the last low-level action;

[0017] The order transfer scheduler is based on each order to be transferred Low-level state Generate instructions for the selected delivery person to accept the order Low-level actions Environments receive layered actions And feed the low-level and high-level rewards back to the order transfer scheduler, and update the state using the corresponding state transition.

[0018] As an improvement of the present invention, the encounter time selector is composed of a high-level participant network Actor and a high-level evaluator network Critic. The high-level participant network Actor generates actions based on the encoded high-level state embedding to decide whether to Transfer orders, specifically:

[0019] Get in Possible long-term rewards of encounter event selection

[0020]

[0021] in, The high-level state at is defined as The high-level action at is defined as The low-level action at is defined as The high-level reward is defined as The discount factor is defined as γ, Indicated by Parameterized low-level strategy; Q l :S×Ω×A→R is a given high-level action and perform low-level actions while observing the state The order transfer value; V h (·) indicates the high-level status value after the transfer order;

[0022] In calculating the possible Q h Finally, the Softmax function is used to generate the probability of action selection and provide high-level strategies To decide the transfer order when encountering the situation, the high-level strategy for:

[0023]

[0024] in, The high-level state at is defined as The high-level action at is defined as are network parameters of the high-level participant network.

[0025] As another improvement of the present invention, the high-level evaluator network Critic is based on the high-level participant network Actor State value when making an encounter event selection decision To measure the long-term reward, the state value Specifically:

[0026]

[0027] Among them, the high-level cumulative reward of all encounter events at time step t is defined as are the network parameters of the high-level evaluator network.

[0028] As another improvement of the present invention, the order transfer scheduler includes a low-level participant network Actor and a low-level evaluator network Critic, wherein the low-level participant network Actor generates actions based on the low-level encoding state embedding to determine each order. exist Transmit to delivery person Specifically: Through three layers of feedforward hidden layers, the transferable value Q of the order is obtained l And output the final low-level action a through the Softmax function l , the order transfer value Indicates that in a given Perform a low-level action when The action value of is defined as:

[0029]

[0030] Where P(·) is the low-level state transition probability, It is executed The low-level state value after Orders The low-level action generation of follows a strategy defined as:

[0031]

[0032] Among them, A is A list of candidate delivery personnel IDs.

[0033] As another improvement of the present invention, the low-level evaluator network Critic is a function approximator that receives the low-level filter state embedding as input and outputs the low-level state value V l To evaluate And feedback to the lower-level participant network Actor to update the strategy network parameters; the order exist The low-level state value function Defined as:

[0034]

[0035] in, are the network parameters of the low-level evaluator network.

[0036] As another improvement of the present invention, the time constraint module includes an encounter state encoder module considering time constraints and an order filtering module. The encounter state encoder module considering time constraints is used to encode the state at the beginning of the encounter and extract the characteristics of the complete encounter process; the order filtering module filters inappropriate orders based on the remaining delivery time and the frequency of transferred orders. The filtering constraints include state constraint values ​​and frequency constraint values. When the state constraint value or the frequency constraint value is 1, the order is filtered.

[0037] As a further improvement of the present invention, the state context is used in the state constraint. To filter out orders with less remaining delivery time, the state context is defined as Order A binary vector of:

[0038]

[0039] in, yes The remaining delivery time, β is the remaining time threshold;

[0040] In the frequency constraint, each order exist Frequency Constraints To filter out inappropriate orders, specifically

[0041]

[0042] in, yes The transfer time, ∈ o is the order frequency constraint for the order.

[0043] Compared with the prior art, the present invention has the following beneficial effects: it provides a real-time cooperative scheduling system for instant delivery based on hierarchical reinforcement learning. First, in the hierarchical design of the encounter event selector and the encounter-considered order transfer scheduler, delivery time constraints (i.e., promised delivery time) are used to calculate platform revenue and delivery driver income as constraints for reward items. The encounter event selector is used to filter out inappropriate delivery driver encounter events, thereby narrowing the state space of reinforcement learning from the perspective of encounter scenarios and reducing scheduling calculation time. At the same time, an encounter state encoder that considers time constraints is used to extract a representation embedding vector of the complete encounter process (e.g., encounter constraint time embedding) to support real-time delivery driver cooperative scheduling. In addition, in the order filtering module that considers time constraints, delivery time constraints and transfer frequency constraints are considered to filter inappropriate orders, thereby accelerating online scheduling and improving overall decision-making performance. This system is based on real-time cooperative scheduling of delivery drivers based on encounter interactions. Without changing the delivery drivers' original delivery routes as much as possible, it efficiently realizes cooperative delivery among delivery drivers, effectively improving overall delivery efficiency and revenue, and reducing order timeout rates. It can be fully utilized in industries such as food delivery, errand running, and express services, and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a structural diagram of the instant delivery real-time cooperative scheduling system based on hierarchical reinforcement learning of the present invention;

[0045] Figure 2 This is a workflow diagram of the simulator module in the instant delivery real-time cooperative scheduling system based on hierarchical reinforcement learning of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0047] Example 1

[0048] The definitions of the relevant symbols involved in this embodiment are as follows:

[0049]

[0050]

[0051] Real-time cooperative scheduling system for instant delivery based on hierarchical reinforcement learning, such as Figure 1As shown in the figure, it includes the following modules: Module 1: Simulator module, which is used to simulate the routine process of delivery personnel in instant delivery and build an environment for training and evaluation of algorithms; through the real-time status of orders, GPS location of delivery personnel, information about delivery personnel encounters, geographical information of the area, and road traffic conditions collected by the platform during the actual delivery process; the features of this part of data are extracted to drive the simulator. The simulator we built can initialize the distribution of delivery personnel and orders based on the input data, and simulate the real delivery environment to allocate orders and schedule delivery personnel. The process is as follows Figure 2 shown.

[0052] First, build the underlying class of the simulated delivery environment to drive the simulator to run. The design is as follows:

[0053] In the simulator, we have built four classes to construct objects for the operations in the simulator, namely: Environment, Delivery Personnel, Order, and Encounter. The following will define the four classes respectively.

[0054] Environment: As the overall simulated delivery environment object class, it is mainly responsible for initialization and updating the overall environmental state based on decisions during the delivery simulation process. Its own attributes include time, the collection of delivery personnel objects in the experimental area, and the collection of order objects. Its main operations include loading daily order record data in the experimental area, loading daily courier delivery records and trajectory data in the experimental area, updating the delivery personnel status at each moment, obtaining the set of delivery personnel who can be assigned orders in real time, and making initial order dispatch decisions.

[0055] Courier: Attributes include the start time, courier ID, current GPS location, the status and information of the order being delivered, the current list of delivery routes, and the delivery time. Key operations include planning delivery routes, assigning orders, receiving order transfers, discarding order transfers, and updating the status of courier acceptance or abandonment. Couriers are one of the main objects implementing cooperative delivery scheduling. While executing their own operations according to the real-time delivery process, they receive order exchange instructions from the cooperative delivery scheduling model, which in turn updates their status.

[0056] Order: Attributes include the order ID, merchant order location, user delivery location, order creation time, merchant arrival time, order delivery time, estimated delivery time, the current order delivery stage, the number of times the order has been transferred, the order price, and the delivery fee. The main operation is to initialize and update the order information one by one using the input data. The Order class is only used to create order objects to facilitate order dispatch and transfer operations. Regarding the order itself, there are no additional operations in the simulator of this patent.

[0057] Encounter: Attributes include a list of the IDs of the couriers who encountered each other, the start and end time of the encounter, the number of orders the couriers had at the time of the encounter, and a list of order objects. Its main operation is to initialize and update its own order information one by one using the input data. The Encounter class acts as a trigger event class. When an encounter between couriers is detected, it obtains the encounter status to support order transfer decisions. Regarding the encounter itself, there are no additional operations in the simulator of this patent.

[0058] In the simulator, we model various events in the instant delivery scenario, including order generation, order dispatch, deliveryman exchange of orders, and deliveryman location updates based on route planning. After receiving a new order, the deliveryman updates his subsequent delivery route based on the specific situation, and then accepts and delivers new orders. The timeline of various events is designed as follows: (1) Add a new order in the simulator, extract order features, deliveryman spatiotemporal distribution features, deliveryman route planning status, and road environment features. (2) Based on the above extracted features, detect encounter events between deliverymen to extract features in each deliveryman encounter scenario. (3) Make order transfer decisions and feedback new deliveryman-order matching pairs. (4) Execute the new deliveryman-order matching results in the simulator, update the status of each deliveryman and the subsequent delivery route selection.

[0059] Based on the delivery driver's historical trajectory data, we extract and calculate the distribution of delivery speeds under various scenarios (different spatiotemporal distributions, order statuses, weather conditions, and traffic conditions). We then set the delivery driver's real-time delivery speed based on these statistics. We also extract the delivery driver's reaction time after receiving an order from the order status record, which is the difference between the order receipt time and the order creation time under the same spatiotemporal distribution. Based on the delivery driver's real-time speed and their route planning, we calculate the travel time required for the delivery driver to stop their delivery and meet to exchange orders. Based on the difference between the delivery driver's arrival time and the time they pick up the order, we extract the processing time characteristics of order exchange between delivery drivers from the order status record. Given the fine-grained temporal nature of delivery encounters, the simulator uses a time step of 1 second. Furthermore, based on practical considerations, the simulator incorporates quantitative constraints on the delivery driver encounter and order exchange process.

[0060] The behavior of the delivery drivers in the simulator is as follows: if no order exchange scheduling decision is made, delivery drivers in the experimental area continue to deliver orders according to their original order assignments and the platform's recommended routes. If a decision is made to exchange orders, delivery drivers in the experimental area find the nearest safe order exchange location recommended by the cooperative delivery scheduling model and transfer the order. After the exchange is complete, the route plan is updated to deliver the new set of orders. The simulator is deployed in the development environment of a real platform. The development environment has the same data flow and interfaces as the real-world production environment. The number of delivery drivers, the status of each delivery driver, and the status of each order change at each time step.

[0061] Module 2: The decision-making module uses a cooperative delivery scheduling model based on hierarchical reinforcement learning. Based on the order status, delivery driver status, and encounter status inputs collected from the simulator module, it extracts features through an actor-critic network and makes delivery driver cooperative delivery decisions based on the encounter interaction. This determines how to transfer orders between the encountering delivery drivers, namely whether to transfer orders in the current encounter scenario, and for each order, whether to transfer and to which encountering delivery driver. Based on the large number of delivery driver encounter events detected, order records, delivery driver trajectory data, and delivery driver encounter information are extracted from the instant delivery platform. Feature analysis, feature extraction, and modeling are performed on the complex encounter scenarios. The spatiotemporal distribution of delivery driver encounters at different time steps, as well as delivery driver status and order status information at the time of the encounter, are obtained. These information is input into the cooperative delivery scheduling model to obtain delivery driver cooperative delivery scheduling decisions. These decisions are then fed back into the simulator loop to train the cooperative delivery scheduling model based on the data.

[0062] The main framework of the cooperative delivery scheduling model method is a hierarchical reinforcement learning technology framework based on the Actor-Critic algorithm, which mainly includes an encounter event selector and an order transfer scheduler. Each layer contains a Critic network and an Actor network to evaluate the strategy network and output the cooperative delivery scheduling strategy. Under different encounter scenarios, the cooperative scheduling of orders considering encounters aims to (1) model complex encounter scenarios and select appropriate encounter scenarios; (2) schedule the order transfer between encounter delivery personnel in the selected encounter scenario. Inspired by the theory of hierarchical abstract machines, we formulate the problem as a hierarchical Markov decision process, that is, decomposing the entire task into two subtasks: (1) high-level task M h :Use the encounter event selector to select the encounter event, (2) low-level task M l :Use the order transfer scheduler to perform order transfer scheduling considering encounters. Specifically, given an encounter event In the state of transfer the order, or (b) which delivery person receives the transferred order ) and uses the feedback provided by the environment (i.e., the real delivery environment) after executing this series of decisions as rewards (e.g., order delivery time (ODT), platform revenue (ORP), driver delivery time (CDT), and driver revenue (CDR). The agent then learns from the rewards and updates its state. Our ultimate goal is to maximize the expected cumulative reward in each round (e.g., a day), that is, to optimize global delivery efficiency, revenue, and order delay rate by locally transferring orders between couriers as they encounter each other.

[0063] Taking into account the dynamic changes in the delivery process, we define the components of hierarchical reinforcement learning as follows: The high-level state s h : The high-level state at is defined as

[0064] – is the encounter state, including the time step and location of the encounter, and the encounter trend embedding To simulate the encounter trend, we construct an encounter distance series Where ξ is the length of the sequence. Each element is the distance between delivery persons within a period of time v (e.g. 30 seconds). We then use the gated recurrent unit network (GRU) to extract the encounter trend embedding.

[0065] – It's the deliveryman. The status of the delivery personnel includes their real-time location, current capacity, their next stop location (the location of the merchant or customer), the remaining delivery time to reach the next stop, and the frequency of the delivery personnel’s order transfer operations.

[0066] – Contains the overall information of the order: (1) the similarity of orders delivered by the same delivery person, (2) the similarity of orders delivered by different delivery persons, (3) the shortest overdue time of an order, and (4) the longest remaining delivery time of an order. The similarity of orders is calculated by the cosine similarity between the vectors of the order embeddings.

[0067] High-level action a h : The high-level action is defined as This is a binary value used to indicate an encounter Whether to transfer the order. It means that No order transfer action was performed.

[0068] High-level rewards h : The high reward is measured by two factors, taking into account the willingness of the deliveryman to transfer orders: (1) the deliveryman’s total delivery time CDT and (2) the deliveryman’s The total revenue CDR is formally defined as

[0069]

[0070] Among them CDT b 、CDT a The total CDT and CDR before and after the deliveryman changed the order respectively b 、CDR a The total CDR of the delivery person for the current order before and after the cooperative delivery. a The shorter the CDR a The higher it is, the bigger the reward. is the weighting factor.

[0071] Low-level states l : The low-level state contains each order exist Real-time order status of transit and matching information between delivery personnel and orders, including Remaining delivery time, The next stop location, similar to Orders, The similarity between the next stop location of the delivery person and the next stop location of the delivery person, the number of orders currently assigned by the delivery person, and the current delivery time CDT of the delivery person.

[0072] Low-level action a l :We will each order exist The low-level action of transfer is defined as Where CID∈A is the delivery driver ID of the selected delivery driver.

[0073] If CID is equal to The original delivery person ID indicates that there is no information about The order transfer action. Low-level reward r l :The low-level reward will be exist The order delivery time ODT and platform revenue ORP are taken into account.

[0074]

[0075] in for ODT before and after transfer, for ORP before and after transfer.

[0076] Dealing with the encounter When , the high-level reinforcement learning agent (i.e., the encounter event selector) first observes the high-level state And give an action To decide whether Transfer order. If the high-level reinforcement learning agent chooses not to transfer the order (i.e. ), it continues to process the next encounter event. Otherwise, it calls the low-level reinforcement learning agent (order transfer scheduler) and receives the delayed reward after executing the last low-level action. The low-level reinforcement learning agent calculates the reward based on each order to be transferred. Low-level state Generate instructions for the selected delivery person to accept the order Low-level actions Environments receive layered actions And feed the low-level and high-level rewards back to the agent. Finally, update the state using the corresponding state transition.

[0077] Based on the rich data collected, the encounter event selector (a high-level reinforcement learning agent) aims to optimize the platform's delivery efficiency in actual delivery. The deliverymen initially make independent deliveries according to the platform's dispatch plan. Once a deliveryman encounter is detected, the system will analyze the current encounter scenario, model and select the encounter scenario to determine whether the deliveryman will transfer the order in the current encounter scenario. The encounter event selector trains the Actor-Critic network based on the state input to select the appropriate encounter scenario for cooperative delivery between deliverymen. Calculate the suitability of order transfers between delivery drivers and select appropriate encounter events to transfer orders, thereby improving delivery efficiency and income for delivery drivers and ensuring the timeliness of each order transfer. It mainly consists of two networks: the high-level actor network and the high-level evaluator network. The specific design is as follows:

[0078] High-level actor network Actor: The high-level actor network generates actions based on the encoded high-level state embeddings (including encounter embeddings, deliveryman embeddings, and general order embeddings) to decide whether to Specifically, we concatenate the above embeddings and obtain the encounter fitness value Q after inputting them into three layers of feed-forward hidden layers. h . Indicates The possible long-term payoff of the encounter event selection is defined as:

[0079]

[0080] in Indicated by Parameterized low-level strategies.

[0081] Q l :S×Ω×A→R is a given high-level action and perform low-level actions while observing the state The order transfer value of V h (·) indicates the high-level status value after the transfer order.

[0082] In calculating the possible Q h Finally, we use the Softmax function to generate the probability of action selection and provide a high-level strategy To determine the encounter occasion transfer order, where are network parameters of the high-level participant network.

[0083]

[0084] High-level evaluator network Critic. High-level evaluator network aims to State value when making an encounter event selection decision To measure long-term rewards. Collect the parameters of the value network (high-level evaluator network).

[0085]

[0086] The order transfer scheduler (low-level reinforcement learning agent) is designed to decide how to select the encounter event selected by the encounter event selector. The specific matching relationship between the delivery person and his order is adjusted in the selected encounter event to decide the exchange action for each order. It mainly includes two networks: the low-level actor network and the low-level evaluator network. The specific design is as follows:

[0087] Low-level participant network Actor: The low-level participant network generates actions based on the low-level encoding state embedding to determine each order exist Which delivery person to transmit Including detailed order embedding and deliveryman order matching embedding. We obtain the transferable value Q of the order through three layers of feedforward hidden layers l And output the final low-level action a through the Softmax function l . Order transfer value Indicates that in a given Perform a low-level action when The action value of is defined as:

[0088]

[0089] where P(·) is the low-level state transition probability, It is executed The low-level state value after . Orders The low-level action generation of follows a strategy defined as:

[0090]

[0091] Where A is A list of candidate delivery personnel IDs.

[0092] Low-level critic network Critic: The low-level critic network is a function approximator that receives the low-level filter state embedding as input and outputs the low-level state value V l To evaluate And feed back to the lower-level participant network to update the strategy network parameters. exist The low-level state value function Defined as:

[0093]

[0094] in are the network parameters of the low-level evaluator network.

[0095] In this step, complex tasks are decomposed into multiple subtasks to achieve better performance solutions. Encounter events that are not suitable for order delivery are pre-filtered from the perspective of encounter scenarios, which reduces some state spaces and accelerates scheduling to meet real-time scheduling requirements. In addition, multiple rewards are considered to measure delivery efficiency, timeliness, and revenue from aspects such as delivery personnel, customers, and the platform.

[0096] Module 3: Time Constraint Module, based on encounter data, extracts the time constraint characteristics of delivery personnel's encounter interactions in a large number of diverse encounter scenarios, comprehensively considers the real-time remaining delivery time of the order, the number of historical order transfers, and the real-time scheduling requirements of instant delivery, and guides the scheduling intensity of the decision-making module to achieve effective delivery personnel encounter transfer order recommendation plans while ensuring real-time performance.

[0097] The time constraint module mainly consists of two submodules, namely the encounter state encoder module considering time constraints and the order filtering module. The specific design is as follows:

[0098] (1) Time-constrained encounter state encoder: Considering the duration of the encounter, we design a time-constrained encounter state encoder to encode the observed state at the initial stage of the deliveryman encounter to capture the complete encounter characteristics (e.g., encounter duration, encounter direction, encounter speed). The specific design is as follows:

[0099] A feature extraction scheme that fuses multi-head self-attention and convolution helps us model the entire encounter process by leveraging their respective strong capabilities in modeling global and local information, which is beneficial to the subsequent decisions of the encounter event selector and the encounter-considered order transfer scheduler.

[0100] First, we use the notation To express The feature set of high-level or low-level states Then for the mth attention head, we use the encoding matrix f u Projected into the mth query, key, value representation. We calculate the following Softmax function:

[0101]

[0102] Where W fis a trainable parameter used to consider the impact of different feature types (i.e., encounter features, deliveryman features, deliveryman-order matching features, and order features), and M is the number of attention heads. z ∈F k,t , we use the Softmax function The output of the attention head is weighted by its value representation, and then all these weighted value representations are added together as the output of the m-th attention head. Finally, we concatenate the outputs of the M attention heads and input the concatenated vector into the nonlinear ReLU activation function to output the u-th state feature embedding f u ′.

[0103]

[0104] This state feature embedding collects weighted information of each state feature to perform global attention modeling, and each feature f u Projection to f u 'middle.

[0105] In addition, we also use convolutional layers to perform local perception of adjacent feature information (i.e., features of the same type), aiming to extract the above-mentioned global attention embedding f u ′∈F′ k,t Detailed local features of .,model the subsequent encounter behavior.,We then provide the processed embeddings to the encounter event,selector and the time-constrained order filtering module.

[0106] (2) Order filtering module considering time constraints: This module is designed to pre-filter specific orders that are not suitable for exchange, aiming to reduce the state space of reinforcement learning and accelerate scheduling to meet time constraints. The specific design is as follows:

[0107] Considering the real-time requirements of scheduling, the filtering module is used to pre-filter inappropriate orders for the low-level reinforcement learning agent, thereby reducing the low-level state space and reducing the computational time of scheduling. It also provides a stable trade-off between performance improvement and order transfer overhead for the exploration of the low-level reinforcement learning agent, which is beneficial to the overall performance of collaborative order scheduling. Specifically, we mainly consider the remaining delivery time and the frequency of order transfer to filter inappropriate orders. If One of the state constraint values ​​and frequency constraint values ​​is 1, we filter for the low-level reinforcement learning agent

[0108] State constraint: The state constraint mainly considers the state of the order and the remaining delivery time of the order. First, we estimate the delivery time of each order by using XGBoost to predict the delivery route of the delivery person. The remaining delivery time is then calculated by calculating the difference between the expected delivery time and the current time. In order to avoid a small amount of optimization caused by the additional order transfer overhead, we use the state context To filter out orders with a short remaining delivery time (e.g. 10 minutes), the state context is defined as Order A binary vector of:

[0109]

[0110] in yes The remaining delivery time is , and β is the remaining time threshold.

[0111] Frequency Constraint: Similarly, to avoid the situation where a specific order is frequently transferred, which will cause more detour time, we use each order exist Frequency Constraints To filter out inappropriate orders.

[0112]

[0113] in yes The transfer time, and ∈ o It is the order frequency constraint of the order (for example, once). The system in this case is based on the support of encounter detection by the deployed virtual Beacon service, which enables the exchange of orders through the encounter of delivery personnel without changing the original delivery behavior of the delivery personnel as much as possible, thereby increasing the number of orders along the way during the delivery process and improving the overall delivery efficiency of the platform while meeting the delivery time constraint. It can be used for resource allocation tasks under time constraints, such as food delivery, errand running, express service, etc.; it can be used for real-time scheduling of cooperation between logistics industry personnel, such as carpooling and shared ride services; it can also be used to solve real-time scheduling problems in dynamically changing scenarios, such as real-time scheduling of ambulances and robot delivery in the field of smart transportation; it can also be used for interactive behavior modeling and analysis and interactive behavior prediction, and use individual data information and interactive data information to establish state encoder model technology to predict future interactive behaviors, for example, it can be used for commercial new business.

[0114] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A real-time collaborative scheduling system for instant delivery based on hierarchical reinforcement learning, characterized by: Including simulator module, decision module and time constraint module, The simulator module includes at least environmental information, delivery driver information, order information, and encounter information. All of this information is manipulated within the simulator module to simulate the routine processes of delivery drivers in instant delivery and to create a training and evaluation environment for the algorithm, thereby allocating orders and dispatching delivery drivers. The encounter information serves as a trigger event class. When an encounter between delivery drivers is detected, the encounter status is obtained to support order transfer decisions. The decision-making module adopts a cooperative delivery scheduling model based on hierarchical reinforcement learning. Based on the delivery person information, order information and encounter information collected by the simulator module, it extracts features through the Actor-Critic network, makes delivery person cooperative delivery decisions based on encounter interactions, and feeds the decision back to the simulator loop. The cooperative delivery scheduling model based on hierarchical reinforcement learning includes an encounter event selector and an order transfer scheduler. The encounter event selector selects the encounter event and observes the high-level state And give an action To determine whether the kth encounter event at time t Transfer order when When the encounter event selector chooses not to transfer the order, it continues to process the next encounter event; otherwise, it calls the order transfer scheduler and receives the delayed reward after executing the last low-level action; the encounter event selector is composed of a high-level participant network Actor and a high-level evaluator network Critic. The high-level participant network Actor generates actions based on the encoded high-level state embedding to decide whether to Transfer orders; The order transfer scheduler is based on each order to be transferred The low level state Generate instructions for the selected delivery person to accept the order Low-level actions Environments receive layered actions And feed the low-level and high-level rewards back to the order transfer scheduler, and update the state using the corresponding state transition; The time constraint module: extracts the encounter interaction time constraint characteristics of the delivery person in the encounter scenario based on the encounter information, comprehensively considers the real-time remaining delivery time of the order, the historical order transfer times of the order, and the real-time scheduling requirements of instant delivery, and schedules and guides the decision-making plan in the decision module; the time constraint module includes an encounter state encoder module considering time constraints and an order filtering module, the encounter state encoder module considering time constraints is used to encode the state at the beginning of the encounter and extract the characteristics of the complete encounter process; the order filtering module filters inappropriate orders based on the remaining delivery time and the frequency of transferring orders, and the filtering constraints include a state constraint value and a frequency constraint value. When the state constraint value or the frequency constraint value is 1, the order is filtered.

2. The instant delivery real-time collaborative scheduling system based on hierarchical reinforcement learning as claimed in claim 1, characterized in that: In the simulator module, The data operations of the environmental information include at least loading daily order record data, daily courier delivery record and trajectory data, updating the delivery driver status at each moment, obtaining the set of delivery drivers who can assign orders in real time, and making initial order dispatch decisions; The data operations of the delivery personnel information include at least delivery personnel delivery route planning, order allocation and order transfer receipt, order transfer discarding, and delivery personnel acceptance or abandonment status update; The data operation of the order information: initialize and update the status of the order information one by one according to the environmental information and delivery person information.

3. The instant delivery real-time collaborative scheduling system based on hierarchical reinforcement learning as claimed in claim 2, characterized in that: The encounter event selector specifically includes: Get in Possible long-term rewards of encounter event selection in, The high-level state at is defined as The high-level action at is defined as The low-level action at is defined as The high-level reward is defined as The discount factor is defined as γ; Indicated by Parameterized low-level strategy; Q l :S×Ω×A→R is a given high-level action and perform low-level actions while observing the state The order transfer value; V h (·) indicates the high-level status value after the transfer order; In calculating the possible Q h Finally, the Softmax function is used to generate the probability of action selection and provide high-level strategies To decide the transfer order when encountering the situation, the high-level strategy for: in, The high-level state at is defined as The high-level action at is defined as are network parameters of the high-level participant network.

4. The instant delivery real-time collaborative scheduling system based on hierarchical reinforcement learning as claimed in claim 3, characterized in that: The high-level evaluator network Critic is based on the high-level participant network Actor State value when making an encounter event selection decision To measure the long-term reward, the state value Specifically: Among them, the high-level cumulative reward of all encounter events at time step t is defined as are the network parameters of the high-level evaluator network.

5. The instant delivery real-time collaborative scheduling system based on hierarchical reinforcement learning as claimed in claim 3, characterized in that: The order transfer scheduler includes a low-level participant network Actor and a low-level evaluator network Critic. The low-level participant network Actor generates actions based on the low-level encoding state embedding to determine each order. exist Transmit to delivery person Specifically: Through three layers of feedforward hidden layers, the transferable value Q of the order is obtained l And output the final low-level action a through the Softmax function l , the order transfer value Indicates that in a given Perform a low-level action when The action value of is defined as: Where P(·) is the low-level state transition probability, It is executed The low-level state value after Orders The low-level action generation of follows the strategy defined below:

6. The instant delivery real-time collaborative scheduling system based on hierarchical reinforcement learning as claimed in claim 5, characterized in that: The low-level evaluator network Critic is a function approximator that receives the low-level filter state embedding as input and outputs the low-level state value V l To evaluate And feedback to the lower-level participant network Actor to update the strategy network parameters; the order exist The low-level state value function Defined as: in, are the network parameters of the low-level evaluator network.

7. The instant delivery real-time collaborative scheduling system based on hierarchical reinforcement learning as claimed in claim 6, characterized in that: In the state constraint, use the state context To filter out orders with less remaining delivery time, the state context is defined as Order A binary vector of: in, yes The remaining delivery time, β is the remaining time threshold; In the frequency constraint, define For each order exist Frequency constraints are used to filter out inappropriate orders. Specifically, in, yes The transfer time, ∈ o is the order frequency constraint for the order.

Images