Order dispatching method, device, electronic device and readable storage medium

Abstract

The disclosed embodiments disclose an order dispatching method, device, electronic device and readable storage medium. The order dispatching method includes: obtaining historical order data, historical courier data and historical delivery process data, wherein the historical delivery process data includes environmental data related to the delivery process; extracting the state of the courier and the action features of the matching action between the courier and the order from the historical order data, historical courier data and historical delivery process data; training based on the time step using the state of the courier and the action features of the matching action between the courier and the order through a preset reinforcement learning model to output the matching value between the courier and the order as an order dispatching decision; dispatching orders to couriers based on the order dispatching decision, and comprehensively considering the current impact and future impact of dispatching orders to achieve efficiency-optimized concurrent order dispatching, ensure the overall delivery volume and improve delivery efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an order dispatching method, device, electronic device and readable storage medium. Background Art

[0002] Instant delivery has developed rapidly in recent years, and its timeliness and convenience have significantly changed people's lifestyles. In instant delivery, the order dispatch process is concurrent. Couriers continuously receive new orders and deliver multiple orders (i.e., a batch of orders) in one delivery process. The delivery times of a batch of orders are often overlapping and interrelated. Due to time constraints and the possibility of timeouts (i.e., the ratio of deliveries not completed within the promised time), the pickup and delivery order of existing orders in a batch of orders will change dynamically. Existing instant delivery systems often only consider the impact of current factors when dispatching orders, such as the optimal distance and optimal timeliness. Therefore, it is difficult for existing instant delivery systems to make effective order dispatch decisions while ensuring the overall delivery volume and delivery efficiency. Summary of the invention

[0003] In order to solve the problems in the related art, the embodiments of the present disclosure provide an order dispatching method, an apparatus, an electronic device and a readable storage medium.

[0004] In a first aspect, an order dispatching method is provided in an embodiment of the present disclosure, comprising:

[0005] Acquire historical order data, historical courier data, and historical delivery process data, wherein the historical delivery process data includes environmental data related to the delivery process;

[0006] Extracting the courier's status and the action features of the matching action between the courier and the order from the historical order data, the historical courier data and the historical delivery process data;

[0007] Using a preset reinforcement learning model, training is performed based on the time step using the state of the courier and the action features of the matching action between the courier and the order to output a matching value between the courier and the order as an order dispatch decision;

[0008] The order is dispatched to the courier based on the order dispatch decision.

[0009] In combination with the first aspect, in a first implementation of the first aspect of the present disclosure, the courier's status includes the courier's personal status at the time step, the global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and the contextual spatiotemporal features based on the time step.

[0010] In combination with the first aspect, in a second implementation of the first aspect of the present disclosure, the action features of the matching action between the courier and the order include order features and matching features between the courier and the order.

[0011] The order characteristics include the merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time.

[0012] The matching features between the courier and the order include the distance and time cost between the merchant and the courier's current location, and the delivery time increased by this matching action.

[0013] In combination with the first aspect, an implementation method of the first aspect, and any one of the second implementation methods, in a third implementation method of the first aspect of the present disclosure, the preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet time constraints, and the deep matching network has a variable action space.

[0014] In combination with the third implementation of the first aspect, in a fourth implementation of the first aspect of the present disclosure, the preset reinforcement learning model is used to train based on the time step using the state of the courier and the action features of the matching action between the courier and the order to output the matching value between the courier and the order as the order dispatching decision, including:

[0015] The time-constrained dispatch action space pruning module obtains a set of candidate couriers for an order based on a time step, estimates the delivery time of the order, and filters out couriers that do not meet the time constraint from the set of candidate couriers, so as to filter out the matching action between the couriers that do not meet the time constraint and the order;

[0016] Based on the state of the courier and the action feature pair of the matching action between the courier and the order that meets the time constraint condition, the deep matching network obtains the matching value between the courier and the order, and calculates the weight of the deep matching network according to the matching value between the courier and the order and a preset matching strategy;

[0017] Based on the state of the courier, the state and the action feature pair, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network.

[0018] In combination with the first aspect, an implementation manner of the first aspect, and any one of the second implementation manners, in a fifth implementation manner of the first aspect of the present disclosure, the method further includes:

[0019] Setting a simulator to simulate the real-time situation of order dispatching by using historical order data, historical courier data and historical delivery process data, so as to train the preset reinforcement learning model in the simulator,

[0020] The preset reinforcement learning model makes order dispatching decisions and feeds back the matching results between the couriers and the orders to the simulator for the simulator to evaluate the order dispatching performance.

[0021] In a second aspect, an order dispatching device is provided in an embodiment of the present disclosure, comprising:

[0022] A historical data acquisition module, configured to acquire historical order data, historical courier data and historical delivery process data, wherein the historical delivery process data includes environmental data related to the delivery process;

[0023] A state and action feature extraction module, configured to extract the state of the courier and the action features of the matching action between the courier and the order from the historical order data, the historical courier data and the historical delivery process data;

[0024] A reinforcement learning module, configured to perform training based on a time step using a preset reinforcement learning model using the state of the courier and the action features of the matching action between the courier and the order to output a matching value between the courier and the order as an order dispatching decision;

[0025] The order dispatching module is configured to dispatch orders to couriers based on the order dispatching decision.

[0026] In combination with the second aspect, in a first implementation method of the second aspect of the present disclosure, the courier's status includes the courier's personal status at the time step, the global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and the contextual spatiotemporal features based on the time step.

[0027] In combination with the second aspect, in a second implementation of the second aspect of the present disclosure, the action features of the matching action between the courier and the order include order features and matching features between the courier and the order.

[0028] The order characteristics include the merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time.

[0029] The matching features between the courier and the order include the distance and time cost between the merchant and the courier's current location, and the delivery time increased by this matching action.

[0030] In combination with the second aspect, an implementation manner of the second aspect, and any one of the second implementation manners, in a third implementation manner of the second aspect of the present disclosure, the preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet time constraints, and the deep matching network has a variable action space.

[0031] In combination with the third implementation of the second aspect, in a fourth implementation of the second aspect of the present disclosure,

[0032] The time-constrained dispatch action space pruning module obtains a set of candidate couriers for an order based on a time step, estimates the delivery time of the order, and filters out couriers that do not meet the time constraint from the set of candidate couriers, so as to filter out the matching action between the couriers that do not meet the time constraint and the order.

[0033] Based on the state of the courier and the action feature pair of the matching action between the courier and the order that meets the time constraint condition, the deep matching network obtains the matching value between the courier and the order, and calculates the weight of the deep matching network according to the matching value between the courier and the order and a preset matching strategy;

[0034] Based on the state of the courier, the state and the action feature pair, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network.

[0035] In combination with the second aspect, an implementation manner of the second aspect, and any one of the second implementation manners, in a fifth implementation manner of the second aspect of the present disclosure, the device further includes:

[0036] A simulator is configured to simulate the real-time situation of order dispatching by using historical order data, historical courier data and historical delivery process data, so as to train the preset reinforcement learning model in the simulator,

[0037] The preset reinforcement learning model makes order dispatching decisions and feeds back the matching results between the couriers and the orders to the simulator for the simulator to evaluate the order dispatching performance.

[0038] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method as described in the first aspect or any one of the first to fifth implementations of the first aspect.

[0039] In a fourth aspect, a computer-readable storage medium is provided in an embodiment of the present disclosure, on which computer instructions are stored. When the computer instructions are executed by a processor, the method described in any one of the first aspect and the first to fifth implementations of the first aspect is implemented.

[0040] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0041] According to the technical solution provided by the embodiment of the present disclosure, historical order data, historical courier data and historical delivery process data are obtained, wherein the historical delivery process data includes environmental data related to the delivery process; the courier's state and the action characteristics of the matching action between the courier and the order are extracted from the historical order data, the historical courier data and the historical delivery process data; a preset reinforcement learning model is used to train based on the time step using the courier's state and the action characteristics of the matching action between the courier and the order to output the matching value between the courier and the order as an order dispatching decision; orders are dispatched to couriers based on the order dispatching decision, and the current and future impacts of the dispatched orders can be comprehensively considered to achieve efficiency-optimized concurrent order dispatching, and the timeliness and supply and demand influencing factors in the delivery process are analyzed through the historical order data, the historical courier data and the historical delivery process data, and then an order dispatching model is established through the reinforcement learning model to generate an order dispatching decision, and the appropriate orders are dispatched to the corresponding couriers according to the order dispatching decision to ensure the overall delivery volume and improve the delivery efficiency.

[0042] According to the technical solution provided by the embodiment of the present disclosure, the courier's status includes the courier's personal status at the time step, the global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and the contextual spatiotemporal features based on the time step. It is possible to comprehensively represent the status of the courier and the local supply and demand system of the order dispatch where he is located in a specific space and time, which is conducive to the training of the preset reinforcement learning model to learn better order dispatch decisions.

[0043] According to the technical solution provided by the embodiments of the present disclosure, the action features of the matching action between the courier and the order include order features and matching features between the courier and the order, wherein the order features include merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time, wherein the matching features between the courier and the order include the distance and time cost between the merchant and the courier's current location, and the delivery time increased by this matching action, which can comprehensively represent the matching action between the courier and the order, and is conducive to the training of the preset reinforcement learning model to learn better order dispatching decisions.

[0044] According to the technical solution provided by the embodiment of the present disclosure, the preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet the time constraint conditions, and the deep matching network has a variable action space, and the time-constrained action space pruning module can be used to implement time constraints on order dispatching decisions generated by the Actor-Critic model, and accelerate the model training and order dispatching process, so that the order dispatching decisions generated by the preset reinforcement learning model can reduce the timeout rate of order delivery, ensure the overall delivery volume and improve delivery efficiency.

[0045] According to the technical solution provided by the embodiment of the present disclosure, the preset reinforcement learning model is used to train based on the time step using the state of the courier and the action features of the matching action between the courier and the order to output the matching value between the courier and the order as the order dispatching decision, including: the time-constrained dispatching action space pruning module obtains a set of candidate couriers for the order based on the time step, estimates the delivery time of the order, and filters out the couriers that do not meet the time constraint from the candidate courier set to filter out the matching actions between the courier and the order that do not meet the time constraint; based on the state of the courier and the action feature pairs of the matching action between the courier and the order that meet the time constraint, the deep matching network obtains the matching value between the courier and the order , and calculate the weight of the deep matching network according to the matching value between the courier and the order and the preset matching strategy; based on the state of the courier, the state and the action feature pair, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network. The time-constrained action space pruning module can be used to implement the time constraint of the order dispatching decision generated by the Actor-Critic model, and accelerate the model training and order dispatching process, so that the order dispatching decision generated by the preset reinforcement learning model can reduce the timeout rate of order delivery, ensure the overall delivery volume and improve the delivery efficiency.

[0046] According to the technical solution provided by the embodiment of the present disclosure, the method also includes: setting up a simulator to simulate the real-time situation of order dispatch using historical order data, historical courier data and historical delivery process data, so as to train the preset reinforcement learning model in the simulator, wherein the preset reinforcement learning model makes order dispatch decisions and feeds back the matching results between the courier and the order to the simulator for the simulator to evaluate the order dispatch performance. The simulator can be established in combination with historical data to simulate real-time concurrent order dispatch activities, so that the simulator can be used to train the reinforcement learning model and evaluate the performance of the order dispatch system, thereby helping to improve the order dispatch system.

[0047] According to the technical solution provided by the embodiment of the present disclosure, a historical data acquisition module is configured to acquire historical order data, historical courier data and historical delivery process data, wherein the historical delivery process data includes environmental data related to the delivery process; a state and action feature extraction module is configured to extract the state of the courier and the action features of the matching action between the courier and the order from the historical order data, the historical courier data and the historical delivery process data; a reinforcement learning module is configured to utilize the state of the courier and the matching action between the courier and the order based on the time step through a preset reinforcement learning model The action features are trained to output the matching value between the courier and the order as the order dispatching decision; the order dispatching module is configured to dispatch orders to couriers based on the order dispatching decision, and can comprehensively consider the current impact and future impact of dispatching orders to achieve efficiency-optimized concurrent order dispatching, analyze the timeliness and supply and demand influencing factors in the delivery process through historical order data, historical courier data and historical delivery process data, and then establish an order dispatching model through a reinforcement learning model to generate order dispatching decisions, and dispatch suitable orders to corresponding couriers according to the order dispatching decisions to ensure the overall delivery volume and improve delivery efficiency.

[0048] According to the technical solution provided by the embodiment of the present disclosure, the courier's status includes the courier's personal status at the time step, the global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and the contextual spatiotemporal features based on the time step. It is possible to comprehensively represent the status of the courier and the local supply and demand system of the order dispatch where he is located in a specific space and time, which is conducive to the training of the preset reinforcement learning model to learn better order dispatch decisions.

[0049] According to the technical solution provided by the embodiments of the present disclosure, the action features of the matching action between the courier and the order include order features and matching features between the courier and the order, wherein the order features include merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time, wherein the matching features between the courier and the order include the distance and time cost between the merchant and the courier's current location, and the delivery time increased by this matching action, which can comprehensively represent the matching action between the courier and the order, and is conducive to the training of the preset reinforcement learning model to learn better order dispatching decisions.

[0050] According to the technical solution provided by the embodiment of the present disclosure, the preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet the time constraint conditions, and the deep matching network has a variable action space, and the time-constrained action space pruning module can be used to implement time constraints on order dispatching decisions generated by the Actor-Critic model, and accelerate the model training and order dispatching process, so that the order dispatching decisions generated by the preset reinforcement learning model can reduce the timeout rate of order delivery, ensure the overall delivery volume and improve delivery efficiency.

[0051] According to the technical solution provided by the embodiment of the present disclosure, a candidate courier set for an order based on a time step is obtained through the time-constrained dispatch action space pruning module, the delivery time of the order is estimated, and the couriers that do not meet the time constraint are filtered out from the candidate courier set, so as to filter out the matching actions between the couriers and the orders that do not meet the time constraint. Based on the state of the courier and the action feature pairs of the matching actions between the courier and the order that meet the time constraint, the deep matching network obtains the matching value between the courier and the order, and calculates the weight of the deep matching network according to the matching value between the courier and the order and a preset matching strategy; Based on the state of the courier, the pair of state and action features, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network. The time-constrained action space pruning module can be used to implement time constraints on the order dispatching decisions generated by the Actor-Critic model, and accelerate the model training and order dispatching process, so that the order dispatching decisions generated by the preset reinforcement learning model can reduce the timeout rate of order delivery, ensure the overall delivery volume and improve delivery efficiency.

[0052] According to the technical solution provided by the embodiment of the present disclosure, a simulator is configured to utilize historical order data, historical courier data, and historical delivery process data to simulate the real-time situation of order dispatch, so as to train the preset reinforcement learning model in the simulator, wherein the preset reinforcement learning model makes order dispatch decisions and feeds back the matching results between the courier and the order to the simulator for the simulator to evaluate the order dispatch performance. A simulator can be established in combination with historical data to simulate real-time concurrent order dispatch activities, so that the simulator can be used to train the reinforcement learning model and evaluate the performance of the order dispatch system, thereby helping to improve the order dispatch system.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0055] Figure 1 A flowchart showing an order dispatching method according to an embodiment of the present disclosure is shown;

[0056] Figure 2 A diagram showing an exemplary implementation architecture of an order dispatching system according to an embodiment of the present disclosure;

[0057] Figure 3 Show Figure 2 A diagram of an exemplary structure of a TCAC (Time Constrained Actor-Critic) model in an order dispatching system is shown;

[0058] Figure 4 Show Figure 3 A view of an exemplary structure of a deep matching network of an Actor module in a TCAC (Time Constrained Actor-Critic) model shown;

[0059] Figure 5 A structural block diagram of an order dispatching device according to an embodiment of the present disclosure is shown;

[0060] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0061] Figure 7 It is a structural diagram of a computer system suitable for implementing the object recommendation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0063] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0064] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0065] In the present disclosure, the user data, merchant data, courier data, order data and other data used for training the model have been anonymized and collected with the consent of the user, merchant and courier. Specifically, the embodiments of the present disclosure take three steps to protect privacy during implementation: (i) Anonymization: All analyzed data are anonymized by the service provider. All identifiable IDs, such as courier IDs, order tracking IDs, merchant IDs, and user IDs are replaced with serial identifiers; (ii) Minimum exposure: Only data useful for the order dispatching scheme involved in the embodiments of the present disclosure is processed, including timestamps, locations, and GPS track records. Then, other information is deleted to obtain the minimum exposure; (iii) Aggregation: The aggregation results are analyzed using the model, and no attention is paid to a single courier or a specific user or a specific merchant. Therefore, the learning model does not leak sensitive information about a specific individual.

[0066] As mentioned above, the existing instant delivery system often only considers the impact of current factors when dispatching orders, such as the current optimal distance, optimal timeliness, etc. Therefore, it is difficult for the existing instant delivery system to make effective order dispatch decisions while ensuring the overall delivery volume and delivery efficiency.

[0067] The disclosed embodiments intend to solve the problem of concurrent order dispatching, and the processing of concurrent order dispatching can be applied to various delivery scenarios, for example, instant delivery scenarios. Due to the rapid development of online digital platforms, instant delivery services have become a popular choice for people to order food, medicine and groceries online. Compared with general delivery, the order dispatching problem of instant delivery has unique characteristics. That is, the delivery time overlap caused by concurrent order dispatching: the order dispatching and delivery processes in instant delivery are concurrent, that is, the courier carries multiple orders at the same time in one trip, and the delivery time of these orders overlaps, especially during peak hours, while existing technologies mainly focus on independent order dispatching. In view of this conflict between massive orders and limited couriers, it is crucial to improve the efficiency of the order dispatching system by promptly resolving concurrent order dispatching (that is, the instant delivery platform assigns orders to couriers). Instant delivery has higher requirements for the overall order delivery volume and delivery efficiency. Therefore, the partial order dispatching scheme proposed in the disclosed embodiments is suitable for instant delivery scenarios.

[0068] In view of the above situation, the present disclosure proposes an order dispatching method.

[0069] Figure 1 A flow chart of an order dispatching method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the order dispatching method includes the following steps S101-S104:

[0070] In step S101, historical order data, historical courier data and historical delivery process data are obtained, wherein the historical delivery process data includes environmental data related to the delivery process;

[0071] In step S102, the action features of the courier's status and the matching action between the courier and the order are extracted from the historical order data, the historical courier data and the historical delivery process data;

[0072] In step S103, a preset reinforcement learning model is used to train based on the time step using the state of the courier and the action features of the matching action between the courier and the order to output a matching value between the courier and the order as an order dispatch decision;

[0073] In step S104, orders are dispatched to couriers based on the order dispatch decision.

[0074] In one embodiment of the present disclosure, historical order data, historical courier data, and historical delivery process data are data from the real world, which can be used to extract the status of the courier and the action features of the matching action between the courier and the order, so as to facilitate the training of the reinforcement learning model. The training result of the preset reinforcement learning model is the matching value between the courier and the order, which is used as the order dispatching decision. The matching value between the courier and the order indicates whether the order is to be dispatched to the courier when the order is dispatched. In one embodiment of the present disclosure, the higher the matching value between a courier and an order, the more suitable the order is to be dispatched to the courier.

[0075] According to the technical solution provided by the embodiment of the present disclosure, historical order data, historical courier data and historical delivery process data are obtained, wherein the historical delivery process data includes environmental data related to the delivery process; the courier's state and the action characteristics of the matching action between the courier and the order are extracted from the historical order data, the historical courier data and the historical delivery process data; a preset reinforcement learning model is used to train based on the time step using the courier's state and the action characteristics of the matching action between the courier and the order to output the matching value between the courier and the order as an order dispatching decision; orders are dispatched to couriers based on the order dispatching decision, and the current and future impacts of the dispatched orders can be comprehensively considered to achieve efficiency-optimized concurrent order dispatching, and the timeliness and supply and demand influencing factors in the delivery process are analyzed through the historical order data, the historical courier data and the historical delivery process data, and then an order dispatching model is established through the reinforcement learning model to generate an order dispatching decision, and the appropriate orders are dispatched to the corresponding couriers according to the order dispatching decision to ensure the overall delivery volume and improve the delivery efficiency.

[0076] In one embodiment of the present disclosure, the courier's status includes the courier's personal status at a time step, global delivery demand and delivery supply information observed from an order dispatch area based on the time step, and contextual spatiotemporal features based on the time step.

[0077] In one embodiment of the present disclosure, the state of the courier at time step t can be defined as s t ={p t ,g t ,c t}, where p t is the courier's personal state at the time step, g t is the global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and c t is the spatiotemporal feature of the context based on the time step. The state of courier k at time step t can be defined as in, is the personal state of courier k at time step t, is the global delivery demand and delivery supply information observed from the order dispatch area at time step t, It is a contextual spatiotemporal feature based on the time step. Since time and / or space factors such as traffic conditions (such as congestion), weather conditions (such as rainy or sunny days), and days of the week (such as weekends or weekdays) will affect the delivery behavior of couriers, the time and / or space factors can be combined into a vector as the context state and used to calculate the time and space factors. t k represents the contextual spatiotemporal features of time step t.

[0078] In one embodiment of the present disclosure, p t is the courier’s personal state at the time step, and we can define p t ={loc t ,t,n c ,DT,route}. Among them, loc t is the real-time location of the courier; t is the current time step, n c is the current capacity of the courier. DT is the total delivery time of the courier's existing orders. route is the route plan of all the stations (i.e., the origin and destination of the order) of the courier's existing orders. In one embodiment, in order to model the courier's route behavior, a GRU (gated recurrent unit) network can be used to encode the courier's route sequence into a vector, where route j is the embedding vector of the total j sites generated by the GRU network, s j is the vector of the jth site, and route j-1 is the hidden vector of the previous j-1 stations. The route is expressed by the following formula (1): j :

[0079] route j =GRU(s j , route j-1 ) (1)

[0080] In one embodiment of the present disclosure, g t is the global delivery demand and delivery supply information observed from the order dispatch area at time step t, that is, the global delivery demand and delivery supply distribution observed from the environment (order dispatch area) at time step t. In one embodiment, g t Given the supply value V at time step t t and the demand value D at time step t t The two matrices capture the distribution of couriers and order requests at time step t, which characterizes the supply and demand at a certain point in time and space.

[0081] In one embodiment of the present disclosure, c t is a contextual spatiotemporal feature based on time step t. In one embodiment of the present disclosure, c t It consists of real-time traffic conditions (e.g., congestion or not), weather conditions (e.g., sunny, cloudy, foggy, haze, heavy rain, light rain, partly cloudy), and date conditions (which day of the week). In one embodiment of the present disclosure, these contextual spatiotemporal features can be processed by one-hot encoding and formed into a vector as an additional prompt for model training.

[0082] Please note that g t and c t Updated at the beginning of each time slot, the personal status information of courier k The order dispatching system according to an embodiment of the present disclosure is updated in real time when a new order is dispatched to courier k.

[0083] According to the technical solution provided by the embodiment of the present disclosure, the courier's status includes the courier's personal status at the time step, the global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and the contextual spatiotemporal features based on the time step. It is possible to comprehensively represent the status of the courier and the local supply and demand system of the order dispatch where he is located in a specific space and time, which is conducive to the training of the preset reinforcement learning model to learn better order dispatch decisions.

[0084] In one embodiment of the present disclosure, the action characteristics of the matching action between the courier and the order include order characteristics and matching characteristics between the courier and the order, wherein the order characteristics include merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time, wherein the matching characteristics between the courier and the order include the distance and time cost between the merchant and the courier's current location, and the delivery time increased by this matching action.

[0085] In one embodiment of the present disclosure, for example, in instant delivery, the number of couriers around each order is different. Therefore, due to the real-time order location, courier location, and the number of couriers nearby, the action space of the reinforcement learning model is variable. In order to solve the variable action space and concurrent order dispatching, a vector can be used to represent the action feature. The action feature consists of two parts: (1) Order features include the merchant location l s 、User location d , price rev i , delivery fee i , the difference between the promised delivery time and the current time (also called the remaining time). (2) The courier-order matching characteristics include the distance and time cost between the merchant and the courier’s current location, as well as the delivery time T that is increased by this matching.in The additional delivery time is defined as T in =DT i+1 -DT i , where DT i is the total delivery time of the courier for existing orders according to the current route plan, DT i+1 is the new total delivery time after the courier receives order i and adds this order to the courier’s route plan.

[0086] According to the technical solution provided by the embodiments of the present disclosure, the action features of the matching action between the courier and the order include order features and matching features between the courier and the order, wherein the order features include merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time, wherein the matching features between the courier and the order include the distance and time cost between the merchant and the courier's current location, and the delivery time increased by this matching action, which can comprehensively represent the matching action between the courier and the order, and is conducive to the training of the preset reinforcement learning model to learn better order dispatching decisions.

[0087] The following further describes the solution to the concurrent order dispatch problem. The inventor models the concurrent order dispatch problem in instant delivery as a sequential decision model and then solves it using reinforcement learning. The technical problem to be solved by the embodiment of the present disclosure is to improve the overall delivery volume and delivery efficiency. In the instant delivery scenario, the problem of order timeout must also be solved, that is, it is necessary to appropriately assign orders to couriers to reduce the timeout rate.

[0088] Considering the above problems, the inventors consider formulating concurrent order dispatch in instant delivery as a Markov decision process. The courier chooses an action at each time step to accept the order and continuously obtains the incentive value. The state of the courier changes when he takes an action, that is, after the courier accepts a new order, the route plan and the total delivery time will change accordingly. Formally, this problem is characterized by four main components: {S, A, R, π}. π is the strategy (matching strategy) for making action selection decisions, which will be discussed later. {S, A, R} are listed below.

[0089] State S, that is, the state s of the courier at time step t t .

[0090] Action A, namely the matching between the courier and the order, can also be called the matching action between the courier and the order, and its meaning is as described above.

[0091] Incentive value R, given the state of courier k and the action of order i The instant incentive value of this match is defined as When an available courier picks up and delivers an order, the courier receives an instant incentive value This is represented by the following formula (2).

[0092]

[0093] in, is the number of time steps that the courier adds to the delivery time when adding a new order to its route plan. γ∈[0,1] is the discount factor. Δt The effect of the added time is taken into account. The longer the added time, the less the corresponding incentive value. When a new order is added to the courier’s route, if T in If the incentive value is high, the incentive value will be discounted. When trying to send the incentive value to courier k, the delivery time of order i is estimated and the delivery time is compared with the ToD of this order. If order i is overdue, the incentive value is set to rev i is the price of order i. N o is the total number of orders in the destination grid of the action (e.g., the merchant grid that fetches the order and reaches the order), N c is the total delivery capacity of all couriers in the destination grid, expressed as is the current capability of courier k. Capturing capacity supply and order demand helps in express order matching considering future order allocation. The higher it is, the more incentive value the courier will get when taking an action, so the courier is more likely to be assigned to areas with high delivery demand.

[0094] State transition: At the beginning of the next time step, the global distribution demand and distribution supply information g can be obtained t+1 When a courier accepts a new order, the route plan will change and the personal status p t will also change accordingly. Then we can integrate {p t+1 ,g t+1 ,c t+1} to the next state of courier k Note that the personal status p t Constantly changing, because couriers are constantly accepting new orders, and the order of order delivery will also change due to new orders. Global demand and supply t At the beginning of each time step, θ changes.

[0095] The following reference Figure 2 An example of the overall architecture of an order dispatching system in an embodiment according to the present disclosure is described. Figure 2 A diagram showing an exemplary implementation architecture of an order dispatching system according to an embodiment of the present disclosure.

[0096] from Figure 2 It can be seen that the order dispatching system in the embodiment of the present disclosure can be composed of two parts (stages), one is a part for performing offline learning (offline learning stage), and the other is a part for performing online dispatching (online dispatching stage). Figure 2 In the order dispatching system shown, historical order data, historical courier data, and historical delivery process data are learned in the offline learning part (stage) to generate order dispatching decisions, and the order dispatching decisions are provided to the online dispatching part (stage) for the online dispatching part (stage) to perform concurrent order dispatching to couriers online based on the order dispatching decisions (i.e., matching values). Afterwards, the order dispatching and corresponding delivery performed in the online dispatching part (stage) will generate new order data, new courier data, and new delivery process data for learning in the offline learning part (stage).

[0097] In an embodiment of the present disclosure, after the online dispatching part dispatches the order to the courier in a specific dispatching area, the courier picks up the goods from the merchant's location and delivers the goods to the user's location. In an embodiment of the present disclosure, the dispatching area refers to the divided delivery area. As a local delivery service, order dispatch in instant delivery is carried out by area. For example, a dispatching area can be a small part of a city, about several square kilometers (such as 5 kilometers * 5 kilometers), because the delivery range of instant delivery is 3 to 5 kilometers. A courier is usually responsible for one area so that the courier is familiar with the area and the road network. In the real-world instant delivery system, each city is divided into several areas, and each area is divided into L lon *L lat (longitude length * latitude length) rectangular grid. Each grid is approximately 100m*100m, and the shape of each grid changes to reflect the presence of buildings, rivers, roads, etc. The real-time distribution of couriers and orders can be counted at the grid level to capture fine-grained demand and supply distribution, and order dispatch is performed at the regional level.

[0098] It should be understood that the above-mentioned concurrent order dispatching is to illustrate the effect of the order dispatching system of the embodiment of the present disclosure. The order dispatching system of the embodiment of the present disclosure can be used for both concurrent order dispatching and individual order dispatching.

[0099] exist Figure 2In the order dispatch system shown, the offline learning part obtains historical order data, historical courier data and historical delivery process data from historical data, and the online dispatch part can generate subsequent data including order data, courier data and delivery process data. Among them, the historical delivery process data includes environmental data related to the delivery process. The environmental data related to the delivery process may include data related to the delivery process such as weather, traffic, date, etc. The state and action feature extraction module of the offline learning part extracts the state of the courier and the action features of the matching action between the courier and the order from the historical order data, historical courier data and historical delivery process data. From the above description of the courier's state and the description of the action features of the matching action between the courier and the order, it can be seen that Figure 2 The state and action feature extraction module shown extracts contextual spatiotemporal features (such as traffic, weather, date), order features, and courier status from the historical order data, historical courier data, and historical delivery process data provided by the online part.<p,g,c> , but this is just an example. The state and action feature extraction module finally extracts the state of the courier and the action feature action of the matching action between the courier and the order required for the reinforcement learning model to be trained, that is, {state, action}. The contextual spatiotemporal feature c in the state of the courier already includes the contextual spatiotemporal features (such as traffic, weather, date).

[0100] exist Figure 2 In the order dispatching system shown, the offline learning part also includes a TCAC (time-constrained Actor-Critic) model as a preset reinforcement learning model. The TCAC (time-constrained Actor-Critic) model is the core part of the order dispatching scheme for implementing the embodiment of the present disclosure. The time-constrained Actor-Critic model takes into account the strict time constraints and the stochastic demand-supply problem in instant delivery, and solves the problem of concurrent order dispatching. In one embodiment of the present disclosure, the Actor of the time-constrained Actor-Critic model can be a deep matching network, which aims to learn the strategy of selecting the best action for each courier. In one embodiment of the present disclosure, the Critic of the time-constrained Actor-Critic model can be a value network, which helps to achieve a more stable and efficient model learning process. The Actor deep matching network selects an action (the state of the courier and the matching action between the courier and the order) and outputs the matching value between the courier and the order. The Critic value network evaluates the action selected by the Actor and updates the Actor.

[0101] In the online order dispatching stage, in each time step, the information of new orders and nearby couriers in the same dispatching area can be collected, and then the matching value Q can be calculated using the TCAC model. Then the couriers and orders can be continuously matched with the Q value. Considering the complex real-time factors and the concurrent characteristics of instant delivery, after the courier accepts a new order, the courier's status will be dynamically updated, and the delivery order of the order will also change dynamically. After completing the order dispatching and delivery, the total order volume and the overdue rate of the order on the network platform can be calculated.

[0102] In one embodiment of the present disclosure, the time-constrained Actor-Critic model may further include a pre-matching module (a time-constrained action space pruning module, or an estimated-timebased action pruning module) (see Figure 3 ) is intended to ensure that the time limits for order dispatch and delivery are met and to accelerate the training and dispatch process. In one embodiment of the present disclosure, the preset reinforcement learning model of the offline learning part can be an Actor-Critic model without including a pre-matching module, that is, the order dispatch decision is generated without emphasizing the time constraints of instant delivery. Such a preset reinforcement learning model is suitable for generating order dispatch decisions in a variety of delivery scenarios with relatively loose time limits. The following will mainly discuss order dispatch based on instant delivery, especially concurrent order dispatch. Those skilled in the art can understand how to generate order dispatch decisions using a time-constrained Actor-Critic model based on the teachings of the embodiments of the present disclosure, and also understand how to generate order dispatch decisions using a general Actor-Critic model.

[0103] When studying the instant delivery technology in related fields, the inventors found that the order dispatching problem of instant delivery has two unique characteristics. (1) Delivery time overlap caused by concurrent dispatching: The order dispatching and delivery process in instant delivery are concurrent, that is, the courier carries multiple orders at the same time in one trip, and the delivery time of these orders overlaps, especially during peak hours, while existing research focuses on independent order dispatching; (2) Delivery time constraints: Instant delivery has strict delivery time limits for each order, while the time limits of traditional logistics and online retail are relatively loose. Therefore, order dispatching decisions should be appropriate to avoid order timeouts. Concurrent order dispatching in instant delivery should consider the time constraints of orders and the time constraint relationship in subsequent order dispatching.

[0104] Concurrent order dispatching decisions in instant delivery are sequential and highly repetitive, resulting in a massive amount of historical order dispatching and delivery route records, which provides new opportunities for training reinforcement learning-based methods to learn excellent order dispatching decisions and improve efficiency. However, finding excellent (concurrent) order dispatching strategies from historical decision data is not simple because of two challenges: (i) there is a conflict between increasing the overall future delivery volume of concurrent orders and reducing the overtime rate of existing orders; (ii) instant delivery imposes strict restrictions on order delivery time limits, and the delivery times of orders in a delivery trip overlap. In order to complete the concurrent delivery process, couriers will "instantly" match orders multiple times. When we dispatch new orders to couriers, the couriers have dynamically changed the delivery order and delivery time of the orders, which may cause the existing orders to time out. In addition, the future delivery volume and the impact on subsequent dispatches are also crucial for concurrent order dispatching. Urgent or long-distance orders leave couriers with little time to accept other orders.

[0105] To overcome these challenges, concurrent order dispatching can be modeled as a sequential decision problem, and an order dispatching system based on TCAC is proposed. Specifically, the following unique parts can be included in TCAC. (i) A deep matching network (DMN) with a variable action space is provided, because when matching couriers with orders in instant delivery, the action space of different agents (i.e., couriers in the disclosed embodiment) in different time periods is not pre-fixed, and the number of actions is based on the real-time order location, the courier location, and the number of couriers nearby, while the traditional multi-agent Actor-Critic reinforcement learning algorithm has a discrete action space (as output) or a continuous action space with a given range, both of which are pre-fixed. Through a variable action space, DMN integrates the time constraints of the order, the real-time status of the courier, the dynamic supply and demand, and other real-time contextual information into a (long-term) matching value. This value can be used to construct a matching graph, and then online order dispatching can be performed. (ii) Considering the overlapping delivery time of existing orders and future orders, a time-constrained action space pruning module is provided to prune the dispatching actions that may cause timeouts and accelerate the training and dispatching process. In summary, the embodiments of the present disclosure point out the following contributions.

[0106] 1. By modeling concurrent order dispatch as a sequential decision problem with strict time constraints, a TCAC-based order dispatch system is provided, i.e., an Actor-Critic order dispatch system based on time constraints. The TCAC-based order dispatch system considers the spatiotemporal distribution of supply and demand, order delivery deadlines, uncertain route behaviors of couriers, and contextual factors in the offline learning phase, and learns an efficient concurrent order dispatch strategy. In the online dispatch phase, TCAC considers real-time environmental information and determines appropriate dispatch decisions between couriers and orders under strict time constraints.

[0107] 2. The disclosed embodiment proposes a time-constrained Actor-Critic order dispatching scheme that integrates a deep matching network DMN (Actor) and a value network (Critic). (i) Considering the nature of concurrent order dispatching, DMN has a variable action space. (ii) DMN takes state embedding features (integrating dynamic environmental factors and courier route behavior embeddings) and vectorized action features (order features and order-courier matching features) as input (i.e., the aforementioned<state,action> ), and outputs a (long-term) matching value. (iii) A time-constrained action space pruning module is provided to meet practical time constraints and filter actions to accelerate training and improve order dispatching efficiency.

[0108] exist Figure 2 In the order dispatching system shown, the offline learning part may also include a simulator. In order to train the deep matching network and evaluate the TCAC-based order dispatching system, a data-driven simulator is provided to simulate real-time concurrent order dispatching activities and is deployed in the development environment of the network platform. The simulator can provide real-world data for the TCAC model, for example, providing a simulation environment including order tracking data and courier trajectory data from the real world for model training. The simulator for instant delivery may have a ToD (i.e., the delivery time promised by the order) estimation model for time prediction, and may also have a route prediction model for route prediction. The simulator and the TCAC model may run as plug-in components. The data-driven simulator simulates the instant delivery scenario with historical data and simulates the real-time situation of order dispatching. The TCAC model makes order dispatching decisions and feeds back the courier-order matching results to the simulator. The simulator and the TCAC model may be implemented in various computer languages, or implemented in computer hardware, or implemented in a combination of software and hardware, and deployed on a computing platform. The inventors evaluated the TCAC model using one month of data from the online platform, achieving an increase in order volume of more than 20% and a reduction in the overtime rate of more than 20%.

[0109] The following reference Figure 3Further description Figure 2 An exemplary structure of the TCAC (Time-Constrained Actor-Critic) model in the order dispatching system is shown. Figure 3 Show Figure 2 A diagram of an exemplary structure of a TCAC (Time-Constrained Actor-Critic) model in an order dispatching system is shown.

[0110] exist Figure 3 In the above example, the courier’s state can be extracted through historical order data, historical courier data, and historical delivery process data (including environmental data involving factors such as weather, events, and roads). That is, the courier’s state s at time step t can be obtained through state embedding. t ={p t ,g t ,c t}, where p t is the courier’s personal state at time step t, g t is the global delivery demand and delivery supply information observed from the order dispatch area at time step t, and c t It is based on the spatiotemporal features of the context at time step t. In the Actor module, the order dispatch decision is made (selecting action a t ), as the order is assigned to the courier for delivery according to the order dispatch decision, new order data, courier data and delivery process data will be generated.

[0111] exist Figure 3 In the pre-matching module, the time-constrained action space pruning module is used. In instant delivery, the real-time order dispatching algorithm is required to be quite efficient (i.e. dispatching orders in a short time) and meet the time constraint (i.e., in the current case, the order will not time out when dispatched to the courier). The inventor's observation of real-world delivery data shows that for an order, due to the constraints of pickup time and waiting time, only nearby couriers can be dispatched. For each new order, the couriers near the order (i.e., usually within a few kilometers) are collected and Defined as the set of candidate couriers for order i at time step t. The courier in may be the person who picks up order i. After estimating the delivery time, you can The couriers who cannot guarantee the time limit (the estimated delivery time is greater than the promised delivery time of the order) are screened out, thereby filtering out the possible courier-order matching pairs ({s t ,a t The time-constrained action space pruning module estimates the delivery time and prunes the matching pairs that do not meet the time constraint, thereby reducing the RL action space and improving learning performance.

[0112] exist Figure 3 In , the Actor Deep Matching Network is used to calculate the (long-term) value of matching couriers and orders. In reinforcement learning, one goal is to maximize the expected reward value of each episode. One day can be set as one episode. At each time step t, the expected discounted reward is where γ is the discount factor. For a new order i, collect nearby couriers and add them to the candidate couriers set for order i at time step t. middle. The courier in may be the one who accepts order i. The traditional deep Q-learning network (DQN) takes the state as input and returns a Q-value vector whose dimension is equal to the dimension of the fixed action space. However, in instant delivery, the number of orders around the courier varies depending on the moment. When servicing an order is defined as an action, the action space cannot be guaranteed to be consistent throughout the episode. In addition, in the process of concurrent order dispatch, the courier accepts multiple orders continuously, and the delivery processes of these orders are overlapping.

[0113] The inventor proposed a deep matching network of Actor, whose network structure is as follows Figure 4 As shown.<state,action> represents the input of the deep matching network, and the courier-order matching value is a single output. The deep matching model first converts the sparse input features of the state (State) into dense features, and then we feed the state embedding and the action vector (Action) including the matching features and order features into the feature embedding layer respectively. Embedding means representing each type of high-dimensional sparse feature by a compact feature vector. Then the two features are concatenated and the matching value Q(s) is obtained after feeding these features into the feedforward hidden layer. t ,a t ). Specifically, the weight θ can be a Defined as the weight of the deep matching network, that is, the Actor (strategy) network. For each order, we get its candidate courier set and collect all<state,action> Yes. Then run the deep matching network to calculate the matching value Q(s t ,a t ), which represents the possible (long-term) matching value of this order dispatch. The larger the matching value, the more likely the courier is to accept the order.

[0114] The following discusses the matching strategy (π). After calculating all possible Q(s,a) through the deep matching network, the strategy can be used Make order dispatch decisions. Given all possible<state,action> Yes, we can calculate the corresponding Then, the policy function Given by the softmax function, the policy function is calculated by the following formula (3):

[0115]

[0116] in is the state of candidate courier k for order i at time step t, θ a is the weight of the policy network (Actor). Since the goal is to obtain the best strategy to make order dispatch decisions, the deep matching network model can be trained and the final weight θ is saved. a , used to make decisions during the order dispatch process.

[0117] Let’s discuss Critic. Critic is another function approximator that receives the state of the agent (courier) as input and outputs the state value (V-value) for a given pair. Figure 3 As shown, Critic is used to evaluate actions and provide feedback to Actor. State value function Represented as a state The expected long-term value of courier k at time step t. For each courier, at time step t, the expected long-term value in the remaining steps of an episode is defined as follows:

[0118]

[0119] where γ∈[0,1] is the discount rate of future incentive values. θ c Collect the weight of the value network (Critic).

[0120] The time difference method can be used to update the critic's weight. Orders can be sent to couriers in order and the conversion Stored to the experience memory D (i.e., storing the agent's experience samples). Then some transformations can be sampled to further train the Critic. With respect to the transformations collected from all agents, the loss function expressed by the following equation (5) is minimized To update the value network weight θ c :

[0121]

[0122] Where θc is the weight of the value network. and status At the next time step t+1, it is used as the input of the Critic network. The output of the Critic network is a scalar V(s), which indicates that the courier is in state s after choosing action a. tThe long-term value at time . Then all available courier-order pairs can be evaluated. The update process of DMN is based on the advantage function expressed by the following formula (6):

[0123]

[0124] The advantage function is used to reduce the high variance of the policy network and stabilize the model. The deep matching network can be used to try to select a good dispatch action. The advantage function tells the excellence of the action through the TD method. That is, the Critic is updated through the TD error. Using the advantage function, the gradient of the Actor can be defined as follows (7): for

[0125]

[0126] Where a is the weight of the Actor model, is the policy probability function. In the offline learning phase, we can transform the batch history by sampling from the replay memory Then, based on the gradient in equation (7), we update the Actor's deep matching network weight θ by the gradient descent rule as follows: a :

[0127]

[0128] Among them, α is the learning rate.

[0129] In the offline learning phase, DMN can be trained through the TCAC framework. In the online order dispatching phase, we make appropriate dispatching decisions based on the learned Actor policy network to increase the overall order delivery volume and reduce the overtime rate.

[0130] When performing online concurrent order dispatching based on TCAC, for each order i at a certain time step, a candidate courier set can be obtained: It consists of couriers around orders in the same area. Then, order dispatch can be formulated as a bipartite graph matching problem, and methods such as the Kuhn and Munkres (KM) method can be used to solve the bipartite graph matching problem. In the online order dispatch stage, the dispatch problem is represented by a directed graph. In this graph, there is a set U of ordered vertices u, a set C of courier vertices c, and a set E of edges e = (u, c). Based on the deep matching network, the long-term value Q(s, a) of each courier-order pair is calculated. , and then Q(s, a) is set as the weight of the edge. Assuming that there are M orders and N couriers in an area at a certain time step, order dispatch is to find a suitable match between M orders and N couriers. For each iteration, the graph contains the vertices of the first N placed orders and N couriers. Then, the goal is to find the best match represented by the following equation (9) and the KM method:

[0131]

[0132] Calculated by Actor-Critic model. If courier k is in the search set c of the i-th order i In the case of ik = 1. This means that courier k is a candidate to take order i. Otherwise, set a ik = 0, then in the order-courier bipartite graph, there is no edge between courier k and order i.

[0133] Considering the large number of instant delivery orders during peak hours and the strict delivery time limit constraints, the online concurrent order dispatching scheme based on TCAC can be accelerated by taking advantage of the time limit in the following two aspects: (i) During the offline learning stage of the TCAC model, the action space can be narrowed down and the solution space can be constrained by the time-constrained action space pruning module. The time-constrained action space pruning module ensures that each exploration of the order dispatching action is efficient and meets the time constraint, thereby reducing the training time of the reinforcement learning algorithm. (ii) During the online order dispatching stage, the time-constrained action space pruning module can eliminate the edges representing the matching actions that may cause timed orders. Due to the time constraint, most of the edges in the matching graph (bipartite graph) described above can be eliminated to reduce the order dispatching time. Experimental results show that after learning the deep matching network and the value network, the time to dispatch a batch of orders to couriers at the regional level is less than 40 milliseconds, which effectively meets the time requirement of order dispatching in instant delivery services.

[0134] In one embodiment of the present disclosure, the preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet the time constraint conditions, and the deep matching network has a variable action space.

[0135] According to the technical solution provided by the embodiment of the present disclosure, the preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet the time constraint conditions, and the deep matching network has a variable action space, and the time-constrained action space pruning module can be used to implement time constraints on order dispatching decisions generated by the Actor-Critic model, and accelerate the model training and order dispatching process, so that the order dispatching decisions generated by the preset reinforcement learning model can reduce the timeout rate of order delivery, ensure the overall delivery volume and improve delivery efficiency.

[0136] In one embodiment of the present disclosure, the preset reinforcement learning model is used to train based on the time step using the state of the courier and the action features of the matching action between the courier and the order to output the matching value between the courier and the order as the order dispatching decision, including:

[0137] The time-constrained dispatch action space pruning module obtains a set of candidate couriers for an order based on a time step, estimates the delivery time of the order, and filters out couriers that do not meet the time constraint from the set of candidate couriers, so as to filter out the matching action between the couriers that do not meet the time constraint and the order;

[0138] Based on the state of the courier and the action feature pair of the matching action between the courier and the order that meets the time constraint condition, the deep matching network obtains the matching value between the courier and the order, and calculates the weight of the deep matching network according to the matching value between the courier and the order and a preset matching strategy;

[0139] Based on the state of the courier, the state and the action feature pair, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network.

[0140] According to the technical solution provided by the embodiment of the present disclosure, the preset reinforcement learning model is used to train based on the time step using the state of the courier and the action features of the matching action between the courier and the order to output the matching value between the courier and the order as the order dispatching decision, including: the time-constrained dispatching action space pruning module obtains a set of candidate couriers for the order based on the time step, estimates the delivery time of the order, and filters out the couriers that do not meet the time constraint from the candidate courier set to filter out the matching actions between the courier and the order that do not meet the time constraint; based on the state of the courier and the action feature pairs of the matching action between the courier and the order that meet the time constraint, the deep matching network obtains the matching value between the courier and the order , and calculate the weight of the deep matching network according to the matching value between the courier and the order and the preset matching strategy; based on the state of the courier, the state and the action feature pair, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network. The time-constrained action space pruning module can be used to implement the time constraint of the order dispatching decision generated by the Actor-Critic model, and accelerate the model training and order dispatching process, so that the order dispatching decision generated by the preset reinforcement learning model can reduce the timeout rate of order delivery, ensure the overall delivery volume and improve the delivery efficiency.

[0141] In one embodiment of the present disclosure, the order dispatching method further includes: setting a simulator to simulate the real-time situation of order dispatching using historical order data, historical courier data and historical delivery process data, so as to train the preset reinforcement learning model in the simulator, wherein the preset reinforcement learning model makes order dispatching decisions and feeds back the matching results between the courier and the order to the simulator for the simulator to evaluate the order dispatching performance.

[0142] According to the technical solution provided by the embodiment of the present disclosure, the method also includes: setting up a simulator to simulate the real-time situation of order dispatch using historical order data, historical courier data and historical delivery process data, so as to train the preset reinforcement learning model in the simulator, wherein the preset reinforcement learning model makes order dispatch decisions and feeds back the matching results between the courier and the order to the simulator for the simulator to evaluate the order dispatch performance. The simulator can be established in combination with historical data to simulate real-time concurrent order dispatch activities, so that the simulator can be used to train the reinforcement learning model and evaluate the performance of the order dispatch system, thereby helping to improve the order dispatch system.

[0143] The order dispatching scheme disclosed in the embodiments of the present disclosure can be applied to other fields, for example, a field of on-demand delivery to reduce the timeout rate and increase the order delivery volume. Similar needs can also be found in other real-time information-physical systems, that is, to properly dispatch orders and formulate schedules to meet strict time constraints and improve efficiency. The TCAC-based order dispatching scheme may be extended to these scenarios with minor changes. For example, drivers or couriers in carpooling and logistics systems need to pick up customers or packages within a limited time window. For another example, an ambulance needs to take patients to the hospital as quickly as possible.

[0144] The following reference Figure 5 A structural block diagram of an order dispatching apparatus according to an embodiment of the present disclosure is described.

[0145] Figure 5 The following is a block diagram of an order dispatching device 500 according to an embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The order dispatching device includes:

[0146] The historical data acquisition module 501 is configured to acquire historical order data, historical courier data and historical delivery process data, wherein the historical delivery process data includes environmental data related to the delivery process;

[0147] A state and action feature extraction module 502 is configured to extract the state of the courier and the action features of the matching action between the courier and the order from the historical order data, the historical courier data and the historical delivery process data;

[0148] A reinforcement learning module 503 is configured to perform training based on a time step using the state of the courier and the action features of the matching action between the courier and the order through a preset reinforcement learning model to output a matching value between the courier and the order as an order dispatching decision;

[0149] The order dispatching module 504 is configured to dispatch orders to couriers based on the order dispatching decision.

[0150] In one embodiment of the present disclosure, the historical data acquisition module 501 and the order dispatch module 504 can be integrated into one module to perform online order dispatch and data acquisition. In one embodiment of the present disclosure, the historical data acquisition module 501 can be part of the simulator, that is, the simulator acquires, stores and provides historical data for the reinforcement learning module to train and evaluate the corresponding training effect; and the order dispatch module 504 can be dedicated to executing order dispatch and generating new order data, new courier data and new delivery process data. The new order data, new courier data and new delivery process data can be provided to the reinforcement learning module for training.

[0151] According to the technical solution provided by the embodiment of the present disclosure, a historical data acquisition module is configured to acquire historical order data, historical courier data and historical delivery process data, wherein the historical delivery process data includes environmental data related to the delivery process; a state and action feature extraction module is configured to extract the state of the courier and the action features of the matching action between the courier and the order from the historical order data, the historical courier data and the historical delivery process data; a reinforcement learning module is configured to utilize the state of the courier and the matching action between the courier and the order based on the time step through a preset reinforcement learning model The action features are trained to output the matching value between the courier and the order as the order dispatching decision; the order dispatching module is configured to dispatch orders to couriers based on the order dispatching decision, and can comprehensively consider the current impact and future impact of dispatching orders to achieve efficiency-optimized concurrent order dispatching, analyze the timeliness and supply and demand influencing factors in the delivery process through historical order data, historical courier data and historical delivery process data, and then establish an order dispatching model through a reinforcement learning model to generate order dispatching decisions, and dispatch suitable orders to corresponding couriers according to the order dispatching decisions to ensure the overall delivery volume and improve delivery efficiency.

[0152] In one embodiment of the present disclosure, the courier's status includes the courier's personal status at the time step, global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and contextual spatiotemporal features based on the time step.

[0153] According to the technical solution provided by the embodiment of the present disclosure, the courier's status includes the courier's personal status at the time step, the global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and the contextual spatiotemporal features based on the time step. It is possible to comprehensively represent the status of the courier and the local supply and demand system of the order dispatch where he is located in a specific space and time, which is conducive to the training of the preset reinforcement learning model to learn better order dispatch decisions.

[0154] In one embodiment of the present disclosure, the action features of the matching action between the courier and the order include order features and matching features between the courier and the order.

[0155] The order characteristics include the merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time.

[0156] The matching features between the courier and the order include the distance and time cost between the merchant and the courier's current location, and the delivery time increased by this matching action.

[0157] According to the technical solution provided by the embodiments of the present disclosure, the action features of the matching action between the courier and the order include order features and matching features between the courier and the order, wherein the order features include merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time, wherein the matching features between the courier and the order include the distance and time cost between the merchant and the courier's current location, and the delivery time increased by this matching action, which can comprehensively represent the matching action between the courier and the order, and is conducive to the training of the preset reinforcement learning model to learn better order dispatching decisions.

[0158] In one embodiment of the present disclosure, the preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet time constraints, and the deep matching network has a variable action space.

[0159] According to the technical solution provided by the embodiment of the present disclosure, the preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet the time constraint conditions, and the deep matching network has a variable action space, and the time-constrained action space pruning module can be used to implement time constraints on order dispatching decisions generated by the Actor-Critic model, and accelerate the model training and order dispatching process, so that the order dispatching decisions generated by the preset reinforcement learning model can reduce the timeout rate of order delivery, ensure the overall delivery volume and improve delivery efficiency.

[0160] In one embodiment of the present disclosure, the time-constrained dispatch action space pruning module obtains a set of candidate couriers for an order based on a time step, estimates the delivery time of the order, and filters out couriers that do not meet the time constraint from the set of candidate couriers, so as to filter out the matching action between the couriers that do not meet the time constraint and the order.

[0161] Based on the state of the courier and the action feature pair of the matching action between the courier and the order that meets the time constraint condition, the deep matching network obtains the matching value between the courier and the order, and calculates the weight of the deep matching network according to the matching value between the courier and the order and a preset matching strategy;

[0162] Based on the state of the courier, the state and the action feature pair, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network.

[0163] According to the technical solution provided by the embodiment of the present disclosure, a candidate courier set for an order based on a time step is obtained through the time-constrained dispatch action space pruning module, the delivery time of the order is estimated, and the couriers that do not meet the time constraint are filtered out from the candidate courier set, so as to filter out the matching actions between the couriers and the orders that do not meet the time constraint. Based on the state of the courier and the action feature pairs of the matching actions between the courier and the order that meet the time constraint, the deep matching network obtains the matching value between the courier and the order, and calculates the weight of the deep matching network according to the matching value between the courier and the order and a preset matching strategy; Based on the state of the courier, the pair of state and action features, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network. The time-constrained action space pruning module can be used to implement time constraints on the order dispatching decisions generated by the Actor-Critic model, and accelerate the model training and order dispatching process, so that the order dispatching decisions generated by the preset reinforcement learning model can reduce the timeout rate of order delivery, ensure the overall delivery volume and improve delivery efficiency.

[0164] In one embodiment of the present disclosure, the order dispatching device further includes:

[0165] A simulator is configured to simulate the real-time situation of order dispatching by using historical order data, historical courier data and historical delivery process data, so as to train the preset reinforcement learning model in the simulator,

[0166] The preset reinforcement learning model makes order dispatching decisions and feeds back the matching results between the couriers and the orders to the simulator for the simulator to evaluate the order dispatching performance.

[0167] According to the technical solution provided by the embodiment of the present disclosure, a simulator is configured to utilize historical order data, historical courier data, and historical delivery process data to simulate the real-time situation of order dispatch, so as to train the preset reinforcement learning model in the simulator, wherein the preset reinforcement learning model makes order dispatch decisions and feeds back the matching results between the courier and the order to the simulator for the simulator to evaluate the order dispatch performance. A simulator can be established in combination with historical data to simulate real-time concurrent order dispatch activities, so that the simulator can be used to train the reinforcement learning model and evaluate the performance of the order dispatch system, thereby helping to improve the order dispatch system.

[0168] Those skilled in the art will understand that referring to Figure 5The technical solutions described can be compared with Figures 1 to 4 The described embodiments are combined to provide a reference Figures 1 to 4 The technical effects achieved by the described embodiments. Figures 1 to 4 The specific content of the description will not be repeated here.

[0169] The above embodiments describe the internal functions and structures of the order dispatching device. In a possible design, the structure of the order dispatching device can be implemented as an electronic device, such as Figure 6 As shown in , the electronic device 600 may include a processor 601 and a memory 602 .

[0170] The memory 602 is used to store a program that supports the electronic device to execute the order dispatching method in any of the above embodiments, and the processor 601 is configured to execute the program stored in the memory 602.

[0171] In one embodiment of the present disclosure, the memory 602 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 601 to implement the above reference Figure 1-4 The steps of the method in the embodiments are described.

[0172] The exemplary embodiments of the present disclosure further provide a computer storage medium for storing computer software instructions used by the positioning device, which includes a program for executing any of the above embodiments, thereby having the technical effects brought by the corresponding method.

[0173] Figure 7 It is a structural diagram of a computer system suitable for implementing an object recommendation method based on a specific time period standard according to an embodiment of the present disclosure.

[0174] like Figure 7 As shown, the computer system 700 includes a processing unit (CPU, GPU, NPU, FPGA, etc.) 701, which can perform various processes in the embodiments shown in the above figures according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The processing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0175] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0176] In particular, according to an embodiment of the present disclosure, the method described above with reference to the accompanying drawings can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium thereof, and the computer program includes program code for executing the method in the accompanying drawings. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 709, and / or installed from the removable medium 711.

[0177] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0179] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure, thereby achieving the technical effects brought by the method.

[0180] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. An order dispatching method, comprising: Acquire historical order data, historical courier data, and historical delivery process data, wherein the historical delivery process data includes environmental data related to the delivery process; Extracting the courier's status and the action features of the matching action between the courier and the order from the historical order data, the historical courier data and the historical delivery process data; Using a preset reinforcement learning model, training is performed based on the time step using the state of the courier and the action features of the matching action between the courier and the order to output a matching value between the courier and the order as an order dispatch decision; Dispatching orders to couriers based on the order dispatching decision; The preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet the time constraint conditions, and the deep matching network has a variable action space; The preset reinforcement learning model is used to train based on the time step using the state of the courier and the action features of the matching action between the courier and the order to output the matching value between the courier and the order as the order dispatching decision, including: The time-constrained action space pruning module obtains a set of candidate couriers for an order based on a time step, estimates the delivery time of the order, and filters out couriers that do not meet the time constraint from the set of candidate couriers, so as to filter out matching actions between couriers that do not meet the time constraint and the order; Based on the state of the courier and the action feature pair of the matching action between the courier and the order that meets the time constraint condition, the deep matching network obtains the matching value between the courier and the order, and calculates the weight of the deep matching network according to the matching value between the courier and the order and a preset matching strategy; Based on the state of the courier, the state and the action feature pair, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network.

2. The method according to claim 1, wherein: The courier's state includes the courier's personal state at the time step, global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and contextual spatiotemporal features based on the time step.

3. The method according to claim 1, wherein: The action features of the matching action between the courier and the order include order features and matching features between the courier and the order. The order characteristics include the merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time. The matching features between the courier and the order include the distance and time cost between the merchant and the current location of the courier, and the delivery time increased by this matching action.

4. The method according to any one of claims 1 to 3, wherein: The method further comprises: Setting a simulator to simulate the real-time situation of order dispatching by using historical order data, historical courier data and historical delivery process data, so as to train the preset reinforcement learning model in the simulator, The preset reinforcement learning model makes order dispatching decisions and feeds back the matching results between the couriers and the orders to the simulator for the simulator to evaluate the order dispatching performance.

5. An order dispatching device, comprising: A historical data acquisition module, configured to acquire historical order data, historical courier data and historical delivery process data, wherein the historical delivery process data includes environmental data related to the delivery process; A state and action feature extraction module, configured to extract the state of the courier and the action features of the matching action between the courier and the order from the historical order data, the historical courier data and the historical delivery process data; A reinforcement learning module, configured to perform training based on a time step using a preset reinforcement learning model using the state of the courier and the action features of the matching action between the courier and the order to output a matching value between the courier and the order as an order dispatching decision; An order dispatching module, configured to dispatch orders to couriers based on the order dispatching decision; The preset reinforcement learning model is a time-constrained Actor-Critic model, wherein the time-constrained Actor-Critic model includes a time-constrained action space pruning module as a pre-matching module, a deep matching network as an Actor, and a value network as a Critic, wherein the time-constrained action space pruning module is used to filter out matching actions between couriers and orders that do not meet the time constraint conditions, and the deep matching network has a variable action space; in, The time-constrained action space pruning module obtains a set of candidate couriers for an order based on a time step, estimates the delivery time of the order, and filters out couriers that do not meet the time constraint from the set of candidate couriers, so as to filter out the matching actions between the couriers that do not meet the time constraint and the order. Based on the state of the courier and the action feature pair of the matching action between the courier and the order that meets the time constraint condition, the deep matching network obtains the matching value between the courier and the order, and calculates the weight of the deep matching network according to the matching value between the courier and the order and a preset matching strategy; Based on the state of the courier, the state and the action feature pair, the incentive value for the courier and the discount on the incentive value, the value network calculation uses a preset advantage function to determine the excellence of the matching action between the courier and the order, and calculates the gradient of the deep matching network to update the weight of the deep matching network.

6. The device according to claim 5, wherein: The courier's state includes the courier's personal state at the time step, global delivery demand and delivery supply information observed from the order dispatch area based on the time step, and contextual spatiotemporal features based on the time step.

7. The device according to claim 5, wherein: The action features of the matching action between the courier and the order include order features and matching features between the courier and the order. The order characteristics include the merchant location, user location, price, delivery fee, and the difference between the promised delivery time and the current time. The matching features between the courier and the order include the distance and time cost between the merchant and the current location of the courier, and the delivery time increased by this matching action.

8. The device according to any one of claims 5 to 7, wherein: The device also includes: A simulator is configured to simulate the real-time situation of order dispatching by using historical order data, historical courier data and historical delivery process data, so as to train the preset reinforcement learning model in the simulator, The preset reinforcement learning model makes order dispatching decisions and feeds back the matching results between the couriers and the orders to the simulator for the simulator to evaluate the order dispatching performance.

9. An electronic device, characterized in that: The method comprises a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps described in any one of claims 1 to 4.

10. A readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.

Images