A method and system for urban last-mile delivery scheduling based on multi-agent reinforcement learning

Abstract

This invention discloses a method and system for scheduling urban last-mile delivery based on multi-agent reinforcement learning, belonging to the field of intelligent logistics technology. The method acquires delivery orders and global state information, generates a global state vector through a global state perception module, generates policy parameters for each delivery agent through a multi-agent policy generation module based on a dynamic value decomposition network, allocates orders and schedules resources based on a Nash equilibrium mechanism through a competitive and cooperative scheduling module, and after executing the delivery task, feeds back the reward value to the policy generation module through an execution feedback optimization module using a Nash social welfare-based reward allocation mechanism to form a closed-loop optimization. This invention achieves coordinated scheduling and complementary advantages of three types of delivery resources: drones, delivery personnel, and smart parcel lockers, significantly improving delivery efficiency and customer satisfaction while reducing delivery costs.

Description

Technical Field

[0001] This invention relates to the field of intelligent logistics technology, and in particular to a method and system for scheduling urban last-mile delivery based on multi-agent reinforcement learning. Background Technology

[0002] With the rapid development of e-commerce and the continuous growth in demand for on-demand delivery, last-mile delivery in cities has become a crucial link in the modern logistics system. Traditional last-mile delivery mainly relies on manual delivery personnel, which suffers from low delivery efficiency, high labor costs, and difficulty in guaranteeing delivery time. In recent years, new delivery methods such as drone delivery and smart parcel lockers have gradually emerged, providing new solutions to improve last-mile delivery efficiency. However, a single delivery method cannot meet diverse delivery needs, and how to achieve the coordinated operation of multiple delivery capabilities has become an urgent technical challenge.

[0003] In the prior art, Chinese patent application CN119671425A discloses a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning. This method acquires delivery tasks and multi-level dynamic airspace information, and constructs an airspace scheduling mechanism based on this information. The airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative pathfinding solver. The altitude layer selector determines the UAV's flight altitude layer based on the delivery task and the multi-level dynamic airspace information, obtaining an altitude layer decision. The multi-UAV collaborative pathfinding solver performs horizontal collaborative path planning at each altitude layer, obtaining a path decision. The UAV executes the delivery task based on the altitude layer decision and the path decision. This method designs an altitude layer scheduling strategy for UAV delivery tasks in the vertical dimension, improving the performance of the delivery system by alleviating airspace congestion.

[0004] However, this method has the following shortcomings: First, it only targets drone delivery scenarios and does not consider the collaborative use of other delivery resources such as delivery personnel and smart parcel lockers, thus failing to fully leverage the advantages of diversified delivery capabilities; second, it adopts a pure collaborative mechanism and does not consider the competitive relationship between delivery resources, lacking an effective incentive mechanism when allocating high-value orders; third, airspace scheduling and route planning are relatively independent, lacking a globally unified value assessment and strategy optimization mechanism, making it difficult to achieve a true synergistic effect; and fourth, it does not establish a closed-loop mechanism for execution feedback and strategy adjustment, making it unable to dynamically optimize scheduling strategies based on real-time delivery results, resulting in insufficient adaptability.

[0005] Therefore, how to build a collaborative scheduling mechanism for diverse delivery resources, realize the complementary advantages and synergistic effects of drones, delivery personnel, and smart parcel lockers, and improve the overall efficiency and service quality of urban last-mile delivery is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for scheduling urban last-mile delivery based on multi-agent reinforcement learning, so as to solve the technical problems of insufficient collaborative utilization of multiple delivery resources, lack of competitive incentive mechanism, weak global optimization capability and poor dynamic adaptability in the prior art.

[0007] To achieve the above objectives, this invention provides a method and system for urban last-mile delivery scheduling based on multi-agent reinforcement learning. The method includes: acquiring delivery orders and global state information; inputting the delivery orders and global state information into a global state perception module for feature extraction and state encoding to generate a global state vector; inputting the global state vector into a multi-agent policy generation module to generate individual value functions and policy parameters for each delivery agent based on a dynamic value decomposition network; inputting the policy parameters into a competition-cooperative scheduling module to allocate orders and schedule resources based on a Nash equilibrium mechanism; executing delivery tasks according to the optimal policy combination and inputting the execution results into an execution feedback optimization module to calculate reward values ​​and distribute the reward values ​​to each delivery agent using a Nash social welfare-based reward allocation mechanism; and feeding the reward values ​​back to the multi-agent policy generation module to dynamically update network parameters and policy parameters, forming a closed-loop feedback optimization mechanism.

[0008] This invention also provides a multi-agent reinforcement learning-based urban last-mile delivery collaborative scheduling system, including a global state perception module for acquiring delivery orders and global state information and generating a global state vector; a multi-agent policy generation module for generating individual value functions and policy parameters for each delivery agent based on a dynamic value decomposition network; a competitive cooperative scheduling module for order allocation and resource scheduling based on a Nash equilibrium mechanism; and an execution feedback optimization module for calculating reward values ​​and distributing reward values ​​to each delivery agent using a Nash social welfare-based reward allocation mechanism to form a closed-loop feedback optimization mechanism.

[0009] The beneficial effects of this invention are as follows:

[0010] By constructing a collaborative scheduling mechanism for diversified delivery capabilities, the advantages of drones, delivery personnel, and smart parcel lockers can be complemented. Drones can leverage their speed advantage to handle urgent orders from remote areas, delivery personnel can leverage their flexibility advantage to handle orders in densely populated areas, and smart parcel lockers can reduce time constraints as transit nodes, thereby significantly improving delivery efficiency and customer satisfaction.

[0011] By introducing a hybrid competition-cooperation mechanism, a competition mechanism is used to incentivize delivery agents to improve service quality for high-value orders, while a cooperation mechanism is used for ordinary orders to achieve balanced resource utilization, thereby improving the fairness and stability of the system while ensuring overall delivery efficiency.

[0012] By designing a multi-agent policy generation module based on a dynamic value decomposition network, the global value function is decomposed into individual value functions of each delivery agent. The decomposition weights are determined through a state-dependent hybrid network, achieving an organic unity between global optimization and individual decision-making. Compared with the independent decision-making or simple coordination mechanisms of existing technologies, this invention can more accurately evaluate the contributions of each delivery agent and generate better collaborative strategies.

[0013] By establishing a closed-loop feedback mechanism between the execution feedback optimization module and the multi-agent strategy generation module, the network parameters and strategy parameters are dynamically updated based on the delivery execution results. This enables the system to have continuous learning and adaptive optimization capabilities, allowing it to cope with dynamic environments such as changes in order distribution, road condition fluctuations, and weather effects, and maintain long-term stable high-performance operation.

[0014] By adopting a reward distribution mechanism based on Nash social welfare, the overall benefits of the system are maximized while ensuring fairness among the various delivery agents, avoiding overwork or resource idleness of some agents, and improving the long-term stability and sustainability of the system. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0016] Figure 1 This is a flowchart illustrating the urban last-mile delivery scheduling method based on multi-agent reinforcement learning according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the structure of a multi-agent reinforcement learning-based urban last-mile delivery collaborative scheduling system according to an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the structure of the dynamic value decomposition network according to an embodiment of the present invention;

[0019] Figure 4 This is a schematic diagram of the workflow of the competition and cooperation scheduling module in an embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram of a collaborative delivery scenario using multiple delivery resources, as described in an embodiment of the present invention. Detailed Implementation

[0021] Please refer to the attached document. Figures 1-5The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, the method of the present invention includes the following steps:

[0023] First, obtain delivery order and global status information. Delivery order information includes order number, recipient address, sender address, order priority, order value, and time window requirements. Global status information includes order distribution information, road condition information, weather information, and delivery resource status information. Order distribution information reflects the order density and characteristics of each location within the delivery area at the current moment. Road condition information includes real-time traffic data such as congestion index, estimated travel time, and toll costs for each road. Weather information includes meteorological factors affecting delivery such as temperature, wind speed, precipitation, and visibility. Delivery resource status information includes the status of drones, delivery personnel, and smart parcel lockers, specifically including the location information, availability, remaining battery or stamina, and current task information for each delivery resource.

[0024] Next, the delivery orders and global status information are input into the global status perception module 1. The global status perception module 1 performs feature extraction and status encoding on the global status information to generate a global status vector. Specifically, the order distribution information is spatially gridded, dividing the delivery area into... The grid cells are statistically analyzed. Order density Order priority and order time window Real-time traffic information is collected to obtain the congestion index for each road. Travel time and toll costs ,in Assign road numbers. Acquire weather information to determine the impact coefficient of current weather conditions on drone flight and delivery personnel delivery. and The system collects delivery resource status information and obtains the location coordinates of each delivery agent. Available status Remaining battery power or stamina Current task number ,in Assign a number to the delivery agent. Perform feature fusion and vectorization encoding on the above information to generate a global state vector. .

[0025] In a preferred embodiment, the global state vector The generation employs a multilayer perceptron for feature fusion. The order grid feature matrix is ​​then used. (Including three channels: order density, priority, and time window), road condition feature vector (Features of K roads), weather feature vector (Including two types of influence coefficients), distribution resource characteristic matrix The five-dimensional features (containing N delivery agents) are flattened and stitched together to obtain the original feature vector. Using a multilayer perceptron with two hidden layers... A nonlinear transformation is performed. The first hidden layer has 512 neurons with ReLU activation, the second hidden layer has 256 neurons with ReLU activation, and the output layer has 128 neurons, resulting in the global state vector. .

[0026] Then, the global state vector is input into the multi-agent policy generation module 2. The multi-agent policy generation module 2, based on a dynamic value decomposition network, generates individual value functions and policy parameters for each delivery agent. The delivery agents include drone agents, delivery personnel agents, and smart parcel locker agents. The dynamic value decomposition network decomposes the global value function into the sum of the individual value functions of each delivery agent and determines the decomposition weights through a state-dependent hybrid network.

[0027] like Figure 3 As shown, the dynamic value decomposition network includes individual value networks, hybrid networks, and global value networks.

[0028] Individual value network for each delivery agent Building independent neural networks ,in For delivery intelligent agents Local observation information, For delivery intelligent agents The action, These are parameters for the individual value network. Local observation information. Including delivery smart agents Location information, nearby order information, and information on nearby delivery agents, etc. Including delivery smart agents The executable discrete actions include, for drone agents, actions such as accepting orders, rejecting orders, heading to the target location, and hovering; for delivery personnel agents, actions such as accepting orders, rejecting orders, heading to the target location, and resting; and for smart parcel locker agents, actions such as receiving orders, rejecting orders, and notifying for pickup. The individual value network adopts a deep Q-network structure, containing three fully connected hidden layers with 128, 64, and 32 neurons respectively. The activation function is ReLU, and the output layer outputs the Q-value of each action.

[0029] Hybrid networks based on global state vector Generate a state-dependent mixed weight matrix ,in These are the parameters of the hybrid network. The hybrid network uses a hypernetwork structure, generating a hybrid weight matrix through two fully connected layers. The first layer has 256 neurons, using ReLU as the activation function, and the second layer has [number of neurons missing]. ( (Total number of delivery agents), the activation function uses an absolute value function to ensure non-negative weights, and the output layer is reshaped as follows: The weight matrix.

[0030] The global value network calculates the global value function based on the individual value functions of each delivery agent and the mixed weight matrix. Global Value Function The calculation uses the following innovative algorithm formula:

[0031] ,

[0032] in, For the global value function, This is the global state vector. For the coordinated actions of all delivery agents, The set of parameters for all individual value networks. For parameters of the hybrid network, For the total number of delivery agents, and For the delivery agent index, The first of the mixed weight matrix One element, For delivery intelligent agents The individual value function, For delivery intelligent agents Local observation information, For delivery intelligent agents The action.

[0033] This algorithm introduces a hybrid weight matrix. This approach considers not only the individual value functions of each delivery agent but also the interactive effects between them, achieving a more accurate value decomposition. The hybrid weight matrix is ​​state-dependent, dynamically adjusting the weights of each delivery agent based on the current global state to adapt to collaborative needs in different scenarios. Preferably, each row of the hybrid weight matrix is ​​normalized, making... This ensures the rationality of value allocation.

[0034] To ensure that the global value function satisfies the monotonicity constraint, i.e., the globally optimal action corresponds to the locally optimal action of each delivery agent, this invention applies a non-negativity constraint to the hybrid weight matrix and requires that the diagonal elements... Significantly larger than off-diagonal elements In a preferred embodiment, this constraint is achieved by adding a regularization term to the loss function of the hybrid network:

[0035] ,

[0036] in, For regularization loss, and The regularization coefficient is denoted by , and the preferred value is . and , The threshold value is set to the minimum value of the diagonal elements, with a preferred value of 0.6. The first term penalizes excessively large off-diagonal elements, while the second term ensures that the diagonal elements are sufficiently large. This regularization mechanism guarantees the monotonicity of value decomposition, allowing the globally optimal strategy to be decomposed into locally optimal strategies for each delivery agent.

[0037] Based on individual value function The softmax function is used to generate the strategy parameters for each delivery agent. :

[0038] ,

[0039] in, For delivery intelligent agents Local observation Select action The probability, Temperature is a parameter used to determine the extent of exploration in the control strategy; a preferred value is [value to be filled in]. The smaller the temperature parameter, the more the strategy tends to select the action with the highest value; the larger the temperature parameter, the more exploratory the strategy. In the early stages of training... Take a larger value to enhance exploration, and gradually decrease it as training progresses. To enhance utilization.

[0040] Next, the strategy parameters are input into the competition-cooperation scheduling module 3. The competition-cooperation scheduling module 3 performs order allocation and resource scheduling based on the Nash equilibrium mechanism. For high-value orders, a competition mechanism is used to select the optimal delivery agent; for ordinary orders, a cooperation mechanism is used to achieve balanced resource allocation. The optimal strategy combination for each delivery agent is determined by solving for the Nash equilibrium point.

[0041] like Figure 4 As shown, the competitive collaborative scheduling module first categorizes delivery orders into high-value orders and ordinary orders based on order priority and order value. The calculation of order value comprehensively considers factors such as order amount, timeliness requirements, and customer level. Preferably, a value threshold is set. Orders with a value greater than this threshold are classified as high-value orders; otherwise, they are classified as ordinary orders.

[0042] For high-value orders, a competitive game model is constructed. Each delivery agent... According to strategy parameters Based on the current status information, calculate the value of high-value orders. Bidding value The bidding value is calculated by comprehensively considering factors such as the delivery agent's delivery capacity, current load, and distance to the order. Specifically, the bidding value calculation formula is as follows:

[0043] ,

[0044] in, For delivery intelligent agents For orders The bid value, For delivery intelligent agents Delivery capability score (drone speed and battery life, delivery person experience and physical strength), ranging from 0-100. For delivery intelligent agents The current load (number of orders received), with a value range of 0-10. For delivery intelligent agents Orders The distance (in kilometers) ranges from 0 to 20. For delivery intelligent agents For orders Preference level (based on historical delivery success rate and customer reviews), with a value range of 0-10. , , , For the weighting coefficient, the preferred value is [value]. , , , The higher the bid value, the more suitable the delivery agent is to execute the order.

[0045] The system selects the delivery agent with the highest bid value to execute the high-value order. If multiple delivery agents have similar bid values ​​(the difference is less than a threshold), the system will not execute the high-value order. If an order is excessively concentrated in one agent, random selection is used to prevent that agent from doing so. This competitive mechanism incentivizes delivery agents to improve their delivery capabilities and service quality, while ensuring that high-value orders receive the optimal delivery resources.

[0046] For regular orders, a collaborative optimization model is constructed. This model comprehensively considers delivery distance, delivery time, resource utilization, and load balancing, aiming to minimize the total system delivery cost and maximize resource utilization efficiency. Assume there are currently... A regular order and A usable delivery agent, defining decision variables. Indicates delivery intelligent agent Order execution? The collaborative optimization model is then:

[0047] Minimize total cost:

[0048] ,

[0049] Constraints: Each order is executed by only one delivery agent.

[0050] ,

[0051] The number of orders for each delivery agent shall not exceed its capacity:

[0052] ,

[0053] in, For delivery intelligent agents Execute orders The costs include delivery distance costs, time costs, and resource consumption costs. Preferably, the Hungarian algorithm is used to solve this optimization problem to obtain the optimal match between orders and delivery agents. The Hungarian algorithm can find the optimal matching solution in polynomial time, ensuring that the overall delivery efficiency of the system is maximized.

[0054] After completing order allocation through both competition and cooperation mechanisms, the Nash equilibrium points of the competition game model and the cooperation optimization model are solved to determine the optimal strategy combination for each delivery agent. The Nash equilibrium point is defined as the strategy combination in the current state where no single delivery agent can obtain a higher reward by changing its strategy alone. Specifically, let the delivery agents... The strategy is The strategies of other delivery intelligent agents are Then the Nash equilibrium point satisfies:

[0055] ,

[0056] in, For delivery intelligent agents The utility function is defined as the delivery agent's utility function. In strategy combination The expected cumulative reward obtained below and This refers to the strategy corresponding to the Nash equilibrium point.

[0057] In the actual solution, an iterative optimal response algorithm is used to calculate the Nash equilibrium point. The policy of each delivery agent is initialized as a random policy, and then the following steps are executed iteratively: For each delivery agent... Strategies for fixing other delivery agents Solve for the delivery intelligent agent Optimal response strategy Update the delivery smart agent The strategy is Repeat the above steps until the policy converges or the maximum number of iterations is reached. The converged policy combination is the Nash equilibrium point. In a preferred embodiment, the maximum number of iterations is set to 100, and the convergence threshold is set to a policy change of less than 0.001.

[0058] Based on the optimal strategy combination, each delivery agent executes delivery tasks. Drone agents handle remote or urgent deliveries, leveraging the speed advantage of drones to complete deliveries quickly. Delivery personnel agents handle deliveries in densely populated areas or for heavy goods, utilizing the flexibility and carrying capacity of delivery personnel. Smart parcel locker agents act as transit nodes, receiving orders delivered by drones or delivery personnel, allowing customers to pick up their packages at convenient times, reducing time constraints.

[0059] like Figure 5 As shown, in urban last-mile delivery scenarios, drone agents, delivery personnel agents, and smart locker agents work collaboratively. Drones take off from the distribution center and fly to remote order locations or smart lockers to complete rapid delivery. Delivery personnel depart from delivery stations and deliver multiple orders sequentially within densely populated areas, or drop orders off at smart lockers. Smart lockers are distributed in communities, office buildings, and other locations, providing customers with 24-hour self-service pickup. These three types of delivery resources achieve intelligent collaboration through a multi-agent reinforcement learning collaborative scheduling system, working together to form an efficient delivery network.

[0060] Then, the execution results are input into the execution feedback optimization module 4. The execution feedback optimization module 4 calculates the reward value based on the execution results and distributes the reward value to each delivery agent using a reward distribution mechanism based on Nash social welfare.

[0061] The execution results include information such as the delivery completion status, delivery timeliness, customer satisfaction, and resource consumption of each delivery agent. Based on the execution results, four key indicators are calculated: delivery timeliness, customer satisfaction, resource utilization, and collaborative efficiency.

[0062] Delivery timeliness Defined as the ratio of actual delivery time to promised delivery time, the formula is as follows:

[0063] ,

[0064] in, For delivery timeliness, This represents the number of completed orders. For orders Actual delivery time (minutes). For orders The promised delivery time (in minutes). The shorter the delivery time, the more timely the delivery. Preferably, This indicates that the delivery timeliness is good.

[0065] Customer satisfaction The calculation formula is based on a combination of customer ratings and complaint rate:

[0066] ,

[0067] in, For customer satisfaction, For orders Customer ratings (1-5 points). The complaint rate is calculated as (number of complained orders / total orders). and This represents the weighting coefficient. Higher customer satisfaction indicates better service quality.

[0068] resource utilization rate Defined as the ratio of the average load to the maximum capacity of the delivery agent, the calculation formula is:

[0069] ,

[0070] in, For resource utilization, For the total number of delivery agents, For delivery intelligent agents The current load, For delivery intelligent agents The maximum capacity. Higher resource utilization indicates more efficient use of distribution resources. Preferably, This indicates that resource utilization is good.

[0071] Collaboration efficiency The efficiency improvement of multi-agent collaboration compared to independent work is measured by the following formula:

[0072] ,

[0073] in, For collaborative efficiency, For the total system revenue, For delivery intelligent agents The cost of working independently. Higher collaborative efficiency indicates greater gains from multi-agent collaboration.

[0074] The global reward value is calculated based on delivery timeliness, customer satisfaction, resource utilization, and collaboration efficiency. :

[0075] ,

[0076] in, This is the global reward value. , , , For the weighting coefficient, the preferred value is [value]. , , , The global reward value reflects the overall performance of the system; a higher value indicates that the delivery system is operating better.

[0077] A reward allocation mechanism based on Nash social welfare is adopted, decomposing the global reward value into individual reward values ​​for each delivery agent. Nash social welfare is defined as the sum of the logarithms of the individual reward values ​​of each delivery agent. The goal of reward allocation is to maximize Nash social welfare, thereby ensuring the fairness and incentive of reward distribution. Specifically, the optimization problem of reward allocation is as follows:

[0078] ,

[0079] Constraints:

[0080] ,

[0081] ,

[0082] in, For delivery intelligent agents The individual reward value.

[0083] The individual reward values ​​of each delivery agent are obtained by solving this optimization problem using the Lagrange multiplier method:

[0084] ,

[0085] in, For delivery intelligent agents The contribution of the delivery intelligent agent is defined as follows: The percentage of completed order value to total order value. For average contribution, The adjustment coefficient is preferably set to a value of [value to be filled in]. This reward distribution mechanism ensures basic fairness (each agent receives an equal reward). At the same time, appropriate adjustments will be made based on the contribution level to incentivize delivery agents to improve service quality and delivery efficiency.

[0086] The individual reward value is used as a feedback signal and input into the multi-agent policy generation module 2 to dynamically update the network parameters and policy parameters of the dynamic value decomposition network, forming a closed-loop feedback optimization mechanism.

[0087] The execution feedback optimization closed-loop mechanism comprises five stages: strategy execution, result feedback, reward calculation, reward allocation, and parameter update, forming a complete closed loop. The delivery agent executes delivery tasks according to the strategy parameters. The execution feedback optimization module 4 collects the execution results and calculates the global and individual reward values. The individual reward values ​​are then fed back to the multi-agent strategy generation module 2. The multi-agent strategy generation module 2 updates the network parameters of the dynamic value decomposition network based on the reward values, thereby optimizing the strategy parameters and enabling the delivery agent to make better decisions in subsequent delivery tasks.

[0088] The network parameters are updated using a temporal difference learning method. The temporal difference error is defined. for:

[0089] ,

[0090] in, For a moment The timing difference error, For a moment The global reward value, The preferred value for the discount factor is [value to be filled in]. , and They are time points and The global state vector, For a moment joint actions, For a moment Optional joint actions.

[0091] The gradient for updating network parameters is:

[0092] ,

[0093] ,

[0094] in, and The learning rate is preferably set to a value of [value to be filled in]. and , and They represent the parameters respectively. and Calculate the gradient. By using gradient descent, the time-series difference error is gradually reduced, resulting in more accurate value function estimation and better policy parameters.

[0095] During training, an experience playback mechanism is employed to improve learning efficiency and stability. The experience gained from each interaction is used to... The experience data is stored in an experience replay pool. Each time the network parameters are updated, a batch of experiences is randomly sampled from the experience replay pool for training. The capacity of the experience replay pool is set to 100,000 experiences, and the batch size for each sampling is 32. The experience replay mechanism breaks the temporal correlation between experiences, avoids oscillations and divergences during training, and improves the stability of learning.

[0096] Furthermore, this invention employs a target network mechanism to further enhance training stability. A target network with the same structure as the dynamic value decomposition network is constructed, and the parameters of the target network are... and When calculating the temporal difference error, the target network is used to calculate the target value:

[0097] ,

[0098] The parameters of the target network are changed every... Step to synchronize with current network parameters:

[0099] ,

[0100] Preferably, The introduction of the target network avoids frequent changes in the target value during training, thus improving training stability and convergence speed.

[0101] When a delivery agent encounters an anomaly while performing a delivery task, a dynamic rescheduling mechanism is triggered. Anomalies include unforeseen events such as drone malfunctions, delivery personnel illness, road closures, and severe weather. The specific steps of the dynamic rescheduling mechanism are as follows: Upon detecting an anomaly, the anomaly information and updated global state information are immediately input into the global state perception module 1; the global state perception module 1 regenerates the global state vector to reflect the current anomaly state; the multi-agent policy generation module 2 regenerates the individual value function and policy parameters of each delivery agent based on the new global state vector; the competition-cooperation scheduling module 3 re-allocates orders and schedules resources according to the new policy parameters, generating a new optimal policy combination; based on the new optimal policy combination, the delivery agent's delivery path is adjusted or orders are reassigned to ensure the smooth completion of the delivery task.

[0102] The dynamic rescheduling mechanism enables the system to cope with emergencies, ensuring the reliability and robustness of delivery tasks under abnormal conditions. In a preferred embodiment, the trigger delay for dynamic rescheduling is set to 5 seconds to ensure that the system can respond quickly to abnormal situations.

[0103] Delivery agents share local information through a communication mechanism to enhance their collaborative capabilities. The specific implementation of this communication mechanism is as follows: each delivery agent periodically broadcasts its own location information, current task information, and status information, preferably every 10 seconds; each delivery agent receives information from other delivery agents within its communication range, preferably with a radius of 3 kilometers; based on the received information, each delivery agent updates its local observation information. Through the attention mechanism, each delivery agent selectively focuses on information from other delivery agents that is highly relevant to its own task.

[0104] The attention mechanism is calculated using the following formula:

[0105] ,

[0106] in, For delivery intelligent agents For delivery intelligent agents Attention weights The relevance function measures the delivery agent. and The task relevance is assessed. The relevance function is implemented using a neural network, with inputs being local observations from two delivery agents and outputting a relevance score. Higher attention weights indicate stronger relevance for the delivery agent. Information for delivery smart agents The more important it is, the more intelligent the delivery system becomes. More attention will be paid to intelligent delivery agents The dynamics.

[0107] The strategy design for the intelligent agent of the smart parcel locker is as follows: based on the location information of the smart parcel locker... ,capacity and current usage rate Determine the order capacity that the smart parcel locker can receive. When orders mailing address The location is close to the smart parcel locker (the distance is less than the threshold). (meters) and the smart parcel locker has sufficient capacity ( When the smart locker receives an order, the smart locker's intelligent agent calculates the revenue generated from that order. ,in For smart parcel lockers to receive orders distance, To save time compared to direct delivery, and The weighting coefficient is used; if the return is positive, the smart locker's intelligent agent decides to accept the order and allocates it to the smart locker for storage; after the drone or delivery person delivers the order to the smart locker, the smart locker notifies the customer to pick up the package via SMS or APP; the customer picks up the package at the smart locker within 24 hours to complete the delivery process.

[0108] By using smart parcel lockers as transit points, asynchronous delivery between drones or delivery personnel and customers can be achieved. Drones or delivery personnel no longer need to wait for customers to be home; they can directly deliver orders to smart parcel lockers, significantly improving delivery efficiency. Customers can pick up their packages at convenient times, reducing time constraints and increasing customer satisfaction. The introduction of smart parcel lockers makes the delivery system more flexible and efficient, especially suitable for scenarios where customers are not home during the day.

[0109] like Figure 2 As shown, the system of the present invention includes four core modules: a global state perception module 1, a multi-agent policy generation module 2, a competition and cooperation scheduling module 3, and an execution feedback optimization module 4.

[0110] The global state perception module 1 is used to acquire delivery orders and global state information, extract features and encode the global state information to generate a global state vector. The global state perception module 1 includes a data acquisition unit, a feature extraction unit, and a state encoding unit. The data acquisition unit is responsible for collecting order distribution information, road condition information, weather information, and delivery resource status information in real time. The feature extraction unit preprocesses and extracts features from the collected raw data, including spatial gridding, temporal feature extraction, and statistical feature calculation. The state encoding unit fuses and vectorizes the extracted features to generate a unified global state vector.

[0111] The multi-agent policy generation module 2 takes the global state vector as input and, based on a dynamic value decomposition network, generates individual value functions and policy parameters for each delivery agent. The multi-agent policy generation module 2 includes an individual value network, a hybrid network, a global value network, and a policy generation unit. The individual value network constructs an independent deep Q-network for each delivery agent, calculating the individual Q-value based on local observation information and actions. The hybrid network generates a state-dependent hybrid weight matrix based on the global state vector. The global value network calculates the global value function based on the individual value functions and the hybrid weight matrix. The policy generation unit generates policy parameters for each delivery agent using a softmax function based on the individual value functions.

[0112] The competitive-cooperative scheduling module 3 takes strategy parameters as input and performs order allocation and resource scheduling based on the Nash equilibrium mechanism. Module 3 includes an order classification unit, a competitive game unit, a cooperative optimization unit, and an equilibrium solving unit. The order classification unit categorizes delivery orders into high-value orders and ordinary orders based on order priority and value. The competitive game unit constructs a competitive game model for high-value orders, calculates the bidding value of each delivery agent, and selects the optimal delivery agent. The cooperative optimization unit constructs a cooperative optimization model for ordinary orders and uses the Hungarian algorithm to achieve optimal matching between orders and delivery agents. The equilibrium solving unit solves for the Nash equilibrium points of the competitive game model and the cooperative optimization model, determining the optimal strategy combination for each delivery agent.

[0113] The execution feedback optimization module 4 executes delivery tasks based on the optimal strategy combination, calculates reward values ​​based on the execution results, and distributes these reward values ​​to each delivery agent using a Nash social welfare-based reward allocation mechanism. The reward values ​​are then fed back to the multi-agent strategy generation module 2, dynamically updating the network parameters and strategy parameters of the dynamic value decomposition network, forming a closed-loop feedback optimization mechanism. The execution feedback optimization module 4 includes a result collection unit, an indicator calculation unit, a reward calculation unit, a reward allocation unit, and a parameter update unit. The result collection unit collects the execution results from each delivery agent. The indicator calculation unit calculates four key indicators based on the execution results: delivery timeliness, customer satisfaction, resource utilization, and collaborative efficiency. The reward calculation unit calculates the global reward value based on the key indicators. The reward allocation unit decomposes the global reward value into individual reward values ​​for each delivery agent using a Nash social welfare-based reward allocation mechanism. The parameter update unit feeds back the individual reward values ​​to the multi-agent strategy generation module 2, updating the network parameters using a temporal difference learning method.

[0114] The four core modules are tightly coupled and form a closed-loop feedback mechanism. The global state awareness module 1 provides state input to the multi-agent policy generation module 2, which in turn provides policy parameters to the competition-cooperation scheduling module 3. The competition-cooperation scheduling module 3 outputs the optimal policy combination to guide the delivery agents in executing tasks. The execution feedback optimization module 4 feeds back reward values ​​to the multi-agent policy generation module 2 based on the execution results for parameter updates. This closed-loop feedback mechanism enables the system to continuously learn and adaptively optimize, allowing it to cope with dynamically changing delivery environments and maintain long-term stable high-performance operation.

[0115] In a specific application example, the collaborative scheduling system of this invention was deployed in a city's last-mile delivery system. This system manages 50 drones, 100 delivery personnel, and 200 smart parcel lockers, covering a 100-square-kilometer delivery area within the city. It processes 10,000-15,000 delivery orders daily. System operation data shows that after adopting the collaborative scheduling method of this invention, delivery timeliness improved by 25%, with the average delivery time decreasing from 45 minutes to 34 minutes; customer satisfaction increased by 18%, with customer ratings rising from 4.2 to 4.95; resource utilization improved by 30%, with the average load rate of drones and delivery personnel increasing from 60% to 78%; and the total system delivery cost decreased by 22%, with the cost per order decreasing from 8.5 yuan to 6.6 yuan. These data fully validate the technical advantages and practical value of this invention.

[0116] This invention can also be applied to various scenarios. In rural areas, drones can cover remote villages, delivery personnel handle main roads, and smart parcel lockers are placed in town centers to achieve efficient last-mile delivery coverage in rural areas. In industrial parks, drones can quickly deliver documents and small items, delivery personnel handle large goods, and smart parcel lockers are placed near factories to improve logistics efficiency within the park. In medical emergency scenarios, drones can quickly deliver medicines and blood, delivery personnel handle medical equipment, and smart parcel lockers serve as emergency supply storage points, building an efficient medical emergency logistics network.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for scheduling urban last-mile delivery based on multi-agent reinforcement learning, characterized in that, include: Obtain delivery orders and global status information, wherein the global status information includes order distribution information, road condition information, weather information, and delivery resource status information, and the delivery resource status information includes drone status, delivery person status, and smart parcel locker status; The delivery order and global status information are input into the global status perception module, and the global status information is subjected to feature extraction and status encoding to generate a global status vector. The global state vector is input into the multi-agent policy generation module. Based on the dynamic value decomposition network, individual value functions and policy parameters are generated for each delivery agent, including drone agents, delivery personnel agents, and smart parcel locker agents. The dynamic value decomposition network is constructed, comprising an individual value network, a hybrid network, and a global value network. The global state vector and local observation information of each delivery agent are input into the individual value network to generate individual value functions for each delivery agent. The global state vector is input into the hybrid network to generate state-dependent hybrid weights. Based on the individual value functions and the hybrid weights, the global value function is calculated through the global value network. The global value function is defined as follows: To determine the expected cumulative reward for all delivery agents taking joint actions under the current global state, an individual value function for each delivery agent is defined as the expected cumulative reward obtained by that agent taking actions under the current local observation. A state-dependent hybrid weight matrix is ​​generated by the hybrid network based on the global state vector. The individual value functions of each delivery agent are multiplied by the hybrid weight matrix and summed to obtain the global value function, which satisfies monotonicity constraints, ensuring that the globally optimal action corresponds to the locally optimal action of each delivery agent. Based on the individual value functions and the global value function, the network parameters of the dynamic value decomposition network are updated using a gradient descent method. Finally, the policy parameters of each delivery agent are generated using a softmax function based on the individual value functions. The strategy parameters are input into the competitive collaborative scheduling module, and order allocation and resource scheduling are performed based on the Nash equilibrium mechanism. Specifically: Delivery orders are divided into high-value orders and ordinary orders based on order priority and value; for high-value orders, a competitive game model is constructed, and each delivery agent calculates its bidding value based on the strategy parameters, selecting the delivery agent with the highest bidding value to execute the high-value order; for ordinary orders, a collaborative optimization model is constructed, comprehensively considering delivery distance, delivery time, resource utilization, and load balancing, and using the Hungarian algorithm to match orders with delivery agents; the Nash equilibrium points of the competitive game model and the collaborative optimization model are solved to determine the optimal strategy combination for each delivery agent, ensuring that no single change in strategy by any delivery agent in the current state will result in a higher return. The delivery task is executed according to the optimal strategy combination, and the execution result is input into the execution feedback optimization module. Based on the execution result, delivery timeliness, customer satisfaction, resource utilization, and collaborative efficiency are calculated. A global reward value is calculated based on the delivery timeliness, customer satisfaction, resource utilization, and collaborative efficiency. A reward distribution mechanism based on Nash social welfare is used to decompose the global reward value into individual reward values ​​for each delivery agent. Nash social welfare is defined as maximizing the sum of the logarithms of the individual reward values ​​of each delivery agent, ensuring the fairness and incentive of reward distribution. The individual reward values ​​are then used as feedback signals and input into the multi-agent strategy generation module. The reward value is fed back to the multi-agent policy generation module to dynamically update the network parameters of the dynamic value decomposition network and the policy parameters.

2. The method according to claim 1, characterized in that, The steps of the global state awareness module in extracting features and encoding the state information to generate a global state vector include: The order distribution information is spatially gridded, the delivery area is divided into grid units, and the order density, order priority and order time window of each grid unit are calculated. The road condition information is collected in real time to obtain the congestion index, travel time and travel cost of each road; The weather information is acquired to determine the impact coefficient of current weather conditions on drone flight and delivery personnel delivery. The status information of the delivery resources is collected to obtain the location information, availability status, remaining power or stamina value, and current task information of each delivery intelligent agent; The global state vector is generated by fusing and vectorizing the order density, order priority, order time window, congestion index, travel time, travel cost, impact coefficient, location information, availability status, remaining battery or stamina value, and current task information.

3. The method according to claim 1, characterized in that, The method further includes: When the delivery agent encounters an abnormal situation while performing a delivery task, a dynamic rescheduling mechanism is triggered; The abnormal situation information and the updated global status information are input into the global status perception module; Re-execute the multi-agent policy generation and the competitive cooperative scheduling to generate a new optimal policy combination; Based on the new optimal strategy combination, the delivery agent adjusts its delivery route or reassigns orders.

4. The method according to claim 1, characterized in that, The delivery agents share local information through a communication mechanism, which includes: Each delivery agent broadcasts its own location information, current task information, and status information; Each delivery agent receives information from other delivery agents within its communication range; Based on the received information, update the local observation information of each delivery agent; Through the attention mechanism, each delivery agent selectively focuses on information from other delivery agents that is highly relevant to its own task.

5. The method according to claim 1, characterized in that, The strategies of the intelligent express locker's intelligent agent include: Determine the order capacity that the smart parcel locker can receive based on its location, capacity, and usage rate. When the delivery address of an order is close to the location of a smart parcel locker and the smart parcel locker has sufficient capacity, the order will be assigned to the smart parcel locker for storage. By using the smart parcel locker as a transit node, asynchronous delivery between drones or delivery personnel and customers can be achieved, reducing the constraints of delivery time windows.

6. A multi-agent reinforcement learning-based urban last-mile delivery collaborative scheduling system, used to implement the method described in any one of claims 1-5, characterized in that, include: The global state perception module is used to acquire delivery orders and global state information, extract features and encode the global state information to generate a global state vector. The global state information includes order distribution information, road condition information, weather information and delivery resource status information. The delivery resource status information includes drone status, deliveryman status and smart express cabinet status. A multi-agent policy generation module is used to take the global state vector as input and generate individual value functions and policy parameters for each delivery agent based on a dynamic value decomposition network. The delivery agents include drone agents, delivery personnel agents, and smart parcel locker agents. The dynamic value decomposition network includes an individual value network, a hybrid network, and a global value network. The individual value network generates individual value functions based on the local observation information of each delivery agent. The hybrid network generates a state-dependent hybrid weight matrix based on the global state vector. The global value network multiplies the individual value functions of each delivery agent with the hybrid weight matrix and sums them to obtain the global value function, satisfying monotonicity constraints. The module also generates policy parameters for each delivery agent using a softmax function. The competitive and collaborative scheduling module is used to take the strategy parameters as input and perform order allocation and resource scheduling based on the Nash equilibrium mechanism. Specifically, delivery orders are divided into high-value orders and ordinary orders according to order priority and order value. For high-value orders, a competitive game model is constructed to select the delivery agent with the highest bid value. For ordinary orders, a collaborative optimization model is constructed and the Hungarian algorithm is used for matching. The optimal strategy combination for each delivery agent is determined by solving the Nash equilibrium point. The execution feedback optimization module is used to execute delivery tasks according to the optimal strategy combination, calculate delivery timeliness, customer satisfaction, resource utilization, and collaborative efficiency based on the execution results, calculate the global reward value, and decompose the global reward value into individual reward values ​​for each delivery agent using a reward distribution mechanism based on Nash social welfare. The Nash social welfare is defined as maximizing the sum of the logarithms of the individual reward values ​​of each delivery agent. The reward value is then fed back to the multi-agent strategy generation module to dynamically update the network parameters of the dynamic value decomposition network and the strategy parameters.

Images