Rural area truck and unmanned aerial vehicle cooperative distribution path planning and resource scheduling method

Abstract

The invention discloses a rural area truck and unmanned aerial vehicle cooperative distribution path planning and resource scheduling method, and belongs to the technical field of path planning. According to the method, a multi-target truck-unmanned aerial vehicle path problem with a flexible time window is modeled as a reward maximization problem in deep reinforcement learning; a time window is allowed to be deviated within tolerance time, and punishment is applied to serious deviation; a multi-agent model is adopted, a distribution unit composed of a truck and a carried unmanned aerial vehicle is regarded as an agent, space-time topological features of a complex logistics network are embedded and captured through a multi-order space-time diagram convolutional network, the attention mechanism of the multi-order space-time diagram convolutional network is utilized to perform weighted summation on adjacent node features to generate specific weights of client nodes, and the state sensing ability of the agent is enhanced; and based on a multi-agent near-end strategy optimization algorithm, the collaborative distribution path of the multiple truck-unmanned aerial vehicle units is optimized, and the parallel processing method is combined with ST-MGCN embedding, so that the action space search dimension is effectively reduced, and the network convergence speed and the solving efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of path planning, and in particular relates to a truck and unmanned aerial vehicle (UAV) cooperative distribution path planning and resource scheduling method for rural areas. BACKGROUND

[0002] With the rapid popularization of e-commerce in rural areas, the demand for terminal logistics distribution has shown explosive growth. However, the rural areas are generally characterized by complex geographical conditions, sparse road network and scattered population distribution, which leads to low efficiency, long distribution cycle and high cost of traditional transportation, seriously restricting customer experience. To address this challenge, the truck and UAV cooperative distribution mode has emerged, which breaks through the limitations of single carrier by integrating the advantages of ground transportation and air delivery. In the truck and UAV cooperative distribution mode, the truck acts as a mobile warehouse and can carry multiple UAVs for dynamic scheduling: the truck relies on large carrying capacity to complete the main line transportation, and the UAV uses the three-dimensional space maneuverability to quickly cover complex terrain areas such as mountains and rivers, significantly improving the distribution network coverage. Compared with the traditional pure truck or pure UAV scheme, this cooperative system has both transportation economy and timeliness advantages, and has become a research hotspot in the field of intelligent logistics.

[0003] Although some progress has been made in existing research, there are still significant technical bottlenecks in existing solutions: first, the spatial span of rural distribution scenarios is large and the customer nodes are scattered, making it difficult for a single truck to carry a UAV group to achieve wide-area coverage due to limited endurance and cargo capacity. Existing models assume that the truck needs to stop near the customer node to launch the UAV, resulting in high redundancy of transportation paths and insufficient dynamic response capability; second, customer satisfaction is a key indicator of service quality, and existing research lacks in-depth exploration of quantitative modeling and multi-objective optimization mechanisms, making it difficult to effectively balance the distribution timeliness, cost control and service quality; third, the rural logistics network has strong space-time coupling and high complexity of multi-agent collaboration, and traditional single-agent deep reinforcement learning (DRL) methods are prone to strategy convergence difficulties when dealing with high-dimensional state-action space, while multi-agent DRL faces challenges such as communication delay and incomplete local observation information, severely restricting the efficiency of collaborative decision-making.

[0004] In summary, there is an urgent need to design a truck-UAV cooperative distribution optimization method for complex rural scenarios to overcome the limitations of existing technology in dynamic path planning, multi-objective resource scheduling and agent collaborative control. SUMMARY

[0005] The present application aims to provide a truck and unmanned aerial vehicle (UAV) cooperative distribution path planning and resource scheduling method for rural areas, which solves the following problems in the prior art: I. The rural distribution scene has a large spatial span and scattered customer nodes. The single truck-mounted drone group is limited by the endurance and cargo capacity, and it is difficult to achieve wide-area coverage. Existing models assume that the truck needs to stop near the customer node to launch the drone, resulting in high redundancy of transportation paths and insufficient dynamic response capability. II. Customer satisfaction, as the core indicator of service quality, lacks in-depth exploration of quantitative modeling and multi-objective optimization mechanism in existing research, making it difficult to effectively balance the delivery timeliness, cost control and service quality. III. The rural logistics network has strong space-time coupling and high complexity of multi-agent collaboration. Traditional single-agent deep reinforcement learning (DRL) method is prone to strategy convergence difficulty when dealing with high-dimensional state-action space, while multi-agent DRL faces challenges such as communication delay and incomplete local observation information, which seriously restricts the efficiency of collaborative decision-making.

[0006] The purpose of the present application can be achieved by the following technical solutions: The rural truck and drone collaborative distribution path planning and resource scheduling method comprises the following steps: Step 1: Initialize actor network parameters, critic network parameters, learning rate, discount factor and other hyperparameters according to the parameter table; Step 2: Empty the global experience pool and the local experience pool of each agent; Specifically, the global experience pool can be emptied periodically during training, and the local experience pool of each agent can be emptied simultaneously; Step 3: Determine whether the training sequence number i is greater than the maximum training sequence number. If it exceeds, the algorithm ends, otherwise go to step 4; Specific implementation: In the training process, determine whether the current training sequence number i is greater than the preset maximum training sequence number. If it exceeds, terminate the algorithm running, otherwise jump to step 4 for continuous execution.

[0007] Step 4: Determine whether the cycle sequence number j is greater than the maximum cycle sequence number. If yes, set i=i+1, j=0, and jump to the previous step, otherwise go to step 5; Step 5: Determine whether the agent sequence number k is greater than the maximum agent sequence number. If yes, set k=0, j=j+1, and jump to step 15, otherwise go to step 6; Step 6: The actor network outputs the actions that the current drone and truck need to perform based on the observed environment state and node weight vector. Each group of trucks and drones forms an agent that learns by combining the node weight information output by the multi-order spatio-temporal graph convolution network and the dynamic delivery environment state. After the feature vector is input, the system performs network inference, uses a mask matrix to mask invalid actions with a value of 0 in the final output network, and then normalizes the remaining valid actions to the [0, 1] interval according to the mask and function network probability distribution. Finally, through the mask rule, the unmasked actions are subjected to a softmax operation, and the complete action decision distribution is output. The mask initialization process is as follows: First, two key lists and are constructed to represent the action masks of drones and trucks, respectively, where represents the observation at time t. The length of these two lists is determined by the actual action dimension to ensure that all possible action options are covered. Subsequently, all elements in the two lists are initialized to 1, which means that in the initial state, all actions are feasible.

[0008] ; ; Step 7: The drone and truck perform the action to obtain the next observation and reward value. Step 8: Update the multi-order spatio-temporal graph convolution network to obtain the next customer node weight vector and perform spatial feature aggregation as follows: ; Next, the gating multi-scale temporal feature modeling is performed to update the gate and reset gate to inject the residual connection of TCN and enhance gradient flow: ; The candidate state introduces the output of the time convolution to improve the modeling of time-dependent sequences: ; Finally, the gated adaptive spatio-temporal feature fusion is performed: ; Step 9: Store the current observation and reward value and the next observation and reward value in the local buffer pool Step 10: Deliver the delivery information to other agents to prevent repeated delivery Step 11: Check if the number of buffer pool elements is greater than the total number of customer nodes. If yes, go to step 12, otherwise go to step 5 Step 12: Randomly sample a small batch of data from the local experience pool and the global experience pool. Step 13: Calculate the advantage function, which quantifies the degree of cumulative reward compared to the expected global reward. The formula for calculating the advantage function is: ; This formula is used to calculate the advantage function estimate value at time step . Wherein, is the immediate reward obtained at the next time step t + 1; and are the estimated state values of the next state and the current state ; is the discount factor, which measures the importance of future rewards relative to current rewards.

[0009] Step 14: Update the parameters of the Actor network and the Critic network using the Adam optimizer for effective optimization strategy and ensure stable reinforcement learning training. Importance sampling can be used to reweight historical data, and the performance gap of the old strategy under the new strategy can be evaluated. The gradient estimation formula at this time is: ; Where represents the mean of the limited batch experience. To make each update amplitude more smooth and reliable, the target function can be clipped, and the clipped target function formula is: ; Where A t is the advantage function, is the clipping hyperparameter, represents the mean of the limited batch experience.

[0010] The target function is processed through this formula to optimize the parameter update process. Where is a pre-set hyperparameter used to define the clipping range. After updating the policy network, the value network also needs to be updated. The local replay buffer corresponds to the trajectory data retrieved from the global replay buffer. The main optimization goal for adjusting the value network parameters is: ; Where V ϕ (s t ) is the estimated value of state st, r t is the immediate reward, and T is the total number of time steps.

[0011] By using the Adam optimizer, the network parameters can be efficiently updated using the stochastic gradient descent method; Step 15: Determine whether the agent number k is greater than the maximum agent number. If yes, jump to step 16, otherwise let k = k + 1, and jump to step 6; Step 16: Store the action, state at this time, reward at the next time, and state into the global experience pool; Step 17: Let k = k + 1, and jump to step 5; In the final output network, the remaining actions are reset to the range [0, 1] according to the mask and the probability distribution of the function network. Then, the action is output by the mask rule , the softmax operation is performed on the action not masked, and the complete action is output. The decision distribution of the output action can be calculated by the formula: ; ; , which is used to calculate the probability of taking action when given observation . The numerator is the exponential probability value of action based on the mask rule and observation , and the denominator is the sum of the exponential probability values of all possible actions based on the same conditions. In this way, the probability distribution of the action is determined to assist decision-making; To evaluate the performance gap of the old policy under the new policy, the historical data is reweighted by importance sampling. The gradient estimation can be represented by the formula: .

[0012] , where represents the expected value of sampling from the experience pool based on policy , is the ratio of the action probability under the new policy to the old policy, is the advantage function, which estimates the gradient of the policy network parameter in this way to optimize the policy.

[0013] To make the magnitude of each update more smooth and reliable, the target function can be clipped. The clipped target function is represented by the formula: . Here, is still the average value of the experience sampled from the experience pool based on policy , is the ratio of the action probability under the new policy to the old policy, is the advantage function, which limits the parameter update magnitude by taking the smaller value, making the update of the policy network parameter more stable.

[0014] After updating the policy network, the value network also needs to be updated. The local replay buffer corresponds to the trajectory data retrieved from the global replay buffer. The main optimization objective for adjusting the value network parameters is given by the formula: This is represented here, by calculating the value network for the state. Valuation With instant rewards The sum of squared differences, averaged, measures the deviation between the estimated and actual rewards, with the goal of minimizing this deviation. The Adam optimizer is employed, using stochastic gradient descent to efficiently update network parameters. After sufficient training, these models can be used in distributed execution phases. Furthermore, the learning rate is continuously varied to fine-tune the network weights.

[0015] The beneficial effects of this invention are: The scheme disclosed herein models the multi-objective truck-drone vehicle routing problem with flexible time windows as a reward maximization problem in deep reinforcement learning, allowing deviations from the time window (early or late delivery) within a tolerance period and penalizing severe deviations. A multi-agent model is employed, treating the delivery unit consisting of trucks and their drones as agents. A multi-level spatiotemporal graph convolutional network (ST-MGCN) embedding is used to capture the spatiotemporal topological features of the complex logistics network. Its attention mechanism is utilized to generate specific weights for customer nodes by weighted summation of adjacent node features, enhancing the agents' state awareness capabilities. Furthermore, the multi-agent near-end policy optimization (MAPPO) algorithm is used to optimize the collaborative delivery path of multiple truck-drone groups. This parallel processing method, combined with ST-MGCN embedding, effectively reduces the dimensionality of the action space search, improving network convergence speed and solution efficiency. Attached Figure Description

[0016] The invention will now be further described with reference to the accompanying drawings.

[0017] Figure 1 This is a flowchart illustrating the path planning and resource scheduling method described in this invention; Figure 2 This is a flowchart illustrating another form of the path planning and resource scheduling method described in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] A method for truck and UAV cooperative delivery path planning and resource scheduling in rural areas, as shown in Figure 1 、 Figure 2 includes the following steps: Step 1: According to the parameter table, initialize the actor network parameters, critic network parameters, and learning rate, discount factor and other related hyperparameters step by step; Step 2: Periodically empty the global experience pool during training, and simultaneously empty the local experience pool of each agent; Step 3: Determine whether the current training sequence number i is greater than the preset maximum training sequence number during training. If it exceeds, terminate the algorithm operation, otherwise jump to step 4 for execution.

[0020] Step 4: Determine whether the current cycle sequence number j is greater than the preset maximum cycle sequence number during training. If it exceeds, reset i=i+1 and j=0 and jump to the previous step, otherwise directly jump to step 5 for execution; Step 5: Determine whether the current agent sequence number k is greater than the set maximum agent sequence number during algorithm execution. If it exceeds, reset k=0 and j=j+1 and jump to step 15, otherwise directly jump to step 6 for operation; Step 6: The actor network outputs the actions to be performed by the current UAV and truck according to the observed environment state and node weight vector. Each group of trucks and UAVs forms an agent that learns from the node weight information output by the multi-order spatiotemporal graph convolutional network (ST-MGCN) and the dynamic delivery environment state. After the feature vector is input, the system performs network inference, uses a mask matrix to mask invalid actions with a value of 0 in the final output network, normalizes the remaining valid actions to the [0,1] interval according to the mask and the probability distribution of the function network, and finally performs a softmax operation on the unmasked actions to output a complete action decision distribution.

[0021] The mask initialization process is as follows: First, construct two key lists m ui and m ηi , which are used to represent the action masks of UAVs and trucks, respectively, where O t represents the observation value at time t. The length of these two lists is determined according to the actual action dimension to ensure that all possible action options are covered. Subsequently, all elements in the two lists are initialized to 1, which means that in the initial state, all actions are feasible; ; ; Step 7: After the UAV and truck cooperatively execute the current action, the next time environment observation value and corresponding reward value are obtained in real time; Step 8: Update the multi-order spatiotemporal graph convolutional network to obtain the client node weight vector at the next time step. First, through a learnable graph structure parameterization module, adaptively infer the k-order neighbor relationship at each time step (instead of relying on a predefined graph) to handle the dynamic changes in the rural network topology. Specifically, the traditional attention weight calculation is upgraded to a dynamic graph learning process. First, through a learnable graph structure parameterization module, the k-order neighbor relationships at each time step are adaptively inferred (rather than relying on a predefined graph), thus handling the dynamic changes in rural network topology. The attention weight calculation is as follows: ; Here, fθ is a lightweight neural network, introducing a time interval Δt. ij As dynamic edge weights, they capture the impact of delivery timeliness on spatial correlation. Spatial feature aggregation enhances representation capabilities through a multi-head attention mechanism. ; in, This represents the attention weight between nodes i and j. The aggregated spatial features are represented by σ, which is the activation function, and W is the spatial feature. k This is the weight matrix. and For node features, Δt ij For time intervals; Here, M represents the number of attention heads, which improves the capture of sparse long-distance connections (such as indirect connections in remote villages) through multi-view aggregation. Gated Multi-Scale Temporal Feature Modeling: To avoid the conventionality of GRU, a hybrid temporal model is constructed by incorporating the gating mechanism of Temporal Convolutional Networks (TCNs). This model captures long-term temporal dependencies through dilated convolutions while utilizing the gating mechanism of GRUs to control short-term dynamics. Update and reset gates are injected into the residual connections of the TCN to enhance gradient flow: ; Introducing candidate states into the output of temporal convolution improves the modeling of time-sensitive sequences: ; Final time characteristics By balancing historical information with current inputs through gating, the uncertainty of delivery time can be effectively addressed.

[0022] Gated adaptive spatiotemporal feature fusion: This method replaces simple linear fusion with learnable gating units, dynamically adjusting the contributions of spatial, temporal, and historical features. The fusion formula is as follows: ; Among them, gating weights and Dynamically generated based on the current context, ensuring that the fusion process adapts to sparse scenarios (such as spatial features dominating at certain times, while historical features are more critical at other times). Step 9: Store the observation values ​​and reward values ​​of the current time and the next time into the local buffer pool; Step 10: Synchronize real-time delivery information with other intelligent agents to avoid duplicate deliveries; Step 11: Check if the number of elements in the buffer pool exceeds the total number of client nodes. If it does, proceed to step 12; otherwise, proceed to step 5. Step 12: Randomly draw small batches of data samples from the local experience pool and the global experience pool; Step 13: Calculate the dominance function, which quantifies the degree of cumulative reward compared to the expected global reward. The formula for calculating the dominance function is: ; This formula is used to calculate the time step. Advantage function estimate .in, It is the instant reward obtained at the next time step t + 1; and The next state and current state The estimated state value; It is a discount factor used to measure the importance of future rewards relative to current rewards.

[0023] Step 14: Update the parameters of the Actor network and the Critic network using the Adam optimizer to achieve an effective optimization strategy and ensure stable reinforcement learning training. Importance sampling can be used to reweight historical data and evaluate the performance difference between the old and new strategies. At this point, the gradient estimation formula is: ; in This represents the empirical mean of a finite number of batches. To ensure that each... The update amplitude is smoother and more reliable, and the objective function can be pruned. The formula for the pruned objective function is as follows: ; This formula is used to process the objective function, thereby optimizing the parameter update process. These are pre-defined hyperparameters used to define the pruning range. The value network also needs to be updated after the policy network is updated. The local playback buffer corresponds to the trajectory data retrieved from the global playback buffer. The main optimization objective for adjusting the value network parameters is: ; By employing the Adam optimizer, network parameters can be updated efficiently using stochastic gradient descent. Step 15: During the algorithm execution, determine whether the current agent number k exceeds the set maximum agent number. If it does, jump directly to step 16; otherwise, execute k=k+1 and then jump to step 6. Step 16: Store the current action, state, and reward for the next moment in the global experience pool; Step 17: Let k = k + 1, then skip to step 5.

[0024] In the context of joint delivery, customer satisfaction refers to the degree of satisfaction customers have with the delivery service, particularly in terms of whether the arrival time of goods meets their expectations. When defining customer satisfaction, the following key factors should be considered: First, given the special nature of rural logistics delivery scenarios, farmers typically work outdoors every day, and receiving packages at home within the expected timeframe best meets their expectations and brings a high level of satisfaction.

[0025] Therefore, if the goods arrive within the time interval expected by the farmer [m] i ,n i If customer satisfaction is set as a constant, it is usually related to the customer's level of importance placed on delivery service, reflecting the customer's expectations for on-time delivery and satisfaction. Secondly, farmers' work is characterized by seasonality and time uncertainty. Their departure and return times may vary depending on the specific agricultural activities they engage in. Therefore, setting a tolerance period is an effective measure.

[0026] Allowing package delivery times to deviate from the expected time window within a predetermined tolerance range can significantly increase the probability of farmers picking up packages at their doorsteps, thereby effectively optimizing service quality. When the arrival time is less than or greater than the expected time, customer satisfaction linearly decreases to 0 as the deviation from the expected time increases. Therefore, we set the formulas for calculating the earliest arrival time, latest arrival time, and allowable time tolerance of packages as follows: ; Furthermore, if packages arrive early, some packages may have time-sensitive requirements (such as fresh food), which could negatively impact farmer satisfaction. However, for some packages, early delivery might provide a positive experience for farmers. Therefore, if a package arrives before the earliest possible time, we set the customer satisfaction level to 0. If the goods arrive so late that they exceed the latest possible time, customer satisfaction becomes negative and decreases linearly at a certain rate. This reflects the extreme level of customer dissatisfaction with significant deviations from the expected arrival time.

[0027] Therefore, customer satisfaction at the time point can be represented by a function: ; Overall customer satisfaction can be calculated as follows: ; Considering the characteristic conditions, the constraints are as follows: ; Constraint (7) ensures that each customer node is accessed only once to avoid duplicate deliveries or missed customers. Constraints (8) and (9) ensure that deliveries always begin and end at the warehouse and can only begin and return from the warehouse once. Each drone is limited to being launched and retrieved from each node at most once in constraints (10) and (11).

[0028] ; Constraint (12) limits the time a drone can deliver goods to the maximum range per trip. Constraints (13)-(15) ensure that the order in which trucks visit nodes must follow the specified route. Constraint (16) ensures that both nodes can be served. It requires trucks to travel from node to node in constraint (17). Constraint (18) states that if a drone takes off from the warehouse and returns from a node, the truck should move to that node, strictly adhering to the prescribed route.

[0029] ; Constraint (19) ensures that if an unmanned aerial vehicle (UAV) lands at a node, the truck is not permitted to leave that node until the UAV is recovered, thus ensuring the safe landing and recovery of the UAV. Constraint (20) stipulates that if an UAV is to take off from a node, the truck is not permitted to leave that node before the UAV takes off. Constraint (21) guarantees that the truck leaves the same customer location at any time it is supposed to arrive at a node.

[0030] The time before trucks and drones reach the next customer node should be greater than the time they leave their current location under constraints (22) and (23) to ensure that the mission proceeds as planned.

[0031] ; Finally, constraints (24) and (25) require that each group of trucks or drones must immediately proceed to the next node after visiting one node, without stalling or ineffective waiting. The time it takes for a truck to reach the last customer is defined as the final delivery time. Therefore, the second objective function is as follows: ; And it is necessary to minimize delivery costs, c t and c u Let the hourly labor costs for trucks and drones be represented respectively. The formula for the third objective function is: ; To minimize the distance, the fourth objective function formula is: ; The objective function is to minimize delivery time and distribution costs while maximizing overall customer satisfaction. To address the multi-objective optimization challenge in coordinating truck-drone delivery, the problem is transformed into a weighted sum problem and solved using different weighting coefficients. In subsequent experiments, we quantitatively evaluated the relative importance of the three functions mentioned above to obtain a Pareto optimal solution. In summary, the overall objective function is designed as follows: ; To apply reinforcement learning to this problem, it needs to be mathematically formulated. Markov Decision Processes (MDPs) play a crucial role in reinforcement learning, providing a fundamental framework for the interaction between agents and their environment.

[0032] It can be represented by a tuple <S,A,P,R>, where: represents the state space, containing all states that the agent can perceive and control; represents the action space, containing all possible actions that can be taken in any given state; represents the state transition probability, defining the probability distribution of transitioning from state to state when an action is taken; and is the reward function R(s,a, s')→R, which evaluates the quality of the action taken in the current state by providing a positive reward or a negative penalty, thus leading to a new state.

[0033] The core of reinforcement learning aims to determine an optimal policy function π(a|s) to achieve the best possible outcome.

[0034] 1) Agent: Each group of trucks and its accompanying drones is an Agent. Each Agent selects actions, transmits information to other Agents, interacts with the surrounding environment, and learns to transmit path strategies.

[0035] 2) State space: State information represents the overall distribution state of the network, and is divided into client state information and time state information.

[0036] Customer status information includes visited customer locations: C v Current customer locations served by drones: C u And customer points served by trucks: C η This reflects the interaction between the agent and the client during task execution.

[0037] Time status information includes timestamps of drone and truck arrival at the customer point: T u T η And the maximum remaining flight time of the drone: Fmax These time-related factors are crucial for task execution. The state can be represented as a tuple: ; 3) Action Space: Consists of the customer nodes for the vehicle's next service. We designed n+1 actions corresponding to the customer and the warehouse: a t ∈ A = {0, 1, ..., n}, where when a t = i, i > 0 indicates that package delivery is performed for the customer, when a t =0 means that all customers have received service and returned to the warehouse.

[0038] 4) State transition function: represents the state transition function of the agent from (s) t ,a t ,r t ) environmental state to s t+1 The change is usually expressed by the conditional probability density P(S′=s t+1 |S = s t A = a t ). represents the randomness of transitioning from one state to the next.

[0039] As mentioned earlier, customer status information and time status information are dynamically changing. The transition process from the current step to the next step is as follows: Drone Changes: The agent selects an unvisited customer node and travels to its coordinates to complete the delivery task. Time expenditure includes delivery time by truck or drone. Upon arrival at the customer's location, the drone immediately unloads the package into a delivery box and promptly fulfills the customer's request. Furthermore, the drone adjusts its remaining flight time to... Where ΔT represents the delivery time from the previous location to this location. Upon returning to the truck, the drone resets its maximum scheduled flight duration.

[0040] The arrival time for this customer has now changed, and the visited customer node has been added to the current access list. It also updates the arrival time for the customer node. The drone updates the customer point it is currently serving to the next node to be visited.

[0041] Truck Changes: When a truck arrives at a customer node, it immediately fulfills the customer's request and updates its arrival time to t+ΔT, where t represents the time it arrived at the previous customer. The visited customer node is added to the current visit list. The truck then updates the next customer node it is serving to the list of nodes to visit.

[0042] 5) Reward Function: This reflects the immediate reward an agent receives after performing an action in a specific state, providing feedback to the agent to evaluate and improve its optimal strategy. The reward function needs to consider the following factors: As shown in Equation 1, to maximize customer satisfaction, we minimize customer delivery time as much as possible. However, considering the navigation time limitations of drones, we require that the delivery time of each drone should not exceed its maximum flight time.

[0043] t represents the time when the agent arrives at the client node, T r represents a constant, and represents the remaining flight time of the drone.

[0044] ; If the maximum flight time is exceeded, a penalty value will be applied.

[0045] In summary, instant reward is defined as: ; The weighted variable λ is used to adjust for the influence of different factors, and Tref is the reference delivery time.

[0046] The complexities of collaborative delivery involving multiple agents across multiple trucks and drones make it difficult to explore optimal policies using simple reinforcement learning (RL) methods. MAPPO is a deep reinforcement learning (DRL) algorithm based on a Markov decision process model with local states. To avoid repetitive visits, we need to ensure that all agents in the environment interact with each other so that other agents know the customer points being visited. Therefore, we combine centralized training with distributed execution and leverage parameterized policies and approximate value functions. The basic concepts of MAPPO involve agents interacting with the environment to obtain feedback rewards, learning action policies, and value functions.

[0047] Then, empirical iterations are used to fine-tune the parameters to improve the performance of drone and truck transportation and optimize routes. Each agent consists of a participant network and a critic network. The critic (value network) uses states or state-action pairs to evaluate the theoretical payoff for the participant's current action. The participant (policy network) generates a probability distribution of actions based on its current state to guide the decision-making process. It uses backpropagation to update its parameters using policy gradients and the critic's value estimate. These two components work together, optimizing alternately until a predefined performance metric or training step size limit is reached.

[0048] The MAPPO algorithm consists of two stages. Design details are as follows.

[0049] During the intensive training phase, each truck and drone group needs to learn by combining the node weight information output by the multi-level spatiotemporal graph convolutional network (ST-MGCN) with the current state of the delivery environment. After inputting the feature vector, network inference is performed. Actions with a mask value of 0 are masked in the final output network. Based on the mask and the probability distribution of the function network, the remaining actions are reset to the range [0, 1]. Finally, the masking rules are applied. For the unmasked actions, a softmax operation is performed, and the complete action is output. The decision distribution of the output action can be represented as:

[0050] An experience replay mechanism is employed to improve learning efficiency and achieve the optimal policy. The experience data of all agents is aggregated and stored in a global replay buffer. In, there are tuples ,in Indicates the state at time t. Indicates the action at time t. This represents the reward received by the intelligent agent. This represents the subsequent state. Updating the neural network by randomly sampling a small batch of data from the global replay buffer D reduces the correlation between consecutive observations and improves learning efficiency. Simultaneously, the evaluation network within each agent assesses the overall state information to ensure coordination among agents. In this case, the advantage function quantifies the cumulative reward compared to the expected global reward. The formula for calculating the advantage function is: ; in Indicates at time The instant reward received. and These are for the next state and current state The estimated state value. This is the discount factor. To effectively optimize the policy and ensure stable reinforcement learning training, we can utilize historical data to evaluate the performance gap between the old policy and the new policy. This is achieved by reweighting the historical data using importance sampling. The gradient estimate can be expressed as: ; in This represents the empirical mean of a finite number of batches. To ensure that each... The update amplitude is smoother and more reliable, and the objective function can be pruned, which can be expressed as: ; After updating the policy network, the value network also needs to be updated. The local replay buffer corresponds to the trajectory data retrieved from the global replay buffer. The main optimization objective for adjusting the value network parameters is: ; The Adam optimizer is employed to efficiently update network parameters via stochastic gradient descent. After sufficient training, these models can be deployed in a distributed execution phase. Furthermore, the learning rate is continuously varied to fine-tune these network weights.

[0051] The distributed execution phase. Here, each agent utilizes a trained network of participants to interact with the environment based on local observations and weight vectors. This process generates specific instructions for drones and trucks and gathers experiential data from the environment. Initially, all agents reside in the warehouse and dynamically update their interactions based on instructions output by the participant network. Each agent executes actions and shares customer access information with others in a timely manner to avoid redundant access to the same customer nodes. If an agent's observations do not meet constraints, they are masked through a masking mechanism. This approach ensures synchronization and consistency among agents, preventing system crashes due to improper behavior by individual agents.

[0052] In a multi-objective truck-drone vehicle routing problem with time windows, each truck travels at a speed of 9 m / s, and each drone has a maximum flight time of 0.4 hours and a flight speed twice that of the truck. The warehouse is located at the origin [0,0]. The weight of each package is randomly generated within the range of 0 to 2 kg. If a package weighs more than 1 kg, it must be transported by truck. Time windows are randomly generated from 0 to 200 minutes based on the expected delivery time and the corresponding number of customers. Furthermore, a two-layer multi-level spatiotemporal graph convolutional network (ST-MGCN) is employed. In the multi-agent system, we choose the same 4-layer fully connected neural network architecture for the policy network and value network of each agent. The number of nodes in each layer of this neural network is set to 128, 512, 128, and 64, respectively.

[0053] For details regarding the simulation process and other parameters of the reinforcement learning model, please refer to Tables 1 through 4.

[0054] Table 1, Set Element List:

[0055] Table 2, Model Parameter Table:

[0056] Table 3, Variable Table:

[0057] Table 4, Parameter Table:

[0058] The above description is merely an example and illustration of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

Claims

1. A delivery route optimization method based on multi-agent reinforcement learning, characterized by, The method comprises the following steps: (a) initializing actor network parameters, critic network parameters, a learning rate, and a discount factor; (b) emptying a global experience pool and a local experience pool of each agent; (c) determining whether a training sequence number i is greater than a maximum training sequence number, and if so, ending the algorithm, otherwise performing step (d); (d) determining whether a cycle sequence number j is greater than a maximum cycle sequence number, and if so, setting i = i + 1, j = 0, and returning to step (c), otherwise performing step (e); (e) determining whether an agent sequence number k is greater than a preset maximum agent sequence number, and if so, setting k = 0, j = j + 1, and performing step (i), otherwise performing step (f); (f) the actor network outputs actions of the unmanned aerial vehicle and the truck according to a current environment state and a node weight vector, wherein invalid actions are shielded by a mask matrix, and a softmax operation is performed on valid actions to generate an action probability distribution; (g) performing the actions to obtain observation values and reward values at a next time; (h) updating a multi-order spatio-temporal graph convolution network to obtain a node weight vector at the next time, comprising spatial feature aggregation, gated multi-scale time feature modeling, and gated adaptive spatio-temporal feature fusion; (i) storing the observation values and the reward values at the current time, and the observation values and the reward values at the next time into the local experience pool; (j) passing delivery information to other agents to prevent repeated delivery; (k) determining whether a number of elements in the local experience pool is greater than a total number of customer nodes, and if so, performing step (i), otherwise returning to step (e); (i) randomly sampling a small batch of data from the local experience pool and the global experience pool; (m) calculating an advantage function based on a current reward, a discount factor, and a state value function of a current state and a next state; (n) updating the actor network parameters using a clipped target function, and updating the critic network parameters using an Adam optimizer; (o) storing the actions, the states, the reward at the next time, and the states into the global experience pool; (p) setting k = k + 1, and returning to step (e). 2.The multi-agent reinforcement learning based delivery route optimization method according to claim 1, wherein, The initialization process of the mask matrix in step (f) comprises: Constructing a drone action mask list m ui and truck action mask list m ηi where all elements are initialized to 1, indicating that all actions are feasible states; according to the observation value Ot, use the mask matrix to mask the invalid actions with a value of 0, and normalize the probability distribution of the remaining valid actions to the interval [0, 1]; through the mask rule, softmax operation is performed on the actions that are not masked, and the complete action decision distribution is output. 3.The multi-agent reinforcement learning based delivery route optimization method according to claim 1, wherein, The spatial feature aggregation in step (h) is achieved by the following formula: ； wherein, denotes the attention weight between nodes i and j, denotes the aggregated spatial features, σ is an activation function, W k is a weight matrix, and is a node feature, Δt ij is a time interval, M denotes the number of attention heads. 4.The multi-agent reinforcement learning based delivery route optimization method according to claim 1, wherein, The gated multi-scale time feature modeling in step (h) comprises calculation of an update gate, a reset gate, and a candidate state, and is achieved by the following formula: ； ； wherein, is an update gate, is a reset gate, is a candidate state, σ g is a sigmoid function, TCN denotes a temporal convolutional network, and denotes an element-wise multiplication. 5.The multi-agent reinforcement learning based delivery route optimization method according to claim 1, wherein, The gated adaptive spatio-temporal feature fusion in step (h) is achieved by the following formula: ； wherein β t and γ t are gating weights, is the fused spatio-temporal feature, is the global feature. 6.The multi-agent reinforcement learning based delivery route optimization method according to claim 1, wherein, The calculation formula of the advantage function in step (m) is: ； wherein, is the immediate reward obtained at the next time step t + 1 ; and are the estimated state values of the next state and the current state respectively; is the discount factor. 7.The multi-agent reinforcement learning based delivery route optimization method of claim 1, wherein, The clipped target function in step (n) is: ； where, is the action probability ratio of the new policy to the old policy, A t is the advantage function, is the clipped hyperparameter, denotes the empirical mean over a finite batch. 8.The multi-agent reinforcement learning based delivery route optimization method of claim 1, wherein, When updating the critic network parameters in step (n), the optimization target is: ； where V ϕ (s t ) is the estimated value of state st, r t is the immediate reward, and T is the total number of time steps. 9.The multi-agent reinforcement learning based delivery route optimization method of claim 1, wherein, The action probability distribution in step (f) is calculated by the following formula: ； This formula is used to compute the probability of taking an action given an observation The numerator is the exponentiated probability of the action based on the mask rules and the observation The denominator is the sum of the exponentiated probabilities of all possible actions given the same conditions.

Images