Dynamic planning method for urban terminal distribution path based on fusion of ant colony algorithm and Q-learning

Abstract

The application discloses a city end distribution path dynamic programming method based on ant colony algorithm and Q-Learning fusion, and belongs to the technical field of intelligent logistics path planning. The method comprises the following steps: collecting real-time distribution task sets and dynamic traffic data of a road network, and constructing a dynamic environment model comprising node states and edge passing time; initializing the number of ants of the ant colony algorithm, the initial concentration of pheromone and the state-action value table of the Q-Learning algorithm; each ant selects a next node according to a probability transfer rule combining a current Q value and heuristic information, and updates the Q value by using a time sequence difference error in a moving process; after completing one round of iteration, pheromone increments are corrected according to the cumulative distribution time of the ant path and the Q value deviation, and the global pheromone is updated; the iteration is repeated until convergence, and the current optimal distribution path is output. The application fuses the online learning ability of Q-Learning and the global search advantage of the ant colony algorithm, and realizes efficient adaptive optimization of the distribution path under a dynamic road network environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent logistics route planning technology, specifically a dynamic planning method for urban last-mile delivery routes based on the fusion of ant colony algorithm and Q-Learning. Background Technology

[0002] Urban last-mile delivery faces multiple dynamic factors, including order fluctuations, traffic congestion, and delivery time window constraints, requiring the generation of high-quality delivery routes that satisfy these constraints within a short period. Existing route planning methods mainly rely on swarm intelligence algorithms such as ant colony optimization and genetic algorithms, or use reinforcement learning algorithms such as Q-Learning and deep Q-networks for single-agent route decision-making. Ant colony optimization gradually approaches the optimal solution through the positive feedback mechanism of pheromones, performing well in static environments. However, in situations with frequent changes in traffic conditions, pheromone updates lag behind environmental changes, easily causing converged paths to lose their timeliness. Furthermore, traditional ant colony optimization only relies on factors such as path length to update pheromones, failing to characterize the long-term value differences of node states. Q-Learning methods alone learn state-action values ​​through trial and error, adapting to dynamic environments. However, when faced with large-scale nodes and multiple delivery tasks, the state and action spaces expand rapidly, resulting in low exploration efficiency for single agents and difficulty in converging to an effective path within an acceptable timeframe.

[0003] Simply combining swarm intelligence and reinforcement learning sequentially, such as generating candidate solutions using ant colonies and then fine-tuning them with Q-Learning, or vice versa, fails to fully utilize the complementary information between the two mechanisms during the search process. The ant colony's exploration lacks awareness of the long-term value of nodes, and the value updates in Q-Learning lack effective guidance from collective experience. How to incorporate learned node values ​​into the ants' step-by-step decision-making in real time, and how to use the differences in learned state values ​​to correct the ant colony's pheromone accumulation, are two key issues for achieving deep integration of the two algorithms and improving the efficiency of dynamic path planning. Summary of the Invention

[0004] This invention provides a dynamic planning method for urban last-mile delivery routes based on the fusion of ant colony algorithm and Q-Learning, to solve the problems of slow route planning optimization speed and inability of pheromone updates to reflect long-term value differences of nodes in the dynamic environment of existing technologies. It realizes online mutual feedback between ant colony exploration and reinforcement learning value assessment, which significantly improves the generation quality and real-time response capability of dynamic delivery routes.

[0005] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a dynamic planning method for urban last-mile delivery routes based on the fusion of ant colony optimization and Q-Learning. This method combines the distributed search capability of ant colony optimization with the adaptive decision-making capability of Q-Learning to generate optimal routes in real time under dynamic traffic conditions, taking into account both delivery efficiency and time window constraints. To achieve this objective, the technical solution of the present invention includes: Step S1: Collect real-time delivery task sets and dynamic traffic data of the road network to construct a dynamic environment model that includes node status and edge travel time. Preferably, when constructing the dynamic environment model, the delivery area is divided into multiple grid nodes, each node is associated with a real-time traffic congestion coefficient and a delivery time window constraint, and the travel time between nodes is dynamically updated as edge weights based on real-time traffic data, thereby accurately reflecting changes in road network status and requirements for delivery timeliness.

[0006] Step S2: Initialize the ant colony algorithm's ant count, initial pheromone concentration, and Q-Learning state-action value table. As a preferred initialization method, define a state space for each grid node, consisting of the remaining duration of the current time window and the node's congestion level; assign a set of possible actions to each state, containing all edges leading to adjacent nodes; and process all states in the Q-table. The initial value of the action pair is set to 0, providing a unified starting point for subsequent learning and searching.

[0007] In step S3, each ant selects the next node based on a probabilistic transition rule combining its current Q-value and heuristic information, updating its Q-value using temporal difference error during the movement. The probabilistic transition rule is designed as follows: for the current ant at node i, the transition probability of each candidate next node j is calculated. This probability is proportional to the product of the pheromone concentration on the edge from node i to node j, the exponential function of the Q-value of action j in state i in the Q-table, and the β power of the heuristic information; the heuristic information is the reciprocal of the edge travel time. Thus, when choosing a path, the ant is guided by both the group's pheromone and the long-term value assessment based on Q-learning, effectively balancing local travel costs and global delivery efficiency.

[0008] When an ant moves from node i to node j, it immediately receives an instant reward, which is the sum of the negative edge travel time and the time window violation penalty. The temporal difference error is calculated, which is equal to the instant reward plus the discount factor multiplied by the maximum Q value of node j, and then minus the current Q value of action j under node i. The error is then multiplied by the learning rate to update the Q value of action j under node i, so that individual experience can be accumulated online and the transfer tendency can be adaptively adjusted.

[0009] As a preferred implementation, each ant also performs a local pheromone update during its movement: each time an ant traverses an edge, it immediately reduces the pheromone content on that edge by a certain percentage, thus reducing the probability of subsequent ants blindly following the same path. The percentage reduction in local pheromone is dynamically adjusted by the Q-value deviation on the current edge. This linkage mechanism, while maintaining population diversity, utilizes the Q-value deviation to sensitively reflect the relative changes in path quality, significantly reducing the risk of the algorithm getting trapped in local optima.

[0010] Step S4: After completing one iteration, the pheromone increment is corrected based on the cumulative delivery time and Q-value deviation of the ant path, thereby updating the global pheromone. The specific correction method is as follows: For each edge traversed by ant k, the baseline pheromone increment for that edge is calculated, which is equal to the reciprocal of the total delivery time for ant k; the Q-value deviation of the node states at both ends of the edge is calculated, where the Q-value deviation is the maximum Q-value of node j minus the maximum Q-value of node i; the baseline pheromone increment is multiplied by a deviation factor, which is 1 plus the product of the Q-value deviation and a preset amplitude, to obtain the corrected pheromone increment. When updating the global pheromone, the pheromone on all edges is evaporated, and the evaporated pheromone is equal to the current pheromone multiplied by the evaporation residue coefficient; for each edge traversed by an ant, the corrected pheromone increment is added to the evaporated pheromone; and edges exceeding the pheromone upper limit are truncated. By correcting the pheromone through Q-value deviation, high-value paths receive stronger positive feedback, while the pheromone of low-value paths is suppressed, thereby accelerating the convergence speed and improving the quality of the solution.

[0011] Step S5: Repeat steps S3 and S4 until convergence, and output the current optimal delivery route. The convergence condition is: the change in the total time of the optimal route over multiple iterations is less than a preset threshold, or the maximum number of iterations is reached. Along with outputting the optimal delivery route, the corresponding Q-value sequence is also output, providing a quantitative basis for subsequent route fine-tuning and rapid replanning.

[0012] The method provided by this invention possesses outstanding dynamic adaptability. In step S1, it also receives real-time information on delivery task changes and traffic events. After convergence in step S5, if there are unfinished delivery tasks or significant changes in the road network status, steps S2 to S5 are retried to dynamically replan the remaining unfinished delivery tasks. In particular, when reinitializing the Q-table, the state-action values ​​of unaffected areas in the original Q-table are retained as prior knowledge, enabling the new planning process to reuse learned experience and quickly converge to a new optimal delivery path in a dynamic environment, significantly improving the real-time responsiveness and robustness of last-mile delivery.

[0013] The technical effects and advantages provided by the present invention in the above technical solution are as follows: In ant node transfer decisions, the probabilistic transfer rule is jointly determined by the pheromone concentration between nodes, the exponential function of the state-action value in the Q-table, and the power of the reciprocal of the edge travel time. The Q-value participates in the transfer probability calculation exponentially, allowing an ant to consider not only historically optimal paths indicated by accumulated pheromones but also its own state-action value judgments gained through online learning when choosing the next node. When an edge has a high value in the Q-table, it still has a high probability of being selected even if the current pheromone concentration has not yet accumulated sufficiently. This injects the individual's forward-looking experience from reinforcement learning into the ant colony's random search process in real time. This mechanism can quickly guide ants to avoid congested areas or nodes with tight time windows when traffic conditions suddenly change, avoiding reliance solely on delayed pheromone feedback and significantly reducing the number of iterations required to rediscover effective paths in dynamic environments.

[0014] In the global pheromone update phase, the pheromone increment is corrected using the Q-value deviation of node states. For each edge traversed by the ant, the inverse of the total delivery time is first used as the baseline pheromone increment. Then, the deviation of the maximum Q-value of each node at both ends of the edge is calculated. This deviation is multiplied by a preset amplitude and added to the baseline increment to form the corrected pheromone increment. The Q-value deviation reflects the difference in long-term value between adjacent nodes: when moving from node i to node j brings a significant value increase, this edge will receive a greater pheromone enhancement than that represented by travel time alone. This pheromone adjustment method driven by value deviation allows the pheromone distribution to not only record the experience of path length but also embed the long-term value assessment of node states. This encourages the ant colony to choose edges that lead to high-value states in subsequent iterations, maintaining the global optimality of the path under dynamic time windows and congestion changes, and suppressing path fragmentation caused by local traffic fluctuations. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0016] Figure 1 This is a flowchart of a dynamic planning method for urban last-mile delivery routes based on the fusion of ant colony algorithm and Q-Learning. Figure 2 This is a flowchart of the local pheromone update process based on Q-value deviation; Figure 3 It is a scatter plot showing the relationship between real-time traffic congestion coefficient and travel time under different baseline travel time intervals; Figure 4These are curves showing how the value of actions in different states changes with the number of iterations. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] See Figure 1 This invention provides a method based on ant colony algorithm and Q-square algorithm. Learning integrates a dynamic programming method for urban last-mile delivery routes, which includes the following steps: Step S1: Collect real-time delivery task sets and dynamic traffic data of the road network to construct a dynamic environment model that includes node status and edge travel time; Step S2: Initialize the number of ants, the initial pheromone concentration, and Q for the ant colony algorithm. The state of the learning algorithm Action Value Table; Step S3: Each ant selects the next node based on the probability transition rule that combines the current Q value with heuristic information, and updates the Q value using temporal difference error during the movement process; Step S4: After completing one iteration, adjust the pheromone increment based on the deviation between the cumulative delivery time of the ant path and the Q value, and update the global pheromone. Step S5: Repeat steps S3 and S4 until convergence, and output the current optimal delivery route.

[0019] Example 1 In the specific implementation, step S1, constructing the dynamic environment model, includes: dividing the delivery area into multiple grid nodes, using a square grid covering the city's delivery range, with the geometric center of each grid as the grid node. Each grid node is associated with a real-time traffic congestion coefficient and a delivery time window constraint. The real-time traffic congestion coefficient is obtained by accessing the city's real-time road traffic information platform, which provides the average driving speed of vehicles on each road segment. The real-time traffic congestion coefficient is calculated using the average driving speed and the road's maximum speed limit. For the edge connecting grid node u and grid node v, the travel time between grid nodes is dynamically updated as the edge weight based on real-time traffic data. The dynamic update process uses the following formula to calculate the travel time. :

[0020] in, The time taken for travel between grid nodes u and v is expressed in minutes. This represents the baseline travel time between grid node u and grid node v under unobstructed road conditions, in minutes. The baseline travel time is obtained by dividing the road length between the two grid nodes by the maximum speed limit of the road. This represents the real-time traffic congestion coefficient for the road segment between grid node u and grid node v. The real-time traffic congestion coefficient ranges from 0 to 1, indicating when the road is completely unobstructed. Set to 0 when the road is completely congested. Take 1; This indicates the amplification factor for the impact of congestion. The value is set to 1.0, ensuring that the real-time traffic congestion coefficient affects travel time linearly. Each time real-time traffic data is updated, the real-time traffic congestion coefficient is recalculated based on the updated average speed, and the travel time is recalculated using the above formula, thus reflecting the latest edge weights in the dynamic environment model. Specifically, the delivery time window constraint sets a latest delivery time for each grid node. If a delivery task must arrive at the grid node before the specified time, the time window constraint takes effect; a delivery time later than the latest delivery time will incur a time window violation penalty.

[0021] In step S2, when initializing the state-action value table, a state space is constructed for each grid node. The state space is composed of the remaining duration of the current time window and the node congestion level. The remaining duration of the current time window is defined as the time from the current moment to the latest delivery time associated with the grid node. The remaining duration of the current time window is divided into three intervals: less than 5 minutes is classified as the first duration level, greater than or equal to 5 minutes and less than 15 minutes as the second duration level, and greater than or equal to 15 minutes as the third duration level. The node congestion level is classified according to the real-time traffic congestion coefficient associated with the grid node: a real-time traffic congestion coefficient less than 0.3 is classified as low congestion level, a real-time traffic congestion coefficient greater than or equal to 0.3 and less than 0.6 is classified as medium congestion level, and a real-time traffic congestion coefficient greater than or equal to 0.6 is classified as high congestion level. The combination of the remaining duration level of the current time window and the node congestion level constitutes a discrete set of states, and the total number of states is equal to the product of the number of remaining duration levels of the current time window and the number of node congestion levels. For each discrete state, a set of possible actions is determined. This set consists of edges from the current grid node to all adjacent grid nodes, with each edge corresponding to one possible action. A state-action value table is constructed, which is a two-dimensional table indexed by states (row indexes) and actions (column indexes). Each element in the table stores the initial state-action value of the corresponding state-action pair. The initial state-action value of all state-action pairs is set to 0, indicating that no pre-defined merit evaluation is given to any state-action pair before path planning begins.

[0022] See Figure 3 The horizontal axis in the graph represents the real-time traffic congestion coefficient between grid nodes. The value ranges from 0 to 0.75, and the vertical axis represents the travel time of the corresponding edge. (Unit: minutes). The legend divides the data points into three categories, each corresponding to a baseline travel time. Different intervals: blue dots represent edges with a base passage time of no more than 3 minutes, orange squares represent edges with a base passage time between 3 and 6 minutes, and green triangles represent edges with a base passage time of 6 minutes or more.

[0023] The trend in the graph shows that the baseline travel time... The blue dot for 3 minutes changes according to the real-time traffic congestion index. The passage time gradually increased from 0 to above 0.6. The congestion on these short-path sections showed a clear linear growth trend, with the overall travel time remaining in the range of 2 to 5 minutes, indicating that the impact of congestion on travel time on these short-path sections was relatively smooth and under control.

[0024] The orange squares represent edges with a baseline travel time in the range of 3 to 6 minutes. The data points are relatively scattered, and the travel time increases with the traffic congestion coefficient. The travel time range is approximately 4 to 9 minutes. This shows that the edges with a medium baseline travel time are prone to large fluctuations in travel time under traffic congestion conditions, reflecting the amplification effect of the product relationship between the baseline travel time and the real-time congestion coefficient in the dynamic environment model.

[0025] The green triangle represents the base travel time. For edges with a 6-minute travel time, data points generally exhibit high travel times, and these times show a significant upward trend with increasing congestion coefficients. Travel times range from approximately 6 to 16 minutes. When the congestion coefficient exceeds 0.4, travel times increase significantly, reflecting a substantial increase in travel costs for long-distance edges under congested conditions. This verifies the formula... The linear impact of the congestion coefficient on travel time.

[0026] Example 2 In practice, the process of each ant selecting the next node from its current node according to the probability transition rule in step S3 is as follows: For the current ant's node, denoted as node r, all nodes adjacent to node r that have not yet been visited by the current ant are designated as candidate nodes, each denoted as node s. The transition probability of the ant moving from node r to node s is calculated. transition probability Calculate using the following formula:

[0027] in, Let be the probability that an ant moves from node r to node s; This represents the pheromone concentration on the edge between node r and node s; In the state-action value table, the state-action value corresponding to the action s selected in the state corresponding to node r is represented. Here, action s is the edge that moves from node r to node s. Similarly; This represents the travel time between nodes r and s. Represents node r and node The travel time between edges is provided in real time by the dynamic environment model; This represents the heuristic information importance factor. The value range is 1 to 5. In this step, The value is set to 2. The reason for this setting is that when the value is 2, the reciprocal of the edge travel time amplifies the difference in the form of a square, so that the candidate node with the shorter edge travel time has a higher transition probability. This allows for full utilization of local heuristic information during the search process, which matches the time-sensitive nature of urban last-mile delivery. Let r be the set of all adjacent nodes starting from node r. Based on the calculated transition probabilities, the ant randomly selects the next node to reach from all candidate nodes using a roulette wheel approach. After selection, the ant moves from node r to the selected node, extending the path.

[0028] Once the ant completes its movement from node r to node s, it immediately receives an instant reward and updates its state-action value. The instant reward consists of two parts: the first part is the negative value of the edge travel time. The second part is the time window violation penalty. The specific method for determining the time window violation penalty is as follows: record the time when the ant arrives at node s. If this time is later than the latest delivery time associated with node s, a time window violation penalty occurs. The value of the time window violation penalty is the timeout duration multiplied by a penalty coefficient, which is set to 10.0. If no time window violation occurs, the time window violation penalty is 0. The immediate reward is calculated as follows: Immediate Reward = Time window violation penalty. After receiving the immediate reward, calculate the temporal difference error. The calculation process for the temporal difference error is as follows: add the immediate reward to the discount factor multiplied by the maximum Q value of node s, and then subtract the current Q value of action s under node r. Here, the discount factor is set to 0.9; the maximum Q value of node s refers to the highest state-action value among all possible actions in the current state of node s in the state-action value table; the current Q value of action s under node r is... Then, the update amount is obtained by multiplying the temporal difference error by the learning rate, and this update amount is added to... The state-action value of action s under node r is updated. The learning rate is set to 0.1. This update enables the state-action value table to incorporate the actual time cost and constraint violation information experienced by the ant, gradually improving the evaluation of action selection.

[0029] See Figure 4 The vertical axis in the graph represents the state-action value. The horizontal axis represents the number of iterations, ranging from 0 to 500. The graph contains three curves, corresponding to the state-action value trends of different states and actions defined in Example 2. The blue solid curve represents the state-action value of "duration greater than 15 minutes and low congestion level" with action A; the red dashed curve represents the state-action value of "duration between 5 and 15 minutes and medium congestion level" with action B; and the purple dotted curve represents the state-action value of "duration less than 5 minutes and high congestion level" with action C.

[0030] As can be seen from the graph, the blue solid line curve represents the state-action value. The overall trend is upward, with rapid growth in the first 100 iterations, followed by a slowdown in growth rate, and then stabilization after about 400 iterations. The state-action value fluctuation remains between 4 and 5.5, indicating that action A is gradually evaluated as the better choice in this state, and the value tends to converge after multiple iterations of learning.

[0031] The state and value of the red dashed curve. The overall trend is downward, initially. Slightly higher than 0, the value gradually decreases and tends to stabilize as the number of iterations increases. After about the 300th iteration, it fluctuates in the range of -3 to -2, indicating that action B is gradually judged as a worse choice in this state, and the system's evaluation of this action continues to decrease.

[0032] The state of the purple dotted curve - action value The value rises in the early stages of iteration, then stabilizes after the 100th iteration, with fluctuations concentrated between 0.5 and 1.5. This indicates that in this state, action C is learned to have a moderately high value and is relatively stable.

[0033] Example 3 In practice, step S4 includes the process of correcting the pheromone increment based on the deviation between the cumulative delivery time of the ant path and the Q value, and the process of updating the global pheromone.

[0034] When correcting the pheromone increment based on the deviation between the cumulative delivery time and Q-value of the ant path, for each ant k, every edge traversed on ant k's path is visited, and the corrected pheromone increment is calculated for each edge. For an edge connecting node a and node b traversed by ant k, the baseline pheromone increment is first calculated. The baseline pheromone increment is equal to the reciprocal of the total delivery time of ant k. The total delivery time of ant k refers to the cumulative time consumed by ant k starting from the delivery starting point, visiting all task nodes sequentially along the path until all delivery tasks are completed. This cumulative time is obtained by summing the travel time of each edge on the path. Then, the Q-value deviation of the node states at both ends of the edge is calculated. The Q-value deviation of the node states is the maximum Q-value of node b minus the maximum Q-value of node a. The maximum Q-value of node b refers to the highest state-action value among all possible actions corresponding to node b in its current state in the state-action value table. The maximum Q value of node a refers to the state-action value that is the largest among all possible actions in the state-action value table for node a in its current state.

[0035] After calculating the baseline pheromone increment and Q-value deviation, calculate the corrected pheromone increment using the following formula. :

[0036] in, To correct the pheromone increment; This represents the total time it takes for ant k to complete the delivery; Indicates the preset amplitude. The value range is from 0.1 to 1.0, and in implementation... The value is set to 0.5. The reason for this setting is that when the value is 0.5, the adjustment strength of the Q value deviation on the pheromone increment is moderate. It can reflect the guiding role of the Q value deviation, and will not cause pheromone update oscillation due to excessive adjustment. It is kept within the same order of magnitude as the pheromone baseline increment. This represents the state-action value of taking action c in the state of node b. This represents the set of possible actions for node b in its current state. This represents the maximum Q-value of node b in its current state; Indicates at node Take action in the state State-action value; Represents a node The set of possible actions in the current state; Represents a node Maximum in the current state value. For the deviation factor, when the node in the deviation factor is... The largest Value greater than node The largest When the value is, The value deviation is positive, and the deviation factor is greater than 1. The corrected pheromone increment is greater than the baseline pheromone increment; when the node The largest Value less than node The largest When the value is, The value deviation is negative, and the deviation factor is less than 1. The corrected pheromone increment is less than the baseline pheromone increment.

[0037] During the global pheromone update process, the pheromone on all edges is first volatilized. For each edge in the road network environment, the volatilized pheromone is equal to the current pheromone concentration multiplied by the volatilization residue coefficient. The volatilization residue coefficient ranges from 0 to 1, and in implementation, it is set to 0.9. After the volatilization process is completed, for each ant k, every edge traversed by ant k is iterated, and the calculated correction pheromone increment for the corresponding edge is added to the volatilized pheromone concentration to obtain the updated pheromone concentration.

[0038] During the accumulation process, the pheromone concentration on any edge is truncated by an upper limit. The upper limit is set to 10.0, based on the following reasons: a pheromone upper limit of 10.0 can prevent the pheromone concentration from growing infinitely due to frequent selection of certain edges, avoiding the algorithm from getting trapped in a local optimum too early, and the pheromone upper limit of 10.0 is kept within a reasonable multiple of the initial pheromone concentration. The truncation process is as follows: when the accumulated pheromone concentration exceeds the upper limit, the pheromone concentration is forcibly set to the upper limit; when the accumulated pheromone concentration does not exceed the upper limit, the accumulated pheromone concentration remains unchanged.

[0039] Example 4 In specific implementation, please refer to Figure 2 In step S3, each ant also performs a local pheromone update during its movement. As soon as an ant moves from its current node to the next node, a local pheromone update is immediately performed on the edge the ant just traversed. The edge the ant just traversed is the edge connecting the starting node g and the destination node h, where starting node g is the ant's original node and destination node h is the ant's new node. The local pheromone update process requires calculating the local pheromone reduction ratio, which is dynamically adjusted by the Q-value deviation on the edge between the starting node g and the destination node h.

[0040] When calculating the Q-value deviation on an edge, first obtain the maximum Q-value of the destination node h. The maximum Q-value of the destination node h is the maximum value among all possible actions in the current state of the destination node h, as defined in the state-action value table. Next, obtain the maximum Q-value of the originating node g. The maximum Q-value of the originating node g is the maximum value among all possible actions in the current state of the originating node g, as defined in the state-action value table. The Q-value deviation on the edge is the maximum Q-value of the destination node h minus the maximum Q-value of the originating node g.

[0041] Then, the local pheromone reduction ratio is calculated based on the Q-value deviation on the edge. The local pheromone reduction ratio is determined according to the following formula:

[0042] in, This represents the percentage reduction in local pheromone along the edge between the starting node g and the destination node h. This indicates the percentage reduction in the base amount. The value range is 0 to 1, and in implementation... The value is set to 0.05. The reason for this setting is that setting the base reduction ratio to 0.05 can produce a mild pheromone decay effect, avoid the rapid loss of pheromone concentration due to excessive decay, and reduce the probability of subsequent ants completely repeating the same path. This represents the Q-value deviation adjustment coefficient. The value range is from 0.01 to 0.1, and in implementation it will be... The value is set to 0.02. The basis for this setting is that the common range of Q-value deviation is from 0 to tens. After setting the Q-value deviation adjustment coefficient to 0.02, the additional reduction ratio caused by Q-value deviation can be controlled to be around 0 to 0.2, so that the adjustment range is on the same order of magnitude as the basic reduction ratio, thus achieving a balance between exploration diversity and convergence speed. This represents the state-action value of taking action x in the state of reaching node h. This represents the set of possible actions to take to reach node h in the current state; This represents the maximum Q value reaching node h; This represents the state-action value of taking action y from the state of the starting node g. Let g represent the set of possible actions for starting node g in its current state; This represents the maximum Q value of the starting node g.

[0043] Obtain the proportion of local pheromone reduction Then, the pheromone concentration on the edge between the starting node g and the arriving node h is updated. The updated pheromone concentration is equal to the original pheromone concentration multiplied by 1. After completing the multiplication operation, a lower limit truncation process is required: the lower limit of pheromone concentration is set to 0.001. The rationale for this setting is that a lower limit of 0.001 can maintain a non-zero minimum pheromone value, preventing some edges from permanently losing the possibility of being selected in subsequent searches due to the complete disappearance of pheromone concentration. When the updated pheromone concentration is lower than the lower limit of 0.001, the pheromone concentration on the edge is forcibly set to 0.001. When the updated pheromone concentration is greater than or equal to the lower limit of 0.001, the updated pheromone concentration remains unchanged.

[0044] In this way, when the maximum Q-value of the reaching node h is greater than the maximum Q-value of the starting node g, the Q-value deviation on the edge is positive, the local pheromone reduction ratio increases, and the pheromone suppression effect on the current edge is stronger, reducing the probability of subsequent ants excessively following the same edge. When the Q-value deviation on the edge is negative or small, the local pheromone reduction ratio is small, preserving the pheromone concentration of the current edge and maintaining the attractiveness of the path. The local pheromone reduction ratio is dynamically adjusted according to the change in Q-value deviation.

[0045] Example 5 In practical implementation, step S5 repeatedly executes steps S3 and S4, forming an iterative loop. After each iteration, the system records the path with the minimum total delivery time among all ant-generating paths in this iteration, takes this path as the best path for this round, and records the corresponding total delivery time. The total delivery time of the best path in this round is compared with the total delivery time of the best paths in the previous several rounds, and the change range is calculated. The change range is calculated by taking the total delivery time of the best paths in the most recent 5 consecutive iterations, removing the maximum and minimum values, calculating the range of the remaining values, and taking the percentage of the ratio of this range to the average of the total delivery time of the best paths in these rounds as the change range. When the change range is lower than a preset threshold, it is determined that the convergence condition is met. The preset threshold is set to 2%, based on the following: when the fluctuation of the total delivery time of the best paths in 5 consecutive rounds is less than 2%, it indicates that the algorithm has entered a stable state in terms of time indicators, and further iterations contribute very little to path improvement, achieving a reasonable balance between solution stability and computational cost. The maximum number of iterations is set to 500. This is based on the following reasoning: for a typical number of nodes in urban last-mile delivery, 500 iterations are sufficient to complete a thorough search; exceeding this number increases computational costs while diminishing returns. When the number of iterations reaches 500, regardless of whether the change is below a preset threshold, it is considered to have met the convergence condition.

[0046] After the convergence condition is met, the current optimal delivery path is output. The current optimal delivery path is the path with the shortest total delivery time recorded across all iterations. When outputting the current optimal delivery path, the corresponding Q-value sequence is also extracted and output. The Q-value sequence is formed by sequentially visiting each node along the current optimal delivery path from the starting point, and between every two consecutive nodes, retrieving the state-action value from the state-action value table that matches the state at the time of leaving the node and the action corresponding to reaching the next node, and arranging these state-action values ​​in the order of visit. Using the Q-value sequence, when minor fluctuations occur in external conditions during subsequent delivery processes, the state-action values ​​in the Q-value sequence can be directly invoked for local path adjustments without restarting the entire path planning process.

[0047] In the dynamic environment model constructed in step S1, delivery task change information and traffic event information are continuously received in real time. Delivery task change information includes adding delivery tasks, canceling existing delivery tasks, and modifying the time window or destination of existing delivery tasks. Traffic event information includes sudden road congestion, reduced traffic capacity due to traffic accidents, and temporary traffic control. Whenever delivery task change information or traffic event information is received, the time window constraints of the corresponding nodes or the edge travel time between nodes in the dynamic environment model are updated immediately.

[0048] Once step S5 outputs the current optimal delivery route, if there are any remaining unfinished delivery tasks in the system, or if the dynamic environment model changes significantly due to receiving new delivery task change information or traffic event information, steps S2 to S5 are retried to dynamically replan the remaining unfinished delivery tasks. Remaining unfinished delivery tasks include tasks that have not yet been served and newly added delivery tasks.

[0049] When step S2 is triggered again to initialize the Q-table, not all state-action values ​​are reset to zero. The system first identifies affected areas based on changes in the dynamic environment model. Affected areas are defined as: nodes where new delivery tasks are added, nodes where delivery tasks are canceled, nodes whose time windows have been modified, and nodes whose edge travel time changes by more than 30% due to traffic events, along with their adjacent nodes. The criterion for a change exceeding 30% is that traffic events often cause delays several times greater than normal travel time; setting the threshold to 30% sensitively captures travel time changes with actual impact, avoiding the neglect of invalid replanning caused by small fluctuations. The remaining nodes not identified as affected areas constitute unaffected areas.

[0050] For state-action pairs in the state-action value table whose corresponding nodes fall within unaffected regions, the original Q-values ​​are retained as prior knowledge and not reset. For state-action pairs whose corresponding nodes fall within affected regions, their state-action values ​​are reset to 0, allowing them to learn from scratch in subsequent iterations. In this way, dynamic replanning can inherit the experience learned in stable regions from previous planning processes, concentrating computational resources to explore new optimal strategies in disturbed regions. When retaining the original Q-values, the corresponding values ​​from the original state-action value table are directly copied to the newly initialized table; state-action values ​​reset to 0 are filled with the value 0. Other initialization operations, including the number of ants and the initial pheromone concentration, are reset in the same way as in step S2.

[0051] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A dynamic planning method for urban last-mile delivery routes based on the fusion of ant colony algorithm and Q-Learning, characterized in that, Includes the following steps: Step S1: Collect real-time delivery task sets and dynamic traffic data of the road network to construct a dynamic environment model that includes node status and edge travel time; Step S2: Initialize the number of ants for the ant colony algorithm, the initial pheromone concentration, and the state-action value table for the Q-Learning algorithm; Step S3: Each ant selects the next node based on the probability transition rule that combines the current Q value with heuristic information, and updates the Q value using temporal difference error during the movement process; Step S4: After completing one iteration, adjust the pheromone increment based on the deviation between the cumulative delivery time of the ant path and the Q value, and update the global pheromone. Step S5: Repeat steps S3 and S4 until convergence, and output the current optimal delivery route.

2. The urban last-mile delivery route dynamic planning method based on the fusion of ant colony algorithm and Q-Learning as described in claim 1, characterized in that, The construction of the dynamic environment model in step S1 includes: dividing the delivery area into multiple grid nodes, with each node associated with a real-time traffic congestion coefficient and a delivery time window constraint; The travel time between nodes is dynamically updated based on real-time traffic data as the edge weight.

3. The urban last-mile delivery route dynamic planning method based on the fusion of ant colony algorithm and Q-Learning as described in claim 2, characterized in that, The initialization of the state-action value table in step S2 includes: defining a state space for each grid node, which consists of the remaining duration of the current time window and the node congestion level; Assign a set of possible actions to each state, the set of possible actions containing the edges leading to adjacent nodes; Set the initial value of all state-action pairs in the Q table to 0.

4. The urban last-mile delivery route dynamic planning method based on the fusion of ant colony algorithm and Q-Learning as described in claim 3, characterized in that, The probability transfer rule in step S3 is as follows: For the current ant's node i, calculate the transfer probability of each candidate next node j. The transfer probability is proportional to the pheromone concentration on the edge from node i to node j multiplied by the exponential function of the Q value of action j in state i in the Q table, and then multiplied by the β power of the heuristic information. The heuristic information is the reciprocal of the edge travel time.

5. The urban last-mile delivery route dynamic planning method based on the fusion of ant colony algorithm and Q-Learning as described in claim 4, characterized in that, The step S3 of updating the Q value using time-series difference error includes: when an ant moves from node i to node j, it obtains an immediate reward, which is the sum of the negative edge travel time and the time window violation penalty; Calculate the temporal difference error, which is equal to the immediate reward plus the discount factor multiplied by the maximum Q value of node j minus the current Q value of action j under node i; use this error multiplied by the learning rate to update the Q value of action j under node i.

6. The urban last-mile delivery route dynamic planning method based on the fusion of ant colony algorithm and Q-Learning as described in claim 5, characterized in that, In step S4, the pheromone increment is corrected based on the deviation between the cumulative delivery time of the ant path and the Q value. Specifically, this includes: for each edge traversed by ant k, the baseline pheromone increment of that edge is calculated. The baseline pheromone increment is equal to the reciprocal of the total delivery time of ant k. Calculate the Q-value deviation of the states of the nodes at both ends of the edge, where the Q-value deviation is the maximum Q-value of node j minus the maximum Q-value of node i; The corrected pheromone increment is obtained by multiplying the baseline pheromone increment by a deviation factor, which is 1 plus the product of the Q-value deviation and a preset amplitude.

7. The urban last-mile delivery route dynamic planning method based on the fusion of ant colony algorithm and Q-Learning as described in claim 6, characterized in that, The step S4 of updating the global pheromone includes: evaporating the pheromone on all edges, and the evaporated pheromone is equal to the current pheromone multiplied by the evaporation residual coefficient. For each edge traversed by an ant, the pheromone increment is added to the evaporated pheromone; edges exceeding the pheromone limit are truncated.

8. The urban last-mile delivery route dynamic planning method based on the fusion of ant colony algorithm and Q-Learning as described in claim 7, characterized in that, In step S3, each ant also performs a local pheromone update during its movement: every time an ant passes an edge, it immediately reduces the pheromone on that edge by a certain percentage to reduce the probability that subsequent ants will follow the same path. The local pheromone reduction percentage is dynamically adjusted by the Q-value deviation on the current edge.

9. The urban last-mile delivery route dynamic planning method based on the fusion of ant colony algorithm and Q-Learning as described in claim 8, characterized in that, The convergence condition in step S5 is: the change in the total time of the best path in multiple consecutive iterations is lower than a preset threshold, or the maximum number of iterations is reached. When outputting the current optimal delivery route, the corresponding Q-value sequence is also output for subsequent route adjustments.

10. The urban last-mile delivery route dynamic planning method based on the fusion of ant colony algorithm and Q-Learning as described in claim 9, characterized in that, Step S1 also includes receiving delivery task changes and traffic event information in real time. After step S5 converges, steps S2 to S5 are retried to dynamically replan the remaining unfinished delivery tasks. When re-initializing the Q table, the state-action values ​​of the unaffected areas in the original Q table are retained as prior knowledge.

Images