Transportation path optimization method based on reinforcement learning

Abstract

The application discloses a transportation path optimization method based on reinforcement learning and belongs to the technical field of intelligent logistics path optimization. In order to solve the problem that a fixed path is difficult to respond in time when the vehicle position, road congestion, distribution time window and loading and unloading waiting state continuously change, the application generates a rolling transportation path, an estimated arrival time and a scheduling instruction through dynamic heterogeneous space-time graph coding, reinforcement learning decision and joint risk mask, and realizes the technical effect of reducing the risks of detour, empty driving and overtime distribution.

Description

Technical Field

[0001] This invention relates to the field of intelligent logistics route optimization technology, and in particular to a transportation route optimization method based on reinforcement learning. Background Technology

[0002] In long-haul transportation and urban delivery scenarios for logistics fleets, vehicle operating status, road conditions, order delivery windows, and loading / unloading queues at depots continuously change over time. Existing route planning methods typically rely on static road networks, fixed delivery sequences, or manual reassignment. When abnormal congestion, depot queuing, and order window compression occur simultaneously, it is difficult to reflect the coupled effects between multiple sources of conditions in a timely manner.

[0003] Existing technologies have at least the following shortcomings: First, the spatiotemporal relationships between vehicles, orders, road segments, and depots are not adequately expressed during the route planning process, making it difficult to reflect in advance the impact of congestion propagation and loading / unloading waiting on subsequent order windows; Second, manual reassignment or fixed route rules usually only handle the local accessibility of the current node, lacking a joint evaluation of the compression of subsequent order window margins and long-term benefits; Third, the candidate route action generation stage lacks a feasibility masking for the coupled risks of time windows, congestion, and loading / unloading waiting, which can easily lead to the infeasibility of subsequent routes when the current action seems feasible.

[0004] Therefore, a transportation route optimization method is needed to address the shortcomings of the existing technologies. Summary of the Invention

[0005] One objective of this invention is to propose a transportation route optimization method based on reinforcement learning. Addressing the problem that existing technologies struggle to respond promptly to continuously changing vehicle locations, road congestion, delivery windows, and loading / unloading waiting states, leading to detours, empty runs, and delayed deliveries, this invention proposes a technical solution that constructs a dynamic heterogeneous spatiotemporal graph, generates vehicle route state vectors through a spatiotemporal graph encoder, outputs candidate actions using a reinforcement learning decision model, and sets a joint risk mask of time window, congestion, and loading / unloading waiting before the final action selection. This invention has the technical advantages of preemptively screening out risky and inactive actions and continuously generating transportation routes, estimated arrival times, and scheduling instructions.

[0006] This invention provides a transportation route optimization method based on reinforcement learning, comprising: S1, collecting transportation status data of vehicles, orders, road segments, and depots and constructing a dynamic heterogeneous spatiotemporal graph; S2, inputting the dynamic heterogeneous spatiotemporal graph into a trained spatiotemporal graph encoder to extract features of congestion propagation, vehicle status, order urgency, and loading / unloading waiting, thereby obtaining a vehicle route state vector; S3, inputting the vehicle route state vector into a reinforcement learning decision model to generate a set of candidate actions, strategy probabilities, and long-term benefit estimates; S4, before the final action selection, using a time window-congestion-loading / unloading waiting joint risk masking module to determine a mask of actionable actions, a risk score, and a repair trigger flag; S5, correcting the strategy probabilities or generating repair actions based on the mask of actionable actions, the risk score, and the repair trigger flag, and determining the action to be executed in conjunction with the long-term benefit estimate, generating a transportation route, estimated arrival time, and scheduling instructions, and continuously updating the dynamic heterogeneous spatiotemporal graph and transportation route as the transportation status data changes.

[0007] Optionally, S1 includes: Generate vehicle node attributes based on vehicle location, vehicle load, and assigned orders; Generate order node attributes based on order delivery time window, order required weight, service duration, pickup and delivery location, and order status; The road segment node attributes are generated based on the road segment congestion status, the predicted road segment travel time, and the predicted road segment travel time distribution. The terminal node attributes are generated based on the terminal loading and unloading waiting queue, terminal service capacity, predicted loading and unloading waiting value, and predicted loading and unloading waiting distribution. The dynamic heterogeneous spatiotemporal graph is obtained by generating timestamped connection edges based on the vehicle-order allocation relationship, the order-road segment passage relationship, and the vehicle-station entry and exit relationship.

[0008] Optionally, S2 includes: The congestion status changes of adjacent road segments are aggregated in the dynamic heterogeneous spatiotemporal graph according to the timestamp order to generate congestion propagation features; Generate vehicle status features based on vehicle node attributes and assigned orders; Order urgency features are generated based on the order delivery window, the estimated service start time, and the service duration. The estimated service start time is determined based on the current transportation route, vehicle location, order sequence, and predicted road segment travel time. Loading and unloading waiting characteristics are generated based on the loading and unloading waiting queues and predicted loading and unloading waiting values ​​at the terminal. The congestion propagation features, vehicle status features, order urgency features, and loading / unloading waiting features are aggregated using graph attention to output the vehicle path state vector. Furthermore, the spatiotemporal map encoder is trained using historical transportation samples, which include historical dynamic heterogeneous spatiotemporal maps, historical vehicle routes, actual arrival times, and actual loading and unloading waiting times. During training, the encoding loss is composed of the prediction arrival time error, the road segment travel time error, and the loading and unloading waiting time error. The encoding loss is jointly updated with the reward loss of the reinforcement learning decision model, so that the vehicle path state vector retains the state information related to congestion propagation, order window, and depot waiting.

[0009] Optionally, S3 includes: The vehicle path state vector is input into the action decision network to generate a set of candidate actions, including the next order action, detour action, waiting action, reassignment action and return action, and a strategy probability is generated for each candidate action. The vehicle path state vector and the candidate action set are input into the value evaluation network, and the long-term revenue estimate is generated based on the candidate action evaluation driving time, candidate action evaluation empty mileage, timeout penalty, waiting penalty and reassignment disturbance penalty. The estimated arrival time for candidate action evaluation is determined based on the corresponding candidate action, the predicted road segment travel time in the dynamic heterogeneous spatiotemporal graph, and the current order sequence. The timeout penalty is determined based on the difference between the estimated arrival time for candidate action evaluation and the end time of the order delivery window. The estimated loading and unloading waiting time for candidate action evaluation is determined based on the corresponding candidate action and the predicted loading and unloading waiting time at the station, and the waiting penalty is determined based on the estimated loading and unloading waiting time for candidate action evaluation. The reassignment disturbance penalty is determined based on the difference between the current order sequence before the reassignment action and the candidate order sequence after the reassignment action.

[0010] Optionally, S4 includes: Input the candidate action, the predicted distribution of road segment travel time, the order delivery window, the service duration, the predicted loading and unloading wait value, and the subsequent order sequence into the window-congestion-loading and unloading wait joint risk masking module; The subsequent order sequence is obtained by sorting the unfinished orders in the current vehicle's assigned orders according to the current transportation route, and is updated when evaluating candidate actions according to the next order, detour section, waiting time, reassigned object or return destination corresponding to the candidate action; For each candidate action, calculate the current risk margin and subsequent time window compression amount after executing the candidate action. The current risk margin is the difference between the remaining time of the order delivery time window and the sum of the waiting time, the target quantile of the predicted distribution of the passage time, the service time, and the predicted loading and unloading waiting time corresponding to the candidate action. The subsequent time window compression amount is the amount by which the remaining time window of the subsequent order is reduced after executing the candidate action. The probability of subsequent timeouts is determined based on the subsequent time window compression amount, the predicted distribution of road segment travel time, the predicted loading and unloading waiting value, and the subsequent order delivery time window. When the current risk margin is less than the dynamic masking threshold, or the subsequent timeout probability is greater than the masking probability threshold, the corresponding candidate action is marked as a risky and unworkable action and the workable action mask, risk score and repair trigger flag are generated. The dynamic masking threshold is determined based on the target quantile of the historical passage time prediction error and the target quantile of the historical loading and unloading waiting prediction error, and the masking probability threshold is determined based on the timeout probability quantile in the historical order timeout sample. Furthermore, the risk score is obtained by weighting the current risk margin, the subsequent time window compression amount, and the subsequent timeout probability. The weight of the current risk margin is determined by the quantile of the remaining time of the historical order timeout sample where the remaining time of the order delivery window is located; the weight of the subsequent time window compression is determined by the number of subsequent orders and the mean of the remaining time windows of subsequent orders; and the weight of the subsequent timeout probability is determined by the variance of the predicted distribution of road segment travel time and the variance of the predicted distribution of loading and unloading waiting time. The loading and unloading waiting prediction distribution is generated based on historical terminal waiting time samples and the current terminal loading and unloading waiting queue.

[0011] Optionally, S5 includes: When the repair trigger flag does not indicate that all candidate action sets are marked as risky and inoperable actions, a risk penalty coefficient is generated based on the risk score. The operable action mask is multiplied by the strategy probability, and the multiplication result is corrected and normalized using the risk penalty coefficient to obtain the masked strategy probability. Then, the operable actions are determined based on the operable actions limited by the operable action mask, the masked strategy probability, and the long-term benefit estimate. The executed actions are then converted into the vehicle's next segment sequence, estimated arrival time, and scheduling instructions. When the repair trigger flag indicates that all candidate action sets are marked as risky and inactive actions, after skipping the mask, the strategy probability normalization and normal execution action are determined, and a repair action that is different from the original candidate action instance is regenerated. The repair action includes a waiting action with recalculation of waiting time, a reassignment action with reselection of reassignment object, an order exchange action with recombination of order order order, or a temporary return action with reselection of return destination. After the repair action meets the repair constraints, the repair action is converted into the vehicle's next segment sequence, estimated arrival time, and scheduling instructions; If the regenerated repair action does not meet the repair constraints, continue to regenerate the repair action, or generate a temporary return dispatch instruction and guide the vehicle to a depot that meets the depot service capacity threshold. Furthermore, the repair constraint conditions include: when the current vehicle load capacity is not less than the load required for the next order, the expected arrival time after waiting (the sum of waiting time, travel time, service time, and loading / unloading waiting predictions) is not greater than the end time of the next order delivery window, and the loading / unloading waiting prediction at the depot is not greater than the depot waiting threshold, the waiting action is determined to meet the repair constraint conditions. When an unexecuted order simultaneously meets the following conditions: the current vehicle's remaining load capacity is not less than the required load of the unexecuted order; the estimated arrival time determined based on the current location and the predicted road travel time is not greater than the end time of the delivery window for the unexecuted order; and the remaining time window of the subsequent order after reassignment is not less than the dynamic masking threshold, the reassignment action is determined to meet the repair constraints. When the remaining time window of each order in the subsequent order sequence after the order exchange is not less than the dynamic masking threshold, and the exchanged orders respectively satisfy that the corresponding vehicle load margin is not less than the corresponding order demand load and the corresponding expected arrival time is not greater than the corresponding order delivery time window end time, it is determined that the order exchange action meets the repair constraint conditions. When the current vehicle load capacity is less than the load required for the next order, or the remaining time window for subsequent orders is less than the dynamic masking threshold, or the current target depot service capacity is less than the depot service capacity threshold, a temporary return action is triggered. When the service capacity of the reselected return destination is not less than the service capacity threshold, the temporary return action is determined to meet the repair constraint. The load requirement for the next order, the load requirement for the unexecuted order, and the load requirement for the corresponding order are all determined by the order node attributes. The station waiting threshold and the station service capacity threshold are determined based on the target quantile of the historical station waiting time and the target quantile of the historical station service vehicle number per unit time period, respectively. Furthermore, it also includes generating a graph update event when receiving vehicle location updates, road congestion status updates, or station loading / unloading waiting queue updates; Update the corresponding vehicle node attributes, road segment node attributes, depot node attributes, and timestamped connection edges according to the graph update event, and record the update version number; After the updated dynamic heterogeneous spatiotemporal graph is re-input into the spatiotemporal graph encoder and the reinforcement learning decision model, the actionable mask, risk score and repair trigger flag are re-determined by the time window-congestion-loading and unloading waiting joint risk masking module, and the transportation route, estimated arrival time and scheduling instructions with the updated version number are generated according to the action determination process in step S5.

[0012] The beneficial effects of this invention are: 1. By constructing vehicle nodes, order nodes, road segment nodes, and depot nodes into a dynamic heterogeneous spatiotemporal graph, vehicle status, order time windows, road segment passage time, and loading / unloading waiting queues can be incorporated into the same spatiotemporal representation, enabling subsequent route decisions to be based on updated transportation status data.

[0013] 2. By extracting congestion propagation, vehicle status, order urgency, and loading / unloading waiting features through a spatiotemporal graph encoder, and inputting the vehicle path state vector into a reinforcement learning decision model, it is possible to simultaneously consider travel time, empty mileage, timeout penalty, waiting penalty, and reassignment disturbance penalty in candidate action generation and long-term benefit estimation.

[0014] 3. By setting a time window-congestion-loading / unloading waiting joint risk masking module before the final action selection, risky infeasible actions can be screened out based on the current risk margin, subsequent time window compression amount and subsequent timeout probability, and local repair actions can be triggered when all candidate actions are masked, thereby reducing the subsequent infeasibility risks caused by congestion, queuing and time window coupling. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a transportation route optimization method based on reinforcement learning.

[0016] Figure 2 This is a flowchart of the window-congestion-loading / unloading waiting joint risk masking module in step S4 of the present invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figures 1-2A reinforcement learning-based transportation route optimization method includes: S1, collecting transportation status data of vehicles, orders, road segments, and depots and constructing a dynamic heterogeneous spatiotemporal graph; S2, inputting the dynamic heterogeneous spatiotemporal graph into a trained spatiotemporal graph encoder to extract features of congestion propagation, vehicle status, order urgency, and loading / unloading waiting, obtaining a vehicle route state vector; S3, inputting the vehicle route state vector into a reinforcement learning decision model to generate a set of candidate actions, strategy probabilities, and long-term benefit estimates; S4, before the final action selection, using a time window-congestion-loading / unloading waiting joint risk masking module to determine a mask of actionable actions, risk scores, and repair trigger flags; S5, correcting the strategy probabilities or generating repair actions based on the mask of actionable actions, the risk scores, and the repair trigger flags, and determining the action to be executed in conjunction with the long-term benefit estimate, generating a transportation route, estimated arrival time, and scheduling instructions, and continuously updating the dynamic heterogeneous spatiotemporal graph and transportation route as the transportation status data changes.

[0019] In this specific embodiment, S1 includes: At the start of the scheduling cycle Execute step S1, scheduling period is taken ,in This represents the timestamp of the current graph snapshot. This indicates the time interval between two consecutive graph constructions; The vehicle location, vehicle load, and assigned orders for each vehicle are collected from the vehicle terminal. The vehicle location is determined by the vehicle's GPS latitude and longitude through map matching using the "shortest distance projection of the road segment centerline" method, which determines the road segment number and the mileage offset on that road segment. The vehicle load is obtained by subtracting the required load of completed orders from the vehicle's initial load. The assigned orders are determined by the vehicle-order binding relationship recorded by the dispatch system, thereby generating vehicle node attributes and forming a set of vehicle nodes. The system collects data on incomplete orders, including order delivery window, order load requirement, service duration, pickup and delivery location, and order status. The order delivery window is stored in two fields: "start time" and "end time". The service duration is stored in seconds. The pickup and delivery locations are stored in the latitude and longitude of pickup and delivery. The order status is limited to four discrete values: unassigned, assigned but not served, in service, and completed. This data is used to generate order node attributes and form a set of order nodes. Traffic information platforms collect road congestion status and historical traffic flow time series data, and then use the most recent data... The length of each observation passage time is: The sliding window, in which This represents the number of observation samples used for statistical prediction. Road congestion is categorized into four levels based on the ratio of "current average speed / free-flow speed," denoted as smooth, lightly congested, moderately congested, and severely congested, respectively. Simultaneously, based on this sliding window, the predicted travel time for the road segment is calculated, and the mean and variance of the predicted travel time distribution are calculated to form a "mean-variance" representation. The predicted travel time is calculated using an exponentially weighted moving average with a smoothing coefficient of [value missing]. The weighting coefficient for the latest observed travel time is represented by the variance of the travel time prediction distribution, which is taken as the exponentially weighted mean of the residual squares and together with the mean to form the road segment node attributes, thus forming a road segment node set. The terminal management system collects data on the terminal's loading / unloading waiting queue, service capacity, and historical waiting time sequences. The loading / unloading waiting queue is defined as the number of vehicles that have arrived but not yet started loading / unloading, and the service capacity is the most recent... The "vehicles / minute" service rate is calculated by converting the actual number of loading and unloading vehicles completed within the time limit. This represents the length of the historical time window used for statistical service rate calculations, and is based on the most recent data from the station waiting time sequence. For each sample, the predicted loading and unloading wait time is calculated, and a loading and unloading wait time prediction distribution is constructed using the "mean-variance" method. This represents the number of observation samples used to count the number of observations awaiting prediction, thereby generating the station node attributes and forming a set of station nodes; After obtaining various types of nodes, construct a dynamic heterogeneous spatiotemporal graph. ,in Indicates time Dynamic heterogeneous spacetime graph, Represents a set of nodes. This represents the set of connected edges with timestamps. This represents a node attribute matrix obtained by stacking vehicle node attributes, order node attributes, road segment node attributes, and depot node attributes in alignment with their node numbers. This indicates that the generation timestamp is recorded for each connecting edge. The resulting timestamp attribute vector, Represents the set of vehicle nodes. Represents the set of order nodes. Represents the set of road segment nodes. Represents the set of station nodes; Connecting edge Generate and record timestamps according to three types of relationships. The first type of edge represents the allocation relationship between vehicles and orders. Directed edges are generated for each assigned order, creating a "vehicle node - order node" relationship to indicate that the order is carried by that vehicle. The second type represents the travel relationship between orders and road segments. The shortest estimated travel time path from the current location to the next order pickup / delivery location and between adjacent order pickup / delivery locations in the subsequent order sequence is decomposed into a road segment sequence. Directed edges are then generated for each road segment sequence, creating a "order node - road segment node" relationship to indicate the route required to execute the order. The third category is the entry and exit relationship edges between vehicles and depots. Directed edges of "vehicle node and depot node" are generated one by one according to the depots that the vehicle is currently scheduled to enter or leave to express the entry or exit event. When a node attribute is missing in the current scheduling cycle, the attribute with the same name in the previous scheduling cycle of the node is used to fill it and the missing flag is retained as an additional field of the node attribute to ensure that the subsequent model can distinguish between the real observation and the filled value, thereby completing the transportation status data collection and dynamic heterogeneous spatiotemporal graph construction described in step S1.

[0020] In this specific embodiment, S2 includes: At any moment Upon entering S2, read the most recent data generated and cached in step S1. The dynamic heterogeneous spacetime graph corresponding to each update version number ,in Indicates timestamp as Dynamic heterogeneous spacetime graph, Indicates the time interval of the graph snapshot and takes , This indicates the number of graph snapshots used to extract temporal features; In the feature construction stage, for each road segment node, its set of adjacent road segments is determined based on the road network topology table, and the adjacent road segments are extracted in timestamp order. The congestion status change sequence in each snapshot is concatenated with "the congestion status of this road segment, the congestion status change sequence of adjacent road segments, the predicted travel time of the road segment, and the mean and variance of the predicted travel time distribution of the road segment" to form a congestion propagation feature, so that the road segment node carries both local congestion and neighborhood congestion propagation trend information. For each vehicle node, the road segment number corresponding to the vehicle location and the mileage offset, vehicle load, load margin, number of allocated but uncompleted orders, and the length of the subsequent order sequence corresponding to the current vehicle are concatenated to form a vehicle status feature. The load margin is obtained by subtracting the current vehicle load from the vehicle's maximum load capacity, and the subsequent order sequence is obtained by arranging the vehicle's allocated but uncompleted orders in the order of the current transportation route. For each order node, the estimated service start time of the order is first calculated based on the current transportation route, vehicle location, order sequence, and the predicted road segment travel time in the dynamic heterogeneous spatiotemporal graph. The calculation process is to accumulate "the estimated travel time from the current location to the pickup / delivery location of the order, the service duration of the preceding order, and the predicted loading / unloading waiting time at the corresponding station of the preceding order" one by one according to the order's order sequence. Then, the order urgency feature is constructed using the order delivery window, the estimated service start time, and the service duration. The order delivery window uses two fields, start time and end time, for calculation. The order urgency feature uses the remaining margin of "the end time of the window minus the estimated service start time and then minus the service duration" and is combined with the order demand load and order status. For each terminal node, the terminal's loading / unloading waiting queue, terminal service capacity, predicted loading / unloading waiting values, and the mean and variance of the predicted loading / unloading waiting distribution are concatenated to form loading / unloading waiting features. This ensures that the terminal node simultaneously carries information on queue size, processing capacity, and waiting uncertainty. In terms of model structure, the spatiotemporal graph encoder employs a deterministic "heterogeneous graph attention network + time..." The structure of "inter-aggregation" is used to output vehicle path state vectors. Specifically, independent linear mapping layers are set for vehicle nodes, order nodes, road segment nodes, and depot nodes to map their respective node attributes to a unified hidden dimension. Independent graph attention parameters are set for the vehicle order allocation edge, the order segment passage edge, and the vehicle yard entry and exit edge to perform two-layer heterogeneous graph attention aggregation, where the number of graph attention layers is taken as... The number of attention heads per layer is taken Nonlinear activation function takes The discard rate is taken as and the timestamp attribute of the edge After being converted to temporal difference encoding, it is incorporated into the attention scoring so that the attention weight changes over time; Regarding time aggregation, for the same node, in the... The hidden representations on the graph snapshots are input into a gated recurrent unit in timestamp order for aggregation. The hidden dimension of the gated recurrent unit is also 128, and the parameters are shared across all node types, thus obtaining the time-stamped representations. The node timing representation; Regarding the output method, the time of each vehicle node is taken. The node time sequence representation is used as the vehicle path state vector and denoted as ,in and This represents the vehicle node number, and the vehicle path state vector is used as input to the reinforcement learning decision model in the subsequent step S3. In terms of training methods, the spatiotemporal graph encoder is trained using historical transportation samples. These samples are generated by slicing them according to scheduling cycles and include a historical dynamic heterogeneous spatiotemporal graph, the historical vehicle routes at corresponding times, actual arrival times, and actual loading / unloading waiting times. The historical vehicle routes are represented by the sequence of road segments traversed by the vehicles and the order in which they arrive at each order. During training, the encoding loss and the reward loss of the reinforcement learning decision model are jointly updated. The total loss from the joint update is: ; in This represents the total loss during joint training. Indicates encoding loss, This represents the reward loss of a reinforcement learning decision-making model, and consists of policy loss, value loss, and entropy regularization term. Indicates the weight of gains and losses and takes Indicates the batch size for each parameter update and takes... Indicates the sample index within the batch. Indicates sample The actual arrival time of the vehicle to the target order. Indicates sample The predicted arrival time is obtained by passing the output of the mid-spatial-temporal map encoder and the arrival time prediction head. Indicates sample The actual travel time of vehicles on the target road segment. Indicates sample The predicted travel time is obtained by outputting the mid-spatial-temporal map encoder and passing through the road segment travel time prediction head. Indicates sample The actual loading and unloading waiting time of vehicles at the target station. Indicates sample The prediction wait time is output from the mid-temporal map encoder and obtained by loading and unloading the prediction head. Indicates the weight of the travel time error term and takes Indicates the weight of the waiting time error term and takes The arrival time prediction head, road segment travel time prediction head, and loading / unloading waiting prediction head all adopt a two-layer fully connected network with a hidden layer dimension of 64 and ReLU activation. Training uses the Adam optimizer and the learning rate is set to... The training rounds are set to 50, thus enabling the vehicle path state vector to... During the joint update process, state information related to congestion propagation, order windows, and station waiting is retained and used for subsequent reinforcement learning decisions.

[0021] In this specific embodiment, S3 includes: At any moment Upon entering S3, for each vehicle node Read the vehicle path state vector output in step S2 The input is then fed into a reinforcement learning decision-making model. This model employs a deterministic actor-critic structure and consists of an action decision network and a value evaluation network. The parameters of the action decision network are denoted as... The parameters of the value evaluation network are denoted as The dimension is 128 and is consistent with the steps Maintain consistency; The candidate action set is generated according to action type and uniformly coded as "action type + action parameter". Action type is limited to next order action, detour action, waiting action, reassignment action, and return action. Action parameters are limited to one or more of five fields: order number, detour route number, waiting time, reassignment target vehicle number, and return destination depot number. The next order action is performed on the vehicle... Select the first one from the subsequent order sequence Each incomplete order is processed and a candidate instance is generated for it. The subsequent order sequence has been constructed in step S2 and is consistent with the current transportation path order. The detour operation starts at the road segment corresponding to the vehicle's current location and ends at the road segment corresponding to the next order's pickup / delivery location, within a dynamic heterogeneous spatiotemporal graph. The road segment submap is operated with "predicted road segment travel time" as the edge weight. The shortest path algorithm obtains three distinct road segment sequences and generates candidate instances one by one. The path number of each road segment sequence is written into the action parameters as the detour path number. The waiting action discretizes the waiting time into Candidate instances are generated one by one, where the waiting time represents the duration during which the vehicle remains stationary at its current location; the reassignment action is based on the current vehicle. The set of assigned but unfulfilled orders is used as the reassignment source set, and vehicles that meet the "order-acceptable status" within the same dispatch area are selected from the set of other vehicles in ascending order of estimated travel time to the pickup and delivery locations of the order. Candidate instances of target vehicles are generated, where the order-acceptable status is determined by the vehicle status features that the load margin is greater than or equal to the order demand load and the vehicle status is schedulable. The return-to-site action starts from the vehicle's current position and selects nodes from the depot node set according to the estimated travel time, from smallest to largest. Candidate instances are generated for each station, where the estimated travel time is calculated by running the shortest path on the road segment subgraph with the "road segment travel time prediction value" as the edge weight; The action decision network outputs an action score for each candidate action in the above set of candidate actions and then normalizes it using softmax to obtain the policy probability of each candidate action. The action decision network adopts a two-layer fully connected structure with a hidden layer dimension of 128, uses ReLU as the activation function, and has a dropout rate of [missing value]. The output layer is a scalar score, and the score is calculated for each candidate action. To enable the value evaluation network to generate long-term return estimates based on the evaluation metrics described in claim 4, for each candidate action... First, calculate the travel time, empty mileage, timeout penalty, waiting penalty, and reassignment disturbance penalty for candidate actions, and form an action evaluation vector. The travel time for a candidate action is determined by the road segment sequence corresponding to that candidate action. The predicted travel time for each road segment is obtained by accumulating the predicted arrival time for each segment. The predicted arrival time for candidate actions is calculated from the "current time". The candidate action evaluation uses "travel time + corresponding waiting time + predicted loading / unloading waiting time at the corresponding terminal + corresponding service time" to calculate the timeout penalty. The timeout penalty is calculated as "the positive part of the difference between the estimated arrival time and the end time of the delivery window for the order targeted by the candidate action," thus only taking a positive value when the estimated arrival time exceeds the end time of the delivery window. The estimated loading / unloading waiting time for the candidate action evaluation is taken as the predicted loading / unloading waiting time at the terminal targeted by the candidate action, or zero if no terminal is involved, and is used to calculate the waiting penalty. The waiting penalty is calculated as "the positive part of the difference between the estimated arrival time and the end time of the delivery window for the order targeted by the candidate action." The loading and unloading waiting time is calculated as the sum of the waiting time corresponding to the candidate action, thus penalizing both queuing and voluntary waiting. The candidate action evaluation is obtained by accumulating the empty mileage segment by segment according to the segment length of the corresponding road segment sequence of the candidate action, and the empty mileage is included in the road segment where the vehicle load is zero, so that only the driving distance in the unloaded state is counted. The reassignment disturbance penalty is calculated based on the difference between the current order sequence before the reassignment action and the candidate order sequence after the reassignment action, and the difference is measured by the number of orders in the sequence whose order number has changed position, thus directly reflecting the order disturbance caused by the reassignment. The value evaluation network concatenates the vehicle path state vector and action evaluation vector as input and outputs a long-term revenue estimate. The value evaluation network employs a three-layer fully connected structure with hidden layer dimensions of 256 and 128 respectively, and the activation function is set to... The output layer is a scalar long-term return estimate and does not use an activation function. The relationship between the concatenation and the output is given by the formula: ; in The parameter is Value evaluation network for vehicles At any moment Execute candidate actions Long-term return estimates, This represents the forward mapping function of the value evaluation network. This represents the vector concatenation operation. Indicates candidate actions The action evaluation vector is composed of candidate action evaluation driving time, candidate action evaluation empty mileage, timeout penalty, waiting penalty and reassignment disturbance penalty in a fixed order, thereby completing the generation of candidate action set, strategy probability and long-term benefit estimate as described in step S3.

[0022] In this specific embodiment, S4 includes: At any moment Upon entering S4, for each vehicle node Read the candidate action set output by S3 and dynamic heterogeneous spacetime graphs The system calculates the predicted distribution of road segment travel time, order delivery window, service duration, and terminal loading / unloading wait time associated with the vehicle, and sorts the currently assigned but uncompleted orders for the vehicle according to the current transportation route to obtain the subsequent order sequence. ,in Indicates vehicle At any moment The set of candidate actions Indicates timestamp as Dynamic heterogeneous spacetime graph, Indicates vehicle The subsequent order sequence; For each candidate action Generate subsequent order sequences with updated actions in the time window congestion-loading / unloading wait joint risk masking module. ,in Indicates candidate actions Acting on The subsequent sequence, specifically, when candidate actions When performing the next order action, The first order is set to the order that the action points to, while keeping the rest in the same order. When the candidate action... To perform a detour, the order order remains unchanged, and the road segment sequence from the current location to the first order pickup / delivery location is replaced with the road segment sequence corresponding to the detour path. This applies to candidate actions. When a waiting action is performed, the order order remains unchanged, and the waiting time is added before the first order arrives. When performing a reassignment action, the reassigned order is transferred from... Remove and reconnect the preceding and following orders to form a new sequence when a candidate action is selected. When performing a return operation, insert the return destination station as the current destination node and set the first order after the return operation as the original node. The first order; When calculating the current risk margin, first determine The first order and recorded as Read the end time of the order delivery window for the first order and record it as... Read the service duration of the first order and record it as... Read candidate actions The corresponding waiting time is recorded as follows: Read candidate actions The predicted loading and unloading waiting time at the terminal corresponding to the first order is recorded as follows. and based on Map the road segment sequence from "current location to first order pickup / delivery location" to a cumulative distribution of the road segment travel time prediction distribution to obtain the target quantile value of travel time, and denot it as... ,in Indicates order The end time of the delivery window, Indicates order Service duration, Indicates candidate actions Waiting time, Indicates candidate actions The corresponding estimated loading and unloading waiting time, Indicates candidate actions The predicted travel time for the corresponding path is distributed at the target quantile. The quantile value at the point is obtained by summing the predicted distributions of each road segment according to the independence assumption to obtain the path mean and path variance, and then using the inverse cumulative distribution function of the normal distribution. The remaining time of the order delivery window is defined as: ; in Indicates the time of the first order. The remaining time of the time window is used to calculate candidate actions. Current risk margin: ; in Indicates the execution of candidate actions The current risk margin of the first order; when calculating the subsequent time window compression, the baseline sequence when no candidate action is executed is used. For comparison, respectively using The predicted arrival times of each subsequent order in the baseline sequence are calculated by recursively using the predicted travel time of the road segment and the predicted loading and unloading waiting time at the terminal, and the baseline remaining time window is obtained. Then, the same recursive method is used to calculate... The estimated arrival time of each subsequent order is calculated, and the remaining time window after the action is obtained. The positive parts of the "baseline remaining time window minus the remaining time window after the action" for each subsequent order are summed to obtain the subsequent time window compression amount. ,in Indicates candidate actions The time window compression amount for subsequent orders as a whole, in seconds; When calculating the probability of subsequent timeouts, based on Construct a stochastic simulation evaluation, with the number of simulations taken as... In each simulation, the travel time for each road segment in the path is sampled from its predicted travel time distribution. For nodes involving depots, the loading / unloading waiting time is sampled from their predicted loading / unloading waiting time distribution. The arrival time of each order is then recursively calculated according to the order sequence. If the arrival time of any order is greater than the end time of its delivery window, the simulation is considered a timeout, and the number of timeout simulations is divided by... Get candidate actions The subsequent timeout probability is denoted as ,in Indicates the number of simulations. Indicates the execution of candidate actions The probability of at least one timeout occurring in subsequent orders; Regarding threshold determination, the dynamic masking threshold is denoted as... Furthermore, the target quantile is obtained by adding the target quantile of the absolute value of the historical travel time prediction error to the target quantile of the historical loading and unloading waiting time prediction error. The target quantile is taken as... The absolute value of the historical travel time prediction error is obtained by taking the absolute value of the difference between the actual travel time and the predicted travel time in the historical sample. The absolute value of the historical loading and unloading waiting time prediction error is obtained by taking the absolute value of the difference between the actual loading and unloading waiting time and the predicted loading and unloading waiting time in the historical sample. This is calculated at midnight each day using near- The historical sample was recalculated once, among which This represents the threshold used to determine whether the current risk margin is insufficient. This indicates the number of historical days used for statistical thresholds; The masking probability threshold is denoted as Furthermore, the probability of timeout is determined based on the quantile value of the timeout probability in the historical order timeout sample, with a quantile value of 0.8. The historical order timeout sample is the set of orders that actually timed out, and the timeout probability is determined by the aforementioned random simulation assessment at the corresponding historical time. The previous replay was available; For each candidate action Feasibility assessment, when or When the candidate action is marked as a risky, non-actionable action and set to zero in the actionable action mask, and At that time, place one in the action mask to obtain the vehicle. At any moment The actionable masking vector; Regarding the calculation of risk scores, for each candidate action... Calculate risk score And used in subsequent steps The risk penalty coefficient is generated by weighting the current insufficient risk margin, the subsequent time window compression, and the subsequent timeout probability, and is calculated using the formula: ; in Indicates candidate actions The risk score and its range are This indicates the weight of the current insufficient risk margin. This indicates the weight of the subsequent window compression. The weights represent the probability of subsequent timeouts. Indicates the dynamic masking threshold. Indicates the current risk margin. Indicates the amount of subsequent window compression. This represents the upper bound of the compression used for normalization and takes... Indicates the probability of subsequent timeouts. This represents the operation of finding the maximum value. This represents the operation of taking the minimum value; in Remaining time of the first order window The quantile intervals of the remaining time in historical order timeout samples are determined and three constant mappings are taken. Take when it is not greater than the 0.25 quantile of the remaining historical time. ,when When the value is greater than the 0.25 quantile but not greater than the 0.75 quantile, take... ,when When greater than the 0.75 quantile, take Based on the number of subsequent orders Average of remaining time window for subsequent orders Determine and take the determined rule, when Time to take And when When Increase by 0.1, when Time to take And when When Increase by 0.1, of which express The number of subsequent orders in addition to the initial order. This represents the average remaining time window for the subsequent orders after the expected arrival time following the action; The variances of the predicted distributions of road segment travel time and loading / unloading waiting time are used to determine the candidate actions according to the established rules. The uncertainty index is obtained by summing the travel time variance of the corresponding route with the waiting variance of the corresponding station. When the uncertainty index is not less than its nearest neighbor... When the historical index reaches its 0.75 quantile, take Otherwise take ; Finally and Normalize the sum of the three values ​​to make the sum equal to 1, and then normalize the sum of the three values ​​for the vehicle. At any moment All candidate actions generate a risk score set; If and only if the vehicle When all elements in the action mask vector are zero, the repair trigger flag is set to true and output to step S5; otherwise, the repair trigger flag is set to false, thereby completing the determination of the action mask, risk score and repair trigger flag as described in step S4.

[0023] In this specific embodiment, S5 includes: At any moment Upon entering S5, for each vehicle node Read the candidate action set output in step S3 The policy probability of each candidate action and long-term return estimates And read the action mask and risk score output in step S4. With repair trigger flag, where Indicates vehicle At any moment The set of candidate actions express One of the candidate actions in the process, Indicates the parameters of the action decision network The given vehicle path state vector Selecting candidate actions under certain conditions The strategy probability, This represents the vehicle path state vector output in step S2. Indicates the parameters of the value evaluation network Given candidate actions Long-term return estimates, This indicates the candidate action risk score calculated in step S4, and the value range is... ; When the repair trigger flag is false, the action mask is expanded into a one-bit indicator for each candidate action and denoted as . ,in and Indicates candidate actions For actionable actions Indicates candidate actions For actions deemed risky and therefore inactive, a risk penalty coefficient is generated based on the risk score, and the strategy probability is masked and normalized to obtain the masked strategy probability. The calculation is as follows: ; in Indicates candidate actions The probability of the policy after masking. Represents an exponential function. Indicates the intensity of risk penalty and takes This represents the index of candidate actions used for summation and normalization. When determining the action to be performed, only The action is selected from the candidate action set and an action score is constructed. The action score is determined by the masked policy probability. Compared with long-term return estimates To jointly determine, specifically, to In all The normalized profit is obtained by performing minimum-maximum normalization on the candidate actions. Then calculate the overall score. The candidate action with the highest comprehensive score is selected as the action to be executed, so that the action to be executed satisfies the action masking constraint and takes into account both strategy preference and long-term benefits. When generating transportation routes, estimated arrival times, and scheduling instructions, the execution actions are parsed into "next segment sequence, action effective time, target order or target terminal, and action type," where the next segment sequence is determined through a dynamic heterogeneous spatiotemporal graph. The shortest path is obtained by running the algorithm on the sub-graph of the road segment with the predicted travel time of the road segment as the edge weight. The estimated arrival time is calculated according to the current time. The execution action is calculated by recursively adding the waiting time corresponding to the action, the predicted travel time of the next road segment sequence, the target order service time, and the predicted loading and unloading waiting time at the target depot, and then updating each subsequent order sequence. The dispatch instructions are sent to the vehicle terminal and the dispatch backend in the form of structured fields, which include vehicle number, action type, target order number or target depot number, next road segment sequence, estimated arrival time list, and map update version number. When the repair trigger flag is true, skip the masking strategy probability normalization and the above comprehensive scoring selection process, and regenerate a set of repair actions that are different from the original candidate action instances. Repair actions are generated in a fixed order and their constraints are verified one by one to ensure executability. The first step is to generate a waiting action with a recalculated waiting time, denoted as . And based on the current risk margin of the first order in step S4 With dynamic masking threshold Confirm and retrieve Then cut off to The interval is limited by an upper bound for waiting. Then, the "estimated arrival time after waiting" is calculated using the same arrival time recursion method as in step S4, and the waiting repair constraints are verified. The waiting repair constraints are: the current vehicle load capacity is not less than the load required for the next order; the estimated arrival time after waiting, after the sum of the waiting time, travel time, service time, and loading / unloading waiting predictions, is not greater than the end time of the next order delivery window; and the target depot loading / unloading waiting prediction is not greater than the depot waiting threshold. ,in Press near The 0.9 quantile of the historical station waiting time sample was recalculated at midnight each day; When the waiting-for-repair constraint is not met, a reassignment action is generated with the option to reselect a reassignment target, and the reassignment repair constraint is checked for each candidate reassignment target vehicle. The reassignment repair constraint is that the unexecuted order being reassigned meets the following conditions: the current vehicle's remaining load capacity is not less than the required load capacity of the order; the estimated arrival time determined based on the current location and the predicted road segment travel time is not greater than the end time of the order's delivery window; and the remaining time window of the subsequent orders after reassignment is not less than the dynamic masking threshold. And add the first reassignment repair action that meets the constraints. And stop generating reassignment repair actions; When both waiting for repair and reassignment repair are not feasible, an order exchange action is generated to recombine the order order sequence. This order exchange action is limited to subsequent order sequences. The former In each order, the orders are swapped pairwise, and the recursive calculation of step S4 is replayed one by one to obtain the remaining time window of the subsequent order sequence after the swap. The order swap repair constraint is that the remaining time window of each order in the subsequent order sequence after the swap is not less than the dynamic masking threshold. Furthermore, the exchanged orders must satisfy the following conditions: the vehicle's remaining load capacity is not less than the required load of the corresponding order, and the estimated arrival time is not greater than the end time of the corresponding order's delivery window. The order exchange action that satisfies these constraints and has the lowest total risk score will be added to the list of exchanged orders. ; When the current vehicle's remaining load capacity is less than the load required for the next order, or the remaining time window for subsequent orders is less than the dynamic masking threshold. Or the current target site service capacity is less than the site service capacity threshold. A temporary return action is triggered, and the service capacity of the selected nodes in the field is not less than [a certain value]. The stations are selected as the set of possible return destinations. The first station is chosen as the return destination station based on the estimated travel time, from smallest to largest, and a temporary return dispatch instruction is generated. Press near The 0.9 quantile of the sample of the number of vehicles served per unit time period at the historical station was recalculated at midnight each day; when Once generated, the first repair action in the set is executed directly according to fixed rules and converted into the vehicle's next segment sequence, estimated arrival time, and scheduling instructions. When consecutive... If both the generation and verification of the secondary repair action fail, a temporary return dispatch instruction will be generated and the vehicle will be guided to a location that meets the depot's service capacity threshold. The station terminated the repair cycle; Based on the rolling update process of transportation status data changes, a graph update event is generated and recorded when vehicle location updates, road segment congestion status updates, or terminal loading / unloading waiting queue updates are received. ,in The scheduling system assigns a monotonically increasing update version number to each graph update, including the event type, target object number, and event arrival timestamp. Based on the graph update event, update the corresponding vehicle node attributes, road segment node attributes, depot node attributes, and timestamped connection edges, and write them to the graph. The new version is then re-inputted into the spatiotemporal graph encoder of step S2 and the reinforcement learning decision model of step S3, and the action mask, risk score and repair trigger flag are recalculated in step S4. Then, the action determination process in step S5 generates a model with the updated version number. The transport route, estimated arrival time, and scheduling instructions are included, and instructions not yet executed in the previous version are covered.

[0024] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0025] This invention converts the state changes of vehicles, orders, road segments, and depots into graph structure inputs that can be processed by a spatiotemporal graph encoder through a dynamic heterogeneous spatiotemporal graph. Then, a reinforcement learning decision model outputs candidate actions and long-term revenue estimates based on the vehicle path state vector, enabling the transportation route generation process to be updated in a rolling manner as vehicle location, road segment congestion status, and depot loading and unloading waiting queues change.

[0026] This invention introduces a joint risk masking module for time window, congestion, and loading / unloading waiting during the action generation stage. The current risk margin, subsequent time window compression, and subsequent timeout probability of the candidate action are input into the final action selection and long-term penalty update. This enables the reinforcement learning strategy to not only focus on current reachability but also to avoid actions that compress the time window of subsequent orders in advance, thereby better achieving the technical effect of transportation route optimization.

Claims

1. A transportation route optimization method based on reinforcement learning, characterized in that, include: S1. Collect transportation status data of vehicles, orders, road segments and stations and construct a dynamic heterogeneous spatiotemporal map; S2. Input the dynamic heterogeneous spatiotemporal graph into the trained spatiotemporal graph encoder to extract features of congestion propagation, vehicle status, order urgency, and loading / unloading waiting, and obtain the vehicle path state vector; S3. Input the vehicle path state vector into the reinforcement learning decision model to generate a set of candidate actions, strategy probabilities, and long-term benefit estimates; S4. Before the final action selection, use the time window-congestion-loading / unloading waiting joint risk masking module to determine the action masking, risk score, and repair trigger flag; S5. Based on the action mask, risk score and repair trigger flag, adjust the strategy probability or generate repair actions, and combine the long-term benefit estimate to determine the execution action, generate the transportation route, expected arrival time and scheduling instructions, and continuously update the dynamic heterogeneous spatiotemporal map and transportation route as the transportation status data changes.

2. The transportation path optimization method based on reinforcement learning according to claim 1, characterized in that, S1 includes: Generate vehicle node attributes based on vehicle location, vehicle load, and assigned orders; Generate order node attributes based on order delivery time window, order required weight, service duration, pickup and delivery location, and order status; The road segment node attributes are generated based on the road segment congestion status, the predicted road segment travel time, and the predicted road segment travel time distribution. The station node attributes are generated based on the station loading and unloading waiting queue, station service capacity, loading and unloading waiting prediction value, and loading and unloading waiting prediction distribution; timestamped connection edges are generated based on the vehicle and order allocation relationship, the order and road segment passage relationship, and the vehicle and station entry and exit relationship, thus obtaining the dynamic heterogeneous spatiotemporal graph.

3. The transportation path optimization method based on reinforcement learning according to claim 1, characterized in that, S2 includes: The congestion status changes of adjacent road segments are aggregated in the dynamic heterogeneous spatiotemporal graph according to the timestamp order to generate congestion propagation features; Generate vehicle status features based on vehicle node attributes and assigned orders; Order urgency features are generated based on the order delivery window, estimated service start time, and service duration. The estimated service start time is determined based on the current transportation route, vehicle location, order sequence, and predicted road travel time. Loading and unloading waiting features are generated based on the terminal loading and unloading waiting queue and predicted loading and unloading waiting times. The congestion propagation features, vehicle status features, order urgency features, and loading / unloading waiting features are aggregated using graph attention to output the vehicle path state vector.

4. The transportation path optimization method based on reinforcement learning according to claim 1, characterized in that, S3 includes: inputting the vehicle path state vector into an action decision network to generate a candidate action set including the next order action, detour action, waiting action, reassignment action, and return action, and generating a strategy probability for each candidate action; inputting the vehicle path state vector and the candidate action set into a value evaluation network to generate the long-term revenue estimate based on the candidate action evaluation travel time, candidate action evaluation empty mileage, timeout penalty, waiting penalty, and reassignment disturbance penalty; wherein, the candidate action evaluation estimated arrival time is determined based on the corresponding candidate action, the road segment travel time prediction value in the dynamic heterogeneous spatiotemporal map, and the current order sequence; the timeout penalty is determined based on the difference between the candidate action evaluation estimated arrival time and the order delivery window end time; the candidate action evaluation estimated loading and unloading waiting time is determined based on the corresponding candidate action and the station loading and unloading waiting time prediction value; the waiting penalty is determined based on the candidate action evaluation estimated loading and unloading waiting time; and the reassignment disturbance penalty is determined based on the difference between the current order sequence before the reassignment action and the candidate order sequence after the reassignment action.

5. The transportation path optimization method based on reinforcement learning according to claim 1, characterized in that, S4 includes: inputting candidate actions, road segment travel time prediction distribution, order delivery window, service duration, loading / unloading waiting prediction value, and subsequent order sequence into the window-congestion-loading / unloading waiting joint risk masking module; the subsequent order sequence is obtained by sorting the incomplete orders in the current vehicle's assigned orders according to the current transportation route, and is updated according to the next order, detour segment, waiting duration, reassigned object, or return destination corresponding to the candidate action when evaluating the candidate action; for each candidate action, calculating the current risk margin and subsequent window compression amount after executing the candidate action, the current risk margin is the remaining time of the order delivery window and the waiting duration, the target quantile of the travel time prediction distribution, the service duration, and the loading / unloading waiting prediction value corresponding to the candidate action. The difference between the sum of the predicted loading and unloading wait values ​​is used to calculate the subsequent time window compression amount, which is the reduction in the remaining time window of the subsequent order after executing the candidate action. The subsequent time window compression amount, the road segment travel time prediction distribution, the loading and unloading wait prediction value, and the subsequent order delivery time window are used to determine the subsequent timeout probability. When the current risk margin is less than the dynamic masking threshold, or the subsequent timeout probability is greater than the masking probability threshold, the corresponding candidate action is marked as a risky and inactive action, and the active action mask, risk score, and repair trigger flag are generated. The dynamic masking threshold is determined based on the target quantile of the historical travel time prediction error and the target quantile of the historical loading and unloading wait prediction error, and the masking probability threshold is determined based on the timeout probability quantile in the historical order timeout sample.

6. The transportation path optimization method based on reinforcement learning according to claim 5, characterized in that, S5 includes: When the repair trigger flag does not indicate that all candidate action sets are marked as risky and inoperable actions, a risk penalty coefficient is generated based on the risk score. The operable action mask is multiplied by the strategy probability, and the multiplication result is corrected and normalized using the risk penalty coefficient to obtain the masked strategy probability. Then, the operable actions defined by the operable action mask, the masked strategy probability, and the long-term benefit estimate are used to determine the action to be executed, and the action to be executed is converted into the vehicle's next segment sequence, estimated arrival time, and scheduling instructions. When the repair trigger flag indicates that all candidate action sets are marked as risky and inoperable actions, the masked strategy probability normalization is skipped. Once the standard and normal execution actions are determined, a repair action distinct from the original candidate action instance is regenerated. The repair action includes a waiting action with recalculated waiting time, a reassignment action with reselection of reassignment object, an order exchange action with recombined order order sequence, or a temporary return action with reselection of return destination. After the repair action meets the repair constraints, the repair action is converted into the vehicle's next segment sequence, estimated arrival time, and dispatch instruction. When the regenerated repair action does not meet the repair constraints, the repair action continues to be regenerated, or a temporary return dispatch instruction is generated and the vehicle is guided to a depot that meets the depot service capacity threshold.

7. The transportation path optimization method based on reinforcement learning according to claim 3, characterized in that, The spatiotemporal map encoder is trained using historical transportation samples, which include historical dynamic heterogeneous spatiotemporal maps, historical vehicle routes, actual arrival times, and actual loading / unloading waiting times. During training, the encoding loss is composed of predicted arrival time error, road segment travel time error, and loading / unloading waiting time error. The encoding loss is jointly updated with the reward loss of the reinforcement learning decision model, so that the vehicle route state vector retains state information related to congestion propagation, order windows, and depot waiting times.

8. The transportation path optimization method based on reinforcement learning according to claim 5, characterized in that, The risk score is obtained by weighting the current risk margin, the subsequent time window compression, and the subsequent timeout probability. The weight of the current risk margin is determined by the quantile of the remaining time in the historical order timeout sample where the remaining time of the order delivery window falls. The weight of the subsequent time window compression is determined by the number of subsequent orders and the mean of the remaining time windows for subsequent orders. The weight of the subsequent timeout probability is determined by the variance of the predicted distribution of road segment travel time and the variance of the predicted distribution of loading and unloading waiting times. The predicted distribution of loading and unloading waiting times is generated based on historical station waiting time samples and the current station loading and unloading waiting queues.

9. The transportation path optimization method based on reinforcement learning according to claim 6, characterized in that, The repair constraints include: when the current vehicle's remaining load capacity is not less than the load capacity required for the next order, the estimated arrival time after waiting (combined with the predicted waiting time, travel time, service time, and loading / unloading waiting time) is not greater than the end time of the next order's delivery window, and the predicted loading / unloading waiting value at the depot is not greater than the depot's waiting threshold, the waiting action is determined to satisfy the repair constraints; when an unexecuted order simultaneously satisfies the following conditions: the current vehicle's remaining load capacity is not less than the load capacity required for that unexecuted order, the estimated arrival time determined based on the current location and road segment travel time prediction is not greater than the end time of the unexecuted order's delivery window, and the remaining time window of the reassigned subsequent order is not less than the dynamic masking threshold, the reassignment action is determined to satisfy the repair constraints; when the remaining time window of each order in the subsequent order sequence after the order exchange is not less than the dynamic masking threshold, and the exchanged orders respectively satisfy the corresponding vehicle... If the vehicle's remaining load capacity is not less than the corresponding order's required load capacity and the corresponding estimated arrival time is not greater than the corresponding order's delivery window end time, the order exchange action is determined to meet the repair constraints. If the current vehicle's remaining load capacity is less than the next order's required load capacity, or the remaining time window for subsequent orders is less than the dynamic masking threshold, or the current target depot's service capacity is less than the depot's service capacity threshold, a temporary return action is triggered. If the depot's service capacity at the reselected return destination is not less than the depot's service capacity threshold, the temporary return action is determined to meet the repair constraints. The next order's required load capacity, the unexecuted order's required load capacity, and the corresponding order's required load capacity are all determined by the order node attributes. The depot waiting threshold and the depot service capacity threshold are determined based on the target quantile of historical depot waiting time and the target quantile of historical depot service vehicle count per unit time period, respectively.

10. The transportation path optimization method based on reinforcement learning according to claim 6, characterized in that, Step S5 further includes: when receiving vehicle location updates, road segment congestion status updates, or terminal loading / unloading waiting queue updates, generating a graph update event; updating the corresponding vehicle node attributes, road segment node attributes, terminal node attributes, and timestamped connection edges according to the graph update event, and recording the update version number; after re-inputting the updated dynamic heterogeneous spatiotemporal graph into the spatiotemporal graph encoder and the reinforcement learning decision model, re-determining the action mask, risk score, and repair trigger flag through the time window-congestion-loading / unloading waiting joint risk masking module, and generating a transportation route, estimated arrival time, and scheduling instructions with the update version number according to the action determination process in step S5.

Images