A heterogeneous vehicle fleet fuel transportation path planning method based on reinforcement learning

Abstract

The application discloses a heterogeneous vehicle fleet fuel transportation path planning method based on reinforcement learning, and belongs to the technical field of logistics distribution and intelligent scheduling. The method comprises the following steps: constructing a distribution area road network and a heterogeneous vehicle fleet simulation environment; generating time-driven dynamic distribution demand; constructing a collaborative scheduling strategy network and a multi-component reward function based on a proximal policy optimization algorithm; planning a vehicle driving path and establishing multiple types of constraints; and outputting an optimal path planning strategy through simulation iteration and strategy network training. The application models the heterogeneous vehicle fleet scheduling as a Markov decision process, adopts a hierarchical action space and a multi-component reward function, effectively realizes collaborative scheduling of large fuel tank trucks and small fuel distribution vehicles, reduces the total driving cost and improves the distribution efficiency under the constraints of meeting the capacity, regional access, multi-trip connection and the like.

Description

Technical Field

[0001] This invention belongs to the field of logistics distribution and intelligent scheduling technology, and in particular relates to a method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning. Background Technology

[0002] Optimizing petroleum product delivery routes is crucial for reducing logistics costs and improving transportation efficiency, and is one of the key technologies for enhancing petroleum logistics operations. Existing research mainly focuses on delivery optimization for single vehicle types, using standard vehicle routing problem models to plan routes for homogeneous fleets. However, in real-world urban petroleum delivery scenarios, large fuel tankers and small fuel delivery vehicles often need to work together, which involves constraints related to vehicle type differences, making it difficult to directly apply traditional optimization models.

[0003] With the development of artificial intelligence technology, reinforcement learning has shown great potential in the field of combinatorial optimization. Compared with traditional mathematical programming methods, reinforcement learning learns optimal decision-making strategies through continuous interaction with the environment, effectively handling high-dimensional state spaces and complex constraints. It has achieved significant results in problems such as vehicle routing and traffic signal control. Especially in multi-vehicle collaborative delivery scenarios, reinforcement learning can adaptively learn the cooperation mechanisms between vehicle types, avoiding the problem of traditional heuristic methods easily getting trapped in local optima.

[0004] For the complex optimization problem of multi-vehicle collaborative scheduling, existing reinforcement learning-based methods mainly address this problem through the following three technical routes: The first type of method is an end-to-end learning framework based on policy gradient and attention mechanisms. This type of method constructs a sequence-to-sequence neural network model, models the vehicle routing problem as a sequential decision-making process, and directly learns constructive heuristic policies using policy gradient algorithms. Kool et al. (2019) proposed a Transformer-based attention model, using the REINFORCE algorithm in conjunction with a greedy baseline for training, which showed superior performance to traditional heuristic methods on various path planning problems. Li et al. (2021) further proposed a heterogeneous attention model, which, through a two-layer structure of vehicle selection decoder and node selection decoder, systematically applied deep reinforcement learning to the vehicle routing problem of heterogeneous fleets for the first time, effectively handling the constraints of vehicle capacity differences. Xu et al. (2021) proposed a reinforcement learning method based on a multi-relationship attention mechanism, which captures the complex relationships between nodes through graph neural networks, improving the quality of path planning. However, the above methods mostly use basic strategy gradient methods such as A2C or REINFORCE, which have problems such as unstable training and low sample efficiency, and are difficult to directly handle complex operational constraints in urban delivery such as regional restrictions and multi-trip connections.

[0005] The second category of methods is based on multi-agent reinforcement learning extensions tailored to the characteristics of heterogeneous vehicle fleets. These methods design specialized state representations and policy network structures to address differences in vehicle capacity, speed, and other attributes. Zhang et al. (2020) proposed a multi-agent reinforcement learning-based method for solving multi-vehicle routing problems, employing soft time window constraints for modeling and iteratively generating multi-vehicle paths through an encoder-decoder framework and attention layers, achieving a good balance between computational efficiency and solution quality. Li et al. (2021) proposed a deep reinforcement learning method for routing problems involving vehicles with heterogeneous capabilities, using a multi-agent architecture to train independent policies for each vehicle type and achieving collaborative decision-making through a centralized value network. Lin et al. (2021) proposed an end-to-end deep reinforcement learning framework for electric vehicle routing problems to handle time window constraints and charging scheduling, using a policy gradient method to optimize vehicle routing and charging decisions. However, pure end-to-end learning methods lack theoretical guarantees for solution feasibility and are prone to generating invalid solutions when dealing with complex constraints such as multi-vehicle coordination, restricted area constraints, and multi-trip connections. Furthermore, existing research has failed to systematically characterize the dynamic coordination mechanism between large fuel tankers and small fuel delivery vehicles.

[0006] The third category of methods is optimization enhancement based on the fusion of hybrid strategies and domain knowledge. These methods improve solution quality and computational efficiency by combining reinforcement learning with traditional optimization techniques such as exact algorithms and local search. Hottung and Tierney (2020) proposed a neural large neighborhood search method, utilizing reinforcement learning to learn the selection strategy for disruptive operators and combining it with a heuristic repair mechanism to solve large-scale CVRP problems. Mak et al. (2023) proposed a deep multi-agent reinforcement learning method for fair collaborative vehicle paths, achieving dynamic electric vehicle charging pricing through multi-agent graph convolutional reinforcement learning. However, existing hybrid methods are mostly designed for single vehicle models or static scenarios, failing to systematically characterize the dynamic connection mechanisms and differentiated constraints in multi-vehicle collaboration. Especially in the oil delivery scenario, existing methods fail to effectively handle multiple constraints from various perspectives, such as restricted areas for large fuel tankers, end-point service fees for small fuel delivery vehicles, and driver safety operating time, and lack a deep reinforcement learning framework for multi-vehicle collaboration. In summary, there is an urgent need for a multi-vehicle collaborative optimization method based on PPO deep reinforcement learning. This method should be able to adaptively capture vehicle collaborative patterns by leveraging the powerful representation learning capabilities of neural networks, and ensure training convergence through a stable policy optimization mechanism. Under the premise of meeting actual operational requirements such as restricted areas, capacity constraints, and vehicle connection, this method can achieve efficient optimization of multi-vehicle collaborative delivery of petroleum products. Summary of the Invention

[0007] To address the problems existing in the prior art, the present invention aims to provide a method for fuel transportation route planning for heterogeneous fleets based on reinforcement learning. This method unifies vehicle scheduling and route planning into a sequential decision problem. Through interactive learning between the agent and the simulation environment, it adaptively optimizes the delivery strategy of heterogeneous fleets, overcoming the technical bottlenecks of high computational complexity in traditional exact algorithms and unstable solution quality in heuristic methods.

[0008] To solve the above problems, the present invention adopts the following technical solution: A method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning includes the following steps: Step 1: Construct the basic simulation environment for the road network and heterogeneous vehicle fleet in the delivery area; The heterogeneous fleet includes large fuel tankers and small fuel delivery vehicles; Road network data for the target city or region is obtained from a public map database and imported into a traffic simulation platform to recreate the road, intersection, and lane structure. In this simulation environment, the locations of oil depots, customer nodes, and transit gas stations are set, and the loading capacity limits, speed limits, area access ranges, and minimum connection times for multi-trips of heterogeneous fleets are configured, thus forming the basis of the heterogeneous fleet simulation environment. Step 2: Generate time-driven delivery demand; Based on the heterogeneous fleet simulation environment constructed in step 1, demand arrival rules are set according to the consumption characteristics of petroleum products and historical order data. During the simulation process, fuel delivery demands with timestamps are generated at each customer node according to the preset time step and predefined time sequence demand table. Demands that have not yet been completed are stored in the delivery queue according to the customer node, thereby forming the real-time dynamic delivery demand required by the collaborative scheduling strategy. Step 3: Construct the reward function and cooperative scheduling policy network for the PPO algorithm based on the near-end policy optimization; Based on the real-time dynamic delivery demand generated in step 2, the delivery process of the heterogeneous fleet is abstracted into a Markov decision process. A hierarchical action space consisting of three types of sub-actions—vehicle selection, customer allocation, and trip termination—is designed. A cooperative scheduling policy network with a single-agent Actor-Critic architecture is constructed under the Proximal Policy Optimization (PPO) algorithm. A multi-component reward function is defined to train the network, enabling the network to output scheduling decisions based on the current global state. Step 4: Plan vehicle travel routes and establish constraints; Based on the scheduling decision output by the collaborative scheduling strategy network in step 3, the shortest path algorithm built into the traffic simulation platform is called to plan the driving path from the current location of the vehicle to the customer node and from the customer node back to the oil depot for the selected vehicle. At the same time, constraints are established based on the loading capacity limit, speed limit, area access range, minimum connection time for multiple trips and uniqueness of customer service of the heterogeneous fleet configured in step 1. Decisions that violate any constraint are rejected and penalized to ensure the physical feasibility of scheduling decisions. Step 5: Simulation iteration and strategy output; In the simulation environment built in step 1, the delivery demand from step 2, the collaborative scheduling strategy network from step 3, the vehicle driving paths planned in step 4, and the established constraints are integrated. The simulation clock is advanced in discrete time steps, and scheduling decisions, shortest paths are calculated, vehicle operating status is updated, and constraints are verified in a loop. The PPO algorithm is used to iteratively update the parameters of the collaborative scheduling strategy network until convergence is achieved on key performance indicators such as total driving cost, delivery completion time, and vehicle type collaboration efficiency. Finally, the trained heterogeneous fleet fuel transportation route planning strategy is obtained.

[0009] Preferably, in step 1: The constraint parameters for the client node include: Each customer node's single fuel demand is considered an indivisible service unit. Demand generation is driven solely by preset time rules or historical demand curves and is unrelated to real-time scheduling results. The geographical location, service priority, and acceptable latest service time of each customer node are stored in the road network attribute library. The constraint parameters for heterogeneous vehicle fleets include: (1) Restrictions on large fuel tank trucks; For the large fuel tanker truck configured in step 1, its maximum fuel carrying capacity is The maximum driving speed is increased to [a certain value]. It is only permitted to operate on roads that meet the regional access requirements, and the actual load when performing delivery tasks must not exceed 95% of the rated capacity; (2) Constraints on small fuel-powered delivery vehicles; For the small fuel delivery vehicle configured in step 1, its maximum fuel carrying capacity is Lift( The maximum driving speed is limited to ( ( ), allowing entry into urban areas where large fuel tankers are prohibited to complete last-mile delivery, but the loading rate must not exceed the corresponding rated capacity when performing the task; (3) Multi-stroke connection constraints; In the simulation environment established in step 1, each vehicle must undergo a connection time of no less than the minimum required after completing a delivery trip and returning to the oil depot. The stops are used for vehicle inspection, refueling, and driver rest. The next delivery trip can only start after the minimum connection time requirement is met, so as to realize the safe and dynamic reuse of vehicle resources between multiple trips.

[0010] Preferably, step 2 includes the following steps: Step 2-1: Setting the simulation period and time step; Define the service runtime segment covered by a single simulation and select a discrete time step of fixed length to advance the timing framework of the simulation clock, thereby providing a time reference for subsequent demand evolution and scheduling response processes; Step 2-2: Generate mixed requirements; Based on the time sequence framework set in step 2-1, this step generates delivery requests at each customer node according to the preset demand arrival rules. For regular customer nodes, a fixed time interval is used to generate requests, while for key customer nodes such as large industrial parks or transportation hub gas stations, requests are released based on a pre-arranged time sequence table or actual order logs. Each generated delivery request record includes at least the demand quantity, customer node number, time of demand generation, priority, and latest service time. Steps 2-3: Demand queue management; First, the delivery requests generated in step 2-2 are sorted according to customer priority and the time the requests were generated, and written into the queue of pending delivery requests of the corresponding customer node. All requests that have not yet been served by vehicles are continuously accumulated in this queue until a request is successfully delivered, at which point it is removed from the corresponding queue. The real-time content and status of the queue of pending delivery requests of each customer node thus formed constitute the core component of the customer node status on which the collaborative scheduling strategy network makes decisions in step 3.

[0011] Preferably, the construction of the reward function and cooperative scheduling policy network for optimizing the PPO algorithm based on the near-end policy in step 3 includes the following technical elements: Step 3-1: Markov decision process modeling; Combining the simulation environment established in step 1 with the dynamic requirements generated in step 2, the heterogeneous fleet delivery process is abstracted into a Markov decision process. A single-agent PPO architecture is adopted to uniformly handle the three types of decisions: heterogeneous vehicle selection, customer node assignment, and trip termination timing. Step 3-2: Layered motion space design; A three-layer hierarchical action space is constructed for the Markov decision process in step 3-1, which consists of the following three types of sub-action combinations: (1) Vehicle selection action Used to select the specific vehicle from the currently available fleet to perform the delivery task for this round. The selected vehicle must meet the loading capacity and regional access constraints corresponding to its vehicle type. (2) Customer allocation action : Used to assign the next customer node for a selected vehicle. This selection must take into account the vehicle’s remaining load capacity, the customer node’s demand priority, and the estimated travel time. (3) Termination of journey Used to determine whether a vehicle should continue serving the next customer or return to the oil depot to end the current journey after completing the current customer service; The above three sub-actions are combined in the order of vehicle selection - customer allocation - trip termination to form a complete dispatching action; Step 3-3: State space construction; Based on the simulation environment defined in step 1 and the delivery requirements generated in step 2, a joint state vector is constructed as the input to the policy network. This state vector includes the following components: (1) Customer node status: including the amount of unfulfilled demand, demand priority, service completion marker and spatial coordinates of each customer node; (2) Vehicle status: including vehicle type, current location, remaining load, current task execution status and end time of the previous journey for each vehicle; (3) Fleet coordination status: including at least the estimated distance matrix between vehicles and between vehicles and customer nodes, as well as the regional access identifiers for different vehicle types; (4) Global environment status: including the number of unfulfilled requirements in the system at the current simulation moment; Steps 3-4: PPO strategy network architecture design; A cooperative scheduling strategy network is constructed using an Actor-Critic structure: (1) Actor network: Input the joint state vector constructed in step 3-3, and output the complete action probability distribution in the hierarchical action space defined in step 3-2; (2) Network: Adopting the same network structure as the Actor network, the input is the joint state vector constructed in step 3-3, and the output is an estimate of the expected cumulative reward for that state; Steps 3-5: Define the objective function for near-end strategy optimization; The policy network parameters described in steps 3-4 are updated using a near-end policy optimization objective function with truncation terms. The objective function is defined as follows: In the formula: It is the probability ratio of the new strategy to the old strategy. It is the estimate of the dominance function. It is a truncation hyperparameter; Steps 3-6: Define the multi-component reward function; The multi-component reward function described in step 3 includes four components: service reward, cost reward, collaboration reward, and completion reward, with immediate rewards... Defined as: ; In the formula As a service reward; As a cost incentive; For collaborative rewards; To complete the reward; (1) Service Rewards Used to measure the timeliness of customer service, it provides a positive reward for each successful delivery, defined as: ; In the formula: For at any time The group of customers whose deliveries were successfully completed; The base reward value earned for successfully serving a single customer; For customers The waiting time between when a demand is generated and when a service is provided; The waiting penalty coefficient; (2) Cost incentive Used to penalize transportation costs, it applies a negative reward to the additional travel distance and vehicle activation caused by each scheduling decision, defined as: ; In the formula: For the time The total additional vehicle travel distance resulting from scheduling decisions, For the current moment The total number of vehicles involved in the delivery mission; The fixed cost of starting up a vehicle, , These are the weighting coefficients for driving distance cost and vehicle activation cost, respectively. (3) Collaborative rewards This is used to characterize the degree of coordination between large fuel tankers and small fuel delivery vehicles. It consists of two parts: rewards that encourage task handover and penalties for load imbalance. It is defined as follows: ; In the formula: For at any time Number of task handovers between large fuel tankers and small fuel delivery vehicles; , These represent the average load rates of large fuel tankers and small fuel delivery vehicles at the current moment, respectively. , These are the corresponding weighting coefficients; (4) Completion Reward: This is used to reflect the overall task completion status. When the delivery needs of all customer nodes are met within the simulation period, a one-time large positive reward is given, defined as: ; In the formula, For a moment The number of unfulfilled customer requests remaining in the system; A preset task completion reward value is set. In addition, when a scheduling action violates any of the following constraints: capacity limits, regional access, minimum connection time for multiple trips, or uniqueness of customer service, a fixed negative value is set as a penalty for the immediate reward of that decision step. ,in, These are the preset penalty parameters.

[0012] Preferably, step 4 specifically includes the following steps: Step 4-1: Path calculation; Based on the scheduling decision output in step 3, the shortest path calculation module is called in the traffic simulation platform to plan the driving path from the current location to the customer node for the selected vehicle. Step 4-2: Constraint check; Before executing the specific route planned in step 4-1, the scheduling action is constrained based on the loading capacity limit, speed limit, area access range and minimum connection time of the heterogeneous fleet configured in step 1. If any scheduling action violates any constraint, it is directly rejected. Step 4-3: Multi-trip management; After a vehicle successfully completes the delivery task that has passed the inspection in step 4-2 and returns to the oil depot, its journey end time is recorded. Based on the minimum connection time parameter for multiple trips configured in step 1, the earliest available time for the vehicle to be dispatched next is calculated. Before this time arrives, the vehicle is marked as unavailable in the dispatch system.

[0013] Preferably, step 5 specifically includes the following steps: Step 5-1: Simulation environment initialization; Load the static information and regional access rules defined in step 1, such as road network, oil depot, customer node, transfer gas station, etc., to initialize the heterogeneous fleet. At the same time, load the initial parameters of the initial delivery demand queue established in step 2 and the collaborative scheduling strategy network constructed in step 3 to complete the startup preparation of the entire simulation training system. Step 5-2: Simulation time progression and decision triggering; In the simulation environment initialized in step 5-1, the simulation clock is advanced according to the preset discrete time step. At each time step, the system makes a judgment based on the real-time status of the vehicle. When the vehicle is in an idle state or has arrived at a customer node and completed the service, a scheduling decision is triggered. At this time, the cooperative scheduling strategy network constructed in step 3 is called to generate a complete scheduling instruction containing vehicle selection, customer allocation and trip termination actions based on the current global state. Step 5-3: Perform scheduling and path updates; For the scheduling instruction generated in step 5-2, the constraint check described in step 4 is first called to verify its feasibility. After the instruction passes all constraint checks, it is executed in the simulation environment. The execution process includes: calling the shortest path calculation module of the traffic simulation platform to drive the selected vehicle along the planned path; synchronously updating the vehicle's position, cumulative travel distance and remaining load capacity; and removing the customer demand that has been successfully served from the queue of undelivered demand established in step 3. Step 5-4: Reward Calculation and Experience Collection; After executing the scheduling and updating the environment state in step 5-3, the immediate reward obtained by executing the scheduling decision is calculated based on the multi-component reward function defined in step 3. Then, the key information of this decision-making process—the state before the decision, the scheduling action executed, the immediate reward obtained, and the new state of the environment after execution—is stored as a complete experience sample in the experience replay buffer. Step 5-5: Policy Update and Training Termination; When the amount of data in the experience replay buffer accumulated in step 5-4 reaches the preset batch size, a batch of data is sampled from it, and the parameters of the cooperative scheduling strategy network constructed in step 3 are updated one or more times using the PPO algorithm. This process forms a loop with steps 5-2 to 5-4, continuously optimizing the strategy. When the preset simulation termination condition is met, training stops. Finally, the various performance indicators recorded during the training process are output, including total driving cost, vehicle utilization rate, delivery completion time, and vehicle type coordination efficiency, to evaluate the overall performance of the path planning method.

[0014] The beneficial effects of this invention are: Compared with the prior art, the advantages of this invention are: (1) This application proposes a method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning. This method models the scheduling process of heterogeneous fleets as a Markov decision process. Under the Proximal Policy Optimization (PPO) framework, a cooperative scheduling strategy network with a single agent Actor-Critic architecture is constructed. The network parameters are optimized through simulation iteration. This method realizes the cooperative scheduling of large fuel tankers and small fuel delivery vehicles under various operational constraints such as regional access and multi-trip connection. It can significantly reduce the total driving cost and shorten the delivery completion time. (2) This application designs a hierarchical action space consisting of three types of sub-actions: vehicle selection, customer allocation, and trip termination. This design unifies the three types of decisions: heterogeneous vehicle selection, customer node assignment, and trip termination timing, and outputs end-to-end decisions through a single collaborative scheduling strategy network. This effectively avoids the information loss and local optima problem caused by hierarchical decision-making in traditional methods. (3) In this application, a multi-component reward function is defined in the PPO algorithm, which includes service reward, cost reward, collaboration reward and completion reward. The specially designed collaboration reward component can effectively guide the policy network to learn the dynamic collaboration mechanism between large fuel tanker trucks and small fuel delivery trucks by quantifying the number of task handovers between vehicle types and balancing the average load rate, without relying on preset fixed collaboration rules. (4) This application deeply integrates the traffic simulation platform with the PPO reinforcement learning algorithm to form a complete simulation training system. In this system, the traffic simulation platform is responsible for providing the road network foundation, executing the shortest path algorithm calculation, updating the vehicle operating status and verifying constraints, while the collaborative scheduling strategy network under the PPO framework is responsible for making macro-schedule decisions based on the joint state vector. The two work together to achieve the optimization of strategy decision-micro simulation-feedback learning. Attached Figure Description

[0015] Figure 1 This is a flowchart of the collaborative delivery and transportation route planning method for large fuel tankers and small fuel delivery vehicles based on reinforcement learning, as described in this application. Figure 2 This is a schematic diagram of the training process of the PPO scheduling policy network in this application; Figure 3 A schematic diagram of the state input and hierarchical action output structure. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] I. Symbol Explanation: The main symbols involved in this embodiment of the invention are explained as follows: A collection of large fuel tank trucks; A collection of small, fuel-powered delivery vehicles; The maximum fuel capacity of a single large fuel tanker truck; : The maximum fuel carrying capacity of a single small fuel tanker truck; The maximum speed of a large fuel tanker truck; Maximum speed of small fuel tanker trucks; The minimum connection time that a vehicle must stay at after completing a journey and returning to the oil depot; End time of the journey; : The i-th customer requirement record, in which Number the customer node For demand, When demand arises Prioritize the requirements. This is the latest service time; Simulation step size; : Total duration of a single simulation run; : A set of nodes; Road set; :time At that time, it is the set of delivery requests accumulated across all customer nodes in the system that have not yet been served; :time At that time, the global state vector of the simulation environment is used as the input to the cooperative scheduling policy network; At time t, the cooperative scheduling strategy network determines the state. The output scheduling action; Action to be executed at time t Then, the immediate total reward for environmental feedback; These are divided into four reward components: service reward, cost reward, collaboration reward, and completion reward. : Actor policy network, parameters are , used to output the probability distribution of actions; Critic value network, parameters are , used to assess the value of a state; PPO cutoff coefficient.

[0018] II. Example 1: like Figure 1-3 As shown, the present invention provides a technical solution: a method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning.

[0019] This embodiment uses the fuel delivery business of a certain city as a background to illustrate the implementation process of the method of the present invention in detail. The method trains an intelligent strategy in a simulation environment through five core steps that can coordinate the scheduling of large fuel tankers and small fuel delivery vehicles.

[0020] Step 1: Construct the basic simulation environment for the road network and heterogeneous vehicle fleet in the delivery area; 1. Road network data acquisition and import: Obtain road network data of the target city area from the public map database OpenStreetMap, and import the data into the traffic simulation platform SUMO to accurately recreate the topology of roads, intersections and lanes.

[0021] 2. Key Node Deployment: In the simulated road network, determine the locations of oil depots, customer nodes, and transit gas stations, and set their type attributes: (1) Oil depot: 1, serving as the starting and return point for all vehicles; (2) Customer nodes: N, representing gas stations that need fuel replenishment; (3) Transfer gas stations: 2.

[0022] Each customer node's single fuel demand is considered an indivisible service unit. Demand generation is driven solely by preset time rules or historical demand curves and is unrelated to real-time scheduling results. The customer node's geographical location, service priority, and acceptable latest service time are stored in the road network attribute library.

[0023] 3. Heterogeneous Fleet Parameter Configuration: Configure the loading capacity limits, speed limits, area access ranges, and minimum connection times for multiple trips of the heterogeneous fleet, thereby forming the foundation of the heterogeneous fleet simulation environment. (1) Large fuel tanker truck collection Configuration Vehicles, set parameters for each vehicle: maximum capacity Rise, top speed The actual load during mission execution must not exceed According to intercity traffic regulations, restricted areas are designated in the simulated road network, and large fuel tankers are only allowed to travel on roads in non-restricted areas; (2) Collection of small fuel delivery vehicles Configuration Vehicles, set parameters for each vehicle: maximum capacity Rise, top speed It is permitted to travel on the entire road network; 4. Multi-trip connection constraint configuration: Set a uniform minimum connection time. Within minutes, any vehicle returning to the oil depot after completing a journey must stop. Only then can the time be rescheduled for vehicle inspections, refueling, and driver rest, thereby enabling the safe and dynamic reuse of vehicle resources across multiple journeys.

[0024] Step 2: Generate time-driven delivery demand; Step 2-1: Simulation Period and Time Step Setting: Set a single simulation to cover one working day, i.e. Minutes, the simulated clock operates in fixed steps. Minutes advance; Step 2-2: Hybrid Requirements Generation: At each simulation time step, requirements are generated at each customer node according to preset rules. (1) Regular customers: Demand is generated on a fixed time interval using a Poisson distribution; (2) Key customers (such as gas stations in large industrial parks or transportation hubs): Release concentrated demand during specific peak periods according to the predefined time-series demand schedule.

[0025] Each generated requirement record Includes: Customer node number Demand When demand arises Priority Latest service time .

[0026] Steps 2-3: Demand Queue Management: The system maintains a queue of deliverable demands for each customer node. Newly generated demands are sorted by their priority and generation time and then inserted into the corresponding queue. When a demand is successfully served by a vehicle, it is immediately removed from the queue. The real-time status of all queues together constitutes the customer node status, which is a key input for subsequent decisions.

[0027] Step 3: Construct the reward function and cooperative scheduling policy network for the PPO algorithm based on the near-end policy optimization (e.g., Figure 3 (as shown) Step 3-1: Markov Decision Process Modeling: The heterogeneous fleet delivery process is abstracted into a Markov decision process, and a policy network observes the environmental state. Output action The environment shifts to a new state And return the reward .

[0028] Step 3-2: Layered Motion Space Design: Design a layered motion space consisting of three layers of sub-motion sequences. (1) Vehicle selection action Select a vehicle from the currently available fleet; (2) Customer allocation action For the selected vehicle, select one of its available customer nodes as the next service target; (3) Termination of journey The decision-making process involves determining whether a vehicle should continue serving the next customer or return to the oil depot to end its journey after serving the current customer.

[0029] Step 3-3: State Space Construction: Construct a joint state vector ,Include: (1) Customer node status: the amount of unfulfilled demand, demand priority, service completion marker and spatial coordinates of each customer node; (2) Vehicle status: vehicle type, current location, remaining load, current task execution status and end time of the previous journey for each vehicle; (3) Fleet coordination status: estimated distance matrix from vehicles to each customer node, and regional access identifiers for different vehicle types; (4) Global environment status: The number of unfulfilled requirements in the system at the current simulation time.

[0030] Steps 3-4: PPO Policy Network Architecture Design: Construct a cooperative scheduling policy network for a single-agent Actor-Critic: (1) Actor network: Input state Output the probability distribution of the three sub-actions: vehicle selection, customer allocation, and trip termination. (2) Network: Same network structure as the Actor network, input state Output a scalar value representing the expected cumulative reward of the current state.

[0031] Steps 3-5: Definition of the near-end policy optimization objective function: The near-end policy optimization objective function with truncation terms is used to update the policy network parameters: In the formula: It is the probability ratio of the new strategy to the old strategy. It is the estimate of the dominance function. It is a truncation hyperparameter.

[0032] Steps 3-6: Definition of a multi-component reward function: Design a multi-component reward function. To guide the agent's learning: (1) Service Rewards Encourage instant service. In the formula: yes A collection of customers who are constantly being served It is a basic reward. Customer Waiting time It is the waiting penalty coefficient; (2) Cost incentive Penalize driving and vehicle usage costs. In the formula: It is the increase in driving distance, It is the total number of vehicles participating in the delivery mission. The fixed cost of starting up a vehicle , These are the weighting coefficients for driving distance cost and vehicle activation cost, respectively. (3) Collaborative rewards Promote collaboration between vehicles of different sizes. In the formula: It is the number of task handovers, and It is the average load factor of large fuel tankers and small fuel delivery vehicles. and These are the weighting coefficients; (4) Completion Reward : Incentivize the completion of all tasks, when hour, Otherwise, it is 0; (5) Constraints and penalties: If an action violates any hard constraints, a penalty will be imposed directly. The punishment.

[0033] Step 4: Plan vehicle travel routes and establish constraints; Step 4-1: Path Calculation: When the policy network outputs an action... The traffic simulation platform calls the shortest path calculation module to calculate the shortest feasible path from the current location to the customer node and the shortest path from the customer node back to the oil depot for the selected vehicle.

[0034] Step 4-2: Constraint Check: Before executing the route, check the loading capacity limit, speed limit, area access range, and minimum connection time of the heterogeneous fleet. If any check fails, the dispatch instruction is rejected and the penalty defined in the reward function is applied.

[0035] Step 4-3: Multi-trip Management: After a vehicle successfully serves a customer and returns to the oil depot, record the trip end time. The vehicle was Marked as unavailable for the period of time, until It only becomes idle again at a certain time and can be rescheduled.

[0036] Step 5: Simulation Iteration and Policy Output: Through the interactive loop between the simulation platform and the cooperative scheduling policy network, the final policy (such as...) is trained. Figure 2 (as shown) Step 5-1: Simulation Environment Initialization: Load the static information and regional access rules of the road network, oil depot, customer nodes, transit gas stations, etc. constructed in the previous steps, as well as the initial parameters of the pending delivery demand queue generated in Step 2 and the collaborative scheduling strategy network constructed in Step 3.

[0037] Step 5-2: Simulation Time Advancement and Decision Triggering: The simulation clock starts from 0 and advances in steps of 0. In the process of advancement, at each time step, the system checks the status of all vehicles. When a vehicle is idle or has just completed service at a customer node, a scheduling decision is initiated. At this time, the current global state is updated. Input the cooperative scheduling policy network, and the network outputs a complete scheduling action. .

[0038] Step 5-3: Perform scheduling and path updates: Perform scheduling actions If the constraint check in step 4 passes, it is executed in the simulation environment. The selected vehicle is driven to travel along the planned shortest path, and the vehicle's position, cumulative travel distance, and remaining load are updated. Upon reaching the customer node, the service is completed, the customer demand queue is updated, and the environment enters a new state. .

[0039] Step 5-4: Reward Calculation and Experience Collection: Calculate the immediate reward obtained by executing this scheduling decision. The key information in this decision-making process includes: the state before the decision, the scheduling actions performed, the immediate rewards obtained, and the new state of the environment after execution. Store in the experience replay buffer.

[0040] Step 5-5: Policy Update and Training Termination; When the amount of data in the experience replay buffer reaches the preset batch size, a batch of data is sampled from it. Using the PPO algorithm, the parameters of the cooperative scheduling strategy network are updated in one round based on the sampled data and the defined objective function. Steps 2 to 5 are repeated until the strategy performance converges. After training, the final cooperative scheduling strategy network is saved. This network is the heterogeneous fleet fuel transportation route planning strategy output by this method that can be used for actual online planning.

[0041] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Construct the basic simulation environment for the road network and heterogeneous vehicle fleet in the delivery area; The heterogeneous fleet includes large fuel tankers and small fuel delivery vehicles; Road network data for the target city or region is obtained from a public map database and imported into a traffic simulation platform to recreate the road, intersection, and lane structure. In this simulation environment, the locations of oil depots, customer nodes, and transit gas stations are set, and the loading capacity limits, speed limits, area access ranges, and minimum connection times for multi-trips of heterogeneous fleets are configured, thus forming the basis of the heterogeneous fleet simulation environment. Step 2: Generate time-driven delivery demand; Based on the heterogeneous fleet simulation environment constructed in step 1, demand arrival rules are set according to the consumption characteristics of petroleum products and historical order data. During the simulation process, fuel delivery demands with timestamps are generated at each customer node according to the preset time step and predefined time sequence demand table. Demands that have not yet been completed are stored in the delivery queue according to the customer node, thereby forming the real-time dynamic delivery demand required by the collaborative scheduling strategy. Step 3: Construct the reward function and cooperative scheduling policy network for the PPO algorithm based on the near-end policy optimization; Based on the real-time dynamic delivery demand generated in step 2, the delivery process of the heterogeneous fleet is abstracted into a Markov decision process. A hierarchical action space consisting of three types of sub-actions—vehicle selection, customer allocation, and trip termination—is designed. A cooperative scheduling policy network with a single-agent Actor-Critic architecture is constructed under the Proximal Policy Optimization (PPO) algorithm. A multi-component reward function is defined to train the network, enabling the network to output scheduling decisions based on the current global state. Step 4: Plan vehicle travel routes and establish constraints; Based on the scheduling decision output by the collaborative scheduling strategy network in step 3, the shortest path algorithm built into the traffic simulation platform is called to plan the driving path from the current location of the vehicle to the customer node and from the customer node back to the oil depot for the selected vehicle. At the same time, constraints are established based on the loading capacity limit, speed limit, area access range, minimum connection time for multiple trips and uniqueness of customer service of the heterogeneous fleet configured in step 1. Decisions that violate any constraint are rejected and penalized to ensure the physical feasibility of scheduling decisions. Step 5: Simulation iteration and strategy output; In the simulation environment built in step 1, the delivery demand from step 2, the collaborative scheduling strategy network from step 3, the vehicle driving paths planned in step 4, and the established constraints are integrated. The simulation clock is advanced in discrete time steps, and scheduling decisions, shortest paths are calculated, vehicle operating status is updated, and constraints are verified in a loop. The PPO algorithm is used to iteratively update the parameters of the collaborative scheduling strategy network until convergence is achieved on key performance indicators such as total driving cost, delivery completion time, and vehicle type collaboration efficiency. Finally, the trained heterogeneous fleet fuel transportation route planning strategy is obtained.

2. The method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning according to claim 1, characterized in that, In step 1: The constraint parameters for the client node include: Each customer node's single fuel demand is considered an indivisible service unit. Demand generation is driven solely by preset time rules or historical demand curves and is unrelated to real-time scheduling results. The customer node's geographical location, service priority, and acceptable latest service time are stored in the road network attribute library. The constraint parameters for heterogeneous vehicle fleets include: (1) Restrictions on large fuel tank trucks; For the large fuel tanker truck configured in step 1, its maximum fuel carrying capacity is The maximum driving speed is increased to [a certain value]. It is only permitted to operate on roads that meet the regional access requirements, and the actual load when performing delivery tasks must not exceed 95% of the rated capacity; (2) Constraints on small fuel-powered delivery vehicles; For the small fuel delivery vehicle configured in step 1, its maximum fuel carrying capacity is Lift( The maximum driving speed is limited to ( ( ), allowing entry into urban areas where large fuel tankers are prohibited to complete last-mile delivery, but the loading rate must not exceed the corresponding rated capacity when performing the task; (3) Multi-stroke connection constraints; In the simulation environment established in step 1, each vehicle must undergo a connection time of no less than the minimum required after completing a delivery trip and returning to the oil depot. The stops are used for vehicle inspection, refueling, and driver rest. The next delivery trip can only start after the minimum connection time requirement is met, so as to realize the safe and dynamic reuse of vehicle resources between multiple trips.

3. The method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning according to claim 1, characterized in that, Step 2 includes the following steps: Step 2-1: Setting the simulation period and time step; Define the service runtime segment covered by a single simulation and select a discrete time step of fixed length to advance the timing framework of the simulation clock, thereby providing a time reference for subsequent demand evolution and scheduling response processes; Step 2-2: Generate mixed requirements; Based on the time sequence framework set in step 2-1, this step generates delivery requests at each customer node according to the preset demand arrival rules. For regular customer nodes, a fixed time interval is used to generate requests, while for key customer nodes such as large industrial parks or transportation hub gas stations, requests are released based on a pre-arranged time sequence table or actual order logs. Each generated delivery request record includes at least the demand quantity, customer node number, time of demand generation, priority, and latest service time. Steps 2-3: Demand queue management; First, the delivery requests generated in step 2-2 are sorted according to customer priority and the time the requests were generated, and written into the queue of pending delivery requests of the corresponding customer node. All requests that have not yet been served by vehicles are continuously accumulated in this queue until a request is successfully delivered, at which point it is removed from the corresponding queue. The real-time content and status of the queue of pending delivery requests of each customer node thus formed constitute the core component of the customer node status on which the collaborative scheduling strategy network makes decisions in step 3.

4. The method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning according to claim 1, characterized in that, Step 3, which describes the construction of the reward function and cooperative scheduling policy network for optimizing the PPO algorithm based on the near-end policy, includes the following technical elements: Step 3-1: Markov decision process modeling; Combining the simulation environment established in step 1 with the dynamic requirements generated in step 2, the heterogeneous fleet delivery process is abstracted into a Markov decision process. A single-agent PPO architecture is adopted to uniformly handle the three types of decisions: heterogeneous vehicle selection, customer node assignment, and trip termination timing. Step 3-2: Layered motion space design; A three-layer hierarchical action space is constructed for the Markov decision process in step 3-1, which consists of the following three types of sub-action combinations: (1) Vehicle selection action Used to select the specific vehicle from the currently available fleet to perform the delivery task for this round. The selected vehicle must meet the loading capacity and regional access constraints corresponding to its vehicle type. (2) Customer allocation action : Used to assign the next service customer node to the selected vehicle. This selection must take into account the vehicle’s remaining load capacity, the customer node’s demand priority, and the estimated travel time. (3) Termination of journey Used to determine whether a vehicle should continue serving the next customer or return to the oil depot to end the current journey after completing the current customer service; The above three sub-actions are combined in the order of vehicle selection - customer allocation - trip termination to form a complete dispatching action; Step 3-3: State space construction; Based on the simulation environment defined in step 1 and the delivery requirements generated in step 2, a joint state vector is constructed as the input to the policy network. This state vector includes the following components: (1) Customer node status: including the amount of unfulfilled demand, demand priority, service completion marker and spatial coordinates of each customer node; (2) Vehicle status: including vehicle type, current location, remaining load, current task execution status and end time of the previous journey for each vehicle; (3) Fleet coordination status: including at least the estimated distance matrix between vehicles and between vehicles and customer nodes, as well as the regional access identifiers for different vehicle types; (4) Global environment status: including the number of unfulfilled requirements in the system at the current simulation moment; Steps 3-4: PPO strategy network architecture design; A cooperative scheduling strategy network is constructed using an Actor-Critic structure: (1) Actor network: Input the joint state vector constructed in step 3-3, and output the complete action probability distribution in the hierarchical action space defined in step 3-2; (2) Network: Adopting the same network structure as the Actor network, the input is the joint state vector constructed in step 3-3, and the output is an estimate of the expected cumulative reward for that state; Steps 3-5: Define the objective function for near-end strategy optimization; The policy network parameters described in steps 3-4 are updated using a near-end policy optimization objective function with truncation terms. The objective function is defined as follows: In the formula: It is the probability ratio of the new strategy to the old strategy. It is the estimate of the dominance function. It is a truncation hyperparameter; Steps 3-6: Define the multi-component reward function; The multi-component reward function described in step 3 includes four components: service reward, cost reward, collaboration reward, and completion reward, with immediate rewards... Defined as: ； In the formula As a service reward; As a cost incentive; For collaborative rewards; To complete the reward; (1) Service Rewards Used to measure the timeliness of customer service, it provides a positive reward for each successful delivery, defined as: ； In the formula: For at any time The group of customers whose deliveries were successfully completed; The base reward value earned for successfully serving a single customer; For customers The waiting time between when a demand is generated and when a service is provided; The waiting penalty coefficient; (2) Cost incentive Used to penalize transportation costs, it applies a negative reward to the additional travel distance and vehicle activation caused by each scheduling decision, defined as: ； In the formula: For the time The total additional vehicle travel distance resulting from scheduling decisions, For the current moment The total number of vehicles involved in the delivery mission; The fixed cost of starting up a vehicle, , These are the weighting coefficients for driving distance cost and vehicle activation cost, respectively. (3) Collaborative rewards This is used to characterize the degree of coordination between large fuel tankers and small fuel delivery vehicles. It consists of two parts: rewards that encourage task handover and penalties for load imbalance. It is defined as follows: ； In the formula: For at any time Number of task handovers between large fuel tankers and small fuel delivery vehicles; , These represent the average load rates of large fuel tankers and small fuel delivery vehicles at the current moment, respectively. , These are the corresponding weighting coefficients; (4) Completion Reward: This is used to reflect the overall task completion status. When the delivery needs of all customer nodes are met within the simulation period, a one-time large positive reward is given, defined as: ； In the formula, For a moment The number of remaining unfulfilled customer requests in the system; A preset task completion reward value is set. In addition, when a scheduling action violates any of the following constraints: capacity limits, regional access, minimum connection time for multiple trips, or uniqueness of customer service, a fixed negative value is set as a penalty for the immediate reward of that decision step. ,in, These are the preset penalty parameters.

5. The method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4-1: Path calculation; Based on the scheduling decision output in step 3, the shortest path calculation module is called in the traffic simulation platform to plan the driving path from the current location to the customer node for the selected vehicle. Step 4-2: Constraint check; Before executing the specific route planned in step 4-1, the scheduling action is constrained based on the loading capacity limit, speed limit, area access range and minimum connection time of the heterogeneous fleet configured in step 1. If any scheduling action violates any constraint, it is directly rejected. Step 4-3: Multi-trip management; After a vehicle successfully completes the delivery task that has passed the inspection in step 4-2 and returns to the oil depot, its journey end time is recorded. Based on the minimum connection time parameter for multiple trips configured in step 1, the earliest available time for the vehicle to be dispatched next is calculated. Before this time arrives, the vehicle is marked as unavailable in the dispatch system.

6. The method for planning fuel transportation routes for heterogeneous fleets based on reinforcement learning according to claim 1, characterized in that, Step 5 specifically includes the following steps: Step 5-1: Simulation environment initialization; Load the static information and regional access rules defined in step 1, such as road network, oil depot, customer node, transfer gas station, etc., to initialize the heterogeneous fleet. At the same time, load the initial parameters of the initial delivery demand queue established in step 2 and the collaborative scheduling strategy network constructed in step 3 to complete the startup preparation of the entire simulation training system. Step 5-2: Simulation time progression and decision triggering; In the simulation environment initialized in step 5-1, the simulation clock is advanced according to the preset discrete time step. At each time step, the system makes a judgment based on the real-time status of the vehicle. When the vehicle is in an idle state or has arrived at a customer node and completed the service, a scheduling decision is triggered. At this time, the cooperative scheduling strategy network constructed in step 3 is called to generate a complete scheduling instruction containing vehicle selection, customer allocation and trip termination actions based on the current global state. Step 5-3: Perform scheduling and path update; For the scheduling instruction generated in step 5-2, the constraint check described in step 4 is first called to verify its feasibility. After the instruction passes all constraint checks, it is executed in the simulation environment. The execution process includes: calling the shortest path calculation module of the traffic simulation platform to drive the selected vehicle along the planned path; synchronously updating the vehicle's position, cumulative travel distance and remaining load capacity; and removing the customer demand that has been successfully served from the queue of undelivered demand established in step 3. Step 5-4: Reward Calculation and Experience Collection; After executing the scheduling and updating the environment state in step 5-3, the immediate reward obtained by executing the scheduling decision is calculated according to the multi-component reward function defined in step 3. Then, the key information of this decision-making process—the state before the decision, the scheduling action executed, the immediate reward obtained, and the new state of the environment after execution—is stored as a complete experience sample in the experience replay buffer. Step 5-5: Policy Update and Training Termination; When the amount of data in the experience replay buffer accumulated in step 5-4 reaches the preset batch size, a batch of data is sampled from it, and the parameters of the cooperative scheduling strategy network constructed in step 3 are updated one or more times using the PPO algorithm. This process forms a loop with steps 5-2 to 5-4, continuously optimizing the strategy. When the preset simulation termination condition is met, training stops. Finally, the various performance indicators recorded during the training process are output, including total driving cost, vehicle utilization rate, delivery completion time, and vehicle type coordination efficiency, to evaluate the overall performance of the path planning method.

Images