A learning type electric logistics vehicle dynamic scheduling method

Abstract

The application relates to a learning type dynamic scheduling method of an electric logistics vehicle, which comprises the following steps: establishing a link energy consumption model between any two nodes in a dynamic path of the vehicle based on real-time road environment and logistics vehicle load information; based on the link energy consumption model, a dynamic electric vehicle path problem is described by using a Markov decision process, which comprises the following steps: constructing a dynamic request logistics traffic model, converting the dynamic request logistics traffic model into a Markov decision process; establishing a minimum energy consumption target function; and solving the target function by using a reinforcement learning algorithm to obtain an optimal energy consumption logistics distribution scheme. The method has high real-time performance for dynamic logistics information, and realizes the improvement of the overall energy efficiency of logistics distribution.

Description

Technical Field

[0001] This invention relates to the field of vehicle dispatching and management technology, and in particular to a learning-based dynamic dispatching method for electric logistics vehicles. Background Technology

[0002] Green logistics represents the future direction and trend of the logistics industry, and electric vehicles are a key research and application technology in this field. Electric vehicles offer advantages such as zero pollution and low noise, but also face technical constraints such as short driving range and long charging times. Furthermore, with the rapid development of the on-demand economy, vehicles often need to respond to dynamic customer requests during operation. Therefore, achieving on-demand, economically feasible, and ecologically sustainable vehicle and logistics operation optimization management under the constraints of electric vehicle technology has become a significant challenge. Optimizing vehicle transportation routes with economic efficiency as the goal is an effective means of reducing costs and increasing efficiency in road freight transportation. Solving the dynamic electric vehicle routing problem is of great significance for the application of advanced control technologies such as energy-saving driving and dynamic logistics scheduling.

[0003] In existing technologies, the optimization control of vehicle transportation routes is mostly aimed at scenarios with fixed start and end points. Existing models are not applicable to situations where transportation and logistics vehicles need to travel between nodes in the route and also meet dynamic customer requests. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a learning-based dynamic scheduling method for electric logistics vehicles, aiming to improve vehicle energy efficiency and logistics transportation efficiency.

[0005] The technical solution adopted in this invention is as follows: A learning-based dynamic scheduling method for electric logistics vehicles includes: S1. Establish the vehicle's position between any two nodes in the dynamic path. , The energy consumption model for the links between them is as follows:

[0006] In the formula, , , They are slave nodes To the node The electrical energy consumption, constant mechanical power requirements, and travel time, To reach the node The quality of goods before unloading; These are the conversion coefficients between vehicle mechanical power and battery discharge power, and battery recycled power, respectively. These are the conversion coefficients between the battery's electrical power and the battery power in the discharge state and the battery power in the recycling state, respectively. S2. Based on the aforementioned link energy consumption model, a Markov decision process is used to describe the dynamic electric vehicle routing problem, including: S21. Construct a dynamic request logistics and transportation model. , For an undirected connected graph consisting of nodes and connecting arcs between nodes, the set of nodes is... , These represent distribution centers, customer collection points, and charging station collection points, respectively. Let the set of arc segments in the diagram be the arc segment set. Divide the time between two consecutive nodes into a time period. Discretize the vehicle's working time into a set of time periods. , This defines that the vehicle can accept new customer requests at any time period, and every... Customer requests are processed within a specific time period. For hyperparameters and ; S22. Convert the dynamic request logistics transportation model into a Markov decision process: Define the vehicle in the time period status , These are the vehicle's current location, battery level, cargo load, number of active customers, and number of customers who have visited the site. Determine the vehicle is in a stable condition. When the decision is made to select a node for access, the possible actions include: , For the current time period The position of the next node to be visited, when , ,when hour, ; Considering dynamic customer requests, establish time periods. When the time period arrives From node To the node The resulting set of state transition equations:

[0007]

[0008]

[0009]

[0010]

[0011] In the system of equations, subscripts Each represents a corresponding time period; This is the new node location; Time period Vehicles from node To the node The energy consumption also represents the reward during the state transition process; The required quality of goods for each customer; For nodes When serving customers, The corresponding customer is removed from the current requesting customers and added to the customer making the new request. The number of customers after the event For nodes When serving customers, nodes The number of customers who have been visited after being added to the list of visited customers; S3. Establish the objective function for minimizing energy consumption:

[0012] And construct the constraints:

[0013]

[0014]

[0015] in, , , For binary decision variables, when the vehicle is in a time period Leave node and reach the node hour, It is 1 if it is true, otherwise it is 0. , These are the last time periods before the vehicle departs and before the mission ends. The number of customers participating in the event This represents the number of customers who were visited during the last time period before the task ended. S4. The objective function is solved using a reinforcement learning algorithm to obtain the optimal energy consumption logistics distribution scheme.

[0016] The further technical solution is as follows: In step S1, from node To the node constant mechanical power demand Determined by the following formula:

[0017] in, , representing the current node reached Overall vehicle weight before unloading It refers to the vehicle's curb weight; These represent the air drag coefficient, air density, frontal area of ​​the vehicle, and rolling friction coefficient, respectively. , The vehicles are located at the nodes. To the node The speed and acceleration that constitute the travel speed on the link For the node To the node The slope angle that constitutes the link.

[0018] In step S2, the number of customers requesting service before the vehicle departs. The known time period after vehicle departure Number of internal request customers It is randomly generated. Node customers during time period Probability of sending a new request for:

[0019] In the formula, for The total probability that a node's clients will send a cargo request. obey Binomial distribution over a time period.

[0020] In step S4, the Q-learning algorithm is used to solve the objective function to obtain the optimal energy consumption logistics distribution scheme. The Bellman equation of the Q table in the Q-Learning algorithm is described as follows:

[0021] In the formula, , They represent and Action value function over a time period; The update formula for table Q is:

[0022] In the formula, For learning rate, This is the attenuation factor.

[0023] The design of the Q-learning algorithm includes approximation of vehicle state values: Discretize the maximum battery capacity and maximum payload as follows: Each level, that is Each level represents the corresponding battery capacity and payload.

[0024] The design of the Q-learning algorithm includes formulating a safe target strategy for battery power. :

[0025] Representing state The optimal action represents the previous target strategy. ; for Time period from node To the node Energy consumption For nodes Go to the nearest charging station Power consumption:

[0026] and If the sum is greater than the current battery charge Then the security target strategy Will cover previous target strategies It selects the charging station closest to the current node to charge the battery, otherwise it maintains the previous target strategy. constant.

[0027] The design of the Q-learning algorithm also includes setting up an offline training method: use - Greedy strategy as exploration strategy, safety target strategy As a target policy, each round of offline training is a complete route that starts from the distribution center, serves all requests, and returns to the distribution center, including nodes such as charging stations. If the battery runs out of power, the round ends.

[0028] The training rounds and corresponding hyperparameters are set to obtain the Q-table after training. During the vehicle's delivery period, when a random customer request occurs, the Q-table is updated according to the corresponding status and the safety objective strategy. Select the next node to visit.

[0029] The beneficial effects of this invention are as follows: This invention builds an energy consumption model for electric logistics vehicles based on real-time road environment and logistics vehicle load information. The model takes into account the impact of real-time cargo weight, making it more consistent with reality.

[0030] This invention transforms the dynamic electric vehicle routing problem into a Markov decision process, which facilitates the solution by the Q-learning algorithm and makes the dynamic electric vehicle routing problem model universally applicable to reinforcement learning algorithms.

[0031] This invention solves the dynamic electric vehicle routing problem using the Q-learning algorithm. The computational cost required for online decision-making is very small, and it has high real-time performance for dynamic logistics information and high solution accuracy, thereby improving the overall energy efficiency of logistics distribution.

[0032] This invention improves the design of the Q-learning algorithm by incorporating a safety objective strategy, thereby significantly reducing the risk of electric logistics vehicles running out of power during operation.

[0033] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0034] Figure 1 This is a dynamic electric vehicle routing problem model according to an embodiment of the present invention.

[0035] Figure 2 This is a flowchart of the scheduling method according to an embodiment of the present invention.

[0036] Figure 3 This is a Markov decision process transformation diagram according to an embodiment of the present invention.

[0037] Figure 4 This is a flowchart of the Q-learning algorithm for the dynamic vehicle routing problem according to an embodiment of the present invention. Detailed Implementation

[0038] The specific embodiments of the present invention are described below with reference to the accompanying drawings.

[0039] This embodiment provides a learning-based dynamic scheduling method for electric logistics vehicles. It constructs an energy consumption model for electric logistics vehicle links, establishes a Markov decision process for dynamic electric vehicle routing problems, and then solves an economical dynamic delivery route scheme using the Q-learning algorithm.

[0040] The dynamic electric vehicle routing problem model is referred to Figure 1 .

[0041] The scheduling method is described in [reference]. Figure 2Specifically, it includes the following steps: S1. Establish the vehicle's position between any two nodes in the dynamic path. , The link energy consumption model between them specifically includes: Determine the rolling resistance during driving : , Indicates the total mass of the vehicle. Represents the gravitational constant. This represents the coefficient of rolling friction (which depends on tire pressure, road conditions, and other factors). Indicates the slope angle; Determine air resistance : , Indicates the vehicle's speed. Indicates the air drag coefficient. Indicates air density, Indicates the area of ​​the front part of the vehicle; Determine the total mechanical power :

[0042] In the above formula, Acceleration of the vehicle; Based on the Quantized State System (QSS) model of electric motors, through regression coefficients Fitted mechanical power and battery discharge power and battery recycling power The relationship between them:

[0043] Simultaneously using regression coefficients An approximate description of the battery's electrical power With battery power Conversion relationship:

[0044] Considering that the energy consumption of electric logistics vehicles is affected by the amount of cargo loaded, in order to solve the link energy consumption model, the vehicle mass is defined as the current cargo loading mass. function , It refers to the vehicle's curb weight. Indicates arrival at node The weight of the goods (before unloading) satisfies the node. The quality of the sum of the needs of the current customer and all subsequent customers.

[0045] From node To the node constant mechanical power demand Determined by the following formula:

[0046] in, , The vehicles are located at the nodes. To the node The speed and acceleration that constitute the travel speed on the link For the node To the node The slope angle that constitutes the link.

[0047] Therefore, the link energy consumption model can be expressed as:

[0048] In the formula, , , They are slave nodes To the node The electrical energy consumption, constant mechanical power requirements, and travel time, To reach the node The quality of goods before unloading; These are the conversion coefficients between vehicle mechanical power and battery discharge power, and battery recycled power, respectively. These are the conversion coefficients between the battery's electrical power and the battery power in the discharge state and the battery power in the recycling state, respectively.

[0049] The link energy consumption model reflects the real-time changes in cargo mass along the dynamic path into the vehicle's mechanical power, which is more consistent with the actual situation.

[0050] The driving cycle map of electric logistics vehicles on this link can be given based on the road traffic flow conditions between nodes (obtained through traffic software). For a given driving link, except for the cargo quality... All variables other than are constants.

[0051] S2. Based on the aforementioned link energy consumption model, a Markov decision process is used to describe the dynamic electric vehicle routing problem, including: S21. Construct a dynamic request logistics and transportation model. , For an undirected connected graph consisting of nodes and connecting arcs between nodes, the set of nodes is... , These represent distribution centers, customer collection points, and charging station collection points, respectively. Let the set of arc segments in the diagram be the arc segment set. in, ,but ; The total number of customers and the total number of charging stations are respectively , ; Divide the time between two consecutive nodes into a time period. Discretize the vehicle's working time into a set of time periods. , This defines that the vehicle can accept customer requests at any time period, and every... Customer requests are processed within a specific time period. For hyperparameters and ; S22. Convert the dynamic request logistics transportation model into a Markov decision process, see [link / reference]. Figure 3 Solid arrows represent deterministic processes, while hollow arrows represent nondeterministic processes, including: Define the vehicle in the time period status , These are the vehicle's node location, battery level, cargo load, number of active customers (number of customers with requests), and number of customers who have been visited (number of customers whose requests have been fulfilled, i.e., served). in, That is, the number of customers who are visited is no longer considered the number of active customers; The initial state of the vehicle is represented as follows: This indicates that the vehicle is in a fully charged state (maximum battery capacity). Departing from the distribution center and fully loaded (maximum effective load) The number of active customers known before the vehicle's departure was The clients we have already served are empty sets. .

[0052] Determine the vehicle is in a stable condition. When the decision is to select the node to visit, the possible actions include: , For the current time period The position of the next node to be visited, when , ,when hour, ; Number of customers requesting vehicle departure The known time period after vehicle departure Number of internal request customers It is randomly generated. Node customers during time period Probability of sending a new request for:

[0053] In the formula, for The total probability that a node's clients will send a cargo request. obey Binomial distribution over a time period.

[0054] Considering dynamic customer requests, establish time periods. When the time period arrives From node To the node The resulting set of state transition equations:

[0055]

[0056]

[0057]

[0058]

[0059] In the system of equations, subscripts Each represents a corresponding time period. The range of values ​​for ; This is the new node location; Time period Vehicles from node To the node The energy consumption also represents the reward during the state transition process; The required quality of goods for each customer; For nodes When serving customers, The corresponding customer is removed from the current requesting customers and added to the new requesting customer list. The number of customers after the event For nodes When serving customers, nodes The number of customers who have been visited after being added to the list of visited customers; When the node is a charging station, the vehicle is fully charged. ; in, This means the current cargo load is less than the maximum effective load. , That is, the current battery level is no greater than the maximum battery capacity. .

[0060] This completes the transformation of the Markov decision process for the dynamic electric vehicle routing problem.

[0061] S3. Establish the objective function for minimizing energy consumption:

[0062] in, , , For binary decision variables, when the vehicle is in a time period Leave node and reach the node hour, It is 1 if it is true, otherwise it is 0. Construct constraints: (Y1) (Y2) (Y3) Equation (Y1) represents the last time period. Number of customers who made requests It is an empty set, meaning that all client requests are satisfied when the task ends; Equation (Y2) represents the last time period. Number of customers who have been visited The number of requesting customers determined before departure and each time period Number of customers that randomly send new requests ; Equation (Y2) represents that the vehicle's battery state is always greater than the minimum level.

[0063] S4. Solve the objective function using a reinforcement learning algorithm to obtain the optimal energy-efficient logistics delivery solution, specifically including: The Q-learning algorithm is used to solve the objective function to obtain the optimal energy-efficient logistics and distribution scheme. The Bellman equation of the Q-table in the Q-Learning algorithm is described as follows:

[0064] In the formula, , They represent and Action value function over a time period; Considering that the Markov decision process for the dynamic electric vehicle routing problem accumulates energy consumption, which should be minimized, the update formula for the Q table is:

[0065] In the formula, For learning rate, This is the attenuation factor.

[0066] Design for Q-learning algorithm, including (1) Approximate the vehicle state values: Discretize the maximum battery capacity and maximum payload as follows: Each level, that is Each level represents the corresponding battery capacity and payload.

[0067] This embodiment The maximum battery capacity and maximum payload will be divided into 10 levels.

[0068] Since battery capacity and payload are continuous variables, discretizing the state can reduce the storage space of the Q-table and improve exploration.

[0069] (2) Formulate a safety target strategy for battery power. :

[0070] Representing state The optimal action represents the previous target strategy. ; for Time period from node To the node Energy consumption For nodes Go to the nearest charging station Power consumption:

[0071] and If the sum is greater than the current battery charge Then the security target strategy Will cover previous target strategies It selects the charging station closest to the current node to charge the battery, otherwise it maintains the previous target strategy. constant.

[0072] The goal of designing safety strategies is to prevent the risk of a vehicle running out of power while in motion.

[0073] (3) Set up offline training method: use - Greedy strategy as exploration strategy, safety target strategy As a target policy, each round of offline training is a complete route that starts from the distribution center, serves all requests, and returns to the distribution center, including nodes such as charging stations. If the battery runs out of power, the round ends.

[0074] The training rounds and corresponding hyperparameters are set to obtain the Q-table after training. During the vehicle's delivery period, when a random customer request occurs, the Q-table is updated according to the corresponding status and the safety objective strategy. Select the next node to visit.

[0075] The Q-learning algorithm process is as follows: Figure 4 As shown. Because the Q-learning algorithm is based on table lookup, the computational cost required for online decision-making is very small, and the algorithm has very high real-time performance. Retraining is only required when the customer or charging station settings change, or when the probability of a customer request changes significantly.

[0076] It will be understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A learning-based dynamic scheduling method for electric logistics vehicles, characterized in that, include: S1. Establish the vehicle's position between any two nodes in the dynamic path. , The energy consumption model for the links between them is as follows: ， In the formula, , , They are slave nodes To the node The electrical energy consumption, constant mechanical power requirements, and travel time, To reach the node The quality of goods before unloading; These are the conversion coefficients between vehicle mechanical power and battery discharge power, and battery recycled power, respectively. These are the conversion coefficients between the battery's electrical power and the battery power in the discharge state and the battery power in the recycling state, respectively. S2. Based on the aforementioned link energy consumption model, a Markov decision process is used to describe the dynamic electric vehicle routing problem, including: S21. Construct a dynamic request logistics and transportation model. , For an undirected connected graph consisting of nodes and connecting arcs between nodes, the set of nodes is... , These represent distribution centers, customer collection points, and charging station collection points, respectively. Let the set of arc segments in the diagram be the arc segment set. Divide the time between two consecutive nodes into a time period. Discretize the vehicle's working time into a set of time periods. , This defines that the vehicle can accept new customer requests at any time period, and every... Customer requests are processed within a specific time period. For hyperparameters and ; S22. Convert the dynamic request logistics transportation model into a Markov decision process: Define the vehicle in the time period status , These are the vehicle's current location, battery level, cargo load, number of active customers, and number of customers who have visited the site. Determine the vehicle is in a stable condition. When the decision is made to select a node for access, the possible actions include: , For the current time period The position of the next node to be visited, when , ,when hour, ; Considering dynamic customer requests, establish time periods. When the time period arrives From node To the node The resulting set of state transition equations: ， ， ， ， ， In the system of equations, subscripts Each represents a corresponding time period; This is the new node location; Time period Vehicles from node To the node The energy consumption also represents the reward during the state transition process; The required quality of goods for each customer; For nodes When serving customers, The corresponding customer is removed from the current requesting customers and added to the customer making the new request. The number of customers after the event For nodes When serving customers, nodes The number of customers who have been visited after being added to the list of visited customers; S3. Establish the objective function for minimizing energy consumption: ， And construct the constraints: ， ， ， in, , , For binary decision variables, when the vehicle is in a time period Leave node and reach the node hour, It is 1 if it is true, otherwise it is 0. , These are the last time periods before the vehicle departs and before the mission ends. The number of customers participating in the event This represents the number of customers who were visited during the last time period before the task ended. S4. The objective function is solved using a reinforcement learning algorithm to obtain the optimal energy consumption logistics distribution scheme.

2. The learning-based dynamic scheduling method for electric logistics vehicles according to claim 1, characterized in that, In step S1, from node To the node constant mechanical power demand Determined by the following formula: ， in, , representing the current node reached Overall vehicle weight before unloading It refers to the vehicle's curb weight; These represent the air drag coefficient, air density, frontal area of ​​the vehicle, and rolling friction coefficient, respectively. , The vehicles are located at the nodes. To the node The speed and acceleration that constitute the travel speed on the link For the node To the node The slope angle that constitutes the link.

3. The learning-based dynamic scheduling method for electric logistics vehicles according to claim 1, characterized in that, In step S2, the number of customers requesting service before the vehicle departs. For known time periods after vehicle departure Number of internal request customers It is randomly generated. Node customers during time period Probability of sending a new request for: ， In the formula, for The total probability that a node's clients will send a cargo request. obey Binomial distribution over a time period.

4. The learning-based dynamic scheduling method for electric logistics vehicles according to claim 1, characterized in that, In step S4, the Q-learning algorithm is used to solve the objective function to obtain the optimal energy consumption logistics distribution scheme. The Bellman equation of the Q table in the Q-Learning algorithm is described as follows: ， In the formula, , They represent and Action value function over a time period; The update formula for table Q is: ， In the formula, For learning rate, This is the attenuation factor.

5. The learning-based dynamic scheduling method for electric logistics vehicles according to claim 4, characterized in that, The design of the Q-learning algorithm includes approximation of vehicle state values: Discretize the maximum battery capacity and maximum payload as follows: Each level, that is Each level represents the corresponding battery capacity and payload.

6. The learning-based dynamic scheduling method for electric logistics vehicles according to claim 4, characterized in that, The design of the Q-learning algorithm includes formulating a safe target strategy for battery power. : ， Representing state The optimal action represents the previous target strategy. ; for Time period from node To the node Energy consumption For nodes Go to the nearest charging station Power consumption: ， and If the sum is greater than the current battery charge Then the security target strategy Will cover previous target strategies It selects the charging station closest to the current node to charge the battery, otherwise it maintains the previous target strategy. constant.

7. The learning-based dynamic scheduling method for electric logistics vehicles according to claim 6, characterized in that, The design of the Q-learning algorithm also includes setting up an offline training method: use - Greedy strategy as exploration strategy, safety target strategy As a target policy, each round of offline training is a complete route that starts from the distribution center, serves all requests, and returns to the distribution center, including nodes such as charging stations. If the battery runs out of power, the round ends.

8. The learning-based dynamic scheduling method for electric logistics vehicles according to claim 7, characterized in that, The training rounds and corresponding hyperparameters are set to obtain the Q-table after training. During the vehicle's delivery time, when a random customer request occurs, the Q-table is updated according to the corresponding status and the safety objective strategy. Select the next node to visit.

Images