Multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning

Abstract

This invention discloses a multi-constraint vehicle path planning method based on an attention mechanism and deep reinforcement learning. First, an objective function is established for a vehicle path planning problem with capacity constraints and a time window. A set of user-node-disjoint routes is found for each vehicle at the minimum cost, enabling it to serve the user within the user's time window. Then, a model is established to determine initial parameter information and conditional constraints. The interaction process between the vehicle and the environment is predicted based on a Markov decision process. Finally, a neural network model is trained using a reinforcement learning method to achieve reasonable vehicle path planning under multi-constraint conditions. This invention takes into account various constraints existing in real-world problems, such as the capacity constraints of delivery vehicles and the user's requirements for delivery time. It rationally constructs a reliable neural network structure based on the attention mechanism, trains it for this problem, and reasonably and effectively solves practical problems.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning. Background Art

[0002] With the development of e-commerce, online shopping and offline logistics have become an integral part of daily life. According to statistics, as of December 2020, the number of online shoppers in my country reached 782 million, an increase of 72.15 million from March 2020, accounting for 79.1% of the total internet population. This has led to continued growth in my country's online shopping market, reaching 11.76 trillion yuan, a year-on-year increase of 10.6%. Freight transportation has become a major issue, and the use of autonomous delivery vehicles can reduce human contact and infection risks, making autonomous delivery a hot research topic.

[0003] Cities have a large number of users, who are unevenly distributed. Different users have varying cargo weights and arrival time requirements. Under these constraints, using only traditional route planning methods for autonomous delivery vehicles can easily lack rationality and systematicity, resulting in energy waste and failure to meet user needs. Therefore, reasonable route planning has become a pressing issue for autonomous delivery vehicles, with high economic value. Summary of the Invention

[0004] This invention addresses the irrational and inadequate system of vehicle routing in existing technologies and provides a multi-constraint vehicle routing method based on an attention mechanism and deep reinforcement learning. First, an objective function is established for a vehicle routing problem with capacity constraints and a time window. A set of user-node-disjoint routes is found for each vehicle at minimum cost, enabling it to serve the user within the user's time window. A model is then established to determine initial parameter information and conditional constraints. The interaction between the vehicle and the environment is predicted using a Markov decision process. Finally, a neural network model is trained using reinforcement learning to achieve reasonable vehicle routing under multiple constraints. This invention considers various real-world constraints, such as the capacity constraints of delivery vehicles and user requirements for delivery time, and rationally constructs a reliable neural network structure based on an attention mechanism. This neural network is trained specifically for this problem, effectively solving the problem.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is: a multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning, comprising the following steps:

[0006] S1. Establish an objective function: The objective function is based on a vehicle routing problem with capacity constraints and a time window. The objective function is to find a set of user-node-disjoint routes for each vehicle at the minimum cost, so that the vehicle can serve the user within the user's time window. The parameters affecting the objective function include at least the length of the vehicle's route and the relationship between the user's demand completion time and the time window.

[0007] S2, model building: Create a mathematical model based on the characteristics of the vehicle routing problem with capacity constraints and time windows, represent the user location coordinates, cargo demand, time windows, and vehicle capacity information using mathematical parameters, and determine the conditional constraint information;

[0008] S3, Markov decision process modeling: predict the interaction process between the vehicle and the environment based on the Markov decision process and determine the interaction sequence;

[0009] S4, neural network model establishment: Build a neural network model as a vehicle path user node selection strategy. The neural network model adopts an encoder-decoder structure. The encoder takes user node information as input and outputs individual user node codes and full-graph user node codes. This code and the vehicle's previous driving path are used as inputs to the decoder. The decoder ultimately outputs the selection probability of each user node and selects the user node with the highest probability as the vehicle's next destination.

[0010] S5, vehicle path solution generation: Based on the neural network model in step S4, user node information is input at each time step T and a user node is eventually obtained as the next destination of the vehicle until there are no more serviceable user nodes waiting to be served. The sequence of user nodes generated at each time step is the path solution planned for the vehicle;

[0011] S6, model training: Design a reward function based on the Markov decision process of step S3, calculate the reward of the path solution generated in step S5, and use the reinforcement learning method to train the neural network model based on the reward. The training of the neural network model uses the REINFORCE gradient descent algorithm to update the training parameters θ:

[0012]

[0013] Among them, θ represents the parameters to be trained; b(s) represents the baseline; L(π) is the parameter in p θ The loss function of π in the case of (π|s), that is, the negative value of the reward value;

[0014] The value of the path greedily generated according to the optimal strategy is used as the current value of b(s), and b(s) is updated as the model is updated;

[0015] S7, Path Planning: The user location coordinates, cargo demand, and time window are input into the trained neural network model as encoders to obtain an attention vector. This attention vector, along with vehicle information, historical routes, and current capacity, is input into the decoder. The decoder outputs a user node as the vehicle's next destination at each time step T until there are no more serviceable user nodes waiting to be served. The resulting user node sequence is the path solution for the vehicle's plan.

[0016] As an improvement of the present invention, the objective function in step S1 is:

[0017]

[0018] Among them, R[1,m] is the route set of the fleet, V is the vehicle set, X is the user node set, and m is the number of vehicles; D(x i ,x j ) represents the distance between user nodes i and j; is a binary variable, which is equal to 1 when vehicle v travels from i to j and equal to 0 otherwise; e i With l i Represents the opening and ending time of the time window of user node i, represents the time when vehicle v arrives at user node i, and represents the time difference between the vehicle arrival time and the time window, and α and β are two constants.

[0019] As an improvement of the present invention, in step S1, if the user's required completion time is outside the time window, the vehicle needs to be penalized in proportion to the lateness time.

[0020] As an improvement of the present invention, step S2 creates a mathematical model based on the characteristics of the vehicle routing problem with capacity constraints and time windows, and represents the user location coordinates, cargo demand, time window, and vehicle capacity information using mathematical parameters, including:

[0021] Denote the user node set as X, where x i ∈X,x i ={(x i ,y i ),d i ,(e i ,l i )},(x i ,y i ) is the user node coordinate, d i For user cargo demand, (e i ,l i ) is the user time window, x 0is the warehouse and the starting point of all vehicles, (x0, y0) are randomly generated coordinates, d0 is set to 0, and (e0, l0) is set to 0 to infinity.

[0022] Denote the vehicle set as V, where is the set of user nodes that vehicle i passes before time t, is the time when vehicle i arrives at the current node, is the cargo load of vehicle i at time t.

[0023] As another improvement of the present invention, the conditional constraints of step S2 include:

[0024] Ensure that each customer is visited only once and by one vehicle;

[0025] Ensure that a route is completed by one vehicle;

[0026] Ensure that if the vehicle is serving a customer, the reduced capacity of the vehicle needs to be equal to the user's demand;

[0027] Ensure that vehicles serving customers have sufficient cargo to meet customer needs and that the load never exceeds the vehicle's capacity;

[0028] Ensure that the time when vehicle v travels from i to j cannot be earlier than the time when vehicle v stops at i plus the service time served at i and the travel time from i to j. The service time for a customer is the time when the vehicle waits for the time window to open. If the vehicle arrives within the time window or later than the time window, the customer will be served immediately;

[0029] Define binary variables;

[0030] Define the variable range to be non-negative, and

[0031] Defines the time variable range.

[0032] As another improvement of the present invention, step S3 further includes:

[0033] S31: Determine the state s at time t based on the dynamic and static characteristics of the user node and the dynamic and static characteristics of the vehicle t ;

[0034] S32: Take the user node output by the neural network as the action a at time t t ;

[0035] S33: Determine the next state s after time t based on the user node characteristics and vehicle characteristics after the vehicle serves the next user node t+1 ;

[0036] S34: Determine and update the reward value based on the previous and next states

[0037] As another improvement of the present invention, step S4 specifically includes:

[0038] S41: Inputting the initial sequence of the user node into the encoder, the encoder interactively calculates the sequence of the user node and the sequences of other user nodes to obtain the attention vector sequence of the user node, thereby obtaining the total attention vector of all user nodes;

[0039] S42: The individual attention vector, total attention vector, current vehicle capacity, and previous path user nodes of the user nodes obtained by the encoder in step S41 are input into the decoder, and the probability between the user nodes is calculated by the decoder;

[0040] S43: The decoder selects a user node as the next destination of the vehicle path according to the probability;

[0041] S44: Update the dynamic features of all user nodes according to the selected user node.

[0042] As a further improvement of the present invention, in step S41, the individual features of the user nodes are linearly mapped to obtain a high-dimensional user node feature vector sequence:

[0043]

[0044] in, represents the user node sequence of user node i; W x and b x is a trainable parameter; x i Represents the static characteristics of user node i, that is, x i ={(x i ,y i ),d i ,(e i ,l i )},(x i ,y i ) represents the two-dimensional coordinate, d i Represents the demand for goods at the user node, (e i ,l i ) represents the user node time window;

[0045] All user node feature vectors Enter the multi-head attention layer and get

[0046] Will Input the feedforward neural network FF to obtain the attention vector h (N) , where m is the number of users and N is the number of multi-head attention layers;

[0047] Take the average of the attention vectors of each user node to obtain the full-graph attention vector Compared with the existing technology, the present invention proposes a multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning, which has the following technical advantages:

[0048] (1) Use neural network training to solve the vehicle path planning problem instead of using traditional heuristic algorithms. In terms of computing time, once the neural network is trained, it can quickly solve the vehicle path planning problem and shorten the computing time.

[0049] (2) Use a deep reinforcement learning model based on the attention mechanism. This model improves the Transformer model and no longer uses position encoding. Instead, it uses linear mapping to process input and adds a residual network and a feedforward neural network after the multi-head attention sublayer to achieve faster convergence and better results.

[0050] (3) A multi-constraint vehicle path planning method using attention mechanism and deep reinforcement learning solves the vehicle path planning problem with limited capacity and time window. The time window constraint is added to the original problem, and the loss function and the parameter structure of the neural network based on the attention mechanism are improved to adapt to the problem. Finally, the vehicle path planning problem with limited capacity and time window is solved using attention mechanism and deep reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart of the steps of the multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning;

[0052] Figure 2 A structural model diagram of a neural network based on attention mechanism and deep reinforcement learning in step S4 of the present invention;

[0053] Figure 3 Schematic diagram of the output solution of the CVRPSTW problem with 50 user nodes in the test example of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0055] Example 1

[0056] Multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning, such as Figure 1 As shown, the method includes the following steps:

[0057] Step S1, establishing an objective function;

[0058] The multi-constrained vehicle routing problem is abstracted into a vehicle routing problem with capacity constraints and time windows, and the objective function is created according to the characteristics of the problem.

[0059] The Capacitated Vehicle Routing Problem with Soft Time Windows (CVRPSTW) can be described as a fleet of identical vehicles starting from the same depot to serve users and returning to the depot. The vehicles have capacity constraints, and the users have their own needs, and can only be served once by one of the vehicles within their time window. The goal is to find a set of user-node-disjoint routes for each vehicle at the lowest cost, ensuring that it can serve the user within the user's time window. If the time window constraints are strict, the user will not be served when the time window is not valid. If the constraints are loose, vehicles will be penalized proportionally for late arrivals and early departures.

[0060] make Represents users and warehouses, where x 0 represents the warehouse, then the user node set is X′= X / {x 0}, where each x i Contains four parameters, namely x i ={(s i ,d i ,(e i ,l i ))},s i Represents user node x i The coordinates of d i Represents user needs, (e i ,l i ) represents the user's time window, e i and l i are the opening and closing time of the time window respectively. represents a fleet of vehicles with the same capacity and speed, R[1,m]=(r(1),r(2),…,r(m)) is the set of vehicle routes, then the objective function of the CVRPSTW problem can be defined as:

[0061]

[0062] Among them, D(x i ,x j ) represents the distance between user nodes i and j; is a binary variable that is equal to 1 when vehicle v travels from i to j and equal to 0 otherwise; represents the time when vehicle v arrives at user node i, and α and β are two constants.

[0063] Step S2, model building: Create a mathematical model based on the characteristics of the vehicle routing problem with capacity constraints and time windows, represent the user location coordinates, cargo demand, time windows, and vehicle capacity information using mathematical parameters, and determine the conditional constraints;

[0064] The present invention uses constraints to meet the problem conditions, including that each user is visited only once, the vehicle cannot be overloaded, each variable can be updated in a timely manner, and the value range of each parameter. The constraints of CVRPSTW can be summarized as follows:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073] in, is the remaining capacity of vehicle v before traveling from user i to user j.

[0074] Among the above constraints, constraint (2) ensures that each customer is visited only once and by one vehicle; constraint (3) ensures that one route is completed by one vehicle; constraint (4) ensures that if a vehicle serves a customer, the reduced capacity of the vehicle needs to be equal to the user's demand; constraint (5) ensures that the vehicle serving the customer has enough cargo to meet the customer's demand and the load will never exceed the vehicle's capacity; constraint (6) ensures that the time taken by vehicle v to travel from i to j should be greater than the sum of the travel time and the service time at customer i. We do not consider the service time of each vertex because we assume it to be a constant to simplify the problem. Therefore, the service time of a customer is the time the vehicle waits for the time window to open. If the vehicle arrives at the customer within the time window or later than the time window, it will be served immediately; constraint (7) defines the binary variable; constraint (8) defines the variable range as non-negative; constraint (9) defines the time variable range.

[0075] Step S3, Markov decision process modeling: Based on the interaction process between the vehicle and the environment in the problem, the vehicle path planning problem is abstracted into a Markov decision process;

[0076] In CVRPSTW, the vehicle constantly interacts with the environment. At each time t, the vehicle receives a state S from the environment. Based on this state S, the vehicle performs an action A. This action then acts on the environment, resulting in the vehicle receiving a reward R and reaching a new state. Therefore, the interaction between the vehicle and the environment actually produces a sequence:

[0077] S0,A0,R1,S1,A1,R2,…

[0078] We call this a sequential decision process, so we can use a Markov decision process (MDP) to model this process. The model can be defined by a tuple (S, A, P, r), where S is the state space, A is the action space, P is the transition function, and r is the rewriting function.

[0079] s t =(V t ,X t )∈S records the vehicle state and node state, where Record the state of vehicle i at time t, is the driving path of vehicle i, Record the driving time of vehicle i, Record the time when the vehicle arrives at the user, The remaining capacity of the vehicle, part 2 Record the user node status, where (x i ,y i ) is the coordinate of user node i, d i is the user's demand, (e i ,l i ) is the time window of user i.

[0080] a t is the action of the vehicle, which here represents the action of selecting a node, so

[0081] Next state s t+1 According to the conversion rule P, t Generated, assuming that vehicle i selects user node j as its next service object, then s t+1 The generation rules are as follows:

[0082]

[0083]

[0084]

[0085]

[0086] in is the last user visited by vehicle i, and it is also The last element in X t+1 The amount of user demand in will also change:

[0087]

[0088] Step S4, neural network model establishment: build a neural network model as a vehicle path user node selection strategy. Figure 2 As shown in the figure, the neural network model consists of an encoder-decoder. The user node information is input from the encoder, and the selection of the next target user of the vehicle is finally obtained from the decoder. The encoder includes a linear transformation, a multi-head attention sublayer and a feedforward neural network sublayer; the encoder includes a multi-head attention sublayer.

[0089] The neural network adopts an encoder-decoder structure. The encoder takes user node information as input and outputs individual user node codes and full-graph user node codes. This code and the vehicle's previous driving path are used as inputs to the decoder. The decoder will eventually output the selection probability of each user node, thereby selecting the user node with the highest probability as the vehicle's next destination.

[0090] The encoder structure is as follows:

[0091] User node {x1,…,x n}First linear projection to 128-dimensional vector Right now Where W x and b x Both are trainable parameters, x1 contains the user's static information, x i = {(x i ,y i ),d i ,(e i ,l i )}, n is the number of user nodes. It will pass through three attention layers in sequence, each of which includes a multi-head attention layer (MHA) and a feed-forward neural network layer (FF). Each sublayer additionally contains a skip-connection and a batch normalization. We call the node encoding output by the lth layer The formula for generating node encoding for the l-layer attention layer is as follows:

[0092]

[0093]

[0094] MHA uses the attention mechanism proposed by Vaswani et al. (Attention is all you need. NIPS, 2017). The FF layer consists of a 512-dimensional hidden sublayer and a ReLu activation function.

[0095] The final user node encoding passes through N attention layers, and the encoder outputs As the code of each node, and the average of all nodes is taken as the coding information of the whole graph, that is,

[0096] The decoder structure is as follows:

[0097] Similar to the encoder, the decoder also consists of an attention layer, but this attention layer only contains a single-head attention network. The input of the decoder is encoded by each node of the encoder output It is composed of the context node code, where the context code is composed of the following formula:

[0098]

[0099] in represents the service node selected by the vehicle at time t-1, is the remaining capacity of the vehicle, and [·,·,·] is the horizontal join operator.

[0100] and They enter the attention layer together as input. Unlike the encoder, which calculates the compatibility between nodes, the decoder's attention layer calculates the context nodes. Coded with each node The compatibility between them. The calculation formula is as follows:

[0101]

[0102]

[0103] p i =p θ (π t =i|s,π 1:t-1 )=softmax(u (c)j ) (15)

[0104] Among them, W Q , W k , W v is a trainable parameter, p i is the probability of each point being selected, the decoder will select p i The largest is π t , that is, the service node of the vehicle at time t.

[0105] Perform a softmax operation on the probability of each user node output by the decoder, set the probability of the inaccessible user node to 0, and select the user node with the highest probability as the next destination according to the greedy algorithm.

[0106] The inaccessible user nodes include: selected user nodes and user nodes whose cargo demand exceeds the vehicle carrying capacity.

[0107] Step S5: designing a reward function according to the Markov decision process and training a neural network model by a reinforcement learning method;

[0108] According to formula (1), the loss of the model is defined as We use the REINFORCE algorithm in the policy gradient algorithm to update the training parameter θ. The formula is as follows:

[0109]

[0110] The choice of b(s) has a significant impact on the learning speed. We choose the value of the path currently generated greedily according to the optimal policy as the current value of b(s). Therefore, b(s) will be updated as the model is updated. At the end of each batch of data training, we compare the advantages and disadvantages of the current policy with the b(s) policy. Only when the improvement is significant on 10,000 individual (evaluation) instances according to a paired t-test (α = 5%) will b(s) be updated to the current policy.

[0111] Test example:

[0112] A randomly generated training set is used for testing. We randomly extract the coordinates of the warehouse and 50 customers within the unit square [1.0]×[1.0]. The customer's demand is randomly selected from the set {1,2,…,9}, and the vehicle capacity is set to 50. Inspired by GoogleOR-Tools, we first randomly obtain the length of the time window from [1*3600,5*3600], and then generate the start time of the time window according to the time window and time range we set. In this test case, the time range is 86400 because it is a 24-hour period. The stop time of the time window is the sum of the start time and the time window. The model is trained in the Adam optimizer with a learning rate of 0.0001. When the decoder outputs the probability of each node, we greedily select the one with the highest probability. In multi-head attention, the parameter W Q and W K Yes (d k ×d h ) matrix, W V Yes (d v ×d h ),

[0113] The route map generated by this example is attached Figure 3 The figure uses solid lines with arrows to draw the route map of the fleet starting from the warehouse and serving users. For the sake of convenience, the routes from the warehouse and back to the warehouse are not drawn in. The upper triangle is the randomly generated warehouse address, and the points of different shapes represent the user locations served by different vehicles. The black dots are users who have not received services. Figure 3 As shown, the triangle represents the warehouse, the starting point for all vehicles. To facilitate visualization of vehicle paths, we hide the routes from and to the warehouse. User nodes served by different vehicles are drawn with points of different shapes. As shown in the figure, unlike the vehicle path diagram in the classic Vehicle-Relay Problem (VRP) problem, when users have time windows, vehicles will no longer choose the user node closest to them as their next destination to minimize penalties for late arrivals, resulting in intersections between vehicle paths.

[0114] In summary, this case uses a multi-constraint vehicle path planning method based on the attention mechanism and deep reinforcement learning to solve the vehicle path planning problem with limited capacity and time windows. By adding the time window constraint, the loss function and the parameter structure of the neural network based on the attention mechanism are improved to adapt to the problem. Ultimately, the vehicle path planning problem with limited capacity and time windows is solved using the attention mechanism and deep reinforcement learning.

[0115] It should be noted that the above content merely illustrates the technical idea of the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning, characterized by: The steps include: S1. Establish an objective function: The objective function is based on a vehicle routing problem with capacity constraints and a time window. The objective function is to find a set of user-node-disjoint routes for each vehicle at the minimum cost, so that the vehicle can serve the user within the user's time window. The parameters affecting the objective function include at least the length of the vehicle's route and the relationship between the user's demand completion time and the time window. S2, model building: Create a mathematical model based on the characteristics of the vehicle routing problem with capacity constraints and time windows, represent the user location coordinates, cargo demand, time windows, and vehicle capacity information using mathematical parameters, and determine the conditional constraint information; S3, Markov decision process modeling: predict the interaction process between the vehicle and the environment based on the Markov decision process and determine the interaction sequence; S4, neural network model establishment: Build a neural network model as a vehicle path user node selection strategy. The neural network model adopts an encoder-decoder structure. The encoder takes user node information as input and outputs individual user node codes and full-graph user node codes. This code and the vehicle's previous driving path are used as inputs to the decoder. The decoder ultimately outputs the selection probability of each user node and selects the user node with the highest probability as the vehicle's next destination. S5, vehicle path solution generation: Based on the neural network model in step S4, user node information is input at each time step T and a user node is eventually obtained as the next destination of the vehicle until there are no more serviceable user nodes waiting to be served. The sequence of user nodes generated at each time step is the path solution planned for the vehicle; S6, model training: Design a reward function based on the Markov decision process of step S3, calculate the reward of the path solution generated in step S5, and use the reinforcement learning method to train the neural network model based on the reward. The training of the neural network model uses the REINFORCE gradient descent algorithm to update the training parameters θ: Among them, θ represents the parameters to be trained; b(s) represents the baseline; L(π) is the parameter in p θ The loss function of π in the case of (π|s), that is, the negative value of the reward value; The value of the path greedily generated according to the optimal strategy is used as the current value of b(s), and b(s) is updated as the model is updated; S7, Path Planning: The user location coordinates, cargo demand, and time window are input into the trained neural network model as encoders to obtain an attention vector. This attention vector, along with vehicle information, historical routes, and current capacity, is input into the decoder. The decoder outputs a user node as the vehicle's next destination at each time step T until there are no more serviceable user nodes waiting to be served. The resulting user node sequence is the path solution for the vehicle's plan.

2. The multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning according to claim 1, characterized in that: The objective function in step S1 is: Among them, R[1,m] is the route set of the fleet, V is the vehicle set, X is the user node set, and m is the number of vehicles; D(x i ,x j ) represents the distance between user nodes i and j; is a binary variable, which is equal to 1 when vehicle v travels from i to j and equal to 0 otherwise; e i With l i Represents the opening and ending time of the time window of user node i, represents the time when vehicle v arrives at user node i, and represents the time difference between the vehicle arrival time and the time window, and α and β are two constants.

3. The multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning according to claim 1, characterized in that: In step S1, if the user's required completion time is outside the time window, the vehicle needs to be penalized in proportion to the lateness time.

4. The multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning according to claim 2, characterized in that: The step S2 creates a mathematical model based on the characteristics of the vehicle routing problem with capacity constraints and time windows, and represents the user location coordinates, cargo demand, time windows, and vehicle capacity information using mathematical parameters, specifically including: Denote the user node set as X, where x i ∈X,x i ={(x i ,y i ),d i ,(e i ,l i )},(x i ,y i ) is the user node coordinate, d i For user cargo demand, (e i ,l i ) is the user time window, x 0 is the warehouse, the starting point of all vehicles, (x0, y0) are randomly generated coordinates, d0 is set to 0, and (e0, l0) is set to 0 to infinity; Denote the vehicle set as V, where is the set of user nodes that vehicle i passes before time t, is the time when vehicle i arrives at the current node, is the cargo load of vehicle i at time t.

5. The multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning according to claim 1, characterized in that: The conditional constraints of step S2 include: Ensure that each customer is visited only once and by one vehicle; Ensure that a route is completed by one vehicle; Ensure that if the vehicle is serving a customer, the reduced capacity of the vehicle needs to be equal to the user's demand; Ensure that vehicles serving customers have sufficient cargo to meet customer needs and that the load never exceeds the vehicle's capacity; Ensure that the time when vehicle v travels from i to j cannot be earlier than the time when vehicle v stops at i plus the travel time from i to j and the service time at i. The service time for a customer is the time when the vehicle waits for the time window to open. If the vehicle arrives within the time window or later than the time window, the customer will be served immediately; Define binary variables; Define the variable range to be non-negative, and Defines the time variable range.

6. The multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning as claimed in claim 1, characterized in that: The step S3 further comprises: S31: Determine the state s at time t based on the dynamic and static characteristics of the user node and the dynamic and static characteristics of the vehicle t ; S32: Take the user node output by the neural network as the action a at time t t ; S33: Determine the next state s after time t based on the user node characteristics and vehicle characteristics after the vehicle serves the next user node t+1 ; S34: Determine and update the reward value based on the previous and next states.

7. The multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning as claimed in claim 1, characterized in that: The step S4 specifically includes: S41: Inputting the initial sequence of the user node into the encoder, the encoder interactively calculates the sequence of the user node and the sequences of other user nodes to obtain the attention vector sequence of the user node, thereby obtaining the total attention vector of all user nodes; S42: The individual attention vector, total attention vector, current vehicle capacity, and previous path user nodes of the user nodes obtained by the encoder in step S41 are input into the decoder, and the probability between the user nodes is calculated by the decoder; S43: The decoder selects a user node as the next destination of the vehicle path according to the probability; S44: Update the dynamic features of all user nodes according to the selected user node.

8. The multi-constraint vehicle path planning method based on attention mechanism and deep reinforcement learning according to claim 7, characterized in that: In step S41, the individual features of the user nodes are linearly mapped to obtain a high-dimensional user node feature vector sequence: in, represents the user node sequence of user node i; W x and b x is a trainable parameter; x i Represents the static characteristics of user node i, that is, x i ={(x i ,y i ),d i ,(e i ,l i )},(x i ,y i ) represents the two-dimensional coordinate, d i Represents the demand for goods at the user node, (e i ,l i ) represents the user node time window; All user node feature vectors Enter the multi-head attention layer and get Will Input the feedforward neural network FF to obtain the attention vector h (N) , where m is the number of users and N is the number of multi-head attention layers; Take the average of the attention vectors of each user node to obtain the full-graph attention vector

Images