Unmanned aerial vehicle and vehicle joint distribution path optimization method and system based on reinforcement learning
By formalizing the joint distribution path planning problem of drone and vehicle into Markov decision-making process, and using the reinforcement learning agent behavior strategy network, the problem of low efficiency in solving large-scale problems is solved, and rapid and efficient Path optimization.
Patent Information
- Application Number
- CN202410948999.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
AI Technical Summary
The existing joint distribution path optimization method of drone and vehicle is inefficient in solving large-scale problems, making it difficult to find better solutions in polynomial time, especially traditional methods are difficult to deal with more than 50 goals.
Using reinforcement learning-based method, the joint distribution path planning problem of drone and vehicle is formalized into Markov decision-making process, and the characterization of reinforcement learning agent behavior strategy network is used to perform feature extraction and information aggregation using encoder and decoder. , combined with deep reinforcement learning algorithms to train the model, generate access node sequences of drones and vehicles to achieve path optimization.
The solution is given in millisecond time, with strong generalization performance, can effectively optimize the joint distribution path between drones and vehicles, and improve the solution efficiency and quality.
Smart Images

Figure CN118917504A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of path planning methods, and in particular to a method and system for optimizing the joint delivery path of unmanned aerial vehicles and vehicles based on reinforcement learning. Background Art
[0002] This section introduces the technical background mastered by the applicant and does not represent that it is prior art.
[0003] The optimization of the joint delivery path of drones and vehicles is a typical combinatorial optimization problem, which involves how to effectively allocate vehicles and drones to visit more than two targets and minimize the time cost while satisfying constraints.
[0004] The joint delivery path optimization of drones and vehicles is widely used in the fields of civil logistics and military logistics support. Compared with traditional vehicle transportation, drones are not affected by ground traffic and can effectively improve delivery efficiency. In addition, drone delivery has the characteristics of small batches and high frequency, which is more suitable for delivery at the end of logistics. However, since drone flights are constrained by load and endurance, the joint delivery of drones and vehicles can achieve more efficient delivery.
[0005] In the field of civilian logistics and distribution, with the continuous growth of e-commerce and transportation industries, the logistics industry has developed rapidly, and higher requirements have been put forward for large-scale real-time logistics dispatching systems. However, the transportation and warehousing costs generated by logistics and distribution remain high, so the search for efficient logistics and distribution models has received widespread attention from academia and industry. In the military field, transportation and delivery are an organic part of combat operations and the central link of rear-end support. They play an irreplaceable role in war. The joint delivery of multiple transportation modes such as drones and vehicles is an important way to strengthen the construction and development of strategic delivery capabilities.
[0006] The optimization of joint delivery paths of drones and vehicles has attracted a large number of domestic and foreign scholars to conduct research. Traditional methods mainly include exact algorithms, polynomial time approximate algorithms and meta-heuristic algorithms. Exact algorithms can obtain the optimal solution, but since the joint delivery path optimization problem is an NP-hard problem, this method is difficult to use to solve problems with more than 50 targets. Polynomial time approximate algorithms can usually obtain solutions with guaranteed quality, but the optimality guarantee is weak, and even the local optimal solution of the problem cannot be obtained. Meta-heuristic algorithms such as wolf pack algorithms, ant colony optimization algorithms, bat algorithms and variable neighborhood search algorithms are widely used due to their high performance, but they usually require customization for specific problems and professional domain knowledge, and it is difficult to find a better solution for large-scale problems in polynomial time. Although a large number of solution strategies have emerged in traditional methods, there is still room for further improvement in solution efficiency. Summary of the invention
[0007] In view of the shortcomings of the prior art, the present invention provides a method and system for optimizing the joint delivery path of drones and vehicles based on reinforcement learning.
[0008] A method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning includes the following steps:
[0009] Step 1: mathematically model the path planning problem of a vehicle and a drone performing joint delivery to more than two targets, represent the warehouse address and the target address as nodes, and construct a complete graph G consisting of a set of nodes. By defining the state s, action a, and reward function R, the joint path planning problem is formalized as a Markov decision process;
[0010] Step 2: Input the graph G and construct a parameterized reinforcement learning agent behavior strategy network consisting of an encoder and a decoder. The encoder is used to extract features and aggregate information, and the decoder is used to solve the node selection strategy to generate the probability distribution p of the nodes to be selected. θ ;
[0011] Step 3: Train the model parameters through a policy gradient-based deep reinforcement learning algorithm to generate a node visit sequence for drones and vehicles based on a trained probability distribution model given a node graph.
[0012] Furthermore, the method of constructing a complete graph and forming a Markov decision process described in step 1 includes:
[0013] Step 1-1, for the path planning problem of a vehicle and a drone for joint delivery to more than two targets, the following problem description and condition constraints are given. Consider a warehouse with a vehicle and a drone. The task is completed by using the vehicle or drone to deliver goods to each target node and return to the warehouse in the shortest time. The constraints include that other factors are not considered, the drone delivers at the maximum flight speed; the drone can only deliver one target each time it is released, and then the drone must meet with the vehicle to replace the battery; when the drone and the vehicle meet at a target node, the one that arrives earlier waits for the other.
[0014] Step 1-2, mathematical modeling of the joint distribution path planning problem. set = A complete graph G consisting of {m, 1, 2, 3…, N}, where node m represents a warehouse, nodes 1, 2, 3…, N represent target nodes, and when the distance from node i to node j is d i,j When , the delivery time of vehicle and drone from node i to node j is and The node position that the UAV and vehicle are about to reach or are at at the tth moment is expressed as and UAVs and vehicles arrive at node locations and The remaining time is expressed as and If the drone or vehicle has arrived, then or V t represents the set of nodes visited by drones or vehicles from time 0 to time t, Indicates the target node location that the drone will go to next at the tth moment, Indicates the target node location that the vehicle will go to next at the tth moment.
[0015] Steps 1-3, formalize the joint path planning problem into a Markov decision process, including the definition of state s, action a and reward function R. The state s at the tth moment is expressed as It consists of the set of visited nodes, the node location to be visited at the current moment, and the remaining time to reach the destination; the state a at the tth moment is expressed as Indicates the node location that the drone and vehicle are going to next; considering the dynamic update of the state over time, when the action at the tth moment is confirmed, the target location that the drone and vehicle choose to go to next is After that, the remaining time for the drone and vehicle to reach the next node location is updated, and the remaining time for the vehicle is expressed as The remaining time of the drone is expressed as Then the interval from the current moment to the next moment should be the time when the drone or vehicle first arrives at a certain node, expressed as Therefore, at the t+1th moment, the vehicle or drone reaches the next target node The remaining time is and At the t+1th moment, the position node of the vehicle or drone is updated to and The set of visited nodes at time t+1 is updated to Define T as the time step for the vehicle and drone to return to the warehouse after delivering all the targets. Our goal is to complete all delivery tasks in the shortest time, so the reward function is defined as Where C is the time to complete the task.
[0016] Furthermore, the method for constructing a parameterized reinforcement learning agent behavior strategy network described in step 2 includes:
[0017] Step 2-1, the node set N set The complete graph G is used as the input of the encoder, and the coordinate position of each node n in the node set is represented as Xn =(x n ,y n ), the encoder encodes the node coordinate position into high-dimensional features, and then extracts features and aggregates information through a multi-head attention network, outputting the feature vector h corresponding to node n n ;
[0018] Step 2-2: transform the feature vector h of each node n n Input to the decoder, the structural features obtained after all node feature vectors are downsampled Indicates the correlation between delivery nodes. The decoder consists of an LSTM network and a probabilistic attention network. The LSTM network performs node feature vector fusion to obtain a more comprehensive node feature representation. Probabilistic Attention Network Input Structure Characteristics and the node feature representation at time t Generate the probability distribution p of the node to be selected θ , realize the node selection strategy solution;
[0019] Furthermore, the method for constructing the encoder described in step 2-1 includes:
[0020] Step 2-1-1, the node set N set The complete graph G is used as the input of the encoder, and the coordinate position of each node n in the node set is represented as X n =(x n ,y n ).
[0021] Step 2-1-2: Use the embedding layer to encode the node coordinate position into the high-dimensional feature F1 n , the expression formula is as follows:
[0022]
[0023] Where φ(·) represents the embedding layer function;
[0024] Step 2-1-3, use the multi-head attention network to convert the high-dimensional feature F1 n As input, and pass messages between nodes. For a node set N set Each node feature vector F1 n , we can get the corresponding query matrix Q n =f Q (F1 n ), the key matrix K n =f K (F1 n ) and value matrix V n =f V (F1n ), where f Q , f K and f V They are respectively the query function, key function and value function shared by all N node targets.
[0025] Step 2-1-4, based on the input of the previous step, calculate the single-head attention feature vector. Based on the query matrix and the key matrix, we can calculate the relationship matrix u between the i-th node and the j-th node i,j , the relationship matrix is used to model the correlation between two nodes, and the calculation formula is:
[0026]
[0027] where d k Denotes the query matrix Q n The dimension of The meaning here is to combine the distribution of attention coefficients with the query matrix Q n The dimension decoupling of , so that the gradient value remains stable during training. Based on the relationship matrix u i,j The corresponding attention weight a can be calculated i,j , the calculation formula is:
[0028]
[0029] Then extract the attention feature vector F2 corresponding to node n n , the calculation formula is:
[0030] F2 n =Σ j a n,j V n
[0031] Step 2-1-5, connect the feature information of multiple single-head attentions through multi-head attention to achieve information association at different levels. Set the number of multi-head attention heads to M, then you can get the multi-head attention feature vector F3 corresponding to node n n , the calculation formula is:
[0032]
[0033] The multi-head attention feature vector F3 n and the initial eigenvector F1 n Fusion and batch normalization BN are performed to obtain the feature vector F4 n , the calculation formula is as follows:
[0034] F4 n =BN(F1 n +F3 n )
[0035] Step 2-1-6: Perform feature aggregation through the feedforward neural network to output the feature vector h corresponding to node n n , the calculation formula is:
[0036] h n =BN(F4 n +ReLU(FC(F4 n ))
[0037] Among them, FC represents the fully connected layer and ReLU represents the activation function.
[0038] Furthermore, the method for constructing a decoder described in step 2-2 includes:
[0039] Step 2-2-1, the feature vector h of each node n n Input to the decoder, the structural features obtained after all node feature vectors are downsampled It represents the correlation between distribution nodes. The calculation formula is as follows:
[0040]
[0041] MEAN means taking the average value of the feature vector, which can reduce the computational complexity of the model.
[0042] Step 2-2-2, transform the feature vector h n Input into the LSTM network for node feature vector fusion to obtain a more comprehensive node feature representation at the tth moment The calculation formula is as follows:
[0043]
[0044] in, Represents the information stored by the LSTM network memory gate. It represents the memory of the input data, including the fusion of all input information of the neural network before time t. Represents the information captured by the output gate.
[0045] Step 2-2-3, Probabilistic Attention Network PA Input Structural Features and the node feature representation at time t Generate probabilistic attention feature vector The calculation formula is as follows:
[0046]
[0047] Among them, τ n,·Represents the time from node n to other nodes. The probabilistic attention network PA is a self-attention layer. Then the probabilistic attention feature vector Perform softmax operation to obtain the probability distribution p of the node to be selected θ , to solve the node selection strategy, the calculation formula is as follows:
[0048]
[0049] Furthermore, the policy gradient-based deep reinforcement learning algorithm described in step 3 includes:
[0050] Step 3-1: By parameterizing the reinforcement learning agent behavior strategy network, the probability distribution π of the delivery node sequence can be determined. θ (V T |G), that is, given a complete graph, the sequence of nodes visited by drones and vehicles can be output. Our goal is to minimize the completion time C, so the objective function J(θ|G) of the training is defined, and the calculation formula is:
[0051]
[0052] in represents the mathematical expectation, b(G) represents the average baseline for reducing the gradient variance, which reflects the quality of multiple sampled sequences, so as to better evaluate the quality of the agent's actions. Therefore, the critic network is used to estimate b(G), and the actor network is used to learn the strategy π θ .
[0053] Step 3-2: Use the policy gradient-based deep reinforcement learning algorithm to train the model parameters, input the training data, and generate a random strategy p through the parameterized reinforcement learning agent behavior strategy network. θ , randomly select nodes according to the probability distribution to construct a valid solution, execute the action T times, generate multiple feasible solution sequences, calculate the average baseline b(G), and then use the Adam optimizer to update the model parameters. After completing the specified number of iterations, the trained model parameter set θ is finally obtained.
[0054] The present invention provides a navigation system, comprising the reinforcement learning-based unmanned aerial vehicle and vehicle joint delivery path optimization method.
[0055] The patent of this invention proposes a new reinforcement learning method for solving the problem of joint delivery between drones and vehicles. The problem is formalized as a Markov decision process for solution. An encoder based on multi-head attention is proposed for feature embedding encoding, and a decoder based on LSTM network is proposed for feature aggregation. The model is trained with a deep reinforcement learning algorithm. It can provide a solution at the millisecond level without being constrained by the scale of the problem, and has strong generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.
[0057] Figure 1 The figure is a schematic flow chart of the steps for implementing the present invention.
[0058] Figure 2 The figure is a schematic diagram of a network structure of an embodiment of the present invention. DETAILED DESCRIPTION
[0059] The present invention provides a method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning. Figure 1 As shown, the specific steps include:
[0060] Step 1: mathematically model the path planning problem of a vehicle and a drone performing joint delivery to more than two targets, represent the warehouse and the target as nodes, and construct a complete graph G consisting of a set of nodes. By defining the state s, action a, and reward function R, the joint path planning problem is formalized as a Markov decision process;
[0061] Step 2: Input the graph G and construct a parameterized reinforcement learning agent behavior strategy network consisting of an encoder and a decoder. The encoder is used to extract features and aggregate information, and the decoder is used to solve the node selection strategy to generate the probability distribution p of the nodes to be selected. θ ;
[0062] Step 3: Train the model parameters through a policy gradient-based deep reinforcement learning algorithm to generate a node visit sequence for drones and vehicles based on a trained probability distribution model given a node graph.
[0063] The method is described with reference to the embodiments, such as Figure 2 As shown, the step 1 includes:
[0064] Step 1-1, for the path planning problem of a vehicle and a drone for joint delivery to more than two targets, the following problem description and condition constraints are given. We consider a warehouse with a vehicle and a drone. The task is completed if the vehicle or drone is used to deliver goods to each target node and return to the warehouse in the shortest time. The constraints include that other factors are not considered, the drone delivers at the maximum flight speed; the drone can only deliver one target each time it is released, and then the drone must meet with the vehicle to replace the battery; when the drone and the vehicle turn at a certain target node, the one that arrives earlier waits for the other.
[0065] Step 1-2, mathematical modeling of the joint distribution path planning problem. set = A complete graph G consisting of {m, 1, 2, 3…, N}, where node m represents a warehouse, nodes 1, 2, 3…, N represent target nodes, and when the distance from node i to node j is d i,j When , the delivery time of vehicle and drone from node i to node j is and To indicate that, in this embodiment, d i,j If the distance is 60 km, the speed of the drone is 120 km / h, and the speed of the vehicle is 60 km / h, then the delivery time of the vehicle is The delivery time of the drone is 1 hour, and the delivery time of the drone is 0.5 hours. The node position that the drone and the vehicle are about to arrive at or are at at the tth moment is expressed as and UAVs and vehicles arrive at node locations and The remaining time is expressed as and If the drone or vehicle has arrived, then or V t represents the set of nodes visited by drones or vehicles from time 0 to time t, Indicates the target node location that the drone will go to next at the tth moment, Indicates the target node location that the vehicle will go to next at the tth moment.
[0066] Steps 1-3, formalize the joint path planning problem into a Markov decision process, including the definition of state s, action a and reward function R. The state s at the tth moment is expressed as It consists of the set of visited nodes, the node location to be visited at the current moment, and the remaining time to reach the destination; the state a at the tth moment is expressed as Indicates the node location that the drone and vehicle are going to next; considering the dynamic update of the state over time, when the action at the tth moment is confirmed, the target location that the drone and vehicle choose to go to next is After that, the remaining time for the drone and vehicle to reach the next node location is updated, and the remaining time for the vehicle is expressed as The remaining time of the drone is expressed as Then the interval from the current moment to the next moment should be the time when the drone or vehicle first arrives at a certain node, expressed as Therefore, at the t+1th moment, the vehicle or drone reaches the next target node The remaining time is and At the t+1th moment, the position node of the vehicle or drone is updated to and The set of visited nodes at time t+1 is updated to Define T as the time step for the vehicle and drone to return to the warehouse after delivering all the targets. Our goal is to complete all delivery tasks in the shortest time, so the reward function is defined as Where C is the time to complete the task.
[0067] In this embodiment, step 2 includes:
[0068] Step 2-1, the node set N set The complete graph G is used as the input of the encoder, and the coordinate position of each node n in the node set is represented as X n =(x n ,y n ), the encoder encodes the node coordinate position into high-dimensional features, and then extracts features and aggregates information through a multi-head attention network, outputting the feature vector h corresponding to node n n ;
[0069] Step 2-2: transform the feature vector h of each node n n Input to the decoder, the structural features obtained after all node feature vectors are downsampled Indicates the correlation between delivery nodes. The decoder consists of an LSTM network and a probabilistic attention network. The LSTM network performs node feature vector fusion to obtain a more comprehensive node feature representation. Probabilistic Attention Network Input Structure Characteristics and the node feature representation at time t Generate the probability distribution p of the node to be selected θ , realize the node selection strategy solution;
[0070] In this embodiment, step 2-1 includes:
[0071] Step 2-1-1, the node set N set The complete graph G is used as the input of the encoder, and the coordinate position of each node n in the node set is represented as X n =(x n ,y n ).
[0072] Step 2-1-2: Use the embedding layer to encode the node coordinate position into the high-dimensional feature F1 n , the expression formula is as follows:
[0073]
[0074] Where φ(·) represents the embedding layer function;
[0075] Step 2-1-3, use the multi-head attention network to convert the high-dimensional feature F1 n As input, and pass messages between nodes. For a node set N set Each node feature vector F1 n , we can get the corresponding query matrix Q n =f Q (F1 n ), the key matrix K n =f K (F1 n ) and value matrix V n =f V (F1 n ), where f Q , f K and f V They are respectively the query function, key function and value function shared by all N node targets.
[0076] Step 2-1-4, based on the input of the previous step, calculate the single-head attention feature vector. Based on the query matrix and the key matrix, we can calculate the relationship matrix u between the i-th node and the j-th node i,j , the relationship matrix is used to model the correlation between two nodes, and the calculation formula is:
[0077]
[0078] where d k Denotes the query matrix Q n The dimension of The meaning here is to combine the distribution of attention coefficients with the query matrix Q n The dimension decoupling of , so that the gradient value remains stable during training. Based on the relationship matrix u i,j The corresponding attention weight a can be calculated i,j , the calculation formula is:
[0079]
[0080] Then extract the attention feature vector F2 corresponding to node n n , the calculation formula is:
[0081] F2 n =Σ j a n,j V n
[0082] Step 2-1-5, connect the feature information of multiple single-head attentions through multi-head attention to achieve information association at different levels. Set the number of multi-head attention heads to M, then you can get the multi-head attention feature vector F3 corresponding to node n n , the calculation formula is:
[0083]
[0084] The multi-head attention feature vector F3 n and the initial eigenvector F1 n Fusion and batch normalization BN are performed to obtain the feature vector F4 n , the calculation formula is as follows:
[0085] F4 n =BN(F1 n +F3 n )
[0086] Step 2-1-6: Perform feature aggregation through a feedforward neural network to output the feature vector h corresponding to node n n , the calculation formula is:
[0087] h n =BN(F4 n +ReLU(FC(F4 n ))
[0088] Among them, FC represents the fully connected layer and ReLU represents the activation function.
[0089] In this embodiment, the step 2-2 includes:
[0090] Step 2-2-1, the feature vector h of each node n n Input to the decoder, the structural features obtained after all node feature vectors are downsampled It represents the correlation between distribution nodes. The calculation formula is as follows:
[0091]
[0092] MEAN means taking the average value of the feature vector, which can reduce the computational complexity of the model.
[0093] Step 2-2-2, transform the feature vector h n Input into the LSTM network for node feature vector fusion to obtain a more comprehensive node feature representation at the tth moment The calculation formula is as follows:
[0094]
[0095] in, Represents the information stored by the LSTM network memory gate. It represents the memory of the input data, including the fusion of all input information of the neural network before time t. Represents the information captured by the output gate.
[0096] Step 2-2-3, Probabilistic Attention Network PA Input Structural Features and the node feature representation at time t Generate probabilistic attention feature vector The calculation formula is as follows:
[0097]
[0098] Among them, τ n,· Represents the time from node n to other nodes. The probabilistic attention network PA is a self-attention layer. Then the probabilistic attention feature vector Perform softmax operation to obtain the probability distribution p of the node to be selected θ , to solve the node selection strategy, the calculation formula is as follows:
[0099]
[0100] In this embodiment, step 3 includes:
[0101] Step 3-1: By parameterizing the reinforcement learning agent behavior strategy network, the probability distribution π of the delivery node sequence can be determined. θ (V T |G), that is, given a complete graph, the sequence of nodes visited by drones and vehicles can be output. Our goal is to minimize the completion time C, so the objective function J(θ|G) of the training is defined, and the calculation formula is:
[0102]
[0103] in represents the mathematical expectation, b(G) represents the average baseline for reducing the gradient variance, which reflects the quality of multiple sampled sequences, so as to better evaluate the quality of the agent's actions. Therefore, the critic network is used to estimate b(G), and the actor network is used to learn the strategy π θ .
[0104] Step 3-2: Train the model parameters through a deep reinforcement learning algorithm based on policy gradient. Input the training data and generate a random strategy p through the parameterized reinforcement learning agent behavior strategy network. θ, randomly select nodes according to the probability distribution to construct a valid solution, execute actions T times, generate multiple feasible solution sequences, calculate the average baseline b(G), and then use the Adam optimizer to update the model parameters. After completing 2000 iterations, the algorithm iteration reward value and the average reward value remain stable, and finally the trained model parameter set θ is obtained. In order to verify the solution effect of the proposed UAV and vehicle joint delivery path optimization algorithm based on deep reinforcement learning, two data sets are constructed by randomly generating examples for model training and testing. The x and y coordinates of each customer node are randomly sampled from a uniform distribution of [1,100]×[1,100]. In order to make the warehouse location at the inflection point, the warehouse coordinates are randomly sampled from a uniform distribution of [0,1]×[0,1]. The reinforcement learning algorithm proposed in this paper is compared with the greedy algorithm. As shown in Table 1, the total path length cost and the percentage of error with the optimal solution show that the reinforcement learning algorithm has higher efficiency and performance.
[0105] Table 1
[0106]
[0107] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of the method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning provided by the present invention and some or all of the steps in each embodiment can be run. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0108] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention are essentially or partly contributed to the prior art can be embodied in the form of a computer program, i.e., a software product, which can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.
[0109] The present invention provides a thought and method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.
Claims
1. A method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning, characterized in that: The steps include: Step 1: Mathematically model the path planning problem of joint delivery of a vehicle and a drone to more than two target addresses. Set the warehouse location and the target location as nodes, construct a complete graph G consisting of a set of nodes, and formalize the joint path planning problem into a Markov decision process by defining the state s, action a and reward function R; Step 2: Input the complete graph G obtained in step 1, construct a parameterized reinforcement learning agent behavior strategy network consisting of an encoder and a decoder, perform feature extraction and information aggregation through the encoder, solve the node selection strategy through the decoder, and generate the probability distribution p of the nodes to be selected θ ; Step 3: Through the policy gradient-based deep reinforcement learning algorithm, the probability distribution model parameters are trained to generate a visit node sequence of drones and vehicles based on the probability distribution model given a node graph, and a joint delivery path of drones and vehicles is generated according to the visit node sequence.
2. The method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning according to claim 1 is characterized in that: The method described in step 1 to construct the complete graph and form the Markov decision process includes: Step 1-1, conditionally constrain the path planning problem of a vehicle and a drone performing joint delivery to more than two targets; Step 1-2, mathematical modeling of the joint distribution path planning problem: Node Set N set = A complete graph G consisting of {m, 1, 2, 3..., N}, where node m represents a warehouse node, nodes 1, 2, 3..., N represent target nodes, and the distance from node i to node j is d i,j ; and denote the delivery time of vehicle and drone from node i to node j respectively; and They represent the node positions that the UAV and vehicle are about to reach or are currently at at the tth moment; and Respectively represent the locations of the drone and vehicle arriving at the node and The remaining time, if the drone or vehicle has arrived, or V t represents the set of nodes visited by drones or vehicles from time 0 to time t; and They represent the target node locations that the UAV and vehicle will go to next at the tth moment respectively; Steps 1-3 formalize the joint path planning problem into a Markov decision process, including the definition of state s, action a, and reward function R.
3. The method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning according to claim 1 is characterized in that: The conditional constraints described in step 1-1 include: Use vehicles or drones to deliver goods to each target node and return to the warehouse in the shortest time possible. The number of vehicles and drones is 1. The drone delivers at the maximum flight speed. The drone can only deliver to one target each time it is launched, and then the drone must meet up with the vehicle. When the drone and vehicle meet at a certain target node, the one that arrives earlier waits for the other.
4. The method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning according to claim 1 is characterized in that: The definitions described in steps 1-3 include: The state s at the tth moment is expressed as It consists of the set of visited nodes, the node location to be visited at the current moment, and the remaining time to arrive; The action a at the tth moment is expressed as Indicates the node location where the drone and vehicle will go next; The state is dynamically updated as time changes. When the action at the tth moment is confirmed, the next target location selected by the drone and the vehicle is Afterwards, the remaining time for the drone and vehicle to reach the next node location is updated, and the remaining time for the vehicle is expressed as The remaining time of the drone is expressed as The interval from the current moment to the next moment is the time when the drone or vehicle first arrives at a certain node, expressed as At time t+1, the vehicle or drone reaches the next target node The remaining time is and At the t+1th moment, the position node of the vehicle or drone is updated to and The set of visited nodes at time t+1 is updated to Define T as the time step for vehicles and drones to return to the warehouse after delivering all targets. The mission goal is to complete all delivery tasks in the shortest time, and the reward function R is defined as the task completion time 5. The method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning according to claim 1 is characterized in that: The method for constructing a parameterized reinforcement learning agent behavior policy network described in step 2 includes: Step 2-1: The node set N obtained in step 1 set The complete graph G is used as the input of the encoder, and the coordinate position of each node n in the node set is represented as X n =(x n ,y n ), the encoder encodes the node coordinate position into high-dimensional features, and then extracts features and aggregates information through a multi-head attention network, outputting the feature vector h corresponding to node n n ; Step 2-2: transform the feature vector h of each node n n Input to the decoder, the structural features obtained after all node feature vectors are downsampled Indicates the correlation between delivery nodes. The decoder consists of an LSTM network and a probabilistic attention network. The LSTM network performs node feature vector fusion to obtain a more comprehensive node feature representation. Probabilistic Attention Network Input Structure Characteristics and the node feature representation at time t Generate the probability distribution p of the node to be selected θ , realize the node selection strategy solution.
6. The method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning according to claim 5 is characterized in that: The method for constructing the encoder described in step 2-1 includes: Step 2-1-1, the node set N set The complete graph G is used as the input of the encoder, and the coordinate position of each node n in the node set is represented as X n =(x n ,y n ); Step 2-1-2: Use the embedding layer to encode the node coordinate position into the high-dimensional feature F1 n , the expression formula is as follows: Where φ(-) represents the embedding layer function; Step 2-1-3, use the multi-head attention network to convert the high-dimensional feature F1 n As input, and passing messages between nodes, for the node set N set Each node feature vector F1 n , we can get the corresponding query matrix Q n , the bond matrix K n and the value matrix V n ; Step 2-1-4, based on the input of step 2-1-3, calculate the single-head attention feature vector, and calculate the relationship matrix u between the i-th node and the j-th node according to the query matrix and the key matrix i,j , the relationship matrix is used to model the correlation between two nodes, and the calculation formula is: where d k Denotes the query matrix Q n The dimension of Represents the distribution of attention coefficients and the query matrix Q n The dimension decoupling makes the gradient value stable during training. Based on the relationship matrix u i,j Calculate the corresponding attention weight a i,j , the calculation formula is: Then extract the attention feature vector F2 corresponding to node n n , the calculation formula is: F2 n =∑ j and n,j In n ; Step 2-1-5, connect the feature information of multiple single-head attentions through multi-head attention to achieve information association at different levels. Set the number of multi-head attention heads to M, and then you can get the multi-head attention feature vector F3 corresponding to node n n , the calculation formula is: The multi-head attention feature vector F3 n and the initial eigenvector F1 n Fusion and batch normalization BN are performed to obtain the feature vector F4 n , the calculation formula is as follows: F4 n =BN(F1 n +F3 n ); Step 2-1-6: Perform feature aggregation through the feedforward neural network to output the feature vector h corresponding to node n n , the calculation formula is: h n =BN(F4 n +ReLU(FC(F4 n )); Among them, FC represents the fully connected layer and ReLU represents the activation function.
7. The method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning according to claim 6 is characterized in that: The query matrix Q n =f Q (F1 n ), the key matrix K n =f K (F1 n ) and the value matrix V n =f V (F1 n ), where f Q , f K and f V They are respectively the query function, key function and value function shared by all N node targets.
8. The method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning according to claim 5 is characterized in that: The method for constructing the decoder described in step 2-2 includes: Step 2-2-1, the feature vector h of each node n n Input to the decoder, the structural features obtained after all node feature vectors are downsampled It represents the correlation between distribution nodes. The calculation formula is as follows: Among them, MEAN means to find the average value of the feature vector; Step 2-2-2, transform the feature vector h n Input into the LSTM network for node feature vector fusion to obtain a more comprehensive node feature representation at the tth moment The calculation formula is as follows: in, represents the information stored by the LSTM network memory gate, Represents the memory of input data, including the fusion of all input information of the neural network before time t. Represents the information captured by the output gate; Step 2-2-3, Probabilistic Attention Network PA Input Structural Features and the node feature representation at time t Generate probabilistic attention feature vector The calculation formula is as follows: Among them, τ n,· Represents the time from node n to other nodes. The probabilistic attention network PA is a self-attention layer, and then the probabilistic attention feature vector Perform softmax operation to obtain the probability distribution p of the node to be selected θ , to solve the node selection strategy, the calculation formula is as follows:
9. The method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning according to claim 1 is characterized in that: The policy gradient-based deep reinforcement learning algorithm described in step 3 includes: Step 3-1: Determine the probability distribution π of the delivery node sequence through the parameterized reinforcement learning agent behavior strategy network θ (V T |G), that is, given a complete graph, output the visited node sequence of the drone and the vehicle, define the training objective function J(θ|G), and the calculation formula is: in represents the mathematical expectation, b(G) represents the average baseline that reduces the gradient variance, the critic network is used to estimate b(G), and the actor network is used to learn the strategy π θ ; Step 3-2: Use the policy gradient-based deep reinforcement learning algorithm to train the model parameters, input the training data, and generate a random strategy p through the parameterized reinforcement learning agent behavior strategy network. θ , randomly select nodes according to the probability distribution to construct a valid solution, execute the action T times, generate multiple feasible solution sequences, calculate the average baseline b(G), and then use the Adam optimizer to update the model parameters. After completing the specified number of iterations, the trained model parameter set θ is obtained.
10. A navigation system, characterized in that: A method for optimizing the joint delivery path of drones and vehicles based on reinforcement learning, comprising any one of claims 1 to 9.
Citation Information
Cited By
Method and device for optimizing automatic wiring path of optical fiber wiring cabinet
CN119442815A
Optical fiber distribution cabinet automatic wiring path optimization method and device
CN119442815B
