Unmanned vehicle distribution path optimization method based on double time scale

Abstract

The application provides a double-time-scale-based unmanned vehicle distribution path optimization method, and belongs to the field of reinforcement learning.The double-time-scale-based unmanned vehicle distribution path optimization method comprises the following steps: establishing a distribution problem into a reinforcement learning environment model, obtaining state information of a current environment, including merchant coordinates, customer coordinates, path distances from the merchants to various customers, distances between various customers, and time for the unmanned vehicle to deliver goods to customers after picking up the goods; then inputting the current state information and action information into a double-time-scale algorithm for training, and through continuous training, an optimal strategy of the unmanned vehicle in a distribution path can be obtained.The algorithm uses a kernel-based value function, which to some extent overcomes the problems of a traditional complex value function and catastrophic interference, so that a more optimal distribution path is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to a double-time-scale-based unmanned vehicle delivery path optimization method, belonging to the field of reinforcement learning. BACKGROUND

[0002] Cargo delivery, as an important link in many industries such as express delivery and take-out industry, has received extensive attention from major platforms and customers. In the delivery process, if only the experience of the delivery personnel is used for delivery, it is easy to cause unreasonable delivery path and unnecessary loss. Therefore, the unmanned vehicle is used to deliver goods, and scientific path planning is needed to improve this phenomenon.

[0003] In existing research, the cargo delivery path optimization problem can be modeled as a Markov decision process, and reinforcement learning is one of the methods specifically used to solve the Markov decision process. Traditional reinforcement learning algorithms usually face the problems of complex value function approximation and catastrophic interference. Therefore, it is hoped to seek an improved algorithm to solve the optimization problem of take-out delivery path.

[0004] Therefore, it is necessary to provide a double-time-scale-based unmanned vehicle delivery path optimization method to solve the above problems. SUMMARY

[0005] The purpose of the present application is to provide a double-time-scale-based unmanned vehicle delivery path optimization method, so that the unmanned vehicle can deliver goods in a shorter path and shorter time, effectively improving the delivery efficiency and having good use value and economic value.

[0006] To achieve the above purpose, the present application provides a double-time-scale-based unmanned vehicle delivery path optimization method for optimizing the cargo delivery path of the unmanned vehicle after the unmanned vehicle obtains goods from a merchant and delivers to multiple customers, comprising the following steps:

[0007] Step 1: Establish a reinforcement learning environment model according to the requirements of the unmanned vehicle delivery path;

[0008] Step 2: Obtain the current environment state information s;

[0009] Step 3: Input the current state information and the optional actions of the unmanned vehicle into the double-time-scale-based algorithm to obtain the value of each action, select the action a by the epsilon-greedy method, execute the action a, and enter the next state s ′ , obtain the reward R;

[0010] Step 4: The unmanned vehicle updates the parameters by using the double-time-scale optimization algorithm, thereby updating the current strategy;

[0011] Step 5: Repeat steps 2 to 4 above until the unmanned vehicle reaches the shortest path requirement or the maximum number of iterative updates is reached.

[0012] As a further improvement of the present invention, step 1 specifically includes: establishing a reinforcement learning environment model according to the requirements of the unmanned vehicle delivery path, requiring the random generation of merchant locations in the environment, and generating a random number of customers and random customer locations near the merchants, which are consistent with the actual goods delivery situation.

[0013] As a further improvement of the present invention, step 2 specifically includes: the current status information s includes the merchant coordinates, customer coordinates, unmanned vehicle coordinates, the path distance from the merchant to each customer, the distance between each customer, the unmanned vehicle delivery time, and the time when the unmanned vehicle delivery is completed.

[0014] As a further improvement of the present invention, step 3 specifically includes: based on the dual time scale algorithm where k(s,s i ) represents the kernel function, a w (s,s i ) represents the attention function.

[0015] As a further improvement of the present invention, step 3 specifically includes: when the unmanned vehicle reaches a customer, it obtains a reward R=-d, which is the negative of the total distance traveled from the merchant to the current customer.

[0016] As a further improvement of the present invention, the step 4 specifically includes:

[0017] Step 41: Construct a valid state dictionary, compare the current state s with the states in the dictionary, and Construct the effective state dictionary D t ,in Represents the current state s and the state s in the dictionary t Do the state space difference. If the distance is far, add the data to the dictionary, otherwise do not add it.

[0018] Step 42: traverse the current state in the dictionary one by one and compare it with all the states in the dictionary. Construct the attention function, where ψ(s,s i ) represents state s and state s i The feature vector between , w represents the weight parameter of the attention function, Indicates transposing the weight parameters, s j is the state in the dictionary D, ψ(s,s j ) represents state s and state s jThe eigenvector between the two is then used to train the slow update of the attention function parameter using reinforcement learning, and the optimization of the attention function is as follows loss function: Where d(s) represents the distribution probability of state s, and R represents the reward, and the updates of theta and w are as follows:

[0019]

[0020]

[0021] Where theta represents the feature weight parameter vector of the updated attention function, t represents the current time step, t+1 represents the next time step of the current time step, and k(s t ,s i ) represents a Gaussian kernel function, and gamma represents a discount factor; alpha and beta are learning rates of theta and w respectively, and delta t is the difference between the current state and the expected state,

[0022] Step 43, the value function parameter theta is updated quickly using the semi-gradient descent method, and the value function is Where k(s,s i ) represents a Gaussian kernel function, and a w (s,s i ) represents an attention function, and the method is used to measure the error between the estimated value V θ,w and the true value V π , The semi-gradient descent method is used to update theta, and the update formula is as follows: Where beta t is the fast update learning rate, delta t is the difference between the current state and the expected state, and

[0023] The beneficial effects of the present application are: a double-time-scale-based unmanned vehicle distribution path optimization method is proposed, which can make the unmanned vehicle deliver in a shorter path and shorter time compared with the traditional method, effectively improve the delivery efficiency, and has good use value and economic value. BRIEF DESCRIPTION OF DRAWINGS

[0024] Fig. 1 The flowchart of the double-time-scale-based unmanned vehicle distribution path optimization method of the present application.

[0025] Fig. 2 The double-time-scale-based unmanned vehicle distribution path optimization method of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0026] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be described in detail below with reference to the drawings and specific embodiments.

[0027] It should be noted here that, in order to avoid obscuring the present application due to unnecessary details, only structures and / or processing steps closely related to the solutions of the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0028] In addition, it should also be noted that the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device.

[0029] As shown in Figs. 1-2 The present application proposes a double-time-scale-based unmanned vehicle delivery path optimization method, which can accelerate the convergence to the optimal strategy by using the slow update attention function and the fast update target function of the double-time scale, avoid the problems of value function approximation and dimension disaster, improve the efficiency of take-out delivery, shorten the time and delivery path, effectively solve the existing problems in the take-out delivery path, and has good use value and economic benefits. The main steps include:

[0030] Step 1, establishing a reinforcement learning environment model according to the requirements of the unmanned vehicle delivery path;

[0031] Step 2, obtaining the current environment state information s;

[0032] Step 3, inputting the current state information and the optional actions of the unmanned vehicle into the double-time-scale-based algorithm to obtain the value value corresponding to each action, selecting the action a through the ε-greedy method, executing the action a, and entering the next state s ′ , obtaining the reward R;

[0033] Step 4, the unmanned vehicle updates the parameters by using the double-time-scale optimization algorithm, thereby updating the current strategy;

[0034] Step 5, repeating the steps 2-4 until the unmanned vehicle reaches the shortest path requirement or reaches the maximum number of iteration updates.

[0035] The steps 1-5 will be described in detail below.

[0036] In step 1, an environment model for reinforcement learning is established according to the requirements of the unmanned vehicle delivery path. Randomly generate merchant locations within the environment, generate random customer quantities near merchants, and generate random customer locations to match actual goods delivery situations. The model will create a map based on the actual situation in a certain area, where each merchant and customer will become a node, and the edges on the map will represent the actual distance between nodes to reflect actual traffic conditions and road networks. In the model, random customer quantities and locations are generated near merchants to simulate actual goods delivery situations.

[0037] In step 2, the current state information s includes the location coordinates of the merchant on the graph, the location coordinates of each customer on the graph, the location coordinates of the unmanned vehicle on the graph, the shortest path distance between the merchant and each customer, the shortest path distance between different customers, the time the unmanned vehicle has been used for delivery tasks, and the time the unmanned vehicle completes delivery for customers who have completed delivery.

[0038] In step 3, the current state information and the available actions of the unmanned vehicle are concatenated into the current state s, and the current state s is input into the double-time scale algorithm: where k(s, s i ) represents the kernel function, a w (s, s i ) represents the attention function, the value is calculated, the action a is selected by the ε-greedy method, the action a is executed, and the next state s ′ is entered. The unmanned vehicle reaches a customer and receives a reward R = -d, where the reward R is the negative of the total distance traveled from the merchant to the current customer within the time.

[0039] In step 4, it specifically includes:

[0040] Step 41, construct an effective state dictionary, compare the current state s with the states in the dictionary, and according to construct an effective state dictionary D t . Where represents the distance between the current state s and the state s t in the dictionary. If the distance is far, the data is added to the dictionary, otherwise it is not added.

[0041] Step 42, traverse the current state in the dictionary one by one, compare it with all the states in the dictionary, and according to construct an attention function, where ψ(s, s i ) represents the feature vector between state s and state s i , w represents the weight parameter of the attention function, represents the transpose of the weight parameter, s j is the state in the dictionary D, and ψ(s, s j) represents the feature vector between state s and state s j The slow update of the attention function parameter is trained by using the reinforcement learning acceleration method, and the optimization of the attention function is as follows loss function: Wherein d(s) represents the distribution probability of state s, R represents the reward, and the update is performed on theta and w respectively, and the final attention function update formula is as follows:

[0042]

[0043]

[0044] Wherein, theta represents the feature weight parameter vector of the updated attention function, t represents the current time step, t+1 represents the next time step of the current time step, and k(s t ,s i ) represents a Gaussian kernel function, and gamma represents a discount factor; alpha and beta are learning rates of theta and w respectively, and delta t is the difference between the current state and the expected state,

[0045] In step 43, the value function parameter theta is updated quickly by using the semi-gradient descent method, and the value function is Wherein, k(s,s i ) represents a Gaussian kernel function, and a w (s,s i ) represents an attention function, and the method is used to measure the error between the estimated value V θ,w and the true value V π , The semi-gradient descent method is used to update theta, and the update formula is as follows: Wherein beta t is a fast update learning rate, delta t is the difference between the current state and the expected state, and

[0046] In step 5, it is judged whether the policy converges or whether the policy training reaches the maximum training iteration number. Preferably, the maximum training iteration number is 10000 times.

[0047] In summary, the method can realize the optimal control strategy by the fast and slow update of the double time scale, effectively avoid the problems of complex value function approximation and catastrophic interference, and make the unmanned vehicle reach in a shorter path and shorter time compared with the traditional method, effectively improve the delivery efficiency, and have good use value and economic value.

[0048] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A dual-time-scale unmanned vehicle delivery path optimization method, which is used for unmanned vehicles to deliver goods to multiple customers after obtaining goods from merchants, and optimize the unmanned vehicle cargo delivery path, characterized by: The following steps are involved: Step 1: Establish a reinforcement learning environment model based on the requirements of the unmanned vehicle delivery route; Step 2: Get the current environment status information ; Step 3: Input the current state information and the optional actions of the unmanned vehicle into the algorithm based on the dual time scale to obtain the corresponding Value, through Method selection action , perform the action , enter the next state , get rewards ; Step 4: The unmanned vehicle uses a dual-time-scale optimization algorithm to update the parameters, thereby updating the current strategy; Step 5: Repeat steps 2 to 4 above until the unmanned vehicle reaches the shortest path requirement or the maximum number of iterations is reached; The step 4 specifically includes: Step 41: Construct a valid state dictionary and set the current state Compare with the status in the dictionary, according to Constructing a valid state dictionary ,in Represents the current state s and the state in the dictionary Do the state space difference. If the distance is far, add the data to the dictionary, otherwise do not add it. Step 42: traverse the current state in the dictionary one by one and compare it with all the states in the dictionary. Construct an attention function, where Indicates status and status The eigenvectors between represents the weight parameter of the attention function, Indicates transposing the weight parameters. For dictionary The state in Indicates status and status Then, the reinforcement learning training acceleration method is used to slowly update the attention function parameters. The optimization loss function of the attention function is as follows: ,in Indicates status The distribution probability of R represents the reward, respectively. After updating, the final attention function update formula is as follows: in, Represents the feature weight parameter vector for updating the attention function, t represents the current time step, and t+1 represents the next time step of the current time step; represents the Gaussian kernel function, represents the discount factor; and They are and The learning rate, is the difference between the current state and the expected state, ; Step 43: Use the semi-gradient descent method to adjust the value function parameters Perform fast update, the value function is ,in, represents the Gaussian kernel function, Represents the attention function, which is a method that measures the estimated value by minimizing the mean square error and the true value The error between , using the semi-gradient descent method, Update, the update formula is as follows: ,in is the fast update learning rate, is the difference between the current state and the expected state, and .

2. The dual-time-scale unmanned vehicle delivery path optimization method according to claim 1 is characterized in that: The step 1 specifically includes: establishing a reinforcement learning environment model according to the requirements of the unmanned vehicle delivery path, requiring the random generation of merchant locations in the environment, and generating a random number of customers and random customer locations near the merchants, which are consistent with the actual goods delivery situation.

3. The dual-time-scale unmanned vehicle delivery path optimization method according to claim 1 is characterized in that: The step 2 specifically includes: current status information It includes the merchant coordinates, customer coordinates, unmanned vehicle coordinates, the path distance from the merchant to each customer, the distance between each customer, the delivery time of the unmanned vehicle, and the time when the unmanned vehicle completes the delivery.

4. The dual-time-scale unmanned vehicle delivery path optimization method according to claim 1 is characterized in that: The step 3 specifically includes: based on the dual time scale algorithm ,in represents the weight parameter, Indicates the current state, Representation dictionary The state in represents the kernel function, represents the attention function.

5. The dual-time-scale unmanned vehicle delivery path optimization method according to claim 1 is characterized in that: The step 3 specifically includes: the unmanned vehicle arrives at a customer and receives a reward , which is the negative of the total distance from the merchant to the current customer.

Images