Multi-AGV path planning method based on DDQN algorithm

Abstract

The invention relates to a multi-AGV path planning method based on a DDQN algorithm. The method comprises the steps of obtaining an AGV transportation task according to a warehouse layout; establishing a state space, an action space and a reward function of the AGV; constructing an AGV path planning model; acquiring a current state from the environment and inputting the current state into the main network; the main network generates a current Q value and inputs the Q value to a DDQN error function; the main network calculates a target Q value and calculates a TD error according to the current Q value and the target Q value; storing the TD error and the experience sample into an experience pool; the target network obtains a next state from the experience pool, correspondingly generates an optimal Q value of the next state, and inputs the optimal Q value of the next state into the DDQN error function; the DDQN error function calculates a loss function value according to the current Q value and the optimal Q value in the next state, calculates an error function gradient through back propagation, and updates network parameters; and carrying out loop iteration until the maximum number of training rounds is reached, obtaining a trained AGV path planning model, and carrying out multi-AGV path planning.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent logistics, and in particular relates to a multi-AGV path planning method based on a DDQN algorithm. Background Art

[0002] With the rapid development of intelligent manufacturing and e-commerce, the scale of workshop automation and express warehousing has gradually expanded, and the efficiency of material handling has become a key factor in improving production and distribution speed. In order to cope with the increasingly complex logistics needs, AGV (Automated Guided Vehicle) is widely used in various warehouses and workshops, using automated handling, sorting, storage and other operations to reduce labor costs and improve production efficiency. However, with the increasing complexity of the warehousing environment and logistics system, AGV faces many challenges in actual operation, such as multi-AGV collision avoidance, increased scheduling time, and path congestion. These problems make it difficult for existing AGV systems to make efficient and flexible decisions during path planning, which seriously affects the overall operational efficiency of logistics and production.

[0003] Currently in the field of AGV path planning, traditional heuristic algorithms, such as A*, Dijkstra and genetic algorithms, can find a better path for AGV, but they have problems such as long calculation time when solving large-scale and complex scheduling problems. At the same time, they have low update efficiency and lack of adaptability when facing dynamic environmental changes.

[0004] In order to solve the above defects, deep reinforcement learning (DRL) has gradually been applied to AGV path planning, such as applying the DQN algorithm to feedback the learning environment to achieve path optimization. However, the above algorithms often require a large number of training samples, slow training convergence, and have problems such as unstable convergence and long path planning time when facing high-dimensional complex environments. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-AGV path planning method based on the DDQN algorithm. The method has a fast training convergence speed, can adapt to the complex multi-AGV storage environment, and has the advantage of high computing speed. It can efficiently complete multi-AGV scheduling and improve storage sorting efficiency.

[0006] In order to achieve the above object, the specific technical solution adopted by the present invention is:

[0007] A multi-AGV path planning method based on DDQN algorithm, characterized by comprising:

[0008] S1. Obtain AGV transportation tasks based on warehouse layout; establish AGV state space, action space and reward function;

[0009] S2, constructing an AGV path planning model; obtaining the current state from the environment and inputting the current state into the main network; the main network generates a current Q value and inputs it into the DDQN error function;

[0010] S3, the main network calculates the target Q value, and calculates the TD error according to the current Q value and the target Q value; the TD error and the experience sample are stored in the experience pool; the experience sample includes the next state;

[0011] S4, the target network obtains the next state from the experience pool and generates the optimal Q value of the next state accordingly, and inputs the optimal Q value of the next state into the DDQN error function;

[0012] S5, the DDQN error function calculates the loss function value according to the current Q value and the optimal Q value of the next state, and back-propagates to calculate the error function gradient and update the network parameters;

[0013] S6. Iterate the loop until the maximum number of training rounds is reached, obtain the trained AGV path planning model and perform multi-AGV path planning.

[0014] As a preferred embodiment of the present invention, the main network includes a policy network and a D* algorithm network; and an exploration rate selection method is used to set the selection weights of the policy network and the D* algorithm network.

[0015] As a preferred embodiment of the present invention, the initial value of the exploration rate in the exploration rate selection method is set to 0.8; and the final value of the exploration rate in the exploration rate selection method is set to 1.

[0016] As a preferred embodiment of the present invention, the D* algorithm network is improved using the AGV turning penalty coefficient to obtain an improved D* algorithm network.

[0017] As a preferred embodiment of the present invention, in S1, the reward function is specifically:

[0018] R=r t +r c +r d

[0019] Among them, R is the reward function value; r t Reward for reaching the target point; c Punishment for illegal behavior; d Reward for distance.

[0020] As a preferred embodiment of the present invention, in S3, the TD error is calculated as follows:

[0021]

[0022] Among them, Q(s,a) is the current state Q value; is the target Q value; r is the current reward; γ is the discount factor; is the optimal Q value for the next state.

[0023] As a preferred embodiment of the present invention, the multi-AGV path planning method further includes a target network synchronization step, specifically:

[0024] The training step counting parameters are set and the synchronization timing is determined according to the training step counting parameters; at the synchronization timing, the parameters of the strategy network are copied to the target network, and the parameters of the target network are replaced.

[0025] As a preferred embodiment of the present invention, the loss functions of the strategy network and the target network are specifically:

[0026]

[0027] Where N is the batch size; Q(s i ,a i ) is in the current state s i The Q value predicted by the model is i is the reward in the current state; is the optimal Q value for the next state.

[0028] As a preferred embodiment of the present invention, in S3, the target Q value is calculated using the Bellman formula.

[0029] As a preferred embodiment of the present invention, in S6, the maximum number of training rounds is 300 rounds.

[0030] In summary, the beneficial effects of the present invention are:

[0031] 1. This method introduces the D* algorithm to directly determine the optimal action, reduces the proportion of random search, effectively improves the convergence speed of model training, solves the problem of low search efficiency of the traditional DDQN algorithm in exploring the action space through random strategies in the initial stage, and significantly shortens the training convergence time.

[0032] 2. The D* algorithm introduces global heuristic search, which enables the algorithm to quickly obtain the approximate optimal path direction in the early stage, providing high-quality experience samples for subsequent Q-value updates, thereby accelerating the optimization of the strategy and improving the globality of the strategy.

[0033] 3. Since turning will increase the running time in the actual warehousing environment, the present invention reduces the number of turns and improves the logistics efficiency by introducing the AGV turning penalty coefficient in the D* algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of the multi-AGV path planning method;

[0035] Figure 2 This is a structural diagram of the AGV path planning model. DETAILED DESCRIPTION

[0036] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0037] The terms "first", "second", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any variation thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0038] like Figure 1 As shown, in this embodiment, the AGV transportation task is first obtained according to the warehouse layout; and the state space, action space and reward function of the AGV are established.

[0039] The reward function value is calculated by the following reward function formula:

[0040] R=r t +r c +r d

[0041] r t =10

[0042] r c =r c1 +r c2 +r c3 +r c4

[0043]

[0044] Among them, R is the reward function value; r t There are three types of rewards for reaching the target point: one is to arrive at the pickup location empty, the second is to lift the shelf to reach the sorting station, and the third is to put the shelf back to its original position; c Punishment for illegal behavior, where r c1 represents the AGV out-of-bounds penalty, d1 represents the distance between the AGV and the grid boundary; r c2represents the penalty for collision between AGV and shelf, d2 represents the distance between AGV and shelf; r c3 represents the penalty for collision between AGV and sorting station, d3 represents the distance between AGV and the boundary of grid map; r c4 represents the penalty for AGV collision, d4 represents the distance between AGVs; r d is the distance reward. If the AGV's next action reduces the distance to the target point, a reward of 0.3 is given. current is the distance from the current point to the target point, d last It is the distance from the previous step to the target point, and the distance is calculated by the D* algorithm.

[0045] Next, construct the AGV path planning model, such as Figure 2 As shown in the figure, the AGV path planning model consists of a main network, a target network, an experience pool, and a DDQN error function.

[0046] The current state is obtained from the environment and input into the main network. The main network includes a policy network and a D* algorithm network. The selection weights of the policy network and the D* algorithm network are set using the exploration rate selection method. Specifically, the initial value of the exploration rate in the exploration rate selection method is set to 0.8 and the final value is set to 1, which makes the action selection from the initial 80% policy network and 20% D* algorithm weights, gradually changing to all selections by the policy network.

[0047]

[0048] Set a relatively high reward function value, where c is the number of times this reward value is reached, and n is the total number of times that need to be reached. When c is equal to n, all decisions are made by the policy network.

[0049] By obtaining map information and AGV transportation tasks, including the coordinates of the current AGV position, target shelf, and target picking station on the map, we can obtain the starting and ending coordinates of the three paths from the current AGV position to the target shelf, from the target shelf to the target picking station, and from the target picking station back to the shelf. Based on the k value, we use the D* algorithm to search for the best path, which is as follows:

[0050] Rating function: k = <min[g(s),rhs(s)]+h(s,s start ),min[g(s),rhs(s)]>

[0051] Where min[g(s), rhs(s)] represents the actual cost from the current point to the end point, rhs(s) represents the actual cost from the current position to the end point, which is accumulated in sequence, and g(s) also represents the actual cost from the current position to the end point, and its value is given by rhs(s); h(s, s start) Represents the estimated cost from the current position to the starting point, using the Manhattan distance as the estimated path cost. k has two elements. First, the size is determined by the first element. Only when the first elements are equal, the size is determined by the second element.

[0052] Put the target point into the OpenList. Define the rhs value of the target point as 0, set g = rhs for the target point, and remove this node from the OpenList and put it into the closedlist. Calculate the rhs values for all neighbor nodes. Determine whether the g value of the current expanded point is the same as its rhs value. If g = rhs, it is in a locally consistent state, indicating that the node is in a stable state with little error between the estimate and the actual situation, and keep it; if g < rhs, it means there is a better path, in a locally over - consistent state, let g = rhs to become a locally consistent state, and at the same time update all adjacent nodes and remove them from the list; if g > rhs, it means this node has been affected by obstacles, let g = ∞, traverse the adjacent nodes of this node to see if it is possible to use a certain adjacent node as the parent node to reduce the rhs value of this node. Conduct the next round of expansion, and loop in turn. After the expansion is completed, find the required path through path backtracking.

[0053] Use the policy network to predict the Q value in the current state s and input it into the DDQN error function. The Q value measures the total future reward that can be obtained after taking action a from state s. The policy network takes the current state s as input and calculates a Q - value vector through a neural network. This vector contains the Q values corresponding to all possible actions, can evaluate the expected value of each possible action in the current state, and select the optimal action according to the Q value to maximize the future cumulative reward.

[0054] Then, the policy network combines the Bellman formula to calculate the target Q value, and then calculates the TD error according to the current Q value and the target Q value; the TD error is the priority error. It is based on the difference between the Q value predicted by the current policy network and the target Q value. The larger the TD error, the greater the contribution of this sample to the training, and it should be preferentially sampled. The specific calculation method of the TD error is as follows:

[0055]

[0056] Among them, Q(s,a) is the current state Q value; is the target Q value; r is the current reward; γ is the discount factor; is the optimal Q value of the next state.

[0057] Then, based on the priority experience replay mechanism, the TD error and experience samples are stored in the experience pool. Experience samples (s, a, r, s') are data recorded after the network model interacts with the environment. They indicate that the network model selects an action (a) at a certain moment (s), and obtains a reward (r) and the next state (s') reached based on the action.

[0058] Then, the target network obtains the next state from the experience pool and generates the optimal Q value for the next state accordingly, and inputs the optimal Q value for the next state into the DDQN error function.

[0059] After that, the DDQN error function calculates the loss function value based on the current Q value and the optimal Q value of the next state, and back-propagates to calculate the error function gradient and updates the network parameters. Back-propagation can transfer the error of the loss function from the output layer to the input layer, and update the network weights layer by layer. Then calculate the gradient of each parameter to the loss function, use the calculated gradient to update the network weights, and adjust the model parameters to reduce the loss.

[0060] Finally, the loop is iterated until the maximum number of training rounds is reached. The length of one training round (episode) is set to 300 steps for system training, the number of training rounds is 300 rounds, all rounds of training are completed, the trained AGV path planning model is obtained, and multi-AGV path planning is performed.

[0061] In another possible embodiment, the D* algorithm network is improved using the AGV turning penalty coefficient to obtain an improved D* algorithm network. Specifically, the improved D* algorithm network is:

[0062] h=h d +h p

[0063] h d =|x i -x s |+|y i -y s |

[0064]

[0065] Where h is the estimated path cost; h d is the Manhattan distance; (x i ,y i ) is the current position; (x s ,y s ) is the starting point; h p is the penalty cost; n is the total number of nodes; (x i-1 ,y i-1 ) represents the parent node; (x i-2 ,y i-2) represents the grandfather node. By introducing the turning penalty coefficient, the number of turns can be reduced and the logistics efficiency can be improved.

[0066] In another possible embodiment, the loss functions of the policy network and the target network are specifically:

[0067]

[0068] Where N is the batch size; Q(s i ,a i ) is in the current state s i The Q value predicted by the model is i is the reward in the current state; is the optimal Q value for the next state.

[0069] In another possible embodiment, the multi-AGV path planning method further includes a target network synchronization step, specifically: setting training pedometer parameters and determining the synchronization timing according to the training pedometer parameters; at the synchronization timing, copying the parameters of the policy network to the target network, and replacing the parameters of the target network. The parameters of the target network are a copy of the policy network, but its update frequency is slower. The target network is used to calculate the optimal Q value of the next state. In addition, after a certain number of training steps, a synchronization opportunity is generated, at which time the parameters of the policy network are synchronized to the target network. Synchronizing at intervals of a certain number of training steps can slow down the speed of updating the target network, avoid problems such as training instability, overfitting or inefficiency caused by frequent network updates, and thus improve training efficiency.

[0070] The above descriptions are only preferred embodiments disclosed in this application and descriptions of the technical principles used. Those skilled in the art should understand that the scope of protection involved in this disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this disclosure (but not limited to) to form a technical solution.

[0071] In addition, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

Claims

1. A multi-AGV path planning method based on DDQN algorithm, characterized in that: include: S1. Obtain AGV transportation tasks based on warehouse layout; establish AGV state space, action space and reward function; S2, constructing an AGV path planning model; obtaining the current state from the environment and inputting the current state into the main network; the main network generates a current Q value and inputs it into the DDQN error function; S3, the main network calculates the target Q value, and calculates the TD error according to the current Q value and the target Q value; the TD error and the experience sample are stored in the experience pool; the experience sample includes the next state; S4, the target network obtains the next state from the experience pool and generates the optimal Q value of the next state accordingly, and inputs the optimal Q value of the next state into the DDQN error function; S5, the DDQN error function calculates the loss function value according to the current Q value and the optimal Q value of the next state, and back-propagates to calculate the error function gradient and update the network parameters; S6. Iterate the loop until the maximum number of training rounds is reached, obtain the trained AGV path planning model and perform multi-AGV path planning.

2. According to the multi-AGV path planning method based on the DDQN algorithm of claim 1, it is characterized in that: The main network includes a policy network and a D* algorithm network; and an exploration rate selection method is used to set the selection weights of the policy network and the D* algorithm network.

3. A multi-AGV path planning method based on DDQN algorithm according to claim 2, characterized in that: The initial value of the exploration rate in the exploration rate selection method is set to 0.8; and the final value of the exploration rate in the exploration rate selection method is set to 1.

4. The multi-AGV path planning method based on the DDQN algorithm according to claim 3 is characterized in that: The D* algorithm network is improved using the AGV turning penalty coefficient to obtain an improved D* algorithm network.

5. The multi-AGV path planning method based on the DDQN algorithm according to claim 1 is characterized in that: In S1, the reward function is specifically: R＝r t +r c +r d Among them, R is the reward function value; r t Reward for reaching the target point; c Punishment for illegal behavior; d Reward for distance.

6. The multi-AGV path planning method based on the DDQN algorithm according to claim 1 is characterized in that: In S3, the TD error is calculated as follows: Among them, Q(s,a) is the current state Q value; is the target Q value; r is the current reward; γ is the discount factor; is the optimal Q value for the next state.

7. The multi-AGV path planning method based on the DDQN algorithm according to claim 2 is characterized in that: This multi-AGV path planning method also includes a target network synchronization step, specifically: The training step counting parameters are set and the synchronization timing is determined according to the training step counting parameters; at the synchronization timing, the parameters of the strategy network are copied to the target network, and the parameters of the target network are replaced.

8. The multi-AGV path planning method based on the DDQN algorithm according to claim 7 is characterized in that: The loss functions of the strategy network and the target network are specifically: Where N is the batch size; Q(s i ,a i ) is in the current state s i The Q value predicted by the model is i is the reward in the current state; is the optimal Q value for the next state.

9. The multi-AGV path planning method based on the DDQN algorithm according to claim 1 is characterized in that: In S3, the target Q value is calculated using the Bellman formula.

10. The multi-AGV path planning method based on the DDQN algorithm according to claim 1 is characterized in that: In S6, the maximum number of training rounds is 300 rounds.

Images