AGV path planning method and system based on ant colony-reinforcement learning

Abstract

The invention provides an AGV path planning method and system based on ant colony-reinforcement learning, and the method comprises the steps: carrying out the global path planning through employing an ant colony algorithm, carrying out the local path planning through employing a DDQN algorithm with a point in the global path planning as a target point, and training a neural network used for the actual path planning; in the DDQN algorithm, when target value network parameters are updated, a weighted double-Q network form is utilized to calculate a target value. The algorithm convergence speed, the network stability and the planned path quality can be improved by using the method.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic guided vehicle path planning, and in particular relates to an AGV path planning method and system based on an ant colony-reinforcement learning algorithm. Background Art

[0002] In the intelligent logistics process, warehousing occupies a core position, and the Automatic Guided Vehicle (AGV) is one of the key technologies of intelligent warehousing. Path planning technology is the core of the warehousing AGV scheduling process, and successful path planning is the primary prerequisite for AGV to complete tasks such as picking. With the development of artificial intelligence, deep reinforcement learning methods have achieved more and more significant results, and applying deep reinforcement learning methods to the field of path planning has become a current research hotspot.

[0003] However, as of now, the path planning method based on reinforcement learning still faces many challenges, including slow algorithm convergence, poor network stability in the later stage of training, and poor quality of the planned path. Summary of the invention

[0004] In view of this, the present invention provides an AGV path planning method based on an ant colony-reinforcement learning algorithm, which can improve the algorithm convergence speed, network stability and planned path quality.

[0005] In order to solve the above technical problems, the present invention is implemented as follows.

[0006] An AGV path planning method based on ant colony-reinforcement learning includes the following steps:

[0007] Step S1: using ant colony algorithm for global path planning;

[0008] Step S2: Taking the point in the global path planning as the target point, the DDQN algorithm is used for local path planning, and the neural network used for actual path planning is trained at the same time; the input of the neural network is the AGV state, and the Q value of the set N optional actions is output; the action corresponding to the maximum Q value is taken as the action that the AGV needs to perform, including angular velocity and linear velocity;

[0009] In the DDQN algorithm, when updating the target value network parameters, the target value is calculated using the weighted double Q network form, and the calculation formula is expressed as:

[0010] y IDQN =r'+γ[βQ(s',a * ;θ)+(1-β)Q(s′,a * θ - )] (1)

[0011] Among them, s and s' are the current AGV state and the AGV state at the next moment in the training sample respectively; y IDQN is the target value, which is the preferred action a when the input AGV state is s' * The ideal output of the corresponding current value network; r' is the immediate reward corresponding to state s', γ is the discount factor; Q(s',a * ; θ) represents the preferred action a among the N output Q values ​​of the network when the current value network θ in the DDQN algorithm is in the case of the input AGV state s' * The corresponding output Q value; Q(s′,a * θ - ) represents the target value network θ in the DDQN algorithm - When the input AGV state is s', the preferred action a in the network's N output Q values * The corresponding output Q value;

[0012] The weighting coefficient β is:

[0013]

[0014] Among them, c is the set hyperparameter, action a * and a L They are respectively the best action corresponding to the maximum Q value and the worst action corresponding to the minimum Q value after the AGV state s' is input into the current value network θ.

[0015] Preferably, c is set as a dynamic adaptive parameter, which is calculated as follows:

[0016] c=15e -x

[0017] Among them, x is the absolute value of the time difference error. The larger x is, the smaller c is;

[0018] x is calculated as follows:

[0019] x=|r+γQ(s′,a * θ - )-Q(s,a;θ)|

[0020] Among them, r and a are the immediate reward and action corresponding to the state s obtained from the training samples.

[0021] Preferably, the discrete action space includes an emergency left turn action, a left turn action, a straight-ahead action, a right turn action and an emergency right turn action.

[0022] Preferably, each action in the discrete action space is set as:

[0023] Emergency left turn: angular velocity is 1.5 rad / s, linear velocity is 0.15 m / s;

[0024] Left turn: angular velocity is 0.75 rad / s, linear velocity is 0.15 m / s;

[0025] Straight motion: angular velocity is 0rad / s, linear velocity is 0.15m / s;

[0026] Right turn: angular velocity is -0.75 rad / s, linear velocity is 0.15 m / s;

[0027] Emergency left turn: angular velocity is -1.5rad / s, linear velocity is 0.15m / s.

[0028] Preferably, taking a point in the global path planning as the target point comprises: selecting a turning point in the global path planning as the target point in the local path planning.

[0029] Preferably, step S2 comprises:

[0030] Step S21: simulate AGV movement and obtain the current AGV state s of the AGV t ;

[0031] Step S22: Select the current AGV status s t The corresponding execution action a t : Use the ∈-greedy strategy to select actions. If the currently generated random number is less than the set threshold ∈, then randomly select an optional action in the discrete action space as the execution action a t ; Otherwise, the current AGV state s t Input the current value network, and the network outputs the action corresponding to the maximum Q value among the N Q values ​​as the execution action a t ;∈ is reduced to 0.99 times of the previous round in each step until it is reduced to the set value;

[0032] Step 23: Execute action a t , get the immediate reward r, and the AGV transfers to the next moment AGV state s t+1 ; Set the current AGV status s t , perform action a t 、AGV state s at the next moment t+1 , immediate reward r, and end flag done are recorded in the training data pool;

[0033] Step S24: Network training: Extract a piece of training data from the training data pool, including AGV state s', action a', and immediate reward r'; Based on the AGV state s', use formula (1) to calculate the ideal output y of the target value network IDQN ;

[0034] At each step, the AGV state s' is used as the network input, the Q value corresponding to the action a' output by the current value network is used as the actual output, and the Q value corresponding to the action a' output by the current value network is used as the actual output. IDQN As the ideal output, update the parameters of the current value network;

[0035] Every set number of steps, the current value network parameter is assigned to the target value network parameter to update the target value network parameter;

[0036] Step S25: Repeat steps S21-S24 until the maximum number of iterations or the end flag indicates the end;

[0037] Step S26: Output the optimal current value network as the neural network for actual path planning.

[0038] Preferably, after the training of step S2 is completed, when selecting the neural network for actual path planning, the parameters of the current value network obtained by updating each step Step and the instant reward r' of the step are selected to form the neural network for actual path planning.

[0039] Preferably, the AGV status includes: radar scanning information, target point position, AGV own position, and heading angle.

[0040] Preferably, the instant reward is calculated as follows:

[0041] R e +R d +R y +R s

[0042] Among them, R e Indicates the reward value for reaching the target point or the penalty value for colliding with obstacles;

[0043] R d Represents the distance bonus value, which is calculated as follows:

[0044]

[0045] Where a is a constant, representing the weight of the reward; d current is the distance between the current robot and the target point, d initial is the distance between the initial position and the target point;

[0046] R y Indicates the yaw reward. The more consistent the movement direction is with the target point, the greater the reward value is. The calculation method is as follows:

[0047] R y = b·cosθ

[0048] Among them, b is a constant value, representing the weight of the reward; θ is the angle between the robot's motion direction and the line connecting the target point;

[0049] R s represents the step penalty, which is beneficial for shorter path lengths and is calculated as follows:

[0050] R s =c·0.5

[0051] Among them, c is a constant value, representing the penalty weight.

[0052] The present invention also provides an AGV path planning system based on ant colony-reinforcement learning, comprising: a global path planning unit and a local path planning unit;

[0053] The global path planning unit is used to perform global path planning using an ant colony algorithm;

[0054] The local path planning unit is used to use the point in the global path planning as the target point, use the DDQN algorithm to perform local path planning, and train the neural network used for actual path planning at the same time; the input of the neural network is the AGV state, and the Q value of the set N optional actions is output; the action corresponding to the maximum Q value is taken as the action that the AGV needs to perform, including angular velocity and linear velocity;

[0055] In the DDQN algorithm, when updating the target value network parameters, the target value is calculated using the weighted double Q network form, and the calculation formula is expressed as:

[0056] y IDQN =r+γ[βQ(s′,a * ;θ)+(1-β)Q(s′,a * θ - )] (1)

[0057] Among them, y IDQN is the target value, which is the preferred action a when the input AGV state is s' * The ideal output of the corresponding current value network; r is the immediate reward, γ is the discount factor; Q(s′,a * ; θ) represents the current value network θ in the DDQN algorithm, when the input AGV state is s', the action a in the network N output Q values * The corresponding output Q value; Q(s′,a * θ - ) represents the target value network θ in the DDQN algorithm - When the input AGV state is s', action a among the N output Q values ​​of the network * The corresponding output Q value;

[0058] The weighting coefficient β is:

[0059]

[0060] Among them, c is the set hyperparameter, action a * and a L They are respectively the best action corresponding to the maximum Q value and the worst action corresponding to the minimum Q value after the AGV state s' is input into the current value network θ.

[0061] Beneficial effects:

[0062] (1) The present invention provides an AGV path planning method based on an ant colony-reinforcement learning algorithm. First, the traditional reinforcement learning method is improved. The reinforcement learning algorithm is improved based on the traditional DQN algorithm, and a weighted double Q network is used for the target value calculation method. The improved DQN algorithm effectively overcomes the problem of Q value overestimation and improves the convergence speed and stability of the later network.

[0063] (2) The present invention further improves the neural network from the original fully connected layer traditional network to a competitive network structure.

[0064] (3) The present invention provides an AGV path planning method based on an ant colony-reinforcement learning algorithm. In view of the problem of poor algorithm convergence effect when facing a large map scene, the ant colony algorithm is combined with the improved DQN algorithm to further optimize the algorithm. First, the ant colony algorithm is used for global path planning to obtain an ideal optimal path, and then the key path points are extracted from it and the improved DQN algorithm is used for real-time local path planning. This improvement improves the convergence performance and path quality of the algorithm in large map scenes, and to a certain extent overcomes the limitations of the reinforcement learning method in the face of large map environments.

[0065] (4) The present invention provides an AGV path planning method based on an ant colony-reinforcement learning algorithm, which can be directly used for local path planning to achieve obstacle avoidance in static environments and dynamic environments. This advantage is one of the advantages of the reinforcement learning method over the traditional path planning method. In addition to the algorithm itself being able to achieve obstacle avoidance in dynamic environments, the decision-making actions of the neural network during the interaction between the AGV and the environment can be directly used as the speed control signal of the AGV, thereby eliminating the subsequent trajectory tracking step. This advantage reflects the superior practicality of the invention compared to the traditional path planning algorithm.

[0066] (5) The present invention provides an AGV path planning method based on an ant colony-reinforcement learning algorithm, in which a reward function for reinforcement learning is optimized and designed, taking into account a variety of factors such as reaching a target point, collision with an obstacle, distance to the target point, and heading angle relative to the target point. This design is conducive to improving the path quality and the obstacle avoidance performance of the AGV.

[0067] (6) The present invention provides an AGV path planning method based on an ant colony-reinforcement learning algorithm, wherein the target value calculation method in the reinforcement learning method is improved, and the parameter c is further adapted on the basis of the weighted DDQN method. This improvement is conducive to obtaining a more accurate target Q value estimate, and to a certain extent overcomes the disadvantage of having to adjust c multiple times in the face of different problems, and improves the generalization ability of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A flow chart of path planning based on the ant colony algorithm-reinforcement learning method provided by the present invention;

[0069] Figure 2 A schematic diagram of an unmanned warehouse scenario provided by the present invention;

[0070] Figure 3 A comparison chart of the reward values ​​of the traditional DQN algorithm and its derivative algorithm and the improved DQN algorithm of the present invention;

[0071] Figure 4 A rasterized schematic diagram of an unmanned warehouse scene provided by the present invention;

[0072] Figure 5 This is a schematic diagram of the path for the global path planning of the ant colony algorithm;

[0073] Figure 6 Comparison diagram of the paths planned by a single reinforcement learning method and an ant colony-reinforcement learning method;

[0074] Figure 7 The figure is a block diagram of the path planning system based on the ant colony algorithm-reinforcement learning method of the present invention.

[0075] Figure 8 This is the grid structure adopted by the present invention. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0077] For the task of path planning of AGV from the starting point to the end point in an unmanned warehouse, the present invention provides a path planning solution based on ant colony-reinforcement learning. After the map, starting point, and end point are initialized, the map is first rasterized and the global path planning is performed using the ant colony algorithm; then, the points in the global path planning are used as the target points, and the improved dual deep Q network (DDQN) algorithm is used for local path planning, and the neural network for actual path planning is trained at the same time. The input of the neural network is the AGV state, which may include, for example, radar scanning information, target point position, AGV's own position, and heading angle, that is, s = (s scan ,s goal ,s position , heading), output the Q value of the set N optional actions; take the action corresponding to the maximum Q value as the action that the AGV needs to perform, including angular velocity and linear velocity. After the network is trained, the trained neural network can be deployed to the actual vehicle to perform tasks.

[0078] Here, the improvement of the DDQN algorithm is mainly to adopt a weighted double Q network form for the target value calculation method. The improved DDQN algorithm effectively overcomes the problem of Q value overestimation and improves the convergence speed and stability of the later network. In a preferred solution, the neural network is also improved from the original fully connected layer traditional network to a competitive network structure, such as Figure 8 shown.

[0079] See also Figure 1 The path planning method based on ant colony-reinforcement learning of the present invention comprises the following steps:

[0080] S0: Perform scene initialization, mainly including map information initialization, start point and end point initialization.

[0081] Map information initialization mainly includes initialization of the AGV's activity space and obstacle positions. In order to better simulate the unmanned warehouse environment, this embodiment uses a rectangular closed range and a series of regularly and evenly arranged shelves as obstacles, such as Figure 2 shown.

[0082] The starting point and end point initialization refers to the initialization of the initial position and the target point position of the AGV.

[0083] S1: Use ant colony algorithm for global path planning;

[0084] It should be noted that the present invention requires specifying a starting point and an end point in advance before performing subsequent planning work. The global path planning based on the ant colony algorithm mainly includes the following steps:

[0085] S11: rasterize the map scene.

[0086] It should be noted that the original map scene is a gazebo scene modeled based on the real environment, which has realistic functions such as collision detection, kinematic simulation, and dynamic simulation. The ant colony algorithm is a planning algorithm based on a grid map, so it is necessary to rasterize the map in advance. The specific method is to cut the gazebo scene into small squares with a side length of 1m using horizontal and vertical lines, so that the position of each intersection can be represented by an integer coordinate pair, thus laying the foundation for the use of the ant colony algorithm. The rasterization result is as follows: Figure 3 shown.

[0087] S12: Initialize the ant colony algorithm.

[0088] The initialization of the ant colony algorithm mainly includes the initialization of algorithm parameters and taboo table. The algorithm parameters mainly refer to the ant population size, maximum number of iterations, pheromone concentration factor, heuristic information factor and pheromone volatility factor; the taboo table refers to the nodes that cannot be accessed under each node, including obstacle nodes and previous nodes.

[0089] S13: Perform node selection and state transfer.

[0090] It should be noted that the node selection adopts a non-deterministic strategy. The calculation formula for the probability of the kth ant going from the i-th node to the j-th node is:

[0091]

[0092] Among them, allowed k is the set of nodes that the kth ant is allowed to choose; π ij is the pheromone concentration left from the i-th node to the j-th node; α is the pheromone concentration factor, which represents the influence of pheromone concentration on node selection. The larger its value, the more ants tend to choose the path nodes that the previous ant population has walked through; η ij is the heuristic information, β is the heuristic information factor, which characterizes the influence of path length on node selection. The larger its value is, the more ants tend to choose nodes with shorter paths, which also means that the importance of pheromone concentration is weakened.

[0093] S14: Update the taboo table.

[0094] It should be noted that each step requires updating the taboo table of the current node. The specific principle is to remove the obstacle nodes and the previous step nodes in the eight-neighborhood, and the next state transfer is selected from the remaining nodes.

[0095] S15: Update pheromones.

[0096] After one iteration, the pheromone of the path needs to be updated in time. In order to avoid the heuristic information being obliterated due to excessive changes in pheromone concentration, incremental updates are adopted; combined with the volatility characteristics of pheromones, the pheromone update formula is shown as follows:

[0097] π ij (t+1)=(1-ρ)π ij (t)+Δπ ij (t)

[0098] Among them, ρ is the volatility factor of pheromone, which takes values ​​between (0,1); Δπ ij (t) is the increment of pheromone concentration. π ij (t) is the pheromone before updating, π ij (t+1) is the updated pheromone.

[0099] Δπ ij (t) The calculation formula is as follows:

[0100]

[0101] Among them, Q is the pheromone concentration constant, which is a fixed constant; Len(t) is the length of this iterative path; path is the set of path nodes.

[0102] S16: If the maximum number of iterations is reached, the planned path is output, otherwise, steps S13-S15 are performed.

[0103] In summary, after the above steps, an ideal optimal path from the starting point to the end point based on the grid map can be obtained.

[0104] S2: Local path planning based on the improved DQN algorithm.

[0105] This step uses the points in the global path planning as the target points and performs local path planning in sections. In practice, the turning points in the global path planning can also be selected as the target points in the local path planning. The local path planning at the turning points can reduce the amount of calculation. At the same time, since the turning points are more characteristic, detailed path planning for the turning points can maintain the quality of path planning while reducing the amount of calculation. Moreover, the turning point data is used as a new training sample to enrich the training data pool, which can further ensure the effectiveness of network training.

[0106] Figure 8It is the competitive network structure of the neural network adopted by the present invention. The AGV state s enters the input layer as the input, and then passes through two fully connected layers (dense). The dropout forgetting network layer is used to prevent overfitting. Then, the idea of ​​the competitive network is adopted to divide the Q value estimation into two parts: one part is the state advantage, which directly evaluates the advantage of the current state, which is a scalar; the other part is the action advantage, which evaluates the corresponding advantages after taking different actions, which is an N-dimensional vector. During processing, when the input is state s, the dropout output layer is programmed to output a 1-dimensional scalar, corresponding to the state advantage, and input to the fully connected layer dense3 for processing; then the dropout output layer is programmed to output an N-dimensional scalar, corresponding to the action advantage, and input to the fully connected layer dense4 for processing; the state advantage and the action advantage are then integrated through the output layer to obtain the Q value. This method makes the Q value estimation more accurate and improves the learning efficiency and stability of the algorithm.

[0107] Finally, the advantages of the two parts are integrated, and the calculation formula for the output Q value can be:

[0108]

[0109] Among them, V(s; θ) is the state advantage, which directly evaluates the advantage of the current state and is a scalar; A(s, a; θ) is the action advantage, which evaluates the corresponding advantages after taking different actions and is an N-dimensional vector. In order to ensure the uniqueness of the separated V and A, A(s, a; θ) is normalized. represents the mean of the N-dimensional components of the action advantage, and a is a set of N actions in an N-dimensional scalar. This method makes the Q value estimation more accurate and improves the learning efficiency and stability of the algorithm.

[0110] It should be noted that in this embodiment, the AGV adopts a discrete action space, and the optional actions in the discrete action space are angular velocity-linear velocity pairs. In a preferred embodiment, a total of N=5 actions are designed, including an emergency left turn action, a left turn action, a straight action, a right turn action, and an emergency right turn action. For example, the specific parameters of the five actions that can be designed are shown in the following table:

[0111]

[0112] The improved DDQN algorithm mainly includes the following steps:

[0113] S21: Algorithm parameter initialization, mainly including replay pool D, learning rate, discount factor, current value network parameter θ, target value network parameter θ - , maximum number of rounds, maximum number of steps initial state.

[0114] S22: Get the current AGV status s of the AGV by interacting with the environment t; Select the current AGV status t The corresponding execution action a t .

[0115] In this step, the ∈-greedy strategy is used to select action a t Generate a random number in the range of 0-1. If the current random number is less than the set threshold ∈, randomly select an optional action in the discrete action space as the execution action a t , for example, select a1 = emergency left turn; otherwise, change the current AGV state to s t Input the current value network in the double Q network, and the current value network outputs the action corresponding to the maximum Q value among the N Q values ​​as the execution action a t ;∈ is reduced to 0.99 times of the previous round in each step until it is reduced to the set value, such as 0.05. The specific calculation formula is as follows:

[0116]

[0117] S23: Execute actions and state transitions.

[0118] The "action" mentioned here refers to one of the five actions designed above, specifically the linear velocity and angular velocity control signals; after executing the action, the AGV will reach a new position according to the action and complete the state transfer.

[0119] Specifically, perform action a t , get the immediate reward r, AGV transfers to the next state s t+1 , and obtain the end flag value done. When the next moment state s t+1 When it is determined that the target point has been reached or a collision has occurred, done = 0, indicating that the iteration needs to be stopped; otherwise done = 1.

[0120] Set the current AGV status t , perform action a t 、AGV state s at the next moment t+1 , immediate reward r, and end mark done are recorded as a training data into the training data pool as a whole, thereby enriching the training data pool.

[0121] Here the calculation formula for the immediate reward r can be:

[0122] r=R e +R d +R y +R s

[0123] Among them, R e Indicates the reward value for reaching the target point or the penalty value for colliding with an obstacle. The calculation method is as follows:

[0124]

[0125] R d Represents the distance bonus value, which is calculated as follows:

[0126]

[0127] Where a is a constant, representing the weight of the reward; d current is the distance between the current robot and the target point, d initial is the distance between the initial position and the target point

[0128] R y Indicates the yaw reward. The more consistent the movement direction is with the target point, the greater the reward value is. The calculation method is as follows:

[0129] R y = b·cosθ

[0130] Among them, b is a constant, representing the weight of the reward; θ is the angle between the robot's movement direction and the line connecting the target point.

[0131] R s represents the step penalty, which is beneficial for shorter path lengths and is calculated as follows:

[0132] R s =c·0.5

[0133] Among them, c is a constant value, representing the penalty weight.

[0134] S24: Network training.

[0135] From the training data pool, extract a piece of training data, including AGV state s, action a, immediate reward r, next moment AGV state s', and end flag done.

[0136] As the number of steps goes by, the current value network also needs to be trained once every step when the AGV performs an action; the current value network parameters need to be assigned to the target value network at certain intervals, thereby completing the update of the target value network; preferably, in order to facilitate the selection of a better network for the final deployment of the actual vehicle, the network parameters (current value network) and the corresponding instant rewards are saved every certain number of steps, such as 10 steps, and stored in file format.

[0137] The specific steps of network training are as follows:

[0138] First, calculate the target value. The calculation formula is as follows:

[0139] y IDQN =r'+γ[βQ(s',a* ;θ)+(1-β)Q(s′,a * θ - )] (1)

[0140] Among them, y IDQN is the target value, which is the preferred action a when the input AGV state is s' * The ideal output of the corresponding current value network. r' is the immediate reward extracted from the data pool, γ is the discount factor, Q(s',a * ; θ) represents the current value of the network θ when the input state is s', and the action a among the N output Q values ​​of the network * The corresponding output Q value; Q(s′,a * θ - ) represents the target value network θ - When the input AGV state is s', action a among the N output Q values ​​of the network * The corresponding output Q value. * See below for definitions.

[0141] The calculation formula of β is:

[0142]

[0143] Among them, c is the set hyperparameter, action a * and a L They are the best action corresponding to the maximum Q value and the worst action corresponding to the minimum Q value after inputting the state s' into the current value network θ, which are defined as follows:

[0144] a * = argmax a′ Q(s′,a′;θ)

[0145] a L =argmin a′ Q(s′,a′;θ)

[0146] The above formula a * = argmax a′ The meaning of Q(s′,a′;θ) is that the state s′ is input into the current value network θ, which outputs N Q values, each of which corresponds to an action a′; find the maximum Q value, and its corresponding action is recorded as a * Similarly, formula a L =argmin a′ The meaning of Q(s′,a′;θ) is that the state s′ is input into the current value network θ, which outputs N Q values, each of which corresponds to an action a′; find the minimum Q value, and its corresponding action is recorded as a L .

[0147] Then refer to formula (2), input the state s′ into the target value network θ, which outputs N Q values, and selects a * The corresponding Q value is Q(s',a * θ - ); Input the state s′ into the target value network θ - , the network outputs N Q values, select a L The corresponding Q value is Q(s',a L θ - ), and then β can be calculated.

[0148] Then refer to formula (1) and input the state s′ into the current value network θ - , the network outputs N Q values, select a * The corresponding Q value is Q(s′,a * ;θ), combined with Q(s',a obtained above * θ - ), the target value y can be obtained by using formula (1) IDQN The target value y IDQN As the ideal output when updating the target value network parameters below.

[0149] Among them, the hyperparameter set by c is preferably a dynamic adaptive parameter calculated as follows:

[0150] c=15e -x

[0151] Among them, x is the absolute value of the time difference error. The larger x is, the smaller c is;

[0152] x is calculated as follows:

[0153] x=|r+γQ(s′,argmax a′ Q(s′,a′;θ);θ - )-Q(s,a;θ)|

[0154] Among them, r is the immediate reward corresponding to state s, argmax a′ Q(s′,a′;θ)=a*; s and a in Q(s,a;θ) are obtained from training samples.

[0155] When the Q value difference corresponding to different actions is large, it means that the Q value difference accumulated after a long period of learning is more reliable, and β is close to 1. At this time, the algorithm is equivalent to using a single network to find the target value; when the Q value difference corresponding to different actions is small, such as the Q value is all 0 in the initial stage of learning, β is close to If c is much larger than 1, then βClose to 0, the algorithm is equivalent to using the DDQN method to find the target value, which can avoid the error accumulation caused by overestimation of the Q value in the initial stage.

[0156] S25: Update current value network parameters.

[0157] In this step, each step uses the AGV state s' as the network input, the Q value corresponding to the action a' output by the current value network as the actual output, and y IDQN As the ideal output, update the parameters of the current value network.

[0158] The updating algorithm adopted in a preferred embodiment is the RMSprop algorithm.

[0159] S26: Update the target value network parameters every set number of steps.

[0160] In this step, the target value network parameters are updated if the current cumulative number of steps is, for example, a multiple of 20. The updating method is to copy the current value network parameters to the target value network.

[0161] S27: If the end flag value done is 0 and the maximum number of steps has not been reached, then S22-S26 are repeated. Otherwise, step S28 is executed.

[0162] S28: When the maximum number of iterations is reached, the process ends and the optimal current value network is output.

[0163] In this step, in order to further avoid iterative oscillation, in the previous step, the network parameters of the current value network and the corresponding instant reward are saved every certain number of steps, such as 10 steps. In this step, the network parameters corresponding to the maximum reward value are selected according to the reward value curve, which is the optimal network.

[0164] The planning result of the embodiment executed according to the above steps is as follows Figure 4-Figure 6 shown.

[0165] If a real vehicle needs to be deployed, the optimal network in the saved neural network should be selected according to the reward value curve. After the selection, the input of the network is the current state, and the output is the five Q values ​​corresponding to the five actions. The action corresponding to the maximum Q value should be selected for the AGV to execute.

[0166] Based on the above method, the present invention also provides an AGV path planning system based on ant colony-reinforcement learning, such as Figure 7 As shown, it includes a global path planning unit and a local path planning unit.

[0167] A global path planning unit is used to perform global path planning in a grid map using an ant colony algorithm;

[0168] The local path planning unit is used to use the points in the global path planning as the target points, adopt the DDQN algorithm to perform local path planning, and train the neural network used for actual path planning at the same time; the input of the neural network is the AGV state, and the output is the Q value of the set N optional actions; the action corresponding to the maximum Q value is taken as the action that the AGV needs to perform, including angular velocity and linear velocity.

[0169] In the DDQN algorithm, when updating the target value network parameters, the target value is calculated using the weighted double Q network form. The calculation formula is shown in formula (1)(2).

[0170] In summary, the present invention has the following advantages over the prior art:

[0171] 1. High authenticity of map scenes: The enhanced method is based on the gazebo map for modeling rather than the common grid map, so that any and real design scenes can be directly carried out in gazebo according to the real environment and kinematic simulation can be performed, making the method more practical than traditional algorithms.

[0172] 2. Strong generalization ability of the method: Since the training process of the reinforcement learning method is to let the AGV continuously explore and try and update the network, the environment is a static obstacle environment or a dynamic obstacle environment. The application of the method has no effect - that is, the method can achieve obstacle avoidance in both static and dynamic environments. In addition, this method can directly output the speed value required by the AGV without the need for an additional trajectory tracking algorithm.

[0173] 3. Ant Colony-Reinforcement Learning Algorithm: In addition to combining the ant colony algorithm, this method also makes corresponding improvements to the reinforcement learning method itself. Based on the DQN algorithm, the target value calculation method is improved to the weighted DDQN method, and the network structure is improved to the competitive network form. These improvements have overcome the various limitations of the reinforcement learning method in the field of path planning to a certain extent, and improved the convergence speed of the algorithm, the stability of the network in the later stage, and the quality of the planned path.

[0174] The above specific embodiments only describe the design principle of the present invention. The shapes and names of the components in the description may be different and are not limited. Therefore, those skilled in the art in the field of the present invention may modify or replace the technical solutions recorded in the above embodiments; and these modifications and replacements do not deviate from the creative purpose and technical solutions of the present invention and should all fall within the protection scope of the present invention.

Claims

1. An AGV path planning method based on ant colony-reinforcement learning, characterized in that: The following steps are involved: Step S1: using ant colony algorithm for global path planning; Step S2: Taking the point in the global path planning as the target point, the DDQN algorithm is used for local path planning, and the neural network used for actual path planning is trained at the same time; the input of the neural network is the AGV state, and the Q value of the set N optional actions is output; the action corresponding to the maximum Q value is taken as the action that the AGV needs to perform, including angular velocity and linear velocity; In the DDQN algorithm, when updating the target value network parameters, the target value is calculated using the weighted double Q network form, and the calculation formula is expressed as: and IDQN =r'+γ[βQ(s′,a * ;θ)+(1-β)Q(s′,a * ;θ - )] (1) Among them, s and s' are the current AGV state and the AGV state at the next moment in the training sample respectively; y IDQN is the target value, which is the preferred action a when the input AGV state is s' * The ideal output of the corresponding current value network; r' is the immediate reward corresponding to state s', γ is the discount factor; Q(s',a * ; θ) represents the preferred action a among the N output Q values ​​of the network when the current value network θ in the DDQN algorithm is in the case of the input AGV state s' * The corresponding output Q value; Q(s′,a * θ - ) represents the target value network θ in the DDQN algorithm - When the input AGV state is s', the preferred action a in the network's N output Q values * The corresponding output Q value; The weighting coefficient β is: Among them, c is the set hyperparameter, action a * and a L They are respectively the best action corresponding to the maximum Q value and the worst action corresponding to the minimum Q value after the AGV state s' is input into the current value network θ.

2. The method according to claim 1, characterized in that c is set as a dynamic adaptive parameter, and is calculated as follows: c=15e -x Among them, x is the absolute value of the time difference error. The larger x is, the smaller c is; x is calculated as follows: x=|r+γQ(s′,a * ;θ - )-Q(s,a;θ)| Among them, r and a are the immediate reward and action corresponding to the state s obtained from the training samples.

3. The method according to claim 1, characterized in that The discrete action space includes an emergency left turn action, a left turn action, a straight-ahead action, a right turn action, and an emergency right turn action.

4. The method according to claim 3, characterized in that Each action in the discrete action space is set as: Emergency left turn: angular velocity is 1.5 rad / s, linear velocity is 0.15 m / s; Left turn: angular velocity is 0.75 rad / s, linear velocity is 0.15 m / s; Straight motion: angular velocity is 0rad / s, linear velocity is 0.15m / s; Right turn: angular velocity is -0.75 rad / s, linear velocity is 0.15 m / s; Emergency left turn: angular velocity is -1.5rad / s, linear velocity is 0.15m / s.

5. The AGV path planning method based on ant colony-reinforcement learning as claimed in claim 1, characterized in that: The taking a point in the global path planning as the target point is: selecting a turning point in the global path planning as the target point in the local path planning.

6. The method according to claim 1, characterized in that The step S2 comprises: Step S21: simulate AGV movement and obtain the current AGV state s of the AGV t ; Step S22: Select the current AGV status s t The corresponding execution action a t :use ∈ -greedy strategy selects actions. If the currently generated random number is less than the set threshold ∈, then a selectable action in the discrete action space is randomly selected as the execution action a t ; Otherwise, the current AGV state s t Input the current value network, and the network outputs the action corresponding to the maximum Q value among the N Q values ​​as the execution action a t ; ∈ At each step, the value is reduced to 0.99 times of the previous round until it reaches the set value; Step 23: Execute action a t , get the immediate reward r, and the AGV transfers to the next moment AGV state s t+1 ; Set the current AGV status s t , perform action a t 、AGV state s at the next moment t+1 , immediate reward r, and end flag done are recorded in the training data pool; Step S24: Network training: Extract a piece of training data from the training data pool, including AGV state s', action a', and immediate reward r'; Based on the AGV state s', use formula (1) to calculate the ideal output y of the target value network IDQN ; At each step, the AGV state s' is used as the network input, the Q value corresponding to the action a' output by the current value network is used as the actual output, and the Q value corresponding to the action a' output by the current value network is used as the actual output. IDQN As the ideal output, update the parameters of the current value network; Every set number of steps, the current value network parameter is assigned to the target value network parameter to update the target value network parameter; Step S25: Repeat steps S21-S24 until the maximum number of iterations or the end flag indicates the end; Step S26: Output the optimal current value network as the neural network for actual path planning.

7. The method according to claim 1 or 6, characterized in that After the training in step S2 is completed, when selecting the neural network for actual path planning, the parameters of the current value network obtained by updating each step Step and the instant reward r' of the step are selected, and the parameters of the current value network with the largest instant reward value are selected to form the neural network for actual path planning.

8. The method according to claim 5, characterized in that The AGV status includes: radar scanning information, target point position, AGV own position, and AGV heading angle.

9. The method according to any one of claims 1 to 8, characterized in that: The instant reward is calculated as follows: R e +R d +R y +R s Among them, R e Indicates the reward value for reaching the target point or the penalty value for colliding with obstacles; R d Represents the distance bonus value, which is calculated as follows: Where a is a constant, representing the weight of the reward; d current is the distance between the current robot and the target point, d initial is the distance between the initial position and the target point; R y Indicates the yaw reward. The more consistent the movement direction is with the target point, the greater the reward value is. The calculation method is as follows: R y =b·cosθ Among them, b is a constant value, representing the weight of the reward; θ is the angle between the robot's motion direction and the line connecting the target point; R s represents the step penalty, which is beneficial for shorter path lengths and is calculated as follows: R s =c·0.5 Among them, c is a constant value, representing the penalty weight.

10. An AGV path planning system based on ant colony-reinforcement learning, characterized in that: include: A global path planning unit and a local path planning unit; The global path planning unit is used to perform global path planning using an ant colony algorithm; The local path planning unit is used to use the point in the global path planning as the target point, use the DDQN algorithm to perform local path planning, and train the neural network used for actual path planning at the same time; the input of the neural network is the AGV state, and the Q value of the set N optional actions is output; the action corresponding to the maximum Q value is taken as the action that the AGV needs to perform, including angular velocity and linear velocity; In the DDQN algorithm, when updating the target value network parameters, the target value is calculated using the weighted double Q network form, and the calculation formula is expressed as: and IDQN =r+γ[βQ(s′,a * ;θ)+(1-β)Q(s′,a * ;θ - )] (1) Among them, y IDQN is the target value, which is the preferred action a when the input AGV state is s' * The ideal output of the corresponding current value network; r is the immediate reward, γ is the discount factor; Q(s′,a * ; θ) represents the current value network θ in the DDQN algorithm, when the input AGV state is s', the action a in the network N output Q values * The corresponding output Q value; Q(s′,a * θ - ) represents the target value network θ in the DDQN algorithm - When the input AGV state is s', action a among the N output Q values ​​of the network * The corresponding output Q value; The weighting coefficient β is: Among them, c is the set hyperparameter, action a * and a L They are respectively the best action corresponding to the maximum Q value and the worst action corresponding to the minimum Q value after the AGV state s' is input into the current value network θ.

Images