Automatic warehouse multi-AGV multi-cycle path planning method based on QMIX algorithm

Abstract

The invention relates to an automatic warehouse multi-AGV multi-cycle path planning method based on a QMIX algorithm, and the method comprises the steps: building a warehouse node graph according to the layout of a warehouse, obtaining part of observation information of AGVs, and determining the initial starting point, terminal point and obstacle information of the AGVs; if the trained path planning model exists at present, inputting the initial starting point, the terminal point and the obstacle information of the AGV into the path planning model in combination with the previous action of the AGV and the ID information of the AGV, and outputting to obtain a driving path corresponding to the AGV; otherwise, constructing a reinforcement learning environment, taking local observation information of the AGVs as input, converting the local observation information into an observation matrix, inputting the observation matrix into a deep neural network, outputting a Q value of each AGV, receiving global information through a parameter network to generate parameters, and finally reading the Q value of each AGV and the parameters of the parameter network by a reasoning network to output an overall Q value. Compared with the prior art, multi-cycle path planning is carried out on multiple AGVs, and the working efficiency and accuracy of the multiple AGVs in a complex environment can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of AGV path planning, and in particular to a multi-AGV multi-cycle path planning method for an automated warehouse based on a QMIX algorithm. Background Art

[0002] In recent years, with the rapid development of e-commerce platforms and changes in consumer habits, the number of online shopping users has continued to rise, driving the rapid growth of the logistics industry. These online-purchased goods usually need to be shipped from warehouses, which have become key commodity transfer stations. However, traditional warehouse management methods, such as workers manually handling or operating mechanical equipment to move large goods, can no longer meet the current growing logistics needs. These traditional methods are not only complex to manage and inefficient, but also have high labor costs. Therefore, finding more efficient and economical warehouse management methods has become an urgent need in the industry.

[0003] In order to cope with the many challenges in warehouse management, intelligent material handling equipment technology has emerged, especially the automated guided vehicle (AGV), which plays an important role in this. AGV uses advanced automation equipment, the Internet of Things, artificial intelligence and big data analysis to realize the automation and intelligent operation of material handling. These intelligent vehicles can automatically complete the handling, loading and unloading and stacking of goods without human intervention.

[0004] AGV not only significantly improves warehouse operation efficiency and order accuracy, reduces labor costs, but also promotes scientific management and optimal configuration of warehouses. Through precise path planning and flexible scheduling systems, AGV can efficiently carry out material handling in warehouses, greatly reducing errors and delays caused by manual operations. The application of AGV technology not only conforms to the development trend of modern logistics, but also adapts to the needs of future intelligent logistics, providing enterprises with stronger competitiveness and sustainable development capabilities. The widespread application of AGV also makes the logistics system more flexible and responsive, which can significantly improve the overall efficiency of the logistics industry.

[0005] In order to ensure the reliable operation of AGV, it is necessary to accurately plan the path of AGV. Path planning is a trajectory design strategy for connecting the starting point and the target point. In the prior art, common path planning algorithms include traditional path planning algorithms, heuristic path planning algorithms, sampling-based path planning algorithms, and intelligent bionics algorithms. However, these AGV path planning algorithms cannot deal with the problem of multi-AGV conflicts well, and have low applicability to unknown and complex environments, resulting in the need for high human participation. As a result, their work and operation efficiency are low, and they are time-consuming and cannot solve large-scale examples. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm, which can perform multi-cycle path planning for multiple AGVs and improve the working efficiency and accuracy of multiple AGVs in complex environments.

[0007] The purpose of the present invention can be achieved by the following technical solution: A multi-AGV multi-cycle path planning method for an automated warehouse based on a QMIX algorithm, comprising the following steps:

[0008] S1. According to the warehouse layout, establish a warehouse node diagram, obtain some observation information of AGV, and determine the initial starting point, end point and obstacle information of AGV;

[0009] S2, check whether there is a trained path planning model. If so, input the AGV's initial starting point, end point and obstacle information, combined with the AGV's previous action and the AGV's ID information, into the path planning model, and output the AGV's corresponding driving path; otherwise, execute step S3;

[0010] S3, build a reinforcement learning environment, input part of the observation information of each AGV into the RNN (Recurrent Neural Network) neural network of the QMIX algorithm, and determine the predicted Q value of each AGV through the ε-greedy strategy;

[0011] The predicted Q value is input into the Mixing Network, and the predicted Q value is evaluated to obtain the overall Q value;

[0012] S4, according to the overall Q value and the reward value of the AGV, guide the AGV to make the next action selection, save the observation information and optimal path of each AGV in the hidden layer, save the overall Q value and the real reward value of the environment, and update the obstacle information observed by the AGV;

[0013] Through loop iteration, a single AGV output can obtain a higher predicted Q value, making the overall Q value closer to the actual reward value of the environment, and constructing a path planning model, and then returning to step S2.

[0014] Furthermore, the partial observation information of the AGV in step S1 is specifically:

[0015] <I,S,A,T,R,O,Ω,γ>

[0016] A={A1,A2,A3,...,A n}

[0017] T(s,a,s′)=P(s′|s,a)

[0018]

[0019] Where I represents the number of agents; S represents the state space, which contains the agent and environment information; A represents the joint action space, where A i represents the local action a that agent i can take i T is the state transition function, which represents the probability function of the agent transferring to state s′ after taking joint action a in state s; R is the reward function, which is used to represent the reward obtained by the agent when state transition occurs; O is the observation space, which represents the set of joint observations, o t is the observation set at time t, Ω is the observation function; γ∈[0,1] is the discount factor.

[0020] Furthermore, the state space is specifically: S = (S 小车 , S 目的地 , S 障碍物 ), where the state of the AGV, i.e., the position coordinates, is given by It is represented by n position coordinate tuples, where n is the number of AGVs;

[0021] The state of the AGV target, i.e. the target position coordinates, is given by Represents, including n position coordinate tuples, S 小车 and S 目的地 Through one-to-one correspondence in sequence, the AGV is guaranteed to correspond to the target point one-to-one;

[0022] The state of the obstacle, i.e. the obstacle position coordinates, is given by It indicates that it contains the position coordinates of all obstacles and consists of m position coordinate tuples, where m is the number of obstacles;

[0023] All AGVs are at their initial positions at time 0. When a coordinate position is occupied by an AGV or an obstacle, other AGVs will not be able to reach that node, that is, the AGV coordinate position and the obstacle coordinate position cannot coincide. When the AGV reaches its target position, the next target position will be generated for it and stored in the target state S. 障碍物 middle.

[0024] Furthermore, the observation space is specifically: O=(O 障碍物 , O 小车 , O 目的地 ), decompose it into three matrices, namely the obstacle matrix, other AGV matrices and target projection matrix. The obstacle matrix O 障碍物Contains obstacle information within the observable distance of the AGV. When there is an obstacle within the observation range, this part of the matrix will be set to 1, otherwise it will be 0;

[0025] AGV Matrix O 小车 Contains information about other AGVs within the observable range of the AGV. When there are other AGVs within the observable range, this part of the matrix will be set to 1, otherwise it will be set to 0;

[0026] The target matrix O 目的地 Contains the location information of the AGV target point. When the target point is not within the observation range of the current AGV, it is the projection of a vector from the AGV to the target point in the matrix, and will be displayed at the edge of the matrix, that is, the edge position is 1 and other positions are 0; when the target point enters the observation space of the AGV, the target point is set to 1 and the others are 0.

[0027] Furthermore, the path planning model in step S2 includes multiple RNN neural networks, the multiple RNN neural networks are respectively connected to the hybrid network, the multiple RNN neural networks correspond to different AGVs, each RNN neural network includes an input layer MLP (Multilayer Perceptron), an intermediate layer GRU (Gated Recurrent Unit) and an output layer MLP, the input of the RNN neural network is the observation space of each AGV, the action code of the AGV and the ID of the AGV, and the output is the Q value of the AGV;

[0028] The hybrid network includes an inference network and a parameter network. The input of the inference network is the Q value of all AGVs, and the output is the global Q value Q tot ,The weights and bias parameters in the inference network are generated by the parameter network;

[0029] The input of the parameter network is the global state s t , the output is the weight W and bias B of the inference network;

[0030] The offset of the last layer of the hybrid network is obtained through a two-layer network and a RELU activation function to obtain a nonlinear mapping network. The inference network directly calculates the global Q value Q after receiving the parameters of the parameter network. tot .

[0031] Furthermore, the working process of the RNN neural network in step S3 includes: inputting the current observation value of each AGV, the action of the previous step, and the one-hot encoding of the ID of each AGV represented in vector form to the input layer MLP, receiving the input of the first layer and hidden historical information through the middle layer GRU, and outputting the predicted Q value through the output layer MLP.

[0032] Furthermore, the calculation formula of the ε-greedy strategy in step S3 is:

[0033]

[0034] Among them, c+1 represents the total number of actions in the AGV action space, a * represents the action that maximizes the Q value, π(a|s) represents the action of the AGV in a certain state in the π strategy, that is, there is a probability of ε to randomly select an action from c+1 actions, including the action that maximizes the Q value, and there is a probability of 1-ε to directly select the action that maximizes the Q value. ε decreases as the training progresses, thereby ensuring that the action selection of the AGV is more exploratory in the early stage of training and more stable in the later stage.

[0035] Furthermore, the reward value of the AGV in step S4 is calculated by the reward function:

[0036] r(s,a)=w t r t (s,a)-w p r p (s,a)-w c r c (s,a)

[0037]

[0038] Among them, r t is the reward value obtained by AGV when it reaches the target point, r p (s,a) represents the movement penalty. When the AGV moves, it will obtain a certain penalty value, thereby encouraging the AGV to reach the target point with as few moves as possible. c (s, a) is the conflict penalty. When a conflict occurs, a negative reward value will be given to make the AGV avoid the conflict as much as possible. When the AGV reaches the target, it will get a reward value of 1, that is, r t (s,a)=1, and the reward value is 0 in other cases. When a conflict occurs, the AGV will not be able to pass the target point and will receive a conflict penalty r. c (s, a).

[0039] Furthermore, the conflicts include node conflicts, follow-up conflicts, transposition conflicts and cycle conflicts.

[0040] Furthermore, the node conflict specifically refers to: the AGV plans to occupy the same node at the same time step;

[0041] The following conflict specifically refers to: an AGV plans to occupy a node that was occupied by another AGV in the previous time step;

[0042] The transposition conflict specifically refers to: two AGVs plan to exchange positions in a single time step;

[0043] The cycle conflict specifically means that each AGV moves to a vertex previously occupied by another AGV, forming a "rotation cycle" pattern, also known as a deadlock.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] The present invention combines reinforcement learning technology, takes the observation space (obstacles, other AGVs, end point) observed by AGV and the previous action, and the ID of AGV as the input of the corresponding RNN neural network, predicts the next action through the ε-greedy strategy, and evaluates the predicted action through the Mixing Network, so as to obtain the optimal next action at the moment. That is, the local observation information of AGV is introduced as input to convert it into an observation matrix, the observation matrix is ​​input into the RNN neural network, the value function Q value of each AGV is output, and then the parameter network receives global information to generate parameters, and finally the inference network reads the Q value of each AGV and the parameters of the parameter network to output the overall Q value, so as to achieve the goal of centralized training and distributed execution. This method solves the problems of multiple AGV conflicts and deadlocks on the path. At the same time, the randomness and blindness of AGV can be reduced through reasonable reward function settings, so that AGV can choose efficient actions, which can effectively solve the problem that the existing path planning algorithm is time-consuming and cannot solve large-scale examples.

[0046] The present invention uses local observation to limit the observation space of each AGV, and defines the observation space as O=(O 障碍物 , O 小车 , O 目的地 ), that is, each AGV can only observe the state information within a matrix around itself, which is decomposed into three matrices, namely the obstacle matrix, the other AGV matrix and the target projection matrix. The three matrices together constitute the observation space of each AGV at time t The observation space of each AGV By converting the matrix into a vector as an input of the corresponding RNN neural network, the RNN network obtained through subsequent training can have better robustness and transferability, so that it can adapt to different complex environments.

[0047] In order to encourage movement, the present invention is designed not to set a penalty value for movement. When the AGV reaches the target, it will receive a reward value of 1, and in other cases it will receive a reward value of 0. When a conflict occurs, the AGV will not be able to pass the target point, thereby encouraging it to complete more tasks within the specified time and improve throughput. Since it corresponds to whether each AGV successfully reaches the target point, it is beneficial to quickly and accurately calculate evaluation indicators such as the overall success rate, and is more conducive to the statistics of the number of tasks completed in multiple cycles.

[0048] The present invention puts multiple AGVs into the list of the same tuple. When an AGV completes a cycle, it will enter the next cycle to complete the next task, thereby avoiding the AGV being set to stop or disappear after completing a cycle in the warehouse. In contrast, the traditional heuristic algorithm is mainly for a single cycle, which is inconsistent with the actual environment. In addition, the present invention introduces an anti-collision mechanism in the QMIX algorithm, thereby reducing or avoiding conflicts between two or more AGVs, effectively improving the working efficiency of multiple AGVs in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0050] Figure 2 It is a schematic diagram of the application process of the embodiment;

[0051] Figure 3 This is a schematic diagram of an automated warehouse in an embodiment;

[0052] Figure 4 This is a schematic diagram of a single AGV multi-cycle application in an embodiment;

[0053] Figure 5 : is the observation space diagram of AGV in the embodiment;

[0054] Figure 6 It is a structural diagram of the QMIX algorithm in the embodiment;

[0055] Figure 7 Figure 1 shows several types of collisions of AGV in the embodiment;

[0056] Figure 8 It is the operation diagram of AGV on the small map in the embodiment;

[0057] Fig. 9 Schematic diagram of the throughput of AGV in other maps in the embodiment. DETAILED DESCRIPTION

[0058] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Example

[0060] like Figure 1 As shown, a multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm includes the following steps:

[0061] S1. According to the warehouse layout, establish a warehouse node diagram, obtain some observation information of AGV, and determine the initial starting point, end point and obstacle information of AGV;

[0062] S2, check whether there is a trained path planning model. If so, input the AGV's initial starting point, end point and obstacle information, combined with the AGV's previous action and the AGV's ID information, into the path planning model, and output the AGV's corresponding driving path; otherwise, execute step S3;

[0063] S3, build a reinforcement learning environment, input part of the observation information of each AGV into the RNN neural network of the QMIX algorithm, and determine the predicted Q value of each AGV through the ε-greedy strategy;

[0064] The predicted Q value is input into the Mixing Network, and the predicted Q value is evaluated to obtain the overall Q value;

[0065] S4, according to the overall Q value and the reward value of the AGV, guide the AGV to make the next action selection, save the observation information and optimal path of each AGV in the hidden layer, save the overall Q value and the real reward value of the environment, and update the obstacle information observed by the AGV;

[0066] Through loop iteration, a single AGV output can obtain a higher predicted Q value, making the overall Q value closer to the actual reward value of the environment, and constructing a path planning model, and then returning to step S2.

[0067] This embodiment applies the above solution, such as Figure 2 As shown, the main ones are:

[0068] Step 1: Build a node diagram based on the actual layout of the automated warehouse area. The layout diagram is as follows: Figure 3 ) to determine the starting and ending points of the AGV loading and unloading tasks. The automated warehouse environment includes AGVs, shelves, goods, and obstacles in the warehouse. In practical applications, the map can be any regular or irregular shape, and the node distance in the map is the length of the AGV per unit time. The constructed map includes equipment such as AGVs and warehouse obstacles in the warehouse.

[0069] Step 2: The information observed by the randomly running AGV (other AGVs, obstacles, end points), the code of each AGV, and the previous action are used to predict its next action through the ε-greedy strategy in the RNN ring neural network, and the predicted action is evaluated through the mixing network to obtain the optimal next action at the moment.

[0070] Step 3: The AGV runs randomly in the environment, saves everything it gets (environment, real reward, action, observation information), and uses this information as input to return to the previous step for a loop, thereby more accurately guiding the next action selection. In this embodiment, the loop is performed about 200 times.

[0071] Step 4: Input the parameters after multiple cycles into the AGV, so that the single AGV output can obtain a higher predicted Q value, making the overall Q value closer to the real reward value of the environment. Similarly, use the obtained data as input and return to step 2 for looping. This loop process is about 2 million times. Through loop iterations, ε becomes smaller and smaller, making the probability of 1-ε larger and larger, so that the guided action will become more and more accurate.

[0072] Step 5: Save the above actions as input and loop back to the first layer, so that the path database has a trained model, which can be quickly called to allow a single AGV to complete the task without stopping or disappearing, and continue to make the AGV complete other tasks (such as Figure 4 shown).

[0073] In step 2, the partial observable of the AGV is represented by an 8-tuple<I,S,A,T,R,O,Ω,γ> Where I represents the number of agents; S represents the state space, which contains the agent and environment information; A = {A1, A2, A3, ..., A n} represents the joint action space, where A i represents the local action a that agent i can take i ; T is the state transition function, T(s,a,s′)=P(s′|s,a) represents the probability function of the agent transferring to state s′ after taking joint action a in state s; R is the reward function, which is used to represent the reward obtained by the agent when state transition occurs; O is the observation space, which represents the set of joint observations. The observation set at time t is is the observation function; γ∈[0,1] is the discount factor.

[0074] 1. Input the observation information into the RNN neural network of the QMIX algorithm, that is, input the current observation value of each AGV, the action of the previous step, and the one-hot encoding of the ID of each AGV represented in vector form into the MLP (input layer), receive the input of the first layer and the hidden historical information through the GRU (middle layer), and then output the predicted Q value through the MLP (output layer). Among them, the ε-greedy strategy formula is as follows:

[0075]

[0076] Where c+1 represents the total number of actions in the action space of AGV i, and a * represents the action that maximizes the Q value, and π(a|s) represents the action of AGV i in a certain state in the π strategy. In the present invention, this formula indicates that there is a probability of ε to randomly select an action (including the action that maximizes the Q value) from c+1 actions, and there is a probability of 1-ε to directly select the action that maximizes the Q value. In the formula, ε will continue to decrease as the training progresses, thereby ensuring that the action selection of AGV is more exploratory in the early stage of training, and the action selection of AGV is more stable in the later stage.

[0077] This embodiment describes in detail the state space, observation space, reward function, state transfer function and network structure of multi-AGV multi-cycle path planning in combination with the characteristics of automated warehouses.

[0078] (1) State space design

[0079] This scheme defines the state S as S = (S 小车 ,S 目的地 ,S 障碍物 ), where the state of the AGV, i.e., the position coordinates, is given by S 小车 It is represented by n position coordinate tuples (n is the number of AGVs), that is,

[0080] The state of the AGV target, i.e. the target position coordinates, is represented by S 障碍物 Indicates that it also includes n position coordinate tuples S 小车 and S 目的地 Through one-to-one correspondence in sequence, it is ensured that the AGV corresponds to the target point one-to-one.

[0081] Finally, the obstacle state, that is, the obstacle position coordinates are given by S 障碍物 It contains the position coordinates of all obstacles, which is also composed of m position coordinate tuples (m is the number of obstacles), that is,

[0082] All AGVs are at their initial positions at time 0. When a coordinate position is occupied by an AGV or an obstacle, other AGVs will not be able to reach that node, that is, the AGV coordinate position and the obstacle coordinate position cannot coincide. When the AGV reaches its target position, the next target position will be generated for it and stored in the target state S 障碍物 This solution is a partially observable problem. Each AGV can only observe part of the information in its vicinity. Therefore, the state space is not directly used as the input of the AGV neural network, but is placed in the input of the Mixing Network to guide the training of the AGV.

[0083] (2) Observation space design

[0084] In order to make the model and algorithm more robust and transferable, this solution uses local observation to limit the observation space of each AGV, that is, each AGV can only observe the state information within a matrix around itself. This solution defines the observation space as O = (O 障碍物 , O 小车 , O 目的地 ), decompose it into three matrices, namely the obstacle matrix, the other AGV matrix and the target projection matrix.

[0085] The first matrix is ​​the obstacle matrix O 障碍物 , contains the obstacle information within the observable distance of the AGV. When there is an obstacle within the observation range, this part of the matrix will be set to 1, otherwise it will be 0.

[0086] The second matrix is ​​the AGV matrix O 上车 , contains the information of other AGVs within the observable range of the AGV. When there are other AGVs within the observation range, this part of the matrix will be set to 1, otherwise it will be 0.

[0087] The third is the target matrix O 目的地 , contains the location information of the AGV target point. When the target point is not within the observation range of the current AGV, it is the projection of a vector from the AGV to the target point in the matrix, and will be displayed at the edge of the matrix, that is, the edge position is 1, and other positions are 0; when the target point enters the observation space of the AGV, the target point is set to 1, and the others are 0.

[0088] The above three matrices together constitute the observation space of each AGV at time t The observation space of each AGV The matrix is ​​converted into a vector as the input of the neural network, such as Figure 5 shown.

[0089] (3) Reward function design

[0090] In order to encourage movement, this solution will not set a penalty value for movement. When the AGV reaches the target, it will receive a reward value of 1, and in other cases it will receive a reward value of 0. When a conflict occurs, the AGV will not be able to pass the target point. This is to encourage it to complete more tasks within the specified time, thereby improving throughput. This reward setting is easy to explain because it corresponds to whether each AGV successfully reaches the target point. It is easy to calculate evaluation indicators such as the overall success rate, and it is more conducive to the statistics of the number of tasks completed in multiple cycles.

[0091] Table 1 Reward function

[0092]

[0093]

[0094] In this scheme, the reward function is:

[0095] (s,a)=w t r t (s,a)-w p r p (s,a)-w c r c (s,a)

[0096] where r t is the reward value obtained by AGV when it reaches the target point, rp(s,a) represents the movement penalty. When the AGV moves, it will obtain a certain penalty value, thereby encouraging the AGV to reach the target point with as few moves as possible. c (s,a) is the conflict penalty. When a conflict occurs, a negative reward value will be given to make the AGV avoid the conflict as much as possible. When the AGV reaches the target, it will get a reward value of 1, that is, r t (s,a)=1. In all other cases, the reward value is 0. When a conflict occurs, the AGV will not be able to pass the target point and will receive a conflict penalty r. c (s, a). In order to make the AGV reach the target point quickly, a movement penalty can be set for the AGV movement. For each time step that does not reach the target point, the AGV will receive a movement penalty r for moving or stopping to wait. p (s, a), you can customize your own reward function by defining the reward (penalty) value and weight coefficient.

[0097] Therefore, the penalty mechanism of the reward function is used to reduce the reward value obtained by the AGV, thereby reducing the conflict between AGVs and achieving collision avoidance between AGVs.

[0098] (4) State transfer function design

[0099] At each decision time point, each AGV will select an action to execute in its own action space according to its own observation space, thus causing state transfer. Since the state space of this scheme is divided into three parts, namely S = (S 小车 , S 目的地 , S 障碍物 ), the state transfer function is divided into two categories:

[0100] (4.1) When all AGVs have executed the action and no AGV has reached the target point, S 小车 will change, namely:

[0101]

[0102] And S 目的地 With S 障碍物 No changes will occur.

[0103] (4.2) After all AGVs have executed the action, when an AGV reaches the target point, S 小车 With S 目的地 Will change, namely:

[0104]

[0105] where i1-i k For the AGV that reaches the target point (without a sequential relationship), S o After the AGV reaches the destination, it will be reassigned a new task without waiting for other AGVs to complete their tasks.

[0106] (5) QMIX algorithm network structure design

[0107] The network structure of the QMIX algorithm in this scheme is mainly divided into three parts. First, each AGV has an RNN network ( Figure 6 The green network in the middle is controlled by a three-layer network, the input layer (MLP, Multilayer Perceptron), the middle layer (GRU, Gate Recurrent Unit), and the output layer (MLP). The input is the observation space of each AGV, the action code of the AGV, and the number (id) of the AGV, and the output is the Q value of the AGV.

[0108] The second is the hybrid network. In order to meet the above monotonicity constraints, all values ​​of the hybrid network are non-negative numbers, and there is no limit on the offset. The hybrid network consists of two networks. The first is the inference network ( Figure 6 The blue network in the middle), the input is the Q value of all AGVs, and the output is the global Q value Q tot, which is different from general neural networks, its weights and bias parameters are not generated by self-training, but are generated by another parameter network. Figure 6 The input of the orange network is the global state s t , the output is the weight W and bias B of the inference network. In order to ensure the non-negativity of the weight, a linear network and an absolute value activation function are used to ensure that the output is not negative. The offset is used in the same way but without the non-negativity constraint. The offset of the last layer of the hybrid network is obtained through a two-layer network and a RELU activation function to obtain a nonlinear mapping network. After the inference network receives the parameters of the parameter network, it directly calculates the global Q value Q tot .

[0109] The final error function of the QMIX algorithm is:

[0110]

[0111] Update the method using TD-learning:

[0112]

[0113] The prediction error is minimized by using the error back propagation gradient descent method, so that the neural network can accurately predict the maximum Q value in each state, and then the action selection can be performed based on the predicted Q value.

[0114] Table 2 QMIX algorithm execution diagram

[0115]

[0116]

[0117] Most of the conflicts encountered in this embodiment are node conflicts, and three types of conflicts are divided from this conflict, namely, follow-up conflict, transposition conflict, and wrap-around conflict (see Figure 7 ).

[0118] Node conflict, that is, AGV plans to occupy the same node at the same time step:

[0119] π i [t] = π i [t]

[0120] Following conflict, that is, an AGV plans to occupy a node that was occupied by another AGV in the previous time step:

[0121] π i [t+1]=π j [t]

[0122] A swap conflict occurs when two AGVs plan to swap positions in a single time step:

[0123] π i [t] = π j [t+1] and π i [t+1]=π j [t]

[0124] Cyclic conflicts, where each AGV moves to a vertex previously occupied by another AGV, forming a "spinning loop" pattern, also known as a deadlock:

[0125] π i (t+1)=π i+1 (t) and π i+1 (t+1)=π i+2 (t)...π j-1 (t+1)=π j (t) and π j (t+1)=π i (t) In addition, the portability of the QMIX algorithm in this solution is explained as follows: In the intelligent agent path planning problem, the environment can be fully observable or partially observable. As the path planning scene map increases, full observability will bring about a massive amount of data input dimensions, and the model trained for a specific scenario cannot adapt to the changing environment. A partially observable environment can train the model to use less information input, which helps to reduce the input dimension of the neural network, and the AGV path planning model trained with a limited field of view is insensitive to changes in the size of the environment and can be well transplanted to a new environment. In this solution, the environment range that the AGV can observe is set to a specific area centered on itself. An example of the AGV field of view is shown in the attached figure. Figure 8 shown. Figure 8 A partial area map of the automated warehouse is shown, where a blue AGV performs loading or unloading tasks, and the field of view in the figure is also centered on the blue AGV. There are four AGVs in total: yellow, green, blue, and gray, and the corresponding color circles are their end points. The red dotted box is a 2x2 field of view of the blue AGV, and the field of view is always centered on the AGV. It will change position as the AGV moves. This area contains obstacle information and information such as the current position of other AGVs and the corresponding end point. In addition to the information in the field of view, the AGV can also obtain the Euclidean distance and direction information to reach its end point, which can ensure that the AGV is traveling in the direction of the end point.

[0126] After completing the iteration of multiple AGVs in one scene, this solution can also be transplanted to different scenes and can effectively improve the throughput. This shows that the reinforcement learning algorithm learns rules rather than simple map information. It has good portability and can be expanded to other warehouse scenes. Fig. 9 It is a random test image.

[0127] Compared with the existing AGV path planning method, the multi-AGV multi-cycle anti-collision path planning method for automated warehouses based on the QMIX algorithm proposed in this scheme introduces the local observation information of the AGV as input to convert it into an observation matrix, inputs the observation matrix into the deep neural network, outputs the value function Q value of each AGV, and then accepts the global information through the parameter network to generate parameters. Finally, the inference network reads the Q value of each AGV and the parameters of the parameter network to output the overall Q value, thereby achieving the goal of centralized training and distributed execution (global state information is used to improve training efficiency during training, and partial observation is taken during execution, that is, when a single AGV completes the task, its observation range is a rectangular range centered on itself, including information about other AGVs, obstacles, and end points, and the next action is determined based on this information).

[0128] In summary, this solution introduces an anti-collision mechanism in the QMIX algorithm in the path planning of multiple AGVs, thereby reducing or avoiding conflicts between two or more AGVs, and effectively improving the working efficiency of multiple AGVs in complex environments.

[0129] This solution puts multiple AGVs into the same tuple list. When an AGV completes a cycle, it will enter the next cycle to complete the next task, thus avoiding the AGV being set to stop or disappear after completing a cycle in the warehouse. In contrast, traditional heuristic algorithms are mainly aimed at single cycles, which is inconsistent with the actual environment.

[0130] The algorithm used in this solution is suitable for discrete environments. It has good portability and can cope with sudden changes in warehouse environments or be expanded to other warehouse environments.

[0131] Since this solution is to solve nonlinear and monotonic problems, the QMIX algorithm is designed to solve it. Compared with other reinforcement learning algorithms, the QMIX algorithm has good convergence when solving such low-complexity problems.

Claims

1. A multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm, characterized in that: The following steps are involved: S1. According to the warehouse layout, establish a warehouse node diagram, obtain some observation information of AGV, and determine the initial starting point, end point and obstacle information of AGV; S2, check whether there is a trained path planning model. If so, input the AGV's initial starting point, end point and obstacle information, combined with the AGV's previous action and the AGV's ID information, into the path planning model, and output the AGV's corresponding driving path; otherwise, execute step S3; S3, build a reinforcement learning environment, input part of the observation information of each AGV into the RNN neural network of the QMIX algorithm, and determine the predicted Q value of each AGV through the ε-greedy strategy; The predicted Q value is input into the Mixing Network, and the predicted Q value is evaluated to obtain the overall Q value; S4, according to the overall Q value and the reward value of the AGV, guide the AGV to make the next action selection, save the observation information and optimal path of each AGV in the hidden layer, save the overall Q value and the real reward value of the environment, and update the obstacle information observed by the AGV; Through loop iteration, a single AGV output can obtain a higher predicted Q value, making the overall Q value closer to the actual reward value of the environment, and constructing a path planning model, and then returning to step S2.

2. According to the QMIX algorithm-based multi-AGV multi-cycle path planning method for an automated warehouse according to claim 1, it is characterized in that: Part of the observation information of the AGV in step S1 is specifically: <I,S,A,T,R,O,Ω,γ> <h2 style=";text-align:left;direction:ltr">A = {A1,A2,A3,...,A<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr">} T(s,a,s′)=P(s′|s,a) Where I represents the number of agents; S represents the state space, which contains the agent and environment information; A represents the joint action space, where A i represents the local action a that agent i can take i T is the state transition function, which represents the probability function of the agent transferring to state s′ after taking joint action a in state s; R is the reward function, which is used to represent the reward obtained by the agent when state transition occurs; O is the observation space, which represents the set of joint observations, o t is the observation set at time t, Ω is the observation function; γ∈[0,1] is the discount factor.

3. The multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm according to claim 2 is characterized in that: The state space is specifically: S = (S 小车 , S 目的地 , S 障碍物 ) Among them, the state of AGV, i.e. the position coordinates, is given by It is represented by n position coordinate tuples, where n is the number of AGVs; The state of the AGV target, i.e. the target position coordinates, is given by Represents, including n position coordinate tuples, S 小车 and S 目的地 Through one-to-one correspondence in sequence, the AGV is guaranteed to correspond to the target point one-to-one; The state of the obstacle, i.e. the obstacle position coordinates, is given by It indicates that it contains the position coordinates of all obstacles and consists of m position coordinate tuples, where m is the number of obstacles; All AGVs are at their initial positions at time 0. When a coordinate position is occupied by an AGV or an obstacle, other AGVs will not be able to reach that node, that is, the AGV coordinate position and the obstacle coordinate position cannot coincide. When the AGV reaches its target position, the next target position will be generated for it and stored in the target state S. 障碍物 middle.

4. The multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm according to claim 2 is characterized in that: The observation space is specifically: O = (O 障碍物 , O 小车 , O 目的地 ), decompose it into three matrices, namely the obstacle matrix, other AGV matrices and target projection matrix. The obstacle matrix O 障碍物 Contains obstacle information within the observable distance of the AGV. When there is an obstacle within the observation range, this part of the matrix will be set to 1, otherwise it will be 0; AGV Matrix O 小车 Contains information about other AGVs within the observable range of the AGV. When there are other AGVs within the observable range, this part of the matrix will be set to 1, otherwise it will be set to 0; The target matrix O 目的地 Contains the location information of the AGV target point. When the target point is not within the observation range of the current AGV, it is the projection of a vector from the AGV to the target point in the matrix, and will be displayed at the edge of the matrix, that is, the edge position is 1 and other positions are 0; when the target point enters the observation space of the AGV, the target point is set to 1 and the others are 0.

5. The multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm according to claim 2 is characterized in that: The path planning model in step S2 includes multiple RNN neural networks, which are respectively connected to the hybrid network, and the multiple RNN neural networks correspond to different AGVs, respectively. Each RNN neural network includes an input layer MLP, an intermediate layer GRU and an output layer MLP. The input of the RNN neural network is the observation space of each AGV, the action code of the AGV and the ID of the AGV, and the output is the Q value of the AGV. The hybrid network includes an inference network and a parameter network. The input of the inference network is the Q value of all AGVs, and the output is the global Q value Q tot ,The weight and bias parameters in the inference network are generated by the parameter network; The input of the parameter network is the global state s t , the output is the weight W and bias B of the inference network; The offset of the last layer of the hybrid network is obtained through a two-layer network and a RELU activation function to obtain a nonlinear mapping network. The inference network directly calculates the global Q value Q after receiving the parameters of the parameter network. tot .

6. The multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm according to claim 5 is characterized in that: The working process of the RNN neural network in step S3 includes: inputting the current observation value of each AGV, the action of the previous step, and the one-hot encoding of the ID of each AGV represented in vector form to the input layer MLP, receiving the input of the first layer and hidden historical information through the middle layer GRU, and outputting the predicted Q value through the output layer MLP.

7. The multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm according to claim 5 is characterized in that: The calculation formula of the ε-greedy strategy in step S3 is: Among them, c+1 represents the total number of actions in the AGV action space, a * represents the action that maximizes the Q value, π(a|s) represents the action of the AGV in a certain state in the π strategy, that is, there is a probability of ε to randomly select an action from c+1 actions, including the action that maximizes the Q value, and there is a probability of 1-ε to directly select the action that maximizes the Q value. ε decreases as the training progresses, thereby ensuring that the action selection of the AGV is more exploratory in the early stage of training and more stable in the later stage.

8. The multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm according to claim 5 is characterized in that: The reward value of the AGV in step S4 is calculated by the reward function: r(s,a)=w t r t (s,a)-w p r p (s,a)-w c r c (s,a) Among them, rt is the reward value obtained by the AGV when it reaches the target point, rp(s,a) represents the movement penalty. When the AGV moves, it will obtain a certain penalty value, thereby encouraging the AGV to reach the target point with as few moves as possible. rc(s,a) is the conflict penalty. When a conflict occurs, a negative reward value will be given to enable the AGV to avoid the conflict as much as possible. When the AGV reaches the target, it will obtain a reward value of 1, that is, at this time r t (s,a)=1, and the reward value is 0 in other cases. When a conflict occurs, the AGV will not be able to pass the target point and will receive a conflict penalty r. c (s, a).

9. The multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm according to claim 8, characterized in that: The conflicts include node conflicts, follow-up conflicts, transposition conflicts and cycle conflicts.

10. The multi-AGV multi-cycle path planning method for an automated warehouse based on the QMIX algorithm according to claim 9, characterized in that: The node conflict specifically refers to: AGV plans to occupy the same node at the same time step; The following conflict specifically refers to: an AGV plans to occupy a node that was occupied by another AGV in the previous time step; The transposition conflict specifically refers to: two AGVs plan to exchange positions in a single time step; The cycle conflict specifically means that each AGV moves to a vertex previously occupied by another AGV, forming a "rotating cycle" pattern, also known as a deadlock.

Images