Multi-AGV path planning model training method and path planning method based on MAPPO algorithm

Abstract

This specification relates to the field of intelligent logistics technology, and to a multi-AGV path planning model training method and a path planning method based on the MAPPO algorithm. The model training method includes: using the MAPPO algorithm that combines a recurrent neural network and a policy gradient algorithm to parameterize the multi-AGV action strategy, and optimizing the parameters to increase the probability of the AGV selecting an action with a greater reward, thereby obtaining a multi-AGV action path with more rewards. The path planning method includes: performing global path planning on a single AGV, and then based on the global path planning, using a multi-AGV path planning model to resolve path conflicts during the action of multiple AGVs, and obtaining the optimal path for the actions of multiple AGVs. The multi-AGV path planning model training method and path planning method based on the MAPPO algorithm in this specification can efficiently and accurately complete path planning under multi-AGV working conditions, can ensure real-time obstacle avoidance of AGVs in complex environments, improve cargo transportation efficiency, and reduce costs.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of intelligent logistics technology, and in particular to a multi-AGV path planning model training method and a path planning method based on the MAPPO algorithm. Background Art

[0002] With the continuous advancement of technology and the demand for high-efficiency, low-cost transportation in warehouses, traditional warehouses are transitioning to unmanned, intelligent models. Intelligent logistics and distribution are a crucial component of intelligent warehouses. AGVs (Automated Guided Vehicles) are widely used in smart manufacturing workshops and logistics and transportation due to their high efficiency and plannable paths. They possess superior path planning capabilities and can, to a certain extent, fully automate the entire operation process. However, a single AGV struggles to efficiently transport goods, so multiple AGVs are required to operate simultaneously. This requires establishing a model of the transportation space, planning conflict-free paths for each AGV, and finding the optimal path while avoiding collisions between AGVs, thereby reducing power consumption and time costs.

[0003] Traditional path conflict elimination methods use heuristic or meta-heuristic algorithms to plan paths for each AGV, predict and analyze AGV collisions, and re-plan routes to reduce collisions. However, as the number of AGVs deployed increases, the smart warehousing environment becomes increasingly complex, and the density of AGVs increases, the probability of collisions increases. Existing path conflict elimination methods are computationally intensive and time-consuming, making it difficult to quickly plan AGV paths during the cargo transportation and picking process, and it is also difficult to ensure the accuracy of path planning. Summary of the Invention

[0004] The embodiments of this specification provide a multi-AGV path planning model training method and a path planning method based on the MAPPO algorithm, aiming to solve one or more of the above-mentioned problems and other potential problems.

[0005] To achieve the above objectives, the following technical solutions are provided:

[0006] According to a first aspect of this specification, a multi-AGV path planning model training method based on the MAPPO algorithm is provided, comprising:

[0007] Obtain the policy network and value network with parameter initialization; obtain the AGV transportation task, the AGV global path planning, the AGV state space and the AGV action space; the AGV state space includes the AGV position, AGV orientation, AGV load, the end point of the AGV path, the static obstacles and dynamic obstacles in the AGV observation space; the AGV action space is the action type of the AGV during movement;

[0008] The AGV's global path planning, AGV's state space, and action space are input into the policy network, which outputs the action probability of the AGV in its current state. The AGV selects an action based on the action probability; the AGV's reward function value is calculated based on the action and stored.

[0009] Based on the AGV state space and the AGV reward function value, the advantage function value of the action corresponding to the AGV's current state is calculated. The advantage function value is used to measure the quality of each action relative to the average level. When the action is better than the average action, the probability of selecting the action is increased.

[0010] Calculate the loss function values ​​of the policy network and the value network, and update the parameters of the policy network and the value network;

[0011] Repeat the above process of selecting AGV actions, calculating reward function values, calculating advantage function values, and updating the policy network and value network until the AGV completes all tasks or reaches the maximum number of steps. The updated policy network and value network are the trained multi-AGV path planning model.

[0012] The multi-AGV path planning model training method based on the MAPPO algorithm in the embodiments of this specification combines neural networks and policy gradient algorithms to continuously update and optimize the parameters in the policy network and value network, and generates a path with greater rewards, that is, the optimal path, based on the reward function; it has fast computing speed and high accuracy, can adapt to path planning in complex multi-AGV transportation environments, ensure the efficiency and safety of multiple AGVs during transportation, and reduce computing and maintenance costs.

[0013] In some embodiments, the reward function value of the AGV is calculated based on the following reward function formula:

[0014] R=r1+r2+r3+r4+r5+r6

[0015] r1=-1

[0016] r2=2Δd

[0017]

[0018]

[0019]

[0020] Among them, R represents the reward function value, r1 represents the step penalty generated during the operation of the AGV; r2 represents the reward generated based on the distance difference between the current state of the AGV and the previous state to the end point, Δd is the distance difference between the current state of the AGV and the previous state to the end point; r3 represents the penalty set for the AGV within the range when other AGVs enter the observation range, d1 is the distance between the AGVs, and rr is the observation range of the AGV; r4 represents the penalty generated by the collision between the AGV and the shelf, d2 is the distance between the AGV and the shelf; r5 represents the guidance reward generated based on the global path planning during the AGV's movement. When the AGV travels to the global path obtained by the A* algorithm, the AGV generates a reward based on the path distance between the starting point and the current position of the path, and deletes the path trajectory between the starting point and the current position of the path, N e is the path distance between the starting point of the path and the current position of the AGV, c r (t+1) represents the position of AGV at the next moment, l G is the global path; r6 represents the reward generated when the AGV obtains the shelf or reaches the end of the path.

[0021] In some embodiments, the advantage function value of the action corresponding to the current state of the AGV is calculated, and the quality of each action relative to the average level is measured based on the advantage function value, including:

[0022] Use the generalized advantage estimate to calculate the advantage function value of the action corresponding to the current state. The formula is as follows:

[0023]

[0024] Among them, A(s t , a t ) indicates that AGV is in s t In the state, take a t The advantage function value at the time of action, t represents time, T represents the length of the experience trajectory, γ represents the discount factor, λ represents the parameter of the generalized advantage estimate, δ t represents the time series differential error at time t, r t represents the immediate reward for GAV at each time step t, V(s t ) represents state s t The value function estimate, V(s t+1 ) represents state s t+1 Estimation of the value function of ;

[0025] When the advantage function value is positive, it means that the current action of the AGV is better than the average action, which increases the probability of the AGV selecting this action.

[0026] In some embodiments, calculating the loss function values ​​of the policy network and the value network and updating the parameters of the policy network and the value network include:

[0027] Calculate the policy network loss function value of AGV, which is the sum of policy loss and policy entropy loss;

[0028] Update the policy network parameters using the gradient ascent method based on the policy network loss function value;

[0029] Calculate the network loss function value of AGV using the following formula:

[0030]

[0031] Among them, L(φ) represents the loss function value of the value network, φ represents the value network parameter, B represents the batch size, n represents the training batch size, V φ (si) represents the state S of the value network for the i-th batch i The value function estimate, V φold (s i ) represents the state S of the value network for the i-th batch i The historical value function estimate of represents the kth power of the state of the i-th batch, R i represents the actual reward of the i-th batch, and ε represents the hyperparameter;

[0032] The gradient descent method is used to update the value network parameters based on the value network loss function.

[0033] According to a second aspect of this specification, a path planning method is provided, comprising:

[0034] Obtain the map information of the warehouse to be picked and the AGV transportation tasks, and obtain a multi-AGV path planning model; the multi-AGV path planning model is trained according to the multi-AGV path planning model training method based on the MAPPO algorithm;

[0035] Based on the map information and AGV transportation tasks, a global path planning is performed for each AGV to obtain the global path of each AGV;

[0036] The AGV transport task, the global path planning of each AGV, and the AGV state space are input into the multi-AGV path planning model to obtain the optimal motion trajectory of each AGV;

[0037] Complete warehouse picking tasks based on the optimal motion trajectory of each AGV.

[0038] In some embodiments, global path planning is performed for each AGV based on map information and AGV transport tasks, including:

[0039] Based on the map information and AGV transport tasks, the coordinates of the AGV starting point, shelf, and picking platform on the map are obtained, and the starting and ending coordinates of the three paths from the AGV starting point to the shelf, shelf to the picking platform, and picking platform to the shelf are obtained;

[0040] Based on the evaluation function, the A* algorithm is used to search for the best path for each of the three paths;

[0041] The evaluation function formula is as follows: f(n) = g(n) + h(n)

[0042] Among them, n represents the current position of the AGV during the movement process, f(n) represents the total path cost of the AGV from the starting point of a path to the current position and then to the end point of the path, g(n) represents the actual path cost of the AGV from the starting point of the path to the current position, and h(n) represents the estimated path cost of the AGV from the current position to the end point of the path.

[0043] In some embodiments, based on the evaluation function, an A* algorithm is used to search for the best path for each of the three paths, including:

[0044] Enter the starting point of the path to be searched into the Open list, search its adjacent nodes based on the search range, and enter the Open list except for the inaccessible adjacent nodes. Move the starting point as the parent node into the Close list. Traverse the adjacent nodes of the parent node in the Open list, calculate the f(n) value of each adjacent node, move the adjacent node with the smallest f(n) value into the Close list, and set it as the current node. The Open list is a storage list of unsearched nodes, and the Close list is a storage list of searched nodes.

[0045] Detect the adjacent nodes of the current node. Except for the inaccessible adjacent nodes and the nodes in the Close list, if the remaining adjacent nodes are not in the Open list, enter these adjacent nodes into the Open list, with the current node as the parent node, calculate the f(n) value of each adjacent node of the parent node, move the adjacent node with the smallest f(n) value into the Close list and set it as the new current node; if the remaining adjacent nodes are in the Open list, recalculate the f(n) value and g(n) value of each adjacent node. If there is an adjacent node with a smaller g(n) value than the current node, recalculate the f(n) value and g(n) value of the adjacent nodes of the parent node, move the adjacent node with the smallest f(n) value into the Close list and set it as the new current node;

[0046] Determine whether the end point of the path to be searched has been found. If not, repeat the previous step. If the end point is found, determine whether the Open list is empty. If the Open list is not empty, start from the end point and search for the parent node of each node in reverse order until the starting point is found. The path formed by the parent nodes is the best path for the path to be searched.

[0047] Use the above steps to search for the best path for each of the three paths.

[0048] In some embodiments, the cost coefficient of AGV turning is introduced into the A* algorithm to make the AGV tend to take a path with fewer turns. In this case, the formula for estimating the path cost h(n) in the evaluation function is as follows:

[0049] h(n)=kh panh (n),

[0050] Among them, h(n) represents the estimated path cost of the AGV from its current position to the end point of its path, k represents the cost estimation coefficient, and h path (n) represents the actual estimated path cost of the AGV from its current position to the end of its path, (x n ,y n ) represents the current node, (x n-1 ,y n-1 ) represents the parent node, (x n+1 ,y n+1 ) represents the next estimated node of the current node.

[0051] In some embodiments, when searching for adjacent nodes, if the AGV is in a loaded state, the coordinate node of the shelf is used as an inaccessible adjacent node; if the AGV is in an unloaded state, the coordinate node of the shelf is used as a accessible adjacent node.

[0052] This specification also provides a computer storage medium storing a readable program, which, when executed by a processor, can execute the above-mentioned multi-AGV path planning model training method and path planning method based on the MAPPO algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and other objects, features and advantages of the embodiments of the present specification will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present specification are shown by way of example and not limitation.

[0054] Figure 1 A flowchart of a multi-AGV path planning model training method based on the MAPPO algorithm according to an embodiment of this specification is shown;

[0055] Figure 2A flow chart showing a path planning method according to an embodiment of the specification is shown;

[0056] Figure 3 Shown Figure 1 The relationship between the average reward and training rounds of the multi-AGV path planning model training;

[0057] Figure 4 Shown Figure 1 The relationship between the loss value and training rounds of the multi-AGV path planning model training;

[0058] Figure 5 Shown Figure 2 An example diagram of the path planning method in

[15] .

[0059] 1-empty AGV, 2-loaded AGV, 21-destination shelf, 3-shelf, 4-picking platform.

[0060] In the various drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0061] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present invention more thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0062] The term "including" and its variations used in this document indicate open inclusion, that is, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one other embodiment". Terms such as "upper", "lower", "front", and "rear" indicating placement or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the principles of this specification, and do not indicate or imply that the referred elements must have a specific orientation, be constructed or operate in a specific orientation, and therefore should not be understood as limiting this specification.

[0063] The multi-AGV path planning model training method and path planning method based on the MAPPO algorithm according to the embodiments of this specification are described in detail below with reference to the accompanying drawings. Figure 1 The flowchart of the multi-AGV path planning model training method based on the MAPPO algorithm of the embodiment of this specification is shown. The multi-AGV path planning model training method based on the MAPPO algorithm of the embodiment of this specification includes:

[0064] S01. Obtain the policy network and value network with parameter initialization; obtain the AGV transportation task, the AGV global path planning, the AGV state space and the AGV action space; the AGV state space includes the AGV position, AGV orientation, AGV load, the end point of the AGV path, the static obstacles and dynamic obstacles in the AGV observation space; the AGV action space is the action type of the AGV during movement;

[0065] S02, input the AGV's global path planning, AGV's state space and action space into the policy network, output the action probability of the AGV in the current state, and the AGV selects an action based on the action probability;

[0066] S03. Calculate the reward function value of the AGV based on the action and store it;

[0067] S04. Based on the AGV state space and the AGV reward function value, calculate and store the advantage function value of the action corresponding to the AGV's current state. Measure the quality of each action relative to the average level based on the advantage function value. When an action is better than the average action, increase the probability of selecting the action.

[0068] S05. Calculate the loss function values ​​of the policy network and the value network, and update the parameters of the policy network and the value network;

[0069] S06. Repeat steps S02, S03, S04, and S05 until the AGV completes all tasks or reaches the maximum number of steps. The updated policy network and value network are the trained multi-AGV path planning model.

[0070] In step S01, the policy network outputs the AGV's action probability based on the AGV's state space and action space, determining the appropriate action for the AGV in the current environment. The value network provides estimated values ​​for each state, which are used to evaluate the quality of the policy network's actions. When the warehouse map for the AGV's actions is a grid map, the AGV's turning, deceleration before the turn, and acceleration after the turn are roughly equivalent to the AGV moving forward one grid. In this case, the AGV's action space includes straight ahead, left turn, right turn, and waiting.

[0071] In step S02, step S03 and step S04, the actions, reward function values ​​and advantage function values ​​selected in each state of the AGV are stored in the established experience pool.

[0072] In step S03, the reward function value is calculated using the following reward function formula:

[0073] R=r1+r2+r3+r4+r5+r6

[0074] r1=-1

[0075] r2=2Ad

[0076]

[0077]

[0078]

[0079] Among them, R represents the reward function value, r1 represents the step penalty generated during the operation of the AGV; r2 represents the reward generated based on the distance difference between the current state of the AGV and the previous state to the end point, Δd is the distance difference between the current state of the AGV and the previous state to the end point; r3 represents the penalty set for the AGV within the range when other AGVs enter the observation range, d1 is the distance between the AGVs, and rr is the observation range of the AGV; r4 represents the penalty generated by the collision between the AGV and the shelf, d2 is the distance between the AGV and the shelf; r5 represents the guidance reward generated based on the global path planning during the AGV's movement. When the AGV travels to the global path obtained by the A* algorithm, the AGV generates a reward based on the path distance between the starting point and the current position of the path, and deletes the path trajectory between the starting point and the current position of the path, N e is the path distance between the starting point of the path and the current position of the AGV, c r (t+1) represents the position of AGV at the next moment, l G is the global path; r6 represents the reward generated when the AGV obtains the shelf or reaches the end of the path.

[0080] When the reward function formula is based on the guidance reward generated by global path planning, the path trajectory that has generated rewards will be deleted to avoid duplicate rewards. When the warehouse map is a grid map, N e Represents the number of grid cells between the AGV's starting point and its current position on the path. Rewards are generated based on the number of grid cells between the path's starting point and its current position. Rewards are generated when the AGV acquires a shelf or reaches the path's end point; they do not need to occur simultaneously. The reward value is 30. The reward value and coefficient in the reward function formula can be modified appropriately based on the specific situation.

[0081] In step S04, the advantage function value of the action corresponding to the current state is calculated using the generalized advantage estimation, and the formula is as follows:

[0082]

[0083] Among them, A(s t , a t ) indicates that AGV is in s t In the state, take a tThe advantage function value at the time of action, t represents time, T represents the length of the experience trajectory, γ represents the discount factor, λ represents the parameter of the generalized advantage estimate, δ t represents the time series differential error at time t, r t represents the immediate reward for GAV at each time step t, V(s t ) represents state s t The value function estimate, V(s t+1 ) represents state s t+1 Estimation of the value function of ;

[0084] When the advantage function value is positive, it means that the current action of the AGV is better than the average action, which increases the probability of the AGV selecting this action.

[0085] The advantage function only encourages actions with values ​​greater than the average value, rather than just actions with positive values. It can evaluate the extent to which an action is better or worse than following the current strategy in a specific state, optimize the action strategy and reduce variance.

[0086] Step S05 includes:

[0087] S501, calculating the policy network loss function value of the AGV, where the policy network loss function value is the sum of the policy loss and the policy entropy loss;

[0088] The policy network loss function formula is as follows:

[0089]

[0090]

[0091]

[0092] Among them, L(θ) represents the loss function value of the policy network, B represents the batch size, m represents the number of AGVs, θ represents the policy network parameters, and r(θ) represents the policy improvement ratio. represents the kth power of the i-th batch when the parameter is θ, represents the kth power of the average level of the i-th batch, ε represents the hyperparameter, σ represents the entropy coefficient, S represents the policy entropy, π θ represents the policy network with parameters θ, φ θlod represents the historical policy network with parameter θ, α represents the learning rate, Represents the k-th power of the state of the i-th batch;

[0093] S502, updating the policy network parameters using a gradient ascent method based on the policy network loss function value;

[0094] S503. Calculate the network loss function value of the AGV. The formula is as follows:

[0095]

[0096] Among them, L(φ) represents the loss function value of the value network, φ represents the value network parameter, B represents the batch size, n represents the training batch size, V φ (si) represents the state S of the value network for the i-th batch i The value function estimate, V φold (s i ) represents the state S of the value network for the i-th batch i The historical value function estimate of represents the kth power of the state of the i-th batch, R i represents the actual reward of the i-th batch, and ε represents the hyperparameter;

[0097] S504: Update the value network parameters using a gradient descent method based on the value network loss function.

[0098] In step S502, when the policy network parameters are updated using the gradient ascent method, the following The update rule is α, which represents the learning rate. In step S504, the network parameters are updated using the gradient descent method, following The update rule is , and γ represents the discount factor.

[0099] In step S06, the model training results are as follows: Figure 3 、 Figure 4 As shown, the relationship between the average reward of the model and the model training round, and the loss value and the model training round are respectively represented.

[0100] Figure 2 The flow chart of the path planning method of the embodiment of this specification is shown. First, a single AGV path is planned, and then a multi-AGV path planning model is used to resolve conflicts that may occur when multiple AGVs move. The path planning method of the embodiment of this specification includes:

[0101] S1. Obtain map information of the warehouse to be picked and the AGV transportation task, and obtain a multi-AGV path planning model; the multi-AGV path planning model is trained according to the multi-AGV path planning model training method based on the MAPPO algorithm;

[0102] S2. Based on the map information and AGV transportation tasks, a global path planning is performed for each AGV to obtain the global path of each AGV;

[0103] S3, input the AGV transport task, the global path planning of each AGV, and the AGV state space into the multi-AGV path planning model to obtain the optimal motion trajectory of each AGV;

[0104] S4. Complete the warehouse picking task based on the optimal motion trajectory of each AGV.

[0105] In step S1, the map of the warehouse to be picked is a grid map. The map information includes shelf coordinates, obstacle coordinates, picking platform coordinates, etc. The AGV transportation task can be automatically assigned by computer system, artificial intelligence, etc. according to the shelf information and cargo demand, or it can be manually input. Figure 5 As shown in the figure, a 27×31 grid map is created based on the warehouse conditions and work environment. The black grid cells in the grid map represent shelf 3, which is densely populated. The hexagons in the grid map represent AGVs, with a total of 10 AGVs transporting goods. The dotted lines between the AGVs represent their paths. The white grid cells in the grid map represent the aisles between shelves 3. The AGV transport task list contains 50 transport tasks, each assigned to 10 AGVs. A transport task includes the coordinates of shelf 3 to be transported and the corresponding picking platform 4.

[0106] In step S2, the A* algorithm is used to perform global path planning for each AGV. The steps include:

[0107] S21. Based on the map information and the AGV transport task, obtain the coordinates of the AGV starting point, shelf, and picking platform on the map, and obtain the starting and ending coordinates of the three paths from the AGV starting point to the shelf, from the shelf to the picking platform, and from the picking platform to the shelf;

[0108] S22. Based on the evaluation function, use the A* algorithm to search for the best path for each of the three paths;

[0109] The evaluation function formula is as follows: f(n) = g(n) + h(n)

[0110] Among them, n represents the current position of the AGV during the movement process, f(n) represents the total path cost of the AGV from the starting point of a path to the current position and then to the end point of the path, g(n) represents the actual path cost of the AGV from the starting point of the path to the current position, and h(n) represents the estimated path cost of the AGV from the current position to the end point of the path.

[0111] In step S21, different paths can be set according to the specific warehouse conditions, not limited to the three paths from the AGV starting point to the shelf, from the shelf to the picking platform, and from the picking platform to the shelf.

[0112] In step S22, the AGV is in a state where it can only move in the forward, backward, left, and right directions but not diagonally. The Manhattan distance is used as the heuristic function for the path distance, and the formula is as follows:

[0113] h(n)=abs(nx-goal.x)+abs(ny-goal.y)

[0114] Where abs represents the absolute value, n represents the current position of the AGV during the movement, goal represents the target position of the AGV during the movement, x represents the horizontal coordinate, and y represents the vertical coordinate.

[0115] During the AGV's turning process, acceleration and deceleration occur, which will take more driving time and consume more electricity. Therefore, the cost coefficient of AGV turning is introduced into the evaluation function to make the AGV tend to take a path with fewer turns. At this time, the formula for estimating the path cost h(n) in the evaluation function is as follows:

[0116] h(n)=kh panh (n),

[0117] Among them, h(n) represents the estimated path cost of the AGV from its current position to the end point of its path, k represents the cost estimation coefficient, and h path (n) represents the actual estimated path cost of the AGV from its current position to the end of its path, (x n ,y n ) represents the current node, (x n-1 ,y n-1 ) represents the parent node, (x n+1 ,y n+1 ) represents the next estimated node of the current node.

[0118] In step S22, the A* algorithm is used to search for the best path for each of the three paths, including:

[0119] S221: Enter the starting point of the path to be searched into the Open list, search its adjacent nodes based on the search range, enter the Open list except for the inaccessible adjacent nodes, and move the starting point as the parent node into the Close list; traverse the adjacent nodes of the parent node in the Open list, calculate the f(n) value of each adjacent node, move the adjacent node with the smallest f(n) value into the Close list, and set it as the current node; the Open list is a storage list of unsearched nodes, and the Close list is a storage list of searched nodes;

[0120] S222. Detect the adjacent nodes of the current node. Except for the inaccessible adjacent nodes and the nodes in the Close list, if the remaining adjacent nodes are not in the Open list, enter these adjacent nodes into the Open list, use the current node as the parent node, calculate the f(n) value of each adjacent node of the parent node, move the adjacent node with the smallest f(n) value into the Close list and set it as the new current node; if the remaining adjacent nodes are in the Open list, recalculate the f(n) value and g(n) value of each adjacent node. If there is an adjacent node with a smaller g(n) value than the current node, recalculate the f(n) value and g(n) value of the adjacent nodes of the parent node, move the adjacent node with the smallest f(n) value into the Close list and set it as the new current node;

[0121] S223. Determine whether the end point of the path to be searched is found. If not, repeat step S222. If the end point is found, determine whether the Open list is empty. If the Open list is not empty, start from the end point and search for the parent node of each node in reverse order until the starting point is found. The path formed by the parent nodes is the optimal path of the path to be searched.

[0122] S224. Use steps S221, S222, and S223 to search for the best path for each of the three paths.

[0123] When the warehouse map is a grid map, the node is the center point of the grid.

[0124] In step S221 and step S222, when detecting inaccessible adjacent nodes, the inaccessible adjacent nodes are different depending on the different conditions of the AGV's loaded state and empty state. If the AGV is in a loaded state, the coordinate node of the shelf is used as an inaccessible adjacent node. If the AGV is in an empty state, the AGV can pass through the bottom of the shelf, and the coordinate node of the shelf is used as a accessible adjacent node, thereby reducing the AGV's detour and reducing power consumption. Figure 5 The diagram shows an example of path planning for 10 AGVs. Unladen AGV 1 can travel under shelf 3, and its path includes the coordinate nodes on shelf 3. Loaded AGV 2 cannot travel under shelf 3, and its path does not include the coordinate nodes on shelf 3. Loaded AGV 2 retrieves the goods from its target shelf 21 and delivers them to the corresponding coordinate node on picking platform 4.

[0125] In step S3, the AGV's state space specifically includes information such as the AGV's observation range, the coordinates of other AGVs and obstacles within the observation range, and the AGV's path endpoint. For example, if the AGV's observation range is set to 3 grids on a grid map, a 7×7 area centered on the AGV is the AGV's observation area. The AGV constructs its state space based on the coordinates of obstacles within the observation area, the coordinates of other AGVs, and the AGV's global path planning information. This AGV's state space is then input into the multi-AGV path planning model.

[0126] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this specification. Certain features described in the context of separate embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable subcombination.

[0127] Although the subject matter has been described using language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. The various embodiments of this specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A multi-AGV path planning model training method based on the MAPPO algorithm is characterized by: include: Obtain the policy network and value network with parameter initialization; obtain the AGV transportation task, the AGV global path planning, the AGV state space and the AGV action space; the AGV state space includes the AGV position, AGV orientation, AGV load, the end point of the AGV path, the static obstacles and dynamic obstacles in the AGV observation space; the AGV action space is the action type of the AGV during movement; The AGV's global path planning, AGV's state space, and action space are input into the policy network, and the action probability of the AGV in the current state is output. The AGV selects an action based on the action probability, and calculates and stores the AGV's reward function value based on the action; Based on the AGV state space and the AGV reward function value, the advantage function value of the action corresponding to the AGV's current state is calculated. The advantage function value is used to measure the quality of each action relative to the average level. When the action is better than the average action, the probability of selecting the action is increased. Calculate the loss function values ​​of the policy network and the value network, and update the parameters of the policy network and the value network; Repeat the above process of selecting AGV actions, calculating reward function values, calculating advantage function values, and updating the policy network and value network until the AGV completes all tasks or reaches the maximum number of steps. The updated policy network and value network are the trained multi-AGV path planning model. Calculating the loss function values ​​of the policy network and the value network, and updating the parameters of the policy network and the value network, include: Calculate the policy network loss function value of AGV, which is the sum of policy loss and policy entropy loss; Update the policy network parameters using the gradient ascent method based on the policy network loss function value; Calculate the network loss function value of AGV using the following formula: Among them, L(φ) represents the loss function value of the value network, φ represents the value network parameter, B represents the batch size, n represents the training batch size, V φ (s i ) represents the state S of the value network for the i-th batch i The value function estimate, V φold (s i ) represents the state S of the value network for the i-th batch i The historical value function estimate of represents the kth power of the state of the i-th batch, R i represents the actual reward of the i-th batch, and ε represents the hyperparameter; The gradient descent method is used to update the value network parameters based on the value network loss function.

2. The method according to claim 1, characterized in that The reward function value of AGV is calculated based on the following reward function formula: R＝r1+r2+r3+r4+r5+r6 r1＝-1 r2=2Δd Among them, R represents the reward function value, r1 represents the step penalty generated during the operation of the AGV; r2 represents the reward generated based on the distance difference between the current state of the AGV and the previous state to the end point, Δd is the distance difference between the current state of the AGV and the previous state to the end point; r3 represents the penalty set for the AGV within the range when other AGVs enter the observation range, d1 is the distance between the AGVs, and rr is the observation range of the AGV; r4 represents the penalty generated by the collision between the AGV and the shelf, d2 is the distance between the AGV and the shelf; r5 represents the guidance reward generated based on the global path planning during the AGV's movement. When the AGV travels to the global path obtained by the A* algorithm, the AGV generates a reward based on the path distance between the starting point and the current position of the path, and deletes the path trajectory between the starting point and the current position of the path, N e is the path distance between the starting point of the path and the current position of the AGV, c r (t+1) represents the position of AGV at the next moment, l G is the global path; r6 represents the reward generated when the AGV obtains the shelf or reaches the end of the path.

3. The method according to claim 1, characterized in that Calculate the advantage function value of the action corresponding to the current state of the AGV, and measure the quality of each action relative to the average level based on the advantage function value, including: using generalized advantage estimation to calculate the advantage function value of the action corresponding to the current state, the formula is as follows: Among them, A(s t , a t ) indicates that AGV is in s t In the state, take a t The advantage function value at the time of action, t represents time, T represents the length of the experience trajectory, γ represents the discount factor, λ represents the parameter of the generalized advantage estimate, δ t represents the time series differential error at time t, r t represents the immediate reward for GAV at each time step t, V(s t ) represents state s t The value function estimate, V(s t+1 ) represents state s t+1 Estimation of the value function of ; When the advantage function value is positive, it means that the current action of the AGV is better than the average action, which increases the probability of the AGV selecting this action.

4. A path planning method, characterized in that: include: Obtain map information of the warehouse to be picked and AGV transportation tasks, and obtain a multi-AGV path planning model; The multi-AGV path planning model is obtained by training according to any one of the methods described in claims 1-3; Based on the map information and AGV transportation tasks, a global path planning is performed for each AGV to obtain the global path of each AGV; The AGV transport task, the global path planning of each AGV, and the AGV state space are input into the multi-AGV path planning model to obtain the optimal motion trajectory of each AGV; Complete warehouse picking tasks based on the optimal motion trajectory of each AGV.

5. The method according to claim 4, characterized in that Based on map information and AGV transportation tasks, global path planning is performed for each AGV, including: Based on the map information and AGV transport tasks, the coordinates of the AGV starting point, shelf, and picking platform on the map are obtained, and the starting and ending coordinates of the three paths from the AGV starting point to the shelf, shelf to the picking platform, and picking platform to the shelf are obtained; Based on the evaluation function, the A* algorithm is used to search for the best path for each of the three paths; The evaluation function formula is as follows: f(n) = g(n) + h(n) Among them, n represents the current position of the AGV during the movement process, f(n) represents the total path cost of the AGV from the starting point of a path to the current position and then to the end point of the path, g(n) represents the actual path cost of the AGV from the starting point of the path to the current position, and h(n) represents the estimated path cost of the AGV from the current position to the end point of the path.

6. The method according to claim 5, characterized in that Based on the evaluation function, the A* algorithm is used to search for the best path for each of the three paths, including: Enter the starting point of the path to be searched into the Open list, search its adjacent nodes based on the search range, and enter the Open list except for the inaccessible adjacent nodes. Move the starting point as the parent node into the Close list. Traverse the adjacent nodes of the parent node in the Open list, calculate the f(n) value of each adjacent node, move the adjacent node with the smallest f(n) value into the Close list, and set it as the current node. The Open list is a storage list of unsearched nodes, and the Close list is a storage list of searched nodes. Detect the adjacent nodes of the current node. Except for the inaccessible adjacent nodes and the nodes in the Close list, if the remaining adjacent nodes are not in the Open list, enter these adjacent nodes into the Open list, with the current node as the parent node, calculate the f(n) value of each adjacent node of the parent node, move the adjacent node with the smallest f(n) value into the Close list and set it as the new current node; if the remaining adjacent nodes are in the Open list, recalculate the f(n) value and g(n) value of each adjacent node. If there is an adjacent node with a smaller g(n) value than the current node, recalculate the f(n) value and g(n) value of the adjacent nodes of the parent node, move the adjacent node with the smallest f(n) value into the Close list and set it as the new current node; Determine whether the end point of the path to be searched has been found. If not, repeat the previous step. If the end point is found, determine whether the Open list is empty. If the Open list is not empty, start from the end point and search for the parent node of each node in reverse order until the starting point is found. The path formed by the parent nodes is the best path for the path to be searched. Use the above steps to search for the best path for each of the three paths.

7. The method according to claim 6, characterized in that The method also includes introducing a cost coefficient for AGV turns into the A* algorithm, so that the AGV tends to take a path with fewer turns. In this case, the formula for estimating the path cost h(n) in the evaluation function is as follows: Among them, h(n) represents the estimated path cost of the AGV from its current position to the end point of its path, k represents the cost estimation coefficient, and h path (n) represents the actual estimated path cost of the AGV from its current position to the end of its path, (x n ,y n ) represents the current node, (x n-1 ,y n-1 ) represents the parent node, (x n+1 ,y n+1 ) represents the next estimated node of the current node.

8. The method according to claim 6 or 7, characterized in that When searching for adjacent nodes, if the AGV is in a loaded state, the coordinate node of the shelf is used as an inaccessible adjacent node; if the AGV is in an unloaded state, the coordinate node of the shelf is used as a accessible adjacent node.

9. A computer storage medium storing a readable program, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 can be executed.

Images