Goods-to-people system multi-AGV path planning method based on deep reinforcement learning

Abstract

The invention relates to a cargo-to-person system multi-AGV path planning method based on deep reinforcement learning in the field of computers, and the method comprises the steps: fusing a double-layer planning architecture of improved A * global guidance and local reinforcement learning, employing a non-uniform penalty heuristic strategy in a global layer, and employing a non-uniform penalty heuristic strategy in a local layer; performing collaborative optimization on the path dispersity through a fixed penalty term calculated by a Dijkstra algorithm and a real-time dynamic penalty term based on a historical path; an asynchronous near-end strategy optimization (APPO) algorithm is designed in a local layer, multi-channel observation states (static obstacles, AGV positions and global path codes) of five continuous time steps are processed in combination with an LSTM network, and a five-degree-of-freedom action strategy is output. A fixed punishment exemption mechanism is set for a storage movable shelf (Pod) area, invalid avoidance is avoided, the task success rate and throughput are improved, the calculation load is reduced, and an efficient and extensible solution is provided for solving the multi-AGV path planning problem of a cargo-to-person system.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular to the field of multi-agents, and more specifically to a multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning. Background Art

[0002] With the development of automation technology, the Goods-to-Person System (Robotic Mobile Fulfillment System, RMFS) has been widely used in the fields of manufacturing and logistics. The Goods-to-Person System uses automated guided vehicles (AGVs) to efficiently transport and store goods in warehouses, which not only greatly reduces the intensity of manual labor, but also improves the accuracy and efficiency of order processing. However, the operation of AGVs in the Goods-to-Person System is highly complex, especially in multi-AGV path planning, which faces many technical challenges. First of all, the path planning environment of AGVs in the Goods-to-Person System is significantly different from that in traditional AGV applications. The AGVs in the Goods-to-Person System need to be transported under the shelves (see Appendix Figure 1 ), and as the shelves move between storage locations and picking stations, the static map will continue to change. Traditional AGVs usually work in a static environment and do not need to consider real-time changing environments and complex space constraints.

[0003] With the continuous development of the goods-to-person system, its application in various industries is becoming more and more extensive, which also brings new challenges to AGV path planning. The environment in the goods-to-person system is highly dynamic, and AGV needs to continuously plan its path based on real-time tasks and environmental conditions. Due to the narrow storage corridors in the system, AGVs must accurately avoid other vehicles during transportation, otherwise it is easy to cause congestion or collisions. In addition, although traditional centralized path planning methods are effective in some cases, they are usually only applicable to single path planning, and it is difficult to cope with the complex and changeable actual application requirements in the system, especially when large-scale AGVs are deployed, and their scalability and flexibility are poor.

[0004] To address these challenges, existing methods usually adopt a two-stage path planning, that is, first planning a path for each AGV separately, and then performing conflict resolution. Common path planning algorithms such as breadth-first search (BFS), Dijkstra, and A* are usually applied to the path planning of a single AGV. However, with multiple AGVs operating simultaneously, the path conflict problem becomes more and more complicated, and conflict resolution technology becomes a key issue in the goods-to-person system.

[0005] Traditional conflict resolution methods include time window conflict detection and priority-based path avoidance, but most of these methods rely on centralized computing and cannot provide real-time path adjustment in dynamic and high-density environments. In order to improve the efficiency and reliability of path planning, reinforcement learning-based methods have gradually attracted attention in recent years, especially in distributed path planning. The decentralized idea of ​​reinforcement learning enables each AGV to autonomously plan a path based on local observations and current environmental status without relying on global information, thereby achieving more flexible and scalable multi-AGV path planning. Multi-Agent Path Finding (MAPF) is an important research direction for solving the path conflict problem in multi-AGV path planning.

[0006] Although existing distributed reinforcement learning methods have made some progress in multi-AGV path planning, most methods still have limitations. For example, learning-based path planning methods often rely on a single global path map and lack adaptability to the complex dynamic environment of the goods-to-person system. At the same time, since the corridor layout in the system is often narrow and highly dense, existing methods cannot effectively avoid AGV congestion when processing path planning, which in turn affects the throughput and computational efficiency of the overall system. Therefore, for the multi-AGV path planning problem in RMFS, the existing technology urgently needs a new solution to achieve an organic combination of global guidance and local planning to overcome the limitations of traditional methods. Summary of the invention

[0007] In view of the above situation, the present invention provides a multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning to solve the distributed conflict-free path planning problem of multiple AGVs in the goods-to-person system, and provide an efficient and scalable solution for the operation planning of warehousing logistics systems.

[0008] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0009] The multi-AGV path planning method for the goods-to-person system based on deep reinforcement learning includes the following steps:

[0010] Step 1: According to the static structure layout of the goods-to-person system pre-deployed by the engineering staff, an improved A* heuristic algorithm based on a non-uniform penalty mechanism is used to plan a global guidance path for multiple AGVs in the system; the non-uniform penalty mechanism includes a fixed penalty item and a real-time dynamic penalty item, wherein the fixed penalty item is based on the Dijkstra algorithm to calculate the average path length from the grid to the unoccupied area, and the real-time dynamic penalty item is dynamically accumulated according to the AGV path history;

[0011] Step 2: Construct the surrounding grid environment based on the point cloud data obtained by the laser radar on each AGV, and convert the real-time scanned map into grid map information; the grid map information at each time step includes local static obstacle information (such as storage station shelves and picking stations), as well as the positions of other AGVs in the field of view;

[0012] Step 3: Based on the extracted grid map information of the goods-to-person system, a local observation state space matrix is ​​constructed, which includes three map matrices, namely, a static obstacle map matrix, a local AGV map matrix, and a global guidance path map matrix;

[0013] Step 4: Use the asynchronous proximal policy optimization (APPO) algorithm to train the local path planner, fuse the time series observation data through the long short-term memory (LSTM) network, output the collision avoidance action strategy and imitate the global guidance path;

[0014] Step 5: For the storage units with movable shelves in the storage area, the fixed penalty is set to +1, and the real-time dynamic penalty is set to zero. That is, for the goods-to-person system, the grid fixed penalty of the storage area containing movable shelves is set to +1, and the real-time penalty is not calculated to avoid interference with path planning.

[0015] Furthermore, in step 1, in order to better disperse the initial path of the AGV and avoid congestion in narrow warehouse passages and high-traffic areas, a non-uniform penalty mechanism is used to adjust the path; the non-uniform penalty mechanism includes a fixed penalty and a real-time dynamic penalty, which is expressed as:

[0016] Penalty(t,v)=P fixed (v)+P real-time (t,v)

[0017] Where P fixed (v) represents a fixed penalty, P real-time (t,v) represents the real-time penalty.

[0018] In addition, when an AGV reaches its target point, the system's real-time penalty will be reset to prevent a grid from being too penalized in real time and causing all AGVs to not choose this grid as a waypoint.

[0019] Furthermore, the fixed penalty is the average path length from each grid to the unoccupied area calculated based on the Dijkstra algorithm:

[0020]

[0021] By calculating the average path length L from each grid v to all unoccupied areas average (v),determine the access difficulty of the grid, thereby assigning a fixed penalty value to each grid.

[0022] Specifically, the fixed penalty value P fixed (v) is calculated by the following formula:

[0023]

[0024] Where, L average (v) represents the average path length from grid v to all unoccupied areas. The larger L average The (v) value indicates that the grid is difficult to reach, thus giving the grid a higher fixed penalty, and the AGV is more inclined to avoid these grids in path planning;

[0025] Specifically, based on the static structural layout of the cargo-to-person system, the Dijkstra algorithm is first used to calculate the average path length L from each grid to all unoccupied areas (such as idle track grids, waiting area grids, etc.) average (v); This calculation assigns a fixed penalty value P to each grid fixed (v) indicates the difficulty of selecting the grid as a path node.

[0026] Furthermore, the real-time penalty P real-time (t,v) is the real-time path update based on AGV; the real-time penalty increases with the number of times AGV passes through, and is calculated by the following formula:

[0027]

[0028] Where AGVGrid(t,v) is a binary function used to determine whether the AGV is located in the grid v at time step t.

[0029] In the goods-to-person access system, AGVs are generally divided into three states: going to the target shelf (state 1), transporting the Pod to the picking station (state 2), and returning the Pod (state 3). In the present invention, for the global guidance module, the AGV in state 1 regards the environment as an obstacle-free environment and directly plans, and the AGVs in states 2 and 3 only regard the storage unit and the picking station as obstacles for global guidance planning.

[0030] If the AGV is located in the grid, AGVGrid(t,v)=1; otherwise, AGVGrid(t,v)=0; in addition, when an AGV reaches the target grid, the real-time penalties of all grids in the system will be reset.

[0031] Furthermore, combined with the real-time penalty value P real-time(t,v), the penalty value will be updated according to the real-time path of the AGV. The real-time penalty reflects the frequency of a grid being occupied in the past path planning process. Frequently passed grids will be given higher penalties to reduce the AGV's tendency to this area.

[0032] Furthermore, the A* algorithm is used to combine the current actual penalty and the heuristic estimated penalty to calculate a global guidance path for each AGV in the system. The AGV in state 1 directly plans the global environment as an obstacle-free environment, that is, it can freely shuttle under the shelves. The AGVs in states 2 and 3 regard the picking stations as static obstacles for global guidance planning. By adopting a non-uniform penalty mechanism, the AGV not only considers the impact of static obstacles when selecting a path, but also gives priority to avoiding areas that have already been congested, so that the AGV path is more evenly distributed, thereby improving the throughput of the entire system.

[0033] Furthermore, in step 3, the local observation state space matrix is ​​constructed, including the static obstacle map matrix, the local AGV map matrix, and the global guide path map matrix. According to the data extracted from the lidar point cloud in step 2, three grid map matrix data are formed, and the static obstacles, local AGVs, and global guide paths are encoded as -1, +1, and +2, respectively. This encoding is for states 2 and 3, that is, the AGV carrying the Pod. The AGV in state 1 has no static obstacles when encoding.

[0034] Specifically, the static obstacle map matrix encodes static obstacles as -1, idle grids as 0, and state 1, that is, unloaded AGVs, all as 0; the local AGV map matrix encodes the AGVs within the radar range and the grids where the AGV is located as +1, and the remaining grids as 0; the global guidance path map matrix encodes the guidance path grids within the detection range calculated in step 1 as +2, and the remaining grids as 0.

[0035] Furthermore, in step 4, the asynchronous proximal policy optimization (APPO) algorithm is based on the proximal policy optimization (PPO) algorithm. A local path planner in a reinforcement learning framework is constructed for each AGV. By updating the policy network and the value network, the AGV path planning strategy is optimized, so that the AGV can complete the path planning task efficiently and without collision.

[0036] The path planner uses the local observation data matrix of the AGV to generate a collision avoidance path and simultaneously imitates the global guidance path to ensure that the AGV follows the global strategy of the entire path planning.

[0037] Furthermore, in order to optimize the path planning strategy, the following reward function is adopted:

[0038]

[0039] In the formula, p i (t+1) is the location information of AGV at time t+1, W i is the target path point in the global guidance path. When the AGV reaches a target node in the global path, it will receive a reward of +1.

[0040] Furthermore, the objective function of PPO is as follows:

[0041]

[0042] In the formula, r t (θ) is the probability ratio between the new and old strategies, defined as:

[0043]

[0044] Furthermore, the behavior of the AGV is generated by the policy network, which decides actions based on its local observations and historical states (via LSTM). Each time the AGV selects an action based on the current state (including environmental information and historical paths).

[0045] Furthermore, the local path planner processes the local observation sequence through a long short-term memory (LSTM) network, uses historical information to make path decisions, and enhances the adaptability of the AGV to dynamic environmental changes.

[0046] Furthermore, the long short-term memory (LSTM) network consists of three parts: a spatial encoder, an LSTM module, and an action decoder; the spatial encoder extracts local environmental features, the LSTM module considers temporal information, and the action encoder generates action strategies and performs action sampling through a Softmax activation function.

[0047] Furthermore, a method for constructing a training data set for a local path planner includes constructing a data set containing 200 maps, 25% of which are random maps, 25% are maze maps, and 50% are warehouse style maps.

[0048] Preferably, random maps and maze maps are generated by the PRIMAL2 project method, with an obstacle density of 20% or 40%, and contain 64 or 128 AGVs and 8 picking stations; warehouse-style maps are generated by randomly adding paths to ensure a target density of 50%.

[0049] Furthermore, the action behaviors of each AGV are generated by the policy network, and the action is decided according to its local observation state and historical state (through LSTM); each AGV selects an action according to the current state (including environmental information and historical path), and the shared policy network is deployed on each AGV, and each step is repeatedly executed in a distributed manner to complete all picking tasks of the goods-to-person system.

[0050] The present invention also includes other components that enable it to be used normally, which are all conventional means in the field. In addition, devices or components not limited in the present invention, such as: A* heuristic algorithm, PPO algorithm, Dijkstra algorithm, LSTM network, etc., all adopt the existing technology in the field.

[0051] The beneficial effects of the present invention are as follows:

[0052] 1. This paper constructs a new hierarchical planning framework that combines heuristic global guidance with reinforcement learning for multi-AGV path planning in goods-to-person systems. This framework solves the limitation of existing methods that often separate path planning from conflict resolution, thereby effectively alleviating the problem of excessive congestion.

[0053] 2. This paper designs a novel non-uniform penalty heuristic method specifically for global path guidance in goods-to-person systems, especially for long corridor layouts and high-density storage environments. The heuristic method combines fixed penalties and real-time penalties to achieve a more uniform distribution of initial paths in the system, thereby reducing congestion in narrow corridors and improving overall system throughput.

[0054] 3. This paper develops an efficient local reinforcement learning path planner that combines asynchronous proximal policy optimization (APPO) and long short-term memory (LSTM) technology. APPO supports fully decentralized online lifelong path planning for multiple AGVs, while LSTM effectively avoids potential collisions and avoids unnecessary detours by utilizing local observation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the structure and path planning of the existing goods-to-person system targeted by the present invention in the background technology.

[0056] Figure 2 It is a schematic diagram for explaining the heuristic algorithm path guidance method for the goods-to-person system in the present invention.

[0057] Figure 3 This is a flow chart of the multi-AGV path planning method for the goods-to-person system based on deep reinforcement learning in the present invention.

[0058] Figure 4 Schematic diagram of the algorithm for constructing a deep reinforcement learning map dataset for the goods-to-person system in the present invention. DETAILED DESCRIPTION

[0059] In order to enable people in the technical research field to better understand the scheme of the present invention, the technical scheme of the present invention will be clearly and completely described and illustrated in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Example

[0061] like Figure 1-4 As shown, the present invention provides a multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning, which uses an improved heuristic search algorithm to globally guide the AGV, and then uses the deep reinforcement learning network to perform local collision avoidance and obstacle avoidance, specifically comprising the following steps:

[0062] Step 1: Based on the static structure layout of the goods-to-person system pre-deployed by the engineering staff, an improved A* heuristic algorithm based on a non-uniform penalty mechanism is used to plan a global guidance path for the AGV that only considers static obstacles but not mutual collision avoidance;

[0063] The non-uniform penalty mechanism includes a fixed penalty term and a real-time dynamic penalty term, wherein the fixed penalty term calculates the average path length from the grid to the unoccupied area based on the Dijkstra algorithm, and the real-time dynamic penalty term is dynamically accumulated according to the AGV path history;

[0064] Step 1.1, set the operation mode of AGV in the grid map of the goods-to-person system. AGV can move in four directions around the grid, and can also choose to stay in the original grid.

[0065] Step 1.2: The fixed penalty value for each grid is pre-calculated before the AGV operation begins. First, the Dijkstra algorithm is used to calculate the average path length of each grid in the system to all grids that are not occupied by Pods and picking stations (the path length from one grid to the adjacent grid is counted as 1):

[0066]

[0067] Where V empty (v) represents the grid set not occupied by Pods and picking stations, |V empty (v)| is the number of elements in the set. Intuitively, L average (v) represents the average path length from grid v to other grids v′, and the smaller L average (v) indicates that it is easier for AGV to reach other grids from grid v, which means that the grid is more likely to become a waypoint for AGV and also a potential congestion blocking node.

[0068] Furthermore, if the AGV chooses to move to this grid, the grid will be subject to a larger penalty, and the fixed penalty can be defined as:

[0069]

[0070] For the fixed penalty, when the grid L average (v) is the largest, that is, the grid is the grid with the lowest probability of being used in the AGV path planning process, P fixed (v) = 1. For other grids that are more frequently used in path planning, P fixed (v)>1. The fixed penalty for each grid does not change unless the number or structure of Pods and picking stations changes. The AGV will tend to choose a grid with a smaller fixed penalty to move.

[0071] Step 1.3: The real-time penalty is based on the path nodes of all AGVs and is dynamically updated during training or testing. The real-time penalty can be defined as:

[0072]

[0073] Where AGVGrid(t,v) is a function about discrete time, indicating whether the grid v is occupied by AGV at discrete time t, and is defined as follows:

[0074]

[0075] In the formula, AGVGrid(t,v) represents the number of times grid v is occupied by AGV from the beginning of this round to time t. This value reflects the frequency of use of the grid; if a grid is frequently occupied by AGV, it is a high-traffic area, so increasing the real-time penalty of the grid can effectively avoid more congestion.

[0076] Furthermore, the system will reset the real-time penalty values ​​of all grids when the AGV reaches the target location. The unlimited growth of the real-time penalty may cause the penalty of some grids to be too high, causing the AGV to completely avoid these grids in subsequent decisions.

[0077] Step 1.4: According to Step 1.2 and Step 1.3, the movement penalty of AGV is uneven, that is, the path length from one grid to the adjacent grid is not equal. The total penalty consists of two parts, fixed penalty and real-time penalty:

[0078] Penalty(t,v)=P fixed (v)+P real-time (t,v) (5)

[0079] In the formula, the fixed penalty P fixed(v) It depends on the layout structure of the warehouse of the goods-to-person system, which is calculated before deployment, and the real-time penalty P real-time (t,v) depends on the motion paths of all current AGVs and changes over time.

[0080] Furthermore, the A* algorithm is used to calculate the global guidance path of each AGV in the system based on the sum of the actual penalty value (g value) of the path and the estimated penalty value (h value) to the target point.

[0081] Furthermore, for the empty AGV heading to the target shelf in state 1, the global environment is considered to be obstacle-free when using the improved A* for planning, that is, it can shuttle under the shelf; for the AGV loaded with shelves in states 2 and 3, the storage station and picking station need to be regarded as static obstacles for global guidance planning.

[0082] Step 2: Construct the surrounding grid environment based on the point cloud data obtained by the laser radar on each AGV, and convert the real-time scanned map into grid map information; the grid map information at each time step includes local static obstacle information (such as storage station shelves and picking stations), as well as the positions of other AGVs in the field of view;

[0083] Furthermore, if Figure 2 As shown in the figure, the upper right corner of the grid represents the fixed penalty, and the lower right corner represents the real-time penalty of the grid. Each time the AGV passes by, the real-time penalty increases by 1. The depth of the color in the figure represents the size of the fixed penalty, and the darker the color, the greater the penalty.

[0084] Furthermore, in order to prevent Figure 1 As shown, the AGV in mode 1 has an impact on system congestion. The present invention sets the grid fixed penalty of the storage area containing the Pod to +1, and does not calculate the real-time penalty, ensuring that the AGV in mode 1 can pass under the shelf first, that is, the AGV with a handling task, that is, Figure 2 AGVs in Mode 2 and 3 can free up planning space, which is beneficial to improving system throughput.

[0085] Step 3: Based on the extracted grid map information of the goods-to-person system, a local observation state space matrix is ​​constructed, which includes three map matrices, namely, a static obstacle map matrix, a local AGV map matrix, and a global guidance path map matrix;

[0086] According to the point cloud data acquired by the laser radar in step 2, the data is converted into a grid map. The present invention encodes the grid map into an observation matrix.

[0087] Specifically, by performing a local scan of the environment of each AGV, a detection radius r with the AGV as the center is obtained. FOV The square area, area (2rFOV +1)×(2r FOV +1), such as Figure 3 As shown. FOV When =3, the AGV’s field of view is a 7×7 grid area.

[0088] Furthermore, the local observation space of each AGV It consists of three different components: static obstacle map matrix Indicates the location of storage stations and picking stations, coded as -1; local AGV (dynamic obstacle) map matrix Indicates the location of other AGVs, coded as +1; Global guidance path map matrix The global guidance path generated based on the method of step 1 of the example of the present invention is coded as +2, which represents the guidance path within the field of view of the AGV.

[0089] Furthermore, for the empty AGV heading to the target shelf state 1, in the static obstacle map matrix When encoding, static obstacles are ignored, that is, it is possible to shuttle under the shelves of the storage station, and all are encoded as 0.

[0090] By encoding in the manner described, the method of the present invention ensures that the AGV can focus on the current local planning and decision-making tasks while following the global guidance path during the autonomous path planning process.

[0091] In the present invention, at each time step t, the local observation space includes not only the observation at the current moment, but also the observation data of the previous four time steps, thereby forming a time series {O t-4 ,O t-3 ,O t-2 ,O t-1 ,O t}.

[0092] This time series provides the AGV with historical observation information, enabling it to recognize and adapt to dynamic changes and trends in the environment.

[0093] Step 4: Use the asynchronous proximal policy optimization (APPO) algorithm to train the local path planner, fuse the time series observation data through the long short-term memory (LSTM) network, output the collision avoidance action strategy and imitate the global guidance path;

[0094] Step 4.1: The guidance path generated by the method described in step 1 is used as the benchmark for AGV path planning, and a reward function is set. At each time step t, when the AGV reaches a path point on the global guidance path, it will receive a reward value of +1.

[0095]

[0096] In the formula, p i (t+1) represents the position of the AGV at time step t+1, W i The reward mechanism of the present invention encourages AGV to closely follow the global guidance path, and the reward structure is simple, which improves the training efficiency. If there is no path point in the global guidance path, the heuristic guidance path planner in step 1 will be reactivated to generate a new global guidance path. This mechanism of the present invention ensures that the AGV can continue to obtain effective navigation guidance in the goods-to-person system.

[0097] Step 4.2: Local observation space synthesized according to step 3 And the historical observation space {O t-4 ,O t-3 ,O t-2 ,O t-1 ,O t}, the present invention converts it into an input tensor of 7×7×3×5. Then the features of the environment state are extracted through the convolution layer and the residual block, and feature flattening is performed, which is the spatial encoder. The flattened features are input into the LSTM module, so that the AGV can retain the memory of historical observations and incorporate spatiotemporal information into the decision-making process.

[0098] Step 4.3: Action decoding is performed on the output of LSTM, which is composed of an Actor-Critic network. Actor network: It consists of two fully connected layers, and the ReLU activation function is applied after the first layer. The Actor network generates the probability distribution of possible actions through the Softmax activation function to achieve action decoding (Action Decoder); Critic network: It also consists of two fully connected layers, and the ReLU activation function is applied after the first layer. The Critic network estimates the value function V t , represents the expected cumulative reward of the current state of the AGV.

[0099] All AGVs share a unified policy network and use the APPO algorithm to optimize the policy and maximize the expected return. The strategy update is adjusted by the clipping parameter ε to ensure the stability of training.

[0100] In step 4, deep reinforcement learning training is involved. The present invention provides a dataset construction method. To solve the multi-AGV path planning problem in the goods-to-person system, the present invention constructs a diversified dataset containing 200 64×64 maps, of which 25% are random maps, 25% are maze maps, and 50% are warehouse-style maps. The warehouse map simulates the layout of a typical goods-to-person system and supports storage and retrieval task training; random and maze maps introduce complex topological structures to enhance the adaptability of the model in complex environments.

[0101] The dataset generates random and maze maps with an obstacle density of 20% or 40% and contains 64 or 128 AGVs and 8 picking stations. The warehouse map generator randomly adds paths to ensure that the target density is 50%. The specific algorithm is shown in Figure 4 The dataset provides a diverse environment for model training, ensuring its robustness and efficiency in complex layouts.

[0102] Step 5: For the storage units with movable shelves in the storage area, the fixed penalty is set to +1, and the real-time dynamic penalty is set to zero. That is, for the goods-to-person system, the grid fixed penalty of the storage area containing movable shelves is set to +1, and the real-time penalty is not calculated to avoid interference with path planning.

[0103] Furthermore, in step 1, in order to better disperse the initial path of the AGV and avoid congestion in narrow warehouse passages and high-traffic areas, a non-uniform penalty mechanism is used to adjust the path; the non-uniform penalty mechanism includes a fixed penalty and a real-time dynamic penalty, which is expressed as:

[0104] Penalty(t,v)=P fixed (v)+P real-time (t,v)

[0105] Where P fixed (v) represents a fixed penalty, P real-time (t,v) represents the real-time penalty.

[0106] In addition, when an AGV reaches its target point, the system's real-time penalty will be reset to prevent a grid from being too penalized in real time and causing all AGVs to not choose this grid as a waypoint.

[0107] Furthermore, the fixed penalty is the average path length from each grid to the unoccupied area calculated based on the Dijkstra algorithm:

[0108]

[0109] By calculating the average path length L from each grid v to all unoccupied areas average(v),determine the access difficulty of the grid, thereby assigning a fixed penalty value to each grid.

[0110] Specifically, the fixed penalty value P fixed (v) is calculated by the following formula:

[0111]

[0112] Where, L average (v) represents the average path length from grid v to all unoccupied areas. The larger L average The (v) value indicates that the grid is difficult to reach, thus giving the grid a higher fixed penalty, and the AGV is more inclined to avoid these grids in path planning;

[0113] Specifically, based on the static structural layout of the cargo-to-person system, the Dijkstra algorithm is first used to calculate the average path length L from each grid to all unoccupied areas (such as idle track grids, waiting area grids, etc.) average (v); This calculation assigns a fixed penalty value P to each grid fixed (v) indicates the difficulty of selecting the grid as a path node.

[0114] Furthermore, the real-time penalty P real-time (t,v) is the real-time path update based on AGV; the real-time penalty increases with the number of times AGV passes through, and is calculated by the following formula:

[0115]

[0116] Where AGVGrid(t,v) is a binary function used to determine whether the AGV is located in the grid v at time step t.

[0117] If the AGV is located in the grid, AGVGrid(t,v)=1; otherwise, AGVGrid(t,v)=0; in addition, when an AGV reaches the target grid, the real-time penalties of all grids in the system will be reset.

[0118] Furthermore, combined with the real-time penalty value P real-time (t,v), the penalty value will be updated according to the real-time path of the AGV. The real-time penalty reflects the frequency of a grid being occupied in the past path planning process. Frequently passed grids will be given higher penalties to reduce the AGV's tendency to this area.

[0119] Furthermore, the A* algorithm is used to combine the current actual penalty and the heuristic estimated penalty to calculate the global guidance path for each AGV in the system. The AGV in state 1 is planned for an environment without static obstacles. By adopting a non-uniform penalty mechanism, the AGV not only considers the impact of static obstacles when selecting a path, but also gives priority to avoiding areas that have already been congested, so that the AGV path is more evenly distributed, thereby improving the throughput of the entire system.

[0120] Furthermore, in step 3, a local observation state space matrix is ​​constructed including a static obstacle map matrix, a local AGV map matrix and a global guidance path map matrix. According to the data extracted from the lidar point cloud in step 2, three types of raster map matrix data are formed, and the static obstacles, local AGVs and global guidance paths are encoded as -1, +1 and +2, respectively.

[0121] Specifically, the static obstacle map matrix encodes static obstacles as -1, idle grids as 0, and state 1, that is, unloaded AGVs, all as 0; the local AGV map matrix encodes the AGVs within the radar range and the grids where the AGV is located as +1, and the remaining grids as 0; the global guidance path map matrix encodes the guidance path grids within the detection range calculated in step 1 as +2, and the remaining grids as 0.

[0122] Furthermore, in step 4, the asynchronous proximal policy optimization (APPO) algorithm is based on the proximal policy optimization (PPO) algorithm. A local path planner in a reinforcement learning framework is constructed for each AGV. By updating the policy network and the value network, the AGV path planning strategy is optimized, so that the AGV can complete the path planning task efficiently and without collision.

[0123] The path planner uses the local observation data matrix of the AGV to generate a collision avoidance path and simultaneously imitates the global guidance path to ensure that the AGV follows the global strategy of the entire path planning.

[0124] Furthermore, in order to optimize the path planning strategy, the following reward function is adopted:

[0125]

[0126] In the formula, p i (t+1) is the location information of AGV at time t+1, W i is the target path point in the global guidance path. When the AGV reaches a target node in the global path, it will receive a reward of +1.

[0127] Furthermore, the objective function of PPO is as follows:

[0128]

[0129] In the formula, r t (θ) is the probability ratio between the new and old strategies, defined as:

[0130]

[0131] Furthermore, the behavior of the AGV is generated by the policy network, which decides actions based on its local observations and historical states (via LSTM). Each time the AGV selects an action based on the current state (including environmental information and historical paths).

[0132] Furthermore, the local path planner processes the local observation sequence through a long short-term memory (LSTM) network, uses historical information to make path decisions, and enhances the adaptability of the AGV to dynamic environmental changes.

[0133] Furthermore, the long short-term memory (LSTM) network consists of three parts: a spatial encoder, an LSTM module, and an action decoder; the spatial encoder extracts local environmental features, the LSTM module considers temporal information, and the action encoder generates action strategies and performs action sampling through a Softmax activation function.

[0134] Furthermore, a method for constructing a training data set for a local path planner includes constructing a data set containing 200 maps, 25% of which are random maps, 25% are maze maps, and 50% are warehouse style maps.

[0135] Furthermore, the action behaviors of each AGV are generated by the policy network, and the action is decided according to its local observation state and historical state (through LSTM); each AGV selects an action according to the current state (including environmental information and historical path), and the shared policy network is deployed on each AGV, and each step is repeatedly executed in a distributed manner to complete all picking tasks of the goods-to-person system.

[0136] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. Without departing from the scope and spirit of the described embodiments, many modifications and changes are obvious to ordinary technicians in this technical field. Any technical deformation made within the spirit and principles of the present invention falls within the protection scope of the present invention.

Claims

1. A multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: According to the pre-deployed static structure layout of the goods-to-person system, an improved A* heuristic algorithm based on a non-uniform penalty mechanism is used to plan a global guidance path for the AGV; Step 2: Construct the surrounding grid environment based on the point cloud data obtained by the laser radar carried by each AGV, and convert the real-time scanned map into grid map information; Step 3: Construct a local observation state space matrix based on the extracted goods-to-person system grid map information; Step 4: Use the asynchronous proximal strategy optimization algorithm to train the local path planner, fuse the time series observation data through the long short-term memory (LSTM) network, output the collision avoidance action strategy and imitate the global guidance path; Step 5: For the movable shelf storage units in the storage area of ​​the goods-to-person system, the fixed penalty is set to +1, and the real-time dynamic penalty is set to zero.

2. The method for multi-AGV path planning in a goods-to-person system based on deep reinforcement learning according to claim 1 is characterized in that: In step 1, the non-uniform penalty mechanism includes fixed penalty and real-time dynamic penalty, which can be expressed as: Penalty(t,v)=P fixed (v)+P real-time (t,v) Where P fixed (v) represents a fixed penalty, P real-time (t,v) represents the real-time penalty.

3. The method for multi-AGV path planning in a goods-to-person system based on deep reinforcement learning according to claim 2 is characterized in that: The fixed penalty is the average path length from each grid to the unoccupied area calculated based on the Dijkstra algorithm: The access difficulty of the grid is determined by calculating the average path length from each grid v to all unoccupied areas, thereby assigning a fixed penalty value to each grid; The calculation formula for the fixed penalty value is: Where, L average (v) represents the average path length from grid v to all unoccupied areas. The larger the fixed penalty value, the more difficult it is to reach the corresponding grid. AGV is more inclined to avoid grids with large fixed penalty values ​​in path planning.

4. The method for multi-AGV path planning in a goods-to-person system based on deep reinforcement learning according to claim 2 is characterized in that: Real-time penalty P real-time (t,v) is based on the real-time path update of AGV; The real-time penalty increases as the number of AGV passes by, and is calculated using the following formula: Where AGVGrid(t,v) is a binary function used to determine whether the AGV is located in the grid v at time step t.

5. The method for multi-AGV path planning in a goods-to-person system based on deep reinforcement learning according to claim 1 is characterized in that: In step 3, a local observation state space matrix is ​​constructed including a static obstacle map matrix, a local AGV map matrix, and a global guidance path map matrix, and the static obstacles, local AGVs, and global guidance paths are encoded as -1, +1, and +2, respectively.

6. The method for multi-AGV path planning in a goods-to-person system based on deep reinforcement learning according to claim 1, characterized in that: In step 4, the asynchronous proximal policy optimization (APPO) algorithm is based on the proximal policy optimization (PPO) algorithm. A local path planner in a reinforcement learning framework is constructed for each AGV, and the AGV path planning strategy is optimized by updating the policy network and the value network.

7. The method for multi-AGV path planning in a goods-to-person system based on deep reinforcement learning according to claim 6 is characterized in that: The local path planner processes local observation sequences through the LSTM network, uses historical information to make path decisions, and enhances the AGV's ability to adapt to dynamic environmental changes.

8. The method for multi-AGV path planning in a goods-to-person system based on deep reinforcement learning according to claim 7 is characterized in that: The LSTM network consists of three parts: a spatial encoder, an LSTM module, and an action decoder; the spatial encoder extracts local environmental features, the LSTM module considers temporal information, and the action encoder generates action strategies and performs action sampling through the Softmax activation function.

9. The method for multi-AGV path planning in a goods-to-person system based on deep reinforcement learning according to claim 8 is characterized in that: A method for constructing a training data set for a local path planner includes constructing a data set containing 200 maps, 25% of which are random maps, 25% are maze maps, and 50% are warehouse style maps.

10. The method for multi-AGV path planning in a goods-to-person system based on deep reinforcement learning according to claim 9, characterized in that: The action behaviors of each AGV are generated by the policy network, and the action is decided based on its local observation state and historical state. Each AGV selects an action according to the current state, and the shared policy network is deployed on each AGV to repeatedly execute each step in a distributed manner to complete all picking tasks of the goods-to-person system.

Images