Multi-agv path planning method for goods-to-person system based on deep reinforcement learning

Abstract

The application relates to a kind of multi-AGV path planning method of goods-to-person system based on deep reinforcement learning in the field of computer, fusion improved A* global guidance and double-layer planning framework of local reinforcement learning, global layer uses non-uniform penalty heuristic strategy, and fixed penalty term calculated by Dijkstra algorithm and real-time dynamic penalty term based on historical path are cooperated to optimize path dispersion;Local layer designs asynchronous proximal policy optimization (APPO) algorithm, combines LSTM network to process multi-channel observation state (static obstacle, AGV position, global path encoding) of continuous 5 time steps, and outputs 5 degrees of freedom action strategy.For the area of movable shelf (Pod) in warehouse, a fixed penalty exemption mechanism is set to avoid invalid avoidance, improve task success rate and throughput, and reduce computing load, providing an efficient and scalable solution for solving the multi-AGV path planning problem of goods-to-person system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, in particular to the field of multi-agent, and more specifically to a multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning. Background Art

[0002] With the development of automation technology, the Goods-to-Person System (Robotic Mobile Fulfillment System, RMFS) has been widely used in the fields of manufacturing and logistics. The Goods-to-Person System uses automated guided vehicles (AGVs) to efficiently transport and store goods in warehouses, which not only greatly reduces the intensity of manual labor, but also improves the accuracy and efficiency of order processing. However, the operation of AGVs in the Goods-to-Person System is highly complex, especially in multi-AGV path planning, which faces many technical challenges. First of all, there is a significant difference between the path planning environment of AGVs in the Goods-to-Person System and traditional AGV applications. The AGVs in the Goods-to-Person System need to be transported under the shelves (see Appendix Figure 1 ), and as the shelves move between storage locations and picking stations, the static map will continue to change. Traditional AGVs usually work in a static environment and do not need to consider real-time changing environments and complex space constraints.

[0003] With the continuous development of goods-to-person systems, their application in various industries is becoming more and more extensive, which also brings new challenges to AGV path planning. The environment in the goods-to-person system is highly dynamic, and AGVs need to continuously plan their paths based on real-time tasks and environmental conditions. Due to the narrow storage corridors in the system, AGVs must accurately avoid other vehicles during transportation, otherwise it is easy to cause congestion or collisions. In addition, although traditional centralized path planning methods are effective in some cases, they are usually only applicable to single path planning and cannot cope with the complex and changing actual application requirements of the system. In particular, their scalability and flexibility are poor when deploying large-scale AGVs.

[0004] To address these challenges, existing methods typically employ a two-stage path planning approach: first, planning a path for each AGV individually, followed by conflict resolution. Common path planning algorithms, such as breadth-first search (BFS), Dijkstra, and A*, are typically applied to single-AGV path planning. However, with multiple AGVs operating simultaneously, path conflicts become increasingly complex, making conflict resolution a critical issue in goods-to-person systems.

[0005] Traditional conflict resolution methods include time window conflict detection and priority-based path avoidance, but most of these methods rely on centralized computing and cannot provide real-time path adjustments in dynamic and high-density environments. In order to improve the efficiency and reliability of path planning, reinforcement learning-based methods have gradually attracted attention in recent years, especially in distributed path planning. The decentralized idea of ​​reinforcement learning enables each AGV to autonomously plan its path based on local observations and current environmental status without relying on global information, thereby achieving more flexible and scalable multi-AGV path planning. Multi-Agent Path Finding (MAPF) is an important research direction for solving the path conflict problem in multi-AGV path planning.

[0006] Although existing distributed reinforcement learning methods have made some progress in multi-AGV path planning, most methods still have limitations. For example, learning-based path planning methods often rely on a single global path map and lack adaptability to the complex dynamic environment of goods-to-person systems. At the same time, because the corridor layout in the system is often narrow and highly dense, existing methods cannot effectively avoid AGV congestion when handling path planning, which in turn affects the throughput and computational efficiency of the overall system. Therefore, for the multi-AGV path planning problem in RMFS, existing technologies urgently need a new solution to achieve an organic combination of global guidance and local planning to overcome the limitations of traditional methods. Summary of the Invention

[0007] In response to the above situation, the present invention provides a multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning to solve the distributed conflict-free path planning problem of multiple AGVs in the goods-to-person system, and provide an efficient and scalable solution for the operation planning of the warehousing logistics system.

[0008] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0009] The multi-AGV path planning method for the goods-to-person system based on deep reinforcement learning includes the following steps:

[0010] Step 1: Based on the static structure layout of the goods-to-person system pre-deployed by engineers, a modified A* heuristic algorithm with a non-uniform penalty mechanism is used to plan a global guidance path for multiple AGVs within the system. The non-uniform penalty mechanism includes a fixed penalty term and a real-time dynamic penalty term. The fixed penalty term uses the Dijkstra algorithm to calculate the average path length from the grid to the unoccupied area, and the real-time dynamic penalty term is dynamically accumulated based on the AGV path history.

[0011] Step 2: Construct a grid environment based on the point cloud data acquired by the LiDAR onboard each AGV, converting the real-time scanned map into grid map information. The grid map information at each time step includes local static obstacle information (such as storage station shelves and picking stations), as well as the positions of other AGVs within the field of view.

[0012] Step 3: Based on the extracted goods-to-person system grid map information, a local observation state space matrix is ​​constructed, which contains three map matrices: a static obstacle map matrix, a local AGV map matrix, and a global guidance path map matrix;

[0013] Step 4: Use the asynchronous proximal policy optimization (APPO) algorithm to train the local path planner, fuse the time series observation data through the long short-term memory (LSTM) network, output the collision avoidance action strategy and imitate the global guidance path;

[0014] Step 5: For the storage units with movable shelves in the storage area, the fixed penalty is set to +1, and the real-time dynamic penalty is set to zero. That is, for the goods-to-person system, the grid fixed penalty of the storage area containing movable shelves is set to +1, and the real-time penalty is not calculated to avoid interference with path planning.

[0015] Furthermore, in step 1, in order to better disperse the initial path of the AGV and avoid congestion in narrow warehouse aisles and high-traffic areas, a non-uniform penalty mechanism is used to adjust the path. The non-uniform penalty mechanism includes a fixed penalty and a real-time dynamic penalty, which is expressed as:

[0016] Penalty(t,v)=P fixed (v)+P real-time (t,v)

[0017] Where, P fixed (v) represents a fixed penalty, P real-time (t,v) represents the real-time penalty.

[0018] In addition, when an AGV reaches its destination, the system's real-time penalty will be reset to prevent a grid from being penalized too much in real time and causing all AGVs to not choose this grid as a waypoint.

[0019] Furthermore, the fixed penalty is the average path length from each grid to the unoccupied area calculated based on the Dijkstra algorithm:

[0020]

[0021] By calculating the average path length L from each grid v to all unoccupied areas average (v),Determine the access difficulty of the grid, thereby assigning a fixed penalty value to each grid.

[0022] Specifically, the fixed penalty value P fixed (v) is calculated by the following formula:

[0023]

[0024] Where, L average (v) represents the average path length from grid v to all unoccupied areas, and the larger L average The (v) value indicates that the grid is difficult to reach, thus giving the grid a higher fixed penalty. The AGV is more inclined to avoid these grids during path planning;

[0025] Specifically, based on the static structural layout of the cargo-to-person system, the Dijkstra algorithm is first used to calculate the average path length L from each grid to all unoccupied areas (such as idle track grids, waiting area grids, etc.) average (v); This calculation assigns a fixed penalty value P to each grid fixed (v) indicates the difficulty of selecting the grid as a path node.

[0026] Furthermore, the real-time penalty P real-time (t, v) is based on the real-time path update of the AGV; the real-time penalty increases with the number of times the AGV passes through, and is calculated by the following formula:

[0027]

[0028] Where AGVGrid(t,v) is a binary function used to determine whether the AGV is located in the grid v at time step t.

[0029] In a goods-to-person storage and retrieval system, AGVs typically operate in three states: heading to the target shelf (state 1), transporting a pod to a picking station (state 2), and returning the pod (state 3). In this invention, the global guidance module treats the environment in state 1 as obstacle-free and performs direct planning. AGVs in states 2 and 3 only consider the storage unit and picking station as obstacles for global guidance planning.

[0030] If the AGV is located in the grid, AGVGrid(t,v)=1; otherwise, AGVGrid(t,v)=0; in addition, when an AGV reaches the target grid, the real-time penalties of all grids in the system will be reset.

[0031] Furthermore, combined with the real-time penalty value P real-time(t, v), which will be updated according to the real-time path of the AGV. The real-time penalty reflects the frequency of a certain grid being occupied in the past path planning process, and the grid that is frequently passed through will be assigned a higher penalty to reduce the AGV's tendency to pass through this area.

[0032] Further, the A* algorithm is used to synthesize the current actual penalty and heuristic predicted penalty to calculate a global guide path for each AGV in the system. The AGV in state 1 directly plans in the global environment as an unobstructed environment, i.e., it can shuttle freely under the shelves, and the AGVs in states 2 and 3 plan a global guide path considering the picking station as a static obstacle. By adopting a non-uniform penalty mechanism, the AGV not only considers the influence of static obstacles when selecting a path, but also preferentially avoids areas that have already appeared to be congested, so that the AGV path is more evenly distributed, thereby improving the throughput of the entire system.

[0033] Further, in step 3, the local observation state space matrix includes a static obstacle map matrix, a local AGV map matrix, and a global guide path map matrix. Based on the data extracted from the laser radar point cloud in step 2, the three kinds of grid map matrix data are composed, and the static obstacle, local AGV, and global guide path are respectively coded as -1, +1, and +2. This coding is for states 2 and 3, i.e., AGVs carrying pods, and state 1 AGVs have no static obstacles when encoding.

[0034] Specifically, the static obstacle map matrix encodes the static obstacles as -1 and the idle grid as 0. State 1, i.e., the empty AGV, is all encoded as 0; the local AGV map matrix encodes the AGVs within the radar range and the grid where the AGV is located as +1, and the remaining grids are encoded as 0; the global guide path map matrix encodes the guide path grid within the detection range calculated in step 1 as +2, and the remaining grids are encoded as 0.

[0035] Further, in step 4, the asynchronous proximal policy optimization (APPO) algorithm is based on the proximal policy optimization (PPO) algorithm, and a local path planner in a reinforcement learning framework is constructed for each AGV. By updating the policy network and the value network, the optimization of the AGV path planning strategy is realized, so that the AGV can efficiently and collision-free complete the path planning task.

[0036] The path planner uses the local observation data matrix of the AGV to generate a collision avoidance path, and at the same time imitates the global guide path to ensure that the AGV follows the global strategy of the entire path planning.

[0037] Further, in order to optimize the path planning strategy, the following reward function is used:

[0038]

[0039] Where p i (t+1) is the location information of AGV at time t+1, W i is the target path point in the global guided path. When the AGV reaches a target node in the global path, it will receive a reward of +1.

[0040] Furthermore, the objective function of PPO is as follows:

[0041]

[0042] Where r t (θ) is the probability ratio between the new and old strategies, defined as:

[0043]

[0044] Furthermore, the AGV’s behavior is generated by the policy network, which decides its actions based on its local observations and historical states (via LSTM). Each time, the AGV selects an action based on its current state (including environmental information and historical paths).

[0045] Furthermore, the local path planner processes local observation sequences through a long short-term memory (LSTM) network, uses historical information to make path decisions, and enhances the AGV's adaptability to dynamic environmental changes.

[0046] Furthermore, the long short-term memory (LSTM) network consists of three parts: a spatial encoder, an LSTM module, and an action decoder; the spatial encoder extracts local environmental features, the LSTM module considers temporal information, and the action encoder generates action strategies and performs action sampling through a Softmax activation function.

[0047] Furthermore, a method for constructing a training dataset for a local path planner includes constructing a dataset containing 200 maps, 25% of which are random maps, 25% are maze maps, and 50% are warehouse-style maps.

[0048] Preferably, random maps and maze maps are generated by the PRIMAL2 project method with an obstacle density of 20% or 40% and contain 64 or 128 AGVs and 8 picking stations; warehouse-style maps are generated by randomly adding paths to ensure a target density of 50%.

[0049] Furthermore, the action behavior of each AGV is generated by the policy network, which decides on its action based on its local observation state and historical state (through LSTM); each AGV selects an action based on its current state (including environmental information and historical path), and the shared policy network is deployed on each AGV, and each step is repeatedly executed in a distributed manner to complete all picking tasks of the goods-to-person system.

[0050] The present application also includes other components that can enable its normal use, which are conventional means in the art, and in addition, the devices or components not defined in the present application, such as: A* heuristic algorithm, PPO algorithm, Dijkstra algorithm, LSTM network, etc., all use the existing technology in the art.

[0051] The beneficial effects of the present application are as follows:

[0052] 1. The present application constructs a new hierarchical planning framework, which combines heuristic global guidance and reinforcement learning, for multi-AGV path planning in a goods-to-person system. This framework solves the limitation of separating path planning and conflict resolution in existing methods, thereby effectively alleviating the problem of excessive congestion.

[0053] 2. The present application designs a new type of non-uniform penalty heuristic method, which is specifically used for global path guidance in a goods-to-person system, especially for long corridor layout and high-density storage environment. This heuristic method combines fixed penalty and real-time penalty to achieve more uniform distribution of initial paths in the system, thereby reducing congestion in narrow corridors and improving overall system throughput.

[0054] 3. The present application develops an efficient local reinforcement learning path planner, which combines asynchronous proximal policy optimization (APPO) and long short-term memory (LSTM) technology. APPO supports fully decentralized online lifelong path planning for multiple AGVs, while LSTM effectively avoids potential collisions and unnecessary detours by utilizing local observation data. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A schematic diagram of the structure and path planning of the existing goods-to-person system targeted by the present application in the background art.

[0056] Figure 2 A schematic diagram of the heuristic algorithm path guidance method for a goods-to-person system in the present application.

[0057] Figure 3 A flowchart of the multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning in the present application.

[0058] Figure 4 A schematic diagram of the deep reinforcement learning map data set construction algorithm for a goods-to-person system in the present application. DETAILED DESCRIPTION

[0059] To help those skilled in the art better understand the present invention, the following will provide a clear and complete description and illustration of the present invention's technical solution, with reference to the accompanying drawings. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by those skilled in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0060] Example

[0061] like Figure 1-4 As shown, the present invention provides a multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning. The method uses an improved heuristic search algorithm to globally guide the AGVs, and then uses the deep reinforcement learning network to perform local collision and obstacle avoidance. The method specifically includes the following steps:

[0062] Step 1: Based on the static structure layout of the goods-to-person system pre-deployed by engineers, an improved A* heuristic algorithm with a non-uniform penalty mechanism is used to plan a global guidance path for the AGV that only considers static obstacles and does not consider mutual collision avoidance;

[0063] The non-uniform penalty mechanism includes a fixed penalty term and a real-time dynamic penalty term, wherein the fixed penalty term calculates the average path length from the grid to the unoccupied area based on the Dijkstra algorithm, and the real-time dynamic penalty term is dynamically accumulated according to the AGV path history;

[0064] Step 1.1: Set the AGV's operation mode in the goods-to-person system grid map. The AGV can move in four directions around the grid, or it can choose to stay in the original grid.

[0065] Step 1.2: The fixed penalty value for each grid is pre-calculated before the AGV operation begins. First, the Dijkstra algorithm is used to calculate the average path length of each grid in the system to all grids that are not occupied by pods and picking stations (the path length from one grid to the adjacent grid is counted as 1):

[0066]

[0067] Where V empty (v) represents the grid set not occupied by Pods and picking stations, |V empty (v)| is the number of elements in the set. Intuitively, L average (v) represents the average path length from grid v to other grids v′, and the smaller L average (v) indicates that it is easier for AGV to reach other grids from grid v, which means that the grid is more likely to become a waypoint for AGV and also a potential congestion blocking node.

[0068] Furthermore, if the AGV chooses to move to this grid, the grid will be subject to a larger penalty. The fixed penalty can be defined as:

[0069]

[0070] For the fixed penalty, when the grid L average When (v) is the largest, that is, the grid is the grid with the smallest probability of use in the AGV path planning process, P fixed (v) = 1. For other grids that are more frequently used in path planning, P fixed (v)>1. The fixed penalty for each grid does not change unless the number or structure of pods and picking stations changes. The AGV will tend to choose grids with smaller fixed penalties for movement.

[0071] Step 1.3: The real-time penalty is based on the path nodes of all AGVs and is dynamically updated during training or testing. The real-time penalty can be defined as:

[0072]

[0073] Where AGVGrid(t,v) is a function about discrete time, indicating whether the grid v is occupied by an AGV at discrete time t, and is defined as follows:

[0074]

[0075] Where AGVGrid(t,v) represents the number of times grid v has been occupied by an AGV from the start of the current round to time t. This value reflects the frequency of use of the grid; if a grid is frequently occupied by AGVs, it is considered a high-traffic area, and increasing the real-time penalty for that grid can effectively avoid further congestion.

[0076] Furthermore, the system will reset the real-time penalty values ​​of all grids when the AGV reaches the target location. The unlimited growth of the real-time penalty may cause the penalty of some grids to be too high, causing the AGV to completely avoid these grids in subsequent decisions.

[0077] Step 1.4: According to Steps 1.2 and 1.3, the AGV's movement penalty is uneven, that is, the path length from one grid to the adjacent grid is not equal. The total penalty consists of two parts: fixed penalty and real-time penalty:

[0078] Penalty(t,v)=P fixed (v)+P real-time (t,v) (5)

[0079] Where, the fixed penalty P fixed(v) It depends on the layout structure of the warehouse of the goods-to-person system, which is calculated before deployment, and the real-time penalty P real-time (t,v) depends on the motion paths of all current AGVs and changes over time.

[0080] Furthermore, the A* algorithm is used to calculate the global guidance path of each AGV in the system based on the sum of the actual penalty value of the path (g value) and the estimated penalty value to the target point (h value).

[0081] Furthermore, for an empty AGV heading to the target shelf in state 1, the global environment is considered to be obstacle-free when planning using the improved A*, that is, it can shuttle under the shelf; for AGVs loaded with shelves in states 2 and 3, the storage station and picking station need to be regarded as static obstacles for global guidance planning.

[0082] Step 2: Construct a grid environment based on the point cloud data acquired by the LiDAR onboard each AGV, converting the real-time scanned map into grid map information. The grid map information at each time step includes local static obstacle information (such as storage station shelves and picking stations), as well as the positions of other AGVs within the field of view.

[0083] Furthermore, if Figure 2 As shown in the figure, the upper right corner of the grid represents the fixed penalty, and the lower right corner represents the real-time penalty of the grid. Each time the AGV passes by, the real-time penalty increases by 1. The color depth in the figure represents the size of the fixed penalty, and the darker the color, the greater the penalty.

[0084] Furthermore, in order to prevent Figure 1 As shown, the AGV in mode 1 has an impact on system congestion. The present invention sets the grid fixed penalty of the storage area containing the Pod to +1 and does not calculate the real-time penalty to ensure that the AGV in mode 1 can pass under the shelf first, that is, the AGV with the transportation task, that is, Figure 2 AGVs in Mode 2 and 3 can free up planning space, which is beneficial to improving system throughput.

[0085] Step 3: Based on the extracted goods-to-person system grid map information, a local observation state space matrix is ​​constructed, which contains three map matrices: a static obstacle map matrix, a local AGV map matrix, and a global guidance path map matrix;

[0086] According to step 2, the point cloud data obtained by the laser radar is converted into a grid map. The present invention encodes the grid map into an observation matrix.

[0087] Specifically, by performing a local scan of the environment of each AGV, a detection radius r with the AGV as the center is obtained. FOV The square area, area (2rFOV +1)×(2r FOV +1), such as Figure 3 As shown. FOV When =3, the AGV’s field of view is a 7×7 grid area.

[0088] Furthermore, the local observation space of each AGV It consists of three different components: static obstacle map matrix Indicates the location of storage stations and picking stations, coded as -1; local AGV (dynamic obstacle) map matrix Indicates the position of other AGVs, coded as +1; global guidance path map matrix The global guidance path generated based on the method of step 1 of the example of the present invention is coded as +2, which represents the guidance path within the field of view of the AGV.

[0089] Furthermore, for the empty AGV heading to the target shelf state 1, in the static obstacle map matrix When encoding, static obstacles are ignored, that is, they can be shuttled under the storage station shelves and all are encoded as 0.

[0090] By encoding in the manner described, the method of the present invention ensures that the AGV can focus on the current local planning and decision-making task while following the global guidance path during the autonomous path planning process.

[0091] In the present invention, at each time step t, the local observation space includes not only the observation at the current moment, but also the observation data of the previous four time steps, thereby forming a time series {O t-4 ,O t-3 ,O t-2 ,O t-1 ,O t}.

[0092] This time series provides historical observation information to the AGV, enabling it to identify and adapt to dynamic changes and trends in the environment.

[0093] Step 4: Use the asynchronous proximal policy optimization (APPO) algorithm to train the local path planner, fuse the time series observation data through the long short-term memory (LSTM) network, output the collision avoidance action strategy and imitate the global guidance path;

[0094] Step 4.1: The guidance path generated by the method described in step 1 is used as the benchmark for AGV path planning, and a reward function is set. At each time step t, when the AGV reaches a path point on the global guidance path, it will receive a reward value of +1.

[0095]

[0096] Where p i (t+1) represents the position of AGV at time step t+1, W i The reward mechanism of the present invention encourages AGV to follow the global guide path closely, and the reward structure is simple, which improves the training efficiency. If there is no path point in the global guidance path, the heuristic guidance path planner in step 1 will be reactivated to generate a new global guidance path. This mechanism of the present invention ensures that the AGV can continue to obtain effective navigation guidance in the goods-to-person system.

[0097] Step 4.2: Local observation space synthesized according to step 3 And the historical observation space {O t-4 ,O t-3 ,O t-2 ,O t-1 ,O t The present invention converts this into a 7×7×3×5 input tensor. It then extracts environmental state features through convolutional layers and residual blocks, and performs feature flattening, which forms the spatial encoder. These flattened features are then fed into the LSTM module, enabling the AGV to retain its memory of historical observations and incorporate spatiotemporal information into its decision-making process.

[0098] Step 4.3: Action decoding is performed on the output of the LSTM, which is composed of an Actor-Critic network. The Actor network consists of two fully connected layers, with a ReLU activation function applied after the first layer. The Actor network generates a probability distribution of possible actions through the Softmax activation function to implement action decoding (Action Decoder); the Critic network also consists of two fully connected layers, with a ReLU activation function applied after the first layer. The Critic network estimates the value function V t , represents the expected cumulative reward of the AGV's current state.

[0099] All AGVs share a unified policy network and use the APPO algorithm to optimize the policy and maximize the expected return. The strategy update is adjusted by the clipping parameter ε to ensure the stability of training.

[0100] In step 4, which involves deep reinforcement learning training, this paper provides a dataset construction method. To address the multi-AGV path planning problem in a goods-to-person system, this paper constructs a diverse dataset consisting of 200 64×64 maps, 25% of which are random maps, 25% are maze maps, and 50% are warehouse-style maps. The warehouse maps simulate the typical layout of a goods-to-person system, supporting storage and retrieval task training. The random and maze maps introduce complex topologies, enhancing the model's adaptability in complex environments.

[0101] The dataset uses the PRIMAL2 method generator to generate random and maze maps with an obstacle density of 20% or 40%, and contains 64 or 128 AGVs and 8 picking stations. The warehouse map generator randomly adds paths to ensure the target density is 50%. The specific algorithm is shown in Figure 4 The dataset provides a diverse environment for model training, ensuring its robustness and efficiency in complex layouts.

[0102] Step 5: For the storage units with movable shelves in the storage area, the fixed penalty is set to +1, and the real-time dynamic penalty is set to zero. That is, for the goods-to-person system, the grid fixed penalty of the storage area containing movable shelves is set to +1, and the real-time penalty is not calculated to avoid interference with path planning.

[0103] Furthermore, in step 1, in order to better disperse the initial path of the AGV and avoid congestion in narrow warehouse aisles and high-traffic areas, a non-uniform penalty mechanism is used to adjust the path. The non-uniform penalty mechanism includes a fixed penalty and a real-time dynamic penalty, which is expressed as:

[0104] Penalty(t,v)=P fixed (v)+P real-time (t,v)

[0105] Where, P fixed (v) represents a fixed penalty, P real-time (t,v) represents the real-time penalty.

[0106] In addition, when an AGV reaches its destination, the system's real-time penalty will be reset to prevent a grid from being penalized too much in real time and causing all AGVs to not choose this grid as a waypoint.

[0107] Furthermore, the fixed penalty is the average path length from each grid to the unoccupied area calculated based on the Dijkstra algorithm:

[0108]

[0109] By calculating the average path length L from each grid v to all unoccupied areas average(v),Determine the access difficulty of the grid, thereby assigning a fixed penalty value to each grid.

[0110] Specifically, the fixed penalty value P fixed (v) is calculated by the following formula:

[0111]

[0112] Where, L average (v) represents the average path length from grid v to all unoccupied areas, and the larger L average The (v) value indicates that the grid is difficult to reach, thus giving the grid a higher fixed penalty. The AGV is more inclined to avoid these grids during path planning;

[0113] Specifically, based on the static structural layout of the cargo-to-person system, the Dijkstra algorithm is first used to calculate the average path length L from each grid to all unoccupied areas (such as idle track grids, waiting area grids, etc.) average (v); This calculation assigns a fixed penalty value P to each grid fixed (v) indicates the difficulty of selecting the grid as a path node.

[0114] Furthermore, the real-time penalty P real-time (t, v) is based on the real-time path update of the AGV; the real-time penalty increases with the number of times the AGV passes through, and is calculated by the following formula:

[0115]

[0116] Where AGVGrid(t,v) is a binary function used to determine whether the AGV is located in the grid v at time step t.

[0117] If the AGV is located in the grid, AGVGrid(t,v)=1; otherwise, AGVGrid(t,v)=0; in addition, when an AGV reaches the target grid, the real-time penalties of all grids in the system will be reset.

[0118] Furthermore, combined with the real-time penalty value P real-time (t, v), this penalty value is updated based on the AGV's real-time path. The real-time penalty reflects how frequently a grid has been occupied during past path planning. Frequently used grids are assigned higher penalties to reduce the AGV's tendency to use these areas.

[0119] Furthermore, the A* algorithm combines the current actual penalty with the heuristically estimated penalty to calculate a global guidance path for each AGV in the system. The AGV in state 1 is planned for an environment without static obstacles. By adopting a non-uniform penalty mechanism, the AGV not only considers the impact of static obstacles when selecting a path, but also prioritizes avoiding congested areas. This results in a more even distribution of AGV paths, thereby improving the throughput of the entire system.

[0120] Furthermore, in step 3, a local observation state space matrix is ​​constructed, including a static obstacle map matrix, a local AGV map matrix, and a global guidance path map matrix. According to the data extracted from the lidar point cloud in step 2, three types of raster map matrix data are formed, and static obstacles, local AGVs, and global guidance paths are encoded as -1, +1, and +2, respectively.

[0121] Specifically, the static obstacle map matrix encodes static obstacles as -1, idle grids as 0, and state 1, that is, unloaded AGVs, all as 0; the local AGV map matrix encodes the grid where the AGV is located within the radar range and the current AGV as +1, and the remaining grids as 0; the global guidance path map matrix encodes the guidance path grid within the detection range calculated in step 1 as +2, and the remaining grids as 0.

[0122] Furthermore, in step 4, the asynchronous proximal policy optimization (APPO) algorithm is based on the proximal policy optimization (PPO) algorithm to build a local path planner in a reinforcement learning framework for each AGV. By updating the policy network and value network, the AGV path planning strategy is optimized, enabling the AGV to complete the path planning task efficiently and without collision.

[0123] The path planner uses the AGV's local observation data matrix to generate a collision avoidance path and simultaneously imitates the global guidance path to ensure that the AGV follows the global strategy of the entire path planning.

[0124] Furthermore, in order to optimize the path planning strategy, the following reward function is adopted:

[0125]

[0126] Where p i (t+1) is the location information of AGV at time t+1, W i is the target path point in the global guidance path. When the AGV reaches a target node in the global path, it will receive a reward of +1.

[0127] Furthermore, the objective function of PPO is as follows:

[0128]

[0129] Where r t (θ) is the probability ratio between the new and old strategies, defined as:

[0130]

[0131] Furthermore, the AGV’s behavior is generated by the policy network, which decides its actions based on its local observations and historical states (via LSTM). Each time, the AGV selects an action based on its current state (including environmental information and historical paths).

[0132] Furthermore, the local path planner processes local observation sequences through a long short-term memory (LSTM) network, uses historical information to make path decisions, and enhances the AGV's adaptability to dynamic environmental changes.

[0133] Furthermore, the long short-term memory (LSTM) network consists of three parts: a spatial encoder, an LSTM module, and an action decoder; the spatial encoder extracts local environmental features, the LSTM module considers temporal information, and the action encoder generates action strategies and performs action sampling through a Softmax activation function.

[0134] Furthermore, a method for constructing a training dataset for a local path planner includes constructing a dataset containing 200 maps, 25% of which are random maps, 25% are maze maps, and 50% are warehouse-style maps.

[0135] Furthermore, the action behavior of each AGV is generated by the policy network, which decides on its action based on its local observation state and historical state (through LSTM); each AGV selects an action based on its current state (including environmental information and historical path), and the shared policy network is deployed on each AGV, and each step is repeatedly executed in a distributed manner to complete all picking tasks of the goods-to-person system.

[0136] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. Without departing from the scope and spirit of the described embodiments, many modifications and changes are obvious to ordinary technicians in this technical field. Any technical deformation made within the spirit and principles of the present invention falls within the scope of protection of the present invention.

Claims

1. A multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning, characterized by: The following steps are involved: Step 1: Based on the pre-deployed static structure layout of the goods-to-person system, an improved A* heuristic algorithm based on a non-uniform penalty mechanism is used to plan a global guidance path for the AGV; The non-uniform penalty mechanism includes fixed penalty and real-time dynamic penalty, which can be expressed as: Where, represents a fixed penalty, Indicates real-time punishment; The fixed penalty is the average path length from each grid cell to the unoccupied area calculated based on the Dijkstra algorithm: By calculating each raster The average path length to all unoccupied areas determines the access difficulty of the grid, thereby assigning a fixed penalty value to each grid; The calculation formula for the fixed penalty value is: Where, Represents a grid The average path length to all unoccupied areas. A larger fixed penalty value indicates that the corresponding grid is more difficult to reach. AGVs tend to avoid grids with large fixed penalty values ​​during path planning. Real-time punishment It is based on real-time path update of AGV; The real-time penalty increases as the number of times the AGV passes through accumulates and is calculated using the following formula: Where, Is a binary function used to determine the time step Is the AGV located in the grid? middle; Step 2: Construct the surrounding raster environment based on the point cloud data obtained by the laser radar on each AGV, and convert the real-time scanned map into raster map information; Step 3: Construct a local observation state space matrix based on the extracted goods-to-person system grid map information; Step 4: Use an asynchronous proximal policy optimization algorithm to train the local path planner, fuse the time series observation data through the long short-term memory (LSTM) network, output the collision avoidance action strategy and imitate the global guidance path; The asynchronous proximal policy optimization algorithm is based on the proximal policy optimization (PPO) algorithm. It builds a local path planner in a reinforcement learning framework for each AGV and optimizes the AGV path planning strategy by updating the policy network and value network. Step 5: For the movable shelf storage units in the storage area of ​​the goods-to-person system, the fixed penalty is set to +1, and the real-time dynamic penalty is set to zero.

2. The multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning according to claim 1 is characterized by: In step 3, the local observation state space matrix is ​​constructed, including the static obstacle map matrix, the local AGV map matrix, and the global guidance path map matrix, and the static obstacles, local AGVs, and global guidance paths are encoded as -1, +1, and +2, respectively.

3. The multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning according to claim 2 is characterized by: The local path planner processes local observation sequences through the LSTM network, uses historical information to make path decisions, and enhances the AGV's adaptability to dynamic environmental changes.

4. The multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning according to claim 3 is characterized by: The LSTM network consists of three parts: a spatial encoder, an LSTM module, and an action decoder; the spatial encoder extracts local environmental features, the LSTM module considers temporal information, and the action encoder generates action strategies and performs action sampling through the Softmax activation function.

5. The multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning according to claim 4 is characterized by: The training dataset construction method for the local path planner includes constructing a dataset containing 200 maps, of which 25% are random maps, 25% are maze maps, and 50% are warehouse-style maps.

6. The multi-AGV path planning method for a goods-to-person system based on deep reinforcement learning according to claim 5 is characterized by: The action behavior of each AGV is generated by the policy network, and the action is decided based on its local observation state and historical state. Each AGV selects an action based on the current state, and the shared policy network is deployed on each AGV, and each step is repeatedly executed in a distributed manner to complete all picking tasks in the goods-to-person system.

Images