Multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN

Abstract

The present invention provides a multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning (DQN), which includes the following steps: constructing a surrounding environment map based on lidar point cloud data and converting it into a raster map; constructing an AGV's observation matrix and state vector based on the raster map information, where the observation matrix records whether there are other AGVs around the AGV and the running directions of these AGVs; constructing a multi-starting point and multi-endpoint path planning model for a single AGV; applying the constructed model to all AGVs in the environment, calculating the pre-action of each AGV without considering other AGVs, obtaining an observation matrix based on the pre-actions of other AGVs within the AGV's observation range, and using the observation matrix to correct the model results. Compared with other heuristic algorithms or methods for constructing multi-AGV path planning using deep reinforcement learning methods, the present invention can ensure that the AGV maintains the optimal action selection while avoiding collisions.

Description

Technical Field

[0001] The present invention relates to a multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning (DQN), belonging to the technical field of mobile robot navigation. Background Art

[0002] In recent years, Automated Guide Vehicles (AGVs) have played an important role as material transportation tools in warehousing systems and manufacturing plants due to their high flexibility and high degree of automation brought by their fast response, strong controllability, high work efficiency and good safety.

[0003] Path planning algorithm research is one of the most important technologies in AGV research. Its goal is to plan an optimal or suboptimal AGV movement route between a known AGV starting point and destination, based on different requirements, to ensure smooth and efficient transportation. As systems become larger and more complex, multiple AGVs must collaborate to ensure orderly and efficient operation. When multiple AGVs are included in a system, in addition to planning a reasonable and efficient path for each AGV to reach its destination, collision avoidance between AGVs must also be considered. A reasonable path must be planned for each AGV to reach its destination, while ensuring that collisions and deadlocks between AGVs are avoided.

[0004] Most existing path planning algorithms for multiple AGVs use heuristic or meta-heuristic algorithms to plan paths for each AGV separately, predict and analyze potential collisions. If an AGV is about to collide, a specific method is used to re-plan the route to avoid the collision. However, this approach presents the following problems: When the environment is complex and the number of AGVs is large, traditional path search algorithms are computationally intensive and have poor real-time performance, making it impossible to quickly determine the optimal AGV operation and obstacle avoidance strategy. Most existing methods analyze collisions between AGVs in four directions: up, down, left, and right. This significantly limits the AGV's operational space and flexibility.

[0005] A multi-robot path planning and obstacle avoidance method (Haodong Li, Tao Zhao, Songyi Dian. Prioritized planning algorithm for multi-robot collision avoidance based on artificial untraversable vertex [J]. Applied Intelligence (2022) 52:429–451.) defines five types of inter-robot collisions based on the robots' motion and position. To avoid these collisions, the concept of artificial untraversable vertices is proposed. Finally, the D*Lite method is combined for path planning and obstacle avoidance. Because the collision avoidance strategy is a limited, artificially defined method, it cannot cover all multi-robot collision scenarios. Furthermore, the optimal path is ignored during obstacle avoidance, and replanning the robots' paths to the destination after obstacle avoidance is more cumbersome. To address these shortcomings, an adaptive obstacle avoidance method based on deep reinforcement learning is proposed. This method can achieve both optimal path selection and obstacle avoidance, improving computational efficiency while significantly reducing computational complexity. Summary of the Invention

[0006] In order to solve the problems existing in the above background technology, the present invention provides a multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN.

[0007] The present invention is achieved through at least one of the following technical solutions.

[0008] A multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN includes the following steps:

[0009] Step 1: Build a map of the surrounding environment based on the LiDAR point cloud data and convert it into a raster map; the information on the raster map includes the location of each AGV and the size of the obstacles;

[0010] Step 2: Construct the AGV's observation matrix and state vector based on the grid map information. The observation matrix records whether there are other AGVs around the AGV and the running direction of other AGVs. The state vector includes the following three parts: the end point position, the relationship between the AGV's current position and the end point, and the surrounding environment information.

[0011] Step 3: Build a multi-starting and multi-destination path planning model for a single AGV based on the deep reinforcement learning algorithm and the AGV’s state vector;

[0012] Step 4: Apply the model constructed in step 3 to all AGVs in the environment, calculate the pre-action of each AGV without considering other AGVs, obtain the observation matrix based on the pre-actions of other AGVs within the AGV's observation range, and use the observation matrix to correct the model results to avoid collisions between AGVs.

[0013] Furthermore, in step 2, the relationship between the current position of the AGV and the end point is expressed in a relative position manner. The directions of the AGV position relative to the end point position include 8 directions: up, upper left, left, lower left, down, lower right, right, and upper right, which are expressed as (-1, 0), (-1, -1), (0, -1), (1, -1), (1, 0), (1, 1), (0, 1), and (-1, 1) respectively.

[0014] Furthermore, in step 2, the encoding method for the environment within a grid around the AGV is: according to the AGV sequentially executing the corresponding actions in the action space {still, up, upper left, left, lower left, down, lower right, right, upper right}, the grid position and environmental attributes are encoded and the surrounding environment information is added to the state vector.

[0015] Furthermore, for the multi-start and multi-destination path planning model of a single AGV, the operation mode of the AGV in the grid map is set, including: the AGV can run in eight directions around it and can choose to stay in the same position. That is, the action space of the AGV is {stop, up, upper left, left, lower left, down, lower right, right, upper right}, which is represented by numbers {0, 1, 2, 3, 4, 5, 6, 7, 8}; the action is selected using the ∈ greedy strategy:

[0016]

[0017] Where ∈∈[0,1] represents the randomness of the strategy, that is, the probability of selecting an action from a random action and a greedy action; c represents a randomly generated decimal greater than 0 and less than 1. When the generated c is greater than the randomness, the greedy action is selected, otherwise the random action is selected. Represents a greedy action, that is, the Q value network Q(s, a, θ) trained by DQN selects the current optimal action a according to the state s, where θ represents the network parameters of the Q value network.

[0018] Furthermore, the deep reinforcement learning algorithm is a DQN algorithm. By collecting the interaction data between the AGV and the virtual environment, including the AGV status, AGV actions and feedback rewards of the environment, the DQN model algorithm is trained to obtain a DQN model, that is, a multi-starting point and multi-endpoint path planning model for a single AGV. The optimal operation strategy suitable for the AGV is learned for the interaction between the AGV and the environment.

[0019] Furthermore, the reward function for AGV’s interaction with the environment is set as:

[0020] R=r1+r2+r3+r4

[0021]

[0022]

[0023]

[0024]

[0025] Where r1 represents the penalty for the AGV running beyond the boundary or hitting an obstacle; r2 represents the penalty for the moving distance balance of the oblique action; r3 represents the heuristic reward, which is given by calculating the Manhattan distance between the AGV and the end point and comparing it with the distance before the operation; r4 represents the reward for the AGV reaching the end point, p A =(x A ,y A ) indicates the current position, x A 、y A Indicates the specific horizontal and vertical coordinates, p′ A =(x′ A ,y′ A ) indicates the position at the previous moment; p G =(x G ,y G ) indicates the end point position, the subscript G represents the end point, and a represents the action performed by the AGV.

[0026] Furthermore, the DQN algorithm combines the neural network algorithm and the Q-learning reinforcement learning algorithm. In Q-learning, the Q value is recorded and updated using a table. The Q value is calculated using the following formula:

[0027] Q π (s,a)=E π [G t |S t =s,A t =a]

[0028]

[0029] Where Q π (s, a) means according to strategy E π , at time t, the state space S t Take action space A in a state s t The value function of an action a in G t Expressed as the reward R for each subsequent step k t+k+1The discounted sum, γ represents the discount factor, and the reward R t+k+1 It is the feedback given by the environment based on the agent's actions, and is a short-term criterion for judging the quality of the strategy at that step.

[0030] Where G(t) represents the reward function at time t, which is a measure of the subsequent impact of executing action a in state s. G is defined as the sum of subsequent reward functions. The DQN algorithm uses the Q-value network to fit the Q-value, parameterizes the Q-value, and updates the Q-value by updating the network parameters:

[0031] Q(s,a,θ)≈Q * (s,a)

[0032] Where θ represents the neural network parameters, Q * (s, a) represents the Q-value function obtained under the guidance of the optimal strategy, and Q(s, a, θ) represents the Q-value calculated by the parameterized Q-value network.

[0033] Furthermore, for AGVs running in eight directions, collisions between AGVs include encounter collisions, intersection collisions, and trip collisions. An encounter collision refers to a collision between two AGVs running into the same grid. An intersection collision refers to a collision between two AGVs that overlap during operation but ultimately are not in the same grid. A trip collision refers to a collision between two AGVs due to their actual size during operation.

[0034] Furthermore, in step 4, the method for solving the action correction coefficient is to first construct the observation matrix of each AGV, extract the sub-observation matrix corresponding to each action, and perform collision analysis on each sub-observation matrix to obtain the correction coefficient under the action.

[0035] Furthermore, a collision analysis is performed on the AGV's different action sub-observation matrices to determine whether the action will cause a collision. If a collision will occur, the action correction is 0; if a collision will not occur, the action correction is 1. This results in a correction coefficient for the AGV's executable action. The value of the AGV obtained in step 3 without considering other AGVs is vector-multiplied by the correction coefficient, and the action represented by the maximum Q value is taken as the AGV's final action.

[0036] Compared with the existing technology, the beneficial effects of the present invention are:

[0037] 1. We built a path planning algorithm based on deep reinforcement learning. The trained model can solve the AGV path planning problem with multiple starting points and multiple destinations. That is, from any starting point in a known environment, the optimal running direction to a given destination can be obtained.

[0038] 2. By using this model in combination with the action correction coefficient, the optimal action to avoid collision can be automatically given when a collision between AGVs is detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the AGV running direction in the embodiment;

[0040] Figure 2 Schematic diagram of the DQN training process in the embodiment;

[0041] Figure 3 This is a diagram of the single AGV path planning process based on DQN in the embodiment;

[0042] Figure 4 Schematic diagrams of several collision modes in the embodiment;

[0043] Figure 5 This is a schematic diagram of multi-AGV collision avoidance in an embodiment;

[0044] Figure 6 This is a flowchart of an embodiment of a multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN;

[0045] Figure 7 This is the AGV path planning and obstacle avoidance result diagram in the 4×4 environment of the embodiment;

[0046] Figure 8 This is the result diagram of AGV path planning and obstacle avoidance in an 8×8 obstacle environment in Example 1. DETAILED DESCRIPTION

[0047] To help those skilled in the art better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It is apparent that the embodiments described are only a portion of the present invention, not all of the embodiments. All other embodiments derived by those skilled in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0048] The present invention provides a multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN, the steps are as follows: Figure 6 As shown, the deep reinforcement learning network is used to embed and map the AGV and its surrounding AGV status information and local environment information, and the direction of the AGV to the end point while avoiding other AGVs and obstacles is obtained. The specific steps include the following.

[0049] Step 1: Build a map of the surrounding environment based on the LiDAR point cloud data and convert it into a raster map. The information enriched in the raster map includes the initial position of each AGV, the size and position of obstacles, and the location of the end point.

[0050] Step 2: Construct the AGV's observation matrix and state vector based on the grid map information. The observation matrix records whether there are other AGVs around the AGV and their running directions; the state vector contains the following three parts: the end point location, the relationship between the AGV's current position and the end point, and the surrounding environment information. The specific implementation consists of the following sub-steps:

[0051] Step 2.1. Add the destination location to the state vector. The destination location consists of coordinates, with one grid cell representing one unit of length. Set the top left corner of the map as the origin, with the right and down axes as positive directions. Set the destination location coordinates based on the number of grid cells.

[0052] Step 2.2: Add the relationship between the AGV's current position and the destination to the state vector.

[0053] As an embodiment, this relationship represents the direction of the AGV position relative to the end point, including eight directions: up, upper left, left, lower left, down, lower right, right, and upper right, which are represented by numbers as (-1, 0), (-1, -1), (0, -1), (1, -1), (1, 0), (1, 1), (0, 1), (-1, 1), such as Figure 1 shown.

[0054] Step 2.3: Encode the surrounding environment information and add it to the state vector. Define the visible range based on the AGV's field of view. The minimum visible range is one grid around it. Encode the environmental attributes within the visible range, where a stationary obstacle grid is -1, a passable grid is 0, and the end point is 1. Encode the grid positions and environmental attributes obtained by the AGV executing the corresponding actions in the action space {stationary, up, upper left, left, lower left, down, lower right, right, upper right}, and add the surrounding environment information to the state vector.

[0055] Step 3: Build a multi-start and multi-destination path planning model for a single AGV based on the deep reinforcement learning DQN and the AGV's state vector. This model ensures that the AGV can plan a smooth path to different destinations from multiple different starting points. The specific construction method is as follows:

[0056] Step 3.1: Set the AGV's operation mode within the grid map. This includes: the AGV can move in eight directions and can choose to stay in position. The AGV's action space is {stop, up, upper left, left, lower left, down, lower right, right, upper right}, represented numerically as {0, 1, 2, 3, 4, 5, 6, 7, 8}; the AGV cannot pass through obstacles diagonally. Action selection uses the ∈ greedy strategy.

[0057]

[0058] Where ∈∈[0,1] represents the randomness of the strategy, that is, the probability of selecting an action from a random action and a greedy action; c represents a randomly generated decimal greater than 0 and less than 1. When the generated c is greater than the randomness, the greedy action is selected, otherwise the random action is selected. Represents a greedy action, that is, the Q value network Q(s, a, θ) trained by DQN selects the current optimal action a according to the state s, where θ represents the network parameters of the Q value network.

[0059] Step 3.2: Construct a reward function for the AGV's environmental feedback during operation. The reward function mainly consists of the following four parts:

[0060] R=r1+r2+r3+r4

[0061]

[0062]

[0063]

[0064]

[0065] Where r1 represents the penalty for the AGV running beyond the boundary or hitting an obstacle; r2 represents the penalty for the moving distance balance of the oblique action; r3 represents the heuristic reward, which is given by calculating the Manhattan distance between the AGV and the end point and comparing it with the distance before the operation; r4 represents the reward for the AGV reaching the end point, p A =(x A ,y A ) represents the current position, x and y represent the specific horizontal and vertical coordinates respectively, the subscript A represents that this is the attribute of AGV, p′ A =(x′ A ,y′ A ) indicates the position at the previous moment; p G =(x G ,y G ) indicates the end point position, the subscript G represents the end point, and a represents the action performed by the AGV.

[0066] Step 3.3: The deep reinforcement learning algorithm used is the DQN algorithm. DQN combines the neural network algorithm and the Q-learning reinforcement learning algorithm. In Q-learning, a table is used to record and update the Q value. The Q value is calculated by the following formula:

[0067] Q π (s,a)=E π [G t |S t =s,A t =a]

[0068]

[0069] Where Q π (s,a) means according to strategy E π , at time t, the state space S t Take action space A in a state s t The value function of an action a in G is represented by the reward G t The expected return G t Expressed as the reward R for each subsequent step k t+k+1 The discount sum, γ represents the discount factor, which is the long-term criterion for judging the quality of the strategy at this step. t+k+1 It is the feedback given by the environment based on the agent's actions, and is a short-term criterion for judging the quality of the strategy at that step.

[0070] Where G(t) represents the reward function at time t, a measure of the subsequent impact of executing action a in state s. G is defined as the sum of subsequent reward functions. Reward functions with greater distances between them have less impact on the current reward function. DQN uses a neural network Q-value network to fit Q-values, parameterize them, and update them by updating network parameters.

[0071] Q(s,a,θ)≈Q * (s,a)

[0072] Where θ represents the neural network parameters, Q * (s, a) represents the Q-value function obtained under the guidance of the optimal policy, and Q(s, a, θ) represents the Q-value calculated by the parameterized Q-value network. If the Q-value calculated by the Q-value network is close to the optimal Q-value, the network will appear to be the optimal policy.

[0073] Step 3.4, such as Figure 2 As shown in Figure 1, the DQN model training is improved to achieve multi-start and multi-destination path planning. The DQN algorithm training process uses the target network and experience replay mechanism to store the experience (s, s′, a, r) ​​of each interaction with the environment in memory and randomly extract it during training to calculate the loss function:

[0074] L(θ)=E[(yQ(s,a,θ)) 2 ]

[0075] y=r+γmax a Q target (s',a,θ - )

[0076] Where target network Q target (s′,a,θ- ) Calculate y as the optimal value of the Q value network's current state s through the next state s'. Parameter θ - is a delayed copy of the Q-value network parameters θ. L(θ) represents the mean squared difference loss between the Q-value network and the target network.

[0077] Step 3.5: Improve the model's generalization performance to implement a path planning algorithm for multiple starting points and destinations. The initial point selection for each training session is not restricted to a single location; instead, different initial points are randomly selected to begin training. This increases the variety of data the agent is exposed to during training, broadens the agent's perception of the environment, and improves the generalization performance of the reinforcement learning model in a single environment. The destination position is added to the state vector, and the destination coordinates are changed during training. The states of different destinations at the same location are treated as different states. This expands the state space while increasing the data dimensionality. This increases the model's generalization performance by increasing the amount of data, enabling path planning to multiple destinations.

[0078] Step 4: Apply the model constructed in step 3 to all AGVs in the environment, calculate the pre-action of each AGV without considering other AGVs, obtain the observation matrix based on the pre-action of other AGVs within the AGV observation range, and use the observation matrix to correct the Q value obtained by the model to avoid collisions between AGVs. Figure 5 The specific correction process is as follows:

[0079] Step 4.1, such as Figure 4 As shown in the figure, under the rule that AGVs can operate diagonally, collisions between AGVs include encounter collisions, intersection collisions, and trip collisions. An encounter collision occurs when two AGVs collide within the same grid. An intersection collision occurs when two AGVs overlap during operation but ultimately end up in different grids. A trip collision occurs when two AGVs collide during operation due to their actual size, and their positions do not completely overlap.

[0080] Step 4.2: Set the AGV's observation range to a minimum of two grid cells around itself. This means the AGV's observation matrix is ​​a 5×5 matrix with its own position in the center. This matrix records the AGV's pre-action status at positions within the two surrounding grid cells. The AGV's position (2, 2) is its pre-action. If there are no other AGVs within the observation range, all other positions are marked as -1. Positions with other AGVs are marked as pre-actions.

[0081] Step 4.3: For each action that the AGV can perform, extract a 3×3 sub-observation matrix from its observation matrix. This sub-observation matrix indicates that when the AGV takes that action, only other AGVs within the sub-observation matrix corresponding to that action will collide with the AGV. Based on the AGV collision scenarios described in Step 4.1, perform a collision analysis on the sub-observation matrices for different AGV actions to determine whether the action will cause a collision. If so, the action's correction factor is 0; if not, the action's correction factor is 1. This results in a correction factor for each action that the AGV can perform.

[0082] Step 4.4: Multiply the Q value of the AGV obtained in step 3 without considering other AGVs (i.e., the value of each action) by the correction coefficient of the AGV action obtained in step 4.3 at the corresponding position of the vector to obtain the final Q value considering other AGVs. The action represented by the maximum Q value is taken as the final action of the AGV.

[0083] As a preferred embodiment, Figure 5 The following figure shows an analysis of the actions to be performed by AGV1. First, in the environment, the darker squares represent static obstacles, the squares filled in other ways represent the running AGVs, and the flag represents the target point of AGV1. The AGV1 state code is constructed based on the environmental information of one square around AGV1 and the end point information, where the environmental information is encoded according to the direction of the arrow in the figure. The encoded state vector is then input into the trained neural network, and the action probability distribution of AGV1, that is, the Q value, is output. Action 7 (rightward) with the largest Q value is selected as the pre-action of AGV1. The same operation is performed on all AGV1 in the environment to obtain their pre-actions.

[0084] An observation matrix is ​​constructed for analysis, capturing environmental information within two grids surrounding AGV1 that does not contain static obstacles. Pre-action information is annotated for other AGV1s within the captured environment. For each AGV1 action, a 3×3 submatrix is ​​obtained from the observation matrix, starting from the stationary position and moving counterclockwise to the upper right. The collision behavior of each submatrix is ​​analyzed, and a correction coefficient of 0 is assigned to actions that could cause a collision, and a correction coefficient of 1 is assigned to actions that would not cause a collision. This constructs an action correction coefficient vector. The Q value vector of AGV1 is multiplied by the corresponding position of the action correction coefficient vector to obtain the final Q value vector. The action with the highest Q value is selected as the AGV's execution action.

[0085] As a preferred embodiment, Figure 7 As shown in the figure, a 4×4 grid environment is constructed, with four AGVs located at four vertices in the environment, and their endpoints are located on the other side of the diagonal. The deep reinforcement learning DQN model is trained for this environment and the endpoint positions, and the training results are shown in the figure. Figure 7The trained model is combined with the action correction factor to perform path planning and obstacle avoidance between multiple AGVs. The results are shown in (a), (b), and (c). Figure 7 As shown in (d), (e), (f), (g), (h), and (i), the AGVs avoided collisions reasonably and finally arrived at their respective destinations effectively.

[0086] Similarly, as a preferred embodiment, Figure 8 As shown in the figure, an 8×8 grid environment with obstacles is constructed, containing three AGVs and their three destinations. The reward function, step count, and loss function during the training process are shown in the figure. The model approached convergence after 110 training rounds. To improve the model's generalization, training was continued until 600 rounds. The figure shows the AGV's obstacle avoidance process in an environment with obstacles, using the model and correction factors. Based on the priority order, if there are other AGVs ahead and no other paths to avoid, the AGV will choose to retreat to avoid the obstacle. This method is also the optimal path in this situation.

[0087] This paper uses deep reinforcement learning to construct a DQN model for AGV action selection. This model can determine the optimal action selection for an AGV at any location in the environment leading to its destination. This model is extended to the action decisions of all AGVs in the environment. Based on collision analysis between multiple AGVs, an action correction factor is constructed for each action, thereby achieving optimal path selection without collisions. Compared with other methods that use deep reinforcement learning to construct multi-AGV path planning, this method can ensure that AGVs maintain relatively optimal action selection while avoiding collisions.

[0088] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN, characterized by: The following steps are involved: Step 1: Build a map of the surrounding environment based on the LiDAR point cloud data and convert it into a raster map; the information on the raster map includes the location of each AGV and the size of the obstacles; Step 2: Construct the AGV's observation matrix and state vector based on the grid map information. The observation matrix records whether there are other AGVs around the AGV and the running direction of other AGVs. The state vector includes the following three parts: the end point position, the relationship between the AGV's current position and the end point, and the surrounding environment information. Step 3: Build a multi-starting and multi-destination path planning model for a single AGV based on the deep reinforcement learning algorithm and the AGV's state vector; the deep reinforcement learning algorithm is the DQN algorithm, which combines the neural network algorithm and the Q-learning reinforcement learning algorithm. In Q-learning, a table is used to record and update the Q value, and the Q value is calculated by the following formula: Q π (s,a)=E π [G t |S t =s,A t =a] Where Q π (s,a) means according to strategy E π , at time t, the state space S t Take action space A in a state s t The value function of an action a in G t Expressed as the reward R for each subsequent step k t+k+1 The discounted sum, γ represents the discount factor, and the reward R t+k+1 It is the feedback given by the environment based on the agent's actions, and is a short-term criterion for judging the quality of the strategy at that step; Where G(t) represents the reward function at time t, which is a measure of the subsequent impact of executing action a in state s. G is defined as the sum of subsequent reward functions. The DQN algorithm uses the Q-value network to fit the Q-value, parameterizes the Q-value, and updates the Q-value by updating the network parameters: Q(s,a,θ)≈Q * (s,a) Where θ represents the network parameters of the Q-value network, Q * (s, a) represents the Q-value function obtained under the guidance of the optimal strategy, and Q(s, a, θ) represents the Q-value calculated by the parameterized Q-value network; For the multi-start and multi-destination path planning model of a single AGV, the operation mode of the AGV in the grid map is set, including: the AGV can run in eight directions around it and can choose to stay in the same position. That is, the AGV's action space is {stop, up, upper left, left, lower left, down, lower right, right, upper right}, which is represented by numbers {0, 1, 2, 3, 4, 5, 6, 7, 8}; the action is selected using the ∈ greedy strategy: Where ∈∈[0,1] represents the randomness of the strategy, that is, the probability of selecting an action from a random action and a greedy action; c represents a randomly generated decimal greater than 0 and less than 1. When the generated c is greater than the randomness, the greedy action is selected, otherwise the random action is selected. Represents a greedy action, that is, the Q value network Q(s, a, θ) trained by DQN selects the current optimal action a according to the state s, where θ represents the network parameters of the Q value network; Step 4: Apply the model constructed in step 3 to all AGVs in the environment, calculate the pre-action of each AGV without considering other AGVs, obtain the observation matrix based on the pre-actions of other AGVs within the AGV's observation range, and use the observation matrix to correct the model results to avoid collisions between AGVs.

2. A multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN as claimed in claim 1, characterized in that: In step 2, the relationship between the current position of the AGV and the end point is expressed in a relative position. The directions of the AGV position relative to the end point include up, upper left, left, lower left, lower, lower right, right, and upper right, which are expressed as (-1, 0), (-1, -1), (0, -1), (1, -1), (1, 0), (1, 1), (0, 1), and (-1, 1) respectively.

3. The multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN as claimed in claim 1, characterized in that: In step 2, the environment within one grid around the AGV is encoded as follows: the grid position and environmental attributes obtained by the AGV sequentially executing the corresponding actions in the action space {still, up, upper left, left, lower left, down, lower right, right, upper right} are encoded and the surrounding environment information is added to the state vector.

4. The multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN as claimed in claim 1, characterized in that: The deep reinforcement learning algorithm is the DQN algorithm. By collecting the interaction data between the AGV and the virtual environment, including the AGV status, AGV actions and feedback rewards from the environment, the DQN model algorithm is trained to obtain the DQN model, which is a multi-starting point and multi-endpoint path planning model for a single AGV. The optimal operation strategy suitable for the AGV is learned based on the interaction between the AGV and the environment.

5. The multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN as claimed in claim 4 is characterized in that: The reward function for AGV's interaction with the environment is set as: R＝r1+r2+r3+r4 Where r1 represents the penalty for the AGV running beyond the boundary or hitting an obstacle; r2 represents the penalty for the moving distance balance of the oblique action; r3 represents the heuristic reward, which is given by calculating the Manhattan distance between the AGV and the end point and comparing it with the distance before the operation; r4 represents the reward for the AGV reaching the end point, p A =(x A ,y A ) indicates the current position, x A 、y A Indicates the specific horizontal and vertical coordinates, p′ A =(x A ′,y A ′) indicates the position at the previous moment; p G =(x G ,y G ) indicates the end point position, the subscript G represents the end point, and a represents the action performed by the AGV.

6. The multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN as claimed in claim 2, characterized in that: For AGVs running in eight directions, collisions between AGVs include encounter collisions, staggered collisions, and trip collisions. An encounter collision refers to a collision between two AGVs running into the same grid. An staggered collision refers to a collision between two AGVs that overlap during operation but ultimately are not in the same grid. A trip collision refers to a collision between two AGVs due to their actual size during operation.

7. A multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning (DQN) according to any one of claims 1 to 6, characterized in that: In step 4, the method for solving the action correction coefficient is to first construct the observation matrix of each AGV, extract the sub-observation matrix corresponding to each action, and perform collision analysis on each sub-observation matrix to obtain the correction coefficient under the action.

8. The multi-AGV path planning and obstacle avoidance method based on deep reinforcement learning DQN as claimed in claim 7, characterized in that: Perform collision analysis on the AGV's different action sub-observation matrices to determine whether the action will cause a collision. If it will cause a collision, the correction of the action is 0; if it will not cause a collision, the correction of the action is 1, thereby obtaining the correction coefficient of the AGV's executable action; vector multiply the value of the AGV obtained in step 3 without considering other AGVs by the correction coefficient, and take the action represented by the maximum Q value as the AGV's final action.

Images