Deep reinforcement learning multi-AGV conflict-free path planning method for storage environment

Abstract

The invention provides a storage environment deep reinforcement learning multi-AGV conflict-free path planning method, and the method comprises the steps: creating a virtual storage simulation training environment, and extracting virtual storage simulation training environment information from the virtual storage simulation training environment; determining a state observation value code of each AGV according to the virtual storage simulation training environment information; setting an action space of the AGV, and constructing a reward function of deep reinforcement learning; and constructing a distributed conflict-free multi-AGV path planner introducing a dynamic grouping mechanism, carrying out iterative training on the distributed conflict-free multi-AGV path planner through a deep reinforcement learning network according to the reward function and the state observation value code and the action space of each AGV, updating network parameters of the distributed conflict-free multi-AGV path planner, and carrying out path planning on the distributed conflict-free multi-AGV path planner according to the network parameters of the distributed conflict-free multi-AGV path planner. A trained distributed conflict-free multi-AGV path planner is obtained; and carrying out conflict-free path planning on a plurality of AGVs in an actual storage environment by using the trained distributed conflict-free multi-AGV path planner to obtain a conflict-free path of each AGV.

Description

Technical Field

[0001] The present invention relates to the field of logistics robot path planning, and in particular to a deep reinforcement learning multi-AGV conflict-free path planning method in a warehousing environment. Background Art

[0002] In recent years, new technologies and new business models have emerged on a large scale, and the traditional logistics industry is evolving from manual to automated. With the rapid development of logistics robots and Internet of Things technologies, more and more companies are beginning to make strategic layouts in the field of smart logistics. In order to improve work efficiency and reduce labor intensity, e-commerce companies have begun to use logistics robots (such as AGV, Automated Guided Vehicle, automatic guided vehicle) to achieve automated picking, and multi-robot path planning is a key link in the scheduling of automated picking systems.

[0003] In the process of realizing the present invention, the applicant found that there are at least the following problems in the prior art: most of the traditional robot path planning research focuses on simple situations where the static environment or the system environment is known, and the models and methods are constructed based on the idea of ​​operations optimization, which is difficult to meet the ever-changing complex environment in the current research context, and cannot fully use the dynamic data of the system, resulting in poor practical application of the models and methods. Most of the research on logistics robot path planning problems based on reinforcement learning has been carried out based on single-agent reinforcement learning methods, emphasizing the interaction and feedback between a single agent and the environment, and less considering the impact of the interaction between agents on the results. In addition, most of the current methods use centralized training, and as the scale of agents increases, the problem of network dimensionality explosion often occurs during the training process. Summary of the invention

[0004] An embodiment of the present invention provides a deep reinforcement learning multi-AGV (Automated Guided Vehicle) conflict-free path planning method in a warehousing environment to solve the dimensionality disaster problem existing in the joint decision-making of multiple AGVs and realize conflict-free path planning of multiple AGVs.

[0005] To achieve the above objectives, an embodiment of the present invention provides a method for deep reinforcement learning multi-AGV conflict-free path planning in a warehouse environment, comprising:

[0006] Creating a virtual warehouse simulation training environment, and extracting virtual warehouse simulation training environment information from the virtual warehouse simulation training environment;

[0007] Determine, according to the virtual warehouse simulation training environment information, a state observation value code of each AGV among a plurality of AGVs in the virtual warehouse simulation training environment;

[0008] Set the action space of AGV and construct the reward function of deep reinforcement learning;

[0009] Constructing a distributed conflict-free multi-AGV path planner that introduces a dynamic grouping mechanism, iteratively training the distributed conflict-free multi-AGV path planner according to the reward function and the state observation value encoding and action space of each AGV, updating the network parameters of the distributed conflict-free multi-AGV path planner, and obtaining a trained distributed conflict-free multi-AGV path planner;

[0010] Using the trained distributed conflict-free multi-AGV path planner to perform conflict-free path planning on multiple AGVs in an actual warehouse environment to obtain a conflict-free path for each AGV;

[0011] The virtual warehouse simulation training environment information includes the initial positions, obstacle positions, shelf positions and road information of multiple AGVs in the virtual warehouse simulation training environment; the state observation value encoding of the AGV includes: the position information and movement direction of the AGV, the target shelf position, whether there are goods, and the position information and movement direction of obstacles and other AGVs within the perception radius; the action space includes: left turn, right turn, forward, loading and unloading; the perception radius is used to define the observation range of the AGV;

[0012] The distributed conflict-free multi-AGV path planner includes a global network model and a local network model of each AGV; the global network model and the local network model of each AGV have the same network structure and correspond to an evaluation network, an attention network, a communication network, and a deep reinforcement learning network;

[0013] The evaluation network is used to fuse the features of other AGVs within the observation range of the AGV into the features of the AGV for each AGV to obtain a feature vector of the AGV, and determine the adjacency matrix of the AGV based on the feature vectors of all AGVs within the observation range of the AGV;

[0014] The attention network is used to calculate, for each AGV, a communication group of the AGV centered on the AGV based on the self-attention mechanism and according to the adjacency matrix of the AGV and the state observation value encoding of the AGV contained in the adjacency matrix of the AGV;

[0015] The communication network is used to perform network communication within a communication group centered on the AGV for each AGV, and update the state observation value code of each AGV in the communication group;

[0016] The deep reinforcement learning network is constructed based on a multi-agent asynchronous advantage actor-critic algorithm, and is used for AGVs that are divided into communication groups according to a dynamic grouping mechanism. Action decisions are made in a policy network according to the state observation value encoding of each AGV, path planning is completed, and rewards are obtained according to a reward function, with the goal of maximizing the reward. The model parameters of the local network model of each AGV are continuously iterated and updated, and the model parameters of the local network model of each AGV are used to update the model parameters of the global network model, wherein the deep reinforcement learning network includes a policy network and a critic network.

[0017] The above technical solution has the following beneficial effects: introducing a dynamic grouping mechanism, dynamically grouping multiple AGVs through an evaluation network and an attention network, communicating between AGVs through a communication network, and training network parameters in combination with deep reinforcement learning, thereby solving the dimensionality disaster problem that occurs when multiple AGVs make joint decisions during the conflict-free path planning process for multiple AGVs, and improving the efficiency of conflict-free path planning for multiple AGVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is a flow chart of a method for deep reinforcement learning multi-AGV conflict-free path planning in a warehouse environment, one of the embodiments of the present invention;

[0020] Figure 2 It is an algorithm flow chart of an evaluation network of a multi-AGV conflict-free path planning method for deep reinforcement learning in a warehouse environment, one of the embodiments of the present invention;

[0021] Figure 3 It is a schematic diagram of the AGV orientation area according to one embodiment of the present invention;

[0022] Figure 4 It is a structural schematic diagram of an attention network of a multi-AGV conflict-free path planning method for deep reinforcement learning in a warehouse environment according to one embodiment of the present invention;

[0023] Figure 5 It is a structural schematic diagram of a communication network of a method for deep reinforcement learning multi-AGV conflict-free path planning in a warehouse environment according to one embodiment of the present invention;

[0024] Figure 6It is a structural schematic diagram of a deep reinforcement learning network of a deep reinforcement learning multi-AGV conflict-free path planning method in a warehouse environment according to one embodiment of the present invention;

[0025] Figure 7 It is a deep reinforcement learning network training flow chart of a deep reinforcement learning multi-AGV conflict-free path planning method in a warehouse environment according to one embodiment of the present invention;

[0026] Figure 8 It is a schematic diagram of the network structure of a distributed conflict-free multi-AGV path planner of a deep reinforcement learning multi-AGV conflict-free path planning method in a warehousing environment according to one of the embodiments of the present invention;

[0027] Fig. 9 It is a reward iteration graph generated by training in a multi-AGV conflict-free path planning task simulation experiment in one embodiment of the present invention;

[0028] Fig.10 is a sample schematic diagram of a virtual warehouse simulation training environment according to one embodiment of the present invention;

[0029] Fig.11 It is a schematic diagram of a virtual warehouse simulation training environment of a warehouse 1 used in a simulation experiment according to one embodiment of the present invention;

[0030] Fig.12 It is a schematic diagram of a virtual warehouse simulation training environment of a warehouse 2 used in a simulation experiment according to one embodiment of the present invention;

[0031] Fig.13 It is a schematic diagram of a virtual warehouse simulation training environment of a warehouse 3 used in a simulation experiment according to one embodiment of the present invention;

[0032] Fig.14 It is a schematic diagram of quantization coding of a virtual warehouse simulation training environment according to one embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] On the one hand, if Figure 1 As shown, an embodiment of the present invention provides a method for deep reinforcement learning multi-AGV conflict-free path planning in a warehouse environment, comprising:

[0035] Step S10, creating a virtual warehousing simulation training environment, and extracting virtual warehousing simulation training environment information from the virtual warehousing simulation training environment;

[0036] Step S11, determining a state observation value code of each AGV among a plurality of AGVs in the virtual warehouse simulation training environment according to the virtual warehouse simulation training environment information;

[0037] Step S12, setting the action space of AGV and constructing the reward function of deep reinforcement learning;

[0038] Step S13, constructing a distributed conflict-free multi-AGV path planner that introduces a dynamic grouping mechanism, iteratively training the distributed conflict-free multi-AGV path planner according to the reward function and the state observation value encoding and action space of each AGV, updating the network parameters of the distributed conflict-free multi-AGV path planner, and obtaining a trained distributed conflict-free multi-AGV path planner;

[0039] Step S14, using the trained distributed conflict-free multi-AGV path planner to perform conflict-free path planning on multiple AGVs in an actual warehouse environment to obtain a conflict-free path for each AGV;

[0040] The virtual warehouse simulation training environment information includes the initial positions, obstacle positions, shelf positions and road information of multiple AGVs in the virtual warehouse simulation training environment; the state observation value encoding of the AGV includes: the position information and movement direction of the AGV, the target shelf position, whether there are goods, and the position information and movement direction of obstacles and other AGVs within the perception radius; the action space includes: left turn, right turn, forward, loading and unloading; the perception radius is used to define the observation range of the AGV;

[0041] The distributed conflict-free multi-AGV path planner includes a global network model and a local network model of each AGV; the global network model and the local network model of each AGV have the same network structure and correspond to an evaluation network, an attention network, a communication network, and a deep reinforcement learning network;

[0042] The evaluation network is used to fuse the features of other AGVs within the observation range of the AGV into the features of the AGV for each AGV to obtain a feature vector of the AGV, and determine the adjacency matrix of the AGV based on the feature vectors of all AGVs within the observation range of the AGV;

[0043] The attention network is used to calculate, for each AGV, a communication group of the AGV centered on the AGV based on the self-attention mechanism and according to the adjacency matrix of the AGV and the state observation value encoding of the AGV contained in the adjacency matrix of the AGV;

[0044] The communication network is used to perform network communication within a communication group centered on the AGV for each AGV, and update the state observation value code of each AGV in the communication group;

[0045] The deep reinforcement learning network is constructed based on a multi-agent asynchronous advantage actor-critic algorithm, and is used for AGVs that are divided into communication groups according to a dynamic grouping mechanism. Action decisions are made in a policy network according to the state observation value encoding of each AGV, path planning is completed, and rewards are obtained according to a reward function, with the goal of maximizing the reward. The model parameters of the local network model of each AGV are continuously iterated and updated, and the model parameters of the local network model of each AGV are used to update the model parameters of the global network model, wherein the deep reinforcement learning network includes a policy network and a critic network.

[0046] The embodiments of the present invention have the following technical effects: a dynamic grouping mechanism is introduced, multiple AGVs are dynamically grouped through an evaluation network and an attention network, and communication between AGVs is carried out through a communication network. The network parameters are trained in combination with deep reinforcement learning, which solves the dimensionality disaster problem that occurs when multiple AGVs make joint decisions during the conflict-free path planning process of multiple AGVs, and improves the efficiency of conflict-free path planning of multiple AGVs.

[0047] Furthermore, we construct a reward function for deep reinforcement learning, including:

[0048] The reward function is expressed as follows:

[0049] R(s,a)=R1+R2+R3 (1)

[0050] Among them, R1 represents the reward value obtained based on the distance between the AGV and the nearest obstacle, R2 represents the reward value obtained by the AGV reaching the target shelf position, and R3 represents the penalty value set for the time spent in the path planning process of the AGV, which is used to measure the cost paid by the AGV in the process of avoiding other intelligent agents and non-target shelves;

[0051] Among them, R1 is determined by formula (2) during the period when no collision occurs:

[0052] R1=u×obs_distance (2)

[0053] , and according to the following collision penalty correction formula (3), a penalty correction is performed on R1 when a collision is identified:

[0054]

[0055] Where u represents the reward and penalty factor set for the distance between the AGV and the obstacle, which is a positive value; obs_distance represents the distance between the current AGV and the nearest obstacle; the collision penalty correction formula represents that when the distance obs_distance from the AGV to the obstacle is less than or equal to the preset minimum collision distance dis crash When it is recognized as a collision, R1 is subtracted by 100, and the value of R1 is penalized and corrected;

[0056] The initial value of R2 is 0, and according to the following arrival reward formula (4), a reward correction R2 is given when it is identified as reaching the target shelf:

[0057]

[0058] Among them, tar_distance represents the distance between the current AGV and the target shelf, reach is the set arrival distance. When the distance between the robot and the target shelf tar_distance is less than the default arrival distance dis reach When , it means that the current AGV has completed the path planning and arrived at the target shelf, and is recognized as arriving at the target shelf;

[0059] R2' is the continuous arrival reward function, specifically formula (5):

[0060]

[0061] Among them, ep_count represents the cumulative value of the number of rounds in which the AGV reaches the target continuously. If the AGV encounters a conflict or collision during the simulation, or reaches the maximum training time step but does not reach the target shelf, ep_count is reset and accumulated again; k' represents the reward coefficient for the number of times the AGV reaches the target shelf continuously, ep_count;

[0062] Determine R3 according to formula (6):

[0063] R3=q×step_count (6)

[0064] Among them, q is the penalty coefficient based on arrival time, and step_count is the cumulative step value.

[0065] Furthermore, the construction introduces a distributed conflict-free multi-AGV path planner with a dynamic grouping mechanism, iteratively trains the distributed conflict-free multi-AGV path planner according to the reward function and the state observation value encoding and action space of each AGV, updates the network parameters of the distributed conflict-free multi-AGV path planner, and obtains the trained distributed conflict-free multi-AGV path planner, including:

[0066] Define a global iteration count, and set an initial value of the global iteration count to 0;

[0067] Increase the global iteration count by one;

[0068] Determine whether the global iteration count is less than a preset global iteration count threshold;

[0069] If it is determined that the global iteration count is less than a preset global iteration count threshold, the following steps are performed:

[0070] For each AGV in the virtual warehouse simulation training environment, the model parameters of the local network model corresponding to the AGV are synchronized using the model parameters of the global network model, and the time step count corresponding to the AGV is set to 0;

[0071] Determine the state observation value code of each AGV in the virtual warehouse simulation training environment according to the virtual warehouse simulation training environment information and the preset perception radius;

[0072] For each AGV in the virtual warehouse simulation training environment, a feature vector of the AGV is determined based on the state observation value encoding of the AGV and the evaluation network of the local network model of the AGV; the feature vector of the AGV represents the environmental information within the perception radius of the AGV;

[0073] Generate an adjacency matrix corresponding to the AGV according to the feature vector of the AGV; the adjacency matrix represents the adjacency relationship between all AGVs within the perception radius of the AGV;

[0074] According to the adjacency matrix corresponding to the AGV and the state observation value encoding of all AGVs in the adjacency matrix, based on the attention network of the local network model of the AGV, the similarity matrix corresponding to the AGV is obtained; according to the similarity matrix corresponding to the AGV, other AGVs having similar motion paths as the AGV are determined, and the AGV and the obtained other AGVs having similar motion paths as the AGV are combined to form a communication group centered on the AGV; wherein each row element in the similarity matrix represents the corresponding relationship between the AGV corresponding to the row and other AGVs having similar running paths;

[0075] Inputting the state observation value codes corresponding to all AGVs in the communication group centered on the AGV into the communication network of the local network model corresponding to the AGV for prediction, and obtaining the predicted state observation value code of the AGV;

[0076] Encoding the predicted state observation value of the AGV and inputting it into the policy network of the deep reinforcement learning network to determine the action to be performed by the AGV;

[0077] Execute the pending action of the AGV to make the AGV interact with the virtual warehouse simulation training environment, update the position information in the state observation value encoding of the AGV, and update the virtual warehouse simulation training environment information according to the updated position information of the AGV;

[0078] According to the state observation value code of the AGV before executing the action to be executed and the updated state observation value code of the AGV, based on a preset reward value calculation function, calculate the reward value corresponding to the current time step count corresponding to the AGV;

[0079] The time step count corresponding to the AGV is increased by one, and it is determined whether the position information in the updated state observation value code of the AGV is equal to the target shelf position in the state observation value code of the AGV, and whether the time step count corresponding to the AGV is greater than the preset local time step threshold; if it is determined that the position information in the updated state observation value code of the AGV is not equal to the target shelf position in the state observation value code of the AGV, and the time step count corresponding to the AGV is less than the preset local time step threshold, then the process returns to the step of determining the state observation value code of each AGV in the virtual warehouse simulation training environment according to the virtual warehouse simulation training environment information and the preset perception radius;

[0080] Determine the expected return of the AGV at the current time step count based on the position information and reward value of the AGV at the current time step count and based on the commentator network in the local network model of the AGV;

[0081] According to the expected return of the AGV at the current time step count, determine the expected return corresponding to each time step count before the current time step count, and according to the expected return corresponding to each time step count before the current time step count, perform gradient update on the model parameters in the local network model of the AGV; the model parameters in the local network model of the AGV include model parameters in the evaluation network, attention network, communication network and deep reinforcement learning network in the local network model;

[0082] According to the model parameters of the local network model of the AGV after the gradient update, the model parameters of the corresponding model in the global network model are updated, and the execution returns to the step of increasing the global iteration count by one; wherein the model parameters of the global network model include: the model parameters of the evaluation network, the attention network, the communication network, and the deep reinforcement learning network in the global network model;

[0083] If it is determined that the global iteration count is greater than or equal to a preset global iteration count threshold, the model parameters of the corresponding model in the global network model are saved.

[0084] Furthermore, the trained distributed conflict-free multi-AGV path planner is used to perform conflict-free path planning on multiple AGVs in an actual warehouse environment to obtain a conflict-free path for each AGV, including:

[0085] Acquire actual warehouse environment information, wherein the actual warehouse environment information includes the positions of multiple AGVs, obstacle positions, shelf positions, and road information in the actual warehouse environment;

[0086] Initialize the model parameters of the local network model of each AGV in the multiple AGVs in the actual warehouse environment using the model parameters of the global network model in the trained distributed conflict-free multi-AGV path planner;

[0087] According to the preset time step interval, periodically build a communication group centered on the AGV at each time step through the evaluation network and attention network of the local network model of each AGV in the actual warehouse environment, and update the state observation value code of each AGV in the communication group centered on the AGV through the communication network in the local network model of the AGV, input the updated state observation value code of the AGV into the policy network in the deep reinforcement learning network of the local network model of the AGV, obtain the action to be executed of the AGV corresponding to the time step, execute the action to be executed, and determine the position information of the AGV corresponding to the time step;

[0088] The AGV position information of each AGV in the actual warehouse environment at each time step is arranged in chronological order to obtain the conflict-free planning path of each AGV.

[0089] Further, according to the virtual warehouse simulation training environment information and the preset perception radius, the state observation value encoding of each AGV in the virtual warehouse simulation training environment is determined, including:

[0090] For each AGV in the virtual warehouse simulation training environment, the obstacles and other AGVs within the perception radius centered on the AGV are determined according to the virtual warehouse simulation training environment information, and the state observation value encoding of the AGV is constructed according to the obstacle position corresponding to the determined obstacle, the position information and movement direction of other AGVs.

[0091] Furthermore, for each AGV in the virtual warehouse simulation training environment, according to the state observation value encoding of the AGV, based on the evaluation network of the local network model of the AGV, a feature vector of the AGV is determined, including:

[0092] According to the state observation value coding of the AGV, all AGVs within the field of the AGV are determined, and the state observation value coding of all AGVs within the AGV is vectorized into corresponding initial eigenvalues; the neighborhood of the AGV represents the range covered by the preset perception radius of the AGV;

[0093] The initial eigenvalues ​​of all AGVs in the neighborhood of the AGV are saved as the last updated eigenvalues;

[0094] Set the initial value of the number of iterations;

[0095] Determine whether the number of iterations is greater than the preset number of iterations. If the number of iterations is greater than the preset number of iterations, end the iteration loop and use the initial feature value of the AGV after the iteration loop ends as the feature vector of the AGV. Otherwise, continue to execute the following iteration loop:

[0096] The initial eigenvalue of the AGV and the previously updated eigenvalues ​​corresponding to other AGVs in the neighborhood of the AGV are averaged, and then multiplied by the weight corresponding to the current number of iterations to obtain an aggregate value, and the aggregate value is nonlinearized by a preset nonlinear function to obtain a nonlinearized aggregate value, and the nonlinearized aggregate value is used to update the initial eigenvalue of the AGV;

[0097] Normalizing the initial characteristic value of the AGV, and using the normalized initial characteristic value of the AGV to update the initial characteristic value of the AGV;

[0098] For each other AGV in the field of the AGV, determining the positional relationship of the other VGA relative to the AGV;

[0099] When it is determined that the other AGV is in the forward area of ​​the AGV, the last updated characteristic value corresponding to the other AGV is updated using the value obtained by multiplying the initial characteristic value of the other AGV by the first coefficient;

[0100] In the case where it is determined that the other AGV is in the rearward area of ​​the AGV, the last updated characteristic value corresponding to the other AGV is updated using the value obtained by multiplying the initial characteristic value of the other AGV by the second coefficient;

[0101] Return to the step of determining whether the number of iterations is greater than the preset number of iterations;

[0102] Wherein, determining the positional relationship of the other VGAs relative to the AGV includes:

[0103] According to the state observation value coding of the AGV, the position information of other AGVs within the perception radius of the AGV is obtained, and the other AGVs within the perception radius of the AGV are divided into AGVs in the facing area of ​​the AGV and AGVs in the rearward area of ​​the AGV; wherein the facing area of ​​the AGV is defined as the first quadrant and the second quadrant of the coordinate system with the position information in the state observation value coding of the AGV as the origin; the rearward area of ​​the AGV is defined as the third quadrant and the fourth quadrant of the coordinate system with the position information in the state observation value coding of the AGV as the origin; the Y-axis of the coordinate system with the position information in the state observation value coding of the AGV as the origin is defined as a straight line pointing from the position information in the state observation value coding of the AGV to the target shelf position in the state observation value coding of the AGV, and the positive direction of the Y-axis is toward the target shelf position in the state observation value coding of the AGV, and the X-axis of the coordinate system with the position information in the state observation value coding of the AGV as the origin is perpendicular to the Y-axis;

[0104] Among them, the first coefficient is greater than the second coefficient, the first coefficient and the second coefficient are both greater than 0 and the sum of the first coefficient and the second coefficient is 1; the weights corresponding to the first coefficient, the second coefficient and all iteration numbers are model parameters of the evaluation network.

[0105] Furthermore, according to the feature vector of the AGV, an adjacency matrix corresponding to the AGV is generated, including:

[0106] The similarity values ​​of the feature vectors of all AGVs within the perception radius of the AGV are calculated, and the corresponding element positions of two AGVs whose similarity values ​​are greater than a preset adjacency similarity threshold in the adjacency matrix are set to 1, otherwise they are set to 0.

[0107] Further, according to the adjacency matrix corresponding to the AGV and the state observation value encoding of all AGVs in the adjacency matrix, based on the attention network of the local network model of the AGV, a similarity matrix corresponding to the AGV is obtained, and according to the similarity matrix corresponding to the AGV, other AGVs having similar motion paths as the AGV are determined, and the AGV and the obtained other AGVs having similar motion paths as the AGV are combined to form a communication group centered on the AGV, including:

[0108] Vectorizing the state observation value codes of all AGVs in the adjacency matrix corresponding to the AGV into corresponding code vectors, and arranging all the obtained code vectors in the order of the AGVs in the adjacency matrix to obtain an observation value code matrix;

[0109] Initialize the query matrix, key matrix and value matrix in the attention network in the local network model of the AGV using the observation value encoding matrix;

[0110] According to the following formula (7), the similarity matrix of the current time step is obtained:

[0111]

[0112] Where M is the adjacency matrix of the current time step; Q is the query matrix, K is the key matrix, d k is the dimension of the key matrix, T represents the transposed matrix of the current matrix;

[0113] The normalized similarity matrix is ​​calculated according to the following formula (8):

[0114]

[0115] Among them, the softmax function is a normalization function; Represents the interaction relationship between AGVs before normalization. It means the sum of the relationship between AGV No. i and other AGV No. k after exponential operation;

[0116] The other AGVs corresponding to the columns or rows corresponding to the elements with a value of 1 on the row or column corresponding to the AGV in Attention (Q, K, V) are divided into a communication group centered on the AGV.

[0117] Further, at a preset time step interval, a communication group centered on the AGV is periodically constructed at each time step through the evaluation network and the attention network of the local network model of each AGV in the actual warehouse environment, and the state observation value encoding of each AGV in the communication group centered on the AGV is updated through the communication network in the local network model of the AGV, and the updated state observation value encoding of the AGV is input into the policy network in the deep reinforcement learning network of the local network model of the AGV to obtain the action to be executed of the AGV corresponding to the time step, execute the action to be executed, and determine the position information of the AGV corresponding to the time step, including:

[0118] For each time step, the state observation value encoding of each AGV in the actual warehouse environment is determined according to the actual warehouse environment information and the preset perception radius; the actual warehouse environment information includes: the location information of multiple AGVs, obstacle locations, shelf locations and road information;

[0119] For each AGV in the actual warehouse environment, determine the feature vector of the AGV according to the state observation value encoding of the AGV and based on the evaluation network in the local network model of the AGV;

[0120] Generate an adjacency matrix corresponding to the AGV according to the feature vector of the AGV;

[0121] According to the adjacency matrix corresponding to the AGV and the state observation value encodings of all AGVs in the adjacency matrix, based on the attention network in the local network model of the AGV, a similarity matrix corresponding to the AGV is obtained;

[0122] According to the similarity matrix corresponding to the AGV, other AGVs having similar motion paths as the AGV are determined, and the AGV and the other AGVs having similar motion paths as the AGV are combined to form a communication group centered on the AGV;

[0123] Inputting the state observation value codes corresponding to all AGVs in the communication group centered on the AGV into the communication network in the local network model of the AGV for prediction, and obtaining the predicted state observation value code of the AGV;

[0124] Encoding the predicted state observation value of the AGV and inputting it into the policy network of the deep reinforcement learning network of the local network model of the AGV to determine the action to be performed by the AGV;

[0125] Execute the pending action of the AGV to enable the AGV to interact with the actual storage environment, update the position information in the state observation value code of the AGV, and update the actual storage environment information according to the updated position information in the state observation value code of the AGV;

[0126] Determine whether the position information in the state observation value code of the AGV is equal to the target shelf position in the state observation value code of the AGV;

[0127] If it is determined that the position information in the state observation value code of the AGV is not equal to the target shelf position in the state observation value code of the AGV, then return to the step of determining the state observation value code of each AGV in the actual storage environment for each time step according to the actual storage environment information and the preset perception radius.

[0128] The above technical solution of the embodiment of the present invention is described in detail below in conjunction with specific application examples. For technical details not introduced during the implementation process, please refer to the relevant description in the previous text.

[0129] The embodiment of the present invention provides a multi-AGV conflict-free path planning method based on deep reinforcement learning and dynamic grouping, simulates the actual warehouse picking scene, constructs a deep reinforcement learning multi-AGV conflict-free path planner that introduces a dynamic grouping mechanism, and solves the dimensional disaster problem of multi-AGV joint decision-making through a dynamic grouping mechanism. It includes:

[0130] Select all orders that have been processed by the warehouse in a certain period of time. After completing the task assignment, the system monitors the physical environment in the warehouse in real time through perception technology, including the current position of the AGV, the distribution of goods, obstacles, etc. The system needs to collect and manage various task requirements, including the type of task, the urgency of the task, the starting point and target location of the task, etc. Build a multi-AGV path planning model, considering multiple factors such as the number of AGVs, speed, task priority, physical environment, etc., in order to plan the path. Path planning includes path generation and real-time path adjustment. Path generation is to generate the best path for each AGV, ensure that the path length is minimized, and avoid conflicts between AGVs so that the task can be executed efficiently; real-time path adjustment is to monitor the position of the AGV and environmental changes during the task execution. If obstacles or emergencies occur, the path can be adjusted in time to deal with them.

[0131] Furthermore, the method also includes: solving the multi-AGV conflict-free path planning model by introducing deep reinforcement learning with a dynamic grouping mechanism, and the overall network architecture of the model includes input, dynamic grouping, collaborative communication and output. The input information of the model includes environmental information, real-time location information of the AGV and the perception radius of the AGV, and the dynamic grouping mechanism is introduced to complete the information interaction between the agents. In order to realize the dynamic grouping of AGVs, the model takes each AGV as the central agent and dynamically determines the communication group centered on the AGV. First, after evaluating the network, the label of the central agent is extracted from the features of the agents within the observation range of the central agent, and an adjacency matrix is ​​constructed at the same time. Assuming that there are N associated agent nodes in the system, each agent node has its own features, and the feature vectors of the agent nodes can form an N×D-dimensional matrix X. Through an N×N-dimensional adjacency matrix, the association relationship between the agent nodes can be clearly displayed. The model uses self-attention units to extract the feature vectors of the agents to calculate the similarity of the observation value encodings between the agents, and uses the adjacency matrix to filter out the agents that lack mutual correlation. Finally, the softmax function is used to calculate the attention weight values ​​between the agents to obtain the final AGV grouping information.

[0132] In order to achieve collaborative communication of AGVs, the model designs a communication network and a deep reinforcement learning network based on a centralized training and decentralized execution architecture. The central agent selects agents within the group to communicate in the communication network and outputs shared communication information, so that the agents within the group can obtain more comprehensive observation information. The agents within the group understand and infer the behavior of other AGVs in the group based on comprehensive observation information, cooperate with other AGVs to make decisions and help each other reach the target shelf, output the action sequence of the AGV, obtain rewards, and accumulate and maximize the rewards of all AGVs in the system.

[0133] like Figure 8 As shown in the figure, the model network structure including evaluation network, attention network, communication network and deep reinforcement learning network is designed, including:

[0134] The evaluation network mainly completes the feature extraction of the central agent and its neighboring agents, predicts the label of the central agent through the feature vectors of the agents within the central agent's observation range, and constructs the adjacency matrix. In the multi-agent cooperative planning problem, it is assumed that the task scenario is a two-dimensional environment (W×H) with obstacles. Assume V = {v1,v2,...,v N} is the set of N agents in the environment, N(v i ) is the agent {v i ,i=1,2,...,N}, the perception radius of each agent is r ob , the observation range matrix of the agent on the map at time t is Where W ob and H ob Represent the width and height of the agent’s observation range respectively. The communication radius of the agent is r com , the communication network is C(V,ε t ),in is a V×V matrix, if (v i ,v j )∈ε t , then it represents v i and v j At the same time, if two agents can communicate with each other at time t, it means ||P i -P j ||≤r com Among them, P i ,P j ∈R 2 , P i ,P j Represents the agent v i ,v j The goal of the model is to effectively navigate the agents to their respective target locations without conflict. The target path is connected by the sequence of actions of the agent at each moment. The evaluation network needs to create a mapping function F to convert the agent's observation range matrix Convert to decision That is C t Represents the communication network C at time t. In order to effectively aggregate the agents in the two-dimensional space, the model uses the mean aggregation function to aggregate information and outputs the feature vector of the agent (AGV).

[0135] like Figure 2 As shown in the figure, the main idea of ​​the evaluation network is that in each iteration, the agent continuously aggregates the state information of the agents within the observation range. In the process of continuous iteration, the agent can continuously accumulate more and more feature vectors. The forward propagation algorithm can be divided into three steps. First, randomly sample the agents adjacent to the central agent; second, summarize the environmental information of the sampled agents from the outside to the inside. Finally, use the aggregated information as the input of the fully connected layer to obtain the label of the central agent. The model obtains the weighted cognition of the agent on the environment and the similarity between the observed value encodings by referring to the self-attention mechanism.

[0136] like Figure 3As shown in the figure, the environmental information of the agents in the system changes dynamically in each training cycle. When the model implements dynamic grouping, it is necessary to introduce the observable characteristics of the target agent itself, that is, the specific position relationship of the agent to screen the neighboring agents of the target agent. First, the evaluation network defines the "toward area" and the "backward area" according to the movement direction of the agent. Secondly, a rectangular coordinate system is constructed based on the above areas, with the central agent as the origin, and the line connecting the central agent and its target position point as the vertical axis. During the movement of the agent, the direction facing the target location is defined as the positive direction of the vertical axis of the coordinate axis. In this coordinate system, the first and second quadrants are the facing area, and the third and fourth quadrants are the backward area. The pseudo code of the evaluation network algorithm in the model is shown in Table 1. The feature vector of the central agent is determined by the following steps:

[0137] According to the state observation value coding of the AGV (here equivalent to the central intelligent agent), all AGVs (here equivalent to all intelligent agents) within the domain of the AGV (here equivalent to the central intelligent agent) are determined, and the state observation value coding of all AGVs within the AGV (here equivalent to the central intelligent agent) is vectorized into corresponding initial eigenvalues; the neighborhood of the AGV (here equivalent to the central intelligent agent) represents the range covered by the preset perception radius of the AGV (here equivalent to the central intelligent agent);

[0138] The initial feature values ​​of all AGVs (here equivalent to all agents) in the neighborhood of the AGV (here equivalent to the central agent) are saved as the last updated feature values;

[0139] Set the initial value of the number of iterations; it can be set to 0 or 1 to control the number of iterations; the preset number of iterations can be used as a hyperparameter for debugging during the actual training process;

[0140] Determine whether the number of iterations is greater than the preset number of iterations. If the number of iterations is greater than the preset number of iterations, the iteration loop ends, and the initial feature value of the AGV (here equivalent to the central intelligent agent) after the iteration loop ends is used as the feature vector of the AGV (here equivalent to the central intelligent agent). Otherwise, continue to execute the following iteration loop:

[0141] The initial feature value of the AGV (here equivalent to the central intelligent agent) and the last updated feature values ​​corresponding to other AGVs (here equivalent to other intelligent agents) in the neighborhood of the AGV (here equivalent to the central intelligent agent) are averaged, and then multiplied by the weight corresponding to the current number of iterations to obtain an aggregate value, and the aggregate value is nonlinearized by a preset nonlinear function to obtain a nonlinearized aggregate value, and the nonlinearized aggregate value is used to update the initial feature value of the AGV (here equivalent to the central intelligent agent);

[0142] Normalizing the initial feature values ​​of the AGV (equivalent to the central intelligent agent here), and using the normalized initial feature values ​​of the AGV (equivalent to the central intelligent agent here) to update the initial feature values ​​of the AGV (equivalent to the central intelligent agent here);

[0143] For each other AGV (equivalent to other agents) within the area of ​​the AGV (equivalent to the central agent), determine the positional relationship of the other VGA (equivalent to other agents) relative to the AGV (equivalent to the central agent);

[0144] When it is determined that the other AGV (here equivalent to other intelligent agents) is in the forward area of ​​the AGV (here equivalent to the central intelligent agent), the last updated characteristic value corresponding to the other AGV (here equivalent to other intelligent agents) is updated using the value obtained by multiplying the initial characteristic value of the other AGV (here equivalent to other intelligent agents) by the first coefficient;

[0145] When it is determined that the other AGV (here equivalent to other intelligent agents) is in the rearward area of ​​the AGV (here equivalent to the central intelligent agent), the last updated characteristic value corresponding to the other AGV (here equivalent to other intelligent agents) is updated using the value obtained by multiplying the initial characteristic value of the other AGV (other intelligent agent) by the second coefficient;

[0146] Return to the step of determining whether the number of iterations is greater than the preset number of iterations;

[0147] Wherein, determining the positional relationship of the other VGAs (here equivalent to other intelligent agents) relative to the AGV (here equivalent to the central intelligent agent) includes:

[0148] According to the state observation value coding of the AGV, the position information of other AGVs within the perception radius of the AGV is obtained, and the other AGVs within the perception radius of the AGV are divided into AGVs in the facing area of ​​the AGV and AGVs in the rearward area of ​​the AGV; wherein the facing area of ​​the AGV is defined as the first quadrant and the second quadrant of the coordinate system with the position information in the state observation value coding of the AGV as the origin; the rearward area of ​​the AGV is defined as the third quadrant and the fourth quadrant of the coordinate system with the position information in the state observation value coding of the AGV as the origin; the position information in the state observation value coding of the AGV is The Y axis of the coordinate system of the origin is defined as a straight line pointing from the position information in the state observation value encoding of the AGV to the target shelf position in the state observation value encoding of the AGV, and the positive direction of the Y axis is toward the target shelf position in the state observation value encoding of the AGV, and the X axis of the coordinate system of the origin with the position information in the state observation value encoding of the AGV is perpendicular to the Y axis; wherein, the first coefficient is greater than the second coefficient, the first coefficient and the second coefficient are both greater than 0 and the sum of the first coefficient and the second coefficient is 1; the weights corresponding to the first coefficient, the second coefficient and all iterations are model parameters of the evaluation network. The nonlinear function includes but is not limited to the ReLu function, the Sigmoid function or the Tanh function.

[0149]

[0150] Table 1 Evaluation network algorithms

[0151] The attention network module determines the similarities and differences between different agents by analyzing the agent feature information. It focuses on the key features of the agent, such as position, speed, state, etc., to identify patterns and trends related to environmental interaction, so that the system can accurately perceive the state and behavior of the agent, and make appropriate preparations for subsequent collaborative communication. Through dynamic grouping, the attention network module can classify agents with similar characteristics or behaviors into the same group to create a more organized agent network, making collaborative communication more efficient. For example, dividing agents with similar tasks or goals into the same group can promote information exchange and collaboration between them. Figure 4 The attention network structure is shown. M is the adjacency matrix extracted by the evaluation network from the feature vector of the agent (AGV). Matrix M indicates that AGV No. 1 can observe AGV No. 3. Similarly, AGV No. 3 can also observe AGV No. 1; AGV No. 1 cannot obtain the feature information of AGV No. 2.

[0152]

[0153] The model obtains the weighted cognition of the agent to the environment and the similarity between the observation encodings by referencing the self-attention mechanism. Among them, the matrix X represents the feature vector transformed from the observation encoding matrix of the agent. In the initial state, the query matrix Q, key matrix K and value matrix V in the self-attention mechanism are the same as the observation encoding matrix X. The similarity value of any two AGV observation encoding matrices can be obtained.

[0154] The similarity matrix of the current time step is obtained according to formula (7), where M is the adjacency matrix of the current time step; Q is the query matrix, K is the key matrix, d k is the dimension of the key matrix, T represents the transposed matrix of the current matrix;

[0155] The normalized similarity matrix is ​​calculated according to formula (8), where the softmax function is a normalization function; Represents the interaction relationship between AGVs before normalization. It means the sum of the relationship between AGV No. i and other AGV No. k after exponential operation;

[0156] The other AGVs corresponding to the columns or rows corresponding to the elements with a value of 1 on the row or column corresponding to the AGV in Attention (Q, K, V) are divided into a communication group centered on the AGV.

[0157] Based on Attention(Q,K,V), when the value of the element in the i-th row and j-th column is equal to 0, AGV i cannot obtain the feature vector of AGV j, such as position, distance and other information; when the value of the element in the i-th row and j-th column is not equal to 0, AGV i can obtain the information of AGV j.

[0158] The communication network module is a key tool for information exchange between intelligent agents. It can help intelligent agents establish effective communication channels and promote cooperation and collaborative work. Figure 5 As shown in the figure, the bidirectional LSTM unit embedded in it is a neural network unit that can capture the characteristics of sequence data. Through its bidirectional recursive structure, it can comprehensively utilize past and future information. It has the ability to selectively output information that promotes cooperation, thereby helping the agent to make more intelligent decisions based on the understanding and prediction of the behavior of other agents. The agent will select group members from the surrounding agents and establish a communication group. When multiple groups select agent i at the same time, the model allows information to flow between different groups. Formula (9) and Formula (10) indicate that agent i first updates the information of the agent in group P once, and then agent i joins the Q communication group and updates its own parameters again. The last updated parameter information will directly affect the parameter update of other agents in group Q.

[0159]

[0160] In the above formula, C represents the communication network. The communication network uses a bidirectional LSTM network. The bidirectional LSTM unit can selectively output information that promotes cooperation and helps the agents make decisions based on understanding and predicting the behavior of other agents.

[0161] In a deep reinforcement learning network, such as Figure 6 As shown in the figure, the global network is a grid model shared by AGVs in the warehouse environment, including the Actor network (strategy network) and the Critic network (commentator network). There are n independent sub-threads under the global network, each sub-thread corresponds to an AGV, and the network structure in each sub-thread is the same as the network structure of the public neural network. During the training process, each sub-thread interacts with the environment independently to obtain experience values, and does not affect each other.

[0162] Since the strategies of related agents in a multi-agent system will affect the final optimal strategy of the current agent, when traditional methods apply reinforcement learning algorithms to solve problems, they exceed the solving capabilities of single-agent Markov stored processes. The model combines the characteristics of multi-AGV systems, considers dynamic environmental changes, and uses deep reinforcement learning with a dynamic grouping mechanism to construct a multi-AGV path planning framework. In this architecture, AGVs can make decisions multiple times in different time periods. In the same time period, the spatial positions of AGVs are used for dynamic grouping. The AGVs in the group are expected to achieve correlation equilibrium so that all AGVs can obtain efficient and conflict-free path planning, which greatly reduces the decision space and solves the dimensionality explosion problem of multi-AGV system path decisions. In the deep reinforcement learning network, each AGV will establish a centralized discriminator Critic, which can obtain all global states and action sets of all AGVs in the group, and output the corresponding value function Q(x,a1,..,a n), alleviating the problem of unstable environment in multi-agent system. At the same time, the Actor network (strategy network) of each AGV only needs to make decisions based on local observation information, realizing distributed management of multi-agent system. Centralized Q-value learning and distributed strategy execution are used to solve the problem. The Q-value can obtain the observation information o and action a of all AGVs in the group, and the strategy π outputs the action of the AGV according to the environmental observation information of each AGV. The sub-thread uses the interaction data with the environment to estimate the loss function of its own neural network, and uses the estimation results to update the parameters of the public neural network model, continuously improving the reliability and accuracy of the model training results. As the number of training times increases, the sub-thread continuously adjusts the parameters of its internal neural network to keep pace with other neural networks. Through interaction with the external environment, the network model in the sub-thread can effectively help itself and other public network models to achieve optimization. During the training process, different agents learn to obtain parameterized experience in interaction with the environment, and each sub-thread will copy the training parameters of the global network before working. After the AGV in the warehouse environment interacts with the environment, the algorithm will calculate the gradient and transmit the gradient back to the central control center to further update the parameters of the global network. During the execution of the algorithm, each actor obtains parameters from the central control center for training, and then the parameters are transmitted back to the central control network after training.

[0163] Furthermore, a multi-agent asynchronous advantage actor-critic algorithm is used to solve the multi-AGV conflict-free path planning model of deep reinforcement learning with dynamic grouping mechanism, and a simulation experiment platform is developed to verify the effectiveness of the model and algorithm. Effectiveness experiments are conducted on multi-AGV warehouse environments of different scales; comparative experiments are conducted on whether to dynamically group AGVs in the simulation environment and multiple deep reinforcement learning solution algorithms.

[0164] 1.1 Algorithm Design: In the case of multi-agent games, the state space and action space of the environment often grow exponentially. When solving the equilibrium solution, if the traditional value function-based training algorithm is used, the algorithm training time is often very long. In order to solve this problem, the embodiment of the present invention creates a strategy-based approximate solution algorithm MAA3C algorithm, and each agent in the system can obtain its own optimal strategy.

[0165] 1.1.1 Multi-agent Asynchronous Advantage Actor-Critic Algorithm: The Actor-Critic Algorithm combines the temporal difference algorithm with the policy gradient algorithm to solve the path planning problem of multi-agents. The Multi-Agent Asynchronous Advantage Actor-Critic (MAA3C) algorithm enriches the theoretical results of the traditional single-agent Actor-Crtic algorithm and extends it to the process of multi-agent games. During the training process, different agents learn to gain parameterized experience in the interaction with the environment, and each sub-thread will copy the training parameters of the global network before working. After the AGV in the warehouse environment interacts with the environment, the algorithm will calculate the gradient and transmit the gradient back to the central control center to further update the parameters of the global network. During the execution of the algorithm, each actor obtains parameters from the central control center and then trains, and then transmits the parameters back to the central control network after training.

[0166] 1.1.2 Solution process: The embodiment of the present invention adopts the MAA3C algorithm to solve the deep reinforcement learning multi-AGV conflict-free path planning model with the introduction of a dynamic grouping mechanism. The input of the MAA3C algorithm is the observation information of the agent, including environmental information and feature information of other agents; the output of the algorithm is the optimal strategy π of the agent. * (s), which is the optimal action value function Q * (s, a) corresponds to the action set. During the algorithm training process, AGV can interact with the environment and obtain changing and unknown environmental information in the surrounding environment. The algorithm solution process is as follows Figure 7 shown.

[0167] First, the system obtains the current state of the AGV as input, and the Actor network outputs the parameters corresponding to the normal distribution. After sampling, the action a of the AGV in the current state is obtained. t Second, the AGV performs action a in step 1. t , the environment feedbacks the state s at the next moment t+1 The Critic network gives an immediate reward r based on the actions and states taken by the agent. t . Third, the Actor network updates parameters based on the immediate reward and combines s t+1 Output new parameters σ, μ, and sample to get the current action a t+1 Fourth, AGV performs action a t+1 , iterate continuously according to steps 2 and 3, and repeat the process until conflict-free path planning is completed.

[0168] 1.2 Experimental design and result analysis: The study conducts effectiveness experiments on multi-AGV application environments of different scales; and conducts comparative experiments on whether to dynamically group AGVs in the simulation environment and multiple deep reinforcement learning solution algorithms.

[0169] 1.2.1 Experimental design: Based on the above experimental running conditions, the embodiment of the present invention uses python code to simulate the warehouse environment and develops a set of experimental simulation environments - multi-AGV warehouse simulation environment. The experimental environment developed by the embodiment of the present invention simulates the warehouse environment of the real world, which provides a guarantee for the implementation of the model and algorithm. In the simulation environment, each AGV is regarded as a particle. The experiment sets a certain number of training rounds. The AGV starts from the initial point, processes the data in each round and inputs it into the model. In the continuous iteration of the algorithm, conflict-free path planning is performed to reach the target shelf position. If the AGV conflicts with other AGVs during the training process or completes the path optimization to reach the target shelf, the round ends. The code follows this logic and keeps looping until the maximum number of training rounds set by the system is reached. In order to prevent the problem of invalid training caused by the inability of the algorithm to converge, the maximum time step threshold is set during the training process to avoid invalid training time. In a multi-robot warehouse environment, the robot's observation range includes all relevant information of the units adjacent to the robot, which is partially observable. By default, the observation range of each agent is the information within a 3×3 square centered on the agent. The robot's visible range can be modified as an environmental parameter. The agent can obtain its own position, rotation and load, as well as the position information of other agents within the visible range and the shelves around the robot. The action space in the multi-robot warehouse environment has four discrete actions to choose from: In this environment simulation, the robot has the following discrete action space: A = {Turn Left, Turn Right, Forward, Load}. The first three actions allow the robot to rotate and move forward. Loading and unloading are only valid when the robot is under the target shelf. When the AGV does not carry any goods, it can shuttle freely under the shelf; but when the AGV carries the shelf, the AGV must travel through the visible channel in the figure. The AGV cannot move to an occupied cell when moving forward or rotating in the grid world. Load the shelf when the AGV is at the location of the target shelf. Fig.10 This is a schematic diagram of the warehouse simulation training environment, where square P1 is the picking station, P2 is the shelf, the total number of AGVs is 8, 1 (rectangular) AGV is not carrying a shelf, 7 (hexagonal) AGVs are carrying shelves, and the number of target shelves P3 requested is 3. The AGV needs to complete conflict-free path planning from the picking station to reach the target shelf and get a reward. This article uses the total cumulative rewards of all agents as the performance indicator. The following is a detailed description of the warehouse environment:

[0170] (1) Observation space: In a multi-robot warehouse environment, the robot's observation range includes all relevant information of the units immediately adjacent to the robot and is partially observable. By default, the observation range of each agent is the information within a 3×3 square centered on the agent. The robot's visible range can be modified as an environmental parameter. The agent can obtain its own position, rotation, and load, as well as the position information of other agents within the visible range and the shelves around the robot. The following array is the detailed information encoding the state observation values ​​of the two AGVs:

[0171] [[8,3,0,0,1,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,1,0,1,0,0,0,0,0,0,1,0,0,0,0,1,0,1,0,1,0,0,1,0,0,1,0,0,1,0,0,1,0,0,0,0,0,1,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,0,0],

[0172] [4,9,0,0,0,1,0,1,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0],...]

[0173] Each element in the array is an observation corresponding to each agent. The first two values ​​correspond to the x and y coordinates of the AGV itself. The third is encoded as "1" or "0", depending on whether the AGV is currently carrying a shelf. If it is carrying a shelf, it is 1, and if it is not carrying a shelf, it is 0. The next four values ​​are the one-hot encoding of the current AGV's facing direction, including up, down, left, and right. For example, (1,0,0,0) can be used to represent up, (0,1,0,0) can be used to represent down, (0,0,1,0) can be used to represent left, and (0,0,0,1) can be used to represent right. Then, a single value of "1" or "0" indicates whether the AGV is in the channel position. If it is a channel, the shelf is not allowed to be placed at this position. The remaining values ​​in the encoding can be divided into 9 groups of 7 elements, each of which corresponds to a square in the observation radius (within the 3x3 observation range centered on the agent, which can be customized according to the parameters). In each group of elements, the first element is 1 or 0, indicating whether there is an AGV at the location; if there is an AGV, it is followed by a unique hot code indicating the driving direction of the adjacent AGV; the sixth element indicates whether there is a shelf at the location, and if there is a shelf, the seventh element indicates whether the shelf is the target shelf.

[0174] (2) Action space: The action space in the multi-robot warehouse environment has four discrete actions to choose from: In this environment simulation, the robot has the following discrete action space: A = {Turn Left, Turn Right, Forward, Load}, the first three actions allow the robot to rotate and move forward. Loading and unloading are only valid when the robot is under the target shelf. When the AGV does not carry any goods, it can shuttle freely under the shelf; but when the AGV carries a shelf, the AGV must travel in the aisle visible in the figure. The AGV cannot move to an occupied cell when moving forward or rotating in the grid world. Load the shelf when the AGV is at the location of the target shelf.

[0175] (3) Reward function: If the AGV successfully reaches the target shelf, it will receive a reward. A major challenge in the environment is how the AGV can complete path planning to achieve efficient and conflict-free arrival at the target shelf. The training goal of reinforcement learning is to maximize the cumulative reward. In reinforcement learning, the reward function can quantitatively evaluate the learning effect of the AGV. Therefore, the design of the reward function is an important factor affecting the effect of reinforcement learning and plays an extremely important role in the final training results. The reward function R(s,a) can be expressed by formula (1); the R1 part of formula (1) describes the shortest distance between the AGV and the obstacle. In formula (2) of R1, obs_distance represents the distance between the current AGV and the nearest obstacle, and u is the reward and penalty factor set for the distance between the AGV and the obstacle. In order for the agent to successfully avoid obstacles, a safe distance needs to be defined. The experiment sets u to a positive value. From the formula, it can be seen that the greater the distance between the obstacle agent and the current AGV, the more rewards the reward function generates and the better the effect. R2 represents the reward for the AGV to reach the target shelf position. In the simulation environment, a small arrival distance is set. If the distance between the AGV and the target shelf is less than the set value, it means that the AGV has completed the conflict-free path planning and reached the target shelf. If the AGV can reach the target shelf several times in a row, it means that during the training process, the AGV is using the experience it has learned. The experiment can optimize the reward function to improve the reward level for the AGV so that the AGV can complete the path planning better. In the simulation experiment, in order to make the agent path planning more efficient and minimize the cost, the experiment sets a reward function to punish the number of steps the agent takes to avoid obstacles. The third part of formula (1), R3, represents the penalty function set for the time spent in the AGV path planning process. R3 is defined as q×step_count, where q is the penalty coefficient based on the arrival time, and step_count is the cumulative step value, which is used to measure the cost paid by the AGV in the process of avoiding other agents and non-target shelves. Therefore, the reward function sets a penalty factor negative q. The larger the cumulative time step step_count during the training process, the larger the penalty imposed on the AGV by the reward function. The reward function is further set based on whether the robot can successfully avoid other intelligent agents on non-target shelves, whether it can reach the target shelf, whether it takes less time, etc., so that the robot can eventually achieve conflict-free path planning in a complex environment with a smooth path and less time overhead. Regarding the obstacle avoidance level, if the robots are always within a safe distance or between the robot and the obstacle, the AGV will receive a reward. If the distance between the robots or between the robot and the shelf is less than the safe distance, a collision occurs by default and the robot will be penalized. According to the collision penalty correction formula (3), a penalty correction is performed on R1 when a collision is identified, and the definition of discrash is the safety distance. If the distance between two AGVs is less than the safety distance, it can be considered that there is a conflict between the AGVs. According to the setting standard of the reward function, the agent is punished. The reward and punishment for whether the AGV can complete the conflict-free path planning during the training process is shown in formula (4). Among them, tar_distance represents the distance between the current AGV and the target shelf, and dis reach is the set arrival distance. When the distance between the robot and the target shelf tar_distance is less than the default arrival distance dis reach , it means that the current AGV has completed the path planning and reached the target shelf. In order to further enable the AGV to reach the target shelf position more quickly and continuously, the continuous arrival reward function R2' is adopted. The specific logic is shown in formula (5). Among them, ep_count represents the cumulative value of the number of rounds in which the AGV reaches the target continuously. If the AGV encounters a conflict or collision during the simulation process or reaches the maximum training time step but does not reach the target shelf, ep_count is reset and accumulated again. The k' in formula (5) represents the reward coefficient for the number of times the AGV reaches the target shelf continuously ep_count in the simulation environment.

[0176] In the simulation experiment, the network structure of the policy network is three layers. The number of neurons in the middle layer of the neural network is set to 128, and the last layer of the neural network outputs the action probability that the AGV can choose. The hyperparameters required during the experiment are shown in Table 2.

[0177] Parameter name describe value hiddendimension Hidden layer dimensions 64 / 128 learningrate Learning Rate 0.0005 rewardStandardisation Reward Standardization true networktype Network Type Fully Connected Network (FC) entropycoefficient Entropy coefficient 0.01 targetupdate Objective function update 0.01(soft update) n-step Maximum number of steps per round 5 γ Discount Factor 1.0 epochsize The number of iterative updates in an epoch 10 batchsize The number of steps taken to update 500

[0178] Table 2 Hyperparameters in the algorithm

[0179] The experiments were replicated using random number seeds and 95% confidence intervals were calculated for the returns. The parameters of the different algorithms were optimized for the different environments used, and kept constant for the rest of the tasks in the same environment. Each combination of hyperparameters was evaluated for different seeds, and the optimal hyperparameter combination was obtained to produce the results to be evaluated. In general, all algorithms were evaluated with approximately the same number of hyperparameter combinations in each environment to ensure consistency.

[0180] Simulation experiment environment settings: The simulation experiment can customize the size of the warehouse, the number of AGVs in the system, the number of shelves, and the location of the target shelves. By default, the number of AGVs in the system is equal to the number of target shelves. The embodiment of the present invention verifies the effectiveness of the MAA3C algorithm by conducting experiments in three warehouses of different sizes (Warehouse 1, Warehouse 2, Warehouse 3). The three warehouses contain 9, 25, and 49 AGVs (intelligent agents) respectively, and the environmental parameters are shown in Table 3.

[0181] Warehouse parameter description parameter Warehouse 1 Warehouse 2 Warehouse 3 Warehouse Code Grid_Code warehouse1 warehouse2 warehouse3 Warehouse size Grid_Sacre small medium large Number of columns in the warehouse Shelf_Colums 3 5 7 Column height in warehouse Column_Height 3 5 7 Number of rows in the warehouse Shelf_Rows 3 5 7 Number of Agents Agent_Number 9 25 49 Target number of shelves Request_Sizes 9 25 49 Agent observation range Sensor_Range 3*3 3*3 3*3 Grid World Height Grid_Height 14 32 56 Grid world width Grid_Width 10 16 22 Grid world size Grid_Size 140 512 1232

[0182] Table 3 Environmental parameters of warehouses of different sizes

[0183] The height formula of the grid world is shown in formula (11):

[0184] Grid_Weight=(2+1)*Shelf_Columns+1 (11)

[0185] The grid world width formula is shown in formula (12):

[0186] Grid_Height=(Column_Height+1)*Shelf_Rows+2 (12)

[0187] According to the parameter input algorithm in the above table, the simulation maps of the virtual warehouse simulation training environment of warehouse 1, warehouse 2 and warehouse 3 can be obtained, such as Fig.11 , Fig.12 and Fig.13 As shown in the figure, the squares filled with diagonal lines represent shelves, the blank squares represent roads, the squares filled with plus signs represent target shelves, and the hexagons represent AGVs.

[0188] 1.2.2 Experimental results and analysis: Taking the experimental environment of warehouse 1 as an example, each grid in the simulation environment is encoded during the experiment to generate a quantitative encoding map of the simulation environment of warehouse 1, such as Fig.14 shown.

[0189] In the simulation environment of warehouse 1, 9 AGVs process 9 target tasks accordingly. Based on the multi-AGV conflict-free path planning model and the MAA3C algorithm, the operation trajectory of each AGV under different task numbers can be obtained, as shown in Table 4.

[0190]

[0191] Table 4 AGV path trajectory table in simulated warehouse 1

[0192] Fig. 9The reward iteration graph generated by training in the simulation experiment of multi-AGV conflict-free path planning task is described, showing the trend of the overall reward of the warehouse system with the number of training steps in simulation environments of different scales. Among them, the horizontal axis represents the time step in the training process, and the vertical axis is the reward value corresponding to each time step. The overall reward curves of warehouse 1, warehouse 2 and warehouse 3 correspond to the curves warehouse1-MAA3C, warehouse2-MAA3C and warehouse3-MAA3C in the figure respectively. From the convergence value returned by the agent, it can be seen that in simulation experimental environments of different scales, the overall reward of the system can achieve final convergence with the increase of training time steps. In the initial round, the reward value is relatively low. With the increase of training steps, the reward obtained by the agent in the process of reaching the target shelf gradually increases, and the overall reward value of the system gradually increases until the final convergence. Due to the setting of the reward function, with the increase of the number of AGVs in the simulation environment and the increase of the environment scale, the reward value of the system fluctuates; however, the increase of the simulation environment scale has no effect on the policy convergence of AGV, and the global policy remains stable, which verifies the effectiveness of the model and algorithm. Based on the model structure analysis, the centralized critic network input includes the characteristics of all AGVs themselves and the local observation characteristics of all AGVs on the environment. In the strategy evaluation process, the model effectively solves the instability problem of the system in the multi-agent warehouse environment based on the above input information. The centralized model network structure improves the strategies of each agent in the system and can achieve overall model convergence.

[0193] The deep reinforcement learning multi-AGV conflict-free path planning model with the introduction of a dynamic grouping mechanism proposed in the embodiment of the present invention adopts the idea of ​​dynamic grouping to realize the information interaction of related AGVs in the warehousing environment, and designs the MAA3C algorithm to solve the model. Therefore, the comparative experiment is carried out from two dimensions: one is whether to introduce the dynamic grouping mechanism of AGV in the experimental simulation environment, and the other is to solve and compare the model based on the algorithm dimensions provided above using different deep reinforcement learning algorithms. The experiment will use the above two dimensions to solve the conflict-free path planning problem of AGV for warehousing environments of different sizes. Tables 5 and 6 show the comparison of the average return values ​​obtained by seven algorithms, namely VDN (Value Decomposition Network), QMIX (Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning), COMA (Counterfactual Multi-Agent Policy Gradients), MADDPG (Multi-Agent Deep Deterministic Policy Gradient), MAPPO (Multi-Agent Proximal Policy Optimization), MAA2C ​​(Multi-Agent Advantage Actor Critic) and MAA3C (Multi-Agent Asynchronous Advantage Actor-Critic), during the code training process without considering dynamic grouping and considering dynamic grouping. By comparing the training process data of agents in different algorithms under non-dynamic grouping and dynamic grouping, it can be found that the average returns of all algorithms under dynamic grouping are higher than those without dynamic grouping in most cases, and the larger the scale, the more obvious the difference. This shows that the introduction of dynamic grouping is very effective in solving large-scale AGV path planning problems in warehousing environments. Regardless of whether dynamic grouping is adopted, the MAA3C algorithm has the highest average return value compared to other algorithms; at the same time, the effect is more significant as the scale increases. This is because the MAA3C algorithm uses a joint action value function to train agents. Compared with the local action value function, centralized training fully considers the behavior of other agents and realizes joint strategy evaluation between agents.The experimental results further illustrate the effectiveness and feasibility of using the MAA3C algorithm with dynamic grouping to solve the model.

[0194]

[0195] Table 5 Average returns and 95% confidence intervals of different algorithms under non-dynamic grouping

[0196]

[0197] Table 6 Average returns and 95% confidence intervals of different algorithms under dynamic grouping

[0198] The embodiments of the present invention have the following technical effects: in the multi-AGV path planning in a warehousing environment, a dynamic grouping mechanism is introduced, multiple AGVs are dynamically grouped through an evaluation network and an attention network, and communication between AGVs is carried out through a communication network. In combination with deep reinforcement learning, network parameters are trained, which solves the dimensionality disaster problem that occurs during joint decision-making in the process of multi-AGV conflict-free path planning, and improves the efficiency of multi-AGV conflict-free path planning.

[0199] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0200] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0201] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0202] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including". In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0203] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A deep reinforcement learning multi-AGV conflict-free path planning method for a warehouse environment, characterized in that: include: Creating a virtual warehouse simulation training environment, and extracting virtual warehouse simulation training environment information from the virtual warehouse simulation training environment; Determine, according to the virtual warehouse simulation training environment information, a state observation value code of each AGV among a plurality of AGVs in the virtual warehouse simulation training environment; Set the action space of AGV and construct the reward function of deep reinforcement learning; Constructing a distributed conflict-free multi-AGV path planner that introduces a dynamic grouping mechanism, iteratively training the distributed conflict-free multi-AGV path planner according to the reward function and the state observation value encoding and action space of each AGV, updating the network parameters of the distributed conflict-free multi-AGV path planner, and obtaining a trained distributed conflict-free multi-AGV path planner; Using the trained distributed conflict-free multi-AGV path planner to perform conflict-free path planning on multiple AGVs in an actual warehouse environment to obtain a conflict-free path for each AGV; The virtual warehouse simulation training environment information includes the initial positions, obstacle positions, shelf positions and road information of multiple AGVs in the virtual warehouse simulation training environment; the state observation value encoding of the AGV includes: the position information and movement direction of the AGV, the target shelf position, whether there are goods, and the position information and movement direction of obstacles and other AGVs within the perception radius; the action space includes: left turn, right turn, forward, loading and unloading; the perception radius is used to define the observation range of the AGV; The distributed conflict-free multi-AGV path planner includes a global network model and a local network model of each AGV; the global network model and the local network model of each AGV have the same network structure and correspond to an evaluation network, an attention network, a communication network, and a deep reinforcement learning network; The evaluation network is used to fuse the features of other AGVs within the observation range of the AGV into the features of the AGV for each AGV to obtain a feature vector of the AGV, and determine the adjacency matrix of the AGV based on the feature vectors of all AGVs within the observation range of the AGV; The attention network is used to calculate, for each AGV, a communication group of the AGV centered on the AGV based on the self-attention mechanism and according to the adjacency matrix of the AGV and the state observation value encoding of the AGV contained in the adjacency matrix of the AGV; The communication network is used to perform network communication within a communication group centered on the AGV for each AGV, and update the state observation value code of each AGV in the communication group; The deep reinforcement learning network is constructed based on a multi-agent asynchronous advantage actor-critic algorithm, and is used for AGVs that are divided into communication groups according to a dynamic grouping mechanism. Action decisions are made in a policy network according to the state observation value encoding of each AGV, path planning is completed, and rewards are obtained according to a reward function, with the goal of maximizing the reward. The model parameters of the local network model of each AGV are continuously iterated and updated, and the model parameters of the local network model of each AGV are used to update the model parameters of the global network model, wherein the deep reinforcement learning network includes a policy network and a critic network.

2. The method for multi-AGV conflict-free path planning in a warehouse environment using deep reinforcement learning as claimed in claim 1, characterized in that: Construct a reward function for deep reinforcement learning, including: The reward function is expressed as follows: R(s,a)=R1+R2+R3 Among them, R1 represents the reward value obtained based on the distance between the AGV and the nearest obstacle, R2 represents the reward value obtained by the AGV reaching the target shelf position, and R3 represents the penalty value set for the time spent in the path planning process of the AGV, which is used to measure the cost paid by the AGV in the process of avoiding other intelligent agents and non-target shelves; Among them, R1 is determined by the following formula during the period when no collision occurs: R1=u×obs_distance And according to the following collision penalty correction formula, a penalty correction is performed on R1 when a collision is identified: Where u represents the reward and penalty factor set for the distance between the AGV and the obstacle, which is a positive value; obs_distance represents the distance between the current AGV and the nearest obstacle; the collision penalty correction formula represents that when the distance obs_distance from the AGV to the obstacle is less than or equal to the preset minimum collision distance dis crash When it is recognized as a collision, R1 is subtracted by 100, and the value of R1 is penalized and corrected; The initial value of R2 is 0, and according to the following arrival reward formula, the reward correction R2 is given when it is recognized as reaching the target shelf: Among them, tar_distance represents the distance between the current AGV and the target shelf, reach is the set arrival distance. When the distance between the robot and the target shelf tar_distance is less than the default arrival distance dis reach When , it means that the current AGV has completed the path planning and arrived at the target shelf, and is recognized as arriving at the target shelf; R2' is the continuous arrival reward function, specifically the formula: Among them, ep_count represents the cumulative value of the number of rounds in which the AGV reaches the target continuously. If the AGV encounters a conflict or collision during the simulation, or reaches the maximum training time step but does not reach the target shelf, ep_count is reset and accumulated again; k' represents the reward coefficient for the number of times the AGV reaches the target shelf continuously, ep_count; Determine R3 according to the following formula: R3=q×step_count Among them, q is the penalty coefficient based on arrival time, and step_count is the cumulative step value.

3. The method for multi-AGV conflict-free path planning in a warehouse environment using deep reinforcement learning as claimed in claim 1, characterized in that: The distributed conflict-free multi-AGV path planner introducing a dynamic grouping mechanism is constructed, and the distributed conflict-free multi-AGV path planner is iteratively trained according to the reward function and the state observation value encoding and action space of each AGV, and the network parameters of the distributed conflict-free multi-AGV path planner are updated to obtain the trained distributed conflict-free multi-AGV path planner, including: Define a global iteration count, and set an initial value of the global iteration count to 0; Increase the global iteration count by one; Determine whether the global iteration count is less than a preset global iteration count threshold; If it is determined that the global iteration count is less than a preset global iteration count threshold, the following steps are performed: For each AGV in the virtual warehouse simulation training environment, the model parameters of the local network model corresponding to the AGV are synchronized using the model parameters of the global network model, and the time step count corresponding to the AGV is set to 0; Determine the state observation value code of each AGV in the virtual warehouse simulation training environment according to the virtual warehouse simulation training environment information and the preset perception radius; For each AGV in the virtual warehouse simulation training environment, a feature vector of the AGV is determined based on the state observation value encoding of the AGV and the evaluation network of the local network model of the AGV; the feature vector of the AGV represents the environmental information within the perception radius of the AGV; Generate an adjacency matrix corresponding to the AGV according to the feature vector of the AGV; the adjacency matrix represents the adjacency relationship between all AGVs within the perception radius of the AGV; According to the adjacency matrix corresponding to the AGV and the state observation value encoding of all AGVs in the adjacency matrix, based on the attention network of the local network model of the AGV, the similarity matrix corresponding to the AGV is obtained, and according to the similarity matrix corresponding to the AGV, other AGVs having similar motion paths as the AGV are determined, and the AGV and the obtained other AGVs having similar motion paths as the AGV are combined to form a communication group centered on the AGV; wherein each row element in the similarity matrix represents the corresponding relationship between the AGV corresponding to the row and other AGVs having similar running paths; Inputting the state observation value codes corresponding to all AGVs in the communication group centered on the AGV into the communication network of the local network model corresponding to the AGV for prediction, and obtaining the predicted state observation value code of the AGV; Encoding the predicted state observation value of the AGV and inputting it into the policy network of the deep reinforcement learning network to determine the action to be performed by the AGV; Execute the pending action of the AGV to make the AGV interact with the virtual warehouse simulation training environment, update the position information in the state observation value encoding of the AGV, and update the virtual warehouse simulation training environment information according to the updated position information of the AGV; According to the state observation value code of the AGV before executing the action to be executed and the updated state observation value code of the AGV, based on a preset reward value calculation function, calculate the reward value corresponding to the current time step count corresponding to the AGV; The time step count corresponding to the AGV is increased by one, and it is determined whether the position information in the updated state observation value code of the AGV is equal to the target shelf position in the state observation value code of the AGV, and whether the time step count corresponding to the AGV is greater than the preset local time step threshold; if it is determined that the position information in the updated state observation value code of the AGV is not equal to the target shelf position in the state observation value code of the AGV, and the time step count corresponding to the AGV is less than the preset local time step threshold, then the process returns to the step of determining the state observation value code of each AGV in the virtual warehouse simulation training environment according to the virtual warehouse simulation training environment information and the preset perception radius; Determine the expected return of the AGV at the current time step count based on the position information and reward value of the AGV at the current time step count and based on the commentator network in the local network model of the AGV; According to the expected return of the AGV at the current time step count, determine the expected return corresponding to each time step count before the current time step count, and according to the expected return corresponding to each time step count before the current time step count, perform gradient update on the model parameters in the local network model of the AGV; the model parameters in the local network model of the AGV include model parameters in the evaluation network, attention network, communication network and deep reinforcement learning network in the local network model; According to the model parameters of the local network model of the AGV after the gradient update, the model parameters of the corresponding model in the global network model are updated, and the execution returns to the step of increasing the global iteration count by one; wherein the model parameters of the global network model include: the model parameters of the evaluation network, the attention network, the communication network, and the deep reinforcement learning network in the global network model; If it is determined that the global iteration count is greater than or equal to a preset global iteration count threshold, the model parameters of the corresponding model in the global network model are saved.

4. The method for multi-AGV conflict-free path planning in a warehouse environment using deep reinforcement learning as claimed in claim 1, characterized in that: The trained distributed conflict-free multi-AGV path planner is used to perform conflict-free path planning on multiple AGVs in an actual warehouse environment to obtain a conflict-free path for each AGV, including: Acquire actual warehouse environment information, wherein the actual warehouse environment information includes the positions of multiple AGVs, obstacle positions, shelf positions, and road information in the actual warehouse environment; Initialize the model parameters of the local network model of each AGV in the multiple AGVs in the actual warehouse environment using the model parameters of the global network model in the trained distributed conflict-free multi-AGV path planner; According to the preset time step interval, periodically build a communication group centered on the AGV at each time step through the evaluation network and attention network of the local network model of each AGV in the actual warehouse environment, and update the state observation value code of each AGV in the communication group centered on the AGV through the communication network in the local network model of the AGV, input the updated state observation value code of the AGV into the policy network in the deep reinforcement learning network of the local network model of the AGV, obtain the action to be executed of the AGV corresponding to the time step, execute the action to be executed, and determine the position information of the AGV corresponding to the time step; The AGV position information of each AGV in the actual warehouse environment at each time step is arranged in chronological order to obtain the conflict-free planning path of each AGV.

5. The method for multi-AGV conflict-free path planning in a warehouse environment using deep reinforcement learning as claimed in claim 3, characterized in that: Determining the state observation value encoding of each AGV in the virtual warehouse simulation training environment according to the virtual warehouse simulation training environment information and the preset perception radius includes: For each AGV in the virtual warehouse simulation training environment, the obstacles and other AGVs within the perception radius centered on the AGV are determined according to the virtual warehouse simulation training environment information, and the state observation value encoding of the AGV is constructed according to the obstacle position corresponding to the determined obstacle, the position information and movement direction of other AGVs.

6. The method for multi-AGV conflict-free path planning in a warehouse environment using deep reinforcement learning as claimed in claim 3 is characterized in that: For each AGV in the virtual warehouse simulation training environment, according to the state observation value encoding of the AGV, based on the evaluation network of the local network model of the AGV, the feature vector of the AGV is determined, including: According to the state observation value coding of the AGV, all AGVs within the field of the AGV are determined, and the state observation value coding of all AGVs within the AGV is vectorized into corresponding initial eigenvalues; the neighborhood of the AGV represents the range covered by the preset perception radius of the AGV; The initial eigenvalues ​​of all AGVs in the neighborhood of the AGV are saved as the last updated eigenvalues; Set the initial value of the number of iterations; Determine whether the number of iterations is greater than the preset number of iterations. If the number of iterations is greater than the preset number of iterations, end the iteration loop and use the initial feature value of the AGV after the iteration loop ends as the feature vector of the AGV. Otherwise, continue to execute the following iteration loop: The initial eigenvalue of the AGV and the previously updated eigenvalues ​​corresponding to other AGVs in the neighborhood of the AGV are averaged, and then multiplied by the weight corresponding to the current number of iterations to obtain an aggregate value, and the aggregate value is nonlinearized by a preset nonlinear function to obtain a nonlinearized aggregate value, and the nonlinearized aggregate value is used to update the initial eigenvalue of the AGV; Normalizing the initial characteristic value of the AGV, and using the normalized initial characteristic value of the AGV to update the initial characteristic value of the AGV; For each other AGV in the field of the AGV, determining the positional relationship of the other VGA relative to the AGV; When it is determined that the other AGV is in the forward area of ​​the AGV, the last updated characteristic value corresponding to the other AGV is updated using the value obtained by multiplying the initial characteristic value of the other AGV by the first coefficient; In the case where it is determined that the other AGV is in the rearward area of ​​the AGV, the last updated characteristic value corresponding to the other AGV is updated using the value obtained by multiplying the initial characteristic value of the other AGV by the second coefficient; Return to the step of determining whether the number of iterations is greater than the preset number of iterations; Wherein, determining the positional relationship of the other VGAs relative to the AGV includes: According to the state observation value coding of the AGV, the position information of other AGVs within the perception radius of the AGV is obtained, and the other AGVs within the perception radius of the AGV are divided into AGVs in the facing area of ​​the AGV and AGVs in the rearward area of ​​the AGV; wherein the facing area of ​​the AGV is defined as the first quadrant and the second quadrant of the coordinate system with the position information in the state observation value coding of the AGV as the origin; the rearward area of ​​the AGV is defined as the third quadrant and the fourth quadrant of the coordinate system with the position information in the state observation value coding of the AGV as the origin; the Y-axis of the coordinate system with the position information in the state observation value coding of the AGV as the origin is defined as a straight line pointing from the position information in the state observation value coding of the AGV to the target shelf position in the state observation value coding of the AGV, and the positive direction of the Y-axis is toward the target shelf position in the state observation value coding of the AGV, and the X-axis of the coordinate system with the position information in the state observation value coding of the AGV as the origin is perpendicular to the Y-axis; Among them, the first coefficient is greater than the second coefficient, the first coefficient and the second coefficient are both greater than 0 and the sum of the first coefficient and the second coefficient is 1; the weights corresponding to the first coefficient, the second coefficient and all iteration numbers are model parameters of the evaluation network.

7. The method for multi-AGV conflict-free path planning in a warehouse environment using deep reinforcement learning as claimed in claim 3, characterized in that: According to the feature vector of the AGV, an adjacency matrix corresponding to the AGV is generated, including: The similarity values ​​of the feature vectors of all AGVs within the perception radius of the AGV are calculated, and the corresponding element positions of two AGVs whose similarity values ​​are greater than a preset adjacency similarity threshold in the adjacency matrix are set to 1, otherwise they are set to 0.

8. The method for multi-AGV conflict-free path planning in a warehouse environment using deep reinforcement learning as claimed in claim 3, characterized in that: According to the adjacency matrix corresponding to the AGV and the state observation value encoding of all AGVs in the adjacency matrix, a similarity matrix corresponding to the AGV is obtained based on the attention network of the local network model of the AGV, and other AGVs having similar motion paths as the AGV are determined according to the similarity matrix corresponding to the AGV, and the AGV and the other AGVs having similar motion paths as the AGV are formed into a communication group centered on the AGV, including: Vectorizing the state observation value codes of all AGVs in the adjacency matrix corresponding to the AGV into corresponding code vectors, and arranging all the obtained code vectors in the order of the AGVs in the adjacency matrix to obtain an observation value code matrix; Initialize the query matrix, key matrix and value matrix in the attention network in the local network model of the AGV using the observation value encoding matrix; The similarity matrix of the current time step is obtained according to the following formula Where M is the adjacency matrix of the current time step; Q is the query matrix, K is the key matrix, d k is the dimension of the key matrix, T represents the transposed matrix of the current matrix; The normalized similarity matrix is ​​calculated according to the following formula: Among them, the softmax function is a normalization function; Represents the interaction relationship between AGVs before normalization. It means the sum of the relationship between AGV No. i and other AGV No. k after exponential operation; The other AGVs corresponding to the columns or rows corresponding to the elements with a value of 1 on the row or column corresponding to the AGV in Attention (Q, K, V) are divided into a communication group centered on the AGV.

9. The method for multi-AGV conflict-free path planning in a warehouse environment using deep reinforcement learning as claimed in claim 4, characterized in that: According to the preset time step interval, periodically at each time step, a communication group centered on the AGV is constructed through the evaluation network and the attention network of the local network model of each AGV in the actual storage environment, and the state observation value encoding of each AGV in the communication group centered on the AGV is updated through the communication network in the local network model of the AGV, and the updated state observation value encoding of the AGV is input into the policy network in the deep reinforcement learning network of the local network model of the AGV to obtain the action to be executed of the AGV corresponding to the time step, execute the action to be executed, and determine the position information of the AGV corresponding to the time step, including: For each time step, the state observation value encoding of each AGV in the actual warehouse environment is determined according to the actual warehouse environment information and the preset perception radius; the actual warehouse environment information includes: the location information of multiple AGVs, obstacle locations, shelf locations and road information; For each AGV in the actual warehouse environment, determine the feature vector of the AGV according to the state observation value encoding of the AGV and based on the evaluation network in the local network model of the AGV; Generate an adjacency matrix corresponding to the AGV according to the feature vector of the AGV; According to the adjacency matrix corresponding to the AGV and the state observation value encodings of all AGVs in the adjacency matrix, based on the attention network in the local network model of the AGV, a similarity matrix corresponding to the AGV is obtained; According to the similarity matrix corresponding to the AGV, other AGVs having similar motion paths as the AGV are determined, and the AGV and the other AGVs having similar motion paths as the AGV are combined to form a communication group centered on the AGV; Inputting the state observation value codes corresponding to all AGVs in the communication group centered on the AGV into the communication network in the local network model of the AGV for prediction, and obtaining the predicted state observation value code of the AGV; Encoding the predicted state observation value of the AGV and inputting it into the policy network of the deep reinforcement learning network of the local network model of the AGV to determine the action to be performed by the AGV; Execute the pending action of the AGV to enable the AGV to interact with the actual storage environment, update the position information in the state observation value code of the AGV, and update the actual storage environment information according to the updated position information in the state observation value code of the AGV; Determine whether the position information in the state observation value code of the AGV is equal to the target shelf position in the state observation value code of the AGV; If it is determined that the position information in the state observation value code of the AGV is not equal to the target shelf position in the state observation value code of the AGV, then return to the step of determining the state observation value code of each AGV in the actual storage environment for each time step according to the actual storage environment information and the preset perception radius.

Images