Multi-agent reinforcement learning and context awareness-based path planning method for multiple automated guided vehicles

Abstract

The invention discloses a multi-agent reinforcement learning and context awareness-based multi-automated guided vehicle path planning method, and belongs to the crossing field of machine learning and intelligent logistics. According to the context sensing information provided by the invention, the environment understanding of the AGV in the training process is fully utilized, and the context sensing information used for assisting other agents in decision making is generated aiming at the observation information of the current AGV through long-time memory of the AGV. When multi-agent communication reinforcement learning is carried out, diversified communication information is generated by using local feature information of the AGV, and then a final decision action probability distribution result is obtained by combining the local feature information of the AGV. On the basis of a centralized training and distributed execution training normal form, the multi-AGV system can well learn an efficient and direct path planning solution, the path planning efficiency and accuracy of the AGVs on the map are effectively improved, and meanwhile collision conflicts between the AGVs are reduced.

Description

Technical Field

[0001] The present invention belongs to the research fields of machine learning multi-agent systems and intelligent logistics, and relates to a path planning method for multi-Automated Guided Vehicles (AGVs) based on context-aware information in multi-agent reinforcement learning. Background Art

[0002] With the rapid development of the logistics industry and intelligent technologies, intelligent warehousing systems have become a key means to improve warehouse management efficiency. In these systems, AGVs rely on autonomous path finding and navigation to complete the efficient transportation of materials. The path planning of multi-AGV systems not only requires parallel path planning for multiple AGVs, but also needs to optimize the collaborative behavior among AGVs to avoid cross conflicts and environmental obstacles, ensuring the efficiency and safety of path planning.

[0003] The multi-AGV path planning problem can be modeled as a multi-agent path search problem and solved by treating AGVs as agents. Facing the high complexity of the multi-agent path search problem, Sharon et al. proposed the CBS algorithm, which achieved efficient path planning in multi-agent systems through a top-level constraint tree splitting strategy and a bottom-level conflict-free path search (Sharon G, Stern R, Felner A, et al. Conflict-based search for optimal multi-agent pathfinding[J]. Artificial intelligence, 2015, 219: 40-66.). Ren et al. further significantly accelerated the execution speed of the CBS algorithm by explicitly estimating the CBS algorithm and combining online learning techniques to optimize the selection of top-level nodes (Ren Z, Rathinam S, Choset H. A Bounded Sub-Optimal Approach for Multi-Agent Combinatorial Path Finding[J]. IEEE Transactions on Automation Science and Engineering, 2024.). The PRIMAL framework proposed by Sartoretti et al. achieved effective cooperation among agents by combining reinforcement learning and imitation learning (Sartoretti G, Kerr J, Shi Y, et al. Primal: Pathfinding via reinforcement and imitation multi-agent learning[J]. IEEE Robotics and Automation Letters, 2019, 4(3): 2378-2385.). Hu et al. adopted the multi-agent deep deterministic policy gradient method in the AGV anti-collision path planning problem of automated container terminals and optimized the algorithm performance through the Gumbel-Softmax policy (Hu H, Yang X, Xiao S, et al. Anti-conflict AGV path planning in automated container terminals based on multi-agent reinforcement learning[J]. International Journal of Production Research, 2023, 61(1): 65-80.).

[0004] Multi-agent communication, as a method that can effectively improve the cooperation level of agents in a multi-agent system, has been widely studied. In the existing knowledge categories of communication, agents generate information based on the currently available local information and share it with other agents. These information include but are not limited to local observation information and historical information. Most of the proposed methods involve agents communicating their respective local observation information, but the ways they process this information vary. For example, MAGIC (Y. Niu, R. R. Paleja, and M. C. Gombolay, “Multi-agent graph-attention communication and teaming.,” in AAMAS, vol. 21, p. 20th, 2021.) uses a graph attention network to learn the communication graph and then derives the communication information. IMMAC (C. Sun, B. Wu, R. Wang, X. Hu, X. Yang, and C. Cong, “Intrinsic motivated multi-agent communication,” in Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1668–1670, 2021.) believes that aimless and random communication will damage the overall performance of the multi-agent system and may even have adverse effects. Therefore, it proposes a method that focuses on communicating “what surprises you” information to enhance cooperation among agents. Regarding the imagined future knowledge, it specifically refers to obtaining the expected actions, strategic tendencies, or planned strategies of agents within a certain period in the future through certain methods and technologies. Then these information are transmitted to other agents to achieve better coordination and cooperation. MBC (S. Han, M. Dastani, and S. Wang, “Model-based sparse communication in multi-agent reinforcement learning,” in Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 439–447, 2023.) uses model-based reinforcement learning techniques to directly predict the communication information that agents will receive. This method reduces the overuse of communication channels and improves the efficiency of policy learning.

[0005] Due to the partial observability of the environment, meaningful communication information is of great value in helping AGVs optimize their strategies. Existing research either relies on agents to communicate their true local information (such as actions) or uses modeling techniques to predict future information and transmit it to other agents for communication. However, as training progresses, agents will continuously optimize their decision-making strategies, thereby gaining a deeper understanding of the environment. This understanding can help agents achieve more effective communication. Summary of the Invention

[0006] The present invention aims to propose a novel communication method and apply it to multi-AGV path planning to help AGVs better understand the environment, transmit auxiliary decision-making information, and optimize the learning process of AGV decision-making strategies.

[0007] The technical solution adopted by the present invention is as follows:

[0008] A multi-automatic guided vehicle path planning method based on multi-agent reinforcement learning and context awareness, the steps are as follows:

[0009] Step 1: Environmental map modeling.

[0010] The grid method is used to model the actual environmental map, and the environment is divided into a series of small regional units with clear boundaries to accurately locate the positions of multi-automatic guided vehicles (AGVs). When using the grid method to model the environmental map, all object information in the environment needs to be fully represented, including the AGV communication area, the area with obstacles, and the AGV target position information. Each AGV's movement on the grid map is limited to adjacent grids.

[0011] Step 2: Definition of AGV state space, action space, and design of reinforcement learning reward function.

[0012] The multi-agent reinforcement learning paradigm is adopted to improve the path planning ability of the AGV group. On the basis of obtaining the grid map, the entire system is modeled as a multi-agent reinforcement learning problem, and the state space and action space of each AGV are specified, and a reasonable reward function is designed. The specific steps are as follows:

[0013] In the multi-AGV path planning problem, there are n AGVs, forming a set V = {v 1 , v 2 ,..., v n}. Each AGV needs to move from the specified starting point b i to the end point e i , and the starting points and end points respectively form sets B = {b 1 , b 2 , b 3 ,...., b n}, and E = {e 1 , e 2 , e 3 ,..., e n}. Path planning is carried out on a two-dimensional grid map containing obstacles. The goal of path planning is to minimize the total length of all AGV paths, and the mathematical representation is shown in formula (1). P i represents the path of v i , and length(P i ) represents the distance from the starting point b i to the ending point e i of the i-th AGV.

[0014]

[0015] The multi-AGV path planning problem is formulated as a partially observable Markov decision process POMDP, aiming to effectively cope with the uncertainty in path planning and the partial observability of the environmental state. POMDP can be represented by the tuple (S, A, T, R, Ω, O, γ), where S is the state space, A is the action space, T is the transition model, R is the reward function, Ω is the observation space, O is the observation model, and γ ∈ (0, 1] is the discount factor. At each time step t, when the environment is in a certain state s and all AGVs take an action a ∈ A, it will transition to the state s′, and the probability is determined by the transition model T(s, a, s′). Subsequently, the AGV receives an observation o ∈ Ω according to the new state s′, and the probability of this observation occurring is given by the observation model O(o, s′, a), indicating the probability of observing o after executing the action a and transitioning to the state s′.

[0016] In the map obtained by the grid method, each AGV can observe the information within a certain range around itself, making the entire system have the property of local observability. Since there are a finite number of grids in the map, the range that each AGV can observe is a square area of size l×l centered on itself, and this area constitutes the observation field of view of the AGV. Each AGV can obtain the obstacle information, the position information of the AGV, the target position information of the current AGV, and the target position information of other AGVs based on the information within its field of view.

[0017] The action space A of the AGV on the grid map is defined as a series of discrete actions, including moving up, down, left, and right, as well as remaining stationary. The AGV is not allowed to perform diagonal movement actions along the grid at each time step. At each time step t, each AGV will obtain the action at ∈ A according to its own strategy π, and the overall action of the entire multi-AGV group is n represents the number of AGVs.

[0018] The existence of the reward function aims to guide the behavior of AGVs through a reward and punishment mechanism. At each time step t, the AGV executes an action in the scope of the grid map environment, and the environment gives the AGV a reward value based on the global environmental information. The goal of the entire AGV group is to maximize the cumulative value of the rewards given by the environment. When the AGV successfully reaches the target point, the environment will feedback a one-time arrival reward with a value of g; when the AGV collides with an obstacle or other AGVs during the movement, it will receive a collision penalty with a value of -g from the environment. Further, in order to encourage the AGV to reach the target point as quickly as possible, a step penalty with a value of -f is introduced in the grid map environment. Before the AGV reaches the target location, it will receive a step penalty for each time step passed. In addition, considering that turning may cause additional time consumption for the AGV, the AGV will receive an additional turning penalty each time it turns, so as to encourage the AGV to take as direct a path as possible and reduce unnecessary turning behaviors. Finally, in order to enable the AGV to continuously move forward towards the target location, the change in the distance d between the AGV before and after executing the action and the target position is also calculated through formula (2), and the AGV can also receive a reward value.

[0019]

[0020] Step 3: Initialization.

[0021] Initialize the AGV network and the hybrid network. The AGV network includes a local feature information network, a context awareness information generation network, a communication information generation network, a communication information hybrid network, and a decision network. The hybrid network receives the global state information s at the current time step t t and the action value probability distribution output by all AGV networks, and outputs the global action value probability distribution Q tot 。

[0022] Copy the parameters of the AGV network and the hybrid network to the target AGV network and the target hybrid network. The target AGV network and the target hybrid network are copies of the AGV network and the hybrid network respectively. Every p rounds, the target AGV network and the target hybrid network will copy the weight information of the AGV network and the hybrid network respectively.

[0023] Initialize the experience replay buffer pool with a capacity of M.

[0024] Set the total number of iteration rounds epoch, and the update frequency p of the target network parameters.

[0025] After the training loop starts, at the beginning of each round, the time step is set to 0, the grid environment is reset, the initial global state information s and the observation information of each initial AGV are obtained and initialize the long-term memory vector m of each AGV as m = {m1 , m 2 ,..., m n} is a zero value.

[0026] Step 4: Generation of multi-AGV local feature information, context awareness information, and communication information.

[0027] 4.1) Generation of AGV local feature information.

[0028] At time step t, the i-th AGV can obtain the local observation information within its own field of view and the action taken at the previous time step t - 1 by itself After that, the local observation information and the historical action are concatenated to obtain the local information as the input to the local feature information network. The local feature information network consists of a fully connected layer and a gated neural network layer GRU, and outputs the local feature information of the i-th AGV For the n AGVs existing in the system, the overall local feature information will be generated accordingly

[0029] Furthermore, in a multi-AGV environment, the local observation information observed by each AGV is a one-dimensional vector with a dimension of 64. The local observation information is transmitted to a fully connected layer with a shape of (64, 32), and then passes through a GRU layer with a shape of (32, 32). Finally, a local feature vector with a dimension of 32 is obtained The overall local feature information has a shape of (n, 32) dimensions.

[0030] 4.2) Generation of AGV context awareness information.

[0031] The generation of context awareness information comes from the AGV's own understanding of the environment. At the current time step t, for the i-th AGV, the context awareness information generation network uses the long-term memory vector, the AGV local observation information and the historical action to generate the context awareness information of the i-th AGV at the current moment The generated context awareness information will be transmitted to the communication information generation network and used as part of the communication information. The specific implementation is as follows:

[0032] The context awareness information generation network first passes the long-term memory vector m i through a fully connected layer to generate the corresponding keyi, the local observation information and the action at the previous time step After passing through the fully connected layer, the queryi information in the attention mechanism is generated, and then the long-term memory vector Local observation information and the action at the previous time step are concatenated, passed through the fully connected layer to generate the value information in the attention mechanism, and then the final context-aware information about agent i is calculated using formula (3). Where is the parameter representing the adjustment amplitude, exp(·) represents the exponential calculation operation, represents the transpose of queryi.

[0033]

[0034] 4.3) AGV communication information generation.

[0035] To generate the communication information, the context-aware information generated by the i-th AGV at the current time step t, and the local information observed by the AGV are concatenated as the input of the communication information generation network. Specifically, since the local information observed by the AGV is actually represented by a vector in the multi-AGV system, but the meanings represented by different parts of this vector are different, the different information contained in the vector is distinguished into self-feature information self_features, friendly feature information ally_features, and obstacle feature information enemy_features. At the same time, these information are respectively reconstructed. Through the fully connected neural network, the one-dimensional feature information is reconstructed into two-dimensional feature information, and the reconstructed information self_feat, ally_feat, and enemy_feat are obtained respectively, as shown in formula (4).

[0036]

[0037] Where, ent i represents the different meaning information divided, f k，z represents the elements in the reconstructed two-dimensional vector, and k and z represent that the reconstructed two-dimensional vector is of (k, z) dimension.

[0038] After that, these three types of reconstructed different feature information are concatenated with the context-aware information at the current time step to obtain which is fused through the multi-head attention mechanism and the self-attention mechanism, and finally the communication information of the i-th AGV at the current time step t and the updated value m of the long-term memory vector i .

[0039] Step 5: Multi-AGV communication information aggregation and AGV action decision-making.

[0040] In the communication stage, a full communication behavior occurs among AGVs. Specifically, at the current time step t, the i-th AGV generates communication information For all AGVs, the generated communication information can be expressed as Since it is a full communication behavior, after the i-th AGV generates the communication information it will send it to all other AGVs. Therefore, after a full communication behavior, all AGVs can obtain the communication information c t . After that, in order to generate diverse communication information, a communication information mixing network is adopted to vary the communication information c received by the AGV t according to its own local feature information The communication information mixing network consists of two fully connected layers. One of the inputs and the resulting value represents the weight w 1 , and one of the inputs and the resulting value represents the offset value bais after taking the absolute value. Through formula (5), the i-th AGV obtains the final communication information c' through communication, where MLP represents the fully connected neural network layer. The communication information c' and the local feature information of the i-th AGV are concatenated as the input of the decision neural network of the i-th AGV. The decision network consists of a GRU and a fully connected layer. Finally, a vector with a dimension of num_action×1 is output, and this vector represents the action value probability distribution Q of all actions that the i-th AGV can take at the current time step t i .

[0041]

[0042] After obtaining the action value probability distribution results of each AGV at time step t, the ∈-greedy strategy shown in formula (6) is used to select specific actions where idx represents the index value of the selected response. These actions will be executed at this time step t, so that the environment transfers to the next state s', and the reward r obtained at the current time step and the information terminated indicating whether the environment terminates are obtained.

[0043]

[0044] Randomly select a number between 0 and 1 as the parameter ε; random action means randomly selecting an action; store the obtained state information s, next state s′, reward r, and whether it is terminated at time step t in the buffer pool.

[0045] Step Six: AGV Action Output and Value Decomposition Architecture Training.

[0046] When the number of data in the buffer pool reaches the pre - specified number, start the training phase. In the training phase, first randomly sample x pieces of data from the buffer pool for training. All the action - value probability distributions generated by the AGVs will be transmitted to the hybrid network. Based on the received global state information s t , generate the corresponding weights and biases, and obtain the final output of the hybrid network as the global action - value probability distribution Q tot , and make it satisfy formula (7), where argmax represents the value of the independent variable when the given function reaches the maximum value, s is the global state information, and a is the global action information.

[0047]

[0048] The hybrid network consists of four fully - connected layers and receives the global state as input at the same time, so as to obtain the weights W 1 , W 2 and biases b 1 , b 2 , and use formula (8) to obtain Q tot , where Q(τ, a)=(Q 1 , Q 2 , …, Q n ).

[0049] Q tot =W 2 [W 1 Q(τ, a)+b 1 +b 2 (8)

[0050] Through the obtained Q tot , use the loss function of formula (9) Train through deep - learning backpropagation, update the AGV network and hybrid network parameters, so as to obtain a multi - agent group control strategy with a high level of cooperation and coordination. In formula (9), r represents the reward, represents the target network, where θ and represent the network parameters, γ represents the discount factor, represents the data set in the buffer pool, s′ represents the global state information at the next time step, and a′ represents the joint action information of all AGVs at the next time step. It means that at the next time step, all possible joint actions a′ are traversed, and the joint action that can maximize Q is selected. tot The maximum joint action.

[0051]

[0052] In addition, every p rounds, the parameters of the target AGV network and the target hybrid network are updated.

[0053] Advantages of the present invention: To solve the problem of the specific composition of the communication content between AGVs, thereby improving the cooperative behavior performance among multiple AGVs, the present invention proposes a communication method that imitates the communication between human team members, utilizes the context-aware information of individuals, and assists AGV decision-making. By making full use of the characteristics of deep neural networks and Markov decision chains, the understanding and memory of the AGV itself about the environment are retained, and the local information observed at the current moment is understood based on the retained long-term understanding and memory, generating context-aware information, and communicating the context-aware information together with the local feature information to assist other AGVs in making decisions, thereby realizing the efficient cooperation of multiple AGV groups and improving the final performance of multiple AGV groups in the multi-AGV system. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a grid-type environmental map.

[0055] Figure 2 is a diagram of the context-aware information generation module.

[0056] Figure 3 is an expanded view of the communication information sending module.

[0057] Figure 4 is an expanded view of the communication information receiving module.

[0058] Figure 5 is a diagram of the overall centralized training and distributed execution architecture.

[0059] Figure 6 is the overall flowchart. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. The specific steps are as follows

[0061] A multi-automatic guided vehicle path planning method based on multi-agent reinforcement learning and context awareness, the steps are as follows:

[0062] Step 1: Environmental map modeling.

[0063] In the multi-AGV (Automated Guided Vehicle) path planning problem, in order to obtain an effective solution, an accurate modeling of the actual environmental map is required. The present invention uses the grid method to model the actual environmental map, dividing the environment into a series of small regional units with clear boundaries, so as to accurately locate the positions of multiple automated guided vehicles (AGVs). As Figure 1 shown, when using the grid method to model the environmental map, the white grids represent the areas where AGVs can pass, and the gray grids indicate the impassable areas with obstacles. The position of each AGV is determined by the grid coordinates where it is located, and the triangle represents the AGV. Its target position is represented as a rhombus. The movement of each AGV is limited to between adjacent grids.

[0064] Step 2: Definition of the AGV state space, action space, and design of the reinforcement learning reward function.

[0065] The multi-agent reinforcement learning paradigm is adopted to improve the path planning ability of the AGV group. On the basis of obtaining the grid map, the entire system is modeled into a multi-agent reinforcement learning problem, and the state space and action space of each AGV are specified, as well as a reasonable reward function is designed. The specific steps are as follows:

[0066] In the multi-AGV path planning problem, there are n AGVs, forming a set V = {v 1 , v 2 ,..., v n}. Each AGV needs to move from the specified starting point b i to the end point e i , and the starting points and end points respectively form sets B = {b 1 , b 2 , b 3 ,...., b n} and E = {e 1 , e 2 , e 3 ,..., e n}. The path planning is carried out on a two-dimensional grid map containing obstacles. The goal of path planning is to minimize the total length of all AGV paths, and the mathematical representation is shown in formula (1), where P i represents the path of v i , and length(P i ) represents the distance of the i-th AGV from the starting point b i to the end point e i .

[0067]

[0068] The multi-AGV path planning problem is formulated as a Partially Observable Markov Decision Process (POMDP) to effectively address the uncertainty in path planning and the partial observability of the environmental state. A POMDP can be represented by the tuple (S, A, T, R, Ω, O, γ), where S is the state space, A is the action space, T is the transition model, R is the reward function, Ω is the observation space, O is the observation model, and γ = 0.99 ∈ (0, 1] is the discount factor. At each time step t, when the environment is in a certain state s and all AGVs take an action a ∈ A, it will transition to state s′, and the probability is determined by the transition model T(s, a, s′). Subsequently, the AGV receives an observation o ∈ Ω based on the new state s′, and the probability of this observation occurring is given by the observation model O(o, s′, a), indicating the probability of observing o after performing action a and transitioning to state s′.

[0069] In the map obtained by the grid method, each AGV can observe the information within a certain range around itself, making the entire system have the property of local observability. Since there are a finite number of grids in the map, the observable range of each AGV is a square area of size l×l centered on itself, and this area constitutes the observation field of view of the AGV. Each AGV can obtain the obstacle information, the position information of the AGV itself, the target position information of the current AGV, and the target position information of other AGVs based on those within its field of view.

[0070] The action space A of the AGV on the grid map is defined as a series of discrete actions, including moving up, down, left, right, and staying still. The AGV is not allowed to move along the grid diagonal at each time step. At each time step t, each AGV will obtain an action a t ∈ A according to its own policy π, and the overall action of the entire multi-AGV group is where n represents the number of AGVs.

[0071] The existence of the reward function aims to guide the behavior of AGVs through a reward and punishment mechanism. At each time step t, an AGV executes an action in the scope of the grid map environment, and the environment gives the AGV a reward value according to the global environmental information. The goal of the entire AGV group is to maximize the cumulative value of the rewards given by the environment. When an AGV successfully reaches the target point, the environment will feedback a one-time arrival reward with a value of 50; when an AGV collides with an obstacle or other AGVs during the movement process, it will receive a collision penalty from the environment with a value of -50. Further, in order to encourage AGVs to reach the target point as quickly as possible, a step penalty with a value of -1 is introduced in the grid map environment. Before an AGV reaches the target location, it will receive a step penalty every time it passes through a time step. In addition, considering that turning may cause additional time consumption for AGVs, each time an AGV turns, it will receive an additional turning penalty (with a value of -1) to encourage AGVs to take as direct a path as possible and reduce unnecessary turning behaviors. Finally, in order to enable AGVs to continuously move forward towards the target location, the change in the distance d between the AGV's position before and after executing an action and the target position is also calculated through formula (2), and the AGV can also receive a reward value.

[0072]

[0073] Step 3: Initialization.

[0074] Initialize the AGV network and the hybrid network. The AGV network includes a local feature information network, a context-aware information generation network, a communication information generation network, a communication information hybrid network, and a decision-making network. The hybrid network receives the global state information s at the current time step t t and the action value probability distribution output by all AGV networks, and outputs the global action value probability distribution Q tot .

[0075] Copy the parameters of the AGV network and the hybrid network to the target AGV network and the target hybrid network. The target AGV network and the target hybrid network are copies of the AGV network and the hybrid network respectively. Every p rounds, the target AGV network and the target hybrid network will copy the weight information of the AGV network and the hybrid network respectively.

[0076] Initialize the experience replay buffer pool with a set capacity of M = 5000.

[0077] Set the total number of iteration rounds epoch, and the update frequency p of the target network parameters.

[0078] After that, after the training loop starts, at the beginning of each round, the time step is set to 0, the grid environment is reset, and the initial global state information s and the observation information of each initial AGV are obtained And initialize the long-term memory vectors m = {m 1 , m 2 ,..., m n} of each AGV to zero values.

[0079] Step 4: Generation of multi-AGV local feature information, context awareness information, and communication information.

[0080] 4.1) Generation of AGV local feature information.

[0081] At time step t, the i-th AGV can obtain the local observation information within its own field of view and the action taken at the previous time step t - 1 by itself After that, the local observation information and the historical action are concatenated to obtain the local information as the input to the local feature information network. The local feature information network consists of a fully connected layer and a gated recurrent unit (GRU) layer, and outputs the local feature information of the i-th AGV. For the n AGVs existing in the system, the overall local feature information

[0082] will be generated accordingly. Further, in a multi-AGV environment, the local observation information observed by each AGV is a one-dimensional vector with a dimension of 64. The local observation information is transmitted to a fully connected layer with a shape of (64, 32), and then passes through a GRU layer with a shape of (32, 32). Finally, a 32-dimensional local feature vector is obtained. The shape of the overall local feature information is (n, 32) dimensions.

[0083] 4.2) Generation of AGV context awareness information.

[0084] The generation of context awareness information comes from the AGV's own understanding of the environment. At the current time step t, for the i-th AGV, the context awareness information generation network uses the long-term memory vector, the AGV local observation information and the historical action to generate the context awareness information of the i-th AGV at the current moment. The generated context awareness information will be transmitted to the communication information generation network and used as part of the communication information. The specific implementation is as follows:

[0085] As Figure 2 shown, the context awareness information generation network first takes the long-term memory vector mi After passing through a fully connected layer, the corresponding keyi and local observation information are generated. and the action at the previous time step After passing through a fully connected layer, the query in the attention mechanism is generated. i information, and then the long-term memory vector local observation information and the action at the previous time step are concatenated, passed through a fully connected layer to generate the value information in the attention mechanism, and then the final context-aware information about agent i is calculated using formula (3). where represents the parameter of the adjustment amplitude, and exp(·) represents the exponential calculation operation. represents the transpose of queryi.

[0086]

[0087] 4.3) Generation of AGV communication information.

[0088] To generate the communication information, the context-aware information generated by the i-th AGV at the current time step t, and the local information observed by the AGV are concatenated as the input to the communication information generation network. Specifically, since the local information observed by the AGV is actually represented by a vector in a multi-AGV system, but the meanings represented by different parts of this vector are different. For example, this vector may contain information such as the speed and position of AGVi itself, as well as information such as the relative speed and relative position of friendly units (other AGVs), and also contains information related to obstacles. To distinguish these information, it is necessary to manually distinguish the different information contained in the vector into self-feature information self_features, friendly feature information ally_features, and obstacle feature information enemy_features, and at the same time, these information are respectively reconstructed. Through a fully connected neural network, the one-dimensional feature information is reconstructed into two-dimensional feature information, and the reconstructed information self_feat, ally_feat, and enemy_feat are obtained respectively, as shown in formula (4).

[0089]

[0090] where, ent i represents the different meaning information of the division, f k，z represents the elements in the reconstructed two-dimensional vector, and k and z represent that the reconstructed two-dimensional vector is of (k,z) dimension.

[0091] After that, these three different reconstructed feature information and the context awareness information at the current time step are concatenated to obtain which is fused through a multi-head attention mechanism (with the number of layers being L, and in the present invention L = 2) and a self-attention mechanism, and finally the communication information of the i-th AGV at the current time step t is obtained As Figure 3 shown, as well as the updated value m of the long-term memory vector i .

[0092] Step Five: Multi-AGV communication information aggregation and AGV action decision-making.

[0093] In the communication stage, a full communication behavior will be carried out among AGVs. Specifically: at the current time step t, the i-th AGV generates communication information For all AGVs, the generated communication information can be expressed as Since it is a full communication behavior, after the i-th AGV generates communication information it will send it to all other AGVs. Therefore, after a full communication behavior, all AGVs can obtain the communication information c t . After that, in order to generate diverse communication information, a communication information mixing network is adopted to vary the communication information c received by the AGV t according to its own local feature information . The communication information mixing network consists of two fully connected layers, where one input and the obtained result represents the weight w 1 , and one input and the obtained result represents the offset value bais after taking the absolute value. Through formula (5), the i-th AGV obtains the final communication information c' through communication, where MLP represents a fully connected neural network layer. The communication information c' and the local feature information of the i-th AGV are concatenated as the input of the decision neural network of the i-th AGV, where the decision network consists of a GRU and a fully connected layer. Finally, a vector with a dimension of num_action × 1 (num_action represents the number of actions that the AGV can take, which is 5) is output, and this vector represents the action value probability distribution Q of all actions that the i-th AGV can take at the current time step t i .

[0094]

[0095] After obtaining the action value probability distribution results of each AGV at time step t, a specific action is selected through the ∈-greedy strategy shown in formula (6) Among them, idx represents the index value of the selected response. These actions will be executed at this time step t, so that the environment transfers to the next state s′, and the reward r obtained at the current time step and the termination information terminated of the environment are obtained.

[0096]

[0097] Randomly select a number between 0 and 1 as the parameter ε; random action represents randomly selecting an action; the state information s, the next state s′, the reward r, and whether it is terminated terminated obtained at time step t are stored in the buffer pool.

[0098] Step 6: AGV action output and value decomposition architecture training.

[0099] When the number of data in the buffer pool reaches 200, the training phase is started. In the training phase, first, 32 pieces of data are randomly sampled from the buffer pool for training. All the action value probability distributions generated by the AGV will be transmitted to the hybrid network. Based on the received global state information s t , the corresponding weights and biases are generated, and the final output of the hybrid network is obtained as the global action value probability distribution Q tot , and make it satisfy formula (7), where argmax represents the value of the independent variable when the given function reaches the maximum value, s is the global state information, and a is the global action information.

[0100]

[0101] The hybrid network consists of four fully connected layers and receives the global state as input at the same time, so as to obtain the weights W 1 , W 2 and the biases b 1 , b 2 , and Q tot is obtained using formula (8), where Q(τ, a) = (Q 1 , Q 2 , …, Q n ).

[0102] Q tot = W 2 [W 1 Q(τ, a) + b 1 + b 2 (8)

[0103] Through the obtained Q tot , the loss function of formula (9) is used Training is carried out through the backpropagation of deep learning to update the parameters of the AGV network and the hybrid network, thereby obtaining a multi-agent group control strategy with a high level of cooperation and coordination. In formula (9), r represents the reward, represents the target network, where θ and represent the network parameters, γ represents the discount factor, represents the data set in the buffer pool, s′ represents the global state information at the next time step, and a′ represents the joint action information of all AGVs at the next time step. represents that at the next time step, all possible joint actions a′ are traversed, and the joint action that can make Q tot the largest is selected.

[0104]

[0105] In addition, every p rounds, the parameters of the target AGV network and the target hybrid network are updated.

Claims

1. A multi-automatic guided vehicle path planning method based on multi-agent reinforcement learning and context perception, characterized in that: Here are the steps: Step 1: Environmental map modeling; Step 2: AGV state space, action space definition and reinforcement learning reward function design; Step 3: Initialization; Step 4: Generate multi-AGV local feature information, context perception information, and communication information; Step 5: Aggregation of multi-AGV communication information and AGV action decision; Step 6: AGV action output and value decomposition architecture training.

2. The multi-automatic guided vehicle path planning method based on multi-agent reinforcement learning and context perception according to claim 1 is characterized in that: Step one is as follows: the actual environment map is modeled using the grid method, and the environment is divided into a series of small area units with clear boundaries, so as to accurately locate the positions of multiple automatic guided vehicles (AGVs); when the environment map is modeled using the grid method, it is necessary to fully represent all the information of objects in the environment, including the AGV communication area, the area with obstacles, and the target location information of the AGV; the movement of each AGV on the grid map is limited to between adjacent grids.

3. The multi-automatic guided vehicle path planning method based on multi-agent reinforcement learning and context perception according to claim 1 is characterized in that: Step 2 uses a multi-agent reinforcement learning paradigm to improve the path planning ability of the AGV group. On the basis of the grid map, the entire system is modeled as a multi-agent reinforcement learning problem, and the state space and action space of each AGV are standardized, and a reasonable reward function is designed. The specific steps are as follows: In the multi-AGV path planning problem, there are n AGVs, forming a set V = {v1,v2,…,v n }; Each AGV needs to start from the designated starting point b i Move to end point e i The starting point and the end point form a set B = {b1, b2, b3, ..., b n } and E={e1,e2,e3,…,e n }; Path planning is performed on a two-dimensional grid map containing obstacles; The goal of path planning is to minimize the total length of all AGV paths, which is mathematically expressed as shown in formula (1), P i Indicates v i The path length(P i ) indicates that the i-th AGV starts from the starting point b i To the end i distance; The multi-AGV path planning problem is formalized as a partially observable Markov decision process POMDP, which aims to effectively deal with the uncertainty in path planning and the partial observability of the environment state; POMDP can be represented by a tuple (S, A, T, R, Ω, O, γ), where S is the state space, A is the action space, T is the transition model, R is the reward function, Ω is the observation space, O is the observation model, and γ∈(0,1] is the discount factor; at each time step t, when the environment is in a certain state s, all AGVs will transition to state s′ after taking action a∈A, and its probability is determined by the transition model T(s, a, s′); then the AGV receives the observation o∈Ω according to the new state s′, and the probability of this observation occurring is determined by the observation model O(o, s ′ ,a) is given, indicating the probability of observing o after executing action a and transitioning to state s′; In the map obtained by the grid method, each AGV can observe information within a certain range around itself, making the entire system locally observable; since there are finite grids in the map, the range that each AGV can observe is an l×l area with its own center as the center of the square, and this area constitutes the AGV's observation field of view; each AGV can obtain obstacle information within its own field of view, AGV location information, current AGV target location information, and target location information of other AGVs within the field of view; The action space A of the AGV on the grid map is defined as a series of discrete actions, including moving up, down, left, right, and staying still; the AGV is not allowed to move along the grid diagonal at each time step; at each time step t, each AGV will get an action a according to its own strategy π t ∈A, the overall action of the entire multi-AGV group is n represents the number of AGVs; The purpose of the reward function is to guide the behavior of AGV through a reward and penalty mechanism. At each time step t, the AGV will execute an action scope grid map environment. The environment will give the AGV a reward value based on the global environment information. The goal of the entire AGV group is to maximize the cumulative value of the reward given by the environment. When the AGV successfully reaches the target point, the environment will feedback a one-time arrival reward with a value of g. When the AGV collides with an obstacle or other AGV during the journey, it will receive a collision penalty with a value of -g from the environment. Furthermore, in order to encourage the AGV to reach the target point as quickly as possible, a step penalty with a value of -f is introduced in the grid map environment. Before the AGV reaches the target location, it will receive a step penalty for each time step. In addition, considering that turning may cause the AGV to consume extra time, the AGV will receive an additional turning penalty each time it turns, so as to encourage the AGV to take the most direct path possible and reduce unnecessary turning behaviors. Finally, in order to enable the AGV to continuously move towards the target location, the change in the distance d between the AGV and the target position before and after the AGV executes the action will be calculated by formula (2), and the AGV will also receive a reward value.

4. The multi-automatic guided vehicle path planning method based on multi-agent reinforcement learning and context perception according to claim 1 is characterized in that: Step three is as follows: Initialize the AGV network and hybrid network; the AGV network includes a local feature information network, a context-aware information generation network, a communication information generation network, a communication information hybrid network, and a decision network; the hybrid network receives the global state information s at the current time step t t And the action value probability distribution output by all AGV networks, output the global action value probability distribution Q tot ; The parameters of the AGV network and the hybrid network are copied to the target AGV network and the target hybrid network; the target AGV network and the target hybrid network are copies of the AGV network and the hybrid network respectively. Every p rounds, the target AGV network and the target hybrid network will copy the weight information of the AGV network and the hybrid network respectively; Initialize the experience replay cache pool and set the capacity M; Set the total number of iterations epoch and the target network parameter update frequency p; After the training cycle starts, the time step is set to 0 at the beginning of each round, the grid environment is reset, and the initial global state information s and the initial observation information of each AGV are obtained. And initialize the long-term memory vector m of each AGV = {m1,m2,…,m n } is zero value.

5. The method for multi-automatic guided vehicle path planning based on multi-agent reinforcement learning and context perception according to claim 1, characterized in that: Step 4 is as follows: (4.1) Generation of AGV local feature information; The i-th AGV can obtain local observation information within its field of view at time step t. , and the action taken at the previous time step t-1 Then the local observation information and historical actions Splice together to get local information As the input of the local feature information network; the local feature information network consists of a fully connected layer and a gated neural network layer GRU, outputting the local feature information of the i-th AGV ; For n AGVs in the system, the overall local feature information will be generated accordingly (4.2) AGV context-aware information generation; The generation of context-aware information comes from the AGV's own understanding of the environment; at the current time step t, for the i-th AGV, the context-aware information generation network uses the long-term memory vector and the AGV's local observation information to generate context-aware information. and historical actions Generate contextual awareness information for the i-th AGV at the current moment Context-aware information generated It will be transmitted to the communication information generation network as part of the communication information. The specific implementation is as follows: The context-aware information generation network first transforms the long-term memory vector m i After a fully connected layer, the corresponding key is generated i , local observation information and the action at the previous time step After the fully connected layer, the query in the attention mechanism is generated i information, and then the long-term memory vector Local observation information and the action at the previous time step After concatenation, the fully connected layer generates the value information in the attention mechanism, and then the final contextual perception information about agent i is calculated using formula (3): in represents the parameter of the adjustment amplitude, exp(·) represents the exponential calculation operation, Represents query i The transpose of (4.3) AGV communication information generation; The generation of communication information requires the context perception information generated by the i-th AGV at the current time step t and the local information observed by the AGV The splicing is used as the input of the communication information generation network; specifically as follows: Since the local information observed by the AGV is actually represented by a vector in the multi-AGV system, but the different parts of this vector represent different meanings, the different information contained in the vector is distinguished and divided into self-feature information self_features, friendly feature information ally_features, and obstacle feature information enemy_features. At the same time, these information are reconstructed separately. Through the fully connected neural network, the one-dimensional feature information is reconstructed into two-dimensional feature information, and the reconstructed information self_feat, ally_feat, and enemy_feat are obtained respectively, as shown in formula (4); Among them, ent i Indicates the different meanings of the division, f k,z represents the elements in the reconstructed two-dimensional vector, k and z represent that the reconstructed two-dimensional vector is of (k, z) dimension; Then, these three reconstructed different feature information are combined with the context-aware information at the current time step Put together to get By integrating the multi-head attention mechanism and the self-attention mechanism, we finally get the communication information of the i-th AGV at the current time step t. And the update value m of the long-term memory vector i .

6. The multi-automatic guided vehicle path planning method based on multi-agent reinforcement learning and context perception according to claim 1 is characterized in that: Step 5 is as follows: In the communication phase, AGVs will perform a full communication behavior. Specifically, at the current time step t, the i-th AGV generates communication information , for all AGVs, the generated communication information can be expressed as Since it is a full communication behavior, the i-th AGV generates communication information Then it will be sent to all other AGVs; so after one full communication behavior, all AGVs can obtain the communication information c t ; Then, in order to generate diversified communication information, a communication information hybrid network is used to convert the communication information received by AGV into t Based on its local feature information Make changes; The communication information hybrid network consists of two fully connected layers, one of which inputs The result represents the weight w1, an input The absolute value of the result is taken to represent the offset value bais; the final communication information c′ of the ith AGV after communication is obtained through formula (5), where MLP represents the fully connected neural network layer; the communication information c′ and the local feature information of the ith AGV are combined The splicing is used as the input of the decision neural network of the i-th AGV, where the decision network consists of a GRU and a fully connected layer; finally, a vector of dimension num_action×1 is output, which represents the action value probability distribution Q of all actions that the i-th AGV can take at the current time step t. i ; After obtaining the probability distribution of the action value of each AGV at time step t, the specific action is selected by the ∈-greedy strategy shown in formula (6): Where idx represents the index value of the selected response; these actions will be executed at this time step t, so that the environment will transfer to the next state s′, and obtain the reward r obtained at the current time step and whether the environment is terminated; Randomly select a number between 0 and 1 as the parameter ε; randomaction means randomly selecting an action; store the state information s, next state s′, reward r and whether it is terminated at time step t into the buffer pool.

7. The method for multi-automatic guided vehicle path planning based on multi-agent reinforcement learning and context perception according to claim 1, characterized in that: Step 6 is as follows: When the number of data in the buffer pool reaches the predetermined number, the training phase begins. In the training phase, x data are randomly sampled from the buffer pool for training. The probability distribution of action values ​​generated by all AGVs will be transmitted to the hybrid network. Based on the received global state information s t , generate corresponding weights and biases, and obtain the final output of the hybrid network as the global action value probability distribution Q tot , and make it satisfy formula (7), where argmax represents the value of the independent variable when the given function reaches the maximum value, s is the global state information, and a is the global action information; The hybrid network consists of four fully connected layers, which receive the global state as input, thereby obtaining weights W1, W2 and biases b1, b2 respectively. Using formula (8), Q tot , where Q(τ,a)=(Q1,Q2,…,Q n ); Q tot =W2[W1Q(τ,a)+b1]+b2 (8) By obtaining Q tot , using the loss function of formula (9) Through deep learning back propagation training, the AGV network and hybrid network parameters are updated to obtain a multi-agent group control strategy with high-level cooperation and coordination. In formula (9), r represents the reward, represents the target network, where θ and represents the network parameters, γ represents the discount factor, represents the data set in the buffer pool, s′ represents the global state information of the next time step, and a′ represents the joint action information of all AGVs in the next time step; Indicates that in the next time step, all possible joint actions a′ are traversed and the one that can make Q tot The biggest joint action; In addition, every p rounds, the target AGV network and target hybrid network parameters are updated.

Images