Multi-AGV distributed path planning method based on graph reinforcement learning

Abstract

The invention provides a multi-AGV distributed path planning method based on graph reinforcement learning, which is used for solving the technical problem of low path planning efficiency of a multi-AGV system in an information limited dynamic environment. The method comprises the following steps: establishing a dynamic heterogeneous graph model in multi-AGV collaborative path planning; constructing a distributed partially observable Markov decision model based on a graph structure in combination with path planning and collaborative decision; on the basis of a multi-agent deep reinforcement learning framework of a graph neural network and a multi-head attention mechanism, a multi-AGV cooperative path planning algorithm for coping with an information-limited dynamic environment is designed, information aggregation is realized on the basis of local perception and neighborhood interaction by adjusting weights of nodes and edges in a dynamic heterogeneous graph, and a multi-AGV cooperative path planning algorithm for coping with an information-limited dynamic environment is realized. And the environment change is updated and adapted in real time. According to the method, the path planning efficiency can be improved through an information aggregation mechanism in an information-limited dynamic environment, multi-AGV cooperative path planning is optimized through the graph neural network, and efficient and stable operation of the system is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of automated logistics and path planning in the field of intelligent manufacturing, in particular to a multi-AGV collaborative path planning method, and more particularly to a distributed path planning method for multi-AGV based on graph reinforcement learning. Background Art

[0002] With the rapid development of intelligent manufacturing and industrial automation, industrial equipment such as automated guided vehicles (AGVs) and robots are widely used in intelligent factories and warehousing logistics systems. To improve production efficiency and flexibility, AGVs usually perform tasks in complex dynamic environments, such as material transportation, goods handling, etc. The multi-AGV system can complete tasks more effectively through coordinated work, but in practical applications, it faces various challenges, especially in path planning.

[0003] In a multi-AGV system, due to the large number and wide distribution of AGVs, how to efficiently and safely plan the path of each AGV has become a key issue. Traditional path planning methods, such as graph search, heuristic algorithms, etc., usually rely on global information and assume that the environment changes relatively slowly, and cannot effectively handle dynamic obstacles and environmental changes. In addition, these methods usually do not consider the cooperation between AGVs, which easily leads to low path planning efficiency, especially in dynamic environments.

[0004] In recent years, methods based on reinforcement learning have gradually been applied to multi-AGV path planning. Reinforcement learning can automatically adjust decision-making strategies by interacting with the environment to adapt to complex dynamic environments. However, directly applying reinforcement learning to a multi-AGV system still faces two major challenges: one is the problem of doubling the system interaction complexity caused by the increase in the number of devices, and the locality of path decisions significantly hinders the overall operation efficiency; the other is the incompleteness of information caused by limited sensing range, and the one-sided environmental perception caused by communication bandwidth and sensor coverage limitations results in a significant reduction in the path planning decision-making efficiency. This limitation of local observation affects the collaborative decision-making efficiency of the system, especially in the case of multi-AGV interaction and dynamic obstacles.

[0005] To solve these problems, methods based on graph neural networks (GNNs) and multi-agent reinforcement learning (MADRL) have gradually attracted attention. Graph neural networks can effectively handle the dynamic changes of nodes (AGVs) and edges (relationships between AGVs) in an AGV system, thereby improving the efficiency of path planning and the collaborative ability of the system. By combining GNNs with MADRL, AGVs can make collaborative decisions based on local perception information and interactions with other AGVs without relying on global information.

[0006] The invention patent with the application number 202111614329.3 discloses an AGV path planning based on deep reinforcement learning. A simulation environment is built according to the ros operating system, and an AGV model is built; the states, actions, and rewards of reinforcement learning are designed, and an algorithm framework for deep reinforcement learning is built; the network is trained to enable the AGV to learn strategies from the environment; the trained network is used to achieve the path planning of the AGV. The above invention obtains the optimal action from the original image obtained from the AGV camera and automatically extracts features; it solves the problem that traditional algorithms are prone to falling into local minima. It does not require prior knowledge of the environment, explores the environment by itself, and learns feasible strategies; when the environment changes in real time or the environmental conditions are complex, it can dynamically make changes to avoid dynamic obstacles and reach the target point. The DDPG algorithm adopted by the above patent is a single-agent algorithm, lacking a mechanism for handling multi-AGV collaborative decision-making; its network input dimension is fixed, unable to adapt to changes in the number of AGVs, with poor generalization; its performance is limited under local observation conditions; when multiple AGVs learn simultaneously, they face a non-stationary environment, with unstable training and low sampling efficiency; it cannot effectively model and utilize the topological relationship between AGVs. Summary of the Invention

[0007] Aiming at the technical problem of low path planning efficiency of multi-AGV systems in information-constrained dynamic environments, the present invention proposes a multi-AGV distributed path planning method based on graph reinforcement learning. Through a dynamic heterogeneous graph model and an information aggregation mechanism, combining graph neural networks and multi-agent reinforcement learning, it can better achieve path planning and collaborative operations in dynamic environments, significantly improving the efficiency and safety of the AGV system, and enhancing the path planning efficiency in complex dynamic environments and the collaborative operation ability between AGVs. The present invention can be used to optimize the AGV path planning and collaborative operation efficiency in dynamic and information-constrained environments.

[0008] In order to achieve the above object, the technical solution of the present invention is realized as follows: A multi-AGV distributed path planning method based on graph reinforcement learning, the steps are as follows:

[0009] Step 1: Establish a dynamic heterogeneous graph model in multi-AGV collaborative path planning;

[0010] Step 2: Combine path planning and collaborative decision-making to construct a distributed partially observable Markov decision model based on a graph structure;

[0011] Step 3: Based on a multi-agent deep reinforcement learning framework of graph neural networks and multi-head attention mechanisms, design a multi-AGV collaborative path planning algorithm for dealing with information-constrained dynamic environments. By adjusting the weights of nodes and edges in the dynamic heterogeneous graph, information aggregation is achieved based on local perception and neighborhood interaction, and the environment changes are updated and adapted in real time.

[0012] Preferably, the architecture of the multi-AGV collaborative path planning includes M AGVs and an edge server. The edge server works in collaboration with the AGVs by centrally training a path planning model. The AGVs are connected to the edge server via wireless communication. The AGVs make collaborative decisions through local perception and neighborhood interaction to form a dynamic heterogeneous graph.

[0013] At time step t, the dynamic heterogeneous graph is represented as G t =(X t , Adj t , W t ), where X t is the node feature matrix, containing the position, velocity, target position, and entity type information of each entity at time step t; Adj t is the adjacency matrix of node connectivity, and the edge weight W t reflects the interaction intensity between entities. The edge weight W t consists of an initial weight W0 and a dynamic weight increment ΔW t . The initial weight W0 constructs a basic interaction structure based on entity types; the dynamic weight increment ΔW t is calculated through a multi-head edge attention mechanism based on entity position offset, speed change, and obstacle influence. During the task execution of the AGV, it dynamically interacts with environmental elements, and the dynamic heterogeneous graph adjusts the connection relationship and interaction intensity between nodes in real time, enabling the AGV to adapt to obstacle changes and speed adjustments in path planning in real time.

[0014] Preferably, the method for constructing a distributed partially observable Markov decision model is as follows:

[0015] In the architecture of multi-AGV collaborative path planning composed of M AGVs, each AGV integrates its own state, the states of other entities, and their interaction relationships within its perception range based on the dynamic heterogeneous graph, and dynamically responds to environmental changes. The implementation method is as follows: At time step t, the m-th AGV m has an observation space of: where represents the speed of the m-th AGV m at time step t, m ∈ {1, 2,..., M}, represents the current position of the m-th AGV m , represents the target position of the m-th AGV m , e m is the entity type of the m-th AGV m , ID m is the unique identifier for distinguishing the m-th AGV m , represents the m-th AGV at time step t mThe interaction relationships with other entities within the perception range and the interaction relationships with other AGVs;

[0016] The overall observation space is represented as: Where, are the observation spaces of the 1st, 2nd... Mth AGVs respectively;

[0017] The observation space is constructed through a dynamic heterogeneous graph, which captures the interaction relationships between each AGV and other entities in the environment and reflects the changes in the environment and system state in real time by dynamically updating the graph structure;

[0018] At time step t, the mth AGV m selects an action according to the existing state where k ∈ {0, 1, 2, 3, 4}, and its values represent different movement directions; the discrete action is converted into a continuous control force through the mapping function g That is affects the acceleration a m of the mth AGV m ;

[0019] At time step t, the composite reward function r m of the mth AGV m (t) is:

[0020]

[0021] Where, is the reward value for encouraging reaching the specified target, is the reward value obtained for maintaining a safe distance, is the reward value for maintaining stable and coordinated movement, is the reward value for efficient movement behavior, is the reward value for following the RRT planned path.

[0022] Preferably, the acceleration where μ represents the environmental friction coefficient, m AGV is the mass of the mth AGV m , is the speed of the mth AGV at time step t + 1 m , and Δt represents the update step size;

[0023] The reward value for reaching the specified target is:

[0024]

[0025] Where, R goal is the reward for the AGV reaching the target position, R timeis the penalty when the target is not reached in each step;

[0026] The reward value obtained by maintaining a safe distance is: where R collsion is the penalty when the distance to an obstacle or other AGV is less than the safe distance;

[0027] The reward value for maintaining stable and coordinated movement is: where Δd is the progress in the current target direction, d actual is the total moving distance, ω eff is the weight coefficient of the path efficiency;

[0028] The reward value for efficient movement behavior is: where ω move is the weight coefficient for controlling the consistency of the movement direction, and cosθ is the cosine value of the angle θ between the current movement direction and the target direction;

[0029] The reward value for following the RRT planned path is expressed as: where is the position of the guiding path where the AGV is closest to the RRT path.

[0030] Preferably, the multi-AGV collaborative path planning algorithm includes a neural network model, and the neural network model includes an Actor network deployed on the AGV and a Critic network deployed on the edge server; the Actor network consists of a first MLP layer, a Noisy layer, a first GRU module, and an output layer connected in sequence. The first MLP layer processes the local environment information from the graph neural network, encodes the obtained local features as the local environment information of the AGV, and transforms the local environment information through a learnable weight matrix W and a non-linear ReLU activation function to obtain local features; the Noisy layer introduces a noise parameter N(0,σ noise ), processes the transformed local features, and introduces Gaussian noise to achieve policy exploration, where σ noise is the standard deviation parameter of the Gaussian noise introduced in the Noisy layer; the GRU module is used to update the historical state, adjusts the information flow between the feature with Gaussian noise introduced and the historical state through an update gate, and generates a final decision through the Softmax function; based on the obtained features and decisions, the output layer generates a path planning strategy π(a t |o t ) of the AGV to achieve distributed collaborative path planning decisions for multiple AGVs;

[0031] The Critic network consists of a second MLP layer, a second GRU module, and a value function layer connected in sequence; the second MLP obtains global features through global information aggregation of the graph neural network, and after being processed by a weight matrix and a non-linear ReLU activation function, extracts global dynamic features; the second GRU module takes the global dynamic features as input, uses the control gate mechanism to combine historical state information, and captures and updates the dynamic evolution of the system; the value function layer combines the hidden state of the second GRU model with the global dynamic features, and calculates the global state value V(s t ,H global ) to provide value evaluation for policy optimization.

[0032] Preferably, during the training of the multi-AGV collaborative path planning algorithm, each AGV outputs an action according to the observation space and obtains a reward using a composite reward function. The Critic network judges the action value according to the reward and updates the network. The implementation steps are as follows:

[0033] S331: Construct graph node features based on the local environment information sensed by each AGV to form a dynamic heterogeneous graph G t , and calculate the interaction strength between nodes through the multi-head attention mechanism to obtain a weighting coefficient, and dynamically update the weight of the edge;

[0034] S332: The graph neural network uses the graph node features of each AGV to update its own local features through weighted aggregation; uses the Transformer convolutional layer to update the graph node features of the AGV through weighted summation and aggregation of neighborhood node features;

[0035] S333: After the local feature update, the graph neural network uses the global pooling operation in global information aggregation to integrate the local features of all AGVs to generate global features, and the global features are input into the Critic network to evaluate the state value of the overall environment;

[0036] S334: The updated local features are input into the Actor network to guide the path planning decision of each AGV.

[0037] Preferably, the update of the local features is realized by the ReLU activation function and weighted summation, and the formula is:

[0038]

[0039] where h m represents the current feature vector of the AGV, W0 is a learnable weight matrix, N(m) is the set of other AGV nodes adjacent to the m-th AGV m , is the weighting coefficient calculated according to the interaction relationship between nodes, hj represents the feature vectors of other AGVs adjacent to the feature vector h m

[0040] Preferably, an Embedding layer is provided between the multi-head attention mechanism and the Transformer convolutional layer to convert each AGV and its corresponding entity type into an embedding vector of a fixed dimension. The embedding vector is concatenated with the original features of the node to generate a feature vector that combines all key information. If there are edge features, they are added to the concatenated feature vector. The graph neural network includes an initial Transformer convolutional layer for processing preliminary feature updates. According to the layer_N parameter, several Transformer convolutional layers are added, and the features are gradually processed and information is aggregated through multiple layers of Transformer convolutions to optimize the feature representation.

[0041] Preferably, the decision-making of path planning is optimized through information aggregation and reinforcement learning, including the following steps:

[0042] 1) Initialize the parameters of the neural network model composed of the Actor network and the Critic network and the graph neural network;

[0043] 2) According to the communication and perception range of the multi-AGV distributed path planning, construct a dynamic heterogeneous graph. Each AGV interacts with the environment according to its perception range, and transmits the features of adjacent AGVs and the information of static and dynamic obstacles in the environment to the node features of the dynamic heterogeneous graph. For AGVs or obstacles not within the perception range, their features are filled with zeros;

[0044] 3) The AGV obtains node and edge information based on the interaction data, and aggregates the state information of the environment and other AGVs through the graph neural network;

[0045] 4) Collect the complete observation information, aggregated information, state, action, and reward of the AGV as an experience trajectory and store them in the buffer as sampling data to train the neural network model;

[0046] 5) Repeat steps 1) to 4) until the number of simulation rounds of training is completed. After training, each AGV makes an optimal path decision based on the local graph structure observation and the real-time changes of the dynamic environment, and collaborates with other AGVs.

[0047] Preferably, the graph neural network aggregates the local information of different AGVs and the environment to obtain aggregated information, dynamically adjusts the edge weights between nodes according to the perception range of each AGV and its interaction relationship with other AGVs, and is dynamically updated according to the relative position and speed difference between the AGV and other entities, including the following steps:​

[0048] (1) Collect the status, actions, and rewards of each AGV and put them into the interaction information. All the interaction information is integrated and constructed into a training dataset;

[0049] (2) Use a graph neural network to construct a dynamic heterogeneous graph to aggregate and update the local information from other AGVs and the environment;

[0050] (3) Use one-hot encoding to classify the feedback signals, and perform feature extraction and integration through a graph neural network;

[0051] (4) Combine the observation data of other AGVs, and the neural network model optimizes the path planning strategy through a centralized training and distributed execution architecture;

[0052] (5) Make path decisions based on real-time environmental changes and system states, and cooperate with other AGVs;

[0053] The feedback signal is obtained after each AGV interacts with the environment during path planning. The AGV compares the actions it executes with the changes in the environment to obtain the feedback signal.

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0055] 1. The multi-AGV distributed path planning method based on graph reinforcement learning proposed by the present invention can improve the path planning efficiency through an information aggregation mechanism in an information-constrained dynamic environment, and optimize the multi-AGV collaborative path planning through a graph neural network (GNN) to ensure the efficient and stable operation of the system.

[0056] 2. The present invention demonstrates strong scalability and can adapt to changes in the number of different AGVs and environmental topologies. When the system scale changes, the present invention can maintain stable performance, showing appropriate generalization ability and environmental adaptability, thus effectively supporting the operation requirements of different scale application scenarios, which reflects the strong scalability of this method.

[0057] 3. The present invention has strong robustness. Through the designed information aggregation mechanism, it ensures that AGVs can effectively cooperate with other AGVs under the condition of limited local information, and improves the robustness of collaborative decision-making through a graph neural network, maintaining an efficient and stable path planning ability in a dynamic environment. Description of the Drawings

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0059] Figure 1 This is the flowchart of the present invention.

[0060] Figure 2 This is the architecture diagram of the AGV warehousing logistics system based on the dynamic heterogeneous graph of the present invention.

[0061] Figure 3 This is the schematic diagram of the dynamic heterogeneous graph model of the present invention.

[0062] Figure 4 This is the schematic diagram of the multi-agent deep reinforcement learning model based on the fusion of graph neural network (GNN) and multi-head attention mechanism of the present invention.

[0063] Figure 5 This is the cumulative reward curve graph of the simulation under different configurations of the present invention.

[0064] Figure 6 This is the cumulative reward curve graph of the simulation under different perception ranges of the present invention. Detailed implementation manners

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0066] As Figure 1 shown, the present invention proposes a multi-AGV distributed path planning method based on graph reinforcement learning, which comprehensively considers the local perception and global cooperation requirements of the multi-AGV system, realizes the maximization of decision-making efficiency while ensuring system safety, ensures the efficient cooperation and stable operation of the multi-AGV system in a dynamic and complex environment, and has high path planning efficiency and system stability. The specific steps are as follows:

[0067] Step 1: Establish a dynamic heterogeneous graph model in multi-AGV collaborative path planning.

[0068] As Figure 3As shown in the figure, in order to alleviate the system dynamic changes caused by the limited local perception of AGVs, a dynamic heterogeneous graph model applicable to multi-AGV collaborative path planning is first established. The architecture of multi-AGV collaborative path planning includes M AGVs and an edge server. The edge server collaborates with the AGVs by centrally training a path planning model. The AGVs are connected to the edge server via wireless communication. The AGVs make collaborative decisions through local perception and neighborhood interaction to form a dynamic heterogeneous graph structure.

[0069] At time step t, the dynamic heterogeneous graph is represented as G t =(X t , Adj t , W t ), where X t is the node feature matrix, which contains the position, speed, target position, and entity type information of each entity in the system at time step t. Adj t is the adjacency matrix of the connectivity between entities. When Adj t (i, j)=1, it means there is a connection between AGVi and AGVj. When Adj t (i, j)=0, it means there is no connection. The edge weight W t reflects the interaction intensity between entities and is composed of the initial weight W0 and the dynamic increment ΔW t : W t (i, j)=W0(i, j)+ΔW t (i, j). The initial weight W0 is based on the entity type to construct a basic interaction structure. The dynamic increment ΔW t is based on the entity position offset, speed change, and obstacle influence to achieve real-time environmental adaptation. The initial weight W0(i, j)=sim(type i , type j ), where sim is the type similarity matrix, and is used to guide the interaction behavior in the multi-agent system, enabling the agents to efficiently find paths and avoid obstacles. Positive values promote the approach between entities (such as an agent searching for a target), negative values prompt the separation of entities (such as an agent avoiding an obstacle), and the numerical magnitude represents the interaction intensity. This design enables the system to achieve collaborative path planning and collision avoidance in a complex environment. The dynamic increment ΔW t is calculated through the multi-head edge attention mechanism:

[0070]

[0071] where the interaction intensity of the h-th attention head is calculated as:

[0072]

[0073] where, and is a learnable weight matrix, h i and h j are the feature representations of nodes i and j respectively (including position, velocity, target position, and entity type information), and d k is the dimension of the h-th attention head. In this way, the system can dynamically adjust the interaction strength between nodes to achieve real-time perception and adaptation to the environment.

[0074] In multi-AGV collaborative path planning, AGVs make decisions through local perception information and neighborhood interaction, forming a dynamic heterogeneous graph model. During the path planning process, the interaction between AGVs and environmental changes have an important impact on path planning. Since each AGV can only perceive local information, the effectiveness of task planning and decision-making depends on the cooperation between AGVs. To cope with the dynamic changes in the environment, the heterogeneous graph can dynamically adjust the connection strength between nodes to ensure that the AGV cluster can respond to the changing factors in the environment in real time. This mechanism enables AGVs to quickly adapt to obstacles on the path and make corresponding speed adjustments, thereby improving the overall system's cooperation efficiency and making path planning more flexible and efficient.

[0075] Figure 3 shows the relationship changes of AGVs, targets, and obstacles in the dynamic heterogeneous graph at time steps t = 0, t = 1, and t = 2. The time step is the change per second, indicating how the heterogeneous graph adjusts the connection relationship and interaction strength between nodes in real time, reflecting the dynamic interaction between AGVs and environmental elements during the task execution process.

[0076] Figure 2 In the AGV warehousing and logistics system architecture based on heterogeneous graphs, in multi-AGV collaborative path planning, AGVs make decisions through local perception information and neighborhood interaction, forming a heterogeneous graph model. In this model, AGVs, shelves, dynamic obstacles, and human workers are used as heterogeneous nodes, and the decision execution and information aggregation paths form the connection relationship of edges. During the path planning process, the interaction between AGVs and environmental changes have an important impact on the path plan, which is reflected in the dynamic adjustment of the graph structure. Since each AGV can only perceive local information, an architecture of "local perception - global training - distributed execution" is adopted: AGVs upload observation data to the edge server, the edge server integrates information to calculate and optimize strategies, and then AGVs receive and execute these optimized strategies. To cope with the dynamic changes of the system and the real-time update of the environment, the structure of the heterogeneous graph continuously adjusts the interaction strength between nodes to ensure that AGVs can adapt to the obstacle changes and speed adjustments in path planning in real time, optimizing the overall cooperation effect.

[0077] Step 2: Considering the local perception and global cooperation requirements in the multi-AGV system, combining path planning and collaborative decision-making, construct a distributed partially observable Markov decision model based on the graph structure.

[0078] Since the distributed AGV is in a partially observable state, each AGV can only observe its own access information and cannot obtain global information. To solve this problem, the present invention first models the path planning problem of the distributed AGV as a distributed partially observable Markov decision model according to the following steps:

[0079] The observable Markov decision model includes a graph-structured observation space, an action space, and a reward function. First, in a system composed of M AGVs, each AGV integrates its own state, the states of other entities, and their interaction relationships within its sensing range based on a dynamic heterogeneous graph structure, dynamically responds to environmental changes, and provides support for path planning decisions.

[0080] At time step t, the m-th AGV m (m ∈ {1, 2,..., M}) has an observation space of The specific definition is as follows:

[0081]

[0082] where, represents the speed of the m-th AGV m at time step t, represents the current position of the m-th AGV m , represents the target position of the m-th AGV m , e m is the entity type of the m-th AGV m and here e m = 0. According to the entity type definition, e m ∈ {0, 1, 2, 3}, where e m = 0 represents an AGV, e m = 1 represents a static obstacle, e m = 2 represents a dynamic obstacle, e m = 3 represents a target point. ID m is the unique identifier for distinguishing the m-th AGV m , represents the m-th AGV at time step t mThe interaction relationships with other entities within the perception range and the interaction relationship with the AGV. Each AGV collects environmental data through its local sensors, such as position, speed, obstacles, target position, etc. This information is used to construct a dynamic heterogeneous graph and is updated at each time step to reflect changes in the environment. The graph model contains node information (such as position, speed, target position, etc.) and the relative relationships with obstacles and other AGVs. The AGV exchanges data with the edge server or other AGVs, including its own position, speed, target position, obstacle information, and relative position with other AGVs, etc. This information helps to achieve collaborative decision-making, enabling the entire system to optimize path planning and ensure the coordination and safety of the multi-AGV system in a dynamic environment.

[0083] The overall observation space of the system can be expressed as:

[0084]

[0085] Where, are the observation information of the 1st, 2nd... Mth AGVs respectively. Each AGV constructs its observation space based on its local sensor data.

[0086] The observation space is constructed through a dynamic heterogeneous graph. The dynamic heterogeneous graph captures the mutual relationships between each AGV and other entities in the environment (such as obstacles, other AGVs, target points, etc.), and reflects the changes in the environment and system state in real time by dynamically updating the graph structure (node features, adjacency matrix, and edge weights). In this way, the AGV can make path planning and decisions based on its own and other AGVs' current states.

[0087] At time step t, the mth AGV m selects an action according to its current state, denoted as where k ∈ {0, 1, 2, 3, 4}, which represent different movement directions respectively. In the specific implementation process, the discrete action is converted into a continuous control force through the mapping function g (that is ), thus affecting the acceleration a m of the mth AGV m , which is defined as follows:

[0088]

[0089] Where, μ represents the environmental friction coefficient, m AGV is the mass of the mth AGV m , is the speed of the mth AGV m at time step t + 1, and Δt represents the update step in time.

[0090] Through the above method, the AGV can adjust its acceleration according to the local environment and status, thus achieving smooth and continuous movement. This dynamic control can adapt to complex and changing environments, ensuring the stability and efficiency of the AGV system in practical applications. This process uses a distributed partially observable Markov decision process (Dec-POMDP) model, enabling the AGV to make decisions based on local observation data and then achieve path planning within a limited sensing range. At the same time, the speed and acceleration control of the AGV smooth its movement, ensuring the coordinated and efficient operation of the system.

[0091] The method for designing the reward function by comprehensively considering the system decision-making efficiency and safety is as follows:

[0092] To improve the path planning efficiency and safety of the AGV, the reward function of the AGV at each time step t consists of multiple parts, each corresponding to different optimization goals. Combining these goals can maximize the overall performance of the system.

[0093] First, to prompt the AGV to reach the target position in the shortest time, It is defined as:

[0094]

[0095] Among them, R goal is the reward for the AGV to reach the target position, and R time is the penalty for each step when the target is not reached, used to encourage completing the task as soon as possible. The reward R goal is a positive value and is significantly greater than the absolute value of the penalty R time , large enough to motivate the behavior of achieving the goal, usually set according to the importance of completing the task; the penalty R time is a negative value, and its value should be such that it does not overwhelm the target reward and is sufficient to effectively encourage efficiency, usually proportional to the expected time to complete the task.

[0096] Secondly, to maintain a safe distance between the AGV and other obstacles (including dynamic obstacles and other AGVs), It is defined as:

[0097]

[0098] Among them, R collsion is the penalty when the distance from the obstacle or other AGV is less than the safe distance. The penalty R col is a negative value, and its absolute value should be large enough to ensure that safety is the top priority. The value can increase the penalty intensity as the distance from the obstacle decreases, ensuring that the AGV maintains a safe distance.

[0099] Then, to guide the stability and coordination of the movement speed and direction of the AGV, It is defined as:

[0100]

[0101] Among them, Δd is the progress in the target direction in the current step, and d actual is the total moving distance, and ω eff is the weight coefficient of the path efficiency; Δd is the progress in the direction of the target in the current step, which can be positive or negative, depending on whether it is moving in the direction of the target, and the unit is usually the same as the measurement unit of the environment. The basic weight coefficient takes a value of 1.5 and increases as the AGV approaches the target distance, so as to pay more attention to the path efficiency when approaching the target.

[0102] Secondly, in order to evaluate the consistency between the moving direction of the AGV and the target direction, the motion efficiency reward at time step t is expressed as:

[0103]

[0104] Among them, ω move is the weight coefficient that controls the consistency of the moving direction in the overall reward function, and cosθ is the cosine value of the angle θ between the current moving direction and the target direction. ω move Generally, a value in the range of [0.5, 0.8] is selected to adjust the alignment degree between the actual moving direction of the AGV and the target direction. When the value approaches the upper limit of 0.8, it will strengthen the accuracy of the agent aligning with the target direction, thereby improving the overall efficiency and accuracy of the path.

[0105] Finally, RRT (Rapidly-exploring Random Tree) can strictly follow the pre-generated global reference trajectory and avoid deviating from the preset path. The RRT guidance reward at time step t is expressed as:

[0106]

[0107] Among them, is the position of the guiding path that is closest to the RRT path of the AGV.

[0108] To sum up, at time step t, the composite reward function r m of the m-th AGV m (t) can be expressed as:

[0109]

[0110] The composite reward function r m(t) is used to train the decision-making model of the AGV. In a multi-AGV system, each AGV adjusts its behavior strategy according to its current state and feedback from the environment (such as goal achievement, obstacle avoidance, efficiency, etc.). Through the multi-agent reinforcement learning algorithm, each AGV optimizes its path planning strategy based on these reward signals. After training, the AGV will be able to make optimal path decisions according to local perception information and environmental changes, thereby improving the overall efficiency and safety of the system.

[0111] Step 3: Based on the multi-agent deep reinforcement learning framework of graph neural network (GNN) and multi-head attention mechanism, a multi-AGV collaborative path planning algorithm for dealing with information-limited dynamic environments is designed. By adjusting the weights of nodes and edges in the dynamic heterogeneous graph G t information aggregation is achieved based on local perception and neighborhood interaction. By updating and adapting to environmental changes in real time, each AGV can flexibly adjust its path decision, ensuring the efficient collaboration and stable operation of the multi-AGV system in a dynamic and complex environment. The overall network architecture of the algorithm is as Figure 4 shown.

[0112] First, build an Actor network deployed on the AGV; the core architecture of the Actor network consists of an MLP layer, a Noisy layer, a GRU module, and an output layer connected in sequence.

[0113] The MLP layer processes the local environmental information from the GNN, and the integrated local features h m are encoded as the local environmental information of the AGV. The local environmental information is transformed through a learnable weight matrix W and a non-linear ReLU activation function. Subsequently, the Noisy layer introduces a noise parameter N(0,σ noise ), processes the transformed features, and introduces Gaussian noise to achieve policy exploration. Specifically, the noise formula is through added to the policy, so that the system conducts a large-scale exploration through noise guidance in the initial stage. As the training process progresses, the noise gradually decreases, avoiding the model falling into local optimal solutions, thereby supporting the stability of the policy. σ noise is the standard deviation parameter of the Gaussian noise introduced in the Noisy layer. A t represents the initial value without added noise; represents the value after adding Gaussian noise. The GRU module is used to update the historical state, and the information flow between the current input and the historical state is regulated through the update gate z t so that the model can flexibly select control weights and generate the final decision through the Softmax function. Finally, based on these features and decision information, the output layer generates the path planning strategy π(a t |o t), thus realizing the distributed cooperative path planning decision-making of multiple AGVs.

[0114] Secondly, build a Critic network deployed on the edge server. The core architecture of the Critic network consists of an MLP layer, a GRU module, and a value function layer connected in sequence.

[0115] The MLP layer first obtains the global feature H through the global information aggregation of the GNN global , and these features summarize the state information of all AGVs. Then, through the processing of the weight matrix and the non-linear ReLU activation function, the global dynamic features are extracted, providing a basis for the subsequent path planning model. Next, the GRU module takes these features as input, uses its control gate mechanism to combine with the historical state information, and captures and updates the dynamic evolution of the system. The value function layer combines the hidden state h of the GRU model t with the global dynamic features, and calculates the global state value V(s t , H global ) through the learned mapping function, providing value evaluation for policy optimization.

[0116] Finally, design a GNN information aggregation mechanism to address the problem of incomplete decision-making caused by limited local perception in a multi-AGV system, that is, insufficient local perception information in a dynamic environment. Integrate the local information from each AGV through the GNN information aggregation mechanism to ensure that the system can effectively cooperate in decision-making and improve the accuracy of path planning. The present invention obtains a multi-AGV collaborative path planning algorithm for coping with information-limited dynamic environments through graph neural networks, Actor networks, and Critic networks. During training, each AGV outputs actions according to the observation space and obtains rewards. The Critic network judges the value of the output actions according to the rewards and updates the network. The specific implementation steps are as follows:

[0117] S331: Based on the local environmental information (such as position, speed, target position, etc.) perceived by each AGV, construct graph node features to form a dynamic heterogeneous graph G t , calculate the interaction intensity between nodes through the multi-head attention mechanism (MHA-Interaction), and dynamically update the weights of the edges. Specifically, the MHA mechanism adjusts the interaction weights according to the relative positions, speed differences, and obstacle effects between nodes, thereby realizing strong dynamic state updates. In this way, the interaction intensity between nodes affects the subsequent path planning calculations, ensuring that each AGV can reflect the changes in the surrounding environment in real time and optimize decisions based on this.

[0118] S332: The GNN utilizes the local node feature information of each AGV and updates its own features through weighted aggregation. During this process, the Transformer convolutional layer is used to update the features of the AGV (such as position, speed, target position, etc.) through weighted summation and aggregation of neighborhood node features. Specifically, the node feature update is achieved through the ReLU activation function and weighted summation, and the formula is:

[0119]

[0120] where h m represents the current feature vector of the AGV, W0 is the learnable weight matrix, N(m) is the set of other AGV node neighbors adjacent to the m-th AGV m , is the weighted coefficient calculated according to the interaction relationship between nodes. h j represents the feature vectors of other AGVs adjacent to the feature vector h m , and these features are obtained by aggregating the information of neighbor nodes through the GNN. represents the interaction intensity or relationship between nodes, and the weighted coefficient is achieved through the MHA mechanism. The MHA mechanism dynamically calculates the interaction intensity between each pair of nodes according to the relative position, speed, and obstacle information between nodes. In addition, h m on the left side of the equation is the updated node feature, and the feature vectors h m and h j on the right side are the features of the node itself and its neighbor nodes respectively. Through this weighted sum and aggregation method, the feature vector h m combines the information of itself and neighbor nodes, thus achieving dynamic feature update to better reflect the changes and states of the environment in subsequent path planning. Through this local information aggregation process, the AGV can capture and respond to the dynamic changes in the environment in real time.

[0121] S333: After local feature update, the GNN integrates the local features of all AGVs using the global pooling operation in global information aggregation to generate the global feature H global . These global features H global capture the dynamic changes of the entire system and provide a global perspective for the subsequent decision-making process. The global feature H global is then input into the Critic network to evaluate the state value of the overall environment.

[0122] S334: After local feature aggregation, it is input into the Actor network to guide the path planning decision of each AGV. At the same time, the global feature H globalInput into the Critic network to evaluate the state value of the system. That is, through the fusion of local information and global information, it can make immediate decisions based on local information while comprehensively considering the global environment, thus ensuring the efficiency and safety of path planning.

[0123] Figure 4 The Embedding in [reference] processes data by converting each AGV and its corresponding entity type into an embedding vector of a fixed dimension. These embedding vectors are concatenated with the original features of the nodes (such as position, speed, etc.) to generate a feature vector that combines all key information. After that, if there are edge features (such as the relative position or speed information between AGVs), these edge features are also added to the concatenated feature vector and processed through multiple linear layers to obtain the updated node features. The number of Transformer convolutional layers is determined by the parameter layer_N of the model. Specifically, the model contains an initial Transformer convolutional layer for processing preliminary feature updates, and then several Transformer convolutional layers are added according to the layer_N parameter. These layers gradually process features and aggregate information through multiple layers of Transformer convolutions to optimize the feature representation.

[0124] The Local Aggregation in information aggregation is obtained through the local information aggregation process of the GNN module. The features of each AGV node (such as position, speed, etc.) and the features of neighboring nodes are weighted and aggregated through MHA-Interaction to form updated local features. These features reflect the interaction state of the current AGV and its neighboring nodes, and after local information aggregation, they provide a solid foundation for subsequent global information processing.

[0125] The objectives include classification objectives and continuous numerical objectives. Among them, the classification objectives include the discrete action selection of AGVs during path planning, while the continuous numerical objectives include local perception information collected by sensors, environmental feedback, and the state value or reward value evaluated by the Critic network, which are used to guide the optimization of path planning decisions. During path planning, the objectives not only include the target position of the AGV (as a classification objective), but also involve continuous numerical objectives, which are based on local environmental information collected by sensors (such as obstacle distance, speed change, etc.). This information guides the optimization of the path planning strategy through the value evaluation and reward signal of the Critic network. Specifically, the Critic network calculates the value of the current state and provides guidance for policy optimization, enabling the AGV to select the optimal path and behavior in a dynamic environment.

[0126] The above content mainly involves the initial settings for path planning, the collection and processing of environmental information, and the preparations for constructing a dynamic heterogeneous graph model, providing a basis for subsequent path planning decisions. How to use graph reinforcement learning for decision optimization and collaborative path planning. As Figure 1 shown, a multi-AGV distributed path planning method based on graph reinforcement learning optimizes path planning decisions through information aggregation and reinforcement learning, mainly including the following steps:

[0127] 1) Initialize the parameters of the neural network model (Actor-Critic network) and the GNN module; including the Actor network (π θ ), the Critic network (V φ ), the learning rate (lr), and the number of attention heads.

[0128] 2) According to the communication and sensing ranges of the multi-AGV system, construct a dynamic heterogeneous graph. Each AGV interacts with the environment according to its sensing range, and transmits the features of adjacent AGVs and the information of static and dynamic obstacles in the environment into the node features of the heterogeneous graph. For AGVs or obstacles outside the sensing range, their features are filled with zeros to ensure the effective integration and transmission of information. The sensing range is usually determined through experiments or system requirements, and a radius that balances computational efficiency and performance is usually selected. Specifically, refer to the experimental results section Figure 6 .

[0129] 3) The AGV aggregates the state information of the environment and other AGVs through the node and edge information obtained from the interaction data. In a dynamic environment with limited information, the local information of different AGVs and the environment is integrated through an information aggregation mechanism to obtain aggregated information, enabling each AGV to make path decisions based on the updated global information.

[0130] In Figure 1 , the decision information aggregation is achieved by judging the mutual relationship between the local observation data of each AGV and the environmental information. Specifically, the node information from other AGVs and the environment is aggregated through a graph neural network (GNN), and the connection strength (edge weight) between nodes is dynamically adjusted according to the sensing range of each AGV and its interaction relationship with other AGVs. These weights are calculated through a multi-head attention mechanism and are dynamically updated based on information such as the relative position and speed difference between the AGV and other entities (such as obstacles, target points, etc.). The aggregation of decision information thus judges and integrates the local perception information of all AGVs, enabling the path planning decision to be optimized from a global perspective.

[0131] The design of the information aggregation mechanism includes the following steps:

[0132] (1)Collect the interaction information such as the status, actions, and rewards of each AGV. All this data will be integrated and constructed into a training dataset to provide a basis for subsequent path planning decisions.

[0133] (2)Adopt a GNN to aggregate and update the local information from other AGVs and the environment, enhancing the global perception ability of each AGV and compensating for the perception limitations of a single AGV in a dynamic environment;

[0134] The GNN processes the local information in a multi-AGV system by constructing a dynamic heterogeneous graph. Each AGV collects environmental information (such as position, speed, target position, etc.) through its sensors and represents this information as node features in the graph. At each time step, the AGV connects the state it perceives and its interactions with other entities (such as obstacles, target points, other AGVs) through edges. The GNN uses MHA-Interaction to dynamically calculate the interaction intensity between nodes and updates the features (the state of the AGV) of each node through weighted aggregation. In this way, real-time updates can be made based on the information of neighboring nodes, enabling flexible decisions to be made in a dynamic environment.

[0135] (3)Use one-hot encoding to classify the feedback signals, and extract and integrate the aforementioned information (such as status, actions, etc.) through a GNN to ensure effective information transmission and decision support; the feedback signals are obtained after each AGV interacts with the environment during path planning. The AGV compares the actions it executes (such as moving direction, speed change, etc.) with the changes in the environment (such as the change in distance from obstacles, the proximity to the target position) to obtain feedback. This feedback signal is usually given in the form of rewards or punishments. For example, when the AGV approaches the target, it will receive a reward; when the AGV gets close to an obstacle, it will be punished. The AGV stores these feedback signals together with its actions and status in the experience trajectory for subsequent neural network training.

[0136] (4)Combine the observation data of other AGVs and optimize the path planning strategy through a neural network to improve the decision-making efficiency and collaborative operation ability of the system in a dynamic environment. The system optimizes the path planning strategy through a centralized training and distributed execution architecture. During the training phase, the AGV aggregates the observation data of itself and other AGVs (such as position, speed, target position, obstacle position, etc.) into a graph structure. In the GNN, each AGV uses its local data (such as the current state, speed, target position, etc.) as node features and transmits and aggregates the information of neighboring AGVs through the connection relationship of edges. When processing this graph data, the Actor-Critic network optimizes the path planning strategy through the aggregation of local information, thus ensuring that the AGV makes the best decision.

[0137] (5) Make path decisions based on real-time environmental changes and system states, cooperate with other AGVs, and ensure the decision-making efficiency and safety of the path planning algorithm in a dynamic environment. During the path planning process, the AGV makes path decisions based on real-time local environmental information. Each AGV senses the surrounding environment through its sensors and constructs a dynamic heterogeneous graph. At each time step, the structure of the graph is dynamically updated, and the edge weights between nodes are also adjusted according to environmental changes (such as obstacles, changes in the positions of other AGVs, etc.). The GNN uses MHA-Interaction to calculate the interaction strength between nodes, so that the AGV can make the most appropriate decision based on its own state and the states of neighboring AGVs. The real-time updated graph structure ensures that the AGV can adapt to dynamic environmental changes in a timely manner, thus optimizing path decisions.

[0138] 4) Collect the complete observation information, aggregated information, state, action, and reward of the AGV as an experience trajectory and store it in a buffer as sampling data to update the neural networks (Actor network and Critic network) of the algorithm.

[0139] At each time step, the AGV aggregates its complete observation information (such as position, speed, target position, distance to obstacles, etc.) and local information from other AGVs into a graph structure. Then, the AGV selects an action based on the observation information and interacts with the environment to obtain a feedback reward. The observation information includes the state, action, reward of the AGV, and aggregated environmental data, which will form an experience trajectory and be stored in the buffer. The Actor network and Critic network in the neural network model will be trained based on these experience trajectories to optimize the path planning strategy. The Actor network optimizes path decisions through reinforcement learning, and the Critic network evaluates the value of the current decision to help the Actor network update the strategy.

[0140] 5) Repeat steps 1) to 4) until the number of simulation rounds for training is completed. After training, each AGV makes an optimal path decision using a distributed path planning algorithm based on local graph structure observations and real-time changes in the dynamic environment, and cooperates with other AGVs to ensure the efficient operation of the entire system. Each AGV makes path decisions based on local perception of graph structure observations (position, speed, target position, etc.) and dynamic environmental changes (such as obstacles and the states of other AGVs). The GNN aggregates local perception information and updates the nodes and edge weights in the graph structure. Then, the Actor network is used to perform path planning based on the updated graph information, and finally cooperate with other AGVs to execute tasks.

[0141] To verify the effectiveness and robustness of the GNMAPP algorithm proposed in the present invention, multiple groups of simulation experiments were carried out to test the impact of different configurations and parameter settings on the system performance. Figure 5It shows the performance of the GNMAPP algorithm under different configurations, including the cases of GNMAPP_no_Noisy, GNMAPP_no_RRT, and GNMAPP_no_GNN. The experimental results show that introducing noise, RRT, and GNN can significantly improve the path planning effect. In particular, the introduction of GNN greatly improves the accuracy and collaborative efficiency of path planning. After removing noise or GNN, although the system convergence speeds are different, the final performance differences are small. When removing RRT, the system shows a slight deficiency in path optimization, indicating that RRT plays an important role in path guidance and optimization.

[0142] In addition, Figure 6 It shows the influence of different perception range settings on the performance of the GNMAPP algorithm. The test results with edge radius ranges set to 0.5, 1, 2, and 5 show that although the smaller edge radius (0.5) has a faster initial learning speed, its final performance is inferior to that of models with larger edge radii (2 and 5). The data shows that the model with an edge radius of 2 achieves a good balance between performance and computational efficiency, can obtain sufficient environmental information, and at the same time avoids the computational overhead brought by too large a perception range. This indicates that the selection of the perception range plays an important balancing role between the final effect of path planning and the learning speed. A smaller perception range may result in a faster learning speed, while a larger perception range is helpful for more accurate path planning. Generally speaking, the GNMAPP algorithm shows good adaptability. Whether specific components are removed or parameter settings are adjusted, it can achieve efficient and accurate path planning in a dynamic environment, demonstrating its application potential in multi-AGV collaborative systems.

[0143] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multi-AGV distributed path planning method based on graph reinforcement learning, characterized in that The steps are as follows: Step 1: Establish a dynamic heterogeneous graph model in multi-AGV collaborative path planning; Step 2: Combine path planning and collaborative decision-making to construct a distributed partially observable Markov decision model based on the graph structure; Step 3: Based on the multi-agent deep reinforcement learning framework of graph neural network and multi-head attention mechanism, design a multi-AGV collaborative path planning algorithm for dealing with information-constrained dynamic environments. By adjusting the weights of nodes and edges in the dynamic heterogeneous graph, information aggregation is achieved based on local perception and neighborhood interaction, and the environment changes are updated and adapted in real time.

2. The multi-AGV distributed path planning method based on graph reinforcement learning according to claim 1, wherein, The architecture of the multi-AGV collaborative path planning includes M AGVs and an edge server. The edge server works with the AGVs through centralized training of the path planning model. The AGVs are connected to the edge server through wireless communication. The AGVs make collaborative decisions through local perception and neighborhood interaction to form a dynamic heterogeneous graph; At time step t, the dynamic heterogeneous graph is represented as G t =(X t , Adj t , W t ), where X t is the node feature matrix, which contains the position, velocity, target position, and entity type information of each entity at time step t; Adj t is the adjacency matrix of node connectivity, and the edge weight W t reflects the interaction intensity between entities. The edge weight W t is composed of the initial weight W0 and the dynamic weight increment ΔW t . The initial weight W0 constructs a basic interaction structure based on entity types; the dynamic weight increment ΔW t is calculated through the multi-head edge attention mechanism based on entity position offset, velocity change, and obstacle influence. During the task execution of the AGV, it dynamically interacts with environmental elements, and the dynamic heterogeneous graph adjusts the connection relationship and interaction intensity between nodes in real time, enabling the AGV to adapt to obstacle changes and speed adjustments in path planning in real time.

3. The multi-AGV distributed path planning method based on graph reinforcement learning according to claim 2, wherein The method for constructing the distributed partially observable Markov decision model is: In the architecture of multi-AGV collaborative path planning composed of M AGVs, each AGV integrates its own state, the states of other entities, and their interaction relationships within its sensing range based on a dynamic heterogeneous graph, and dynamically responds to environmental changes. The implementation method is as follows: At time step t, the m-th AGV m has an observation space of: where represents the velocity of the m-th AGV m at time step t, m ∈ {1, 2,..., M}, represents the current position of the m-th AGV m , represents the target position of the m-th AGV m , e m is the entity type of the m-th AGV m ID m is the unique identifier that distinguishes the m-th AGV m , represents the interaction relationships between the m-th AGV m and other entities within its sensing range and the interaction relationships with other AGVs; The overall observation space is represented as: where are the observation spaces of the 1st, 2nd,... Mth AGVs respectively; The observation space is constructed through a dynamic heterogeneous graph. The dynamic heterogeneous graph captures the mutual relationships between each AGV and other entities in the environment, and reflects the changes in the environment and system state in real time by dynamically updating the graph structure; At time step t, the m-th AGV m selects an action according to the current state where k ∈ {0, 1, 2, 3, 4}, and its values represent different movement directions respectively; the discrete action is converted into a continuous control force through the mapping function g That is, g: affects the acceleration a m of the m-th AGV m ; At time step t, the m-th AGV m has a composite reward function r m (t) given by: Among them, is the reward value for encouraging reaching the specified goal, is the reward value obtained for maintaining a safe distance, is the reward value for maintaining stable and coordinated movement, is the reward value for efficient movement behavior, is the reward value for following the RRT planned path.

4. The multi-AGV distributed path planning method based on graph reinforcement learning according to claim 3, wherein The acceleration where μ represents the environmental friction coefficient, m AGV is the mass of the m-th AGV m , is the velocity of the m-th AGV at time step t + 1 m , and Δt represents the update step size; The reward value for reaching the specified target is as follows: Among them, R goal is the reward for the AGV to reach the target position, and R time is the penalty for each step when the target is not reached; The reward value obtained by maintaining a safe distance is as follows: where R collsion is the penalty when the distance to an obstacle or another AGV is less than the safe distance; The reward value for maintaining stable and coordinated movement is as follows: where Δd is the progress of the current target direction, d actual is the total moving distance, and ω eff is the weight coefficient of the path efficiency; The reward value of the efficient motion behavior is as follows: where ω move is the weight coefficient for controlling the consistency of the motion direction, and cosθ is the cosine value of the angle θ between the current motion direction and the target direction; The reward value of the path following the RRT plan is expressed as: where is the position of the guiding path that is closest to the RRT path for the AGV.

5. The multi-AGV distributed path planning method based on graph reinforcement learning according to any one of claims 1-4, characterized in that, The multi-AGV collaborative path planning algorithm includes a neural network model, and the neural network model includes an Actor network deployed on the AGV and a Critic network deployed on the edge server; the Actor network consists of a first MLP layer, a Noisy layer, a first GRU module, and an output layer connected in sequence. The first MLP layer processes the local environment information from the graph neural network, encodes the obtained local features as the local environment information of the AGV, and transforms the local environment information through a learnable weight matrix W and a non-linear ReLU activation function to obtain local features; the Noisy layer noise introduces a noise parameter N(0,σ noise ), processes the transformed local features, and introduces Gaussian noise to achieve policy exploration. Among them, σ noise is the standard deviation parameter of the Gaussian noise introduced in the Noisy layer; the GRU module is used to update the historical state, regulates the information flow between the features with Gaussian noise introduced and the historical state through an update gate, and generates the final decision through the Softmax function; based on the obtained features and decisions, the output layer generates the path planning policy π(a t |o t ) of the AGV to achieve the distributed collaborative path planning decision of multiple AGVs; The Critic network consists of a second MLP layer, a second GRU module, and a value function layer connected in sequence; the second MLP obtains global features through global information aggregation of the graph neural network, and after processing by the weight matrix and the non-linear ReLU activation function, extracts global dynamic features; the second GRU module takes the global dynamic features as input, uses the control gate mechanism to combine historical state information, and captures and updates the dynamic evolution of the system; the value function layer combines the hidden state of the second GRU model with the global dynamic features, and calculates the global state value V(s t ,H global ) through the learned mapping function to provide value evaluation for policy optimization.

6. The multi-AGV distributed path planning method based on graph reinforcement learning according to claim 5, wherein During the training of the multi-AGV collaborative path planning algorithm, each AGV outputs an action according to the observation space and obtains a reward using a composite reward function. The Critic network judges the value of the output action according to the reward and updates the network. The implementation steps are: S331: Construct graph node features based on the local environment information perceived by each AGV to form a dynamic heterogeneous graph G t , and calculate the interaction strength between nodes through the multi-head attention mechanism to obtain the weighting coefficient, and dynamically update the weight of the edge; S332: The graph neural network uses the graph node features of each AGV to update its local features through weighted aggregation; the Transformer convolutional layer is used to update the graph node features of the AGV through weighted summation and aggregation of neighborhood node features; S333: After the local feature update, the graph neural network uses the global pooling operation in global information aggregation to integrate the local features of all AGVs to generate global features. The global features are input into the Critic network to evaluate the state value of the overall environment; S334: The updated local features are input into the Actor network to guide the path planning decision of each AGV.

7. The multi-AGV distributed path planning method based on graph reinforcement learning according to claim 6, wherein The update of the local features is achieved through the ReLU activation function and weighted summation. The formula is: Among them, h m represents the current feature vector of the AGV, W0 is a learnable weight matrix, and N(m) is the set of other AGV m node sets adjacent to the m-th AGV is the weighted coefficient calculated according to the interaction relationship between nodes, h j represents the feature vectors of other AGVs adjacent to the feature vector h m node sets adjacent to the m-th AGV 8. The multi-AGV distributed path planning method based on graph reinforcement learning according to claim 7, characterized in that, An Embedding layer is provided between the multi-head attention mechanism and the Transformer convolutional layer to convert each AGV and its corresponding entity type into an embedding vector of a fixed dimension. The embedding vector is concatenated with the original features of the node to generate a feature vector that combines all key information. If there are edge features, they are added to the concatenated feature vector. The graph neural network includes an initial Transformer convolutional layer for processing preliminary feature updates. According to the layer_N parameter, several Transformer convolutional layers are added, and the features are gradually processed and information is aggregated through multiple layers of Transformer convolution, so as to optimize the feature representation.

9. The multi-AGV distributed path planning method based on graph reinforcement learning according to claim 8, wherein Optimizing the path planning decision through information aggregation and reinforcement learning includes the following steps: 1) Initialize the parameters of the neural network model composed of the Actor network and the Critic network and the graph neural network; 2) Construct a dynamic heterogeneous graph according to the communication and perception ranges of multi-AGV distributed path planning. Each AGV interacts with the environment according to its perception range, and transfers the features of adjacent AGVs and the information of static and dynamic obstacles in the environment into the node features of the dynamic heterogeneous graph. For AGVs or obstacles outside the perception range, their features are filled with zeros; 3) The AGV obtains node and edge information according to the interaction data, and aggregates the state information of the environment and other AGVs through the graph neural network; 4) Collect the complete observation information, aggregated information, state, action, and reward of the AGV as an experience trajectory and store it in the buffer as sampling data to train the neural network model; 5) Repeat steps 1) to 4) until the number of simulation rounds for training is completed. After training is completed, each AGV makes an optimal path decision using distributed path planning based on the local graph structure observation and the real-time changes of the dynamic environment, and collaborates with other AGVs.

10. The multi-AGV distributed path planning method based on graph reinforcement learning according to claim 9, wherein The graph neural network aggregates the local information of different AGVs and the environment to obtain aggregated information, dynamically adjusts the edge weights between nodes according to the perception range of each AGV and its interaction relationship with other AGVs, and is dynamically updated according to the relative position and speed difference between the AGV and other entities. The steps are as follows: (1) Collect the state, action, and reward of each AGV into the interaction information, and all the interaction information is integrated and constructed into a training data set; (2) Use the graph neural network to construct a dynamic heterogeneous graph to aggregate and update the local information from other AGVs and the environment; (3) Use one-hot encoding to classify the feedback signal, and perform feature extraction and integration through the graph neural network; (4) Combining the observation data of other AGVs, the neural network model optimizes the path planning strategy through a centralized training and distributed execution architecture; (5) Make a path decision based on the real-time environmental changes and system state, and collaborate with other AGVs; The feedback signal is obtained by each AGV after interacting with the environment during path planning. The AGV compares the action it executes with the changes in the environment to obtain the feedback signal.

Images