Multi-AGV path planning method and system for intelligent factory

Abstract

The invention discloses a multi-AGV path planning method and system for an intelligent factory. The method comprises the following steps: modeling a static environment map by adopting a grid method, initializing an initial position of an intelligent agent and distributing a target; modeling multi-agent cluster scheduling into a partially observable Markov decision process; constructing a network structure of a strategy network and an evaluation network; training neural network parameters by using a method of combining deep reinforcement learning and imitation learning; and during execution, inputting part of observation information into the unmanned vehicle strategy network, outputting action probability distribution, and sampling the probability distribution to output a decision action. According to the system, a recurrent neural network and an attention mechanism are introduced into a neural network, imitation learning is introduced into a training strategy, a high cluster path planning success rate in a four-neighborhood two-dimensional grid map can be achieved after training, the convergence speed is high, strategy updating is stable, and the exploration ability is high; and the adaptability to the change of the map size and the barrier density is high.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent factory automation, and particularly relates to a multi-AGV path planning method for an intelligent factory. BACKGROUND

[0002] With the rapid development of intelligent factories, multi-AGV systems are widely used in material handling, warehouse management and other links to realize automation and flexible production. However, traditional multi-AGV path planning methods face many challenges.

[0003] Limitations of environment modeling: Early methods such as grid or graph models can model the environment statically, but they are difficult to handle dynamic obstacles and partially observable scenes. For example, centralized planning algorithms (such as A* or conflict search algorithm CBS) have a sharp increase in computational complexity when the number of agents increases, and cannot adapt to real-time changes in map size or obstacle density.

[0004] Deficiency of collaborative decision-making: Multi-agent path planning is often simplified as a fully observable Markov Decision Process (MDP), ignoring the fact that AGVs can only obtain local observation information (such as a limited field of view centered on themselves) in the actual environment. This leads to path conflicts, deadlocks or low efficiency, especially when the agent density is high, the collision risk increases significantly.

[0005] Defects of learning methods: Although existing deep reinforcement learning (DRL) methods can handle sequential decision-making, they have poor training stability, slow convergence, and are sensitive to sparse reward scenarios (such as agents needing to explore for a long time to reach the target). Pure DRL methods are prone to local optimization, while imitation learning can use expert trajectories to speed up training, but lacks the ability to generalize to unknown environments.

[0006] Real-time and adaptability bottleneck: Traditional methods rely on global information communication, which not only increases system overhead, but also makes it difficult to ensure real-time performance in distributed execution. In addition, fixed policy network structures (such as ordinary convolutional neural networks) cannot effectively capture spatiotemporal dependencies, resulting in weak adaptability to complex factory environments. SUMMARY

[0007] The technical problem to be solved by the present application is: In view of the problems involved in the above background technology, combining multi-agent systems, deep reinforcement learning and imitation learning techniques, the problem of collaborative path planning for multiple AGVs in dynamic environments in intelligent factories is solved, and a multi-AGV path planning method and system for intelligent factories are proposed to improve the success rate, adaptability and efficiency of path planning.

[0008] To solve the problems in the above prior art, the present application adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a multi-AGV path planning method for smart factories, specifically including the following steps:

[0010] S1. A static environment map is modeled using the grid method. The map is initialized, and the initial position of the unmanned vehicle and its target point position are determined.

[0011] S2. Model the multi-driver path planning problem as a partially observable Markov decision process.

[0012] S3. Introduce recurrent neural networks and attention mechanisms to construct the network structure of policy networks and evaluation networks.

[0013] S4. Train neural network parameters using a combination of deep reinforcement learning and imitation learning.

[0014] S5. When performing actual multi-vehicle path planning tasks, the current partial observation information is input into the policy network, the policy network outputs the action probability distribution, and then the probability distribution is sampled to output the decision action.

[0015] Furthermore, in the method proposed in this invention, the specific steps of step S1 are as follows:

[0016] S11. Model the smart factory as a two-dimensional grid map using the grid method. Each grid cell corresponds to one of the following: free space, obstacles, smart agent, and smart agent target point.

[0017] S12. Establish mapping rules for each grid cell of the two-dimensional mesh to correspond to different roles (such as free space, obstacles).

[0018] S13. Generate a complex dataset, in which the starting point and target point of each agent will be randomly generated, and different 2D grid map sizes, obstacle densities and the number of agents will be randomly combined.

[0019] Furthermore, in the method proposed in this invention, step S2 specifically includes:

[0020] The POMDP is used to describe the decision model for multi-AGV path planning. This model can be defined as a seven-tuple.<S,A,R, P, Z, O,γ> .

[0021] Action space: Because the experiment was conducted in a grid environment, the action space is discrete, consisting of five actions for each agent: up, down, left, right, and stay.

[0022] Reward Functions: Movement Reward: A negative reward is given when the agent performs any movement action (up, down, left, right) to incentivize finding the shortest path and reduce ineffective movements. Collision Penalty: A high negative reward is given when the agent collides with static obstacles or other agents in the environment to prioritize the safety of path planning. Goal Reach Reward: A positive reward is given when the agent first reaches its designated goal point to guide it in completing its individual task. Static Strategy Reward: The reward is 0 when the agent is already at the goal point and chooses to remain stationary, avoiding unnecessary intervention in its completed task state. A negative reward is given when the agent is not at the goal point but chooses to remain stationary to prevent it from stagnating at non-goal points and getting trapped in local optima. Global Completion Reward: A high positive reward is given when all agents reach their respective goal points and the entire task is completed to promote collaboration among agents and achieve global optimum.

[0023] Partial observation information includes: a matrix of location information for nearby obstacles; a matrix of location information for nearby agents; the current agent's location information and a matrix of target location information for nearby agents; the matrix of its own target location information within the field of view, and the direction vector of the target point. All five pieces of information are matrices of dimension (m, n).

[0024] State space: composed of partial observation information from all agents.

[0025] Furthermore, in the method proposed in this invention, step S3 specifically comprises:

[0026] S31. Policy Network Construction:

[0027] Input layer design: The input is a local observation sequence of a single AGV, where each observation contains 5×5 grid data of 5 channels. The sequence length k is adjusted according to historical dependency requirements. The input shape is (batch_size, k, 5, 5, 5).

[0028] Spatial Feature Extraction Module: Uses a three-layer convolutional neural network (CNN) to extract spatial features.

[0029] First convolutional layer: Conv2D input channels 5, output channels 32, convolution kernel 3×3, stride 1, padding 1, followed by ReLU activation function; output feature map shape is (batch_size, k, 32, 5, 5);

[0030] The second convolutional layer has 32 input channels and 64 output channels, with a 3×3 kernel, a stride of 1, padding of 1, followed by a ReLU activation function and 2×2 max pooling with a stride of 2; the output feature map shape is (batch_size, k, 64, 2, 2).

[0031] The third convolutional layer has 64 input channels and 128 output channels, with a 3×3 kernel, a stride of 1, padding of 1, and a ReLU activation function. The output feature map shape is (batch_size, k, 128, 2, 2).

[0032] The feature map is flattened into a vector and compressed to 256 dimensions through a fully connected layer, followed by a ReLU activation function; the output shape is (batch_size, k, 256).

[0033] Temporal dependency processing module: Inputs the 256-dimensional feature sequence into a single-layer gated recurrent unit (GRU) with 128 hidden units, and outputs the 128-dimensional hidden state at the final time step to capture the historical dependencies in the AGV's motion; the output shape is (batch_size, 128).

[0034] Attention weighting module: Uses a multi-head self-attention mechanism with 8 heads, each head having 16 dimensions, to weight the features output by the GRU and dynamically highlight key information; the output shape is (batch_size, 128).

[0035] Policy output layer: The 128-dimensional features are mapped to 64 dimensions through a fully connected layer (128→64, ReLU activation), and then output as a 5-dimensional action probability distribution through a fully connected layer (64→5). Finally, it is normalized by the Softmax function. The output shape is (batch_size, 5), corresponding to the probabilities of five actions: up, down, left, right, and stay.

[0036] S32. Evaluate network construction:

[0037] Dual-branch input design: Observation branch: The input is the same as the policy network, processed by the same CNN module, and outputs 128-dimensional features; Action branch: The input is the discrete actions of the AGV, which are mapped to 32-dimensional features through a fully connected layer (5→32), followed by a ReLU activation function; The output shape is (batch_size, k, 32);

[0038] Feature fusion module: The observation features and action features are concatenated in the feature dimension to obtain a 160-dimensional fused feature, which is then compressed to 128 dimensions through a fully connected layer (160→128), followed by a ReLU activation function; the output shape is (batch_size, k,128);

[0039] Temporal dependency processing module: Input the fused 128-dimensional feature sequence into a single-layer GRU with 64 hidden units, and output the hidden states and shapes (batch_size, k, 64) for all time steps to evaluate the long-term value dependency of state-action pairs;

[0040] Attention weighting module: Uses a multi-head self-attention mechanism with 4 heads, each head having 16 dimensions, to weight the sequence features output by the GRU, focusing on key time step information that affects the Q value; the output shape is (batch_size, k, 64).

[0041] Value assessment module: Take the final time step features of the attention-weighted sequence, map them to 32 dimensions through a fully connected layer (64→32, ReLU activation), and finally output the Q value through a fully connected layer (32→1); the output shape is (batch_size, 1).

[0042] Furthermore, in the method proposed in this invention, step S4 specifically comprises:

[0043] S41. Parameter Initialization and Global Settings: Initialize the parameters of the policy network and value network, set the training hyperparameters, including the maximum number of training steps Tmax and the discount factor γ; initialize the generation mechanism of the switching probability p between imitation learning and reinforcement learning.

[0044] S42, Training Loop Control: Start the training loop. When the global training steps T is less than Tmax, repeat the process from S43 to S45:

[0045] S43, Imitation Learning Branch: If the current random probability p < 0.5, then execute the imitation learning branch:

[0046] Expert trajectory generation: Invoke the expert path planning algorithm to generate the global optimal trajectory for the current environment state, and extract the agent's observations and expert suggestions for the optimal action at each time step;

[0047] Imitation loss calculation: Input the observations into the policy network to obtain the action probability distribution; calculate the imitation learning loss function, which aims to maximize the likelihood probability between the policy network's output action and the expert's action;

[0048] Network parameter update: The policy network parameters are updated by minimizing the imitation loss through the backpropagation algorithm, so that the agent's behavior can quickly approach the expert level.

[0049] S44, Reinforcement Learning Branch: If the current random probability p ≥ 0.5, then execute the reinforcement learning branch:

[0050] Experience collection: The environment is reset to the initial state; at each time step, each agent samples and executes actions according to the current policy, obtains rewards and the next state, and stores the experience quadruple into the experience replay pool;

[0051] Cumulative Rewards and Gradient Calculation: Data is sampled from the experience replay pool and inverse time series calculation is performed; for each sampled trajectory, the cumulative reward is calculated starting from the endpoint; subsequently, the policy gradient and value gradient are accumulated.

[0052] Centralized training and distributed execution: During the optimization process, the value network uses global information from all agents for centralized training to learn cooperative policies; while the policy network relies only on local observations to achieve distributed execution.

[0053] Network parameter synchronization update: Using the calculated policy gradient and value gradient, the network parameters of the global policy network Actor and value network Critic are updated synchronously.

[0054] S45. Training Termination and Model Saving: When the number of training steps T reaches Tmax or the policy performance converges, the training process is terminated, and the parameters of the finally trained policy network are saved for subsequent actual path planning tasks.

[0055] Furthermore, in the method proposed in this invention, step S5 specifically includes the following steps:

[0056] S51. Real-time Acquisition and Preprocessing of Observation Information: At the beginning of each planning cycle, each AGV acquires its local observation information in real time through sensors or a communication system. This includes the position information matrix of nearby obstacles, the position information matrix of other nearby AGVs, the position information matrix of the current AGV itself, the target position information matrix of other nearby AGVs, and the direction vector information matrix of its own target point. All observation information is an m×n grid matrix. Subsequently, the observation information is preprocessed, including data format standardization and serialization. The current observation is combined with the most recent k-1 historical observations to form an observation sequence of length k, resulting in an input tensor of shape (1, k, 5, 5, 5).

[0057] S52. Policy Network Forward Inference: The preprocessed observation sequence is input into the trained policy network. The policy network first extracts spatial features through the convolutional neural network (CNN) module, then processes temporal dependencies through the gated recurrent unit (GRU), and then weights key information through a multi-head self-attention mechanism, finally outputting a 5-dimensional action probability distribution, corresponding to the probability values ​​of five actions: up, down, left, right, and stay. The probability distribution is normalized by the Softmax function to ensure that the sum of the probabilities of all actions is 1.

[0058] S53. Action Selection and Decision Execution: Based on the action probability distribution output by the strategy network, a random sampling method is used to select the action of the current AGV. After the action is selected, the AGV executes the action and interacts with other AGVs and the environment.

[0059] S54. Status Update and Termination Condition Judgment: After executing the action, update the position status and environment status of all AGVs; check the termination condition: if all AGVs have reached their respective target points, the planning task is completed and the process is terminated; otherwise, return to S51 to continue the next planning cycle, repeat the observation, reasoning and execution process until the task is completed or the maximum number of steps is reached.

[0060] Secondly, the present invention provides a multi-AGV path planning system for smart factories, comprising:

[0061] Module: Used to construct a factory environment map using a grid method and to establish a mathematical model for AGV path planning with the goal of maximizing cumulative rewards;

[0062] The solver module is used to transform the mathematical model of AGV path planning into a partially observable Markov decision process and construct the corresponding state space, action space and reward function.

[0063] Training Module: Used to train the Actor network corresponding to the AGV based on deep reinforcement learning and imitation learning. At the beginning of each training round, based on the output value of a set random function, the module selects whether to perform imitation learning or deep reinforcement learning for that round. Deep Reinforcement Learning: Data generated from random interactions between the AGV and the environment is used to fill the experience replay pool, and data is randomly read from the experience replay pool using the MASAC algorithm for network training. Imitation Learning: Based on the trajectory planned by an expert algorithm (such as M*), state-action pair information is obtained for training the Actor network.

[0064] Planning module: Used to deploy the trained multi-agent model to the workshop and perform real-time path planning for multiple AGVs based on the workshop environment map.

[0065] Thirdly, the present invention provides a computer device comprising:

[0066] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-AGV path planning method for a smart factory described above.

[0067] Secondly, the present invention proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the aforementioned multi-AGV path planning method for smart factories.

[0068] Finally, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the multi-AGV path planning method for smart factories described in the present invention.

[0069] Compared with the prior art, the present invention, by adopting the above technical solution, has the following technical effects:

[0070] (1) In some observable environments (such as grid maps of smart factories), agents can only obtain limited field-of-view information, which makes them prone to getting stuck in local optima or repeating errors (such as dead ends). Furthermore, traditional neural networks have difficulty handling the temporal logic in path planning (such as the continuity of agent movement), leading to unstable strategies. This invention introduces recurrent neural networks (such as GRU) into the neural network to capture long-term dependencies in the time series, which makes up for the agent's deficiency of only having local observation information. This allows the agent to have a certain memory and inference ability for global map information through the GRU network.

[0071] (2) To address the problems that training a network solely through reinforcement learning requires a large amount of exploration data, results in sparse rewards, has low training efficiency, and is prone to collisions or path conflicts between agents in complex environments, this invention introduces imitation learning during network training to reduce the amount of training computation, accelerate network training, and improve the success rate of path planning.

[0072] (3) In view of the problem that traditional neural networks process all features equally, resulting in non-important information (such as distant obstacles) interfering with decision-making and key information (such as the risk of collision at close range) being diluted, this invention introduces an attention mechanism. By allowing the agent to autonomously focus on key information in the environment, it effectively solves the problems of information overload and inefficient collaboration in complex multi-agent tasks of the traditional MARL algorithm, and improves the success rate of path planning and average reward.

[0073] In summary, the multi-AGV path planning method for smart factories proposed in this invention has a fast convergence speed, stable strategy updates, strong exploration capabilities, a high path planning success rate, and strong environmental adaptability, making it suitable for scenarios such as warehousing logistics and production line material distribution. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of the overall implementation process of the present invention.

[0075] Figure 2 This is a schematic diagram of the structure of the actor network and the critic network in this invention.

[0076] Figure 3 This is a schematic diagram of the simulated map information of the present invention.

[0077] Figure 4 This is a schematic diagram illustrating the combination of reinforcement learning and simulation learning methods in step S4 of the present invention.

[0078] Figure 5This refers to the MASAC algorithm framework used in step S4 of the present invention.

[0079] Figure 6 This is a logic diagram of the execution path planning task of a certain AGV in step S5 of the present invention. Detailed Implementation

[0080] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0081] This invention significantly improves the success rate and robustness of multi-AGV path planning by modeling multi-AGV path planning as a partially observable Markov decision process (POMDP), introducing a neural network that integrates recurrent neural networks and attention mechanisms, and combining a hybrid training method of deep reinforcement learning and imitation learning.

[0082] Example 1: As Figure 1 and Figure 2 The diagram illustrates the method flow and specific algorithm network structure of this invention. The multi-AGV path planning method for smart factories proposed in this invention includes the following steps:

[0083] S1. Use the grid method to model a static environment map, initialize the map, and determine the initial position of the unmanned vehicle and its target point position:

[0084] S11. Model the smart factory as a two-dimensional grid map using the grid method. Each grid cell corresponds to one of the following: free space, obstacles, smart agent, and smart agent target point.

[0085] S12. Establish mapping rules for each grid cell of the two-dimensional mesh to correspond to different roles (such as free space, obstacles).

[0086] S13. Generate a complex dataset, in which the starting point and target point of each agent will be randomly generated, and different 2D grid map sizes, obstacle densities and the number of agents will be randomly combined.

[0087] The grid map is a square with sides of 10, 20, or 40; the obstacle density is the percentage of obstacle grids in the total map grid, selectable as 0%, 10%, 30%, or 50%; the number of agents can be selected as 4, 8, 16, or 32. The generated map iterates through these combinations. To ensure a valid path from the agent's starting position to the target point, a breadth-first search algorithm is used to determine all reachable grids or locations from the current agent position. By randomly selecting the target location within the reachable area of ​​the agent's starting position, a feasible path between the target point and the agent's starting position is guaranteed. It is crucial to ensure that the target point does not overlap with obstacles in the environment; that is, the target location must be located in an open area, avoiding selection of the target point within obstacles or impassable areas, ensuring the agent can successfully reach the target. Figure 3 This is a schematic diagram of the simulated map information of the present invention.

[0088] S2. Model the multi-driver path planning problem as a partially observable Markov decision process:

[0089] The POMDP is used to describe the decision model for multi-AGV path planning. This model can be defined as a seven-tuple.<S,A,R, P, Z, O,γ> .

[0090] Action space: Because the experiment was conducted in a grid environment, the action selection was limited and explicit. Therefore, the action space was discrete, consisting of five actions for each agent: up, down, left, right, and stay.

[0091] Reward Functions: Movement Reward: When an agent performs any movement action (up, down, left, right), a negative reward of -0.3 is given to incentivize it to find the shortest path and reduce ineffective movements. Collision Penalty: When an agent collides with a static obstacle or other agent in the environment, a high negative reward of -2.0 is given to prioritize the safety of path planning. Goal Reach Reward: When an agent reaches its designated goal point for the first time, a positive reward of +1 is given to guide it in completing its individual task. Static Strategy Reward: When an agent is already at the goal point and chooses to remain stationary, the reward is 0 to avoid unnecessary intervention in its completed task state. When an agent is not at the goal point but chooses to remain stationary, a negative reward of -0.5 is given to prevent it from stagnating at a non-goal point and getting trapped in a local optimum. Global Completion Reward: When all agents have reached their respective goal points and the entire task is completed, a high positive reward of +20 is given to promote cooperation among agents and jointly achieve global optimum.

[0092] Partial observation information includes: a matrix of location information for nearby obstacles; a matrix of location information for nearby agents; the current agent's location information and a matrix of target location information for nearby agents; the matrix of its own target location information within the field of view, and the direction vector of the target point. All five pieces of information are matrices of dimension (5,5).

[0093] State space: composed of partial observation information from all agents.

[0094] S3. Introduce recurrent neural networks and attention mechanisms to construct the network structure of the policy network and evaluation network:

[0095] S31. Policy Network (Actor) Construction:

[0096] Input layer design: The input is a local observation sequence of a single AGV, where each observation contains 5×5 grid data of 5 channels (corresponding to obstacles, the position of nearby AGVs, the current position of the AGV, the target point of nearby AGVs, and the direction vector of its own target point). The sequence length k is set to 4~8 (adjusted according to historical dependency requirements), and the input shape is (batch_size, k, 5,5, 5).

[0097] Spatial Feature Extraction Module: A three-layer Convolutional Neural Network (CNN) is used to extract spatial features: First Convolutional Layer: Conv2D input channels 5, output channels 32, kernel size 3×3, stride 1, padding 1, followed by ReLU activation; output feature map shape is (batch_size, k, 32, 5, 5). Second Convolutional Layer: Conv2D input channels 32, output channels 64, kernel size 3×3, stride 1, padding 1, followed by ReLU activation and 2×2 max pooling (MaxPool2d), stride 2; output feature map shape is (batch_size, k, 64, 2, 2). Third Convolutional Layer: Conv2D input channels 64, output channels 128, kernel size 3×3, stride 1, padding 1, followed by ReLU activation; output feature map shape is (batch_size, k, 128, 2, 2). The feature map is flattened into a vector and compressed to 256 dimensions through a fully connected layer, followed by a ReLU activation function; the output shape is (batch_size, k, 256).

[0098] Temporal dependency processing module: Input the 256-dimensional feature sequence into a single-layer gated recurrent unit (GRU) with 128 hidden units, and output the 128-dimensional hidden state at the final time step to capture historical dependencies in AGV motion (such as avoiding repeated paths or oscillating behavior); the output shape is (batch_size, 128).

[0099] Attention weighting module: Uses multi-head self-attention mechanism with 8 heads, each head with 16 dimensions, to weight the features output by GRU, dynamically highlighting key information (such as dynamic obstacles or target direction) and improving the targeting of the strategy; the output shape is (batch_size, 128).

[0100] Policy output layer: The 128-dimensional features are mapped to 64 dimensions through a fully connected layer (128→64, ReLU activation), and then output as a 5-dimensional action probability distribution through a fully connected layer (64→5). Finally, it is normalized by the Softmax function. The output shape is (batch_size, 5), corresponding to the probabilities of five actions: up, down, left, right, and stay.

[0101] S32. Construction of the Critic Network:

[0102] Dual-branch input design: Observation branch: The input is the same as the policy network (local observation sequence, shape (batch_size, k, 5, 5, 5)), processed by the same CNN module (structure same as S312, but with independent weights), outputting 128-dimensional features. Action branch: The input is the discrete actions of the AGV (5-dimensional one-hot vectors), mapped to 32-dimensional features through a fully connected layer (5→32), followed by a ReLU activation function; the output shape is (batch_size, k, 32).

[0103] Feature fusion module: The observation features (128-dimensional) and action features (32-dimensional) are concatenated in the feature dimension to obtain 160-dimensional fused features, which are then compressed to 128 dimensions through a fully connected layer (160→128), followed by a ReLU activation function; the output shape is (batch_size, k, 128).

[0104] Temporal dependency processing module: Input the fused 128-dimensional feature sequence into a single-layer GRU with 64 hidden units, and output the hidden states (shape (batch_size, k, 64)) for all time steps to evaluate the long-term value dependency of state-action pairs.

[0105] Attention weighting module: Uses a multi-head self-attention mechanism (4 heads, 16 dimensions per head) to weight the sequence features output by GRU, focusing on key time step information that affects the Q value (such as conflict warning or target approach time); the output shape is (batch_size, k, 64).

[0106] Value assessment module: Take the final time step features (64-dimensional) of the attention-weighted sequence, map them to 32-dimensional through a fully connected layer (64→32, ReLU activation), and finally output the Q value through a fully connected layer (32→1); the output shape is (batch_size,1).

[0107] S4. Train neural network parameters using a combination of deep reinforcement learning and imitation learning. Figure 4 (For the corresponding logic diagram)

[0108] S41. Parameter Initialization and Global Settings: Initialize the parameters of the policy network (Actor) and the value network (Critic). Set the training hyperparameters, including the maximum number of training steps Tmax and the discount factor γ. Initialize the generation mechanism of the switching probability p between imitation learning and reinforcement learning (e.g., p is a random number uniformly distributed in the interval (0,1)).

[0109] S42, Training Loop Control: Start the training loop. When the global training steps T is less than Tmax, repeat the process from S43 to S45.

[0110] S43, Imitation Learning Branch (Guided by Expert Demonstration): If the current random probability p < 0.5, then execute the imitation learning branch:

[0111] Expert trajectory generation: Invoke an expert path planning algorithm (such as the M* algorithm) to generate a globally optimal trajectory for the current environment state, and extract the agent's observations and expert suggestions for optimal actions at each time step.

[0112] Imitation loss calculation: Input the observations into the policy network to obtain the action probability distribution. Calculate the imitation learning loss function, which aims to maximize the likelihood probability between the policy network's output action and the expert's action.

[0113] Network parameter update: The policy network parameters are updated by minimizing the imitation loss through the backpropagation algorithm, so that the agent's behavior can quickly approach the expert level.

[0114] S44, Reinforcement Learning Branch (Co-optimization based on MASAC): If the current random probability p ≥ 0.5, then execute the reinforcement learning branch:

[0115] Experience collection: The environment is reset to its initial state. At each time step, each agent samples and executes actions according to the current policy, obtains a reward and the next state, and stores the experience quadruple into the experience replay pool.

[0116] Cumulative Rewards and Gradient Calculation: Data is sampled from the experience replay pool and inverse time-series calculations are performed. For each sampled trajectory, the cumulative reward is calculated starting from the endpoint; subsequently, the policy gradient (including an entropy regularization term to encourage exploration) and the value gradient are accumulated.

[0117] Centralized training and distributed execution: During the optimization process, the evaluation network (Critic) is trained centrally using global information from all agents to learn cooperative policies; while the policy network (Actor) relies only on local observations to achieve distributed execution.

[0118] Network parameter synchronization update: Using the calculated policy gradient and value gradient, the global Actor and Critic network parameters are updated synchronously.

[0119] S45. Training Termination and Model Saving: When the number of training steps T reaches Tmax or the policy performance converges (e.g., the average reward fluctuation is less than the threshold), the training process is terminated, and the parameters of the finally trained policy network are saved for subsequent actual path planning tasks.

[0120] S5. When performing actual multi-vehicle path planning tasks, the current partial observation information is input into the policy network, the policy network outputs the action probability distribution, and then the probability distribution is sampled to output the decision action:

[0121] S51. Real-time Acquisition and Preprocessing of Observation Information: At the beginning of each planning cycle, each AGV acquires its local observation information in real time through sensors or a communication system. This includes the position information matrix of nearby obstacles, the position information matrix of other nearby AGVs, the position information matrix of the current AGV itself, the target position information matrix of other nearby AGVs, and the direction vector information matrix of its own target point. All observation information is a 5×5 grid matrix. Subsequently, the observation information is preprocessed, including data format standardization and serialization. The current observation is combined with the most recent k-1 historical observations to form an observation sequence of length k (k is the sequence length set during training, usually 4~8), forming an input tensor of shape (1, k, 5, 5, 5) to meet the input requirements of the policy network.

[0122] S52. Policy Network Forward Inference: The preprocessed observation sequence is input into the trained policy network (Actor network). The policy network first extracts spatial features through a convolutional neural network (CNN) module, then processes temporal dependencies through a gated recurrent unit (GRU), and finally weights key information through a multi-head self-attention mechanism, ultimately outputting a 5-dimensional action probability distribution, corresponding to the probability values ​​of five actions: up, down, left, right, and stay. The probability distribution is normalized using the Softmax function to ensure that the sum of the probabilities of all actions is 1.

[0123] S53. Action Selection and Decision Execution: Based on the action probability distribution output by the policy network, a random sampling method (such as multinomial sampling) is used to select the current AGV's action. The sampling process introduces a degree of randomness to enhance the policy's robustness and avoid getting trapped in local optima. After selecting an action, the AGV executes that action (such as moving or stopping) and interacts with other AGVs and the environment.

[0124] S54. Status Update and Termination Condition Judgment: After executing the action, update the position status and environment status of all AGVs. Check the termination condition: If all AGVs have reached their respective target points, the planning task is completed and the process terminates; otherwise, return to S51 to continue the next planning cycle, repeating the observation, reasoning, and execution process until the task is completed or the maximum number of steps is reached.

[0125] Example 2: This example provides a multi-AGV path planning system for smart factories, including:

[0126] Module: Used to construct a factory environment map using a grid method and to establish a mathematical model for AGV path planning with the goal of maximizing cumulative rewards;

[0127] The solver module is used to transform the mathematical model of AGV path planning into a partially observable Markov decision process and construct the corresponding state space, action space and reward function.

[0128] Training Module: Used to train the Actor network corresponding to the AGV based on deep reinforcement learning and imitation learning. At the beginning of each training round, based on the output value of a set random function, the module selects whether to perform imitation learning or deep reinforcement learning for that round. Deep Reinforcement Learning: Data generated from the random interactions between the AGV and its environment is used to fill the experience replay pool. Data is then randomly read from the experience replay pool using the MASAC algorithm for network training. (See reference...) Figure 5 As shown; Imitation learning: Based on the trajectory planned by expert algorithms (such as M*), state-action pair information is obtained to train the Actor network.

[0129] Planning module: Used to deploy the trained multi-agent model to the workshop and perform real-time path planning for multiple AGVs based on the workshop environment map. See details... Figure 6 As shown.

[0130] Example 3: This example provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that when the processor executes the computer program, it corresponds to the specific steps of the method provided in this embodiment of the invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this example can be found in the method provided in this embodiment of the invention.

[0131] Example 4: This example proposes a computer-readable storage medium storing a computer program. It should be noted that when the computer program is run by a processor, it corresponds to the specific steps of the method provided in this embodiment of the invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this example can be found in the method provided in this embodiment of the invention.

[0132] Example 5: This example proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implements the steps of the multi-AGV path planning method for smart factories described in this invention.

[0133] It should be noted that the processing flow of embodiments 2-5 corresponds to the specific steps of the method provided in embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the method provided in embodiment 1 of the present invention.

[0134] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0136] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent factory-oriented multi-AGV path planning, characterized in that, The application relates to a method for multi-AGV path planning in smart factories. Step S1: a two-dimensional grid model of a factory environment is established by using a grid method, a map is initialized, and starting positions and target positions of AGVs are determined; Step S2: a multi-AGV path planning problem is modeled as a partially observable Markov decision process (POMDP), a state space, an action space, a reward function and partial observation information are defined; Step S3: a policy network and an evaluation network are constructed, wherein the policy network comprises a convolutional neural network for extracting spatial features, a gated recurrent unit (GRU) for capturing time-dependent information, and a multi-head self-attention mechanism for weighting key information; Step S4: a hybrid training method combining deep reinforcement learning and imitation learning is used to train the policy network and the evaluation network, and during the training process, imitation learning branches or reinforcement learning branches are selected to be executed according to a random probability; Deep reinforcement learning is based on data generated by random interaction between AGVs and the environment to fill an experience replay pool, and the MASAC algorithm is used to randomly read data from the experience replay pool for network training; Imitation learning is based on trajectories planned by an expert algorithm to obtain state-action pair information for training of an actor network; Step S5: in an actual path planning task, current local observation information of the AGV is input into the trained policy network, an action probability distribution is output, and an action is selected and executed through sampling. 2.The smart factory-oriented multi-AGV path planning method according to claim 1, wherein: The specific steps of step S1 are as follows: S11: a smart factory is modeled as a two-dimensional grid map by using a grid method, each grid unit corresponds to one of a free space, an obstacle, an intelligent agent AGV or a target point; S12: a mapping rule of each grid unit of the two-dimensional grid corresponding to different roles is established; S13: starting points and target points of each intelligent agent, different two-dimensional grid map sizes, obstacle densities and intelligent agent quantities are randomly combined to form a training data set. 3.The smart factory oriented multi-AGV path planning method of claim 1, wherein: The specific content of step S2 is as follows: The action space comprises five discrete actions of up, down, left, right and stop; The reward function comprises a moving negative reward, a collision penalty, a target arrival positive reward, a static strategy reward and a global completion positive reward; The partial observation information comprises at least one of obstacle positions, nearby AGV positions, a self position, nearby AGV target positions and a self target direction vector, and is represented in the form of a matrix with dimensions (m, n); The state space is formed by splicing the partial observation information of all AGVs.

4. The intelligent factory-oriented multi-AGV path planning method according to claim 1, characterized in that: The specific steps of step S3 are as follows: The input of the policy network is a local observation sequence of the AGV, spatial features are extracted by a convolutional neural network, time sequence information is processed by a GRU, and feature weighting is performed by a multi-head self-attention mechanism, and finally an action probability distribution is output; the evaluation network has a double-branch structure, and observation information and action information are processed respectively, and Q values are output after feature fusion, GRU time sequence processing and attention weighting.

5. The intelligent factory-oriented multi-AGV path planning method according to claim 1, characterized in that: The specific steps of step S4 are as follows: The imitation learning branch uses expert trajectory data to update network parameters by maximizing the likelihood probability of the output of the policy network and the expert action; The reinforcement learning branch is based on the multi-agent flexible actor-critic MASAC algorithm, which collects environmental interaction experience, calculates cumulative returns, and updates policy network and evaluation network parameters; During the training process, the two learning modes are switched at a random probability. 6.The smart factory oriented multi-AGV path planning method according to claim 1, wherein: The specific steps of step S5 are as follows: Real-time acquisition of AGV local observation information and sequence preprocessing; The preprocessed observation sequence is input into the policy network, and the action probability distribution is obtained through forward calculation; According to the action probability distribution, the action is sampled and selected, and then executed; Update the environment state, and judge whether all AGVs have reached the target point. If yes, end the planning; otherwise, continue to execute.

7. A smart factory-oriented multi-AGV path planning system, characterized in that, It comprises: The construction module is used to construct the factory environment map by using the grid method, and to establish an AGV path planning mathematical model with the goal of maximizing cumulative rewards; The solving module is used to convert the AGV path planning mathematical model into a partially observable Markov decision process, and to construct the corresponding state space, action space and reward function; The training module is used to train the policy network and the evaluation network based on a hybrid training method combining deep reinforcement learning and imitation learning. In each training round, the output value of the set random function is selected to perform imitation learning or deep reinforcement learning training. Deep reinforcement learning: based on the data generated by the random interaction of AGV and environment, the experience replay pool is filled, and the network is trained based on the MASAC algorithm to randomly read data from the experience replay pool; Imitation learning: based on the trajectory planned by the expert algorithm, the state-action pair information is obtained, and the Actor network is trained; The planning module is used to deploy the trained multi-agent model to the job shop, and to perform real-time path planning for multiple AGVs according to the workshop environment map.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to realize the steps of the method of any one of claims 1 to 6.

Images