Multi-AGV path planning algorithm based on dynamic exploration and course learning

Abstract

The invention provides a multi-AGV path planning algorithm based on dynamic adaptive exploration and course learning, and relates to the technical field of automation and intelligent logistics. According to the method, although a traditional deep reinforcement learning method is preliminarily applied to multi-AGV system path planning, the limitations of low efficiency, poor dynamic adaptability, insufficient cooperative competition relation processing and the like still exist, and the specific expressions are low exploration efficiency, insufficient sample utilization, slow convergence speed and even non-convergence. For this purpose, a multi-agent depth deterministic strategy gradient algorithm (AECL-MADDPG) based on adaptive exploration and course learning is designed, and centralized training is adopted. A distributed execution framework is adopted, the obstacle avoidance capability and the implicit cooperation efficiency of the AGV in a high-density environment are enhanced by sensing the environment congestion degree and decision uncertainty in real time to dynamically adjust the exploration strength, meanwhile, a course learning-based priority experience playback mechanism is constructed, a training normal form from easy to difficult is combined with key experience priority sampling, and the accuracy and the robustness of the AGV are improved. Model convergence is remarkably accelerated; and the robustness of a final strategy is improved. The algorithm established and designed under the actual operation condition in the automation and intelligent logistics field shows significant advantages in key indexes such as convergence speed, task success rate, average path length and the like, and an efficient and reliable solution is provided for the multi-agent path planning problem in a complex dynamic environment.

Description

Technical Field

[0001] This invention provides a dynamic adaptive exploration and learning multi-AGV path planning algorithm, which relates to the fields of automation and intelligent logistics technology. Background Technology

[0002] In recent years, the explosive growth of e-commerce and the acceleration of global trade integration have propelled the logistics industry into a "three-high" development stage characterized by "high order volume, high turnover efficiency, and high service quality." According to data from the China Federation of Logistics and Purchasing, China's total social logistics volume exceeded 350 trillion yuan in 2023, a year-on-year increase of 6.2%. Warehousing, as the core hub of the logistics chain, directly impacts the overall response speed of the supply chain. Traditional warehousing models rely on manual sorting and handling, which suffers from high labor intensity and high human error rates. Against this backdrop, intelligent warehousing systems, centered on automated storage and retrieval systems (AS / RS), AGVs (Automated Guided Vehicles), and intelligent sorting systems, have become a key technological direction for overcoming the bottlenecks of traditional models. As the "mobile carrier" of intelligent warehousing systems, AGVs have been widely used in e-commerce warehousing, automobile manufacturing, and pharmaceutical distribution due to their advantages such as high flexibility, high positioning accuracy (millimeter level), and ability to operate continuously for 24 hours. However, with the expansion of warehousing scale and the increase in the number of AGVs used, the number of AGVs deployed in a single warehouse has gradually increased from 10-20 units in the early days to more than 100 units. The path planning problem in multi-AGV collaborative operation scenarios has become the core bottleneck restricting the further improvement of system efficiency.

[0003] Traditional path planning algorithms, such as graph search-based A* and D* algorithms, and artificial potential field methods based on virtual force fields, have become mature solutions for path planning in single-agent or static environments. However, when faced with large-scale, dynamic warehousing environments and the real-time collaboration requirements of multiple AGVs, these methods reveal significant limitations: the A* coupled algorithm, when solving the MAPF problem, suffers from an exponentially increasing state space with the number of agents, easily encountering the "curse of dimensionality" and struggling to find the optimal solution within a finite time; while the artificial potential field method has low computational complexity, it is prone to getting trapped in local optima, causing AGVs to stagnate in areas with complex obstacles. Therefore, this invention proposes an improved multi-agent deep deterministic policy gradient algorithm (AECL-MADDPG) based on adaptive exploration and learning, aiming to improve the path planning performance of multi-AGV systems in large-scale intelligent warehousing environments by optimizing the exploration mechanism and learning paradigm. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing multi-AGV path planning methods by proposing an adaptive exploration and learning multi-AGV path planning algorithm.

[0005] An adaptive exploration and learning-based multi-AGV path planning algorithm is proposed. By dynamically adjusting the exploration intensity in real time based on environmental congestion and decision uncertainty, the algorithm enhances the obstacle avoidance capability and implicit collaboration efficiency of AGVs in high-density environments. At the same time, a priority experience playback mechanism based on learning-based learning is constructed, which combines the "from easy to difficult" training paradigm with priority sampling of key experiences, significantly accelerating model convergence and improving the robustness of the final strategy. Finally, simulation experiments are carried out in a simulated warehouse environment to compare and verify the proposed algorithm with baseline algorithms such as MADDPG and DDPG.

[0006] Within the MADDPG framework, during the training phase, each agent can utilize global information (observations and actions) from all agents to guide its learning; during the execution phase, each agent makes decisions based solely on its own local observations. This mechanism significantly improves algorithm stability and multi-agent collaborative capabilities. MADDPG maintains a deterministic policy network (Actor) and an action-value network (Critic) for each agent. The network parameter update principle is inherited from DDPG, as follows: The Critic network updates by minimizing the temporal difference (TD) error, and the loss function is defined as the mean squared Bellman error (MSBE): in, Represents the joint state of all agents. For experience replay pool; target value The formula for calculation is: In the formula, and The target networks are the Critic and the Actor, respectively. This is the discount factor. The Critic network input is global information, enabling a comprehensive understanding of the environmental state.

[0007] The Actor network updates its policy gradient, with the optimization objective being to maximize the Critic network's evaluation value for its output action. The policy gradient... The calculation formula is: Update the strategy network parameters along the direction of increasing Q value. Actor networks can learn to select actions that bring higher long-term returns.

[0008] An adaptive exploration and learning multi-AGV path planning algorithm, specifically including the following steps: S1: Initialization steps: Load parameters such as the total number of training rounds and the maximum number of AGVs; build a warehouse environment model to complete AGV deployment, collision detection, and reward / penalty feedback; initialize multiple AGV agents, each of which includes an Actor network for action decision-making and a Critic network for value evaluation; build an experience replay pool to store the experience data generated by the AGVs during the interaction process. S2: Training main loop: After initialization, the training round loop is entered. Each round corresponds to a complete multi-AGV path planning training process. S3: Environment Interaction Steps: At the beginning of each training round, the environment is reset according to the current course difficulty level and a corresponding number of AGVs and their target positions are generated; then, a time step loop is entered. In each time step, the AGV obtains the environment state information and selects an action based on the adaptive exploration strategy of dynamic congestion awareness. The environment updates its state according to the action and returns the next state, reward information and event type until the task is completed or the preset timeout condition is reached. S4: Experience storage step: Encapsulate the experience samples formed by the AGV during the interaction with the environment, consisting of the state, action, reward and the next state, and add course difficulty level label and event type label to each experience sample before storing it in the experience playback pool. S5: Learning and updating steps: When the number of samples in the experience replay pool reaches the preset threshold, training samples are sampled from the experience replay pool according to the comprehensive priority, and the parameters of the Actor network and Critic network are updated using the sampled samples through the backpropagation algorithm. S6: Course learning steps; After each training round, the average task success rate of the most recent preset number of rounds is calculated. When the average task success rate exceeds a preset threshold, the course difficulty level is increased, and the number of AGVs or the task complexity is increased in the next training round.

[0009] Preferably, in the strategy of dynamically adjusting the exploration intensity by real-time sensing of environmental congestion and decision uncertainty, the AGV ultimately executes the action. It consists of the basic action output by the policy network and an adaptive noise term, rather than being determined by fixed noise. The specific expression is as follows: in, For deterministic actions output by the Actor network, Standard Gaussian noise, This is a dynamic exploration factor, the value of which is obtained by multiplying three parts, comprehensively reflecting the degree of exploration demand: in, The baseline exploration rate gradually decreases as training progresses; For the AGV at its current or target position Congestion level at the location; For AGV in status Decision-making uncertainty; and These are the congestion-aware weighting coefficient and the uncertainty-aware weighting coefficient, respectively. Setting a base exploration rate provides a baseline exploration intensity that gradually decays as training progresses, ensuring high exploration levels in the early stages to explore the environment, and gradually favoring the use of existing strategies in later stages. In the experiment, the initial value was set to 0.7, with a decay coefficient of 0.9999 per step, decreasing to a minimum of 0.05. This parameter setting references common strategies for exploration rate decay in multi-agent reinforcement learning.

[0010] The congestion Defined as the number of other AGVs within a neighborhood of the current or target position of the AGV with a radius of R, its calculation formula is: For indicator functions, when The value is 1 if the condition is met, and 0 otherwise. The congestion perception weight coefficient (valued at 0.6~0.8 in the experiment) increases with the degree of congestion, thus increasing the exploration factor and encouraging the AGV to find alternative paths. The uncertainty The difference in value assessment of the same state-action pair is measured using two independent Critic networks, and is defined as follows: The uncertainty perception weight coefficient (valued at 0.4~0.6 in the experiment) balances the influence of uncertainty perception and congestion perception, avoiding a single factor dominating the exploration strategy.

[0011] Preferably, for building a priority experience playback mechanism based on course learning, the training process is divided into multiple course levels with increasing difficulty. Each course level corresponds to a different number of AGVs; when the average task success rate of the agent at the current course level exceeds a preset threshold, the course level is automatically upgraded. Within the course learning framework, the traditional priority experience replay is enhanced by calculating a comprehensive priority for each sample in the experience pool, consisting of three weighted parts: for each sample in the experience pool... Calculate the overall priority It consists of three weighted parts: in, The base priority is calculated based on the timing difference error; ε is a constant to prevent the priority from being zero; α is a priority adjustment factor. Assign weights to courses; Weights for event types, The calculation formula is: To avoid tiny constants with a priority of 0, The priority adjustment factor balances the wealth gap in priority levels and avoids oversampling of some high-priority samples. The course weights are used to guide the algorithm to prioritize sampling experiences related to the difficulty of the current training course, ensuring the "gradualness" of the learning process. They are defined as the sample difficulty level. Compared to the current training difficulty level Difference Gaussian functions: The standard deviation of the Gaussian function is 0.5, which is used in the experiment. This value is applied when the sample difficulty level matches the current training level. Take the maximum value of 1; when the difficulty difference is 1 level, Approximately 0.606; when the difference is ≥ level 2, ≤0.135 significantly reduces the sampling probability of low-relevance samples; event weights assign higher sampling priority to special events that are crucial to policy optimization, forcing the model to focus on key experiences; defined as a piecewise function: Among them, collision events have the highest weight (2.0) because collision samples contain key information about AGV obstacle avoidance failures and need to be studied in order to reduce subsequent conflicts; task success events have the second highest weight (1.8) to strengthen optimal path decision-making; regular driving events have a weight of 1.0 to ensure the normal learning of basic experience.

[0012] The beneficial effects of this invention are: (1) The improved MADDPG algorithm (AECL-MADDPG) based on adaptive exploration and curriculum learning described in this invention has a good solution to the three core problems of "inefficient exploration", "slow convergence" and "insufficient collaboration" in the multi-AGV path planning of large-scale intelligent warehouses; (2) The present invention proposes to dynamically adjust the exploration intensity by sensing the environmental congestion and decision uncertainty in real time, so that the AGV can actively detour in high-density areas, and the average number of collisions is reduced by 88.6% compared with the standard MADDPG, thus solving the problem of blindness in traditional fixed exploration; (3) The course learning mechanism proposed in this invention makes the algorithm convergence speed 50% faster than the standard MADDPG and improves the sample utilization rate by about 40%, solving the stagnation problem of training "from scratch" in complex tasks; (4) The combination of AE and CL-PER designed in this invention enables the algorithm to achieve a task success rate of 99.1% in a high-density scenario with 15 AGVs, and the average path length is shortened by 11.6%. Its overall performance is better than baseline algorithms such as DDPG and MADDPG, providing an efficient solution for multi-AGV collaborative path planning. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the framework of an adaptive exploration and learning multi-AGV path planning algorithm according to the present invention. Figure 2 A flowchart illustrating the progression logic of course learning proposed in this invention; Figure 3 The average reward change curve of the algorithm compared in this invention over 5000 training rounds. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0016] See attached document Figure 1 —3. Detailed description of the embodiments of the present invention: An adaptive exploration and curriculum-based multi-AGV path planning algorithm. First, by dynamically adjusting the exploration intensity in real time based on environmental congestion and decision uncertainty, the algorithm enhances the obstacle avoidance capability and implicit collaborative efficiency of AGVs in high-density environments. Simultaneously, it constructs a priority experience replay mechanism based on curriculum learning, combining an "easy-to-difficult" training paradigm with priority sampling of key experiences, significantly accelerating model convergence and improving the robustness of the final strategy. Finally, simulation experiments are conducted in a simulated warehouse environment to compare and verify the proposed algorithm with baseline algorithms such as MADDPG and DDPG.

[0017] The MADDPG framework is used to maintain a deterministic policy network (Actor) and an action value network (Critic) for each agent. The Critic network is updated by minimizing the temporal difference (TD) error, and the loss function is defined as the mean squared Bellman error (MSBE). in, Represents the joint state of all agents. For experience replay pool; target value The formula for calculation is: In the formula, and The target networks are the Critic and the Actor, respectively. This is the discount factor. The Critic network input is global information, enabling a comprehensive understanding of the environmental state.

[0018] The Actor network updates its policy gradient, with the optimization objective being to maximize the Critic network's evaluation value for its output action. The policy gradient... The calculation formula is: Update the strategy network parameters along the direction of increasing Q value. Actor networks can learn to select actions that bring higher long-term returns.

[0019] Regarding the strategy of dynamically adjusting the exploration intensity by real-time sensing of environmental congestion and decision uncertainty, the AGV ultimately executes the action. It consists of the basic action output by the policy network and an adaptive noise term, rather than being determined by fixed noise. The specific expression is as follows: in, For deterministic actions output by the Actor network, Standard Gaussian noise, This is a dynamic exploration factor, the value of which is obtained by multiplying three parts, comprehensively reflecting the degree of exploration demand: Setting a base exploration rate provides a baseline exploration intensity that gradually decays as training progresses, ensuring high exploration levels in the early stages to explore the environment, and gradually favoring the use of existing strategies in later stages. In the experiment, the initial value was set to 0.7, with a decay coefficient of 0.9999 per step, decreasing to a minimum of 0.05. This parameter setting references common strategies for exploration rate decay in multi-agent reinforcement learning.

[0020] The congestion sensing term is configured to dynamically adjust the exploration intensity based on environmental density. Among these, congestion level... Quantitative AGV Current or target location The level of activity in the surrounding area is defined as the radius. The number of other AGVs within the range is calculated using the following formula: For indicator functions, when The value is 1 if the condition is met, and 0 otherwise. The congestion perception weight coefficient (valued at 0.6~0.8 in the experiment) increases with the degree of congestion, thereby increasing the exploration factor and encouraging the AGV to find alternative paths.

[0021] An uncertainty perception term is set to measure the agent's state. The "degree of certainty" in making decisions. Drawing inspiration from the TD3 algorithm's approach to handling overestimation of the value function, two independent Critic networks are introduced. and When two networks have significantly different value assessments of the same state-action pair, it indicates high decision-making uncertainty, necessitating enhanced exploration. Uncertainty Defined as the standard deviation of the Q-values ​​output by the two Critic networks: The uncertainty perception weight coefficient (valued at 0.4~0.6 in the experiment) balances the influence of uncertainty perception and congestion perception, avoiding a single factor dominating the exploration strategy.

[0022] To address the aforementioned priority experience playback mechanism based on course learning, a systematic improvement in training efficiency is achieved through a training paradigm that progresses from easy to difficult and a sampling strategy that focuses on key aspects. The entire training process is broken down into a series of courses with increasing difficulty, and different difficulty levels are defined for each course. Each level corresponds to a different number of AGVs (in the experiment) Level 1 corresponds to 2-3 AGVs, Level 2 corresponds to 5-8 AGVs, and Level 3 corresponds to 10-15 AGVs. When the agent is at the current level... Average task success rate Exceeding the preset threshold (When set to 80% in the experiment), the course difficulty automatically increases: Within the course learning framework, the traditional priority experience replay is enhanced by calculating a comprehensive priority for each sample in the experience pool, consisting of three weighted parts: for each sample in the experience pool... Calculate the overall priority It consists of three weighted parts: The basic priority follows the core idea of ​​traditional priority experience playback, based on TD error. To measure the "unexpectedness" of a sample, The larger the value, the higher the learning value of the sample. The calculation formula is: To avoid tiny constants with a priority of 0, The priority adjustment factor balances the wealth gap in priority levels and avoids oversampling of some high-priority samples.

[0023] Course weights guide the algorithm to prioritize sampling experiences relevant to the difficulty of the current training course, ensuring the "gradualness" of the learning process. They are defined as the difficulty level of the samples. Compared to the current training difficulty level Difference Gaussian functions: The standard deviation of the Gaussian function (0.5 in the experiment) is used when the sample difficulty level matches the current training level. Take the maximum value of 1; when the difficulty difference is 1 level, Approximately 0.606; when the difference is ≥ level 2, ≤0.135, significantly reducing the sampling probability of low-relevance samples.

[0024] Event weights assign higher sampling priority to special events that are crucial to policy optimization, forcing the model to focus on key experiences. This is defined as a piecewise function. Among them, collision events have the highest weight (2.0) because collision samples contain key information about AGV obstacle avoidance failures and need to be studied in order to reduce subsequent conflicts; task success events have the second highest weight (1.8) to strengthen optimal path decision-making; regular driving events have a weight of 1.0 to ensure the normal learning of basic experience.

[0025] The specific steps of the algorithm are as follows: Step 1: Input the storage environment Env, the total number of training rounds M=5000, and the maximum number of steps per round T=200; Step 2: Initialize course level c_level=1, maximum course level C_max=3; Step 3: Initialize the number of AGVs N = 2~3 (corresponding to c_level = 1) Step 4: Initialize each AGV_i: Actor network μ_θi (input: observation s_i, output: action a_i) Dual Critic networks Q_ωi1 and Q_ωi2 (Input: (s_i, a_i), Output: Value Q) Target networks μ_θi', Q_ωi1', Q_ωi2' (parameters initially synchronized with the main network) Step 5: Initialize the CL-PER experience pool D (capacity = 5 × 10) 4 The storage tuple is: (s_i, a_i, r_i, s_i', c_level, e_type) Step 6: Set hyperparameters: γ=0.98 (discount factor), τ=0.005 (soft update coefficient), batch_size=256 (batch size), ε_base=0.7 (initial exploration rate). Step 7: Main training loop (per round); for episode = 1 to M do Step 8: Reset Environment Env: Generate a 30×30 grid map (20% obstacles), randomly assign AGV start / end points. Step 9: Obtain the initial observation set S = {s_1, s_2, ..., s_N} Step10: total_reward = 0, success_flag = False Step 11: Single-round step loop (execution phase) for step = 1 to T do Step 12: AGV Distributed Decision-Making: Generate Actions, Action Set A = {} Step13: for each AGV_i in 1..N do Step 14: Calculate the congestion degree C and decision uncertainty U Step 15: Output the final strategy Step16: return {μ_θ1, μ_θ2, ..., μ_θN} The key control parameters considered in the multi-AGV path planning algorithm for adaptive exploration and curriculum learning in this invention are described in detail below: Map specifications: 30×30 two-dimensional grid, with the grid size corresponding to a 0.5m×0.5m area in the actual warehouse; Obstacle setting: Randomly generate static obstacles (simulating shelves) that occupy 20% of the total area; the boundaries cannot be crossed. AGV parameters: maximum moving speed 1 grid per step, no turning delay, collision detection is grid overlap; Dynamic elements: Add dynamic shelves to the advanced course (level 3), which move one grid (non-obstacle area) randomly every 50 steps. State space: The local observation dimension of each AGV is 22, including: its own normalized coordinates (x, y): 2D; the target point relative vector (distance normalized to [0,1], angle normalized to [-1,1]): 2D; lidar observation (16 directions, detection distance 0~3 grids, normalized to [0,1]): 16D (8 directions to detect obstacles, 8 directions to detect other AGVs); task progress (distance traveled / total distance): 2D.

[0026] Action space: Discrete action space, containing 5 actions: {move up, move down, move left, move right, stay}. After the action is executed, the AGV position is updated by 1 grid (the position remains unchanged when staying).

[0027] Reward Function: Designing a Composite Reward Function to Guide AGV Efficient Learning. : AGVs receive a +20 bonus upon reaching the finish line, and an additional +50 bonus for all AGVs completing the mission. Collision penalty with obstacles or other AGVs: -10, stop for 1 step after collision; A base penalty of -0.01 is applied to each step to prevent the AGV from wandering in place. : When the distance to the target point decreases after each move, the reward is +0.1 × the distance decreased; when the distance increases, the penalty is -0.1 × the distance increased (distance unit is grid).

[0028] See attached document Figure 3 Three mainstream deep reinforcement learning algorithms, DDPG, MADDPG, and MADDPG+PER, were selected as baselines to ensure fairness in the comparison. The average reward change curves of the four algorithms over 5000 training rounds are presented (smoothed using a 50-round sliding window).

[0029] The AECL-MADDPG of this invention converges the fastest, achieving an average reward of 12.5 after 1500 training rounds and stabilizing at around 13.2 after 2000 rounds. MADDPG+PER converges the second fastest, stabilizing at 10.8 after 3000 rounds. Standard MADDPG converges more slowly, stabilizing at 8.6 after 4000 rounds. DDPG performs the worst, failing to fully converge even after 5000 rounds, with an average reward of only 5.4 in the stable phase. This convergence advantage stems from the CL-PER mechanism of AECL-MADDPG: a "from easy to difficult" curriculum design reduces initial learning difficulty while prioritizing the sampling of key samples such as collisions and successes, improving data utilization efficiency; adaptive exploration reduces invalid exploration, further shortening the convergence time.

[0030] Table 1 shows the key performance indicators of the four algorithms' training mature models after 100 independent tests under the highest difficulty course (Level 3, 15 AGVs).

[0031] Table 1. Performance Comparison of Algorithms in a High-Density Scenario with 15 AGVs Based on the data in Table 1 for the most difficult course, the following conclusions can be drawn: Task success rate: AECL-MADDPG achieved 99.1%, an improvement of 13.4 percentage points compared to MADDPG and 7.9 percentage points compared to MADDPG+PER, indicating that it has extremely strong robustness in high-density dynamic environments and can almost completely avoid task failures caused by collisions and path congestion.

[0032] Average path length: AECL-MADDPG is the shortest (36.4 grids), which is 11.6% shorter than MADDPG and 7.8% shorter than MADDPG+PER, proving that the adaptive exploration strategy can guide the AGV to choose a better path and avoid taking a detour due to blind exploration.

[0033] Average number of collisions: AECL-MADDPG has only 0.12 collisions per round, which is 88.6% lower than MADDPG and 86.4% lower than MADDPG+PER. This is because dynamic congestion perception can avoid high-density areas in real time and reduce conflicts between AGVs.

[0034] Average completion time: AECL-MADDPG is the fastest (89.6 steps), which is 26.3% shorter than MADDPG, demonstrating its advantage in path planning efficiency and significantly improving the throughput of the warehousing system.

[0035] The core reason for the performance advantage lies in the fact that the adaptive exploration strategy of AECL-MADDPG enables the AGV to dynamically adjust its behavior (such as actively detouring in congested areas), while the CL-PER mechanism ensures that the model prioritizes learning key experiences such as collision avoidance and optimal paths. The two work together to optimize "decision efficiency" and "strategy safety".

[0036] The above descriptions are merely some embodiments of the present invention. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of the present invention. Any modifications, equivalent substitutions, and improvements made within the inventive concept of the present invention are all within the protection scope of the present invention.

Claims

1. A dynamic adaptive exploration and learning multi-AGV path planning algorithm, comprising the following steps: S1: Initialization steps: Load parameters such as the total number of training rounds and the maximum number of AGVs; build a warehouse environment model to complete AGV deployment, collision detection, and reward / penalty feedback; initialize multiple AGV agents, each of which includes an Actor network for action decision-making and a Critic network for value evaluation; build an experience replay pool to store the experience data generated by the AGVs during the interaction process. S2: Training main loop: After initialization, the training round loop is entered. Each round corresponds to a complete multi-AGV path planning training process. S3: Environment Interaction Steps: At the beginning of each training round, the environment is reset according to the current course difficulty level and a corresponding number of AGVs and their target positions are generated; then, a time step loop is entered. In each time step, the AGV obtains the environment state information and selects an action based on the adaptive exploration strategy of dynamic congestion awareness. The environment updates its state according to the action and returns the next state, reward information and event type until the task is completed or the preset timeout condition is reached. S4: Experience storage step: Encapsulate the experience samples formed by the AGV during the interaction with the environment, consisting of the state, action, reward and the next state, and add course difficulty level label and event type label to each experience sample before storing it in the experience playback pool. S5: Learning and updating steps: When the number of samples in the experience replay pool reaches the preset threshold, training samples are sampled from the experience replay pool according to the comprehensive priority, and the parameters of the Actor network and Critic network are updated using the sampled samples through the backpropagation algorithm. S6: Course Learning Steps; After each training round, the average task success rate of the most recent preset number of rounds is calculated. When the average task success rate exceeds a preset threshold, the course difficulty level is increased, and the number of AGVs or the task complexity is increased in the next training round.

2. The multi-AGV path planning algorithm for dynamic adaptive exploration and curriculum learning according to claim 1, characterized in that, In the aforementioned dynamic congestion-aware adaptive exploration strategy, the final action executed by the AGV is composed of the basic action output by the Actor network and an adaptive noise term, and its expression is as follows: in, For deterministic actions output by the Actor network, Standard Gaussian noise, This is a dynamic exploration factor, the value of which is obtained by multiplying three parts and is used to adjust the exploration intensity.

3. The multi-AGV path planning algorithm for dynamic adaptive exploration and curriculum learning according to claim 2, characterized in that, The dynamic exploration factor Determined by the basic exploration term, congestion perception term, and uncertainty perception term, its expression is: in, The baseline exploration rate gradually decreases as training progresses; For the AGV at its current or target position Congestion level at the location; For AGV in status Decision-making uncertainty; and These are the congestion-aware weighting coefficient and the uncertainty-aware weighting coefficient, respectively. The congestion Defined as the number of other AGVs within a neighborhood of the current or target position of the AGV with a radius of R, its calculation formula is: For indicator functions, when The value is 1 if it is true, and 0 otherwise. The congestion perception weight coefficient (valued at 0.6~0.8 in the experiment) increases with the degree of congestion, thereby increasing the exploration factor and encouraging the AGV to find alternative paths. The uncertainty The difference in value assessment of the same state-action pair is measured using two independent Critic networks, and is defined as follows: The uncertainty perception weight coefficient (valued at 0.4~0.6 in the experiment) balances the influence of uncertainty perception and congestion perception, avoiding a single factor dominating the exploration strategy.

4. The multi-AGV path planning algorithm for dynamic adaptive exploration and curriculum learning according to claim 1, characterized in that, The aforementioned mechanism for prioritizing experience replay based on course learning divides the training process into multiple course levels with increasing difficulty. Each course level corresponds to a different number of AGVs; when the average task success rate of the agent at the current course level exceeds a preset threshold, the course level is automatically upgraded. Within the course learning framework, the traditional priority experience replay is enhanced by calculating a comprehensive priority for each sample in the experience pool, consisting of three weighted parts: for each sample in the experience pool... Calculate the overall priority It consists of three weighted parts: in, The base priority is calculated based on the timing difference error; ε is a constant to prevent the priority from being zero; α is a priority adjustment factor. Assign weights to courses; Weights for event types, The calculation formula is: To avoid tiny constants with a priority of 0, The priority adjustment factor balances the wealth gap in priority levels and avoids oversampling of some high-priority samples. The course weights are used to guide the algorithm to prioritize sampling experiences related to the difficulty of the current training course, ensuring the "gradualness" of the learning process, and are defined as the sample difficulty level. Compared to the current training difficulty level Difference Gaussian functions: The standard deviation of the Gaussian function is 0.5, which is used in the experiment. This value is applied when the sample difficulty level matches the current training level. Take the maximum value of 1; when the difficulty difference is 1 level, Approximately 0.606; when the difference is ≥ level 2, ≤0.135 significantly reduces the sampling probability of low-relevance samples; event weights assign higher sampling priority to special events that are crucial to policy optimization, forcing the model to focus on key experiences; defined as a piecewise function: Among them, collision events have the highest weight (2.0) because collision samples contain key information about AGV obstacle avoidance failures and need to be studied in order to reduce subsequent conflicts; task success events have the second highest weight (1.8) to strengthen optimal path decision-making; regular driving events have a weight of 1.0 to ensure the normal learning of basic experience.

Images