Intelligent safe single-AGV and multi-AGV efficient scheduling strategy for complex dynamic environment

Abstract

The invention discloses a complex dynamic environment-oriented intelligent security single AGV and multi-AGV efficient scheduling strategy. Dynamic path planning and obstacle avoidance of a single AGV are realized based on a 3D convolutional neural network in combination with an adaptive exploration DQN algorithm; cooperative path planning is carried out on multiple AGVs in a complex environment based on a curriculum learning mechanism and a QMIX algorithm, and cooperative learning efficiency and stability are accelerated based on a distributed reinforcement learning algorithm and group relative strategy optimization; meanwhile, a DeepSeek large model is introduced, real-time cargo source matching and return empty vehicle matching are realized based on a DeepSeek intelligent scheduling system, and a warehousing automation revolution from'people find goods' to'AI drive goods find people 'is realized based on a DeepSeek-Vision image recognition technology. The method can significantly improve the obstacle avoidance capability and scheduling efficiency of the AGV in a complex dynamic environment, and is suitable for the field of intelligent storage and automatic manufacturing.

Description

Technical Field

[0001] The present invention relates to scheduling and path planning technologies involving AGV systems, in particular single-AGV and multi-AGV scheduling algorithms based on deep reinforcement learning, which are used for efficient path planning and collaborative obstacle avoidance in complex dynamic environments. Background Art

[0002] With the rapid development of intelligent manufacturing and logistics automation, Automated Guided Vehicle (AGV) systems are increasingly being used in smart warehousing and automated production. Traditional AGV scheduling algorithms often use static path planning methods such as A* and Dijkstra. While these methods perform well in simple environments, they are often inefficient when faced with dynamic obstacles and multiple AGV systems, making them difficult to cope with complex and changing environments. Therefore, deep reinforcement learning, an algorithm that can interact with its environment and learn autonomously, offers a new solution for AGV scheduling.

[0003] In a single AGV system, the AGV typically needs to autonomously navigate, avoid obstacles, and plan paths in dynamic environments. Traditional path planning algorithms, such as reinforcement learning methods based on Q-learning, have slow convergence in complex scenarios due to their balance between exploration and exploitation. To address this, the Deep Q-Network (DQN) combined with a Convolutional Neural Network (CNN) was introduced to enhance the adaptability and learning capabilities of AGVs in dynamic path planning.

[0004] In multi-AGV systems, path conflicts and coordinated obstacle avoidance become the core difficulties of scheduling. The scheduling of multiple AGVs requires coordinating the path planning of multiple intelligent agents to ensure that collisions and deadlocks are avoided. To address this issue, distributed reinforcement learning algorithms such as QMIX have gradually become the mainstream method for solving multi-AGV scheduling. Through a decentralized strategy, QMIX enables each AGV to make independent decisions based on its own local environment, while performing joint optimization through the global network, thereby improving the scheduling efficiency of the multi-AGV system. In complex multi-AGV systems, improving training speed and training stability are key. To meet this demand, the distributed reinforcement learning framework and group relative strategy optimization algorithms have become important solutions, improving training efficiency through parallel learning and experience sharing, as well as intra-group and inter-group constraints. In actual application scenarios, the deployment of large language models such as DeepSeek can help make intelligent decisions. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent, safe, and efficient scheduling strategy for single and multiple AGVs in complex dynamic environments, which can significantly improve the obstacle avoidance capability and scheduling efficiency of AGVs in complex dynamic environments and is suitable for the fields of intelligent warehousing and automated manufacturing.

[0006] The technical solution of the present invention is: an intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments, including:

[0007] The first strategy is based on the combination of 3D convolutional neural network and adaptive exploration DQN algorithm for single AGV path planning;

[0008] The second strategy is based on the combination of curriculum learning and QMIX algorithm for efficient scheduling of multiple AGVs.

[0009] In the above technical solution, the first strategy includes the following steps:

[0010] Step A1: Acquire environmental information of the working area where the AGV is located, and rasterize the environmental information into a multi-frame dynamic raster map with a time dimension;

[0011] Step A2: extracting spatial and temporal features of the environmental information based on a 3D convolutional neural network, and dynamically predicting the movement trajectory of the obstacle;

[0012] Step A3: Dynamically adjust the ε-greedy strategy to balance exploration and exploitation based on the adaptive exploration DQN algorithm;

[0013] Step A4: Based on the heuristic reward mechanism, combined with dynamic distance and target point rewards, the AGV obstacle avoidance path is optimized.

[0014] In the above technical solution, step A1 includes:

[0015] Step A11: The AGV obtains current environmental information through sensors, where the environmental information includes the locations of static and dynamic obstacles.

[0016] Step A12: Divide the working area into grid units of equal size and construct a dynamic grid map. The grid map is updated in real time according to sensor input to reflect dynamic changes in the current environment.

[0017] In the above technical solution, step A2 includes:

[0018] Step A21: Set the number of skipped training frames to d, let the AGV move randomly d times, and store the state diagram of each frame in sequence into a double-ended sequence of length d;

[0019] Step A22: Initialize the 3D convolutional neural network and start training. After each AGV movement, pass the new state diagram to the end of the double-ended sequence and remove the oldest image data.

[0020] Step A23: In each training, a sequence of images including d frames of AGV current position, target position and obstacle position is used as the state value St Input 3D convolutional neural network, the sequence length d is the depth D of the 3D image data;

[0021] Step A24: The AGV performs path planning in a complex dynamic environment based on the features extracted by the 3D convolutional neural network, predicting and avoiding dynamic obstacles in advance.

[0022] In the above technical solution, step A3 includes:

[0023] The adaptive ε-greedy strategy dynamically adjusts the ε value based on the learned experience at each training step by setting the attenuation factor d. The formula is:

[0024] ε=(ε max -ε min ) -k / d .

[0025] In the above technical solution, step A4 includes:

[0026] The heuristic exploration strategy is based on the actual cost reward function R g and R h Estimated comprehensive reward function R composed of cost reward function f As the reward obtained by AGV during normal driving, the reward function is as follows:

[0027]

[0028] Actual cost reward function R g : Based on the change in distance from the AGV's current position to the starting point, the formula is:

[0029] R g =d start (s t+1 )-d start (s t )

[0030]

[0031] Estimated cost reward function R h : Based on the distance change between the AGV's current position and the target point, the formula is:

[0032] R h =d end (s t+1 )-d end (s t )

[0033] Among them, k g and k hare the weight coefficients of the actual cost reward function and the estimated cost reward function, respectively. The deep Q network is trained by reinforcement learning. During the training process, the number of interactions between the AGV and the environment is set to multiple training rounds. Each training round corresponds to a complete process from the starting point to the end point or a collision. The deep Q network optimizes the path planning strategy by updating the Q-value function.

[0034] In the above technical solution, the second strategy includes the following steps:

[0035] Step B1: Obtain the local environment information of each AGV and update the environment status based on the rasterized model;

[0036] Step B2: Construct a local path planning strategy for each AGV based on a distributed partially observable Markov model;

[0037] Step B3: The global hybrid network based on the QMIX algorithm jointly optimizes the local Q values ​​of multiple AGVs to ensure global path optimality and collaborative work;

[0038] Step B4: Based on the course learning mechanism, gradually increase the task difficulty and optimize the AGV's task completion rate;

[0039] Step B5: Improve training stability based on the GRPO method;

[0040] Step B6: Deploy DeepSeek to help make intelligent decisions and intelligent scheduling.

[0041] In the above technical solution, step B1 includes:

[0042] Step B11: The work area is divided into multiple grid cells of equal size by raster modeling. Each grid cell is marked according to whether it is occupied by an obstacle or whether other AGVs have entered;

[0043] Step B12: Set the initial and target positions of multiple AGVs, as well as the positions of obstacles. Each AGV obtains the local state of the surrounding environment through sensors, including the positions of obstacles and other AGVs, and updates the grid map in real time based on this information.

[0044] In the above technical solution, step B2 includes:

[0045] Step B21: Establish the Dec-POMDP model and define the state, observation, action, reward and conflict type;

[0046] Step B22: Independently learn local strategies. Each AGV independently learns its local strategy under the distributed reinforcement learning framework. Based on local state information and local Q-values, each AGV updates its strategy to maximize future rewards.

[0047] Step B23: Sharing key information. Through the communication mechanism, each AGV shares local state information and local Q value to ensure the collaborative work between multiple AGVs. The global hybrid network integrates the local Q value of each AGV to form a global Q value to achieve the optimality of global path planning.

[0048] Step B24: Decomposing the global path planning problem into multiple local optimization problems using a distributed computing framework.

[0049] Step B25: Parallel computing, each AGV calculates its local path planning strategy in parallel on the distributed computing nodes.

[0050] In the above technical solution, step B3 includes:

[0051] Step B31: Local Q-value calculation. Each AGV independently observes its local state and calculates a local Q-value function based on its action. The local Q-value represents the expected value of reaching the target state after selecting an action under the current state of the AGV. The update of the local Q-value is based on the Bellman equation:

[0052]

[0053] Among them, r i is the immediate reward; γ is the discount factor used to weigh the current reward against future rewards; s i 'For action a i The new state after Indicates that in state s i 'The maximum Q value of taking the optimal action;

[0054] Step B32: Global Q value combination. The local Q value of each AGV is combined into a global Q value through the global hybrid network, which represents the comprehensive performance of the entire system. The calculation formula of the global Q value is:

[0055] Q total =f(Q1(s1,a1),Q2(s 2, a2),...,Q n (s n ,a n ))

[0056] The global hybrid network uses a nonlinear function f to perform weighted combination of local Q values ​​to ensure the overall collaborative optimality of the multi-AGV system;

[0057] Step B33: Gradient update, the global Q value is updated by gradient descent. The algorithm uses the following loss function to minimize the difference between the current Q value and the target Q value:

[0058]

[0059] Among them, θ and θ' are the parameters of the current and target networks respectively; r i is the immediate reward; N is the number of AGVs in training; by minimizing the loss function and updating the global Q value, the system can converge to the optimal solution after multiple iterations;

[0060] Step B34: Action selection. Each AGV selects an action based on its local Q-value, using an ε-greedy strategy to balance exploration and exploitation. The ε value is initially large, encouraging random exploration. As training progresses, the ε value is gradually reduced, and decision-making relies more on the learned optimal strategy. During algorithm execution, the AGV selects the optimal action based on its local state and Q-value at each time step, ultimately completing the task.

[0061] In the above technical solution, step B4 includes:

[0062] Step B41: Initially, a smaller allocation radius is set to allow the AGV to perform path planning within a smaller range;

[0063] Step B42: As training progresses, when multiple AGVs achieve a certain completion rate in a relatively low-difficulty environment, the system gradually increases the number of obstacles and the complexity of their distribution, lengthens the navigation path, and increases the AGV's distribution radius. As the task difficulty increases, the AGV needs to handle more dynamic obstacles and complex interactions. The completion rate is calculated as follows:

[0064]

[0065] is the number of AGVs that successfully reach the target location, and N is the total number of AGVs;

[0066] Step B43: Learning curve optimization, specifically:

[0067] Local learning: AGV learns how to avoid obstacles and navigate in simple environments in the early stages;

[0068] Global optimization: As the task difficulty increases, each AGV not only needs to make independent decisions based on local observation information, but also needs to optimize overall path planning and conflict avoidance through a global hybrid network;

[0069] Step B44: Dynamic feedback mechanism. During the course learning process, the system dynamically adjusts the AGV task settings based on the results of each round of training. If the completion rate does not meet expectations, the system will temporarily reduce the task difficulty to prevent the training from falling into a local optimal or overly complex environment.

[0070] In the above technical solution, step B5 includes:

[0071] Step B51: Consistency constraints within the group ensure that the policy update direction of the AGVs in the group is consistent to avoid deviation of a single policy;

[0072] Step B52: A smooth policy update mechanism limits the magnitude of each policy update to ensure smooth and consistent policy updates for AGVs within the group.

[0073] Step B53: Distributed experience replay. Each group independently maintains a distributed experience replay buffer to store historical exchange data within the group and reduce training deviation.

[0074] The above technical solution introduces the DeepSeek big model, and uses the DeepSeek big model's intelligent scheduling system to achieve "dynamic carpooling optimization", real-time matching of cargo sources and return empty vehicles, and realize cargo distribution based on DeepSeek-Vision's image recognition technology. With the powerful reasoning ability of the big model, it fully considers multiple factors and intelligently plans the optimal transshipment plan.

[0075] The advantages of the present invention are:

[0076] The present invention can significantly improve the obstacle avoidance capability and scheduling efficiency of AGV in complex dynamic environments, and is suitable for the fields of intelligent warehousing and automated manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0078] Figure 1 This is a flow chart of the efficient scheduling strategy for single AGV and multiple AGVs of the present invention.

[0079] Figure 2 Modeling the dynamic grid map of the single AGV system of the present invention.

[0080] Figure 3 This is the global dynamic path planning result diagram of the single AGV system of the present invention.

[0081] Figure 4 This is the test map of the multi-AGV system of the present invention.

[0082] Figure 5 Schematic diagram of the observation matrix in the multi-AGV system embodiment of the present invention.

[0083] Figure 6 Schematic diagram of conflict detection in a multi-AGV system embodiment of the present invention.

[0084] Figure 7 This is the convergence curve diagram of the single AGV path planning experiment of the present invention.

[0085] Figure 8It is the training completion rate of the single AGV path planning model proposed in this invention.

[0086] Figure 9 It is the training success rate of the multi-AGV efficient scheduling model proposed in this invention.

[0087] Figure 10 This is a test chart comparing the completion rates of the multi-AGV efficient scheduling strategy algorithm proposed in this invention and the PTIMAL algorithm. DETAILED DESCRIPTION

[0088] To make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features therein can be combined with each other without conflict.

[0089] The following description sets forth numerous specific details to facilitate a thorough understanding of the present invention. The embodiments described are merely examples of some aspects of the present invention, and are not exhaustive. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0090] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0091] Example:

[0092] See also Figures 1 to 10 As shown, the present invention relates to an intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments, including:

[0093] The first strategy is based on the combination of 3D convolutional neural network and adaptive exploration DQN algorithm for single AGV path planning;

[0094] The second strategy is based on the combination of curriculum learning and QMIX algorithm for efficient scheduling of multiple AGVs.

[0095] The specific steps of the first strategy include:

[0096] Step A1: Acquire environmental information of the working area where the AGV is located, and rasterize the environmental information into a multi-frame dynamic raster map with a time dimension; specifically, Step A1 includes:

[0097] Step A11: The AGV obtains the current environment status information through sensors (such as laser radar, camera), and the environment information includes the location of static and dynamic obstacles. Figure 2 As shown in the figure, the static obstacles are mainly modeled in the tallying area and the general cargo storage area, which are represented by box I; the dynamic obstacles move back and forth along a specific trajectory within the defined motion range, which are represented by box II.

[0098] Step A12: Divide the work area into grid cells of equal size and construct a dynamic grid map. The grid map is updated in real time based on sensor input to reflect dynamic changes in the current environment. The dynamic grid map is local environmental information acquired by the AGV with itself as the center, and is updated in real time as the AGV moves.

[0099] Step A2: extracting spatial and temporal features of the environmental information based on a 3D convolutional neural network (3D-CNN) to dynamically predict the movement trajectory of the obstacle. Specifically, step A2 includes:

[0100] Step A21: Set the number of skipped training frames to d, let the AGV move randomly d times, and store the state diagram of each frame in sequence into a double-ended sequence of length d;

[0101] Step A22: Initialize the 3D convolutional neural network and start training. After each AGV movement, pass the new state diagram to the end of the double-ended sequence and remove the oldest image data.

[0102] Step A23: In each training, a sequence of images including d frames of AGV current position, target position and obstacle position is used as the state value S t Input 3D convolutional neural network, the sequence length d is the depth D of the 3D image data;

[0103] Step A24: The AGV performs path planning in a complex dynamic environment based on the features extracted by the 3D convolutional neural network, predicting and avoiding dynamic obstacles in advance.

[0104] Step A3: Dynamically adjust the ε-greedy strategy to balance exploration and exploitation based on the adaptive exploration DQN algorithm; specifically, step A3 includes:

[0105] Step A31: Use the Deep Q Network (DQN) algorithm to select the path of the AGV. The input is the environmental features extracted by 3D-CNN, and the output is the possible path selection of the AGV (such as forward, left, right, etc.).

[0106] Step A32: DQN uses an adaptive ε-greedy strategy to balance exploration and exploitation, and the greed factor is dynamically adjusted:

[0107] ε=(ε max -ε min )- k / d

[0108] Where k represents the total number of steps (times) that the AGV moves during the entire training, and d represents the decay factor of ε. The above method helps to balance the trade-off between exploration and exploitation at different stages of training. The relevant parameters are set as follows:

[0109]

[0110] Step A4: Based on the heuristic reward mechanism, combined with dynamic distance and target point rewards, optimize the AGV obstacle avoidance path. Specifically, the step A4 includes:

[0111] Step A41: The reward function is designed to change dynamically according to different states, as follows:

[0112]

[0113] Among them, k g R g +k h R h k in g and k h Both are set to 1.

[0114] Step A42: Model training through reinforcement learning. The AGV repeatedly performs path planning tasks in a simulated environment, and the DQN continuously updates the Q-value function through interaction with the environment. The number of interactions between the AGV and the environment during training is set to a number of episodes, each representing the entire process of the AGV from its starting point to its destination or collision. Through multiple iterations, the AGV gradually learns to choose the optimal path. To verify the path planning performance of a single AGV system, the following evaluation metrics are designed:

[0115] (a) Success rate: measures the proportion of times the AGV successfully reaches the target point.

[0116] (b) Average path length: The average travel distance of the AGV from the starting point to the destination point is calculated to evaluate the efficiency of path planning.

[0117] (c) Average time steps: Statistics of the average time steps required for the AGV to complete the task.

[0118] (d) Number of collisions: records the number of collisions that occur during the path planning process of the AGV to evaluate its obstacle avoidance capability.

[0119] exist Figure 2The AGV path planning effect is tested in the simulation environment shown in the figure. The results show that after multiple dynamic and static obstacle avoidances and reaching the end point, the global dynamic path planning test of this round is successfully completed. The final path result is as follows Figure 3 In addition, after 1000 rounds of testing, 825 rounds of global dynamic path planning without collision were successfully completed, and the final success rate of the model was P success It is 82.5%.

[0120] The feasibility and effectiveness of this model and adaptive local environment strategy in dynamic path planning were verified.

[0121] The specific steps of the second strategy include:

[0122] Step B1: Modeling the multi-AGV environment using a grid method, obtaining local environmental information of each AGV, and updating the environmental status based on the grid model; specifically, step B1 includes:

[0123] Step B11: Randomly generate K×K working maps with different shapes, defined by the map size K∈{10,40,80} and the obstacle density δ∈{0,0.1,0.2,0.3}, which defines the proportion of unoccupiable locations in the map.

[0124] Step B12: Set the initial and target positions of multiple AGVs. All agents start from random positions and are assigned a radius R. c Randomly assign targets. Each AGV obtains the local state of the surrounding environment through observation, including the location of obstacles and the location of other AGVs, and updates the grid map in real time based on this information.

[0125] Step B2: Constructing a local path planning strategy for each AGV based on a distributed partially observable Markov model (Dec-POMDP); specifically, step B2 includes:

[0126] Step B21: Establish a Dec-POMDP model and define states, observations, actions, rewards, and conflict types. Specifically, the following are included:

[0127] (a) Define the state space. In this experiment, the state space includes the current position of the AGV in the working area and the environmental information collected by the AGV through observation.

[0128] (b) Define the action space. In this experiment, the AGV's action space is set to the four main directions of movement: east, south, west, and north. Therefore, the action set can be expressed as: Ω(A) = {up 1 square, down 1 square, left 1 square, right 1 square, no movement}.

[0129] (c) Define the observation set. The observation matrix serves as the model input. The observation matrix has five channels: obstacle position; other AGV target positions; other AGV positions; current AGV target position; and current AGV direction. The direction is encoded as the Manhattan distance to the target location and indicates the direction to the target location. Each AGV can wait or move in all cardinal directions.

[0130] (d) Define the reward function. This embodiment gives different rewards to the AGV for different actions. The specific settings are as follows:

[0131]

[0132] (e) Define conflict types. During the path planning process, the system detects and prevents potential conflicts by observing the movement paths of each AGV. If a conflict is detected, the system resolves the problem by adjusting the path or replanning. Detected conflicts include:

[0133] (f) Vertex collision: occurs when two or more AGVs try to enter the same grid cell at the same time.

[0134] (g) Edge collision: occurs when the paths of two AGVs intersect while moving.

[0135] Step B22: Independently learn local policies. Each AGV independently learns its local policy within a distributed reinforcement learning framework. Based on local state information (e.g., position, velocity, sensor data) and local Q-values, each AGV updates its policy to maximize future rewards.

[0136] Step B23: Sharing Key Information. Through the communication mechanism, each AGV shares key information such as local state and local Q-value, ensuring collaborative operation among multiple AGVs. The global hybrid network integrates the local Q-values ​​of each AGV to form a global Q-value, achieving optimal global path planning.

[0137] Step B3: Jointly optimize the local Q values ​​of multiple AGVs based on the global hybrid network of the QMIX algorithm to ensure global path optimality and collaborative work; specifically, step B3 includes:

[0138] Step B31: Local Q-value calculation. Each AGV independently observes its local state and calculates a local Q-value function based on its actions. This local Q-value represents the expected value of reaching the target state after selecting an action under the AGV's current state. The update of the local Q-value is based on the Bellman equation:

[0139]

[0140] Among them, r iis the immediate reward; γ is the discount factor used to weigh the current reward against future rewards; s i 'For action a i The new state after Indicates that in state s i 'The maximum Q value for taking the optimal action.

[0141] Step B32: Global Q value combination. The local Q value of each AGV is combined into a global Q value through the global hybrid network, which represents the comprehensive performance of the entire system. The calculation formula of the global Q value is:

[0142] Q total =f(Q1(s1,a1),Q2(s 2, a2),...,Q n (s n ,a n ))

[0143] The global hybrid network uses a nonlinear function f to perform weighted combination of local Q values ​​to ensure the overall collaborative optimality of the multi-AGV system.

[0144] Step B33: Gradient update. The global Q value is updated by gradient descent. The algorithm uses the following loss function to minimize the difference between the current Q value and the target Q value:

[0145]

[0146] Among them, θ and θ' are the parameters of the current and target networks respectively; r i is the immediate reward; N is the number of AGVs in training. By minimizing the loss function and updating the global Q value, the system converges to the optimal solution after multiple iterations. In addition, the main parameters set in the model include the observation space size, discount factor, learning rate, hidden layer size, number of episodes per round, time limit, and number of training rounds. The specific settings are shown in the following table:

[0147]

[0148] Step B34: Action Selection. Each AGV selects an action based on its local Q-value, employing an ε-greedy strategy to balance exploration and exploitation. Initially, the ε value is large, encouraging random exploration. As training progresses, the ε value is gradually reduced, relying more on the learned optimal strategy for decision-making. During algorithm execution, the AGV selects the optimal action based on its local state and Q-value at each time step, ultimately completing the task.

[0149] Step B4: Based on the course learning mechanism, gradually increase the task difficulty and optimize the AGV task completion rate; specifically, the step B4 includes

[0150] Step B41: Initially, a smaller allocation radius is set to allow the AGV to perform path planning within a smaller range.

[0151] Step B42: As training progresses, when multiple AGVs achieve a certain completion rate in a relatively low-difficulty environment, the system gradually increases the number of obstacles and their distribution complexity, lengthens the navigation path, and increases the AGV's distribution radius. As the task difficulty increases, the AGV needs to handle more dynamic obstacles and complex interactions. The completion rate is calculated as follows:

[0152]

[0153] In this experiment, a statistical method is used to decide whether to increase the allocation radius. After training E rounds, the average completion rate at the completion of each round can be evaluated:

[0154]

[0155] Assuming that the completion rate follows a normal distribution, if μ-η σ ≥U, then the allocation radius is increased by 1 each time, where U∈(0,1) is the decision threshold for course learning and η>0 is the deviation factor for the specified confidence level.

[0156] Step B43: Optimize the learning curve. Specifically:

[0157] Local learning: AGV learns how to avoid obstacles and navigate in simple environments in the early stages.

[0158] Global optimization: As the task difficulty increases, each AGV not only needs to make independent decisions based on local observation information, but also needs to optimize overall path planning and conflict avoidance through a global hybrid network.

[0159] Step B44: Dynamic Feedback Mechanism. During the course learning process, the system dynamically adjusts the AGV's task settings based on the results of each round of training. If the completion rate does not meet expectations, the system will temporarily reduce the task difficulty to prevent training from falling into a local optimum or an overly complex environment. In addition, a CACTUS-OPLEX algorithm with the same properties was developed. The CACTUS-QMIX algorithm was compared with the CACTUS-OPLEX algorithm and the PRIMAL (Pathfinding via Reinforcement and Imitation Multi-Agent Learning) algorithm to verify the effectiveness of the model.

[0160] Test results show that when the number of AGVs is 8 and the training is completed for 5000 rounds, the completion rate of the OPLEX model approaches convergence around 2000 rounds and fluctuates around 0.5. Compared to the QMIX model, the convergence rate is slower and the completion rate is lower. Experimental results confirm that the curriculum-based QPLEX model performs inferior to the QMIX model. When the number of AGVs n is between 4 and 128, the completion rate of CACTUS-QMIX is higher than that of PRIMAL. When the number of AGVs n is 4, 8, and 16, the completion rate is 1, indicating that CACTUS-QMIX has stronger stability and better performance than PRIMAL. However, when the number of AGVs n is 256 and 512, due to the limited space and the large number of agents, the performance of both algorithms is relatively poor. Overall, the CACTUS-QMIX algorithm is more stable for small-scale agents and also has a higher completion rate and better performance for large-scale agents.

[0161] Step B5: Improving training stability based on the GRPO method; specifically, step B5 includes:

[0162] Step B51: Constrain the intra-group policy consistency to ensure that the policy update direction of the AGVs in the group is consistent and avoid deviation of a single policy.

[0163] Step B52: A smooth policy update mechanism limits the magnitude of each policy update to ensure smooth and consistent policy updates for AGVs within the group.

[0164] Step B53: Distributed experience replay. Each group independently maintains a distributed experience replay buffer to store historical exchange data within the group and reduce training deviation.

[0165] Step B6: Deploy DeepSeek to help make intelligent decisions and intelligent scheduling. Specifically, step B6 includes:

[0166] Step B61: Dynamic carpooling optimization and cargo allocation. The DeepSeek intelligent dispatching system is deployed to collect real-time cargo information (including cargo type, volume, weight, and destination) and vehicle status (including location, load capacity, and return route). Using deep learning algorithms, multi-dimensional matching is performed to efficiently combine cargo with empty return vehicles to reduce empty trip rates. DeepSeek's image recognition technology is also integrated to automatically identify and classify cargo, accurately extracting cargo features.

[0167] Step B62: Intelligent reasoning and scheduling optimization, deploy the DeepSeek reasoning engine, comprehensively consider the cargo type, vehicle capacity, time window, transportation cost and external environmental factors, and generate the optimal transshipment plan.

[0168] Step B63: Full-process optimization and intelligent scheduling, deploy dynamic matching, cargo allocation and scheduling modules in stages to achieve full-process optimization from data collection to scheduling planning.

[0169] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the present invention and implement it accordingly. They are not intended to limit the scope of protection of the present invention. Any modifications made within the spirit of the main technical solution of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. An intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments, characterized by: include: The first strategy is based on the combination of 3D convolutional neural network and adaptive exploration DQN algorithm for single AGV path planning; The second strategy is based on the combination of curriculum learning and QMIX algorithm for efficient scheduling of multiple AGVs.

2. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 1 is characterized in that: The first strategy includes the following steps: Step A1: Acquire environmental information of the working area where the AGV is located, and rasterize the environmental information into a multi-frame dynamic raster map with a time dimension; Step A2: extracting spatial and temporal features of the environmental information based on a 3D convolutional neural network, and dynamically predicting the movement trajectory of the obstacle; Step A3: Dynamically adjust the ε-greedy strategy to balance exploration and exploitation based on the adaptive exploration DQN algorithm; Step A4: Based on the heuristic reward mechanism, combined with dynamic distance and target point rewards, the AGV obstacle avoidance path is optimized.

3. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 2 is characterized in that: The step A1 comprises: Step A11: The AGV obtains current environmental information through sensors, where the environmental information includes the locations of static and dynamic obstacles. Step A12: Divide the working area into grid units of equal size and construct a dynamic grid map. The grid map is updated in real time according to sensor input to reflect dynamic changes in the current environment.

4. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 2 is characterized in that: The step A2 comprises: Step A21: Set the number of skipped training frames to d, let the AGV move randomly d times, and store the state diagram of each frame in sequence into a double-ended sequence of length d; Step A22: Initialize the 3D convolutional neural network and start training. After each AGV movement, pass the new state diagram to the end of the double-ended sequence and remove the oldest image data. Step A23: In each training, a sequence of images including d frames of AGV current position, target position and obstacle position is used as the state value S t Input 3D convolutional neural network, the sequence length d is the depth D of the 3D image data; Step A24: The AGV performs path planning in a complex dynamic environment based on the features extracted by the 3D convolutional neural network, predicting and avoiding dynamic obstacles in advance.

5. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 2 is characterized in that: The step A3 comprises: The adaptive ε-greedy strategy dynamically adjusts the ε value based on the learned experience at each training step by setting the attenuation factor d. The formula is: e=(e max -e min ) -k / d 。 6. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 2 is characterized in that: The step A4 comprises: The heuristic exploration strategy is based on the actual cost reward function R g and the estimated cost reward function R h The comprehensive reward function R f As the reward obtained by AGV during normal driving, the reward function is as follows: Actual cost reward function R g : Based on the change in distance from the AGV's current position to the starting point, the formula is: R g =d start (s t+1 )-d start (s t ) Estimated cost reward function R h : Based on the distance change between the AGV's current position and the target point, the formula is: R h =d end (s t+1 )-d end (s t ) Among them, k g and k h are the weight coefficients of the actual cost reward function and the estimated cost reward function, respectively. The deep Q network is trained by reinforcement learning. During the training process, the number of interactions between the AGV and the environment is set to multiple training rounds. Each training round corresponds to a complete process from the starting point to the end point or a collision. The deep Q network optimizes the path planning strategy by updating the Q-value function.

7. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 1 is characterized in that: The second strategy includes the following steps: Step B1: Obtain the local environment information of each AGV and update the environment status based on the rasterized model; Step B2: Construct a local path planning strategy for each AGV based on a distributed partially observable Markov model; Step B3: The global hybrid network based on the QMIX algorithm jointly optimizes the local Q values ​​of multiple AGVs to ensure global path optimality and collaborative work; Step B4: Based on the course learning mechanism, gradually increase the task difficulty and optimize the AGV's task completion rate; Step B5: Improve training stability based on the GRPO method; Step B6: Deploy DeepSeek to help make intelligent decisions and intelligent scheduling.

8. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 7 is characterized in that: The step B1 comprises: Step B11: The work area is divided into multiple grid cells of equal size by raster modeling. Each grid cell is marked according to whether it is occupied by an obstacle or whether other AGVs have entered; Step B12: Set the initial and target positions of multiple AGVs, as well as the positions of obstacles. Each AGV obtains the local state of the surrounding environment through sensors, including the positions of obstacles and other AGVs, and updates the grid map in real time based on this information.

9. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 7 is characterized in that: The step B2 comprises: Step B21: Establish the Dec-POMDP model and define the state, observation, action, reward and conflict type; Step B22: Independently learn local strategies. Each AGV independently learns its local strategy under the distributed reinforcement learning framework. Based on local state information and local Q-values, each AGV updates its strategy to maximize future rewards. Step B23: Sharing key information. Through the communication mechanism, each AGV shares local state information and local Q value to ensure the collaborative work between multiple AGVs. The global hybrid network integrates the local Q value of each AGV to form a global Q value to achieve the optimality of global path planning. Step B24: Decomposing the global path planning problem into multiple local optimization problems using a distributed computing framework; Step B25: Parallel computing, each AGV calculates its local path planning strategy in parallel on the distributed computing nodes.

10. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 7 is characterized in that: Described step B3 comprises: Step B31: Local Q-value calculation. Each AGV independently observes its local state and calculates a local Q-value function based on its action. The local Q-value represents the expected value of reaching the target state after selecting an action under the current state of the AGV. The update of the local Q-value is based on the Bellman equation: Among them, r i is the immediate reward; γ is the discount factor used to weigh the current reward against future rewards; s i 'For action a i The new state after Indicates that in state s i 'The maximum Q value of taking the optimal action; Step B32: Global Q value combination. The local Q value of each AGV is combined into a global Q value through the global hybrid network, which represents the comprehensive performance of the entire system. The calculation formula of the global Q value is: Q total =f(Q1(s1,a1),Q2(s 2, a2),...,Q n (s n ,a n )) The global hybrid network uses a nonlinear function f to perform weighted combination of local Q values ​​to ensure the overall collaborative optimality of the multi-AGV system; Step B33: Gradient update, the global Q value is updated by gradient descent. The algorithm uses the following loss function to minimize the difference between the current Q value and the target Q value: Among them, θ and θ' are the parameters of the current and target networks respectively; r i is the immediate reward; N is the number of AGVs in training; by minimizing the loss function and updating the global Q value, the system can converge to the optimal solution after multiple iterations; Step B34: Action selection. Each AGV selects an action based on its local Q value, using the ε-greedy strategy to balance exploration and exploitation.

11. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 7 is characterized in that: Step B4 includes: Step B41: Initially, a smaller allocation radius is set to allow the AGV to perform path planning within a smaller range; Step B42: As training progresses, when multiple AGVs achieve a certain completion rate in a relatively low-difficulty environment, the system gradually increases the number of obstacles and the complexity of their distribution, lengthens the navigation path, and increases the AGV's distribution radius. As the task difficulty increases, the AGV needs to handle more dynamic obstacles and complex interactions. The completion rate is calculated as follows: N goal is the number of AGVs that successfully reach the target location, and N is the total number of AGVs; Step B43: Learning curve optimization, specifically: Local learning: AGV learns how to avoid obstacles and navigate in simple environments in the early stages; Global optimization: As the task difficulty increases, each AGV not only needs to make independent decisions based on local observation information, but also needs to optimize overall path planning and conflict avoidance through a global hybrid network; Step B44: Dynamic feedback mechanism. During the course learning process, the system dynamically adjusts the AGV task settings based on the results of each round of training. If the completion rate does not meet expectations, the system will temporarily reduce the task difficulty to prevent the training from falling into a local optimal or overly complex environment.

12. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 7 is characterized in that: Step B5 includes: Step B51: Consistency constraints within the group ensure that the policy update direction of the AGVs in the group is consistent to avoid deviation of a single policy; Step B52: A smooth policy update mechanism limits the magnitude of each policy update to ensure smooth and consistent policy updates for AGVs within the group. Step B53: Distributed experience replay. Each group independently maintains a distributed experience replay buffer to store historical exchange data within the group and reduce training deviation.

13. The intelligent and safe single AGV and multi-AGV efficient scheduling strategy for complex dynamic environments according to claim 7 is characterized in that: By introducing the DeepSeek big model and leveraging its intelligent scheduling system, we achieve "dynamic carpooling optimization," matching cargo sources and empty return vehicles in real time, and allocating cargo based on DeepSeek-Vision's image recognition technology. Leveraging the powerful reasoning capabilities of the big model, we can fully consider multiple factors and intelligently plan the optimal transshipment plan.

Images