Multi-AGV path planning method and system based on deep reinforcement learning

Abstract

The invention belongs to the technical field of AGV path planning. According to the multi-AGV path planning method and system based on deep reinforcement learning, an MDP model of an AGV path planning problem is established, a reward function is constructed in a reward remodeling mode, and the AGVs are guided to evaluate real-time actions more accurately; an improved PPO algorithm is provided, an Actor-Critic network is established for an AGV path planning problem, and a strategy network is updated by using a synchronization strategy updating mode; different scales of task tests prove that the MAPPO algorithm provided by the invention has better solving performance compared with a PPO algorithm in a large-scale storage environment; in addition, compared with a traditional path planning method, the method has the advantages that the length of the solved average path is shorter, the number of inflection points is smaller, and higher solving quality is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of AGV path planning, and in particular to a multi-AGV path planning method and system based on deep reinforcement learning. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] The rapid development of mobile Internet technology has led to a profound change in e-commerce. The vigorous development of e-commerce has put forward higher requirements for the service level of the logistics industry, especially the warehousing system. While ensuring service quality, enterprises are also faced with the dual challenges of reducing service costs and improving service efficiency.

[0004] Automated Guided Vehicle (AGV) significantly reduces the use of human resources and improves cargo handling efficiency by autonomously performing cargo loading, unloading and handling tasks. AGV is equipped with a high-precision navigation system to ensure that it can move quickly and accurately within the warehouse. Its large-scale use in warehousing systems helps to improve warehouse space utilization. The efficient operation of the AGV system and its seamless integration with advanced logistics management systems provide a reliable and efficient solution for handling tasks in intelligent warehousing systems.

[0005] Path planning is a crucial part of the AGV's handling process, involving a trade-off of multiple objectives, such as driving distance, time, energy consumption, etc. Its accuracy and efficiency directly determine the quality of the handling work in the warehousing system. Therefore, in a complex and changeable warehousing environment, it is of great significance to study how to efficiently plan conflict-free paths for AGVs. At present, a variety of algorithms have been successfully applied to AGV path planning problems, such as A* algorithms, swarm intelligence algorithms, etc., but these methods are inefficient when facing complex environments, and are prone to fall into local optimal solutions, lacking sufficient flexibility and adaptability. Summary of the invention

[0006] In order to address the shortcomings of the prior art, the present invention provides a multi-AGV path planning method and system based on deep reinforcement learning, establishes an MDP model of the AGV path planning problem, constructs a reward function by means of reward reshaping, and guides the AGV to evaluate real-time actions more accurately; establishes an Actor-Critic network for the AGV path planning problem, and uses a synchronous policy update method to update the policy network, which greatly improves the path planning accuracy.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a multi-AGV path planning method based on deep reinforcement learning.

[0009] A multi-AGV path planning method based on deep reinforcement learning includes the following processes:

[0010] Obtaining AGV local environment observation information and path planning related information, extracting spatial features in the local environment observation information according to a convolutional neural network, extracting path features in path planning related information according to a multi-layer perceptron, and combining the spatial features and the path features through a fully connected layer to form an observation code;

[0011] According to the observation coding and the pre-trained deep reinforcement learning algorithm, the execution strategy of each AGV is obtained, so that each AGV performs the corresponding action according to the execution strategy, obtains the reward and the state at the next moment, until the control cycle is reached.

[0012] As a further limitation of the first aspect of the present invention, the pre-trained deep reinforcement learning algorithm includes a policy network and a value network. The policy network generates a probability distribution of possible actions based on the observation coding and processed by a multi-layer perceptron. Each AGV selects the optimal action based on the probability distribution to guide the movement of the AGV in the environment. The value network outputs an estimated value of the current state based on the observation coding and processed by a multi-layer perceptron to guide the policy update of the policy network.

[0013] As a further limitation of the first aspect of the present invention, the policy network shares some parameters with the value network, all AGVs share the same policy network, and all agents make decisions based on the same strategy at each time step.

[0014] As a further limitation of the first aspect of the present invention, the goal of the pre-trained deep reinforcement learning algorithm is to find an optimal policy function so that the AGV obtains the maximum cumulative discounted reward G t : Among them, R t represents the immediate reward at time t, R t+1 represents the immediate reward at time t+1, R t+2 represents the immediate reward at time t+2, R t+k represents the immediate reward at time t+k, γ∈[0,1] is the discount factor, which indicates the influence of future state on current decision.

[0015] As a further limitation of the first aspect of the present invention, the instant reward at any time is the sum of the basic reward and the reshaping reward: R(s,a)=r base (s,a)+r reshape(s,a), where r base (s,a) is the basic reward under state s and action a, r reshape (s,a) is the reshape reward for state s and action a.

[0016] As a further limitation of the first aspect of the present invention, the remodeling reward r reshape (s,a) is the potential reward f(s,a,s ′ ) and curiosity reward r curiosity The sum of (s,a);

[0017] in, is the discount factor, Φ(s) is the potential function of state s, and Φ(s ′ ) is the state s ′ The potential function of

[0018] Among them, λ is the weight of curiosity reward, is the predicted probability of the agent for the next state, P(s ′ |s,a) is the actual observed state transition probability, s ′ is the next state of state s.

[0019] As a further limitation of the first aspect of the present invention, Φ(s)=dist(p current ,p goal )=|x current -x goal |+|y current -y goal |, where p current is the current position of the AGV, p goal is the target position of AGV, x current With y current is the coordinate of the current position of the AGV, x goal With y goal is the coordinate of the target position of the AGV.

[0020] In a second aspect, the present invention provides a multi-AGV path planning system based on deep reinforcement learning.

[0021] A multi-AGV path planning system based on deep reinforcement learning, comprising:

[0022] The observation code generation unit is configured to: obtain the AGV local environment observation information and path planning related information, extract the spatial features in the local environment observation information according to the convolutional neural network, extract the path features in the path planning related information according to the multi-layer perceptron, and combine the spatial features and the path features through a fully connected layer to form an observation code;

[0023] The path planning unit is configured to obtain the execution strategy of each AGV according to the observation coding and the pre-trained deep reinforcement learning algorithm, so that each AGV performs the corresponding action according to the execution strategy, obtains the reward and the state at the next moment, until the control cycle is reached.

[0024] A third aspect provides a computer device, comprising: a processor and a computer-readable storage medium;

[0025] a processor adapted to execute a computer program;

[0026] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the multi-AGV path planning method based on deep reinforcement learning as described in the first aspect of the present invention is implemented.

[0027] In a fourth aspect, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the multi-AGV path planning method based on deep reinforcement learning as described in the first aspect of the present invention.

[0028] In a fifth aspect, the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the multi-AGV path planning method based on deep reinforcement learning as described in the first aspect of the present invention.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention innovatively proposes a multi-AGV path planning strategy based on deep reinforcement learning, establishes an MDP model for the AGV path planning problem, constructs a reward function by means of reward reshaping, and guides the AGV to more accurately evaluate real-time actions; an improved PPO algorithm is proposed, an Actor-Critic network is established for the AGV path planning problem, and the policy network is updated by means of synchronous policy updates; after tests on tasks of different scales, it is proved that the MAPPO algorithm proposed in the present invention has better solution performance than the PPO algorithm in a large-scale warehousing environment; in addition, compared with the traditional path planning method, the path planning method based on deep reinforcement learning has a shorter average path length, fewer inflection points, and higher solution quality.

[0031] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0033] Figure 1 A schematic diagram of the basic framework of reinforcement learning provided in Example 1 of the present invention;

[0034] Figure 2 A schematic diagram of a two-dimensional grid map provided in Example 1 of the present invention;

[0035] Figure 3 A schematic diagram of a Markov decision process provided in Example 1 of the present invention;

[0036] Figure 4 A schematic diagram of the objective function restriction range provided in Example 1 of the present invention;

[0037] Figure 5 A schematic diagram of a solution process based on the MAPPO algorithm provided in Example 1 of the present invention;

[0038] Figure 6 A schematic diagram of an Actor-Critic network provided in Example 1 of the present invention;

[0039] Figure 7 A schematic diagram showing a comparison of success rates and reward values ​​provided in Example 1 of the present invention;

[0040] Figure 8 A schematic diagram of a multi-AGV path planning system based on deep reinforcement learning provided in Example 2 of the present invention;

[0041] Fig. 9 A schematic diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0044] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0045] Embodiment 1:

[0046] Deep Reinforcement Learning (DRL) combines the advantages of deep neural networks and reinforcement learning. It uses neural networks to efficiently extract features from complex environmental states and uses reinforcement learning to guide AGVs to make optimal decisions in warehouse environments. Compared with traditional algorithms, DRL performs well in processing high-dimensional and dynamically changing environments. It can adjust path planning strategies in real time in complex warehouse layouts through end-to-end learning, providing strong support for the efficient operation of AGV systems.

[0047] Reinforcement Learning (RL) is a machine learning method inspired by psychology. Its theoretical basis originates from dynamic programming theory. Its main framework is as follows: Figure 1 As shown. In the reinforcement learning process, the agent interacts with the environment, obtains the state of the system at a specific moment, and selects actions in the current state according to the strategy. After executing the action, the environment will feedback rewards. The agent evaluates the quality of the action based on the feedback rewards and then updates the state. The environment continuously provides new states to the agent, forming a state-action-reward sequence. The goal of reinforcement learning is to optimize the strategy, that is, the mapping function from state to action, through continuous trial and error and learning, to guide the agent to take actions that can obtain the maximum cumulative reward in a specific environment. Reinforcement learning has achieved remarkable results in games, robot control, finance, autonomous driving, and smart homes. Through reinforcement learning, the system can adapt to dynamically changing environments and continuously optimize decisions, thereby achieving highly autonomous and intelligent operations.

[0048] However, in traditional reinforcement learning, the intelligent agent relies too much on manually extracted features and limited state representation, and is only suitable for solving low-dimensional or discrete problems. Deep reinforcement learning combines the advantages of deep learning and reinforcement learning, aiming to address the limitations of traditional reinforcement learning in dealing with high-dimensional problems. By introducing deep learning methods, DRL can automatically extract features in high-dimensional state space and approximate complex value functions or policy functions, thereby achieving effective policy optimization in large-scale and continuous spaces. This end-to-end learning method simplifies the tedious steps of manually extracting and processing features, while improving the adaptability and decision-making capabilities of intelligent agents in complex dynamic environments.

[0049] In the intelligent warehousing system, multiple AGVs shuttle in the warehouse, and realize the "goods to person" picking mode by transporting the shelves to the picking workbench. However, in the actual working environment, due to the large number of AGVs and the limited and narrow number of warehouse lanes, path conflicts may occur when AGVs work together to complete the handling tasks. These conflicts may cause congestion, deadlock and other problems in the system, which will seriously affect the overall handling efficiency. Therefore, planning a set of conflict-free paths for multiple AGVs performing handling tasks is crucial to improving system work efficiency and reducing operating costs.

[0050] The multi-agent path finding (MAPF) problem involves multiple autonomous agents completing tasks independently or collaboratively in a shared environment. The core is to design effective strategies so that all agents can efficiently move from the starting position to the target position without conflicts. The study of multi-agent path planning problems is of great significance to improving the operational efficiency of AGVs and the overall performance of the system. This paper abstracts the AGV path planning problem into a multi-agent path planning problem, and studies the multi-AGV picking path planning problem in the process of intelligent warehouse entry and exit.

[0051] The two-dimensional grid map is an environmental representation method widely used in the field of robot navigation, especially for environmental modeling and path planning. By discretizing the warehouse environment into regular and uniformly sized cells, the two-dimensional grid map simplifies the movement process of the AGV to the operation of moving from one cell to an adjacent cell, thereby reducing the complexity of decision-making and improving the feasibility of the algorithm. Each cell is marked as passable or impassable according to the actual warehouse layout and obstacles (such as shelves, workstations, or walls) to simulate obstacles in the real environment, such as Figure 2 shown.

[0052] Markov Decision Process (MDP) provides a basic mathematical framework for describing the interaction between the environment and the agent. Figure 3As shown. MDP is mainly composed of a state set (State Space, S), an action set (Action Space, A), a state transition probability (State Transition Probability, P), a reward function (Reward Function, R) and a discount factor γ. MDP has obvious Markov properties, that is, the future state is independent of the past state and action, and is only related to the current state and current action. Its basic goal is to find a strategy π that enables the agent to obtain the maximum cumulative reward in the long term. This implementation modeled the multi-AGV path planning problem as an MDP. The Markov decision process of n AGVs consists of five tuples:<S,A,P,R,γ> express.

[0053] S represents the state space, including the location information of each AGV, the target location, and the location of obstacles in the environment. For the i-th AGV, its state s i It can be expressed as:

[0054] s i ={p i , g i , o1, o2, …, o m} (1)

[0055] Among them, p i is the current position of the i-th AGV, g i is its target position, {o1,o2,...,o m} represents the position of m obstacles in the environment. Therefore, the state space S of the entire system can be expressed as a combination of all AGV states:

[0056] S={(s1,s2,…,s n )|s i ={p i ,g i ,o1,o2,…,o m},i=1,2,...,n} (2);

[0057] A represents the action space, that is, all possible actions that the AGV can perform in each state. For the i-th AGV, in state s i Next, its action a i It can be expressed as:

[0058]

[0059] in, is the set of actions that the i-th AGV can choose in the current state. The action space A of the entire system is the combination of all AGV actions:

[0060]

[0061] State transition probability P(s ′ |s,a) is a conditional probability indicating that given the current state s and action a, the system will transition to the next state s ′ Possibilities:

[0062]

[0063] Among them, S t and S t+1 Represent the state at the current time t and the next time t+1 respectively, A t represents the action performed at time t.

[0064] The design of the reward function directly affects the learning direction and final performance of the agent, so it is crucial in deep reinforcement learning. In traditional reinforcement learning methods, sparse rewards are often used to guide the actions of the agent. However, since the agent lacks immediate reward feedback in most cases, it is difficult for it to accurately evaluate the effect of the actions taken. The present invention designs the reward function by reshaping the reward, and guides the agent to more accurately evaluate real-time actions by adding an intrinsic reward mechanism.

[0065] The essence of the AGV path planning problem is how to find the shortest path for each AGV to reach its destination while avoiding collisions. Therefore, the design of the reward function should follow the following three criteria: avoiding collisions, optimizing the path, and reaching the destination. The reward function is divided into two parts: the basic reward function and the reward reshaping strategy.

[0066] First, define the basic reward function to guide the basic behavior of AGV. The basic reward function includes the global reward r w (s,a) and the single-step reward r s (s,a) two parts:

[0067] r base (s,a)=r w (s,a)+r s (s,a) (6);

[0068] Global Rewards w (s,a) includes the task completion reward r f (s,a) and collision reward r c (s,a) two parts:

[0069] r w (s,a)=r f (s,a)+r c (s,a) (7);

[0070] Among them, the task completion reward is r f (s,a) means that when the AGV reaches the end point, a large positive reward is given and the current round ends:

[0071]

[0072] The collision reward means that when the AGV collides with an obstacle, a negative reward is given and the round ends:

[0073]

[0074] Single step reward r s (s,a) includes the guiding reward r g (s,a), turn penalty r t (s,a) and the step penalty r d (s,a):

[0075] r s (s,a)=r g (s,a)+r t (s,a)+r d (s,a) (10);

[0076] Among them, the guidance reward provides positive rewards for each step the AGV takes to approach the correct target position:

[0077] r g (s,a)=-α·dist(p current , p goal ) (11);

[0078] The turning penalty gives the AGV a slight negative reward every time it turns, encouraging the AGV to choose a straighter path and reduce unnecessary turns:

[0079]

[0080] The step penalty gives a slight negative reward for each step execution, encouraging the AGV to complete the task in fewer steps:

[0081] r d (s,a)=-δ (13).

[0082] In order to further optimize the learning process of AGV, in addition to the basic reward function, this paper introduces a reward reshaping strategy r reshape (s,a), by adding potential reward mechanism and curiosity mechanism, the agent's exploration ability and strategy optimization in the learning process are enhanced:

[0083] r reshape (s,a)=f(s,a,s′)+rcuriosity (s,a) (14).

[0084] The potential reward function guides the AGV to move faster towards the target direction by increasing the value of favorable states:

[0085]

[0086] in, is the discount factor, Φ(s) is the potential function of state s, and the role of the potential reward function is to increase the attractiveness of favorable states, making AGV more inclined to choose a path that can approach the target. Manhattan distance is used as a heuristic function to calculate the current AGV position pcurren t The shortest path distance to the target position pgoal. The calculation formula of Φ(s) is:

[0087] Φ(s)=dist(p current , p goal )=|x current -x goal |+|y current -y goal | (16);

[0088] In order to motivate AGV to explore unknown areas and underutilized paths in the environment, this paper introduces a curiosity mechanism as part of the intrinsic reward. The basic idea of ​​the curiosity mechanism is to reward AGV for exploring states with large prediction errors, thereby guiding it to discover new paths and strategies during the learning process.

[0089] Curiosity Rewards curiosity (s,a) Based on the prediction error of the agent on the environment. When the behavior result of the AGV in a certain state does not match its expectations, additional rewards are given to encourage it to continue exploring:

[0090]

[0091] Among them, λ is the weight of curiosity reward, is the agent's predicted probability of the next state, and P(s′|s,a) is the actual observed state transition probability. The larger the prediction error, the higher the agent's curiosity reward, which encourages it to continue exploring this state space.

[0092] By integrating the basic reward function and the reward reshaping mechanism, the reward R(s,a) obtained by the AGV in a certain state s is as follows:

[0093] R(s,a)=r base (s,a)+r reshape (s,a) (18);

[0094] The goal of deep reinforcement learning is to find an optimal policy function that enables AGV to obtain the maximum cumulative reward G. t :

[0095]

[0096] Among them, R t represents the immediate reward at time t, G t represents the cumulative discounted reward starting at time t.

[0097] γ∈[0,1] is a discount factor, which indicates the influence of future states on current decisions.

[0098] The Proximal Policy Optimization (PPO) algorithm is a deep reinforcement learning algorithm based on Policy Gradient (PG). Compared with the traditional PG method, the PPO algorithm adopts the Actor-Critic network architecture, which retains the advantages of the PG algorithm in continuous state space and continuous action space. In addition, PPO introduces a truncation mechanism to improve the stability and efficiency of the training process by limiting the amplitude of policy updates. Its objective function is as follows:

[0099]

[0100] Among them, θ is the optimizable strategy parameter, It is an advantage function, which is used to measure the advantage of the current action relative to the average action, that is, the degree of goodness of a certain action in a certain state.

[0101] r t (θ) is the importance sampling ratio, which represents the probability ratio of the new strategy to the old strategy. It is used to adjust the expectations of the new and old strategies so that the updated strategy does not deviate too much from the original strategy:

[0102]

[0103] clip(r t (θ),1-∈,1+∈) is the clipping function. t (θ) is limited to the interval [1-∈, 1+∈] to prevent the strategy from being updated too much. ∈ is a small positive number used to control the update amplitude of the strategy. In this invention, it is taken as 0.2. The objective function limit range is as follows: Figure 4 shown.

[0104] In order to solve the path conflict and improve the task success rate, this paper uses an improved algorithm based on PPO - Multi-Agent PPO (MAPPO), referring to the Actor-Critic architecture of the basic PPO algorithm, and makes corresponding extensions and improvements for the AGV path planning problem. The solution process based on the MAPPO algorithm is as follows: Figure 5 shown.

[0105] The input of the AGV includes observation information of the local environment and features related to path planning. Convolutional Neural Network (CNN) and Multi-layer Perceptron (MLP) layers are used to encode these input information respectively.

[0106] CNN is used to process the observable local two-dimensional environment of the AGV and extract spatial features, including surrounding obstacles, the positions of other AGVs, etc. MLP is used to encode the path target vector of the AGV (the direction vector to the target position) and other path-related features. The outputs of these layers are combined through the fully connected layer (Concatenation Layer) to form a unified observation code.

[0107] The Actor-Critic network structure of MAPPO is as follows: Figure 6 As shown in the figure, it contains two main network components: the policy network (Actor) and the value network (Critic). The policy network receives the combined observation encoding, and after further processing by the MLP layer, it generates the probability distribution of each possible action. Each AGV selects the optimal action based on the probability distribution to guide its movement in the environment; the value network also receives the combined observation encoding, and after processing by the MLP layer, it outputs the estimated value of the current state, thereby guiding the update of the policy. In the MAPPO algorithm, these two networks share some parameters to improve sample efficiency and reduce computational overhead.

[0108] A key feature of MAPPO is synchronous policy update, that is, all AGVs share the same policy network. At each time step, all agents make decisions based on the same policy. In this way, the policies of all AGVs remain consistent during training, avoiding the instability caused by asynchronous updates. This synchronous update mechanism enables MAPPO to show higher stability and efficiency when dealing with complex dynamic environments, and can effectively improve the sample utilization of multi-agent systems.

[0109] The experiment of the present invention is based on the Windows 11 operating system. The training process uses the i9-13900KF CPU and the NVIDIA GeForce RTX4090 GPU. The program is developed using Python 3.8, and the reinforcement learning environment is built based on OpenAI gym 0.21.0.

[0110] To facilitate analysis and research, the following assumptions are made:

[0111] (1) The models and parameters of AGVs are the same;

[0112] (2) Each piece of cargo is handled by only one AGV;

[0113] (3) The turning cost of the AGV is greater than the straight-moving cost;

[0114] (4) The acceleration and deceleration process of AGV is not considered;

[0115] (5) Without considering order priority, all handling tasks are randomly generated and assigned.

[0116] The key parameter settings in the reward function are shown in Table 1.

[0117] Table 1: Key parameters of AGV reward function.

[0118]

[0119] The parameter settings of the MAPPO algorithm training process are shown in Table 2.

[0120] Table 2: MAPPO algorithm training process parameters.

[0121]

[0122] The conflict-based search (CBS) algorithm was selected as a comparison among the traditional path planning algorithms, and the experiment was conducted with the trained PPO algorithm and MAPPO algorithm in the same map environment. The map size was 30×30, the number of AGVs was 20, the number of tasks was 50, and the proportion of inaccessible areas in the scene was 50%. The average path length and average number of inflection points of different algorithms were compared, and the experimental results are shown in Table 3.

[0123] Table 3: Comparison of solution quality of different algorithms.

[0124]

[0125] As shown in Table 3, all three algorithms can successfully complete the AGV path planning task on a map of a given scale. The average path lengths solved by the PPO algorithm and the MAPPO algorithm are better than those of the CBS algorithm. The main reason is that the planning goal of the CBS algorithm is the single optimal path, while the optimization goal of the PPO algorithm and the MAPPO algorithm is to maximize the overall reward value. In addition, due to the synchronous strategy update mechanism, the average path length solved by the MAPPO algorithm is slightly lower than that of the PPO algorithm. As for the number of inflection points, due to the introduction of the reward reshaping mechanism, the number of inflection points of the PPO algorithm and the MAPPO algorithm is smaller than that of the CBS algorithm.

[0126] Compare the performance of the MAPPO algorithm and the PPO algorithm. The test was conducted on a 30x30 2D grid map, with 20 AGVs, 50 missions, and 50% of the inaccessible areas in the scene. The success rate and average reward of the PPO and MAPPO algorithms under different training generations were compared. The experimental results are shown in Figure 2. Figure 7 shown.

[0127] Depend on Figure 7 It can be seen that when the training generation is less than 4000 rounds, the average success rate and average cumulative reward of PPO and MAPPO are similar; as the training generation continues to increase, the proposed MAPPO algorithm is generally superior to the PPO algorithm in terms of success rate and average reward value.

[0128] Comparative experiments between the MAPPO algorithm and the PPO algorithm were carried out under different numbers of tasks and AGVs to test the performance of the PPO and MAPPO algorithms. The experimental results are shown in Table 4.

[0129] Table 4: Algorithm performance test under different maps and task scales.

[0130]

[0131]

[0132] It can be seen from the data in Table 4 that the success rate of the MAPPO algorithm proposed in the present invention is better than that of the PPO algorithm in six groups of experiments of different scales. In the first two groups of experiments, the success rate of the two algorithms was 1.0, which does not mean that the performance of the two algorithms is the same. The main reason is that the problem scale is small and the solution space is not large. In addition, for experiments of different scales, the MAPPO algorithm is also better than the PPO algorithm in terms of the average reward. From the above analysis, it can be seen that the proposed MAPPO algorithm is effective after verification by experiments of different scales.

[0133] Embodiment 2:

[0134] like Figure 8 As shown, this implementation provides a multi-AGV path planning system based on deep reinforcement learning, including:

[0135] The observation code generation unit is configured to: obtain the AGV local environment observation information and path planning related information, extract the spatial features in the local environment observation information according to the convolutional neural network, extract the path features in the path planning related information according to the multi-layer perceptron, and combine the spatial features and the path features through a fully connected layer to form an observation code;

[0136] The path planning unit is configured to obtain the execution strategy of each AGV according to the observation coding and the pre-trained deep reinforcement learning algorithm, so that each AGV performs the corresponding action according to the execution strategy, obtains the reward and the state at the next moment, until the control cycle is reached.

[0137] The specific working methods of each unit of the above-mentioned multi-AGV path planning system are described in Example 1 and will not be repeated here.

[0138] It is understandable that the above-mentioned units can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the multi-AGV path planning system can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0139] According to another embodiment of the present application, the system described in this embodiment can be constructed and the multi-AGV path planning method of the embodiment of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general-purpose computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0140] Embodiment 3:

[0141] like Fig. 9 As shown, this implementation provides an electronic device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.

[0142] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0143] Processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0144] The processor 1001 is configured to execute the following process:

[0145] Obtaining AGV local environment observation information and path planning related information, extracting spatial features in the local environment observation information according to a convolutional neural network, extracting path features in path planning related information according to a multi-layer perceptron, and combining the spatial features and the path features through a fully connected layer to form an observation code;

[0146] According to the observation coding and the pre-trained deep reinforcement learning algorithm, the execution strategy of each AGV is obtained, so that each AGV performs the corresponding action according to the execution strategy, obtains the reward and the state at the next moment, until the control cycle is reached.

[0147] The specific working method is described in Example 1 and will not be repeated here.

[0148] Embodiment 4:

[0149] This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0150] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0151] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:

[0152] Obtaining AGV local environment observation information and path planning related information, extracting spatial features in the local environment observation information according to a convolutional neural network, extracting path features in path planning related information according to a multi-layer perceptron, and combining the spatial features and the path features through a fully connected layer to form an observation code;

[0153] According to the observation coding and the pre-trained deep reinforcement learning algorithm, the execution strategy of each AGV is obtained, so that each AGV performs the corresponding action according to the execution strategy, obtains the reward and the state at the next moment, until the control cycle is reached.

[0154] The specific working method is described in Example 1 and will not be repeated here.

[0155] Embodiment 5:

[0156] The present implementation provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the following process:

[0157] Obtaining AGV local environment observation information and path planning related information, extracting spatial features in the local environment observation information according to a convolutional neural network, extracting path features in path planning related information according to a multi-layer perceptron, and combining the spatial features and the path features through a fully connected layer to form an observation code;

[0158] According to the observation coding and the pre-trained deep reinforcement learning algorithm, the execution strategy of each AGV is obtained, so that each AGV performs the corresponding action according to the execution strategy, obtains the reward and the state at the next moment, until the control cycle is reached.

[0159] The specific working method is described in Example 1 and will not be repeated here.

[0160] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instruction can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that contains one or more available media integration. Available media can be magnetic media (for example, floppy disk, hard disk, tape), optical media (for example, DVD), or semiconductor media (for example, solid-state drive (Solid State Disk, SSD)) and the like.

[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-AGV path planning method based on deep reinforcement learning, characterized in that: The process includes: Obtaining AGV local environment observation information and path planning related information, extracting spatial features in the local environment observation information according to a convolutional neural network, extracting path features in path planning related information according to a multi-layer perceptron, and combining the spatial features and the path features through a fully connected layer to form an observation code; According to the observation coding and the pre-trained deep reinforcement learning algorithm, the execution strategy of each AGV is obtained, so that each AGV performs the corresponding action according to the execution strategy, obtains the reward and the state at the next moment, until the control cycle is reached.

2. The multi-AGV path planning method based on deep reinforcement learning as claimed in claim 1, characterized in that: The pre-trained deep reinforcement learning algorithm includes a policy network and a value network. The policy network generates a probability distribution of possible actions based on the observation encoding and after processing by a multi-layer perceptron. Each AGV selects the optimal action based on the probability distribution to guide the movement of the AGV in the environment; the value network outputs an estimated value of the current state based on the observation encoding and after processing by a multi-layer perceptron, which is used to guide the policy update of the policy network.

3. The multi-AGV path planning method based on deep reinforcement learning as claimed in claim 2, characterized in that: The strategy network shares some parameters with the value network, all AGVs share the same strategy network, and all agents make decisions based on the same strategy at each time step; The goal of the pre-trained deep reinforcement learning algorithm is to find an optimal policy function that enables the AGV to obtain the maximum cumulative discounted reward G t : Among them, R t represents the immediate reward at time t, R t+1 represents the immediate reward at time t+1, R t+2 represents the immediate reward at time t+2, R t+k represents the immediate reward at time t+k, γ∈[0,1] is the discount factor, which indicates the influence of future state on current decision.

4. The multi-AGV path planning method based on deep reinforcement learning as claimed in claim 3, characterized in that: The instant reward at any time is the sum of the basic reward and the reshaping reward: R(s,a)=r base (s,a)+r reshape (s,a), where r base (s,a) is the basic reward under state s and action a, r reshape (s,a) is the reshape reward for state s and action a.

5. The multi-AGV path planning method based on deep reinforcement learning as claimed in claim 4, characterized in that: The remodeling reward reshape (s,a) is the potential reward f(s,a,s ′ ) and curiosity reward r curiosity The sum of (s,a); in, is the discount factor, Φ(s) is the potential function of state s; Among them, λ is the weight of curiosity reward, is the predicted probability of the agent for the next state, P(s ′ |s,a) is the actual observed state transition probability, s ′ is the next state of state s.

6. The multi-AGV path planning method based on deep reinforcement learning as claimed in claim 5, characterized in that: Φ(s) = dist(p current , p goal ) = |x current - x goal | + |y current - y goal |, where p current is the current position of the AGV, p goal is the target position of AGV, x current With y current is the coordinate of the current position of the AGV, x goal With y goal is the coordinate of the target position of the AGV.

7. A multi-AGV path planning system based on deep reinforcement learning, characterized in that: include: The observation code generation unit is configured to: obtain the AGV local environment observation information and path planning related information, extract the spatial features in the local environment observation information according to the convolutional neural network, extract the path features in the path planning related information according to the multi-layer perceptron, and combine the spatial features and the path features through a fully connected layer to form an observation code; The path planning unit is configured to obtain the execution strategy of each AGV according to the observation coding and the pre-trained deep reinforcement learning algorithm, so that each AGV performs the corresponding action according to the execution strategy, obtains the reward and the state at the next moment, until the control cycle is reached.

8. A computer device, characterized in that: include: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the multi-AGV path planning method based on deep reinforcement learning as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the multi-AGV path planning method based on deep reinforcement learning as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the multi-AGV path planning method based on deep reinforcement learning as described in any one of claims 1 to 6.

Images