A dynamic path planning method for AGV based on distributed reinforcement learning

Abstract

The present invention discloses a method for dynamic path planning of AGV based on distributed reinforcement learning, comprising the following steps: S1, building a simulated intelligent factory, and setting an intelligent agent with computing, storage and communication functions at each path selection node in the simulated intelligent factory; S2, setting the parameters of the intelligent agent and the parameters of the AGV running on the path; S3, designing the state, action, reward, and policy in reinforcement learning, and S4, training the intelligent agent. Compared with the centralized algorithm, when the number of AGVs is controlled within a certain range, the performance is almost the same as the centralized effect, but the number of calculations of the present invention is significantly lower than that of the centralized one, in which we assume that each calculation of the Q value, i.e., Q[(d,i),j],#imgabs0#, during the simulation process is counted as one calculation. From the results, it can be seen that the distributed algorithm proposed by the present invention achieves similar effects at a lower cost.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an AGV dynamic path planning method based on distributed reinforcement learning. Background Art

[0002] In recent years, the country has launched the Manufacturing 2025 plan, which uses information technology and artificial intelligence technology to transform and upgrade the manufacturing industry. As a representative of technology integration and technological innovation, smart workshops are the main development direction of the manufacturing industry in the new era. Among them, smart workshops include advanced automated processing equipment and automated logistics storage and transportation systems with multiple automatic navigation vehicles (Automatic Guidance Vehicle, AGV). AGV has the characteristics of high flexibility, strong scalability, and easy maintenance, and is widely used in smart workshops. However, the AGV path planning method of traditional workshops is no longer applicable to smart workshops. When facing more complex production environments, a more flexible AGV path planning method is needed.

[0003] Path planning has always been an important part of the AGV technology center. If you want to further improve the production capacity of the workshop, first of all, the production efficiency of the equipment is a very important part, and secondly, the number of AGVs also has a great impact. Another part that cannot be ignored is the efficiency of AGV transportation. Only by reasonably planning the AGV transportation path and reducing road congestion can the production equipment be kept in working condition, thereby improving efficiency.

[0004] Path planning can be divided into global path planning and local path planning according to the degree of understanding of environmental information, and can be divided into static path planning and dynamic path planning according to needs. In more complex dynamic path planning problems, AGV needs to face more complex uncertainties. Traditional algorithms such as artificial potential field method and A* algorithm are easy to fall into local optimality or fail to plan a path.

[0005] To address the above issues, most current smart workshops adopt centralized computing, which is easy to deploy and simple to operate. However, the disadvantages of centralized computing are also obvious. For the central computing unit, only one CPU is needed to deal with massive amounts of computing data, which not only places very high requirements on the CPU performance, but also makes it very expensive.

[0006] Large-scale smart factories will use distributed networks as a better choice. Each machine in the distributed network can store and process data, which reduces the requirements for CPU performance, so there is no need to purchase expensive high-performance machines, greatly reducing hardware investment costs. Summary of the invention

[0007] In view of the shortcomings of traditional path planning algorithms and centralized computing methods, the present invention proposes an AGV dynamic path planning method based on distributed reinforcement learning, which uses a distributed network to store and process data, reduces the requirements on CPU performance, and greatly reduces hardware investment costs.

[0008] In order to solve the above technical problems, the technical solution of the present invention is:

[0009] A distributed reinforcement learning-based AGV dynamic path planning method comprises the following steps:

[0010] S1. Build a simulated smart factory and set up an intelligent agent with computing, storage and communication functions at each path selection node in the simulated smart factory;

[0011] S2. Set the parameters of the agent and the AGV running on the path

[0012] All intelligent agents are represented by set I, and AGVs are represented by set M. Assume that there are n intelligent agents and m AGVs.

[0013] For a certain agent, represented by i, i∈I, the agent i stores Q i 、E i , δ i , Where Q i represents the Q table stored in agent i. The horizontal axis of the Q table represents the state of the node, and the vertical axis represents the action of the node. i represents the qualification trace table stored in agent i, the format is consistent with Q, δ i Represents the TD-error saved in agent i, which is a vector with a length of |δ i |=m, It represents the static time for agent i to reach the adjacent node j, that is, the shortest time from node i to node j when there is no congestion on the track.

[0014] S3. Design the state, action, reward, and policy in reinforcement learning.

[0015] Among them, state is represented by the symbol (d,i), where d represents the target node and i represents the current node.

[0016] Action is represented by the symbol A(d,i), which represents the set of adjacent nodes that the current node i can reach.

[0017] Reward is represented by the symbol R[(d,i),j], and the reward function is R[(d,i),j]=t(i,j), where t(i,j) represents the actual driving time of the AGV from the current node i to the node j.

[0018] Each element in the Q table can be used express;

[0019] S4. Train the agent.

[0020] Preferably, the simulated intelligent factory is constructed using AnyLogic simulation software.

[0021] As a preferred embodiment, the parameter setting of the AGV is as follows: the nodes reached each time are stored in the AGV, and then the model established for the network is:

[0022] G(N,ε)={g1(N1,ε1),...,g m (N m ,ε m )}

[0023] Where G represents the graph, N represents the node set, ε represents the edge set, and g m Represents the graph of the mth AGV, N m represents the set of nodes that the mth AGV has passed through, ε m It represents the set of edges that the mth AGV has walked through. G is composed of many subgraphs, and the subgraphs are formed by the trajectories that the AGV has walked through.

[0024] Preferably, the policy in step S3 is:

[0025] The policy uses Boltzmann softmax, which is expressed as follows: Boltzmann softmax

[0026]

[0027]

[0028] Boltzmann(i) means that agent i uses the Boltzmann softmax policy to sample an action based on the calculated probability distribution.

[0029] Preferably, step S4 includes the following sub-steps:

[0030] S4-1. Initialize the Q stored in each agent i 、E i ,δ i ;

[0031] S4-2, when AGV (m, m∈M) arrives at node h, if h is the starting node, execute step S4-3, if h is the end point, execute step S4-4, if h is neither the starting node nor the end point, execute step S4-5;

[0032] S4-3, δ h (m) = -1000, node h selects k' through the Boltzmann softmax policy, controls m to go to k', N m ∪{h}, repeat step S4-2 until AGV (m, m∈M) delivers the goods to the destination;

[0033] S4-4, AGV (m, m∈M) completes the task, goes to the starting node of the next task, and repeats step S4-2;

[0034] S4-5. Control AGV (m, m∈M) to go to k' and repeat step S4-2 until AGV (m, m∈M) delivers the goods to the destination.

[0035] Preferably, in step S4-1, the initialization method is:

[0036]

[0037]

[0038] Until ε is a sufficiently small number

[0039]

[0040]

[0041] Preferably, the specific method of step S4-4 is:

[0042] When AGV numbered m arrives at node h, it adds node h and edge (w,h) to topology graph g. m In, N m ∪{h},ε m ∪{(w,h)},w∈n m,h , n m,h In Figure g m The neighbors of node h,

[0043] Agent h sets the TD-error value corresponding to the mth AGV to a sufficiently small number, thereby clearing the historical record.

[0044] δ h (m)=-1000

[0045] All in the topology diagramm The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform min operation after receiving it, and then pass the -1000 set above to all neighbors in g m The intelligent agent in the system can achieve the purpose of clearing historical data.

[0046]

[0047] The actual driving time of the AGV from node w to node h is used as a reward.

[0048] R[(d,w),h]=t(w,h),w∈n m,h

[0049] Agent h calculates TD-error according to the TD-error calculation formula. Node h is the end point, and TD-error is consistent with the reward.

[0050] δ h (m)=R[(d,w),h]

[0051] All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform the max operation after receiving it, thus passing the newly calculated TD-error calculated above to all neighbors in g m The intelligent agents in the network can transfer TD-error between intelligent agents.

[0052]

[0053] In the topology diagram g m The agent in the eligibility trace updates the eligibility trace according to the eligibility trace formula rules

[0054] E l [(d,l),j]=E l [(d,l),j]*γλ, where and

[0055] E w [(d,w),h]=E w [(d,w),h]+1,w∈N m And w∈n m,l

[0056] In the topology diagram g m The agent in the update Q value according to the Q value update formula

[0057]

[0058] N m =φ,ε m =φ

[0059] The AGV (m, m∈M) task is completed, and it goes to the starting node of the next task and repeats step S4-2.

[0060] Preferably, the specific method of step S4-5 is:

[0061] When AGV numbered m arrives at node h, it adds node h and edge (w,h) to topology graph g. m middle,

[0062] N m ∪{h},ε m ∪{(w,h)},w∈n m,h

[0063] Agent h sets the TD-error value corresponding to the mth AGV to a sufficiently small number, thereby clearing the historical record.

[0064] δ h (m)=-1000

[0065] All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform min operation after receiving it, and then pass the -1000 set above to all neighbors in g m The intelligent agent in the system can achieve the purpose of clearing historical data.

[0066]

[0067] According to the Q table saved in the agent, greedily take the action k with the smallest Q value *

[0068] k * =argmin k∈A(d,h) Q h [(d,h),k]

[0069] Agent h calculates action k' based on Boltzmann softmax

[0070] k'=Boltzmann(h)

[0071] R[(d,w),h]=t(w,h),w∈n m,h

[0072] Agent h calculates TD-error according to the TD-error calculation formula

[0073] δ h (m)=R[(d,w),h]+γ*Q h [(d,h),k * ]-Q w [(d,w),h]

[0074] All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform the max operation after receiving it, thus passing the newly calculated TD-error calculated above to all neighbors in g m The intelligent agents in the network can transfer TD-error between intelligent agents.

[0075]

[0076] In the topology diagram g m The agent in the eligibility trace updates the eligibility trace according to the eligibility trace formula rules

[0077] E l [(d,l),j]=E l [(d,l),j]*γλ, where and

[0078] E w [(d,w),h]=E w [(d,w),h]+1,w∈N m And w∈n m,l

[0079] In the topology diagram g m The agent in the update Q value according to the Q value update formula

[0080]

[0081] If the action k obtained * If it is inconsistent with the action k' calculated by Boltzmann softmax, the historical trajectory saved in the mth AGV is cleared

[0082] If k * ≠k', then execute the following (1)-(3):

[0083] and

[0084] (2)N m =φ,ε m =φ

[0085] (3)N m ∪{h}

[0086] Control AGV (m, m∈M) to go to k' and repeat step S4-2 until AGV (m, m∈M) delivers the goods to the destination.

[0087] The present invention has the following characteristics and beneficial effects:

[0088] A distributed reinforcement learning method is proposed to dynamically plan the path of AGV. Compared with the centralized algorithm, when the number of AGVs is controlled within a certain range, the performance is almost the same as the centralized effect, but the number of calculations of the invention is significantly lower than that of the centralized method. We assume that each Q value is calculated during the simulation process. It is counted as one calculation. From the results, it can be seen that the distributed algorithm proposed in the present invention achieves a similar effect at a lower cost.

[0089] It does not pursue the Q-table convergence of the intelligent agent, because the environment is dynamically changing and can adapt to more complex environments. Compared with the general reinforcement learning algorithm that pursues the Q-table convergence, it has stronger adaptability to the environment and less computation.

[0090] Compared with traditional algorithms that will not fall into local optimality and use traditional algorithms to plan AGV paths according to the shortest path in a fixed AGV track environment, the number of AGVs is limited. The present invention can increase the number of AGVs by alleviating congestion and thus transport more goods. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0092] Figure 1 This is a track model diagram of a simulated smart factory modeled using AnyLogic in an embodiment of the present invention.

[0093] Figure 2 This is a relationship diagram of the total amount of goods transported by the number of AGVs within 36,000 seconds when the shortest path algorithm is used in an embodiment of the present invention.

[0094] Figure 3 This is a relationship diagram of the total amount of goods transported by the number of AGVs within 36,000 seconds when the shortest path algorithm, centralized algorithm, and distributed algorithm are used in an embodiment of the present invention.

[0095] Figure 4This is a diagram of the calculation times of the centralized algorithm and the distributed algorithm within 36000 seconds when there are 100 AGVs in an embodiment of the present invention. DETAILED DESCRIPTION

[0096] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0097] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0098] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0099] The present invention provides an AGV dynamic path planning method based on distributed reinforcement learning, comprising the following steps:

[0100] S1. Build a simulated smart factory and set up an intelligent agent with computing, storage and communication functions at each path selection node in the simulated smart factory.

[0101] Among them, the simulated smart factory was built using AnyLogic simulation software.

[0102] Specifically, in this embodiment, a simulated intelligent factory is built using AnyLogic simulation software. Figure 1As shown in the figure, the track model diagram of the simulated smart factory built using the AnyLogic simulation platform has physical meaning. The simulated smart factory corresponds to a rectangle with a length of 120 meters, a width of 25 meters, and an area of ​​3,000 square meters. Figure 1 The nodes s1-s5, n1-n28, and d1-d5 are regarded as intelligent entities with computing, storage, and communication capabilities.

[0103] S2. Set the parameters of the agent and the AGV running on the path

[0104] All intelligent agents are represented by set I, and AGVs are represented by set M. Assume that there are n intelligent agents and m AGVs.

[0105] For a certain agent, represented by i, i∈I, the agent i stores Q i 、E i , δ i , Where Q i represents the Q table stored in agent i. The horizontal axis of the Q table represents the state of the node, and the vertical axis represents the action of the node. i represents the qualification trace table stored in agent i, the format is consistent with Q, δ i represents the TD-error stored in agent i, which is a vector |δ i |=n, It represents the static time for agent i to reach the adjacent node j, that is, the shortest time from node i to node j when there is no congestion on the track.

[0106] Furthermore, the parameter setting of the AGV is as follows: the nodes reached each time are stored in the AGV, and then the model of the network is established:

[0107] G(N,ε)={g1(N1,ε1),...,g m (N m ,ε m )}

[0108] Where G represents the graph, N represents the node set, ε represents the edge set, and g m Represents the graph of the mth AGV, N m represents the set of nodes that the mth AGV has passed through, ε m It represents the set of edges that the mth AGV has walked through. G is composed of many subgraphs, and the subgraphs are formed by the trajectories that the AGV has walked through.

[0109] S3, reinforcement learning of the event selection of the agent and AGV, where the event includes state, action, reward, and policy.

[0110] Among them, state is represented by the symbol (d,i), where d represents the target node and i represents the current node.

[0111] Action is represented by the symbol A(d,i), which represents the set of adjacent nodes that the current node i can reach.

[0112] Reward is represented by the symbol R[(d,i),j], and the reward function is R[(d,i),j]=t(i,j), where t(i,j) represents the actual driving time of the AGV from the current node i to the node j.

[0113] Each element in the Q table can be used express;

[0114] The reinforcement learning method in step S3 is:

[0115] The policy uses Boltzmann softmax, which is expressed as follows: Boltzmann softmax

[0116]

[0117]

[0118] Boltzmann(i) means that agent i uses the Boltzmann softmax policy to sample an action based on the calculated probability distribution.

[0119] S4. Training the Agent

[0120] Specifically, the following sub-steps are included:

[0121] S4-1. Initialize the Q stored in each agent i 、E i , δ i ,

[0122]

[0123]

[0124] Until Among them, ε is a sufficiently small number,

[0125]

[0126]

[0127] S4-2, when AGV (m, m∈M) arrives at node h, if h is the starting node, execute step S4-3, if h is the end point, execute step S4-4, if h is neither the starting node nor the end point, execute step S4-5;

[0128] S4-3, δ h (m) = -1000, node h selects k' through the Boltzmann softmax policy, controls m to go to k', N m ∪{h}, repeat step S4-2 until AGV (m, m∈M) delivers the goods to the destination;

[0129] S4-4. When AGV numbered m arrives at node h, it adds node h and edge (w, h) to topology graph g. m middle,

[0130] N m ∪{h},ε m ∪{(w,h)},w∈n m,h , n m,h In Figure g m The neighbors of node h,

[0131] Agent h sets the TD-error value corresponding to the mth AGV to a sufficiently small number, thereby clearing the historical record.

[0132] δ h (m)=-1000

[0133] All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform min operation after receiving it, and then pass the -1000 set above to all neighbors in g m The intelligent agent in the system can achieve the purpose of clearing historical data.

[0134]

[0135] The actual driving time of the AGV from node w to node h is used as a reward.

[0136] R[(d,w),h]=t(w,h),w∈n m,h

[0137] Agent h calculates TD-error according to the TD-error calculation formula. Node h is the end point, and TD-error is consistent with the reward.

[0138] δ h (m)=R[(d,w),h]

[0139] All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform the max operation after receiving it, thus passing the newly calculated TD-error calculated above to all neighbors in g m The intelligent agents in the network can transfer TD-error between intelligent agents.

[0140]

[0141] In the topology diagram g m The agent in the eligibility trace updates the eligibility trace according to the eligibility trace formula rules

[0142] E l [(d,l),j]=E l [(d,l),j]*γλ, where and

[0143] E w [(d,w),h]=E w [(d,w),h]+1,w∈N m And w∈n m,l

[0144] In the topology diagram g m The agent in the update Q value according to the Q value update formula

[0145]

[0146] N m =φ,ε m =φ

[0147] The AGV (m, m∈M) task is completed, and it goes to the starting node of the next task and repeats step S4-2.

[0148] S4-5. When AGV numbered m arrives at node h, it adds node h and edge (w, h) to topology graph g. m middle,

[0149] N m ∪{h},ε m ∪{(w,h)},w∈n m,h

[0150] Agent h sets the TD-error value corresponding to the mth AGV to a sufficiently small number, thereby clearing the historical record.

[0151] δ h (m)=-1000

[0152] All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform min operation after receiving it, and then pass the -1000 set above to all neighbors in g m The intelligent agent in the system can achieve the purpose of clearing historical data.

[0153]

[0154] According to the Q table saved in the agent, greedily take the action k with the smallest Q value *

[0155] k * =argmin k∈A(d,h) Q h [(d,h),k]

[0156] Agent h calculates action k' based on Boltzmann softmax

[0157] k'=Boltzmann(h)

[0158] R[(d,w),h]=t(w,h),w∈n m,h

[0159] Agent h calculates TD-error according to the TD-error calculation formula

[0160] δ h (m)=R[(d,w),h]+γ*Q h [(d,h),k * ]-Q w [(d,w),h]

[0161] All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform the max operation after receiving it, thus passing the newly calculated TD-error calculated above to all neighbors in g m The intelligent agents in the network can transfer TD-error between intelligent agents.

[0162]

[0163] In the topology diagram g m The agent in the eligibility trace updates the eligibility trace according to the eligibility trace formula rules

[0164] E l [(d,l),j]=E l[(d,l),j]*γλ, where and

[0165] E w [(d,w),h]=E w [(d,w),h]+1,w∈N m And w∈n m,l

[0166] In the topology diagram g m The agent in the update Q value according to the Q value update formula

[0167]

[0168] If the action k obtained * If it is inconsistent with the action k' calculated by Boltzmann softmax, the historical trajectory saved in the mth AGV is cleared

[0169] If k * ≠k', then execute the following (1)-(3):

[0170] (1) and

[0171] (2)N m =φ,ε m =φ

[0172] (3)N m ∪{h}

[0173] Control AGV (m, m∈M) to go to k' and repeat step S4-2 until AGV (m, m∈M) delivers the goods to the destination.

[0174] In this embodiment, in combination with the above technical solution, Figure 1 The five rectangles at the top are where the goods are generated. They will be generated at a rate of 2,000 pieces per hour. Different goods are represented by different colors. There are a total of 5 colors representing 5 types of goods. Goods will be randomly generated in one of the five rectangles at the top. Following the first-in-first-out principle, the first generated goods will be transported by the AGV first. Figure 1 The leftmost side is the starting point of the AGV. The length and width of the AGV are both 0.5m, the height is 0.3m, the fastest speed is 1 meter per second, and the minimum distance between AGVs is 0.2m. Figure 1There are 5 conveyor belts at the bottom. The AGV needs to deliver the corresponding color goods from the place of production to the corresponding conveyor belt. The total number of goods delivered will increase by 1 for each delivery to the conveyor belt within the specified 36,000 seconds. In the example process, only the shortest path, centralized, and distributed algorithms are used to plan the path for the AGV that obtains the goods. The scheduling of the AGV that does not obtain the goods uses the method provided by the AnyLogic simulation platform. When the AGV arrives at a certain intelligent body, the steps are consistent with S3 in the invention technical solution until the specified time is met.

[0175] pass Figure 3 and Figure 4 It can be seen that the distributed reinforcement learning method provided in this embodiment is to dynamically plan the path of AGV. Compared with the centralized algorithm, when the number of AGVs is controlled within a certain range, the performance is almost the same as the centralized effect. The number of calculations in this embodiment is significantly lower than that of the centralized one. We assume that each calculation of the Q value during the simulation is It is counted as one calculation. From the results, it can be seen that the distributed algorithm proposed in the present invention achieves a similar effect at a lower cost.

[0176] In addition, through Figure 2 and Figure 3 It can be seen that compared with the traditional algorithm, which will not fall into the local optimum and uses the traditional algorithm to plan the AGV path according to the shortest path in a fixed AGV track environment, the number of AGVs is limited. This embodiment can achieve the purpose of increasing the number of AGVs by alleviating congestion and thus transporting more goods.

[0177] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A method for AGV dynamic path planning based on distributed reinforcement learning, characterized in that: The steps include: S1. Build a simulated smart factory and set up an intelligent agent with computing, storage and communication functions at each path selection node in the simulated smart factory; S2. Set the parameters of the agent and the AGV running on the path All intelligent agents are represented by set I, and AGVs are represented by set M. Assume that there are n intelligent agents and m AGVs. For a certain agent, represented by i, i∈I, the agent i stores Q i 、E i , δ i , Where Q i represents the Q table stored in agent i. The horizontal axis of the Q table represents the state of the node, and the vertical axis represents the action of the node. i represents the qualification trace table stored in agent i, the format is consistent with Q, δ i Represents the TD-error saved in agent i, which is a vector with a length of |δ i |=m, It represents the static time for agent i to reach the adjacent node j, that is, the shortest time from node i to node j when there is no congestion on the track. S3. Design the state, action, reward, and policy in reinforcement learning. Among them, state is represented by the symbol (d,i), where d represents the target node and i represents the current node. Action is represented by the symbol A(d,i), which represents the set of adjacent nodes that the current node i can reach. Reward is represented by the symbol R[(d,i),j], and the reward function is R[(d,i),j]=t(i,j), where t(i,j) represents the actual driving time of the AGV from the current node i to the node j. Each element in the Q table can be used express; S4. Train the agent.

2. The AGV dynamic path planning method based on distributed reinforcement learning according to claim 1 is characterized in that: The simulated smart factory is built using AnyLogic simulation software.

3. The AGV dynamic path planning method based on distributed reinforcement learning according to claim 1 is characterized in that: The parameter setting of the AGV: The AGV stores the nodes reached each time, and then builds a model for the network: G(N,ε)={g1(N1,ε1),...,g m (N m ,he m )} Where G represents the graph, N represents the node set, ε represents the edge set, and g m Represents the graph of the mth AGV, N m represents the set of nodes that the mth AGV has passed through, ε m It represents the set of edges that the mth AGV has walked through. G is composed of many subgraphs, and the subgraphs are formed by the trajectories that the AGV has walked through.

4. The AGV dynamic path planning method based on distributed reinforcement learning according to claim 3 is characterized in that: The policy in step S3 is: The policy uses Boltzmann softmax, which is expressed as follows: Boltzmann softmax Boltzmann(i) means that agent i uses the Boltzmann softmax policy to sample an action based on the calculated probability distribution.

5. The AGV dynamic path planning method based on distributed reinforcement learning according to claim 4 is characterized in that: The step S4 includes the following sub-steps: S4-1. Initialize the Q stored in each agent i 、E i , δ i ; S4-2, when AGV (m, m∈M) arrives at node h, if h is the starting node, execute step S4-3, if h is the end point, execute step S4-4, if h is neither the starting node nor the end point, execute step S4-5; S4-3, δ h (m) = -1000, node h selects k' through the Boltzmann softmax policy, controls m to go to k', N m ∪{h}, repeat step S4-2 until AGV (m, m∈M) delivers the goods to the destination; S4-4, AGV (m, m∈M) completes the task, goes to the starting node of the next task, and repeats step S4-2; S4-5. Control AGV (m, m∈M) to go to k' and repeat step S4-2 until AGV (m, m∈M) delivers the goods to the destination.

6. The AGV dynamic path planning method based on distributed reinforcement learning according to claim 5 is characterized in that: In step S4-1, the initialization method is: Until 7. The AGV dynamic path planning method based on distributed reinforcement learning according to claim 5 is characterized in that: The specific method of step S4-4 is: When AGV numbered m arrives at node h, it adds node h and edge (w,h) to topology graph g. m middle, N m ∪{h},ε m ∪{(w,h)},w∈n m,h , n m,h In Figure g m Neighbors of node h; Agent h sets the TD-error value corresponding to the mth AGV to a sufficiently small number, thereby clearing the historical record. All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the agents in the topology graph g. m The neighbors in g perform min operation after receiving it, and then pass the sufficiently small number set above to all neighbors in g m The intelligent agent in the system can achieve the purpose of clearing historical data; The actual driving time of the AGV from node w to node h is used as a reward; Agent h calculates TD-error according to the TD-error calculation formula. Node h is the end point, and TD-error is consistent with the reward. All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform the max operation after receiving it, thus passing the newly calculated TD-error calculated above to all neighbors in g m The intelligent agents in the network can transfer TD-error between intelligent agents. In the topology diagram g m The agent in updates the eligibility trace according to the eligibility trace formula rules; In the topology diagram g m The agent in updates the Q value according to the Q value update formula; The AGV (m, m∈M) task is completed, and it goes to the starting node of the next task and repeats step S4-2.

8. The AGV dynamic path planning method based on distributed reinforcement learning according to claim 7 is characterized in that: The specific method of step S4-5 is: When AGV numbered m arrives at node h, it adds node h and edge (w,h) to topology graph g. m middle, N m ∪{h},ε m ∪{(w,h)},w∈n m,h , n m,h In Figure g m Neighbors of node h; Agent h sets the TD-error value corresponding to the mth AGV to a sufficiently small number, thereby clearing the historical records; All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the agents in the topology graph g. m The neighbors in g perform min operation after receiving it, and then pass the sufficiently small number set above to all neighbors in g m The intelligent agent in the system can achieve the purpose of clearing historical data; According to the Q table saved in the agent, greedily take the action k with the smallest Q value * ; Agent h calculates action k' based on Boltzmann softmax; Agent h calculates TD-error according to the TD-error calculation formula; All in the topology diagram m The agents in the topology graph communicate with each other and pass their own δ(m) to the m The neighbors in g perform the max operation after receiving it, thus passing the newly calculated TD-error calculated above to all neighbors in g m The intelligent agents in the network can transfer TD-error between intelligent agents. In the topology diagram g m The agent in updates the eligibility trace according to the eligibility trace formula rules; In the topology diagram g m The agent in updates the Q value according to the Q value update formula; If the action k obtained * If the action k' is inconsistent with the action k' calculated by Boltzmann softmax, the following (1)-(3) are executed: (1) and (2)N m =φ,e m =φ (3)N m ∪{h} Control AGV (m, m∈M) to go to k' and repeat step S4-2 until AGV (m, m∈M) delivers the goods to the destination.

Images