Intelligent factory-oriented reinforcement learning path planning method for multiple unmanned vehicles

Abstract

The invention provides a reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory. The method comprises the following steps: building a feeding path planning simulation environment; taking each unmanned vehicle as an intelligent agent, interacting the intelligent agent with the environment to extract information, and constructing a state space, an action space and a reward function of the Markov decision model according to the information; building an IPPO algorithm network, and training the strategy of the Markov decision model based on the IPPO algorithm network and the reward function; during training, a strategy network loss function with a rollback strategy is obtained through improvement by combining cutting loss with KL divergence in a mode of introducing hyper-parameters; and determining the current state at each moment according to the actual current position of the unmanned vehicle, inputting the current state into the strategy network, and outputting the current action to be executed by the unmanned vehicle. The method is high in convergence speed, stable in strategy updating, high in exploration capacity, capable of effectively reducing path blockage and high in environment adaptability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and specifically refers to a reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory. It is for multiple unmanned vehicles in an intelligent factory and is implemented based on the PPO algorithm, especially the IPPO (Independent Proximal Policy Optimization) algorithm. Background Art

[0002] The problem of material distribution for multiple unmanned vehicles in an intelligent factory is essentially a path planning problem for a group of unmanned vehicles. Each unmanned vehicle has a different task, and each unmanned vehicle has to move from a given starting position to a corresponding target position while avoiding collisions with other unmanned vehicles and obstacles.

[0003] The main planning methods for the intelligent factory material distribution problem include traditional path planning algorithms, centralized reinforcement learning planning algorithms, and distributed reinforcement learning planning algorithms. Among them, traditional path planning algorithms face problems such as poor dynamic obstacle avoidance effect and low planning efficiency; centralized reinforcement learning planning algorithms also have problems such as high computational complexity, poor scalability, and sparse reward function distribution.

[0004] Based on the actual requirements of intelligent factory material distribution and the problems existing in the above algorithms, using a distributed reinforcement learning algorithm would be a better choice. In the distributed algorithm, each machine can store and process data, facilitating the handling of its own tasks. At the same time, it has a low requirement for CPU computing power, reducing the actual usage cost. However, the distributed reinforcement learning algorithm has the problem of obtaining local information and no communication between multiple unmanned vehicles, resulting in path conflicts, and further improvement is needed to avoid path conflicts. Summary of the Invention

[0005] To solve the problems in the above-mentioned prior art, the present invention provides a reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory. This method has a relatively fast convergence speed, stable policy update, strong exploration ability, can effectively reduce path congestion, and has a strong environmental adaptability.

[0006] To achieve the above purpose, the present invention provides a reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory, including:

[0007] S1: Obtain the actual layout of the workstations in the factory and the initial conditions of the unmanned vehicles, and build a simulation environment for the feeding path planning;

[0008] S2: Take each unmanned vehicle as an agent, interact the agent with the feeding path planning simulation environment to extract the real-time observation information of the unmanned vehicle and the position of the material demand point, and accordingly construct the state space and action space of the Markov decision model, and establish an optimization goal and a reward function with high task completion, few blockages and collisions, and as short a path as possible;

[0009] S3: Build an IPPO algorithm network based on the Actor-Critic architecture, and train the policy of the Markov decision model based on it and the reward function; during training, by introducing hyperparameters to combine the clipped loss with the KL divergence, thereby improving the policy network loss function with a rollback policy to limit the amplitude of the policy network update;

[0010] S4: When performing the actual path planning task of multi-unmanned vehicle material distribution, determine the current state according to the actual current position of the unmanned vehicle at each moment, input the current state into the policy network, and output the current action through the policy network for the unmanned vehicle to execute.

[0011] The specific steps of S1 include:

[0012] S11: According to the actual layout of the workstations in the intelligent factory, perform grid modeling at a fixed ratio to obtain a two-dimensional grid, and each grid cell of the two-dimensional grid corresponds to one of the idle area, obstacle, unmanned vehicle starting point, and material demand point;

[0013] S12: Establish the mapping rules of each grid cell of the two-dimensional grid and the current position of the unmanned vehicle in the two-dimensional array for mapping the two-dimensional grid into a multi-channel two-dimensional array;

[0014] S13: Write a generator for fixed obstacles and random target points, which is used to set the obstacle positions as fixed, and randomly initialize the unmanned vehicle starting point and material demand point as random target points at the beginning of each round to dynamically simulate the changes in the environment;

[0015] S14: Set the movement speed of the unmanned vehicle and the size of the observable window. The movement speed of the unmanned vehicle is used for simulation at each time step after the start of the round, and the observable window is used to extract the real-time observation information of the unmanned vehicle.

[0016] In S12, the mapping rules of each grid cell of the two-dimensional grid and the current position of the unmanned vehicle in the two-dimensional array are: the idle area is marked as 0; the obstacle is marked as 1; the position of its own unmanned vehicle is marked as 3; the positions of other unmanned vehicles are marked as 2; the position of its own target point is marked as 5; the positions of other target points are marked as 4.

[0017] Construct the state space and action space of the Markov decision model, specifically including: taking each unmanned vehicle as an agent, constructing the path planning task of multi-unmanned vehicle material distribution in the intelligent factory as a Markov decision process, using the local environmental information of the observable window of the unmanned vehicle as the current state, and all possible combinations of the current states as the state space; taking the action set of the unmanned vehicle as the action space, which includes five actions: up, down, left, right, and stop.

[0018] The optimization goal is:

[0019]

[0020] The reward function includes:

[0021] R(s t ,a t )=R ext (s t ,a t )+β·R int (s t ),

[0022] R ext (s t ,a t )=R arrive +R collsion +R step ,

[0023] R int (s t )=α(t)*b1(s t )+(1-α(t))*b2(s t ),

[0024]

[0025]

[0026]

[0027]

[0028] N(s t )=N(s t )+1,

[0029] Among them, R(s t ,a t ) is the total reward; E π represents the expected cumulative reward of all possible state and action combinations under strategy π, γ is the discount factor; s t Indicates the current state of the unmanned vehicle in the environment at the current time step t, a t represents the current action selected by the autonomous vehicle at the current time step t according to the current strategy π; R ext (s t ,a t ) represents the external reward, which is the immediate reward provided by the environment after the agent interacts with the environment. It is composed of the arrival reward R at each time step. arrive , collision rewards R between agents and between agents and obstaclescollsion , the step reward R corresponding to the consumed steps step ; β is the internal reward weight; R int (s t ) represents the internal reward, α(t) represents the internal weight of the internal reward, b1(s t ) represents the error reward deviating from the mean; b2(s t ) represents the reward for the state access frequency; T k represents the output feature of the k-th target network for the state, μ(s t ) represents the mean of the output features of a group of target networks for the state, f predict (s t ) represents the output feature of the prediction network for the state, N(s t ) represents the state access frequency, T is the decay time constant, and K is the number of target networks.

[0030] The internal reward is obtained based on distributed random network distillation. When calculating the internal reward based on distributed random network distillation, two sets of networks with the same structure are built as the target networks {T1, T2,..., T k} and the evaluation network f predict . Multiple unmanned vehicles share a group of target networks and independently have a trainable prediction network.

[0031] The S3 specifically includes:

[0032] S31: Initialize the policy network and value network of the Actor-Critic architecture;

[0033] S32: Enable the agent to interact with the environment according to the current policy to obtain trajectory data. Each sampling point of the trajectory data includes (s t , a t , R(s t , a t ), s t+1 ), s t is the current state, a t is the current action, R(s t , a t ) is the total reward, and s t+1 is the next state;

[0034] S33: Calculate the importance sampling ratio r t (θ) and the KL divergence and calculate the advantage function

[0035] S34: According to the importance sampling ratio r t (θ) and the advantage function To determine the policy update item According to the policy update item KL divergence To obtain the policy network loss function L with a rollback policy Rollback-IPPO (π);

[0036] S35: According to the policy network loss function L with a rollback policy Rollback-PPO (π) to update the policy network;

[0037] S36: Update the value network according to the value network loss function.

[0038] The policy network loss function L with a rollback policy Rollback-IPPO (θ) is:

[0039]

[0040]

[0041] Wherein, is the KL divergence; δ is the divergence threshold; is the policy update item; β is the penalty coefficient; is the single state-action pair loss, and the loss function L Rollback-IPPO (π) is defined by averaging all single state-action pair losses in the entire batch or trajectory.

[0042] The importance sampling ratio r t (θ) is:

[0043]

[0044] Wherein, π θ (a t |s t ) is the probability distribution of the new policy taking action a t under the current state s t ; is the probability distribution of the old policy taking action a t under the current state s t ;

[0045] KL divergence is:

[0046]

[0047] Wherein, D KL is the KL divergence, and the KL divergence is used to measure the deviation between the new and old policies, which is the difference in the probability distributions of the new and old policies. The KL divergence measures the difference in action selection between the new and old policies in each state; is the expectation;

[0048] Advantage function is:

[0049] δ t+l = r t+l + γV(s t+l+1 ) - V(s t+l ),

[0050]

[0051] where δ t+l is the temporal difference error at time step t + l, r t+l is the total reward at time step t + l, γ is the discount factor, V(s t ) is the prediction of the value network for the current state, representing the expected future return starting from this state; V(s t+l ) is the prediction of the value network for the state at time step t + l, V(s t+l+1 ) is the prediction of the value network for the state at time step t + l + 1, t is the current time step, l represents the future offset relative to the current time step t, and λ is the hyperparameter in generalized advantage estimation; T is the decay time constant.

[0052] The present invention optimizes the maximization of the cumulative discounted reward by incorporating a clipping constraint and a KL divergence constraint into the policy network loss function, while preventing drastic changes in the policy. During the update process of the policy network, more stable and efficient policy optimization is achieved by calculating the ratio of the old and new policies, the advantage function, and the KL divergence, with a relatively fast convergence speed, stable policy updates, strong exploration ability, and effective reduction of path congestion, and strong environmental adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is the overall algorithm framework diagram of a reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory according to an embodiment of the present invention.

[0054] Figure 2 is the path planning result display diagram of the path planning task of the multi-unmanned vehicle material distribution obtained by the reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory of the present invention. In the figure, the large-sized square points are the starting points of the unmanned vehicles, and the circular points are the material demand points.

[0055] Figures 3A to 3C is the result comparison diagram of the reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory using the IPPO algorithm network of the present invention and the existing reinforcement learning path planning method for multiple unmanned vehicles using the conventional PPO algorithm. Among them, Figure 3A shows the change in the average number of completed material distributions of the unmanned vehicles with the number of episodes, Figure 3BShows the change of the average reward of the driverless vehicle with the number of rounds. Figure 3C Shows the change of the average number of steps required for the driverless vehicle to complete tasks with the number of rounds. Specific implementation manners

[0056] The following further describes the present invention in conjunction with specific embodiments. It should be understood that the following embodiments are only used to illustrate the present invention and not to limit the scope of the present invention.

[0057] As Figure 1 Shown is a reinforcement learning path planning method for multiple driverless vehicles for an intelligent factory according to an embodiment of the present invention. In the present invention, there are multiple driverless vehicles, and the experimental scenarios discussed in this study are generally for 4 driverless vehicles to work simultaneously.

[0058] As Figure 1 Shown, the reinforcement learning path planning method for multiple driverless vehicles for an intelligent factory includes:

[0059] Step S1: Obtain the true layout of the workstations in the factory and the initial conditions of the driverless vehicles, and build a feeding path planning simulation environment 100;

[0060] Among them, the true layout of the workstations in the factory includes the free area and the positions of obstacles; the initial conditions of the driverless vehicles include the positions of the starting points and material demand points of the driverless vehicles, the movement speed of the driverless vehicles, and the size of the observable window.

[0061] The said S1 specifically includes:

[0062] S11: According to the true layout of the workstations in the intelligent factory, perform grid modeling at a fixed ratio to obtain a two-dimensional grid, and each grid cell of the two-dimensional grid corresponds to one of the free area, obstacles, starting points of driverless vehicles, and material demand points.

[0063] Among them, grid modeling is a method of digitizing and discretely representing the real-world map in the form of a two-dimensional grid. In the two-dimensional grid discussed in the present invention, each grid cell matches the fixed size of the actual workstation and the size of the driverless vehicle (1 meter × 1 meter), where the size of the driverless vehicle exactly occupies one grid cell of the two-dimensional grid. Different driverless vehicles cannot be located in the same grid cell. If the next actions of two driverless vehicles point to the same grid cell, or the positions of two driverless vehicles are adjacent and they are moving towards each other in the next step, it is regarded as a blockage. The free area represents the area where the driverless vehicle can pass. If there is no driverless vehicle at the starting point and material demand point of the driverless vehicle, it is also available for the driverless vehicle to pass.

[0064] In this embodiment, the driverless vehicle adopts an automatic guided vehicle (AGV). The size of the two-dimensional grid is 18×18.

[0065] S12: Establish the mapping rules of each grid cell of the two-dimensional grid and the current position of the driverless vehicle in the two-dimensional array, which are used to map the two-dimensional grid into a multi-channel two-dimensional array.

[0066] The mapping rules of each grid cell of the two-dimensional grid and the current position of the driverless vehicle in the two-dimensional array are as follows: The free area (blank cell) is marked as 0; the obstacle (fixed obstacle) is marked as 1; the position of its own driverless vehicle is marked as 3; the positions of other driverless vehicles are marked as 2; the position of its own target point (i.e., the material demand point of its own driverless vehicle) is marked as 5; the positions of other target points (i.e., the material demand points of other driverless vehicles) are marked as 4.

[0067] Among them, each driverless vehicle has its own perspective. From its own perspective, the positions of other driverless vehicles are 2, the position of its own driverless vehicle is 3, the corresponding target points of other driverless vehicles are 4, and the corresponding target point of its own is 5.

[0068] Step S13: Write a generator for fixed obstacles and random target points through a python program. The generator is used to set the obstacle positions as fixed and randomly initialize the starting points and material demand points of the driverless vehicles as random target points at the beginning of each round to dynamically simulate the changes in the environment.

[0069] The generator for fixed obstacles and random target points is obtained according to the actual material demand scenario, so the obstacle positions are set as fixed.

[0070] Step S14: Set the movement speed of the driverless vehicle and the size of the observable window. The movement speed of the driverless vehicle is used for the simulation of each time step after the start of the round, and the observable window is used to extract the real-time observation information of the driverless vehicle.

[0071] Set the driverless vehicle to move at a constant speed. The movement speed of the driverless vehicle is a fixed value (such as 1 m / s), without considering the acceleration and deceleration in the actual process. It is stipulated that the size of the observable window of the driverless vehicle is 7×7. Thus, in the following simulation process, within each time step, the driverless vehicle moves one grid unit (1 m).

[0072] Therefore, the feeding path planning simulation environment 100 for the intelligent factory not only includes the real layout of the workstations in the intelligent factory, but also includes the number of driverless vehicles, movement speed, etc.

[0073] Step S2: Take each driverless vehicle as an agent, interact the agent with the feeding path planning simulation environment 100 to extract the real-time observation information and the position of the material demand point of the driverless vehicle, and accordingly construct the state space and action space of the Markov decision model (MDP), and establish an optimization goal and a reward function 200 with high task completion rate, few blockages and collisions, and as short a path as possible.

[0074] In a Markov decision model, a round refers to a complete sequence of interactions between an agent and the environment. It usually starts from an initial state, goes through a series of state transitions and action selections, and finally reaches a termination state or meets specific termination conditions. The autonomous vehicle determines its path planning target point based on the location of the material demand point at the beginning of each round in the Markov decision model.

[0075] A Markov decision process includes: state, action, transition probability, reward, and discount factor.

[0076] Specifically, S2 includes:

[0077] S21: Construct the state space and action space of the Markov decision model, specifically including: taking each autonomous vehicle as an agent, constructing the path planning task of multi-autonomous vehicle material distribution in the intelligent factory as a Markov decision process; taking the local environmental information of the observable window of the autonomous vehicle as the current state, and combining all possible current states as the state space; taking the action set of the autonomous vehicle as the action space, which includes five actions: up, down, left, right, and stop.

[0078] Taking the local environmental information of the observable window of the autonomous vehicle as the state space of the Markov decision process specifically includes: based on the real-time position of the autonomous vehicle, extracting the local environmental information of the observable window with a size of 7×7 centered on the autonomous vehicle to generate a three-dimensional tensor containing 5 channels as the current state in the state space. The shape of the generated three-dimensional tensor is 5×7×7, where 5 represents the number of channels, each channel corresponds to a type of mapping rule, and 7×7 represents the size of the observable window. Therefore, it can be understood that the autonomous vehicle can see the surrounding environment of 7×7 centered on itself in an 18×18 map. The actions of the autonomous vehicle can be up, down, left, right, and stop, and can be represented as a one-dimensional vector.

[0079] In other embodiments, the size of the two-dimensional grid is 18×18. When the size of the two-dimensional grid changes, the size of the observable space of the autonomous vehicle also changes. Specifically, conclusions can be obtained through continuous attempts during the experimental training process, and there is no clear data relationship. The present invention only considers an 18×18 grid.

[0080] It should be noted that the absolute position of the autonomous vehicle does not need to be reflected in the two-dimensional grid because, based on the problem background we set, the autonomous vehicle can only see the 7×7 area centered on itself. 5×7×7 is actually 5 two-dimensional arrays of 7×7, that is, a three-dimensional array with a size of 5×7×7.

[0081] In the present invention, the six types of mapping rules include free areas, obstacles, the position of the self-driving vehicle itself, the positions of other self-driving vehicles, the position markers of the self-goal points, and the positions of other goal points. Considering that the six types are mutually exclusive, only five channels are required.

[0082] The specific information of each channel is as follows:

[0083] Channel 1: Environmental obstacle information (Obstacle Map).

[0084] It is used to record the positions of fixed obstacles.

[0085] If there is an obstacle in a certain cell of the grid, the value is set to 1; otherwise, it is set to 0.

[0086] Channel 2: The positions of other AGVs (Other-AgentMap).

[0087] It is used to record the position information of other AGVs.

[0088] If a certain grid cell is occupied by other AGVs, the value is set to 1; otherwise, it is set to 0.

[0089] Channel 3: The position of the current AGV (Self-Agent Map).

[0090] It is used to record the position of the current AGV to avoid collisions.

[0091] If the grid cell is the position where the current AGV is located, the value is set to 1; otherwise, it is set to 0.

[0092] Channel 4: The goal point information of other AGVs (Other-GoalMap).

[0093] It is used to record the goal point positions of other AGVs to avoid competition conflicts and improve the task completion rate.

[0094] If a certain grid cell is the goal point of other AGVs, the value is set to 1; otherwise, it is set to 0.

[0095] Channel 5: The goal point information of the current AGV (Self-Goal Map).

[0096] It is used to record the goal point position of the current AGV for path planning.

[0097] If a certain grid cell is the goal point of the current AGV, the value is set to 1; otherwise, it is set to 0.

[0098] In this embodiment, since there are a total of four self-driving vehicles and their corresponding four goal points, each self-driving vehicle acts as an agent, having its own independent state space and action space, forming a complete state-action space pair.

[0099] S22: Establish an optimization objective and a reward function 200 with a high task completion rate, few blockages and collisions, and as short a path as possible, so as to guide the unmanned vehicle to quickly plan the material delivery path.

[0100] Among them, the optimization objective is to maximize the expected value of the cumulative reward for each round.

[0101] In the embodiment, the optimization objective is expressed as:

[0102]

[0103] Among them, R(s t ,a t ) is the total reward, and R(s t ,a t ) is equal to R ext (s t ,a t ) + β·R int (s t ), R int (s t ) is the internal reward, and R ext (s t ,a t ) is the external reward; E π represents the expected cumulative reward for all possible state and action combinations under the policy π, γ is the discount factor, γ ∈ [0, 1), which is used to balance short-term rewards and long-term rewards; s t represents the current state of the unmanned vehicle in the environment at the current time step t, and a t represents the current action selected by the unmanned vehicle at the current time step t according to the current policy π.

[0104] The policy π(a|s) is a probability distribution, indicating the probability distribution of the agent taking the action a t under s t , that is

[0105]

[0106] The policy is obtained through reinforcement learning training and is represented in the form of a neural network.

[0107] That is to say, the optimization objective is to obtain an optimized policy π by maximizing the expected value of the cumulative reward to achieve efficient decision-making and learning.

[0108] In the task of multiple unmanned vehicles transporting materials simultaneously, there are problems such as the dynamic change of target points in each round, the relatively sparse traditional reward function, insufficient exploration, and the existence of unpredictable dynamic obstacles in the environment. The present invention designs an internal reward of Distributional Random Network Distillation (DRND) to optimize the reward function 200. The reward function 200 includes an internal reward 201 based on distributional random network distillation and an external reward 202. The external reward focuses on collision avoidance, step penalty, and arrival reward, while the internal reward solves the problem of reward sparsity and improves the implicit cooperation efficiency of multiple unmanned vehicles. For distributional random network distillation, refer to the literature [Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. 2024. Exploration and anti-exploration with distributional random network distillation. In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 2328, 56397–56421].

[0109] Among them, the reward function is as follows:

[0110] R(s t ,a t ) = R ext (s t ,a t ) + β·R int (s t )

[0111] R ext (s t ,a t ) = R arrive + R collsion + R step

[0112] R int (s t ) = α(t)*b1(s t ) + (1 - α(t))*b2(s t )

[0113]

[0114]

[0115]

[0116]

[0117] N(s t ) = N(s t ) + 1

[0118] Among them, R ext (s t , a t ) represents the external reward, which is the immediate reward provided by the environment after the agent interacts with the environment. It is composed of the arrival reward R arrive at each time step, the collision reward R collsion between agents and between the agent and obstacles, and the step reward R step corresponding to the consumed number of steps; β is the internal reward weight, which is a hyperparameter used to balance the relative importance of the external reward and the internal reward. β is generally between 0.1 - 0.2; R int (s t ) represents the internal reward, and α(t) represents the internal weight of the internal reward, specifically indicating the dynamic change of the weights of the two rewards b1(s t ) and b2(s t ) over time. b1(s t ) represents the error reward for deviating from the mean, which increases as the error value deviating from the mean decreases; b2(s t ) represents the reward for the state access frequency, which decreases as the state access frequency N(s t ) increases to avoid repeated exploration; T k (s t ) represents the output feature of the k-th target network for the state; μ(s t ) represents the mean of the output features of a group of target networks for the state, specifically indicating the intensity of the target feature, f predict (s t ) represents the output feature of the prediction network for the state, specifically indicating the intensity of the prediction feature, N(s t ) represents the state access frequency, that is, how many times this grid cell has been passed through in this round. T is the decay time constant. The decay time constant T is generally set to the maximum limited number of steps in the whole round. In the present invention, the decay time constant T is generally set to 400, and K is the number of target networks, generally set to 10.

[0119] In this embodiment, 4 unmanned vehicles are used as agents, each having its own independent internal reward and external reward functions. In the external reward, when the unmanned vehicle reaches the target point, the reward R arrive is +2.0, and for each effective movement action executed, the reward R stepis -0.1, and the reward R for hitting an obstacle collsion is -1.0. If the action of the autonomous vehicle is invalid (such as attempting to move into an obstacle or being blocked by other autonomous vehicles), the reward R collsion is -0.2.

[0120] It should be noted that when calculating the internal reward based on the distributed random network distillation, two sets of networks with the same structure are built as the target networks {T1, T2,..., T k} and the evaluation network f predict , and multiple autonomous vehicles share a set of target networks and independently have a trainable prediction network. The internal reward is generated by obtaining the state space of a single autonomous vehicle, using it as the input of the target network and the prediction network, and generating an error reward b1(s t ) that deviates from the mean according to the error between the output features of the target network and the prediction network, and combining the error between the output features of the target network and the prediction network with the state access frequency to generate a reward b2(s t ) for the state access frequency to avoid repeated access to the explored states.

[0121] The target network does not participate in training and thus does not update its parameters. Specifically, the target network is randomly generated and does not update its parameters. It provides a consistent and stable reference point for the prediction networks of 4 autonomous vehicles. The high-dimensional output feature μ(s t ) is obtained by using the mean value extracted from the state by this set of target networks and has no actual physical meaning. The output feature of the prediction network attempts to approximate the output feature of the target network. It makes the prediction network learn the high-dimensional representation of the state under the target network by minimizing the error between the two, gradually fitting the target feature by minimizing the difference from the output of the target network, and calculating the deviation error between the mean of the target network and the output of the prediction network. The error value is used to generate the internal reward for the unexplored state, that is, b1(s t ). Since each autonomous vehicle has an independent prediction network, this enables each agent to adjust its exploration strategy based on its own experience. When different autonomous vehicles learn on different paths, there will be different prediction errors, so different paths will be explored, thus avoiding all autonomous vehicles choosing the same path and forming an implicit cooperation. Therefore, the internal reward adopted by the present invention enables multiple autonomous vehicles to not only efficiently explore the environment but also cooperate to complete tasks, improving the overall efficiency of path planning.

[0122] The internal weight α(t) of the internal reward is dynamically adjusted to adapt to different training stages. As t starts from 0 and increases with the number of steps of the autonomous vehicle, at the initial stage of training, the internal weight α(t) of the internal reward is relatively large, which means that the error reward b1(s t) has a larger weight. When a certain state is visited less frequently, the error of the prediction network is larger. This error can be used as the main part of the internal reward to motivate the autonomous vehicle to explore new states. As the training progresses, the internal weight α(t) of the internal reward will gradually decrease. At this time, the influence of the error reward b1(s t ) gradually weakens, while the reward b2(s t ) of the state visit frequency gradually increases. After a certain state is frequently visited, the error b2(s t ) of the prediction network will decrease, thereby reducing the internal reward and helping the agent better allocate rewards according to the state visit frequency, avoiding excessive repeated visits.

[0123] S3: Build an IPPO algorithm network 300 based on the Actor-Critic architecture, and train the policy of the Markov decision model based on it and the reward function, so as to quickly realize the path planning from the starting point to the target point of the autonomous vehicle after each update of the material demand point.

[0124] For the IPPO algorithm network 300, please refer to the literature [Witt C SD, Gupta T, Makoviichuk D, et al. Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge[J]. 2020. DOI: 10.48550 / arXiv.2011.09533.].

[0125] During training, by introducing hyperparameters, the clipped loss is combined with the KL divergence to improve the policy network loss function with a rollback policy, restricting the update amplitude of the policy network.

[0126] Among them, the Markov decision process (MDP) provides a basic mathematical framework for the entire path planning task, defining the state space, action space, reward function, and optimization goal in the task. Specifically, MDP describes the interaction mode between the autonomous vehicle and the environment, that is, how the autonomous vehicle selects actions in different states and adjusts its future decisions according to the rewards obtained after executing the actions. MDP provides a clear definition for the path planning task, clarifying the decision-making problem that the agent needs to solve.

[0127] The IPPO algorithm network 300 based on the Actor-Critic architecture is a network architecture used to optimize policies in reinforcement learning algorithms, specifically designed for maximizing cumulative rewards through interaction with the environment. Among them, the Actor-Critic architecture, as the basic network architecture of the PPO algorithm network, specifically learns and optimizes policies through the policy network 301 (Actor), and evaluates the quality of policies through the value network 302 (Critic), thereby guiding the optimization of policies. The PPO algorithm uses the Actor-Critic architecture to achieve effective learning of policies, and combines the state space, action space, and reward function provided by the MDP to guide the training process. The IPPO algorithm network 300 enables each unmanned vehicle to act as an agent, and multiple agents make independent decisions simultaneously.

[0128] Specifically, S3 includes:

[0129] S31: Initialize the policy network 301 (Actor) and value network 302 (Critic) of the Actor-Critic architecture.

[0130] In this embodiment, a feature extraction module composed of a 5-layer convolutional network and a 1-layer fully connected layer is used as the basic network of the policy network 301 and value network 302.

[0131] In addition, the IPPO algorithm network 300 uses an LSTM network in the policy network. After the state space enters the policy network and is extracted by the convolutional layer, it is input into the LSTM network through the fully connected layer. The LSTM processes these sequence data and gives the output to the next fully connected layer, and then the fully connected layer outputs the action probability distribution through the softmax function.

[0132] S32: Enable the agent to interact with the environment according to the current policy to obtain trajectory data. Each sampling point of the trajectory data includes (s t , a t , R(s t , a t ), s t+1 ), s t is the current state, a t is the current action, R(s t , a t ) is the total reward, and s t+1 is the next state;

[0133] Among them, the current state s of the unmanned vehicle tThe convolutional network of each respective policy network 301 performs feature extraction and outputs a 120-dimensional feature vector. The feature vector is input into the fully connected layer of the policy network 301, and the action probability distribution π(a|s) is output through the Softmax activation function. An exploratory sampling action a is taken through the probability distribution. t , for execution. After executing the action a t , the next state s is obtained through the environment t+1 and the external reward R is calculated ext (s t , a t ) and the internal reward R int (s t ) to obtain the total reward R(s t , a t ). Record the current state s t , the executed action a t , the immediate reward R(s t , a t ), and the updated state s t+1 , for subsequent training.

[0134] S33: Calculate the importance sampling ratio r t (θ) and the KL divergence and calculate the advantage function

[0135] In step S33, calculate the importance sampling ratio and the KL divergence based on the data of the sampling points Meanwhile, the value network 302 obtains the prediction V(s t ) of the value network for the current state according to the current state s t ), and calculates the advantage function according to the prediction V(s t ) of the value network for the current state

[0136] Among them, the importance sampling ratio r t (θ) is:

[0137]

[0138] Among them, π θ (a t |s t ) is the probability distribution of the new policy taking the action a t under the current state s t , is the probability distribution of the old policy taking the action a t under the current state s t . Both are the probability distributions of taking the action a t under the current state st The probability distribution of , but one is the old strategy and the other is the current strategy, because each round of optimization strategy will be updated, that is, the importance sampling ratio r t (θ) means taking action a under the new and old strategies t The probability ratio.

[0139] KL divergence Used to measure the difference between the probability distributions of new and old strategies to avoid drastic changes in strategies.

[0140] KL divergence It represents the overall change in the probability distribution of the new and old strategies on all actions, which is used to measure the new strategy π θ and old strategies The overall distribution difference in a certain state, that is, not only focusing on a single state, but also focusing on the changes in policy distribution in all states during the entire training process.

[0141] KL divergence for:

[0142]

[0143] in, is the KL divergence, which is used to measure the deviation between the new and old strategies. It is the difference in the probability distribution of the new and old strategies. The KL divergence measures the difference in action selection between the new and old strategies in each state; π θ (a t |s t ) is the new strategy in the current state s t The probability distribution of taking action a under is the old policy in the current state s t The probability distribution of taking action a under For expectation.

[0144] Advantage function The generalized advantage value method is used to obtain:

[0145] δ t+l =r t+l +γV(s t+l+1 )-V(s t+l )

[0146]

[0147] Among them, δ t+l is the time difference error of time step t+l, r t+l is the total reward at time step t+l, γ is the discount factor, γ∈[0,1), V(s t ) is the value network’s prediction of the current state, indicating the expected future return starting from this state; V(st+l ) is the prediction of the value network for the state at time step t + l, V(s t+l+1 ) is the prediction of the value network for the state at time step t + l + 1. t is the current time step, and l represents the future offset relative to the current time step t, that is, the TD error contribution from t to t + l. λ is a hyperparameter in the Generalized Advantage Estimation (GAE) used to balance the bias and variance between the TD error and the Monte Carlo estimate.

[0148] When calculating the advantage function , λ affects the smoothness of the time difference error, and λ is 0.95. T is the decay time constant; is the advantage function used to measure the quality of the current action relative to the average policy; if indicates that the current action is better than the average behavior, the policy should increase the probability of this action; otherwise, it should be decreased.

[0149] S34: Determine the policy update term t (θ) and the advantage function to determine the policy update term According to the policy update term KL divergence to obtain the policy network loss function L Rollback-IPPO (π);

[0150] S34 is the innovation of the IPPO algorithm network 300 adopted by the present invention compared with the existing IPPO algorithm. In the existing IPPO algorithm, the update of the policy network 301 only needs to perform backpropagation gradient ascent according to the clipped loss L IPPO (θ). The existing IPPO algorithm itself uses the clipped loss to limit the policy update amplitude. The clipped loss can ensure that the ratio of the new and old policies will not be too large, but still cannot completely prevent the drastic fluctuations of the policy.

[0151] The policy update term of the IPPO algorithm is:

[0152]

[0153] The policy update term is the core calculation term for IPPO to measure the improvement amplitude of the current policy relative to the old policy under a specific state-action pair. The loss function L IPPO (π) of the existing IPPO is obtained based on the clipping of the policy update term:

[0154]

[0155] Among them, ∈ is a hyperparameter for clipping, which is used to control the amplitude of the policy update; r t(θ) is the importance sampling ratio; A t is the advantage function; clip(r t (θ), 1 - ∈, 1 + ∈) is a clipping operation, aiming to limit the ratio of the old and new policies within a predetermined range to prevent excessive policy updates. Specifically, if the importance sampling ratio r t (θ) exceeds 1 - ∈, 1 + ∈, it will be clipped to ensure that the change of the policy is not too large. The clipped loss L IPPO (π) of the IPPO algorithm balances exploration and exploitation by minimizing the policy update magnitude and maximizing the advantage action selection.

[0156] In the present invention, in the perspective of the driverless vehicle, other driverless vehicles can be regarded as dynamically moving obstacles, and due to their different target point positions, their trajectories are difficult to predict. Therefore, the state space obtained in the observable perspective of the driverless vehicle changes greatly. The conventional truncation method may not be able to limit the drastic fluctuations of the policy network 301. Therefore, when the IPPO algorithm network 300 is trained, by introducing the policy update term combines the KL divergence by introducing hyperparameters and takes the average value of the current batch of samples, thereby improving the policy network loss function L Rollback-IPPO (π) with a rollback policy, limiting the magnitude of the policy network update and ensuring the stability of the training process.

[0157] In the present invention, the policy network loss function L Rollback-IPPO (π) with a rollback policy is as follows:

[0158]

[0159]

[0160] Among them, is the KL divergence; δ is the divergence threshold; is the policy update term; β is the penalty coefficient; is the single state-action pair loss, and the policy network loss function L Rollback-IPPO (π) with a rollback policy is defined by averaging all single state-action pair losses in the entire batch or trajectory.

[0161] Thus, the original IPPO policy network parameter update only performs backpropagation on the clipped loss L IPPO (π), and now it is conditionally combined with the KL divergence by the penalty coefficient β and averaged by batch to obtain the policy network loss function L Rollback-IPPO (π) with a rollback policy.

[0162] The divergence threshold δ and the penalty coefficient β control the restriction intensity and deviation amplitude of policy update, and are dynamically adjusted according to different training stages. In the initial stage of training, to encourage policy exploration and allow the policy to be updated with a large amplitude, the divergence threshold δ is set to a large constant (for example, δ = 0.03) to reduce the restriction on policy update. At the same time, the penalty coefficient β is set to a small value β = 1 to weaken the constraint of KL divergence on policy update and promote diverse exploration of the policy. In the later stage of training, as the policy gradually converges, to prevent the policy from fluctuating violently and improve the convergence stability, the divergence threshold δ is tightened to a small value δ = 0.01 to ensure that the policy update does not deviate from the learned optimal policy. At this time, the penalty coefficient β is increased to 2 to strengthen the penalty for excessive deviation, prompting the policy to be updated stably within the trust region and finally achieving convergence.

[0163] Thus, the present invention combines the IPPO policy update term and the KL divergence constraint in the policy network loss function to optimize the maximization of the cumulative discounted reward while preventing drastic changes in the policy. During the update process of the policy network 301, by calculating the ratio r t (θ) of the new and old policies, the advantage function and the KL divergence, more stable and efficient policy optimization is achieved, with a relatively fast convergence speed, stable policy update, strong exploration ability, which can effectively reduce path congestion and has a strong environmental adaptability.

[0164] S35: Update the policy network 301 (Actor) according to the policy network loss function L Rollback-IPPO (π) with a rollback policy to continue guiding the action selection of the autonomous vehicle.

[0165] S36: Update the value network 302 according to the value network loss function.

[0166] This can better evaluate the state value and assist the update of the policy network 301.

[0167] When updating the value network 302, the value network loss function L Critic (π) satisfies:

[0168]

[0169]

[0170] Among them, G t is the cumulative discounted reward, is the advantage function, V1(s t ) is the predicted value of the previously trained value network for the current state s t , V2(s t ) is the predicted value of the current value network for the state, is the expectation. It should be noted that V(s t ) used to calculate Gt and V(s Critic ) used to calculate L t (π) are not the same thing. V1(s t ) is the predicted value of the state by the previously trained value network, and V2(s t ) represents the predicted value of the state by the currently updated value network. Therefore, the two will not cancel each other out.

[0171] Step S4: When performing the path planning task of actual multi-unmanned vehicle material distribution, determine the current state according to the actual current position of the unmanned vehicle at each moment, and input the current state s t into the policy network 301 of the IPPO algorithm network 300, and output the current action a t through the policy network 301 for the unmanned vehicle to execute.

[0172] The path planning result display diagram of the multi-unmanned vehicle material distribution path planning task obtained is as shown in Figure 2 .

[0173] Figures 3A to 3C This is the result comparison diagram between the reinforcement learning path planning method for multi-unmanned vehicles in an intelligent factory using the IPPO algorithm network of the present invention and the reinforcement learning path planning method for multi-unmanned vehicles using the conventional PPO algorithm. Among them, Figure 3A shows the change in the average number of material distributions completed by the unmanned vehicle with the number of rounds, Figure 3B shows the change in the average reward of the unmanned vehicle with the number of rounds, Figure 3C shows the change in the average number of steps required for the unmanned vehicle to complete the task with the number of rounds, Figures 3A to 3C The yellow line in is the reinforcement learning path planning method for multi-unmanned vehicles in an intelligent factory using the IPPO algorithm network of the present invention, and the gray line is the reinforcement learning path planning method for multi-unmanned vehicles using the conventional PPO algorithm. Among them, the dark yellow line and gray line are the smoothed trend diagrams, which are convenient for seeing the performance gap between the two algorithms, and the light yellow line and gray line represent the original data.

[0174] Figures 3A to 3C All illustrate that the reinforcement learning path planning method for multi-unmanned vehicles in an intelligent factory using the IPPO algorithm network of the present invention has strong anti-interference ability, stable policy update, reasonable reward function design, can better and faster adapt to the dynamically changing environment, and can quickly achieve path planning.

[0175] The above are only the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various changes can be made to the above embodiments of the present invention. Any simple, equivalent changes and modifications made in accordance with the claims and the content of the specification of the present invention application shall fall within the scope of protection of the claims of the present invention patent. Those not described in detail in the present invention are all conventional technical contents.

Claims

1. A reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory, characterized in that, Including: S1: Obtain the actual layout of workstations in the factory and the initial conditions of the AGVs, and build a simulation environment for the material delivery path planning; S2: Take each AGV as an agent, interact the agent with the material delivery path planning simulation environment to extract the real-time observation information of the AGV and the positions of material demand points, and accordingly construct the state space and action space of the Markov decision model, and establish an optimization objective and a reward function with high task completion rate, few blockages and collisions, and as short a path as possible; S3: Build an IPPO algorithm network based on the Actor-Critic architecture, and train the policy of the Markov decision model based on it and the reward function; during training, improve the policy network loss function with a rollback strategy by combining the clipped loss with the KL divergence by introducing hyperparameters, and limit the update amplitude of the policy network; S4: When performing the path planning task of the actual multi-AGV material delivery, determine the current state according to the actual current position of the AGV at each moment, input the current state into the policy network, and output the current action through the policy network for the AGV to execute.

2. The reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory according to claim 1, characterized in that The specific steps of S1 include: S11: According to the actual layout of workstations in the smart factory, perform grid modeling at a fixed ratio to obtain a two-dimensional grid, and each grid cell of the two-dimensional grid corresponds to one of the free area, obstacle, AGV starting point, and material demand point; S12: Establish the mapping rules of each grid cell of the two-dimensional grid and the current position of the AGV in a two-dimensional array, which are used to map the two-dimensional grid into a multi-channel two-dimensional array; S13: Write a generator for fixed obstacles and random target points, which is used to set the obstacle positions as fixed, and randomly initialize the AGV starting point and material demand points as random target points at the beginning of each round to dynamically simulate the changes in the environment; S14: Set the movement speed of the AGV and the size of the observable window. The movement speed of the AGV is used for the simulation of each time step after the start of each round, and the observable window is used to extract the real-time observation information of the AGV.

3. The reinforcement learning path planning method for multiple unmanned vehicles for intelligent factories according to claim 2, characterized in that, In S12, the mapping rules of each grid cell of the two-dimensional grid and the current position of the AGV in the two-dimensional array are: the free area is marked as 0; the obstacle is marked as 1; the position of its own AGV is marked as 3; the positions of other AGVs are marked as 2; the position of its own target point is marked as 5; the positions of other target points are marked as 4.

4. The reinforcement learning path planning method for multi-unmanned vehicles for intelligent factories according to claim 1, characterized in that, Constructing the state space and action space of the Markov decision model specifically includes: taking each AGV as an agent, constructing the path planning task of multi-AGV material delivery in the smart factory as a Markov decision process, taking the local environmental information of the observable window of the AGV as the current state, and all possible combinations of the current states are the state space; taking the action set of the AGV as the action space, which includes five actions: up, down, left, right, and stop.

5. The reinforcement learning path planning method for multi-unmanned vehicles in an intelligent factory according to claim 1, characterized in that, The optimization objective is: The reward function includes: R(s t ,a t ) = R ext (s t ,a t ) + β·R int (s t ), R ext (s t ,a t )=R arrive +R collsion +R step , R int (s t ) = α(t) * b1(s t ) + (1 - α(t)) * b2(s t ), N(s t ) = N(s t ) + 1, where R(s t , a t ) is the total reward; E π represents the expected cumulative reward for all possible state and action combinations under policy π, and γ is the discount factor; s t represents the current state of the autonomous vehicle in the environment at the current time step t, and a t represents the current action selected by the autonomous vehicle according to the current policy π at the current time step t; R ext (s t , a t ) represents the external reward, which is the immediate reward provided by the environment after the agent interacts with the environment. It is composed of the arrival reward R arrive , the collision reward R collsion between agents and between agents and obstacles, and the step reward R step corresponding to the number of steps consumed; β is the internal reward weight; R int (s t ) represents the internal reward, α(t) represents the internal weight of the internal reward, b1(s t ) represents the error reward for deviating from the mean; b2(s t ) represents the reward for the state visit frequency; T k (s t ) represents the output feature of the k-th target network for the state; μ(s t ) represents the mean of the output features of a group of target networks for the state, f predict (s t ) represents the output feature of the prediction network for the state, N(s t ) represents the state visit frequency, T is the decay time constant, and K is the number of target networks.

6. The reinforcement learning path planning method for multi-unmanned vehicles for intelligent factories according to claim 5, characterized in that The internal reward is obtained based on distributed random network distillation. When calculating the internal reward based on distributed random network distillation, two sets of networks with the same structure are built as the target networks {T1, T2,..., T k} and the evaluation network f predict , and multiple unmanned vehicles share a set of target networks and independently have a trainable prediction network.

7. The reinforcement learning path planning method for multi-unmanned vehicles in an intelligent factory according to claim 1, characterized in that, The specific steps of S3 include: S31: Initialize the policy network and value network of the Actor-Critic architecture; S32: Have the agent interact with the environment according to the current policy to obtain trajectory data, and each sampling point of the trajectory data includes (s t , a t , R(s t , a t ), s t+1 ), s t is the current state, a t is the current action, R(s t , a t ) is the total reward, s t+1 is the next state; S33: Calculate the importance sampling ratio r based on the data of the sampling points t (θ) and KL divergence and calculate the advantage function S34: Determine the policy update term according to the importance sampling ratio r t (θ) and the advantage function to determine the policy update term According to the policy update term KL divergence to obtain the policy network loss function L with the rollback policy Rollback-IPPO (π); S35: Update the policy network according to the policy network loss function L Rollback-IPPO Rollback-IPPO (π); S36: Update the value network according to the value network loss function.

8. The reinforcement learning path planning method for multiple unmanned vehicles in an intelligent factory according to claim 1, characterized in that, The policy network loss function \(L\) with a rollback strategy Rollback-IPPO (\(\pi\)) is as follows: Among them, is the KL divergence; δ is the divergence threshold; is the policy update term; β is the penalty coefficient; is the single state-action pair loss, and the loss function L Rollback-IPPO (π) is defined by averaging all single state-action pair losses in the entire batch or trajectory.

9. The reinforcement learning path planning method for multi-unmanned vehicles for intelligent factories according to claim 8, wherein Importance sampling ratio r t is as follows: where π θ (a t |s t ) is the probability distribution of taking action a t under the current state s t , is the probability distribution of taking action a t under the current state s t by the old policy; Kullback-Leibler divergence is defined as: Among them, is the KL divergence, which is used to measure the deviation between the new and old policies, representing the difference in the probability distributions of the new and old policies. The KL divergence measures the difference in action selection by the new and old policies in each state; is the expectation; Advantage function is as follows: δ t+l = r t+l + γV(s t+l+1 ) - V(s t+l ), where δ t+l is the time difference error at time step t + l, r t+l is the total reward at time step t + l, γ is the discount factor, V(s t ) is the prediction of the value network for the current state, representing the expected future return starting from this state; V(s t+l ) is the prediction of the value network for the state at time step t + l, V(s t+l+1 ) is the prediction of the value network for the state at time step t + l + 1, t is the current time step, l represents the future offset relative to the current time step t, λ is the hyperparameter in the generalized advantage estimation; T is the decay time constant.

Images