Multi-AGV reinforcement learning scheduling method for smart factory
By building a simulation environment and deep reinforcement learning algorithm in the AGV system, the path planning and coordination problems of traditional AGV system are solved, and efficient AGV scheduling strategy training and productivity improvement are achieved.
Patent Information
- Application Number
- CN202411318147.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-12-31
AI Technical Summary
Traditional AGV systems have problems such as low efficiency, difficult training and high real-time requirements in terms of path planning, real-time adaptability and coordination, and it is difficult to effectively solve the problem of production congestion prevention in multiple AGV systems.
Using multi-AGV reinforcement learning scheduling method for smart factories, we use the factory real environment information, build a material delivery simulation environment, build a Markov decision model, and set a reward function for optimization goals, and build an Actor-Critic deep reinforcement learning algorithm framework. The final AGV scheduling strategy is trained.
The AGV scheduling method has a fast convergence speed, less training difficulty, and has low real-time requirements, which improves production efficiency and system coordination.
Smart Images
Figure CN119228045A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of smart factory logistics technology, and more specifically to a multi-AGV reinforcement learning scheduling method for smart factories. Background Art
[0002] In modern manufacturing, automated guided vehicles (AGVs) are widely used for material transportation and delivery tasks within factories. As factories expand in size and production processes become more complex, traditional AGV systems often face a series of challenges, including path planning efficiency, real-time adaptability, overall system coordination, and prevention of production congestion.
[0003] In the existing technology, there have been some studies and practices that apply reinforcement learning to multi-AGV systems, including path planning based on Q-learning, intelligent feeding strategy based on deep reinforcement learning, collaborative feeding strategy based on collaborative deep reinforcement learning, and congestion prevention based on model predictive control. These implementations are all trying to use reinforcement learning technology to solve problems such as path planning, feeding decision-making, collaborative work, and congestion prevention in multi-AGV systems. However, they all have some limitations, such as slow convergence speed, greater training difficulty, and high real-time requirements. Summary of the invention
[0004] In order to solve the above-mentioned problems in the prior art, the present invention provides a multi-AGV reinforcement learning scheduling method for smart factories. The method has a fast convergence speed, low training difficulty, and low real-time requirements.
[0005] The present invention provides a multi-AGV reinforcement learning scheduling method for a smart factory, comprising:
[0006] Step S1, obtaining the real factory environment information and building a feeding simulation environment for the smart factory;
[0007] Step S2, obtaining the real-time data of the trolley and the real-time data of the task from the feeding simulation environment, constructing a Markov decision model, and setting a reward function with high workstation importance and short task completion time as the optimization goal;
[0008] Step S3, building an Actor-Critic deep reinforcement learning algorithm framework, performing training based on the reward function, and obtaining the final AGV scheduling strategy when the reward function value meets the requirements.
[0009] Furthermore, the step S1 comprises:
[0010] Step S11, obtaining factory map information, vehicle inherent parameters and historical feeding data from the factory's built-in automation system;
[0011] Step S12, based on the historical feeding data, fitting the demand time intervals of different workstations and the demand probabilities of parts at different workstations;
[0012] Step S13, inputting the factory map information, the inherent parameters of the trolley, and the fitting results of the demand time intervals of different workstations and the demand probability of parts at different workstations into the existing simulation software to build a feeding simulation environment.
[0013] Furthermore, the step S2 comprises:
[0014] Step S21, converting the real-time data of the car and the real-time data of the task into the state matrix of the Markov decision process, and converting the scheduling action of the car into the action matrix of the Markov decision process;
[0015] Step S22, constructing an optimization goal with high workstation importance and short task completion time;
[0016] Step S23, establishing a reward function corresponding to the optimization target.
[0017] Furthermore, the optimization objective is expressed as follows:
[0018] Where, T A Indicates the completion time of all tasks; T W Indicates the difference between the target station demand time and the material collection time. represents the target station demand time when the nth car performs the ath task to transport the i-th box of parts; T D It represents the time required for all vehicles to complete all transportation tasks. It means that the nth vehicle delivers the i-th box of parts when performing the a-th mission. It indicates the time when the nth vehicle starts to deliver the i-th box of parts when performing the a-th task. It indicates the time it takes for the nth vehicle to complete the delivery of the i-th box of parts when performing the a-th task; It indicates the number of workstations corresponding to N boxes of parts on the vehicle when the vehicle performs the ath task; represents the importance per unit value of the z-th workstation, z∈[1,Z]; P z represents the actual value of the importance of the zth workstation, Y z represents the expected output of the z-th station on that day, X z Indicates the number of types of parts required for the zth station, R z j represents the usage ratio of the jth type of parts in the zth station, j∈[1,X z ], Indicates the number of the part type of the jth box of parts that appears at the zth station, f k Indicates the frequency of occurrence of the k-th type of parts in the historical feeding data.
[0019] Furthermore, the reward function is expressed as follows:
[0020] In the formula, R n-step T represents the reward for the nth car to complete a single box delivery. n-step It represents the time taken by the nth car to complete this task, Z i-step It indicates the importance of the target workstation in this task.
[0021] Furthermore, the step S3 comprises:
[0022] Step S31, using a neural network as a strategy network, and considering the feeding sequence of the AGV after the parts are loaded as a decision point;
[0023] Step S32, inputting the state matrix S in step S21 into the strategy network, and the strategy network converts the state matrix S at time t into t Mapped to action a t The probability distribution of to output the action matrix A;
[0024] Step S33, the value network is based on the state s at time t t and action a t To calculate the state value v at time t t , according to the state value v t The PPO algorithm is used to iteratively update the parameters of the policy network, and when the reward function value is maximized, the final AGV scheduling strategy is obtained.
[0025] The method of the present invention has a fast convergence speed, is less difficult to train, and has low real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a flow chart of the multi-AGV reinforcement learning scheduling method for smart factories according to the present invention. DETAILED DESCRIPTION
[0027] The preferred embodiments of the present invention are given below in conjunction with the accompanying drawings and described in detail.
[0028] like Figure 1 As shown, the multi-AGV reinforcement learning scheduling method for smart factories of the present invention includes the following steps:
[0029] Step S1, obtaining the real factory environment information and building a feeding simulation environment for the smart factory. Specifically, step S1 includes:
[0030] Step S11, obtaining factory map information, trolley inherent parameters and historical feeding data from the factory's built-in automation system. The trolley inherent parameters include the trolley loading capacity, for example, the trolley loading capacity is 10 boxes of parts each time, and each box is loaded with different types of parts.
[0031] Step S12, based on the historical material delivery data, fitting the time intervals of different station requirements and the probability of parts demand at different stations. The time intervals of different station requirements refer to the time interval between the time when the trolley currently delivers to a certain station and the time when the trolley needs to deliver to the station next time.
[0032] In this embodiment, the QQ (Quantile-Quantile) graph test method is used to fit the time intervals of different station demands. The QQ graph compares the quantiles of the observed data with the quantiles of the theoretical distribution to evaluate the degree of agreement between the two. This method intuitively shows the difference between the observed data and the theoretical distribution, and can clearly display potential distribution mismatch problems. Specifically, by analyzing the historical feeding data and the time intervals of different station demands, a station demand time interval graph is obtained, and a QQ graph test is performed on it compared with the common distribution. From the test results, it can be seen that the time intervals of different station demands are fitted with the lognormal distribution.
[0033] In this embodiment, based on historical material feeding data, the frequency of each part appearing at different workstations is used to fit the demand probability of parts at different workstations. Since different workstations require different parts, this method can more accurately reflect the demand for different parts in the factory, thereby better simulating the actual material feeding scenario.
[0034] Step S13, input the factory map information, the inherent parameters of the trolley, and the fitting results of the demand time intervals of different workstations and the demand probability of parts at different workstations into the existing simulation software to build a feeding simulation environment.
[0035] In this embodiment, the simulation software for building the feeding simulation environment is Flexsim, a process simulation software based on discrete event simulation technology. In Flexsim, the temporary storage area is used instead of the workstation, and the modeling is performed 1:1 according to the factory map information. For example, the longitudinal length of the factory is about 51m, and the transverse length is about 100m; the longitudinal length of the AGV planning line is 50m, and the transverse length is about 86m. Since the time intervals required for different workstations are fitted with the lognormal distribution, the time intervals required for different workstations are fitted in Flexsim through the built-in lognormal distribution.
[0036] Step S2, obtain the real-time data of the trolley and the real-time data of the task from the feeding simulation environment, build a Markov decision model, and set a reward function with high workstation importance and short task completion time as the optimization goal.
[0037] The Markov decision process includes: state, action, policy, reward and return. In the feeding simulation environment, the real-time data and task data of the trolley are converted into a state matrix to describe the current state in the Markov decision process. At the same time, the trolley scheduling action is converted into an action matrix so that it can get the next state matrix after inputting an action in the simulation environment.
[0038] Specifically, step S2 includes:
[0039] Step S21, convert the acquired real-time data of the car and the real-time data of the task into the state matrix S of the Markov decision process, and convert the scheduling action of the car into the action matrix A of the Markov decision process. Among them, the real-time data of the car includes the number AGVID of the car that is currently executing the task (hereinafter referred to as the "current car") and the position AVGnLOC of all cars, n = 1, 2, ..., M, M is the number of cars set in the feeding simulation environment. The real-time data of the task includes the workstation status and the part status. The workstation status includes the target workstation workstation of the current car, the location workstation_loc of the target workstation, the importance workstation_importance of the target workstation, and the information workstation_done on whether the current car transports the parts in the car to the target workstation. The part status includes the name of the part in the current car and the demand time schtime of the parts in the current car. Taking 10 cars as an example, the vectors in the state matrix are shown in the following table:
[0040] Therefore, the state matrix S is defined as (s 1 ,s 2 ,…,s 81 The current order of feeding 10 boxes of parts in the trolley is the trolley scheduling action. Therefore, the action matrix A is defined as (a 1 ,a 2 ,…,a 10 ). Combining the state space with the action space forms a complete state-action space pair. The actions that can be taken in each state determine the decision space of the system, while the representation of the state affects the observation space of the model.
[0041] Step S22, constructing an optimization goal with high workstation importance and short task completion time.
[0042] Step S22 specifically includes: constructing candidate optimization targets, using data analysis and statistical methods to determine the importance of each candidate optimization target; analyzing historical data, and determining the importance weight (i.e., importance per unit value) of each target based on the analysis results, to obtain the final optimization target with high station importance and short task completion time. Thus, more weight is allocated to the target that has a greater impact on system performance.
[0043] Candidate optimization objectives include minimizing the idle time of AGV, maximizing the task completion rate, minimizing energy consumption, etc.
[0044] Importance can be related to the urgency of the task, customer needs, task type, etc. In this embodiment, the actual importance value is a function related to the analysis results of the frequency of the parts appearing in the historical delivery records, the usage ratio of the parts, and the expected output of the day, and the importance per unit value is obtained by normalizing the actual importance values of each candidate optimization target.
[0045] Suppose there are M cars transporting parts to Z stations, and each car has Q tasks per day. n represents the number of tasks for the nth car. Each car is fully loaded with N boxes of parts each time it sets out. The number of types of parts that need to be transported every day is K. The final optimization goal of high station importance and short task completion time is as follows:
[0046] Where, T A Indicates the completion time of all tasks; T W Indicates the difference between the target station demand time and the material collection time. represents the target station demand time when the nth car performs the ath task to transport the i-th box of parts; T D It represents the time required for all vehicles to complete all transportation tasks. It means that the nth vehicle delivers the i-th box of parts when performing the a-th mission. It indicates the time when the nth vehicle starts to deliver the i-th box of parts when performing the a-th task. It indicates the time it takes for the nth vehicle to complete the delivery of the i-th box of parts when performing the a-th task; It indicates the number of workstations corresponding to N boxes of parts on the vehicle when the vehicle performs the ath task; represents the importance per unit value of the z-th workstation, z∈[1,Z]; P z represents the actual value of the importance of the zth workstation, Yz represents the expected output of the z-th station on that day, X z Indicates the number of types of parts required for the zth station, represents the usage ratio of the jth type of parts in the zth station, j∈[1,X z ], Indicates the number of the part type of the jth box of parts that appears at the zth station, f k Indicates the frequency of occurrence of the k-th type of parts in the historical feeding data.
[0047] Step S23, establishing a reward function corresponding to the optimization target of step S22.
[0048] For the optimization goal, it needs to be expressed algorithmically and converted into the form of a reward function for optimization during the training process. These optimization goals can be expressed through reward functions, for example, positive rewards are given when the task is completed and negative rewards are given when the task is delayed, so as to learn the optimal scheduling strategy below.
[0049] The task completion time is crucial to the efficiency of the smart factory. Therefore, the reciprocal of the task completion time is used as the reward, that is, the faster the task is completed, the higher the reward. Such a reward function can motivate the agent to handle all tasks as quickly as possible and improve production efficiency. Since the optimization objective formula contains both the station importance and the completion time of all tasks, the reward function is expressed as follows:
[0050] In the formula, R n-step T represents the reward for the nth car to complete a single box delivery. n-step It represents the time taken by the nth car to complete this task, Z i-step It indicates the importance of the target workstation in this task.
[0051] Step S3, build the Actor-Critic deep reinforcement learning algorithm framework, perform training based on the reward function set in step S2, and obtain the final AGV scheduling strategy when the reward function value meets the requirements. Specifically, step S3 includes:
[0052] In step S31, a convolutional neural network is used as a policy network (i.e., an Actor network), and the feeding sequence of the AGV after parts loading is regarded as a decision point.
[0053] Step S32: at the decision point, the state matrix S in step S21 is input into the policy network, and the policy network converts the state s at time t into t Mapped to action a t The probability distribution of , thus outputting the action matrix A;
[0054] The policy network corresponds to a policy π, which represents the action a taken in state s.
[0055] Step S33: the value network (i.e., the Critic network) is based on the state s at time t. t and action a t To calculate the state value v at time t t , according to the state value v t The PPO (Proximal Policy Optimization) algorithm is used to iteratively update the parameters θ of the policy network. When the reward function value is maximized, the final AGV scheduling strategy is obtained, that is, the optimal feeding sequence after the AGV trolley loads parts.
[0056] The principle of the PPO algorithm is as follows:
[0057] (1) Policy network replica θ old :Interact with the environment to sample batch data for iterative update of policy network parameters. Interact with the environment, select action A according to the current state S, and get reward value R; S, A, R are stored in the experience replay pool. Network parameters θ old Copied regularly from Theta.
[0058] (2) Policy network: Update the network parameters θ multiple times based on the data in the experience replay pool.
[0059] (3) Value network: evaluate state value and estimate advantage function.
[0060] The above is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. The above embodiment of the present invention can also be modified in various ways. That is, all simple, equivalent changes and modifications made according to the claims and the description of the present invention fall within the scope of protection of the claims of the present invention. The contents not described in detail in the present invention are all conventional technical contents.
Claims
1. A multi-AGV reinforcement learning scheduling method for smart factories, characterized in that: include: Step S1, obtaining the real factory environment information and building a feeding simulation environment for the smart factory; Step S2, obtaining the real-time data of the trolley and the real-time data of the task from the feeding simulation environment, constructing a Markov decision model, and setting a reward function with high workstation importance and short task completion time as the optimization goal; Step S3, building an Actor-Critic deep reinforcement learning algorithm framework, performing training based on the reward function, and obtaining the final AGV scheduling strategy when the reward function value meets the requirements.
2. The multi-AGV reinforcement learning scheduling method for smart factories according to claim 1 is characterized in that: The step S1 comprises: Step S11, obtaining factory map information, vehicle inherent parameters and historical feeding data from the factory's built-in automation system; Step S12, based on the historical feeding data, fitting the demand time intervals of different workstations and the demand probabilities of parts at different workstations; Step S13, inputting the factory map information, the inherent parameters of the trolley, and the fitting results of the demand time intervals of different workstations and the demand probability of parts at different workstations into the existing simulation software to build a feeding simulation environment.
3. The multi-AGV reinforcement learning scheduling method for smart factories according to claim 1 is characterized in that: The step S2 comprises: Step S21, converting the real-time data of the car and the real-time data of the task into the state matrix of the Markov decision process, and converting the scheduling action of the car into the action matrix of the Markov decision process; Step S22, constructing an optimization goal with high workstation importance and short task completion time; Step S23, establishing a reward function corresponding to the optimization target.
4. The multi-AGV reinforcement learning scheduling method for smart factories according to claim 3 is characterized in that: The optimization objective is expressed as follows: T A =T W +T D , Where, T A Indicates the completion time of all tasks; T W Indicates the difference between the target station demand time and the material collection time. represents the target station demand time when the nth car performs the ath task to transport the i-th box of parts; T D It represents the time required for all vehicles to complete all transportation tasks. It means that the nth vehicle delivers the i-th box of parts when performing the a-th mission. It indicates the time when the nth vehicle starts to deliver the i-th box of parts when performing the a-th task. It represents the time it takes for the nth vehicle to complete the delivery of the i-th box of parts when performing the a-th task; D na ∈(1,N] represents the number of workstations corresponding to N boxes of parts on the nth vehicle when the vehicle performs the ath task; represents the importance per unit value of the z-th workstation, z∈[1,Z]; P z represents the actual value of the importance of the zth workstation, Y z represents the expected output of the z-th station on that day, X z Indicates the number of types of parts required for the zth station, represents the usage ratio of the jth type of parts in the zth station, j∈[1,X z ], Indicates the number of the part type of the jth box of parts that appears at the zth station, f k Indicates the frequency of occurrence of the k-th type of parts in the historical feeding data.
5. The multi-AGV reinforcement learning scheduling method for smart factories according to claim 4 is characterized in that: The reward function is expressed as follows: In the formula, R n-step T represents the reward for the nth car to complete a single box delivery. n-step It represents the time taken by the nth car to complete this task, Z i-step It indicates the importance of the target workstation in this task.
6. The multi-AGV reinforcement learning scheduling method for smart factories according to claim 3 is characterized in that: The step S3 comprises: Step S31, using a neural network as a strategy network, and considering the feeding sequence of the AGV after the parts are loaded as a decision point; Step S32, inputting the state matrix S in step S21 into the strategy network, and the strategy network converts the state matrix S at time t into t Mapped to action a t The probability distribution of to output the action matrix A; Step S33, the value network is based on the state s at time t t and action a t To calculate the state value v at time t t , according to the state value v t The PPO algorithm is used to iteratively update the parameters of the policy network, and when the reward function value is maximized, the final AGV scheduling strategy is obtained.