Improved PPO scheduling method, system, medium and equipment in AGV composite operation mode of electronic component workshop

Abstract

The invention relates to the field of electronic component workshop AGV logistics scheduling, and discloses an improved PPO scheduling method and system in an electronic component workshop AGV composite operation mode, a medium and equipment, and the method comprises the steps: carrying out the rasterization of a workshop environment map through grid modeling, enabling the AGV to carry out an A * search algorithm in a grid map through a plurality of task orders, obtaining batch sample data, and carrying out the calculation of the A * search algorithm in the grid map; pre-training is completed according to model parameters of the sample pre-training strategy network; according to the new task order, the AGV interacts with the environment, an observation state, an action space and a reward value are obtained, after matrix processing is carried out on the observation state, the observation state serves as input data of a PPO algorithm, a strategy network loss value and a value network loss value of the PPO algorithm are calculated, network model parameters are updated through a gradient descent method, and PPO training is completed; and generating a real-time task order on line, and guiding the AGV to complete path planning and scheduling tasks through the decision network models trained in the first two stages. According to the method, the electronic component workshop AGV scheduling strategy can be obtained through rapid learning, the learning efficiency is remarkably improved, and stability is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of AGV logistics scheduling in an electronic component workshop, and in particular to an improved PPO scheduling method, system, medium and equipment in an AGV composite operation mode in an electronic component workshop. Background Art

[0002] In the manufacturing of electronic components, due to the requirements of the manufacturing environment, it is necessary to maintain a clean space. AGV (Automated Guided Vehicle), as an important part of the automated logistics system of electronic component workshops, has been widely used in intelligent manufacturing and automated warehousing. With the development of automation technology, the task of transferring raw materials and workpieces in the manufacturing of electronic component workshops is gradually undertaken by AGV. Completing the task of transferring workpieces and raw materials in the earliest time and improving the utilization rate of production resources in the workshop has become the key to the development of the manufacturing industry.

[0003] Traditional AGV path planning and task scheduling algorithms can be roughly divided into heuristic algorithms and intelligent optimization algorithms, such as genetic algorithms, ant colony algorithms, Dijkstra algorithms, and A* algorithms. The above algorithms and their variants have been widely used to solve AGV pathfinding and scheduling problems. Traditional algorithms or algorithm variants can sometimes quickly obtain workshop scheduling solutions, but when the manufacturing workshop environment and task complexity increase, infeasible solutions may be generated. In order to improve the quality of the solution, it is usually necessary to set complex heuristic rules, and these rules constrain each other, and sometimes even fall into a loop or contradict each other. Therefore, it is difficult to find a feasible solution that meets all rules.

[0004] In recent years, with the development of artificial intelligence technology, in order to overcome the limitations of heuristic search algorithms in multi-AGV scheduling in complex environments, reinforcement learning (RL) is a method that can automatically learn through interaction with the environment. The trial-and-error exploration and feedback evaluation of the environment enable the multi-AGV system to have environmental learning and adaptability, which is conducive to global path planning for multiple AGVs and shows great potential in solving AGV scheduling problems. Such as Q-learning, DQN (Deep Q-Network), PPO (Proximal Policy Optimization), etc. An effective and well-performing adaptive memetic algorithm disclosed in the existing literature combines Q-learning to solve the AGVDP problem. However, the AGVDP problem of this method only considers the planning of the production cycle and does not consider the obstacles and conflicts in the vehicle path process. Whether it is a reinforcement learning algorithm based on value or policy network, in a complex environment, the initial exploration of the intelligent agent is prone to fall into the local optimal solution. In the complex environment of a manufacturing workshop with many obstacles and many types of production, the computational complexity of the reinforcement learning method increases, and it is easy to fall into the local optimal solution, resulting in poor decision-making performance and slow training speed. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide an improved PPO scheduling method, system, medium and equipment in the AGV composite operation mode of an electronic component workshop, which can quickly learn and obtain scheduling strategies, significantly improve learning efficiency and have stability.

[0006] To achieve the above-mentioned purpose, in the first aspect, the technical solution adopted by the present invention is: an improved PPO scheduling method under the composite operation mode of AGV in an electronic component workshop, which includes: using grid modeling to rasterize the environmental map of the electronic component workshop, and using multiple task orders to enable the AGV to perform an A* search algorithm on the grid map to obtain batches of sample data, and pre-train the model parameters of the strategy network according to the samples to complete pre-training; according to the new task order, the AGV interacts with the environment to obtain the observation state, action space and reward value, and after matrixing the observation state, it is used as the input data of the PPO algorithm, and the strategy network and value network loss values ​​of the PPO algorithm are calculated, and the network model parameters are updated by the gradient descent method to complete the PPO training; real-time task orders are generated online, and the decision network models trained in the first two stages are used to guide the AGV to complete the path planning and scheduling tasks.

[0007] Furthermore, the grid modeling is used to grid the environment map of the electronic component workshop. Through multiple task orders, the AGV performs the A* search algorithm on the grid map to obtain batches of sample data. The model parameters of the strategy network are pre-trained according to the samples, including:

[0008] Use the A* algorithm to guide AGV to continuously complete batch tasks in the grid map, save the node information of each task action trajectory, and obtain batch sample experience;

[0009] N sample data are extracted and passed into the Actor network, and the network output result is the probability distribution of multiple sample action spaces;

[0010] Select actions based on probability to calculate the logarithmic probability value of the action, calculate the loss value of the Actor network, perform backpropagation based on the loss value to find the gradient, and update the model parameters based on the gradient.

[0011] Furthermore, according to the new task order, the AGV interacts with the environment to obtain the observation state, action space and reward value. After matrix processing, the observation state is used as the input data of the PPO algorithm, including:

[0012] After pre-training the model parameters, the Actor network and Actor network model are further trained through the PPO algorithm;

[0013] Before training, new tasks are assigned, and the A* search algorithm guides the AGV to interact with the grid map environment, saves the state and action trajectory node information of each interaction, and generates the layout matrix, current position matrix, target position matrix and action list according to the environment layout, AGV state and action information as the input data of the PPO algorithm.

[0014] Furthermore, the current observation state s is stored according to the number of consecutive interaction steps t , corresponding action a t , batch action probability distribution and the batch reward value r t+1 ;

[0015] According to s t and a t The new strategy for computing π θ (a t |s t ) and the old strategy Calculate the ratios, calculate the cumulative discounted reward value R based on the reward value and the discount factor γ, and thus calculate the advantage function value A_k;

[0016] Through ratios, A_k, R, state value function V Ф (s t ) and the cutoff value ∈ calculate the loss function L(θ) of the Actor network and the loss function L(Ф) of the Critic network, and use the gradient descent method to update the model parameters of the Actor network and the Critic network.

[0017] Furthermore, the advantage function value A_k is given by the advantage function A t The advantage function A is calculated t for:

[0018] A t =Q t (s t ,a t )-V t (s t )

[0019] In the formula, Q t (s t ,a t ) is the action value function, V t (s t ) is the value function at time t, and the advantage function A t Indicates taking action a t Relative to the average action in state s t additional returns.

[0020] Furthermore, the loss function is:

[0021] L(*)=min(ratios×A t ,clip(ratios,1-∈,1+∈)A t )

[0022] Where * is the current strategy parameter, which is θ or Ф; θ k is the parameter of the old strategy; ∈ is a hyperparameter, which refers to the update amplitude between the new strategy and the old strategy; the clipping function clip(ratios,1-∈,1+∈) limits the value of ratios to the range of [1-∈,1+∈]; the strategy ratio ratios is the new strategy in state s t Next select action a t The probability of the old policy in state s t Next select action a t The ratio of the probabilities.

[0023] Furthermore, after the pre-training is completed for a set number of times, PPO training is performed. When the agent is trained to the point where it can obtain multiple maximum reward values ​​in succession, the pre-training is terminated.

[0024] After pre-training, the PPO algorithm is used for training. When the agent can continuously and stably obtain the maximum cumulative reward, the training ends, the optimal strategy is output, and the Actor and Critic network models are saved.

[0025] In the second aspect, the technical solution adopted by the present invention is: an improved PPO scheduling system under the composite operation mode of AGV in an electronic component workshop, which includes: a pre-training model, which uses grid modeling to rasterize the workshop environment map, and uses multiple task orders to enable the AGV to perform an A* search algorithm on the grid map to obtain batches of sample data, and pre-train the model parameters of the strategy network according to the samples to complete pre-training; a PPO training model, according to the new task order, the AGV interacts with the environment to obtain the observation state, action space and reward value, and after matrixing the observation state, it is used as the input data of the PPO algorithm, and the strategy network and value network loss values ​​of the PPO algorithm are calculated, and the network model parameters are updated by the gradient descent method to complete the PPO training; a real-time scheduling module, which generates real-time task orders online, and guides the AGV to complete the path planning and scheduling tasks through the decision network models trained in the first two stages.

[0026] In a third aspect, the technical solution adopted by the present invention is: a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device executes any one of the above methods.

[0027] In a fourth aspect, the technical solution adopted by the present invention is: a computing device, comprising: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0028] The present invention adopts the above technical solution, which has the following advantages:

[0029] 1. The present invention uses the AGV scheduling experience samples obtained by the A* algorithm to initialize the training convolutional neural network model optimization parameters, and uses PPO to train the intelligent agent, which can quickly learn and obtain the scheduling strategy. The single AGV scheduling reward value is increased by 4.67 times, and the multi-AGV scheduling reward value is increased by 4.5 times. The learning efficiency of the current algorithm is significantly improved and stable.

[0030] 2. Compared with the optimized DQN, AC and PG algorithms, the convergence speed of the improved PPO algorithm of the present invention is increased by 50.6%, 28.3% and 83.4% respectively. The improved PPO algorithm is not only suitable for the scheduling task of a single AGV, but also has strong adaptability and superiority for the complex scheduling task of multiple AGVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is an overall flow chart of the improved PPO scheduling method in the AGV composite operation mode of the electronic component workshop in an embodiment of the present invention;

[0032] Figure 2 is a detailed flow chart of pre-training and PPO training in an embodiment of the present invention;

[0033] Figure 3 : is the training loss curve of the PPO algorithm before and after improvement in the embodiment of the present invention;

[0034] Figure 4 is the reward value convergence curve of the A* algorithm pre-training + PPO algorithm in the embodiment of the present invention;

[0035] Figure 5 It is a comparison chart of the average task completion time of different numbers of AGVs in the PPO algorithm before and after improvement in the embodiment of the present invention. DETAILED DESCRIPTION

[0036] The scheduling efficiency of automated guided vehicles (AGVs) plays a vital role in the intelligent manufacturing logistics operations of electronic component workshops. Under the complex operation mode, AGVs need to complete a variety of complex tasks, and shortening the total task time becomes the core of scheduling optimization. To this end, a scheduling method based on the improved proximal policy optimization (PPO) algorithm is proposed. First, the A* (A-star) algorithm is used to obtain scheduling experience samples in a complex workshop environment, and the neural network parameters are initialized. Then, the improved PPO algorithm is used to train the intelligent agent to optimize the scheduling strategy. Experimental results show that the improved PPO algorithm significantly improves the scheduling efficiency compared with the traditional method.

[0037] In view of the intelligent manufacturing of electronic components, the equipment is divided into clean rooms and non-clean rooms. The workshop has high requirements for the storage of work-in-progress products, and the product varieties and batches are diverse. The complex logistics environment leads to the complex and changeable AGV logistics handling targets. The traditional heuristic algorithm does not meet the demand for solving the scheduling problem. The present invention proposes an improved PPO scheduling method, system, medium and equipment under the AGV composite operation mode of the electronic component workshop. First, the A* algorithm is used to obtain the experience knowledge pre-training strategy network to solve the problem that reinforcement learning is easy to fall into the local optimal solution in the early stage; secondly, the PPO algorithm is used to continue the training of the intelligent agent under real-time random multi-tasks to enhance the adaptability to the environment and improve the learning efficiency and performance of the intelligent agent. The present invention realizes the shortest total time for random multi-task scheduling of different numbers of AGVs by improving the AGV scheduling method of the PPO algorithm. By designing the state space, action space and convolutional neural network model of reinforcement learning, the intelligent agent can make decisions quickly to complete the target task. The improved PPO algorithm is compared with other reinforcement learning algorithms to highlight its advantages in decision-making in the AGV scheduling process.

[0038] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0040] In one embodiment of the present invention, an improved PPO scheduling method in an AGV composite operation mode in an electronic assembly workshop is provided. In this embodiment, an example of application in a manufacturing workshop is used for explanation. Figure 1 As shown, the method comprises the following steps:

[0041] 1) Use grid modeling to rasterize the workshop environment map, and use multiple task orders to make the AGV perform the A* search algorithm on the grid map to obtain batches of sample data. According to the model parameters of the sample pre-training strategy network, complete the pre-training;

[0042] 2) According to the new task order, AGV interacts with the environment to obtain the observation state, action space and reward value. After matrix processing, the observation state is used as the input data of the PPO algorithm, the policy network and value network loss values ​​of the PPO algorithm are calculated, and the network model parameters are updated through the gradient descent method to complete the PPO training;

[0043] 3) Generate real-time task orders online, and guide the AGV to complete path planning and scheduling tasks through the decision network model trained in the first two stages.

[0044] In the above step 1), according to the structure of the PPO algorithm, the observation space of the agent, the action space of the agent, the reward value structure and the convolutional neural network structure for processing the observation space are designed to adapt to the AGV pathfinding and handling tasks in the manufacturing workshop with complex environment.

[0045] In this embodiment, the A* algorithm is a node traversal method based on path finding, which determines the shortest path by searching all possible nodes. This feature of comprehensively searching all possible nodes in a certain order and storing them in memory enables the A* algorithm to generate the best path, and it is superior to most other algorithms.

[0046] According to the A* algorithm, the action trajectory node information of the AGV and the environment interacting successfully to reach the target point when executing each task can be obtained. It is saved in the sample pool for pre-training of the PPO algorithm strategy network.

[0047] In this embodiment, since the exploration space of the electronic component manufacturing workshop is large and the state is complex, during the initial PPO algorithm training process, the agent is prone to fall into a local loop of suboptimal strategies, making it difficult to further optimize the policy network. To solve this problem, the present invention introduces behavioral cloning or other imitation learning algorithms before the formal training of the improved PPO algorithm to pre-train the policy network so that it learns the expert's strategy. After pre-training, the policy network has better initial capabilities and can explore and optimize strategies more efficiently in subsequent PPO training, thereby improving the overall training effect.

[0048] In this embodiment, the workshop environment map is rasterized by using grid modeling, and the AGV is made to perform the A* search algorithm on the grid map through multiple task orders to obtain batches of sample data. The model parameters of the strategy network are pre-trained according to the samples, such as Figure 2 As shown, the following steps are included:

[0049] 1.1) Use the A* algorithm to guide AGV to continuously complete batch tasks in the grid map, save the node information of each task action trajectory, and obtain batch sample experience;

[0050] 1.2) N sample data are extracted and passed into the Actor network, and the network output is the probability distribution of multiple sample action spaces;

[0051] 1.3) Select actions based on probability to calculate the logarithmic probability value of the action, calculate the loss value of the Actor network, perform backpropagation based on the loss value to find the gradient, and update the model parameters based on the gradient.

[0052] Specifically, the A* algorithm is used to guide the AGV to interact with the environment to complete 14 tasks, save the state and action trajectory node information of each interaction, and generate the layout matrix, current position matrix, target position matrix and action list according to the environment layout, AGV state and action information. This is a sampling process. Then, the program will randomly regenerate different assigned tasks until 600 sample data are sampled and 512 samples are extracted to pre-train the policy network. A set of experience data obtained by the A* algorithm is extracted, including N (512) samples, each sample consists of state s i and the corresponding action a i Composition. The policy network of the model is given a state s i Produce action a i The probability of i |si ). Then for each sample i, the corresponding logarithmic probability is logπ(a i |s i ). Then the loss function calculation formula of the policy network is as follows:

[0053]

[0054] For L BC Perform backpropagation and update the parameters of the policy network model, and use this new policy network for reinforcement learning.

[0055] The PPO algorithm is an online strategy algorithm that mainly solves high-dimensional decision-making problems in continuous action space. It uses the clipped probability ratio to limit the amplitude of each strategy network parameter update, making the new strategy better than the old strategy. The update steps of the PPO algorithm are as follows:

[0056] (1) Collect sample data, perform several simulations according to the current strategy π(a|s), and record the state, action, reward, and next state.

[0057] (2) Calculate the advantage function A t , use a value function to estimate the current strategy, the calculation formula is as follows:

[0058] Q t (s t ,a t )=∑ s′ P(s′|s t ,a t )[R(s t ,a t ,s′)+γV t (s′)] (1)

[0059] V t (s t )=max a Q t (s t ,a) (2)

[0060] A t =Q t (s t ,a t )-V t (s t ) (3)

[0061] Among them, Q t (s t ,a t ) is a function (action-value function) that represents the expected return after taking an action starting from state at time , where P(s′|s t,a t ) refers to the state s at time t t Take action a t The probability of transitioning to state s′ is R(s t ,a t ,s′) is the corresponding reward function, γ is the discount factor, V t (s′) is the value function at state s′; V t (s t ) is the value function at time t, that is, at time t, in state s t The maximum expected return is obtained by maximizing the Q function of all possible actions a; the advantage function A t Indicates taking action a t Relative to the average action in state s t additional returns.

[0062] (3) Calculate the strategy ratios, that is, the new strategy in state s t Next select action a t The probability of the old policy in state s t Next select action a t The probability ratio is calculated as follows:

[0063]

[0064] (4) Calculate the objective function and use the clipping function clip(ratios,1-∈,1+∈) to limit the value of ratios to the range of [1-∈,1+∈] to limit the amplitude of the policy update:

[0065] L(s,a,θ k ,θ)=min(ratios×A t ,clip(ratios,1-∈,1+∈)A t ) (5)

[0066] θ is the current policy parameter, θ k is the parameter of the old strategy; ∈ is a hyperparameter, which refers to the update amplitude between the new strategy and the old strategy.

[0067] (5) Obtain the next strategy parameter θ by optimizing the objective function k+1 , using gradient descent:

[0068]

[0069] In the above step 2), according to the new task order, the AGV interacts with the environment to obtain the observation state, action space and reward value, and the observation state is matrixed as the input data of the PPO algorithm, including the following steps:

[0070] 2.1.1) After pre-training the model parameters, continue to train the Actor network and Actor network model through the PPO algorithm;

[0071] 2.1.2) Before training, new tasks are assigned and the A* search algorithm guides the AGV to interact with the grid map environment. The state and action trajectory node information of each interaction is saved. According to the environment layout, the state and action information of the AGV, the layout matrix, current position matrix, target position matrix and action list are generated as the input data of the PPO algorithm.

[0072] In the above step 2), the policy network and value network loss values ​​of the PPO algorithm are calculated, and the network model parameters are updated by the gradient descent method, such as Figure 2 As shown, the following steps are included:

[0073] 2.2.1) Store the current observation state s of small batches according to the number of consecutive interaction steps t , the corresponding batch action a t , batch action probability distribution and the batch reward value r t+1 ;

[0074] Among them, the batch action probability distribution The strategy π refers to the probability θ of executing action a under the observation state at time t; the batch reward value r t+1 It refers to the reward value of executing strategy π, and the reward value is obtained at time t+1.

[0075] 2.2.2) According to s t and a t The new strategy for computing π θ (a t |s t ) and the old strategy Calculate the ratios, calculate the cumulative discounted reward value R based on the batch reward value and the discount factor γ, and thus calculate the advantage function value A_k;

[0076] In this embodiment, in the manufacturing workshop layout, after the AGV selects an action, the system will give corresponding reward feedback. The results of executing the action behavior are generally divided into three categories: the first category is to reach the target position, and the corresponding reward value is 1. The second category is to perform abnormal actions, including collision with obstacles and driving out of the boundary. The corresponding reward value in this case is -1. The third category is normal exploration, which neither reaches the target position nor encounters abnormal situations, and the corresponding reward value is 0. The reward value R for executing the task is defined as follows:

[0077]

[0078] 2.2.3) Through ratios, A_k, R, state value function V Ф (s t ) and the cutoff domain value ∈ calculate the loss function L(θ) of the Actor network and the loss function L(Ф) of the Critic network, and use the gradient descent method to update the model parameters of the Actor network and the Critic network.

[0079] In this embodiment, according to the above formula (5), the loss function can be obtained as follows:

[0080] L(*)=min(ratios×A t ,clip(ratios,1-∈,1+∈)A t )

[0081] Where * is the current strategy parameter, which is θ or Ф.

[0082] In the above embodiment, PPO training is performed after pre-training is completed a set number of times, and the pre-training is terminated when the agent is able to obtain the maximum reward value multiple times in succession after training; for example, the pre-training is terminated when the agent is able to obtain the maximum reward value 10 times in succession after training for a certain period of time.

[0083] After pre-training, the PPO algorithm is used for training. When the agent can continuously and stably obtain the maximum cumulative reward, the training ends, the optimal strategy is output, and the Actor and Critic network models are saved. For example, the optimal strategy is output when the maximum cumulative reward is obtained 30 times in a row.

[0084] In one embodiment of the present invention, an improved PPO scheduling system in an AGV composite operation mode of an electronic assembly workshop is provided, which includes:

[0085] Pre-training model, using grid modeling to rasterize the workshop environment map, through multiple task orders, the AGV performs the A* search algorithm on the grid map, obtains batches of sample data, and completes pre-training based on the model parameters of the sample pre-training strategy network;

[0086] PPO training model, according to the new task order, AGV interacts with the environment to obtain the observation state, action space and reward value, and after matrix processing the observation state, it is used as the input data of the PPO algorithm, and the policy network and value network loss values ​​of the PPO algorithm are calculated. The network model parameters are updated through the gradient descent method to complete the PPO training;

[0087] The real-time scheduling module generates real-time task orders online and guides the AGV to complete path planning and scheduling tasks through the decision network model trained in the first two stages.

[0088] In the above embodiment, in the pre-training model, the workshop environment map is rasterized by using grid modeling, and the AGV is made to perform the A* search algorithm on the grid map through multiple task orders to obtain batches of sample data. The model parameters of the strategy network are pre-trained according to the samples, including:

[0089] Use the A* algorithm to guide AGV to continuously complete batch tasks in the grid map, save the node information of each task action trajectory, and obtain batch sample experience;

[0090] N sample data are extracted and passed into Actor Net, and the network output result is the probability distribution of multiple sample action spaces;

[0091] Select actions based on probability to calculate the logarithmic probability value of the action, calculate the loss value of the Actor network, perform backpropagation based on the loss value to find the gradient, and update the model parameters based on the gradient.

[0092] In the above embodiment, according to the new task order, the AGV interacts with the environment to obtain the observation state, action space and reward value, and the observation state is matrixed as the input data of the PPO algorithm, including:

[0093] After pre-training the model parameters, the Actor network and Actor network model are further trained through the PPO algorithm;

[0094] Before training, new tasks are assigned, and the A* search algorithm guides the AGV to interact with the grid map environment, saves the state and action trajectory node information of each interaction, and generates the layout matrix, current position matrix, target position matrix and action list according to the environment layout, AGV state and action information as the input data of the PPO algorithm.

[0095] In the above embodiment, the policy network and value network loss values ​​of the PPO algorithm are calculated, and the network model parameters are updated by the gradient descent method, including:

[0096] Store the current observation state s of a small batch according to the number of consecutive interaction steps t , the corresponding batch action a t , batch action probability distribution and the batch reward value r t+1 ;

[0097] According to s t and a t The new strategy for computing π θ (a t |s t ) and the old strategy Calculate the ratios, calculate the cumulative discounted reward value R based on the batch reward value and the discount factor γ, and thus calculate the advantage function value A_k;

[0098] In the above embodiment, PPO training is performed after the pre-training is completed for a set number of times, and the pre-training is terminated when the agent can obtain the maximum reward value 10 times in a row after a certain period of training;

[0099] After pre-training, the PPO algorithm is used for training. When the agent can continuously and stably obtain the maximum cumulative reward, the training ends, the optimal strategy is output, and the Actor and Critic network models are saved.

[0100] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0101] like Figure 3 The following is a comparison of the training loss curves of the PPO algorithm before and after the improvement. Figure 4 This is the reward value convergence curve of A* algorithm pre-training + PPO algorithm, which shows that the improved PPO algorithm has significant advantages in convergence speed and loss value stability.

[0102] like Figure 5 As shown in the figure, the blue column represents the PPO algorithm, and the orange column represents the improved PPO algorithm. The red arrows in the figure indicate the improvement of each set of data, indicating that the improved PPO algorithm can greatly improve the efficiency of task completion, especially in multi-AGV scheduling. This provides strong support for the application of intelligent scheduling algorithms in complex logistics environments.

[0103] In summary, the scheduling efficiency of automated guided vehicles (AGVs) plays a vital role in the logistics operation of smart factories. Under the complex operation mode, AGVs need to complete a variety of complex tasks, and shortening the total task time becomes the core of scheduling optimization. To this end, a scheduling method based on the improved proximal policy optimization (PPO) algorithm is proposed. First, the A* (A-star) algorithm is used to obtain scheduling experience samples in a complex workshop environment, and the neural network parameters are initialized. Then, the improved PPO algorithm is used to train the intelligent agent to optimize the scheduling strategy. The experimental results show that the improved PPO algorithm significantly improves the scheduling efficiency compared with the traditional method. The actual factory application results show that the improved PPO algorithm effectively shortens the average completion time of different numbers of AGVs in random tasks, with a maximum reduction of 42.7%.

[0104] A computing device provided in one embodiment of the present invention may be a terminal, which may include: a processor, a communications interface, a memory, a display screen, and an input device. The processor, the communications interface, and the memory communicate with each other through a communication bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory, wherein the non-volatile storage medium stores an operating system and a computer program, and the computer program is executed by the processor to implement the methods in the above embodiments; the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a management network, NFC (near field communication) or other technologies. The display screen may be a liquid crystal display screen or an electronic ink display screen, and the input device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computing device, or an external keyboard, touchpad or mouse, etc. The processor may call the logic instructions in the memory.

[0105] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0106] In one embodiment of the present invention, a computer program product is provided, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments.

[0107] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores server instructions, and the computer instructions enable a computer to execute the methods provided in the above embodiments.

[0108] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An improved PPO scheduling method in the AGV composite operation mode of an electronic component workshop, characterized in that: include: The electronic component workshop environment map is rasterized using grid modeling. Through multiple task orders, the AGV performs the A* search algorithm on the grid map to obtain batches of sample data. The pre-training is completed based on the model parameters of the sample pre-training strategy network. According to the new task order, AGV interacts with the environment to obtain the observation state, action space and reward value. After matrix processing, the observation state is used as the input data of the PPO algorithm, the policy network and value network loss values ​​of the PPO algorithm are calculated, and the network model parameters are updated through the gradient descent method to complete the PPO training; Generate real-time task orders online, and use the decision network model trained in the first two stages to guide the AGV to complete path planning and scheduling tasks.

2. The improved PPO scheduling method in the AGV composite operation mode of the electronic component workshop as claimed in claim 1 is characterized in that: The grid modeling is used to grid the environment map of the electronic component workshop. Through multiple task orders, the AGV performs the A* search algorithm on the grid map to obtain batches of sample data. The model parameters of the strategy network are pre-trained according to the samples, including: Use the A* algorithm to guide AGV to continuously complete batch tasks in the grid map, save the node information of each task action trajectory, and obtain batch sample experience; N sample data are extracted and passed into the Actor network, and the network output result is the probability distribution of multiple sample action spaces; Select actions based on probability to calculate the logarithmic probability value of the action, calculate the loss value of the Actor network, perform backpropagation based on the loss value to find the gradient, and update the model parameters based on the gradient.

3. The improved PPO scheduling method in the AGV composite operation mode of the electronic component workshop as claimed in claim 1 is characterized in that: According to the new task order, the AGV interacts with the environment to obtain the observation state, action space and reward value. After matrix processing, the observation state is used as the input data of the PPO algorithm, including: After pre-training the model parameters, the Actor network and Actor network model are further trained through the PPO algorithm; Before training, new tasks are assigned, and the A* search algorithm guides the AGV to interact with the grid map environment, saves the state and action trajectory node information of each interaction, and generates the layout matrix, current position matrix, target position matrix and action list according to the environment layout, AGV state and action information as the input data of the PPO algorithm.

4. The improved PPO scheduling method in the AGV composite operation mode of the electronic component workshop as claimed in claim 1 is characterized in that: Calculate the policy network and value network loss values ​​of the PPO algorithm, and update the network model parameters by gradient descent, including: Store the current observation state s according to the number of consecutive interaction steps t , corresponding action a t , batch action probability distribution and the batch reward value r t+1 ; According to s t and a t The new strategy for computing π θ (A t |s t ) and the old strategy Calculate the ratios, calculate the cumulative discounted reward value R based on the reward value and the discount factor γ, and thus calculate the advantage function value A_k; Through ratios, A_k, R, state value function V Φ (s t ) and the cutoff domain value ∈ calculate the loss function L(θ) of the Actor network and the loss function L(Φ) of the Critic network, and use the gradient descent method to update the model parameters of the Actor network and the Critic network.

5. The improved PPO scheduling method in the AGV compound operation mode of the electronic component workshop as claimed in claim 4 is characterized in that: The advantage function value A_k is given by the advantage function A t The advantage function A is calculated t for: A t =Q t (s t ,a t )-V t (s t ) In the formula, Q t (s t , a t ) is the action value function, V t (s t ) is the value function at time t, and the advantage function A t Indicates taking action a t Relative to the average action in state s t additional returns.

6. The improved PPO scheduling method in the AGV compound operation mode of the electronic component workshop as claimed in claim 4 is characterized in that: The loss function is: L(*)=min(ratios×A t ,clip(ratios,1-∈,1+∈)A t ) Where * is the current policy parameter, which is θ or Φ; θ k is the parameter of the old strategy; ∈ is a hyperparameter, which refers to the update amplitude between the new strategy and the old strategy; the clipping function clip(ratios, 1-∈, 1+∈) limits the value of ratios to the range of [1-∈, 1+∈]; the strategy ratio ratios is the new strategy in state s t Next select action a t The probability of the old policy in state s t Next select action a t The ratio of the probabilities.

7. The improved PPO scheduling method in the AGV compound operation mode of the electronic component workshop as claimed in claim 1 is characterized in that: After the pre-training is completed for the set number of times, PPO training is performed. When the agent can obtain the maximum reward value multiple times in a row during training, the pre-training ends. After pre-training, the PPO algorithm is used for training. When the agent can continuously and stably obtain the maximum cumulative reward, the training ends, the optimal strategy is output, and the Actor and Critic network models are saved.

8. An improved PPO scheduling system in the AGV composite operation mode of an electronic component workshop, characterized in that: include: The pre-training model uses grid modeling to rasterize the workshop environment map. Through multiple task orders, the AGV performs the A* search algorithm on the grid map to obtain batches of sample data. The pre-training is completed according to the model parameters of the sample pre-training strategy network: PPO training model, according to the new task order, AGV interacts with the environment to obtain the observation state, action space and reward value, and after matrix processing the observation state, it is used as the input data of the PPO algorithm, and the policy network and value network loss values ​​of the PPO algorithm are calculated. The network model parameters are updated through the gradient descent method to complete the PPO training; The real-time scheduling module generates real-time task orders online and guides the AGV to complete path planning and scheduling tasks through the decision network model trained in the first two stages.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 7.

10. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the methods described in claims 1 to 7.

Images