Improved ppo scheduling method, system, medium and equipment under agv complex operation mode of electronic component workshop

Abstract

The present application relates to the field of electronic component workshop AGV logistics scheduling, and discloses an improved PPO scheduling method, system, medium and equipment under the AGV complex operation mode of an electronic component workshop, which comprises: using grid modeling to rasterize the workshop environment map, making the AGV perform A* search algorithm on the grid map through multiple task orders, obtaining batch sample data, and completing pre-training according to the model parameters of the sample pre-training strategy network; according to a new task order, the AGV interacts with the environment, obtains the observation state, action space and reward value, processes the observation state into a matrix, and uses the matrix as the input data of the PPO algorithm; the loss value of the strategy network and the value network of the PPO algorithm is calculated, the network model parameters are updated through the gradient descent method, and the PPO training is completed; online real-time task orders are generated, and the decision network model trained in the previous two stages is used to guide the AGV to complete path planning and scheduling tasks. The present application can quickly learn the AGV scheduling strategy of an electronic component workshop, significantly improves the learning efficiency, and has stability.

Description

Technical Field

[0001] This invention relates to the field of AGV logistics scheduling technology in electronic component workshops, and in particular to an improved PPO scheduling method, system, medium and equipment for AGV composite operation mode in electronic component workshops. Background Technology

[0002] In electronic component manufacturing, maintaining a clean environment is crucial. AGVs (Automated Guided Vehicles), as an important component of automated logistics systems in electronic component workshops, have been widely applied in intelligent manufacturing and automated warehousing. With the development of automation technology, AGVs are increasingly taking over the transfer of raw materials and workpieces in electronic component manufacturing. Completing the transfer of workpieces and raw materials in the earliest possible time, thereby improving the utilization rate of production resources in the workshop, has become a key aspect of manufacturing development.

[0003] Traditional AGV path planning and task scheduling algorithms can be broadly categorized into heuristic algorithms and intelligent optimization algorithms, such as genetic algorithms, ant colony algorithms, Dijkstra's algorithm, and A* algorithm. These algorithms and their variants have been widely used to solve AGV pathfinding and scheduling problems. While traditional algorithms or their variants can sometimes quickly yield shop floor scheduling solutions, they may become infeasible as the complexity of the manufacturing environment and tasks increases. To improve solution quality, complex heuristic rules are often required, and these rules are often interdependent, sometimes even leading to loops or contradictions. Therefore, it is difficult to find a feasible solution that satisfies all the rules.

[0004] In recent years, with the development of artificial intelligence technology, reinforcement learning, as a method that can automatically learn through interaction with the environment, has emerged as a promising approach to solving AGV scheduling problems. This method, through trial-and-error exploration and feedback evaluation, enables multi-AGV systems to learn and adapt to the environment, facilitating global path planning for multiple AGVs. Examples include Q-learning, Deep Q-Network (DQN), and Proximal Policy Optimization (PPO). Existing literature discloses an effective and high-performance adaptive meme algorithm combined with Q-learning to solve the AGVDP problem. However, this method only considers the production cycle planning and does not address obstacles and conflicts during the vehicle path. Regardless of whether the reinforcement learning algorithm is based on value or policy networks, the agent's initial exploration in complex environments is prone to getting trapped in local optima. In complex environments such as manufacturing workshops with many obstacles and diverse production types, reinforcement learning methods suffer from increased computational complexity, easily getting trapped in local optima, leading to poor decision-making performance and slow training speed. Summary of the Invention

[0005] To address the aforementioned problems, the purpose of this invention is to provide an improved PPO scheduling method, system, medium, and equipment for AGV composite operation mode in electronic component workshops. This method can quickly learn and acquire scheduling strategies, significantly improving learning efficiency while maintaining stability.

[0006] To achieve the above objectives, in a first aspect, the technical solution adopted by the present invention is as follows: an improved PPO scheduling method under the composite operation mode of AGV in an electronic component workshop, comprising: rasterizing the environmental map of the electronic component workshop using grid modeling; using multiple task orders to enable the AGV to perform A* search algorithm on the grid map to obtain batch sample data; completing pre-training based on the model parameters of the pre-trained strategy network according to the sample data; interacting with the environment according to new task orders to obtain the observation state, action space, and reward value; matrixing the observation state and using it as input data for the PPO algorithm; calculating the loss values ​​of the strategy network and value network of the PPO algorithm; updating the network model parameters using gradient descent to complete the PPO training; generating real-time task orders online; and guiding the AGV to complete path planning and scheduling tasks through the decision network model trained in the first two stages.

[0007] Furthermore, the workshop environment map of the electronic component factory is rasterized using grid modeling. Multiple task orders are used to enable AGVs to perform an A* search algorithm on the raster map to obtain batches of sample data. Based on these samples, the model parameters of the strategy network are pre-trained, including:

[0008] The A* algorithm is used to guide AGVs to continuously complete batch tasks in a grid map, and the motion trajectory node information of each task is saved to obtain batch sample experience.

[0009] N sample data points are extracted and fed into the Actor network. The network output is the probability distribution of the action space of multiple samples.

[0010] The algorithm selects actions based on probabilities, calculates the log probability of each action, calculates the loss value of the Actor network, performs backpropagation based on the loss value to obtain the gradient, and updates the model parameters based on the gradient.

[0011] Furthermore, based on the new task order, the AGV interacts with the environment to obtain the observation status, action space, and reward value. The observation status is then matrix-processed and used as input data for the PPO algorithm, including:

[0012] The pre-trained model parameters are then used to further train the Actor network and the Actor network model using the PPO algorithm.

[0013] Before training, new tasks are assigned, and the A* search algorithm guides the AGV to interact with the grid map environment. The state and action trajectory node information of each interaction are saved. Based on the environment layout, the state and action information of the AGV, a layout matrix, a current position matrix, a target position matrix and an action list are generated as input data for the PPO algorithm.

[0014] Furthermore, the current observation state s is stored based on the number of consecutive interaction steps. t The corresponding action a t Batch action probability distribution And the batch reward value r t+1 ;

[0015] According to s t and a t A new strategy for computation π θ (a t |s t ) and old strategies Calculate the ratios, and then calculate the cumulative discount reward value R based on the reward value and the discount factor γ, thereby calculating the advantage function value A_k;

[0016] Using ratios, A_k, R, and the state value function V Ф (s t The loss functions L(θ) of the Actor network and L(Ф) of the Critic network are calculated using the cutoff threshold ∈ , and the model parameters of the Actor network and Critic network are updated using gradient descent.

[0017] Furthermore, the dominant function value A_k is determined by the dominant function A t The calculated advantage function A is... t for:

[0018] A t =Q t (s t ,a t )-V t (s t )

[0019] In the formula, Q t (s t ,a t ) is the action value function, V t (s t Let A be the value function at time t, and let A be the advantage function. t Indicates taking action a t Relative to the average action in state s t Additional rewards.

[0020] Furthermore, the loss function is:

[0021] L(*)=min(ratios×A t ,clip(ratios,1-∈,1+∈)A t )

[0022] In the formula, * represents the current policy parameter, which is θ or Ф; θ k is the parameter of the old policy; ∈ is a hyperparameter, referring to the update magnitude between the new and old policies; the clipping function clip(ratios, 1-∈, 1+∈) restricts the value of ratios to the range [1-∈, 1+∈]; the policy ratio ratios are the new policy in state s. t Choose action a t The probability of the old policy in state s t Choose action a t The ratio of the probability of .

[0023] Furthermore, after the pre-training is completed a set number of times, PPO training is performed. When the agent is trained to the point that it can continuously obtain the maximum reward value multiple times, the pre-training ends.

[0024] After pre-training, the PPO algorithm is used for training. When the agent can continuously and stably obtain the maximum cumulative reward, the training ends, the optimal policy is output, and the Actor and Critic network models are saved.

[0025] Secondly, the technical solution adopted by this invention is as follows: an improved PPO scheduling system for AGV composite operation mode in an electronic component workshop, comprising: a pre-trained model, which uses grid modeling to rasterize the workshop environment map, and uses multiple task orders to enable the AGV to perform A* search algorithm on the grid map to obtain batch sample data, and completes pre-training based on the model parameters of the pre-trained strategy network; a PPO training model, in which the AGV interacts with the environment according to new task orders, obtains the observation state, action space and reward value, and uses the matrix processing of the observation state as the input data of the PPO algorithm, calculates the loss value of the strategy network and value network of the PPO algorithm, and updates the network model parameters through gradient descent to complete PPO training; and a real-time scheduling module, which generates real-time task orders online, and guides the AGV to complete path planning and scheduling tasks through the decision network model trained in the first two stages.

[0026] Thirdly, the technical solution adopted by the present invention is: a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0027] Fourthly, the technical solution adopted by the present invention is: a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0028] The present invention has the following advantages due to the adoption of the above technical solutions:

[0029] 1. This invention utilizes AGV scheduling experience samples obtained by the A* algorithm to initialize and train a convolutional neural network model to optimize parameters. The agent is trained using PPO, which can quickly learn and obtain scheduling strategies. The single AGV scheduling reward value is increased by 4.67 times, and the multi-AGV value is increased by 4.5 times. The current algorithm has significantly improved learning efficiency and stability.

[0030] 2. Compared with the optimized DQN, AC, and PG algorithms, the improved PPO algorithm of this invention achieves convergence speed improvements of 50.6%, 28.3%, and 83.4%, respectively. The improved PPO algorithm is not only suitable for single AGV scheduling tasks, but also demonstrates strong adaptability and superiority for complex multi-AGV scheduling tasks. Attached Figure Description

[0031] Figure 1 This is an overall flowchart of the improved PPO scheduling method under the composite operation mode of AGV in the electronic component workshop in this embodiment of the invention;

[0032] Figure 2 This is a detailed flowchart of the pre-training and PPO training in an embodiment of the present invention;

[0033] Figure 3 These are the training loss curves of the PPO algorithm before and after the improvement in this embodiment of the invention;

[0034] Figure 4 This is the reward value convergence curve of the A* algorithm pre-training + PPO algorithm in the embodiment of the present invention;

[0035] Figure 5 This is a comparison chart of the average task completion time of different numbers of AGVs in the PPO algorithm before and after the improvement in this embodiment of the invention. Detailed Implementation

[0036] The scheduling efficiency of Automated Guided Vehicles (AGVs) plays a crucial role in the intelligent manufacturing logistics operations of electronic component workshops. In complex operation modes, AGVs need to complete multiple tasks, making the reduction of total task time the core of scheduling optimization. To address this, a scheduling method based on an improved Proximal Policy Optimization (PPO) algorithm is proposed. First, the A* (A-star) algorithm is used to obtain scheduling experience samples in a complex workshop environment to initialize the neural network parameters. Then, the improved PPO algorithm is used to train the agent to optimize the scheduling strategy. Experimental results show that the improved PPO algorithm significantly improves scheduling efficiency compared to traditional methods.

[0037] For intelligent manufacturing of electronic components, equipment is categorized into cleanroom and non-cleanroom facilities. Workshops have high requirements for work-in-process storage, with a wide variety of products and batches. The complex logistics environment leads to complex and ever-changing AGV logistics handling targets. Traditional heuristic algorithms for solving scheduling problems are insufficient. This invention proposes an improved PPO scheduling method, system, medium, and equipment for AGV composite operation modes in electronic component workshops. First, the A* algorithm is used to acquire experiential knowledge to pre-train the policy network, solving the problem of reinforcement learning easily getting trapped in local optima in the early stages. Second, the PPO algorithm is used to continue training the agent under real-time random multi-task conditions, enhancing environmental adaptability and improving the agent's learning efficiency and performance. This invention achieves the shortest total time for scheduling random multi-tasks with different numbers of AGVs by improving the PPO algorithm's AGV scheduling method. By designing a reinforcement learning state space, action space, and convolutional neural network model, the agent can quickly make decisions to complete the target task. The improved PPO algorithm is compared with other reinforcement learning algorithms, highlighting its advantages in decision-making during AGV scheduling.

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0040] In one embodiment of the present invention, an improved PPO scheduling method is provided for a composite operation mode of AGVs in an electronic component workshop. This embodiment is illustrated using a manufacturing workshop scenario as an example. Figure 1 As shown, the method includes the following steps:

[0041] 1) Utilize grid modeling to rasterize the workshop environment map, and use multiple task orders to enable AGVs to perform A* search algorithm on the grid map to obtain batch sample data. Based on the sample pre-training strategy network model parameters, complete the pre-training.

[0042] 2) Based on the new task order, the AGV interacts with the environment to obtain the observation status, action space and reward value. After the observation status is processed into a matrix, it is used as the input data of the PPO algorithm. The loss values ​​of the policy network and value network of the PPO algorithm are calculated, and the network model parameters are updated by gradient descent to complete the PPO training.

[0043] 3) Generate real-time task orders online, and guide the AGV to complete path planning and scheduling tasks through the decision network model trained in the first two stages.

[0044] In step 1) above, based on the structure of the PPO algorithm, the observation space, action space, reward value structure, and convolutional neural network structure for processing the observation space of the agent are designed to adapt to the AGV pathfinding and handling tasks in the complex environment of the manufacturing workshop.

[0045] In this embodiment, the A* algorithm is a node traversal method based on pathfinding, which determines the shortest path by searching all possible nodes. This characteristic of comprehensively searching all possible nodes in a certain order and storing them in memory enables the A* algorithm to generate optimal paths, and it outperforms most other algorithms.

[0046] The A* algorithm can be used to obtain the motion trajectory node information of the AGV successfully reaching the target point through interaction with the environment during each task execution. This information is stored in a sample pool for pre-training of the PPO algorithm policy network.

[0047] In this embodiment, due to the large exploration space and complex states in the electronic component manufacturing workshop, the agent is prone to getting stuck in local loops of suboptimal policies during the initial PPO algorithm training process, making it difficult for the policy network to be further optimized. To solve this problem, this invention introduces behavioral cloning or other imitation learning algorithms into the improved PPO algorithm before formal training to pre-train the policy network, enabling it to learn the policies of experts. After pre-training, the policy network has better initial capabilities and can explore and optimize policies more efficiently in subsequent PPO training, thereby improving the overall training effect.

[0048] In this embodiment, the workshop environment map is rasterized using grid modeling. Multiple task orders are used to enable the AGV to perform an A* search algorithm on the grid map to obtain batches of sample data. Based on the sample pre-trained strategy network model parameters, such as... Figure 2 As shown, it includes the following steps:

[0049] 1.1) Use the A* algorithm to guide the AGV to continuously complete batch tasks in the grid map, save the action trajectory node information of each task, and obtain batch sample experience;

[0050] 1.2) Extract N sample data and input them into the Actor network. The network output is the probability distribution of the action space of multiple samples.

[0051] 1.3) Select actions based on probabilities and calculate the log probability of the actions. Calculate the loss value of the Actor network, perform backpropagation based on the loss value to obtain the gradient, and update the model parameters based on the gradient.

[0052] Specifically, the A* algorithm guides the AGV to interact with the environment to complete 14 tasks, saving the state and action trajectory node information of each interaction. Based on the environment layout, AGV state, and action information, a layout matrix, current position matrix, target position matrix, and action list are generated; this is one sampling process. Then, the program will randomly generate different assigned tasks until 600 sample data points are collected, and 512 samples are extracted to pre-train the policy network. A set of empirical data obtained by the A* algorithm is extracted, containing N(512) samples, each sample consisting of a state s. i and the corresponding action a i Composition. The policy network of the model is given state s. i Generate action a i The probability is π(a) i |si Therefore, for each sample i, the corresponding log probability is logπ(a). i |s i The loss function of the policy network is calculated as follows:

[0053]

[0054] For L BC Perform backpropagation and update the parameters of the policy network model, then use this new policy network for reinforcement learning.

[0055] The PPO algorithm is an online policy algorithm that primarily solves high-dimensional decision-making problems in a continuous action space. It uses a pruning probability ratio to limit the magnitude of each policy network parameter update, ensuring that the new policy outperforms the old one. The update steps of the PPO algorithm are as follows:

[0056] (1) Collect sample data, perform several simulations according to the current policy π(a|s), and record the state, action, reward and next state.

[0057] (2) Calculate the dominance function A t The current policy is estimated using a value function, calculated as follows:

[0058] Q t (s t ,a t )=∑ s′ P(s′|s t ,a t )[R(s t ,a t ,s′)+γV t (s′)] (1)

[0059] V t (s t ) = max a Q t (s t ,a) (2)

[0060] A t =Q t (s t ,a t )-V t (s t (3)

[0061] Among them, Q t (s t ,a t ) is a function (action-value function), which represents the expected reward starting from state 1 after taking an action at time 2, where P(s′|s) t,a t () refers to the state s at time t. t Take action a t The probability of transitioning to state s′ is R(s). t ,a t ,s′) is the corresponding reward function, γ is the discount factor, and V t (s′) is the value function of state s′; V t (s t ) is the value function at time t, that is, the state s at time t. t The maximum expected return is obtained by maximizing the Q function of all possible actions a; the advantage function A t Indicates taking action a t Relative to the average action in state s t Additional rewards.

[0062] (3) Calculate the policy ratios, i.e., the new policy in state s. t Choose action a t The probability of the old policy in state s t Choose action a t The ratio of the probabilities is calculated using the following formula:

[0063]

[0064] (4) Calculate the objective function and use the clipping function clip(ratios, 1-∈, 1+∈) to restrict the value of ratios to the range [1-∈, 1+∈], which limits the magnitude of policy updates.

[0065] L(s,a,θ k ,θ)=min(ratios×A t ,clip(ratios,1-∈,1+∈)A t (5)

[0066] θ is the current policy parameter, θ k ∈ represents the parameters of the old strategy; ∈ represents the hyperparameter, which indicates the update magnitude between the new and old strategies.

[0067] (5) Obtain the next policy parameter θ by optimizing the objective function. k+1 Using gradient descent:

[0068]

[0069] In step 2) above, based on the new task order, the AGV interacts with the environment to obtain the observation status, action space, and reward value. After matrix processing of the observation status, it is used as the input data for the PPO algorithm, including the following steps:

[0070] 2.1.1) After pre-training, the model parameters are used to further train the Actor network and the Actor network model using the PPO algorithm;

[0071] 2.1.2) Before training, new tasks are assigned, and the A* search algorithm guides the AGV to interact with the grid map environment. The state and action trajectory node information of each interaction are saved. Based on the environment layout, the state and action information of the AGV, a layout matrix, a current position matrix, a target position matrix and an action list are generated as input data for the PPO algorithm.

[0072] In step 2) above, the loss values ​​of the policy network and value network of the PPO algorithm are calculated, and the network model parameters are updated using the gradient descent method, such as... Figure 2 As shown, it includes the following steps:

[0073] 2.2.1) Store the current observation state s in mini-batches based on the number of consecutive interaction steps. t The corresponding batch action a t Batch action probability distribution And the batch reward value r t+1 ;

[0074] Among them, the probability distribution of batch actions The policy π refers to the probability θ of performing action a under the observed state at time t; the batch reward value r t+1 This refers to the reward value for executing strategy π, which is obtained at time t+1.

[0075] 2.2.2) According to s t and a t A new strategy for computation π θ (a t |s t ) and old strategies Calculate the ratios, and then calculate the cumulative discount reward value R based on the batch reward value and the discount factor γ, thereby calculating the advantage function value A_k;

[0076] In this embodiment, within the manufacturing workshop layout, after the AGV selects an action, the system provides corresponding reward feedback. The results of the action execution are generally divided into three categories: the first category, reaching the target position, corresponds to a reward value of 1. The second category, executing abnormal actions, including colliding with obstacles and going out of bounds, corresponds to a reward value of -1. The third category, normal exploration, neither reaching the target position nor encountering any abnormal situations, corresponds to a reward value of 0. The reward value R for executing the task is defined as follows:

[0077]

[0078] 2.2.3) Using ratios, A_k, R, and the state value function V Ф (s t The loss functions L(θ) of the Actor network and L(Ф) of the Critic network are calculated using the cutoff threshold ∈ , and the model parameters of the Actor network and Critic network are updated using the gradient descent method.

[0079] In this embodiment, the loss function can be obtained according to the above formula (5):

[0080] L(*)=min(ratios×A t ,clip(ratios,1-∈,1+∈)A t )

[0081] In the formula, * represents the current strategy parameter, which is θ or Ф.

[0082] In the above embodiments, PPO training is performed after the pre-training is completed a set number of times. When the agent is trained to the point that it can continuously obtain the maximum reward value multiple times, the pre-training ends; for example, when it is trained to the point that it can continuously obtain the maximum reward value 10 times after a certain period of time, the pre-training ends.

[0083] After pre-training, the PPO algorithm is used for training. Training ends when the agent can consistently and stably obtain the maximum cumulative reward, the optimal policy is output, and the Actor and Critic network models are saved. For example, the optimal policy is output when the agent obtains the maximum cumulative reward 30 times consecutively.

[0084] In one embodiment of the present invention, an improved PPO scheduling system for AGV composite operation mode in an electronic component workshop is provided, comprising:

[0085] The pre-trained model uses grid modeling to rasterize the workshop environment map. Through multiple task orders, the AGV performs the A* search algorithm on the grid map to obtain a batch of sample data. Based on the sample pre-training strategy network model parameters, the pre-training is completed.

[0086] The PPO training model involves the AGV interacting with the environment based on new task orders to obtain observation states, action spaces, and reward values. After matrix processing of the observation states, they are used as input data for the PPO algorithm. The loss values ​​of the policy network and value network of the PPO algorithm are calculated, and the network model parameters are updated using the gradient descent method to complete the PPO training.

[0087] The real-time scheduling module generates real-time task orders online and guides the AGV to complete path planning and scheduling tasks through the decision network model trained in the first two stages.

[0088] In the above embodiments, the pre-trained model utilizes grid modeling to rasterize the workshop environment map. Multiple task orders are used to enable the AGV to perform an A* search algorithm on the grid map to obtain batches of sample data. Based on these samples, the model parameters of the pre-trained strategy network are included:

[0089] The A* algorithm is used to guide AGVs to continuously complete batch tasks in a grid map, and the motion trajectory node information of each task is saved to obtain batch sample experience.

[0090] N sample data points are extracted and fed into the Actor Net. The network output is the probability distribution of the action space of multiple samples.

[0091] The algorithm selects actions based on probabilities, calculates the log probability of each action, calculates the loss value of the Actor network, performs backpropagation based on the loss value to obtain the gradient, and updates the model parameters based on the gradient.

[0092] In the above embodiments, based on new task orders, the AGV interacts with the environment to obtain the observation status, action space, and reward value. The observation status is then matrix-processed and used as input data for the PPO algorithm, including:

[0093] The pre-trained model parameters are then used to further train the Actor network and the Actor network model using the PPO algorithm.

[0094] Before training, new tasks are assigned, and the A* search algorithm guides the AGV to interact with the grid map environment. The state and action trajectory node information of each interaction are saved. Based on the environment layout, the state and action information of the AGV, a layout matrix, a current position matrix, a target position matrix and an action list are generated as input data for the PPO algorithm.

[0095] In the above embodiments, the loss values ​​of the policy network and value network of the PPO algorithm are calculated, and the network model parameters are updated using the gradient descent method, including:

[0096] Store the current observation state s in mini-batches based on the number of consecutive interaction steps. t The corresponding batch action a t Batch action probability distribution And the batch reward value r t+1 ;

[0097] According to s t and a t A new strategy for computation π θ (a t |s t ) and old strategies Calculate the ratios, and then calculate the cumulative discount reward value R based on the batch reward value and the discount factor γ, thereby calculating the advantage function value A_k;

[0098] In the above embodiments, PPO training is performed after the pre-training is completed a set number of times. When the agent can continuously obtain the maximum reward value 10 times after a certain period of training, the pre-training ends.

[0099] After pre-training, the PPO algorithm is used for training. When the agent can continuously and stably obtain the maximum cumulative reward, the training ends, the optimal policy is output, and the Actor and Critic network models are saved.

[0100] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0101] like Figure 3 The image shows a comparison of the training loss curves of the PPO algorithm before and after the improvement. Figure 4 The curve shows the reward value convergence curve of the A* algorithm pre-training + PPO algorithm, demonstrating that the improved PPO algorithm has significant advantages in convergence speed and loss value stability.

[0102] like Figure 5 As shown, the blue bars represent the PPO algorithm, and the orange bars represent the improved PPO algorithm. The red arrows in the figure indicate the improvement margin for each data set, demonstrating that the improved PPO algorithm can significantly improve task completion efficiency, especially in multi-AGV scheduling. This provides strong support for the application of intelligent scheduling algorithms in complex logistics environments.

[0103] In summary, the scheduling efficiency of Automated Guided Vehicles (AGVs) plays a crucial role in the logistics operations of smart factories. In complex operation modes, AGVs need to complete multiple tasks, making the reduction of total task time the core of scheduling optimization. To address this, a scheduling method based on an improved Proximal Policy Optimization (PPO) algorithm is proposed. First, the A* (A-star) algorithm is used to obtain scheduling experience samples in a complex workshop environment to initialize the neural network parameters. Then, the improved PPO algorithm is used to train the agent to optimize the scheduling strategy. Experimental results show that the improved PPO algorithm significantly improves scheduling efficiency compared to traditional methods. Real-world factory application results demonstrate that the improved PPO algorithm effectively reduces the average completion time of different numbers of AGVs in random tasks, with a maximum reduction of 42.7%.

[0104] In one embodiment of the present invention, a computing device is provided, which can be a terminal and may include: a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs, which, when executed by the processor, implement the methods described in the above embodiments. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.

[0105] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.

[0107] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.

[0108] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0109] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An improved PPO scheduling method under the composite operation mode of AGVs in an electronic component workshop, characterized in that, include: The environment map of the electronic component workshop is rasterized using grid modeling. Multiple task orders are used to guide AGVs to perform A* search on the grid map to obtain batch sample data. Pre-training is then completed based on the model parameters of the network using the sample pre-training strategy. This includes: using the A* algorithm to guide the AGVs to continuously complete batch tasks on the grid map, saving the action trajectory node information for each task, and acquiring batch sample experience; extracting N sample data and feeding them into the Actor network, the network output being the probability distribution of multiple sample action spaces; selecting actions based on probabilities to calculate the log probability value of the actions, calculating the loss value of the Actor network, performing backpropagation based on the loss value to calculate the gradient, and updating the model parameters based on the gradient. Based on the new task order, the AGV interacts with the environment to obtain the observation status, action space and reward value. After the observation status is processed into a matrix, it is used as the input data of the PPO algorithm. The loss values ​​of the policy network and value network of the PPO algorithm are calculated, and the network model parameters are updated by gradient descent to complete the PPO training. Real-time task orders are generated online, and the decision network model trained in the first two stages guides the AGV to complete path planning and scheduling tasks. Specifically, based on the new task order, the AGV interacts with the environment to obtain the observation status, action space, and reward value. The observation status is then matrix-processed and used as input data for the PPO algorithm, including: The pre-trained model parameters are then used to further train the Actor network and the Actor network model using the PPO algorithm. Before training, new tasks are assigned, and the A* search algorithm guides the AGV to interact with the grid map environment. The state and action trajectory node information of each interaction are saved. Based on the environment layout, the state and action information of the AGV, a layout matrix, a current position matrix, a target position matrix and an action list are generated as input data for the PPO algorithm. Calculate the loss values ​​of the policy network and value network in the PPO algorithm, and update the network model parameters using gradient descent, including: Store the current observation state based on the number of consecutive interaction steps. Corresponding actions Batch action probability distribution and batch reward value ; according to and New strategies for computation and old strategies Calculate the ratio Based on reward value and discount factor Calculate the cumulative discount reward value R, and then calculate the advantage function value. ; pass State value function and truncation threshold Calculate the loss function of the Actor network Loss function of Critic network The model parameters of the Actor network and Critic network are updated using gradient descent.

2. The improved PPO scheduling method under the AGV composite operation mode in the electronic component workshop as described in claim 1, characterized in that, Advantage function value From the advantage function The calculated advantage function is... for: ； In the formula, It is an action value function. It is the value function and advantage function at time t. Indicates taking an action Compared to the average action in the state Additional rewards.

3. The improved PPO scheduling method under the AGV composite operation mode in the electronic component workshop as described in claim 1, characterized in that, The loss function is: ； In the formula, This is the current strategy parameter, which is or ; It is the truncation threshold, referring to the update magnitude between the new and old policies; pruning function. ,Will The value is limited to Within the range; strategy ratio Is the new strategy in the state? Select action The probability of the old policy in state Select action The ratio of the probability of .

4. The improved PPO scheduling method under the AGV composite operation mode in the electronic component workshop as described in claim 1, characterized in that, After the pre-training is completed a set number of times, PPO training is performed. When the agent is trained to the point that it can continuously obtain the maximum reward value multiple times, the pre-training ends. After pre-training, the PPO algorithm is used for training. When the agent can continuously and stably obtain the maximum cumulative reward, the training ends, the optimal policy is output, and the Actor and Critic network models are saved.

5. An improved PPO scheduling system for a composite operation mode of AGVs in an electronic component workshop, used to implement the improved PPO scheduling method for a composite operation mode of AGVs in an electronic component workshop as described in any one of claims 1-4, characterized in that, include: The pre-trained model uses grid modeling to rasterize the workshop environment map. Through multiple task orders, the AGV performs the A* search algorithm on the grid map to obtain a batch of sample data. Based on the sample pre-training strategy network model parameters, the pre-training is completed. The PPO training model involves the AGV interacting with the environment based on new task orders to obtain observation states, action spaces, and reward values. After matrix processing of the observation states, they are used as input data for the PPO algorithm. The loss values ​​of the policy network and value network of the PPO algorithm are calculated, and the network model parameters are updated using the gradient descent method to complete the PPO training. The real-time scheduling module generates real-time task orders online and guides the AGV to complete path planning and scheduling tasks through the decision network model trained in the first two stages.

6. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in claims 1 to 4.

7. A computing device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described in claims 1 to 4.

Images