An agv cluster adaptive rescheduling method based on deep reinforcement learning

Abstract

The application belongs to the technical field of intelligent manufacturing logistics scheduling and relates to an AGV cluster adaptive rescheduling method based on deep reinforcement learning. Initial workshop scheduling data is initialized, and an intelligent optimization algorithm is used to generate an initial scheduling scheme; the AGV cluster is controlled to execute a distribution task and real-time workshop state information is acquired; when a faulty AGV is detected, a DQN state vector is constructed according to the workshop state information at the time of the fault; the DQN state vector is input into a preset DQN network to obtain Q values corresponding to a global rescheduling mode, an insertion rescheduling mode, a replacement mode and a local rescheduling mode respectively, the rescheduling mode corresponding to the maximum Q value is determined as the selected rescheduling mode; and the current scheduling scheme is updated according to the selected rescheduling mode and is issued to the AGV cluster to continue executing the distribution task. The application can adaptively select the optimal rescheduling mode according to the fault scenario and improve the fault recovery efficiency of the AGV cluster.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing logistics scheduling technology, and in particular relates to an adaptive rescheduling method for AGV clusters based on deep reinforcement learning. Background Technology

[0002] With the development of intelligent manufacturing and intelligent logistics technologies, AGVs, with their flexible operation and high degree of automation, have been widely used in production workshops, warehousing centers, and other scenarios to complete material handling and logistics distribution tasks. In actual operation, multiple AGVs usually need to work together to perform delivery tasks, and the AGV cluster is uniformly scheduled based on task requirements, vehicle status, and delivery progress. A reasonable AGV cluster scheduling method can coordinate the task allocation relationship between AGVs, reduce vehicle idle waiting and task backlog, and improve logistics distribution efficiency and equipment utilization.

[0003] However, most existing AGV cluster scheduling methods are based on the normal operating status of AGVs for task allocation and scheduling planning. During the execution of delivery tasks, when some AGVs fail due to equipment malfunctions, communication failures, or other unforeseen circumstances, the failed AGVs may be unable to continue their original delivery tasks, leading to task interruptions, execution delays, and the ineffective execution of the original scheduling scheme. Simultaneously, the unfinished tasks of the failed AGVs need to be continued by other normally operating AGVs. If task adjustments and reallocations cannot be made in a timely manner based on the real-time operating status of the cluster, it can easily lead to task imbalance, uneven vehicle load, and decreased cluster operating efficiency. Existing scheduling methods still lack adaptability to sudden failure scenarios and struggle to quickly formulate reasonable and executable rescheduling schemes after a failure occurs. Therefore, researching a reinforcement learning-driven adaptive rescheduling method for AGV clusters has significant theoretical and engineering value. Summary of the Invention

[0004] This invention addresses the technical problems existing in the background art by proposing an adaptive rescheduling method for AGV clusters based on deep reinforcement learning. It solves the problems of task interruption, execution delay, and decreased operating efficiency caused by sudden failures during AGV cluster delivery, achieving the goal of rapid recovery and efficient rescheduling of AGV clusters in fault scenarios.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] S1. Initialize workshop scheduling data, use intelligent optimization algorithm to generate initial scheduling scheme, and determine the initial scheduling scheme as the current scheduling scheme;

[0007] S2. Control the AGV cluster to execute delivery tasks according to the current scheduling plan, and obtain workshop status information in real time during task execution;

[0008] S3. Determine whether all delivery tasks have been completed based on the workshop status information. If all delivery tasks have been completed, end the scheduling process and output the final total transportation cost of the AGV cluster; otherwise, proceed to step S4.

[0009] S4. Determine whether there is a faulty AGV based on the workshop status information. If there is no faulty AGV, return to step S2. If there is a faulty AGV, determine the current workshop status information as the workshop status information when the fault occurred, and execute step S5.

[0010] S5. Construct a deep Q-network (DQN) state vector based on the workshop state information at the time of the fault.

[0011] S6. Input the DQN state vector into the preset DQN network to obtain the Q value corresponding to each rescheduling method, and determine the rescheduling method corresponding to the maximum Q value as the selected rescheduling method.

[0012] S7. Update the current scheduling scheme according to the selected rescheduling method, send the updated current scheduling scheme to the AGV cluster, control the AGV cluster to continue to execute the delivery task according to the updated current scheduling scheme, and return to step S2 until all delivery tasks are completed.

[0013] Preferably, the workshop scheduling data in step S1 includes the total number of delivery tasks, the task number of each delivery task, the task location, the material call time, the material requirement, and the optimal delivery time of the materials. The task location is represented by the horizontal and vertical axes. The scheduling scheme uses a set... This indicates that m represents the number of AGVs participating in the delivery in the scheduling plan, and m increases accordingly after dispatching a new AGV. This represents the delivery task sequence corresponding to the k-th AGV in the AGV cluster. Let i be the i-th task of the k-th AGV, where... Let i be the task number of the i-th delivery task performed by the k-th AGV. Let x and y be the x and y coordinates of the task position of the i-th delivery task performed by the k-th AGV, respectively. The material calling time for the i-th delivery task performed by the k-th AGV. Let i be the material requirement for the i-th delivery task performed by the k-th AGV. The optimal delivery time for materials for the i-th delivery task performed by the k-th AGV. This represents the number of delivery tasks for the k-th AGV. The sum of the number of delivery tasks for each AGV equals the total number of delivery tasks.

[0014] Preferably, in step S1, when generating the initial scheduling scheme using the intelligent optimization algorithm, the optimization objective is to minimize the total transportation cost of the AGV cluster. Within a preset optimization time, candidate scheduling schemes are continuously searched, and the candidate scheduling scheme with the minimum total transportation cost of the AGV cluster is determined as the initial scheduling scheme. The formula for calculating the total transportation cost of the AGV cluster is as follows:

[0015] ,in, Let be the total transportation cost of the AGV cluster, representing the sum of the transportation costs of m AGVs in the AGV cluster. The index is denoted by 0, where 0 is the index of the repository node, and 1 to 1. Let the index of the delivery task for the k-th AGV be the index. This is the cost coefficient per unit travel distance for AGVs. This is the penalty cost coefficient for early or late delivery of materials. A cost coefficient is applied to each AGV. Enable variables for AGVs, when the k-th AGV is enabled. =1 otherwise 0, Let be a binary decision variable, representing that if the travel path of the k-th AGV includes the route from the delivery task... To the delivery task ,but =1 otherwise 0, For the kth AGV from the delivery task To the delivery task The driving distance between them For the kth AGV to arrive at the delivery task The actual delivery time of the materials. For delivery tasks The absolute time deviation between the optimal delivery time of materials and the actual delivery time of materials.

[0016] Preferably, the workshop status information in step S2 includes the AGV number, real-time location, operating status, remaining amount of material being transported, execution progress of the delivery task sequence, and current workshop timestamp for each AGV in the AGV cluster. The operating status includes normal status and fault status, and the execution progress of the delivery task sequence includes the completed delivery task subsequence and the incomplete delivery task subsequence for each AGV. The incomplete delivery task subsequence includes the delivery task currently being executed and the remaining delivery tasks to be executed.

[0017] Preferably, in step S5, the delivery tasks contained in the unfinished delivery task subsequence of all AGVs are determined as all delivery tasks to be reassigned, and the delivery tasks contained in the unfinished delivery task subsequence of the faulty AGV are determined as delivery tasks to be reassigned for the faulty AGV.

[0018] Preferably, the DQN state vector in step S5 includes the AGV number, real-time location, running status, current remaining amount of material being carried, information on all delivery tasks to be reassigned, and the time of the fault occurrence for each AGV in the AGV cluster.

[0019] Preferably, in step S6, the preset DQN network is a deep Q network trained based on a preset training example set. During training, the DQN state vector is used as the state input, the rescheduling method is used as the action space, and the reward value is determined according to the total transportation cost of the AGV cluster after completing the delivery task using each rescheduling method. The smaller the total transportation cost of the AGV cluster, the larger the corresponding reward value. The network parameters of the deep Q network are iteratively updated according to the reward value until the preset training termination condition is met, and the preset DQN network is obtained.

[0020] As a preferred option, the rescheduling method in step S6 includes global re-optimization, insertion rescheduling, substitute succession and local re-optimization.

[0021] The global re-optimization method is as follows: using the intelligent optimization algorithm in step S1, starting from the real-time position of each normal AGV, all delivery tasks to be reassigned are reassigned. When the normal AGVs cannot form a feasible allocation scheme covering all delivery tasks to be reassigned, a new AGV is dispatched from the warehouse. Starting from the real-time position of each normal AGV and the warehouse position of the new AGV, the intelligent optimization algorithm in step S1 is used to jointly allocate all delivery tasks to be reassigned.

[0022] The insert-type rescheduling method involves inserting the delivery task to be reassigned corresponding to the faulty AGV into a feasible position in the incomplete delivery task subsequence of each normal AGV, while maintaining the relative execution order of the original delivery tasks of each normal AGV.

[0023] The replacement and succession method is as follows: a new AGV is dispatched from the warehouse, and the new AGV continues to execute the delivery tasks to be reassigned corresponding to the faulty AGV according to the execution order of the unfinished delivery task sub-sequences of the faulty AGV, while keeping the unfinished delivery task sub-sequences of each normal AGV unchanged.

[0024] The local re-optimization method involves dispatching a new AGV from the warehouse, using the intelligent optimization algorithm in step S1, taking the warehouse as the starting point of the new AGV, re-optimizing the delivery tasks to be reassigned corresponding to the faulty AGV, generating a delivery task sequence corresponding to the new AGV, and keeping the unfinished delivery task subsequences of each normal AGV unchanged.

[0025] Preferably, in step S7, based on the selected rescheduling method, the completed delivery task subsequences of each AGV in the current scheduling scheme are retained, the delivery tasks to be reassigned corresponding to the faulty AGV are deleted from the uncompleted delivery task subsequences of the faulty AGV, and the uncompleted delivery task subsequences of each AGV are adjusted. When the selected rescheduling method requires dispatching a new AGV, the new AGV is added to the AGV cluster, and the delivery task sequence corresponding to the new AGV is added to the current scheduling scheme to obtain the updated current scheduling scheme. The updated current scheduling scheme is then sent to each AGV participating in the delivery, and each AGV is controlled to continue executing the delivery task according to its corresponding uncompleted delivery task subsequence, and the process returns to step S2.

[0026] Compared with existing technologies, the advantages and positive effects of this invention are that it can adaptively select a suitable rescheduling method based on the real-time status of the workshop at the time of failure, reducing the impact of sudden AGV failures on the continuous execution of delivery tasks, improving the scheduling recovery efficiency, operational continuity, and scheduling stability of AGV clusters in failure scenarios, and helping to reduce the total transportation cost of AGV clusters. Compared with a fixed single rescheduling method, the method of this invention achieves the smallest relative percentage deviation value, the confidence interval does not overlap with the results of four fixed rescheduling methods, and the obtained results have smaller relative percentage deviations and narrower fluctuation ranges. It is superior to a single rescheduling method in terms of solution quality and stability, effectively solving the problems of task interruption, execution delay, and failure of the original scheduling scheme caused by sudden AGV failures. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0029] Figure 2 This is a schematic diagram of the scheduling sequence for the global re-optimization method in this invention;

[0030] Figure 3This is a schematic diagram of the scheduling sequence for the insertion-based rescheduling method in this invention;

[0031] Figure 4 This is a schematic diagram of the scheduling sequence for the substitute succession method in this invention;

[0032] Figure 5 This is a schematic diagram of the scheduling sequence for the local re-optimization method in this invention;

[0033] Figure 6 This is a schematic diagram of a workshop logistics and distribution scenario in a specific embodiment of the present invention;

[0034] Figure 7 This is a comparison chart of the confidence intervals of the relative percentage deviation between the method of the present invention and four fixed rescheduling methods;

[0035] Figure 8 This is a box plot showing the relative percentage deviation between the method of the present invention and four fixed rescheduling methods. Detailed Implementation

[0036] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0037] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0038] In practice, with the rapid development of intelligent manufacturing and intelligent warehousing logistics, Automated Guided Vehicles (AGVs) have been widely used in production workshop material distribution and warehouse cargo transfer due to their high degree of automation and flexible operation. Multi-AGV cluster collaborative scheduling is a core element in improving logistics efficiency and reducing operating costs. Existing scheduling methods are mostly geared towards ideal, fault-free operating conditions, generating static scheduling schemes with the goal of minimizing total transportation costs, without fully considering unforeseen anomalies during task execution. In actual operation, AGVs are prone to exiting operation due to sudden situations such as equipment failure, communication interruption, and path collision, leading to the interruption of assigned tasks and the failure of the original scheduling scheme. Existing technologies often employ a single, fixed rescheduling strategy, which cannot match scenarios with different levels of failure, remaining task size, and time constraints: global re-optimization solutions are time-consuming and difficult to meet real-time requirements; insert scheduling has limited adaptability and is prone to overload under multi-task failures; and substitute succession methods easily lead to resource idleness. Existing methods struggle to balance rescheduling response speed and optimization quality, easily causing task delays, load imbalances, and increased transportation costs, failing to meet the actual needs of high-reliability logistics distribution. Figure 1A flowchart of the implementation of this invention is provided.

[0039] Step 1: Initialize workshop scheduling data and generate an initial scheduling plan based on an intelligent optimization algorithm. Specifically, initializing the workshop scheduling data involves setting the total number of delivery tasks, the task number of each delivery task, the task location, the material call time, the material demand, the optimal material delivery time, and the preset optimization time t=5s for the intelligent optimization algorithm. In this embodiment, the intelligent optimization algorithm uses the artificial bee colony algorithm.

[0040] The scheduling scheme adopts a set This indicates that m is the number of AGVs participating in the delivery in the scheduling plan. This is the delivery task sequence corresponding to the k-th AGV in the AGV cluster. Delivery task: Let i be the i-th task of the k-th AGV, where... Let i be the task number of the i-th delivery task performed by the k-th AGV. Let x and y be the x and y coordinates of the task position of the i-th delivery task performed by the k-th AGV, respectively. The material calling time for the i-th delivery task performed by the k-th AGV. Let i be the material requirement for the i-th delivery task performed by the k-th AGV. The optimal delivery time for materials for the i-th delivery task performed by the k-th AGV. This represents the number of delivery tasks for the k-th AGV, and the sum of the delivery tasks for all AGVs equals the total number of delivery tasks. If... ={{2,3,10,18,29,648},{1,4,6,0,29,630},{3,4,3,21,28,621},{4,9,8,23,29,653}} indicates that the k-th AGV in the AGV cluster sequentially executes delivery tasks numbered 2, 1, 3, and 4. The initial scheduling scheme and the current scheduling scheme are represented by the same scheduling scheme data structure. The initial scheduling scheme is generated based on the workshop scheduling data and is used for the first execution of delivery tasks by the AGV cluster. The current scheduling scheme is the actual scheduling scheme used by the AGV cluster during the execution of delivery tasks.

[0041] When generating the initial scheduling scheme, the artificial bee colony algorithm aims to minimize the total transportation cost of the AGV cluster. The formula for calculating the total transportation cost of the AGV cluster is as follows:

[0042] ,in, Let be the total transportation cost of the AGV cluster, representing the sum of the transportation costs of m AGVs in the AGV cluster. The index is denoted by 0, where 0 is the index of the repository node, and 1 to 1. Let the index of the delivery task for the k-th AGV be the index. This is the cost coefficient per unit travel distance for AGVs. This is the penalty cost coefficient for early or late delivery of materials. A cost coefficient is applied to each AGV. Enable variables for AGVs, when the k-th AGV is enabled. =1 otherwise 0, Let be a binary decision variable, representing that if the travel path of the k-th AGV includes the route from the delivery task... To the delivery task ,but =1 otherwise 0, For the kth AGV from the delivery task To the delivery task The driving distance between them For the kth AGV to arrive at the delivery task The actual delivery time of the materials. For delivery tasks The absolute time deviation between the optimal delivery time of materials and the actual delivery time of materials.

[0043] Step 2: Control the AGV cluster to execute delivery tasks according to the current scheduling plan, and monitor workshop status information during the delivery task execution process. Specifically, workshop status information includes the AGV number, real-time location, operating status, current remaining amount of material being transported, execution progress of the delivery task sequence, and the current workshop timestamp for each AGV in the AGV cluster. The operating status includes normal and fault states, and the execution progress of the delivery task sequence includes the completed and incomplete delivery task sub-sequences for each AGV. The incomplete delivery task sub-sequences include the currently executing delivery tasks and the remaining unexecuted delivery tasks.

[0044] Step 3: Determine whether all delivery tasks have been completed based on the workshop status information. If all delivery tasks have been completed, end the scheduling process and output the final total transportation cost of the AGV cluster; otherwise, proceed to step 4.

[0045] Step 4: Determine if there is a faulty AGV based on the workshop status information. If no faulty AGV exists, return to Step 2; otherwise, determine the current workshop status information as the workshop status information at the time of the fault and proceed to Step 5. In essence, AGVs in a faulty operating state are identified as faulty AGVs, and AGVs in a normal operating state are identified as normal AGVs.

[0046] Step 5: Construct a Deep Q-Network (DQN) state vector based on the workshop status information at the time of the fault. Specifically, the system extracts the unfinished delivery task subsequences of each AGV based on the execution progress of each AGV's delivery task sequence, and identifies all delivery tasks contained in these unfinished delivery task subsequences as all delivery tasks to be reassigned. Simultaneously, the system identifies the delivery tasks contained in the unfinished delivery task subsequences of the faulty AGV as delivery tasks to be reassigned for the faulty AGV. For each delivery task to be reassigned, the system extracts its AGV number, task number, task location, material call time, material demand, and optimal material delivery time. For each AGV in the AGV cluster, the system extracts its AGV number, real-time location, operating status, and current remaining material load. Finally, the system combines the status information of each AGV, the information of all delivery tasks to be reassigned, and the time of the fault to construct the DQN state vector. The DQN state vector characterizes the operating status, remaining tasks, and time information of the AGV cluster at the time of the fault, and serves as the state input for subsequent DQN network rescheduling.

[0047] Step 6: Input the DQN state vector into a preset DQN network to obtain the Q-values ​​corresponding to each rescheduling method, and determine the rescheduling method corresponding to the largest Q-value as the selected rescheduling method. Specifically, the system inputs the DQN state vector constructed in Step 5 into the preset DQN network. The preset DQN network is a deep Q-network pre-trained based on a training example set, whose input is the DQN state vector and whose output is the Q-values ​​corresponding to different rescheduling methods. Each Q-value represents the degree of superiority or inferiority of adopting the corresponding rescheduling method under the current fault scenario.

[0048] Rescheduling methods include global re-optimization, insertion-based rescheduling, substitute succession, and local re-optimization. The system obtains the Q-values ​​corresponding to each of these rescheduling methods through a preset DQN network, compares the magnitudes of the Q-values, and determines the rescheduling method corresponding to the largest Q-value as the rescheduling method to be selected under the current fault scenario.

[0049] The global re-optimization method refers to redistributing all delivery tasks to be reassigned, starting from the real-time position of each AGV in normal status. When each AGV in normal status cannot complete all delivery tasks to be reassigned, a new AGV is dispatched from the warehouse, and the new AGV participates in the task redistribution together with the AGVs in normal status. Figure 2As shown, the first and second AGVs depart from the warehouse to perform delivery tasks. Before the failure, the delivery task sequence for the first AGV was 9→79→89→90→70→60→30→20→19→5, and for the second AGV it was 34→25→49→37→36→56→85→81→71→41. After the first AGV completed delivery tasks numbered 9 and 79, and the second AGV completed delivery task number 34, the second AGV failed and could not continue to perform subsequent delivery tasks. At this point, the unfinished tasks of the first and second AGVs were combined into a set of delivery tasks to be reassigned, and a global re-optimization was performed using the current position of the first AGV and the warehouse departure position of the newly added third AGV as the starting point for rescheduling. After the reassignment, the first AGV continued to perform delivery tasks numbered 89, 36, 70, 25, 85, 20, 49, and 5, while the newly added third AGV performed delivery tasks numbered 90, 60, 30, 19, 37, 56, 81, 71, and 41, thus completing the rescheduling of all remaining delivery tasks in the event of the failure.

[0050] Insertive rescheduling refers to inserting the delivery tasks corresponding to the faulty AGV into feasible positions within the incomplete delivery task subsequence of each normal AGV, while maintaining the relative execution order of the original incomplete delivery tasks of each normal AGV. For example... Figure 3 As shown, after the first AGV completes delivery tasks numbered 9, 79, 89, 90, 70, 60, 30, 20, and 19, and the second AGV completes delivery tasks numbered 34, 25, 49, 37, 36, 56, 85, and 81, the first AGV malfunctions and cannot continue with delivery task number 5. At this point, task number 5 is designated as a delivery task to be reassigned and inserted into the second AGV's incomplete delivery task subsequence using an insert-type rescheduling method. When the remaining material quantity carried by the second AGV is greater than or equal to the material requirement of task number 5, task number 5 meets the insertion execution condition and can be inserted after the second AGV's original incomplete delivery task subsequence 71→41. After reassignment, the second AGV's delivery task sequence is adjusted to 71→41→5, maintaining the relative execution order of its original incomplete delivery tasks 71→41 while completing the insert-type rescheduling of the malfunctioning AGV's incomplete delivery tasks.

[0051] The replacement and succession method refers to dispatching a new AGV from the warehouse. This new AGV continues to execute the delivery tasks corresponding to the failed AGV, following the original execution order of the unfinished delivery task sub-sequences of the failed AGV, while maintaining the unfinished delivery task sub-sequences of each normal AGV unchanged. For example... Figure 4As shown, after the first AGV completes delivery task number 9, the second AGV malfunctions and cannot continue its corresponding delivery task. In this case, a replacement and rescheduling method is used. The sequence of unfinished delivery tasks for the first AGV remains unchanged: 79→89→90→70→60→30→20→19→5. Simultaneously, a newly added third AGV is dispatched from the warehouse to take over the second AGV's unfinished delivery tasks. The newly added third AGV executes tasks numbered 34, 25, 49, 37, 36, 56, 85, 81, 71, and 41 in the order of the second AGV's original unfinished delivery task sequence. This allows the unfinished delivery tasks of the malfunctioning AGV to be continued without altering the original task execution order of the normally functioning AGVs.

[0052] Local re-optimization refers to dispatching a new AGV from the warehouse and, using the warehouse as the starting point for the new AGV, re-optimizing the delivery tasks to be reassigned for the faulty AGV, generating a delivery task sequence for the new AGV, while keeping the incomplete delivery task sub-sequences of each normal AGV unchanged. For example... Figure 5 As shown, after the first AGV completes delivery tasks numbered 9 and 79, it malfunctions and cannot continue its subsequent delivery tasks. At this time, the unfinished delivery task subsequence for the first AGV is 89→90→70→60→30→20→19→5. The second AGV is in normal condition, and its unfinished delivery task subsequence remains unchanged. A local re-optimization approach is adopted for rescheduling, that is, a new third AGV is dispatched from the warehouse, and the warehouse is used as the scheduling starting point for the third AGV. Only the delivery tasks corresponding to the first AGV that need to be reassigned are re-optimized. After re-optimization, the newly added third AGV executes delivery tasks numbered 60, 19, 89, 30, 5, 70, 20, and 90; the second AGV continues to execute delivery tasks according to the original unfinished delivery task subsequence 25→49→37→36→56→85→81→71→41. Therefore, while maintaining the original task execution order of AGVs in normal condition, a partial re-optimization allocation of delivery tasks not completed by faulty AGVs is achieved.

[0053] During the training of the pre-defined DQN network, DQN state vectors constructed under different fault scenarios are used as state inputs, and global re-optimization, insertional rescheduling, substitute succession, and local re-optimization are used as action spaces. The pre-defined DQN network includes an input layer, hidden layers, and an output layer. The number of nodes in the input layer corresponds to the dimension of the DQN state vector. The hidden layer consists of two fully connected layers: the first hidden layer contains 128 neurons, and the second hidden layer contains 64 neurons. Each hidden layer uses the ReLU function as its activation function. The output layer contains four neurons, corresponding to the Q-values ​​of the global re-optimization, insertional rescheduling, substitute succession, and local re-optimization methods, respectively. During training, multiple fault scenario samples are randomly generated based on a pre-defined training example set. The system inputs the DQN state vectors constructed based on the workshop state information at the time of the fault into the DQN network. The DQN network uses... The strategy selects the rescheduling method, in which the initial training phase Set to 1, minimum exploration rate to 0.05, and gradually reduce ε value according to preset decay coefficient 0.995 to balance action exploration and strategy utilization during training.

[0054] For the selected rescheduling method, the system updates the current scheduling plan according to the rescheduling method and calculates the total transportation cost of the AGV cluster after completing all remaining delivery tasks using this rescheduling method. The reward value is determined based on the total transportation cost of the AGV cluster, where a smaller total transportation cost results in a larger reward value. The reward value *r* is calculated according to the formula: The system calculates and stores the DQN state vector, the selected rescheduling method, the reward value, and the updated workshop state information into an experience replay pool with a capacity of 5000. During network parameter updates, a mini-batch of training samples is randomly selected from the experience replay pool for training. The mini-batch size is set to 64, and a discount factor is applied. The learning rate is set to 0.95 and the Adam optimizer is used to update the DQN network parameters. The DQN network consists of a current Q-network and a target Q-network. The current Q-network outputs the Q-values ​​corresponding to each rescheduling method, and the target Q-network calculates the target Q-value. After every 100 parameter updates, the parameters of the current Q-network are copied to the target Q-network. The loss function for network training is the mean squared error between the current Q-value and the target Q-value. The system iteratively updates the DQN network parameters by minimizing this loss function.

[0055] The preset training termination conditions include: the number of training rounds reaches a preset maximum number of training rounds, or the change in the average reward value during a preset number of consecutive training rounds is less than a preset reward change threshold. In this embodiment, the maximum number of training rounds is set to 1000 rounds. If the change in the average reward value obtained from 50 consecutive training rounds is less than 0.001, the DQN network training is determined to have reached convergence, and training is terminated, resulting in the preset DQN network. Through the above training process, the preset DQN network can learn the correspondence between different fault scenario states and rescheduling methods. When a fault occurs, it adaptively outputs the Q value corresponding to each rescheduling method based on the workshop state at the time of the fault, thereby selecting the rescheduling method suitable for the current fault scenario, reducing the reliance on manually set scheduling rules, and improving the adaptability and operational efficiency of AGV cluster rescheduling under fault scenarios.

[0056] Step 7: Update the current scheduling scheme according to the selected rescheduling method, distribute the updated scheme to the AGV cluster, and control the AGV cluster to continue executing delivery tasks according to the updated scheme. Then return to Step 2 until all delivery tasks are completed. Specifically, the system adjusts the current scheduling scheme based on the rescheduling method determined in Step 6. During the adjustment, the system retains the completed delivery task sub-sequences for each AGV and removes the delivery tasks to be reassigned corresponding to the faulty AGV from its incomplete delivery task sub-sequence. Subsequently, the system reassigns the delivery tasks to be reassigned to normal AGVs or to newly added AGVs according to the selected rescheduling method. When the selected rescheduling method requires dispatching a new AGV, the system adds the new AGV to the AGV cluster and adds the corresponding delivery task sequence to the current scheduling scheme. After the adjustment is complete, the system obtains the updated current scheduling scheme and distributes it to all participating AGVs. Each AGV continues to execute the delivery task according to the updated sequence of unfinished delivery tasks. The system returns to step 2 and continues to monitor the workshop status information until all delivery tasks are completed.

[0057] In summary, this invention monitors workshop status information in real time during task execution and determines whether to end the scheduling process based on the completion status of delivery tasks. When an AGV malfunction is detected, the current workshop status information is determined as the workshop status information at the time of the malfunction. Based on the unfinished delivery task subsequences of each AGV, all delivery tasks to be reassigned and the delivery tasks to be reassigned for the malfunctioning AGV are determined. A DQN state vector is constructed by combining the real-time position, operating status, remaining amount of material currently being carried, and the time of the malfunction of each AGV. Subsequently, the DQN state vector is input into a preset DQN network to obtain the Q values ​​corresponding to the global re-optimization method, the insertion re-scheduling method, the substitute succession method, and the local re-optimization method, respectively. The re-scheduling method corresponding to the maximum Q value is determined as the re-scheduling method selected under the current malfunction scenario. Finally, the current scheduling scheme is updated according to the selected re-scheduling method, retaining the completed delivery task subsequences of each AGV, adjusting the unfinished delivery task subsequences of each AGV, dispatching new AGVs to participate in delivery when necessary, and distributing the updated current scheduling scheme to the AGV cluster so that the AGV cluster continues to execute the remaining delivery tasks until all delivery tasks are completed. This invention can adaptively select a suitable rescheduling method based on the real-time status of the workshop when a fault occurs, reducing the impact of sudden AGV failures on the continuous execution of delivery tasks, improving the scheduling recovery efficiency, operational continuity and scheduling stability of AGV clusters in fault scenarios, and helping to reduce the total transportation cost of AGV clusters.

[0058] The following is a specific example to further describe and illustrate the invention. For example... Figure 6 As shown, this example illustrates a workshop logistics delivery scenario consisting of a single warehouse and multiple AGVs of the same model. Based on the delivery demands generated by the material demand locations within the workshop over a certain time period, workshop scheduling data is initialized. This data includes the total number of delivery tasks, the task number of each task, the task location, the material call time, the material demand quantity, and the optimal delivery time. The task location is represented by horizontal and vertical axes. The goal is to determine a scheduling scheme that reduces the total transportation cost of the AGV cluster and control the AGV cluster to execute delivery tasks according to this scheme. The total transportation cost of the AGV cluster includes AGV travel costs, penalty costs for early or late material delivery, and AGV activation costs. In this embodiment, the warehouse node coordinates are (0, 0), the AGV departure time is 365s, the maximum loading capacity is 300kg, the travel speed is 1m / s, and the unloading time is 15s. The material demand locations within the workshop are arranged in a 10×10 grid, with a horizontal distance of 5m and a vertical distance of 8m between adjacent material demand locations.

[0059] The application of this invention is further illustrated below by listing the fifth specific instance T20I5, which includes 20 task points in the test examples of this invention. Specific data for instance T20I5 are shown in Table 1:

[0060] Table 1: Detailed Data Table for Example T20I5

[0061]

[0062] To verify the effectiveness of the Deep Q-Network-based Adaptive Rescheduling Method (DQN-ARS) in fault scenarios, it was compared with four fixed rescheduling methods: Global Re-optimization (GRO), Insertion Rescheduling (IRS), Replacement Succession (RCS), and Local Re-optimization (LRO). These four fixed rescheduling methods consistently use the same rescheduling method after an AGV failure, while the proposed DQN-ARS method adaptively selects a rescheduling method suitable for the current fault scenario based on the workshop status information at the time of the fault. To reduce experimental error, test instances were selected, with each instance repeated 30 times to generate statistical results. This embodiment uses Relative Percentage Deviation (RPD) as the performance evaluation metric. The formula for calculating RPD is:

[0063] ,in, This represents the minimum total AGV cluster transportation cost obtained by repeatedly executing a certain rescheduling method 30 times when resolving a specific fault scenario instance. This represents the minimum total AGV cluster transportation cost obtained by DQN-ARS and the four fixed rescheduling methods when solving this instance. Clearly, the smaller the RPD value, the closer the current method is to the optimal rescheduling result after a failure, indicating that the method has better performance in controlling the total AGV cluster transportation cost.

[0064] like Figure 7 As shown, the method proposed in this invention achieves the smallest relative percentage deviation value, and its confidence interval does not overlap with the confidence intervals obtained by using the four rescheduling methods individually. This indicates that the method of this invention can adaptively select the appropriate rescheduling method according to the workshop status under different fault scenarios, and has a better optimization effect than using a fixed rescheduling method. Figure 8As shown, compared with the methods of global re-optimization, insertion re-scheduling, substitute succession, and local re-optimization, the proposed method has the most compact bin and the overall position of the whiskers representing the maximum and minimum values ​​of the data is relatively low. This indicates that the relative percentage deviation and fluctuation range of the results obtained by the proposed method are smaller, and it is superior to the single re-scheduling method in terms of solution quality and stability.

[0065] Through verification of specific embodiments of the present invention, it is demonstrated that the present invention can effectively solve the problems of task interruption, execution delay, and difficulty in continuing the original scheduling scheme due to sudden failures during the delivery process of AGV clusters. When an AGV fails, the present invention can construct a DQN state vector based on the workshop state information at the time of the failure, and adaptively select an appropriate rescheduling method through a preset DQN network, thereby updating the current scheduling scheme in a timely manner and enabling the AGV cluster to continue executing the remaining delivery tasks. Through the above process, the present invention can improve the efficiency of task reassignment and scheduling recovery capability after a failure, and reduce the impact of failure on the continuous operation of the AGV cluster. Therefore, the reinforcement learning-driven adaptive rescheduling method for AGV clusters proposed in this invention has significant improvements in task reassignment capability, scheduling adaptability, and total transportation cost control of AGV clusters in failure scenarios.

[0066] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An adaptive rescheduling method for AGV clusters based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Initialize workshop scheduling data, use intelligent optimization algorithm to generate initial scheduling scheme, and determine the initial scheduling scheme as the current scheduling scheme; S2. Control the AGV cluster to execute delivery tasks according to the current scheduling plan, and obtain workshop status information in real time during task execution; S3. Determine whether all delivery tasks have been completed based on the workshop status information. If all delivery tasks have been completed, end the scheduling process and output the final total transportation cost of the AGV cluster; otherwise, proceed to step S4. S4. Determine whether there is a faulty AGV based on the workshop status information. If there is no faulty AGV, return to step S2. If there is a faulty AGV, determine the current workshop status information as the workshop status information when the fault occurred, and execute step S5. S5. Construct a deep Q-network (DQN) state vector based on the workshop state information at the time of the fault. S6. Input the DQN state vector into the preset DQN network to obtain the Q value corresponding to each rescheduling method, and determine the rescheduling method corresponding to the maximum Q value as the selected rescheduling method. S7. Update the current scheduling scheme according to the selected rescheduling method, send the updated current scheduling scheme to the AGV cluster, control the AGV cluster to continue to execute the delivery task according to the updated current scheduling scheme, and return to step S2 until all delivery tasks are completed.

2. The adaptive rescheduling method for AGV clusters based on deep reinforcement learning according to claim 1, characterized in that, In step S1, the workshop scheduling data includes the total number of delivery tasks, the task number of each delivery task, the task location, the material call time, the material requirement, and the optimal delivery time for the materials. The task location is represented by the horizontal and vertical axes. The scheduling scheme uses a set... This indicates that m represents the number of AGVs participating in the delivery in the scheduling plan, and m increases accordingly after dispatching a new AGV. This represents the delivery task sequence corresponding to the k-th AGV in the AGV cluster. Let i be the i-th task of the k-th AGV, where... Let i be the task number of the i-th delivery task performed by the k-th AGV. Let x and y be the x and y coordinates of the task position of the i-th delivery task performed by the k-th AGV, respectively. The material calling time for the i-th delivery task performed by the k-th AGV. Let i be the material requirement for the i-th delivery task performed by the k-th AGV. The optimal delivery time for materials for the i-th delivery task performed by the k-th AGV. This represents the number of delivery tasks for the k-th AGV. The sum of the number of delivery tasks for each AGV equals the total number of delivery tasks.

3. The adaptive rescheduling method for AGV clusters based on deep reinforcement learning according to claim 1, characterized in that, In step S1, when generating the initial scheduling scheme using the intelligent optimization algorithm, the optimization objective is to minimize the total transportation cost of the AGV cluster. Within a preset optimization time, candidate scheduling schemes are continuously searched, and the candidate scheduling scheme with the minimum total transportation cost of the AGV cluster is determined as the initial scheduling scheme. The formula for calculating the total transportation cost of the AGV cluster is as follows: ,in, Let be the total transportation cost of the AGV cluster, representing the sum of the transportation costs of m AGVs in the AGV cluster. The index is denoted by 0, where 0 is the index of the repository node, and 1 to 1. Let the index of the delivery task for the k-th AGV be the index. This is the cost coefficient per unit travel distance for AGVs. This is the penalty cost coefficient for early or late delivery of materials. A cost coefficient is applied to each AGV. Enable variables for AGVs, when the k-th AGV is enabled. =1 otherwise 0, Let be a binary decision variable, representing that if the travel path of the k-th AGV includes the route from the delivery task... To the delivery task ,but =1 otherwise 0, For the kth AGV from the delivery task To the delivery task The driving distance between them For the kth AGV to arrive at the delivery task The actual delivery time of the materials. For delivery tasks The absolute time deviation between the optimal delivery time of materials and the actual delivery time of materials.

4. The adaptive rescheduling method for AGV clusters based on deep reinforcement learning according to claim 1, characterized in that, In step S2, the workshop status information includes the AGV number, real-time location, operating status, remaining amount of material being transported, execution progress of the delivery task sequence, and current workshop timestamp for each AGV in the AGV cluster. The operating status includes normal status and fault status. The execution progress of the delivery task sequence includes the completed delivery task subsequence and the incomplete delivery task subsequence for each AGV. The incomplete delivery task subsequence includes the currently executing delivery task and the remaining delivery tasks to be executed.

5. The adaptive rescheduling method for AGV clusters based on deep reinforcement learning according to claim 1, characterized in that, In step S5, the delivery tasks contained in the unfinished delivery task subsequence of all AGVs are identified as all delivery tasks to be reassigned, and the delivery tasks contained in the unfinished delivery task subsequence of the faulty AGV are identified as delivery tasks to be reassigned for the faulty AGV.

6. The adaptive rescheduling method for AGV clusters based on deep reinforcement learning according to claim 1, characterized in that, In step S5, the DQN state vector includes the AGV number, real-time location, running status, remaining amount of material currently being carried, information on all delivery tasks to be reassigned, and the time of the fault occurrence for each AGV in the AGV cluster.

7. The adaptive rescheduling method for AGV clusters based on deep reinforcement learning according to claim 1, characterized in that, In step S6, the preset DQN network is a deep Q network trained based on a preset training example set. During training, the DQN state vector is used as the state input, the rescheduling method is used as the action space, and the reward value is determined according to the total transportation cost of the AGV cluster after completing the delivery task using each rescheduling method. The smaller the total transportation cost of the AGV cluster, the larger the corresponding reward value. The network parameters of the deep Q network are iteratively updated according to the reward value until the preset training termination condition is met, and the preset DQN network is obtained.

8. The adaptive rescheduling method for AGV clusters based on deep reinforcement learning according to claim 1, characterized in that, The rescheduling methods in step S6 include global re-optimization, insertion rescheduling, substitute succession, and local re-optimization. The global re-optimization method is as follows: using the intelligent optimization algorithm in step S1, starting from the real-time position of each normal AGV, all delivery tasks to be reassigned are reassigned. When the normal AGVs cannot form a feasible allocation scheme covering all delivery tasks to be reassigned, a new AGV is dispatched from the warehouse. Starting from the real-time position of each normal AGV and the warehouse position of the new AGV, the intelligent optimization algorithm in step S1 is used to jointly allocate all delivery tasks to be reassigned. The insert-type rescheduling method involves inserting the delivery task to be reassigned corresponding to the faulty AGV into a feasible position in the incomplete delivery task subsequence of each normal AGV, while maintaining the relative execution order of the original delivery tasks of each normal AGV. The replacement and succession method is as follows: a new AGV is dispatched from the warehouse, and the new AGV continues to execute the delivery tasks to be reassigned corresponding to the faulty AGV according to the execution order of the unfinished delivery task sub-sequences of the faulty AGV, while keeping the unfinished delivery task sub-sequences of each normal AGV unchanged. The local re-optimization method involves dispatching a new AGV from the warehouse, using the intelligent optimization algorithm in step S1, taking the warehouse as the starting point of the new AGV, re-optimizing the delivery tasks to be reassigned corresponding to the faulty AGV, generating a delivery task sequence corresponding to the new AGV, and keeping the unfinished delivery task sub-sequences of each normal AGV unchanged.

9. The adaptive rescheduling method for AGV clusters based on deep reinforcement learning according to claim 1, characterized in that, In step S7, based on the selected rescheduling method, the completed delivery task subsequences of each AGV in the current scheduling scheme are retained. The delivery tasks to be reassigned corresponding to the faulty AGV are deleted from the incomplete delivery task subsequences of the faulty AGV, and the incomplete delivery task subsequences of each AGV are adjusted. When the selected rescheduling method requires dispatching a new AGV, the new AGV is added to the AGV cluster, and the delivery task sequence corresponding to the new AGV is added to the current scheduling scheme to obtain the updated current scheduling scheme. The updated current scheduling scheme is then sent to each AGV participating in the delivery, and each AGV is controlled to continue executing the delivery task according to its corresponding incomplete delivery task subsequence, and the process returns to step S2.

Images