A coal sample transfer robot scheduling method based on reinforcement learning

Abstract

The present application relates to the technical field of distribution scheduling, and discloses a coal sample transfer robot scheduling method based on reinforcement learning, comprising: constructing a scheduling environment model of coal sample transfer, including state information of a plurality of coal sample tasks, heterogeneous transport carriers and processing nodes; defining a state space and an action space of a reinforcement learning intelligent agent based on the scheduling environment model; the state space represents the real-time state of the scheduling environment, and the action space covers an effective scheduling action combination composed of sample types, transport carriers and target devices; designing a reward function with the goal of optimizing scheduling efficiency, training the intelligent agent using reinforcement learning until the strategy converges, obtaining an optimal scheduling strategy, deploying the scheduling strategy to a coal sample transfer robot system, and outputting a scheduling action according to the real-time environment state through a greedy strategy to complete the coal sample transfer task. The problems of uneven task distribution and untimely path conflict processing are solved, and the purposes of optimizing resource allocation and improving scheduling efficiency are achieved.

Description

Technical Field

[0001] This invention relates to the field of allocation and scheduling technology, and in particular to a scheduling method for coal sample transfer robots based on reinforcement learning. Background Technology

[0002] In the process of intelligent transformation of the modern coal mining industry, coal sample transfer is a key process connecting production and testing. Its efficiency and accuracy directly affect the timeliness of coal quality analysis results, the efficiency of subsequent production process adjustments, and the level of coal product quality control. With the expansion of coal mining scale and the improvement of automation level, the traditional coal sample transfer mode relying on manual labor or fixed routes is no longer suitable for the operational needs of modern coal enterprises in terms of multiple sampling points, high transfer frequency, and dynamic task priorities. It has gradually exposed technical bottlenecks such as uneven task allocation, untimely handling of path conflicts, low efficiency, and heavy reliance on the experience judgment of dispatchers. Therefore, researching robot task scheduling methods adapted to the characteristics of coal sample transfer scenarios has become a key technical requirement for promoting the intelligent upgrading of coal sample transfer in coal mines. Current technologies include designing task scheduling models for robot task scheduling problems in automated coal sample preparation systems and solving them using genetic algorithms and particle swarm optimization algorithms; and classifying various tasks in the sample preparation process according to priority, with high-priority tasks completed before low-priority tasks, ultimately realizing the scheduling of coal robot sample preparation systems. While heuristic algorithms have seen some application in optimizing scheduling tasks, the time required for algorithm iteration prevents them from achieving the high real-time performance required for coal sample transfer scheduling, thus failing to address the challenges of dynamic multi-node, multi-batch coal sample transfer. In contrast to traditional heuristic algorithms, reinforcement learning methods do not rely on manually designed heuristic rules. Instead, they iteratively update strategies through interaction with the environment, enabling real-time optimal scheduling under multi-task, multi-resource, and dynamic operating conditions. Especially in coal sample transfer scenarios, where task batches are numerous and priorities change frequently, reinforcement learning methods can adaptively cope with the complex and ever-changing environment, improving scheduling efficiency.

[0003] For example, Chinese patent CN118536783B discloses a logistics robot scheduling method based on deep reinforcement learning, providing the following technical solution: First, a deep state-value network (DVN) is constructed and trained. Through offline learning of historical order data, the state-value function is updated using a temporal difference method, and the scheduling strategy is optimized using a multi-logistics robot deep reinforcement learning method. Then, a combined DVN and KM optimization algorithm is used to achieve real-time matching between logistics robots and orders. Finally, idle logistics robots are scheduled to go to areas with higher expected returns to wait for the next round of scheduling. This invention achieves efficient scheduling of logistics orders by hospital logistics robots through the construction of a deep state-value network (DVN) and the use of a combined optimization method. This invention has significant advantages in improving order response speed and reducing order waiting time, and is applicable to various hospital scenarios, effectively improving logistics scheduling efficiency and optimizing resource allocation. However, the aforementioned deep reinforcement learning-based logistics robot scheduling method is limited to logistics scenarios and cannot adapt to the complex scheduling needs of industrial environments such as coal mines. Its scheduling method lacks the ability to coordinate multiple devices and dynamically adjust priorities. The reward function is based on simple time-consuming optimization and is difficult to handle multiple constraint problems, such as common equipment load and real-time conflict avoidance. Summary of the Invention

[0004] This invention addresses the problems of uneven task allocation, untimely handling of path conflicts, and low efficiency in existing technologies. It proposes a coal sample transfer robot scheduling method based on reinforcement learning, which achieves the goals of optimizing resource allocation, reducing transportation conflicts, and improving scheduling efficiency.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A reinforcement learning-based scheduling method for coal sample transfer robots includes:

[0007] Construct a scheduling environment model for coal sample transfer, including several types of coal sample tasks, heterogeneous transport carriers, and status information of processing nodes;

[0008] Based on the scheduling environment model, the state space and action space of the reinforcement learning agent are defined; the state space represents the real-time state of the scheduling environment, and the action space covers the effective combination of scheduling actions consisting of sample type, transport vehicle and target device.

[0009] A reward function is designed to optimize scheduling efficiency. The agent is trained using reinforcement learning until the policy converges to obtain the optimal scheduling policy. This policy is then deployed to the coal sample transfer robot system. The system outputs scheduling actions based on the real-time environmental state using a greedy policy to complete the coal sample transfer task.

[0010] By comprehensively modeling the scheduling environment and applying reinforcement learning, intelligent and adaptive optimization of the coal sample transfer process was achieved, effectively solving the bottleneck problems of uneven task allocation and untimely handling of path conflicts in traditional methods, and significantly improving scheduling efficiency and resource utilization.

[0011] Preferably, the construction of the scheduling environment model for coal sample transfer specifically includes: defining the core elements of coal sample transfer, wherein the core elements include coal sample task characteristics, transportation resource information, and the status of each equipment node.

[0012] Preferably, the coal sample task features include analytical sample preparation and transportation tasks, full water sample delivery and transportation tasks, and retrieval sample storage and transportation tasks; the transportation resource information includes several automated guided vehicles (AGV) units and at least one pneumatic transmission device; the status of each equipment node includes the working status of the retrieval sample cabinet, the full water equipment and the sample preparation machine, and the analytical sample unit, and each node has a preset processing time.

[0013] The specific task characteristics and resource information make the scheduling model more in line with the actual coal sample transfer scenario, taking into account the collaborative work of multiple task types and heterogeneous carriers, thus improving the targeting and reliability of the scheduling.

[0014] Preferably, the state space specifically includes: the number of samples to be transferred for various coal sample tasks, the range of which is limited by the total number of samples in a single task; the remaining load capacity of each AGV unit, the range of which is limited by the maximum load capacity of the AGV; the working status of the pneumatic transmission device, including idle, busy, and switching; the number of idle slots in each processing node, the range of which is limited by the maximum number of slots in the node; the cumulative time elapsed from task start to the current moment; the total number of coal samples allocated to the AGV but not yet reaching the batch quantity, and tracking the partial load status.

[0015] Comprehensive state variables cover key dynamic factors of the scheduling environment, ensuring that the agent can perceive the system state in real time, make accurate decisions, avoid resource conflicts and inefficiencies, and improve the response speed and stability of scheduling.

[0016] Preferably, the action space is a discrete action space that covers all valid scheduling action combinations that conform to the system rules. Each action consists of three parts: sample type, transport vehicle, and target device. The sample type covers all defined coal sample task categories, the transport vehicle includes all available AGV units and pneumatic transmission devices, and the target device includes all applicable processing nodes. The action space excludes invalid actions through a constraint check mechanism.

[0017] Discrete action space simplifies the decision-making process, and constraint checking mechanism ensures the feasibility and safety of actions, reduces invalid operations, and makes scheduling more efficient and stable, making it suitable for complex and ever-changing real-world application scenarios.

[0018] Preferably, the reward function specifically includes: a first reward component with minimizing the total completion time of coal sample transfer as the core objective, wherein the total completion time is defined as minimizing the maximum value among the completion times of all types of coal sample tasks, and the completion time of each type of task is dynamically calculated as a function of the earliest time the sample arrives at the equipment and the equipment processing time and transportation time, and the transportation time is determined according to the path distance, carrier speed and sample quantity; and a second reward component that introduces constraints to penalize scheduling actions that violate system rules, including applying negative rewards when the action causes the transport carrier to be overloaded, the equipment to conflict, or the path to be invalid.

[0019] The hybrid reward function aims to optimize the total completion time while guiding the agent to learn compliance strategies through constraint penalties. This effectively balances efficiency and rule adherence, promotes long-term performance optimization, and avoids local optima.

[0020] Preferably, the reinforcement learning specifically includes: the agent adopts an Actor-Critic structure, collects experience data by interacting with the environment, estimates and calculates the policy gradient using generalized advantage, and stabilizes the training process through a pruning mechanism. The training hyperparameters include the network learning rate, discount factor, and smoothing coefficient. During the training process, the agent gradually optimizes the policy until the reward curve converges, indicating that a stable scheduling capability has been obtained.

[0021] The Actor-Critic structure combined with the PPO algorithm improves training efficiency and stability, while the pruning mechanism prevents policy mutations and ensures smooth convergence of the learning process.

[0022] Preferably, the deployment to the coal sample transfer robot system specifically includes: loading the parameters of the trained and converged agent model and setting it to evaluation mode; initializing the scheduling environment; inputting the amount of coal samples to be transferred; the agent acquiring the environmental status in real time; selecting the most probable effective action to execute based on a greedy strategy; recording the sample ID type, carrier, device, and timestamp after each coal sample transfer is completed; and generating a detailed scheduling log; and outputting the total completion time after all coal samples have completed the entire transfer process to evaluate the scheduling performance.

[0023] As a preferred option, the greedy strategy specifically includes: prioritizing the dispatch of idle devices; when multiple devices are idle, they are sorted and assigned according to their idle duration; when all devices are busy, they are sorted and processed according to their arrival time; the system iteratively calculates the intervention effect of AGV and pneumatic transmission; and always selects the carrier with the shorter expected time.

[0024] The greedy strategy achieves real-time and efficient scheduling through simple priority rules. Prioritizing idle devices reduces waiting time, and the intervention of iterative computing carriers optimizes resource selection, significantly improving task completion speed and system throughput.

[0025] Preferably, the AGV unit has a preset maximum load capacity, sample handling time, and transport speed, the pneumatic transmission device has a preset transmission speed, and there is a preset switching time between the pneumatic and AGV units.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows.

[0027] 1. This invention, through dynamic optimization of task allocation and path planning, can respond in real time to the demands of multiple sampling points and high transfer frequency, effectively reducing waiting time between tasks. Compared with traditional methods relying on manual labor or fixed paths, it utilizes the adaptive characteristics of reinforcement learning to avoid path conflicts and equipment idleness, thereby significantly shortening the overall time consumption of coal sample testing. This efficient scheduling ensures timely output of coal quality analysis results, supports rapid adjustments to subsequent processes, and comprehensively improves the continuity and timeliness of coal mine production.

[0028] 2. This invention improves the overall utilization rate of transportation resources by constructing a hybrid reward function to intelligently balance the use of AGV robots and pneumatic transmission units. Real-time monitoring of equipment load and carrier status avoids resource waste or over-concentration, ensuring balanced load across nodes. This dynamic scheduling solves problems such as uneven load distribution among equipment nodes and path conflicts for transport carriers in traditional methods, increasing system throughput, reducing energy consumption, and providing a sustainable resource allocation solution for large-scale coal mine operations.

[0029] 3. This invention utilizes reinforcement learning to enable the system to autonomously learn optimal strategies, reducing reliance on human experience. Through interactive and iterative updates with the environment, it can cope with dynamic operating conditions such as changes in workload and fluctuations in equipment status, providing stable and reliable scheduling decisions. This not only reduces labor costs but also enhances the system's stability and flexibility. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of a coal sample transfer robot scheduling method based on reinforcement learning according to the present invention.

[0031] Figure 2 The left half of the system structure diagram of the coal sample transfer robot scheduling method based on reinforcement learning of the present invention is shown.

[0032] Figure 3 The right half of the system structure diagram of the coal sample transfer robot scheduling method based on reinforcement learning of the present invention is shown.

[0033] Figure 4 This is a flowchart of the PPO algorithm update for a coal sample transfer robot scheduling method based on reinforcement learning, as described in this invention.

[0034] Figure 5This is a scheduling training curve diagram in one embodiment of the coal sample transfer robot scheduling method based on reinforcement learning of the present invention.

[0035] Figure 6 This is a Gantt chart for scheduling coal sample transfer robots using the optimal strategy in one embodiment of the reinforcement learning-based coal sample transfer robot scheduling method of the present invention. Detailed Implementation

[0036] See Figures 1-6 As shown, a reinforcement learning-based scheduling method for coal sample transfer robots includes:

[0037] Construct a scheduling environment model for coal sample transfer, including several types of coal sample tasks, heterogeneous transport carriers, and status information of processing nodes;

[0038] Based on the scheduling environment model, the state space and action space of the reinforcement learning agent are defined; the state space represents the real-time state of the scheduling environment, and the action space covers the effective combination of scheduling actions consisting of sample type, transport vehicle and target device.

[0039] A reward function is designed to optimize scheduling efficiency. The agent is trained using reinforcement learning until the policy converges to obtain the optimal scheduling policy. This policy is then deployed to the coal sample transfer robot system. The system outputs scheduling actions based on the real-time environmental state using a greedy policy to complete the coal sample transfer task.

[0040] The purpose of this invention is to overcome the problems of low resource utilization, poor batch coordination, multi-node conflicts (such as uneven equipment node load, transport carrier path conflicts, and scheduling strategies not strictly following sequential logic) in existing coal sample transfer scheduling technologies, as well as insufficient dynamic adaptability to cope with real-time changes in task volume, equipment status, and operating conditions. This invention provides a task scheduling method for coal sample transfer robots based on reinforcement learning. This invention constructs a coal sample transfer task environment model involving full water, sample preparation, and sample analysis, abstracting the coal sample transfer scenario into a reinforcement learning environment. Subsequently, the state space and action space of coal sample transfer are designed, taking into account the selection and switching between robot AGV units and pneumatic transmission units. A hybrid reward function is constructed, balancing completion time minimization and resource utilization. The PPO algorithm in deep reinforcement learning is used for training, combining the advantages of AGV and pneumatic transmission, enabling the robot to autonomously learn the optimal task scheduling strategy in the coal sample transfer environment, improving overall resource utilization and achieving dynamic allocation.

[0041] This invention enables dynamic optimization and scheduling of nodes from sample preparation equipment to sample storage cabinets and analysis units, improving the utilization rate of transportation resources, reducing batch transfer waiting time, resolving multi-node scheduling conflicts, and ultimately shortening the overall coal sample testing process. It meets the demands for high efficiency and real-time performance in coal sample testing, reduces reliance on manual experience, and provides an intelligent scheduling solution for coal sample testing in coal-related enterprises such as coal-fired power plants. This invention is applicable to transfer scenarios involving multiple devices, multiple batches, and multiple transportation modes in the coal sample testing process.

[0042] like Figures 1-4 In one embodiment shown, Figure 1 This is an overall flowchart of a method for encoding the unloading status of power inspection tasks according to the present invention. Figure 2 The left half of the system structure diagram of the coal sample transfer robot scheduling method based on reinforcement learning of the present invention is shown. Figure 3 The right half of the system structure diagram of the coal sample transfer robot scheduling method based on reinforcement learning of the present invention is shown. Figure 4 This is a flowchart illustrating the PPO algorithm update process of a reinforcement learning-based coal sample transfer robot scheduling method according to the present invention. First, a scheduling environment model for coal sample transfer is constructed. This model includes several coal sample tasks, heterogeneous transport vehicles, and the state information of processing nodes. Specifically, coal sample task characteristics include analytical sample preparation and transport tasks, full water sample delivery and transport tasks, and retrieval sample storage and transport tasks. Transport resource information includes several automated guided vehicle (AGV) units and at least one pneumatic transport device. The state of each equipment node includes the working status of the retrieval sample cabinet, full water equipment, sample preparation machine, and analytical sample unit, and each node has a preset processing time. The AGV unit has a preset maximum load capacity, sample handling time, and transport speed. The pneumatic transport device has a preset transport speed, and there is a preset switching time between the pneumatic and AGV units.

[0043] Then, based on the scheduling environment model, the state space and action space of the reinforcement learning agent are defined. The state space represents the real-time state of the scheduling environment, specifically including the number of samples to be transferred for various coal sample tasks, the range of which is limited by the total number of samples in a single task; the remaining load capacity of each AGV unit, the range of which is limited by the maximum load capacity of the AGV; the working status of the pneumatic transmission device, including idle, busy, and switching; the number of idle slots in each processing node, the range of which is limited by the maximum number of slots in the node; the cumulative time from task start to the current moment; the total number of coal samples allocated to the AGV but not yet reaching the batch quantity; and the tracking of partial load status.

[0044] The action space is a discrete action space that covers all valid scheduling action combinations that conform to the system rules. Each action consists of three parts: sample type, transport vehicle, and target device. The sample type covers all defined coal sample task categories, the transport vehicle includes all available AGV units and pneumatic transmission devices, and the target device includes all applicable processing nodes. The action space excludes invalid actions through a constraint checking mechanism.

[0045] Next, a reward function is designed to optimize scheduling efficiency. The first reward component of the reward function has the core objective of minimizing the total completion time of coal sample transfer. The total completion time is defined as minimizing the maximum completion time among all types of coal sample tasks. The completion time for each type of task is dynamically calculated as a function of the earliest arrival time of the sample at the equipment, the equipment processing time, and the transportation time. The transportation time is determined based on the path distance, carrier speed, and sample quantity. Simultaneously, a second reward component with constraints is introduced to penalize scheduling actions that violate system rules, including imposing negative rewards when actions lead to carrier overload, equipment conflicts, or invalid paths.

[0046] Subsequently, reinforcement learning was used to train the agent, which adopted an Actor-Critic structure. The agent collected experience data by interacting with the environment, calculated the policy gradient using generalized advantage estimation, and stabilized the training process through a pruning mechanism. The training hyperparameters included the network learning rate, discount factor, and smoothing coefficient. During the training process, the agent gradually optimized the policy until the reward curve converged, indicating that a stable scheduling capability had been obtained.

[0047] Finally, the system is deployed to the coal sample transfer robot system, which uses a greedy strategy to output scheduling actions based on real-time environmental conditions. The greedy strategy specifically includes prioritizing idle equipment for task assignment, sorting and assigning tasks based on idle duration when multiple devices are idle, and sorting tasks based on arrival time when all devices are busy. The system iteratively calculates the intervention effects of AGVs and pneumatic transport, always selecting the carrier with the shorter estimated time. During deployment, the parameters of the trained and converged agent model are loaded and set to evaluation mode. The scheduling environment is initialized, the amount of coal samples to be transferred is input, the agent obtains the environmental status in real time, and selects the most probable effective action based on the greedy strategy. After each coal sample transfer is completed, the sample ID type, carrier and equipment, and timestamp are recorded, generating a detailed scheduling log. When all coal samples have completed the entire transfer process, the total completion time is output to evaluate the scheduling performance.

[0048] In another embodiment, the specific implementation of the present invention is as follows:

[0049] Step 1: First, define the core elements of coal sample transfer scheduling and construct a scheduling environment model, including coal sample task characteristics, transportation resource information, and the status of each equipment node;

[0050] Step 2: Set up the reinforcement learning environment and define the environment state and the scheduling actions that the agent can perform;

[0051] Step 3: Design a hybrid reward function with the objective of minimizing the total completion time of coal sample transfer, and set constraints to guide the learning process; then use reinforcement learning methods for offline training to guide the agent to learn the optimal strategy until the training results converge.

[0052] Step 4: Once the model reaches convergence, output the optimal scheduling strategy and deploy it to the robot system to run and execute the scheduling process. Apply the obtained optimal strategy to the task scheduling of the coal sample transfer robot to obtain the optimal scheduling scheme.

[0053] The coal sample transfer robot task scheduling system is constructed as a multi-node environment comprising a sample preparation machine (C), a sample storage cabinet (A), a full-water equipment (B), and an analysis unit (D). It also considers a hybrid transport system consisting of three AGVs (AGV0 / AGV1 / AGV2) and a pneumatic transfer device. Samples are transferred in a fixed flow: "Sample Preparation Machine C → Sample Storage Cabinet A → Analysis Unit D → Sample Storage Cabinet A." This process only considers two collaborative scheduling modes: AGV and pneumatic transfer, eliminating uncertainties caused by manual analysis.

[0054] The specific implementation logic of task scheduling is as follows: The task of transporting samples to the storage cabinet mainly includes analytical sample preparation and transportation, whole water sample delivery and transportation, and storage and transportation of the storage cabinet. Each task involves analysis, transportation, and processing time. First, the system calculates the theoretical time for each task based on the preset pneumatic system, taking into account sample distance, transportation path, and the time consumed at each stage. Then, the system considers the possibility of AGV intervention. By calculating the time taken with AGV intervention compared to pure pneumatic methods, if the time is shorter with AGV intervention, the AGV is selected as the optimization solution. When dispatching tasks, the system prioritizes idle and available equipment. If multiple devices are idle simultaneously, the system sorts them by their idle time, prioritizing tasks assigned to devices with longer idle times. If all devices are working, the system sorts them by their start time, prioritizing tasks that arrive earlier. The system iteratively calculates across multiple devices and tasks in this way to ensure the shortest time after AGV intervention, thus outputting the optimal task dispatch plan.

[0055] For step 1, the task scheduling optimization problem of the coal sample transfer robot aims to minimize the total completion time of all sample transfers and processing, therefore, optimization is required:

[0056] minZ equals min{max(TimeA, TimeB, TimeC)};

[0057] Where TimeA is max{E(1,A),E(2,A)}, TimeB is max{E(1,B),E(2,B)}, plus T B In addition, max{(80n)} agl -Bd(A,B) / S A n conv -Bd(A,B) / S C}, where l is 0, 1, or 2.

[0058] TimeC is max{E(1,C),E(2,C)}, plus T C In addition to {n agv -C(d(A,C)+d(A,D))} / S C In addition to T D In addition, {d(D,A)n all -C} / S C}

[0059] E(1, i) equals n conv-i d(0,i) / S C Let i be {A, B, C};

[0060] E(2, i) equals (800 n) agv-i ) / load, plus d(0,i) / S A Plus 80n agv-i Plus d(0,i) / S C , where i is {A, B, C}.

[0061] Where, n conv-i Add n agv-i equal to n all-i Let i be {A, B, C}; n agv-i Greater than or equal to 0, less than or equal to load max Let i be {A, B, C}, and l be 0, 1, 2.

[0062] Where i = {A, B, C} is the task type identifier, corresponding to the tasks of sample retrieval, whole water sample, and sample preparation / analysis, respectively; n conv-i n is the number of samples allocated to the pneumatic delivery unit in the i-th task category; agv-i n is the number of samples assigned to the robot unit AGV in the i-th task; all-i The total number of samples assigned to the i-th type of task; load max S represents the maximum single-batch load capacity of the AGV. A S C , i,j, and d(i,j) represent the transmission speeds of the AGV and the pneumatic conveying unit, respectively; d(i,j) is the path length between node i and node j; TB T C T D These represent the single-sample processing time for the whole-water equipment (target equipment for Category B tasks), the sample preparation equipment (target equipment for Category C tasks), and the analytical sample equipment (subsequent target equipment for Category A tasks), respectively. TimeA, TimeB, and TimeC represent the completion times for Category A, Category B, and Category C tasks, respectively.

[0063] When constructing the scheduling environment model in step 1, firstly, the functional positioning of core nodes such as the sample preparation machine, the sample storage cabinet, and the analysis sample unit is clarified. Then, the fixed transfer process of coal samples "sample preparation machine C → sample storage cabinet A → analysis sample unit D → sample storage cabinet A" is determined. Next, the characteristics of two transportation methods, AGV robots (which can transport in batches but take a long time to pick up and put down coal samples) and pneumatic transmission (which can transport only one sample at a time but takes less time), are clarified. When there is a conflict in transportation resources, such as when all AGVs are occupied, pneumatic transmission can be called to fill the gap, or the sample can enter the AGV waiting queue. Finally, the processing progress of each node is recorded in real time (such as the remaining preparation time of sample preparation machine C, the number of coal samples to be tested in analysis sample unit D), and the length of the waiting queue (such as the number of coal samples to be transferred at the exit of sample preparation machine C), so as to provide an environmental basis for subsequent scheduling decisions.

[0064] Step 2 builds upon the model established in Step 1 by creating an environment suitable for training the reinforcement learning agent. The state space must comprehensively cover the key state information of the task, carrier, and equipment to ensure the agent can obtain a complete decision context. The agent adopts an Actor-Critic structure, where the Actor network is responsible for outputting specific action strategies such as task allocation based on the environmental state, while the Critic network evaluates the value and provides performance feedback on the actions output by the Actors. This forms a closed loop with the environment through state, action, and reward, guiding the optimization and improvement of the strategy and continuously refining the scheduling strategy. The coal sample testing scheduling environment model on the right details the entire process from sampling points, transportation routes, transfer stations, laboratories to the robot scheduling system. The environment model also includes key aspects such as task generation, robot state acquisition, and robot load monitoring, providing the agent with real-time, complete, and dynamic input data, enabling it to automatically generate task scheduling schemes based on actual working conditions. The entire system structure shows that the agent and the coal sample testing scheduling environment form an interactive closed loop through three channels: state, reward, and action. After receiving environmental feedback, the agent continuously updates its strategy parameters until it outputs a stable and optimal scheduling decision.

[0065] The system adapts the number of A / B / C class samples to be transferred to the changes in task volume n. The remaining capacity of the AGV and the remaining slots of the equipment reflect the load status of the carrier and equipment. The current total system time provides a time schedule reference and supports dynamic decision-making. The defined state variables include: the number of A / B / C class coal samples to be transferred, the remaining load capacity of AGV0 / AGV1 / AGV2, the pneumatic transfer status, the remaining number of slots of equipment B (containing two sub-equipments, B0 and B1, each with 48 parallel slots), the remaining number of slots of equipment C (containing two sub-equipments, C0 and C1, each with 3 parallel slots), the remaining number of slots of equipment D (containing 33 parallel slots), the current total system time, and the total number of AGV samples to be batched, as shown in Table 1.

[0066] Table 1. State variable information in the reinforcement learning training environment.

[0067] .

[0068] The action space consists of discrete actions, covering all "sample-carrier-device" combinations that conform to the system rules. Constraint checks are used to define fixed action structures and exclude invalid actions (such as actions that cannot be selected when the AGV is fully loaded), ensuring that action execution strictly conforms to the actual operational constraints of the system. The action space involves sample types, transport carriers, and target devices, as detailed in Table 2.

[0069] Table 2. Action variable information under reinforcement learning training environment

[0070] .

[0071] Step 3, in designing the reward function for reinforcement learning, since this method cannot directly solve the constrained objective function from Step 1, minimizes the total completion time as the core objective. Simultaneously, it introduces timeout penalties and positive rewards for task completion to construct a comprehensive reward function. Furthermore, conditions such as the AGV's maximum load, path feasibility, safe distance, and time window are used as constraints to filter out illegal actions or add penalties during training. Therefore, the reward function is set as follows: R equals R0. base Add R res Add R batch Add R penalty Among them, R base The basic reward directly reflects the time required to transport the current sample, incentivizing the agent to prioritize transportation options with shorter processing times, thereby fundamentally reducing the total completion time; R res This refers to resource utilization rewards to promote the efficient use of transportation resources (AGVs, pneumatic conveyors) and reduce time wastage caused by resource idleness; R batch To provide batch-based collaborative rewards and avoid extending the overall time caused by the dispersed transportation of coal samples within a batch; R penaltyThis represents a penalty term. If the AGV is overloaded or the queue length for the sample preparation machine or storage cabinet exceeds its maximum capacity, a penalty is imposed. Simultaneously, it ensures that batch transport follows the sequence of "sample preparation machine → storage cabinet → analysis unit → storage cabinet". Subsequently, the PPO agent parameters are initialized, offline training is conducted, and the total reward metric is observed until convergence, at which point the optimal policy is derived. Here, s, a, r, and s' represent the state, action, reward, and next-time state, respectively, while ω and θ are the parameters of the Critic network and Actor network, respectively.

[0072] The deployment and scheduling described in step 4 includes: first, loading the parameters of the PPO network that has been trained and converged, setting the agent to evaluation mode, and using a greedy strategy for action selection (i.e., selecting the action with the highest probability from the available actions each time to ensure the stability and optimality of the scheduling decision); then, performing real-time scheduling, first completing the environment initialization and inputting the amount of coal samples to be transferred, n, and then the agent obtains the current status of transportation resources (AGV, pneumatic transmission) and the status of each node (sample preparation machine C, sample storage cabinet A, analysis sample unit D) in real time, outputting the optimal scheduling action based on the greedy strategy and executing it, while simultaneously updating the environment status, and after each coal sample is transferred, recording the sample ID, sample type (analytical sample / storage sample), transport carrier (specific AGV number / pneumatic transmission), target equipment (such as storage cabinet A, analysis sample unit D), transfer start time and end time, forming a complete scheduling log; finally, after the environment terminates (all coal samples complete the entire process of "sample preparation machine → storage cabinet → analysis sample unit → storage cabinet"), the total completion time of coal sample transfer is output.

[0073] In one embodiment, taking a fully automated intelligent coal quality analysis and testing system of a testing company as an example, the implementation process of a task scheduling method for a coal sample transfer robot based on reinforcement learning is as follows:

[0074] In this coal sample transfer robot task scheduling system, device B has a processing time of 18,000 s / sample, device C has a processing time of 1,500 s / sample, and device D has a processing time of 14,400 s / sample. The maximum load of the robot AGV unit is 10 samples per batch, the time for picking up and placing a single sample is 40 s, the transport speed is 0.5 m / s, the pneumatic transmission speed is 5 m / s, and the switching time between pneumatic and AGV is 20 s.

[0075] For this scheduling task, the PPO algorithm was used as the scheduling agent. The agent's input is the environment state vector, and its output is a selectable scheduling action. The core hyperparameters of the agent are set as follows: the learning rate for both the actor and critic networks is 0.0003, the discount factor is 0.99, the smoothing coefficient for generalized advantage estimation is 0.95, the pruning threshold is 0.2, each update consists of 10 training epochs, and the batch size is 64. During training, the agent interacts with the environment, accumulates rewards, and updates the algorithm through the PPO policy to improve long-term scheduling performance. The training function is set to a maximum of 1000 epochs, with a maximum of 500 steps per epoch, to ensure that the agent can fully explore and learn within the limited simulation time.

[0076] During the training process of scheduling tasks, the performance of the reinforcement learning agent in making scheduling decisions is as follows: Figure 5 As shown, Figure 5 This is a training curve diagram illustrating the scheduling of a coal sample transfer robot based on reinforcement learning, as described in one embodiment of the present invention. The training curve shows that the agent scores low in the initial stage, but the average score gradually increases as training progresses, and stabilizes after about 15,000 rounds, eventually converging at a relatively optimal level. This indicates that the proposed method can effectively learn a reasonable scheduling strategy.

[0077] Figure 6 This is a Gantt chart illustrating the scheduling of coal sample transfer robots using the optimal strategy in one embodiment of the reinforcement learning-based coal sample transfer robot scheduling method of the present invention. It shows the time allocation and workflow of different transport vehicles in the coal sample transfer task. The blue, orange, pink, and green bars correspond to the running time intervals of AGV0, AGV1, AGV2, and pneumatic transport, respectively. The A / B / C characters on each bar chart correspond to the storage sample, full water sample, and sample preparation / analysis sample, respectively, indicating the destination of each sample after using different transport methods. For example, sample 0 is transported to the full water sample using AGV1 within a running time interval of approximately 400-500, and then to the storage sample. Sample 7 is transported using AGV2, first arriving at the sample preparation machine within a running time interval of 400-500, and then to the storage sample. Since the maximum load of the AGV is 10, 10 samples can be processed at once. It can be observed that there is overlap in the time intervals between different vehicles. For example, AGV0 is in operation during periods of approximately 300-500 and 700-800, AGV1 undertakes tasks in the ranges of 400-1400 and 1800-2500, AGV2 undertakes tasks in the range of 300-6300, while the transportation operations of the pneumatic transmission unit are more dispersed.

[0078] The overall results verify the feasibility and effectiveness of the designed method in complex scheduling scenarios. It has a high resource utilization rate, significantly reduced transportation conflicts, and reflects the coal sample-transport carrier-time allocation logic after adopting the method of this invention, ensuring that the coal sample is transferred to the target equipment in an orderly and efficient manner.

Claims

1. A coal sample transfer robot scheduling method based on reinforcement learning, characterized in that, include: Construct a scheduling environment model for coal sample transfer, including several types of coal sample tasks, heterogeneous transport carriers, and status information of processing nodes; Based on the scheduling environment model, the state space and action space of the reinforcement learning agent are defined; the state space represents the real-time state of the scheduling environment, and the action space covers the effective combination of scheduling actions consisting of sample type, transport vehicle and target device. The design employs a reward function that aims to minimize the maximum completion time for all types of coal sample tasks. For each type of coal sample task, the completion time is calculated by combining the sample quantity allocated to the AGV and the sample quantity allocated to the pneumatic conveyor, the batch processing capacity and transport speed of the AGV, the transport speed and switching time of the pneumatic conveyor, the single sample processing time of each processing node, and the transport path distance. The agent is trained using reinforcement learning until the policy converges, and the optimal scheduling policy is obtained and deployed to the coal sample transfer robot system. The system outputs scheduling actions based on the real-time environmental state through a greedy policy to complete the coal sample transfer task.

2. The coal sample transfer robot scheduling method based on reinforcement learning according to claim 1, characterized in that, The construction of the scheduling environment model for coal sample transfer specifically includes: defining the core elements of coal sample transfer, which include coal sample task characteristics, transportation resource information, and the status of each equipment node.

3. The method for scheduling a coal sample transfer robot based on reinforcement learning according to claim 2, characterized in that, The coal sample task features include analytical sample preparation and transportation tasks, full water sample delivery and transportation tasks, and retrieval sample storage and transportation tasks; the transportation resource information includes several automated guided vehicles (AGV) units and at least one pneumatic transmission device; the status of each equipment node includes the working status of the retrieval sample cabinet, the full water equipment and the sample preparation machine, and the analytical sample unit, and each node has a preset processing time.

4. The coal sample transfer robot scheduling method based on reinforcement learning according to claim 3, characterized in that, The state space specifically includes: The number of samples to be transferred for various coal sample tasks is limited by the total number of samples in a single task. The remaining load capacity of each AGV unit is limited by the maximum load capacity of the AGV. The operating status of the pneumatic transmission device, including idle, busy, and switching states; The number of free slots in each processing node is limited by the maximum number of slots in the node. The cumulative time elapsed from task initiation to the current moment; The total number of coal samples allocated to AGVs but not yet reaching the batch quantity is tracked for partial load status.

5. The coal sample transfer robot scheduling method based on reinforcement learning according to claim 4, characterized in that, The action space is a discrete action space, covering all valid scheduling action combinations that conform to the system rules. Each action consists of three parts: sample type, transport vehicle, and target device. The sample type covers all defined coal sample task categories, the transport vehicle includes all available AGV units and pneumatic transmission devices, and the target device includes all applicable processing nodes. Invalid actions are excluded through constraint checking mechanisms.

6. A coal sample transfer robot scheduling method based on reinforcement learning according to claim 4 or 5, characterized in that, The reward function specifically includes: a first reward component with minimizing the total completion time of coal sample transfer as its core objective, wherein the total completion time is defined as minimizing the maximum value among the completion times of all types of coal sample tasks, and the completion time of each type of task is dynamically calculated as a function of the earliest time the sample arrives at the equipment, the equipment processing time, and the transportation time, with the transportation time determined based on the path distance, carrier speed, and number of samples; and a second reward component that introduces constraints to penalize scheduling actions that violate system rules, including applying negative rewards when actions cause overloading of the transport carrier, equipment conflicts, or invalid paths.

7. The method for scheduling a coal sample transfer robot based on reinforcement learning according to claim 6, characterized in that, The reinforcement learning specifically includes: the agent adopts an Actor-Critic structure, collects experience data by interacting with the environment, estimates and calculates the policy gradient using generalized advantage, and stabilizes the training process through a pruning mechanism. The training hyperparameters include the network learning rate, discount factor, and smoothing coefficient. During the training process, the agent gradually optimizes the policy until the reward curve converges, indicating that a stable scheduling capability has been obtained.

8. The method for scheduling a coal sample transfer robot based on reinforcement learning according to claim 7, characterized in that, The deployment to the coal sample transfer robot system specifically includes: loading the parameters of the trained and converged agent model and setting it to evaluation mode; initializing the scheduling environment; inputting the amount of coal samples to be transferred; the agent acquiring the environmental status in real time; selecting the most probable effective action to execute based on a greedy strategy; recording the sample ID type, carrier, device, and timestamp for each coal sample transfer; and generating a detailed scheduling log; and outputting the total completion time after all coal samples have completed the entire transfer process to evaluate the scheduling performance.

9. A coal sample transfer robot scheduling method based on reinforcement learning according to claim 1 or 8, characterized in that, The greedy strategy specifically includes: prioritizing idle devices for task assignment; sorting and assigning tasks based on idle duration when multiple devices are idle; sorting and processing tasks based on arrival time when all devices are busy; iteratively calculating the intervention effect of AGV and pneumatic transmission; and always selecting the carrier with the shorter expected time.

10. A coal sample transfer robot scheduling method based on reinforcement learning according to claim 3, characterized in that, The AGV unit has a preset maximum load capacity, sample handling time, and transport speed. The pneumatic transmission device has a preset transmission speed, and there is a preset switching time between the pneumatic and AGV units.

Images