Underground mining truck dispatching method and device based on state machine and reinforcement learning

Abstract

The invention provides an underground mining truck dispatching method and device based on a state machine and reinforcement learning, and relates to the technical field of underground mine transportation system dispatching optimization. The method comprises the following steps: acquiring a scene model of an underground mine transportation system, constructing a topological structure model, constructing an underground mine truck fleet operation problem according to the topological structure model and a conflict type, and further constructing a fleet scheduling decision model based on a state machine and reinforcement learning; wherein each truck is used as an intelligent agent of the motorcade scheduling decision-making model, and a state machine is configured for the intelligent agent; and solving the motorcade scheduling decision-making model by taking maximization of transportation efficiency, reduction of waiting time and reduction of conflicts as targets to obtain a scheduling result of the mining truck in the underground transportation working face with the bidirectional single-lane driving characteristic. By combining the state machine and the reinforcement learning technology, intelligent dispatching of the underground mining truck fleet is achieved, so that the transportation efficiency is improved, traffic conflicts are reduced, and energy consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of underground mine transportation system scheduling optimization, and in particular to an underground mine truck scheduling method and device based on a state machine and reinforcement learning. Background Art

[0002] As intelligent mining equipment has been deeply researched and developed, advanced technologies such as data collection, positioning navigation, and intelligent and autonomous control of underground mining trucks have been widely used in mining operations at home and abroad. The introduction of these technologies has greatly improved the level of intelligence in mine operations. However, underground mining operations involve the coordinated operation of multiple equipment, and efficient and safe production requires coordinated operation between equipment. The different stages of underground mining production are interconnected through transportation and lifting processes, forming a complete mining production cycle. In this cycle, the underground mining truck fleet, as an important part of the transportation link, is responsible for shuttling between the upper and lower mine tunnels and performing ore transfer tasks. This multi-vehicle collaborative transportation system puts higher requirements on scheduling management.

[0003] Existing research on underground mining truck fleet management focuses on two scheduling strategies: vehicle-by-vehicle scheduling and global scheduling. The vehicle-by-vehicle scheduling strategy assigns a new task to a truck after it has completed its current transportation task and is ready to set off again, which is similar to the automated guided vehicles in the manufacturing industry. The goal of this strategy is to find the shortest conflict-free route without modifying the existing routes reserved by other trucks. Although this method can solve the conflict problem to a certain extent, it is difficult to adapt to the changes in loading / unloading nodes and the fluctuations in transportation tasks because it requires the construction of a discrete graph.

[0004] Existing methods for underground mining truck fleet scheduling have failed to fully exploit the potential of automated intelligent devices, especially large-load autonomous mining trucks that can accurately respond to operational scheduling instructions and adjust driving speed. In recent years, reinforcement learning, a machine learning method for solving decision-making problems in the interaction between systems and the environment, has demonstrated its advantages in scheduling problems. However, existing RL-based methods mainly focus on scheduling problems in open-pit mine operations and mesh structures, and have not yet been deeply explored in the bidirectional single-channel underground mine transportation scenario. Summary of the invention

[0005] In order to solve the technical problem that the existing RL-based methods mainly focus on scheduling problems in open-pit mine operations and mesh structures, and have not yet explored their application in two-way single-channel underground mine transportation scenarios, the embodiment of the present invention provides a method and device for scheduling underground mine trucks based on state machines and reinforcement learning. The technical solution is as follows:

[0006] On the one hand, a method for dispatching underground mining trucks based on a state machine and reinforcement learning is provided, the method being implemented by an underground mining truck dispatching device, the method comprising:

[0007] S1. Obtain a scenario model of an underground mine transportation system, construct a topological structure model of the underground mine transportation system according to the scenario model, and construct an underground mine truck fleet operation problem according to the topological structure model and the conflict type.

[0008] S2. According to the operation problems of underground mining truck fleets, a fleet scheduling decision model based on state machine and reinforcement learning is constructed; wherein each truck in the underground mining truck fleet is used as an intelligent agent of the fleet scheduling decision model, and the intelligent agent is equipped with a state machine.

[0009] S3. With the goal of maximizing transportation efficiency, reducing waiting time and reducing conflicts, the fleet scheduling decision model is solved to obtain the scheduling results of mining trucks in the underground transportation working face with two-way single-lane driving characteristics.

[0010] Optionally, the fleet scheduling decision model based on state machine and reinforcement learning is constructed in S2, including:

[0011] Take the stages of truck transportation as states and design state transition triggers.

[0012] The action space is adjusted according to the truck's driving range and the state of the state machine.

[0013] Design the observation space, which includes the horizontal and vertical coordinates of each truck and the state number of the state machine where each truck is located.

[0014] Set the reward function and termination condition.

[0015] Optionally, the state machine reinforcement learning system model in S2 is as shown in the following formula (1):

[0016] (1)

[0017] In the formula, represents the state machine reinforcement learning system equation, represents the system state transition trigger matrix, represents the system finite state matrix, Indicates the vehicle position status. represents the observation space of the agent in reinforcement learning, represents the initial state of the vehicle, Represents the initial state of the state machine, Indicated in The corresponding action space vector in the state, represents the system transfer equation vector, Indicates the position of the vehicle, Indicates the empty condition of the vehicle. Indicates the loading condition of the vehicle. Indicates the current vehicle position. Represents the state of the vehicle in the state machine, represents the horizontal coordinate of the vehicle at the initial moment, represents the vehicle ordinate at the initial moment, represents the loading status of the vehicle at the initial moment, Reinforcement learning The observation space of the moment, Indicates time, Indicates the time at which the truck driving status, express The state transition trigger condition at the moment, Indicates that the state machine is The state at any moment, Reinforcement learning The action space at the moment, Represents reinforcement learning action sampling.

[0018] Optionally, the state transition trigger includes a trigger of completing a specific task, a trigger of arriving at a specific location, or a trigger of satisfying a time or capacity constraint.

[0019] The trigger formula of the state transition trigger is shown in the following equations (2)-(4):

[0020] (2)

[0021] (3)

[0022] (4)

[0023] In the formula, Indicates that dimensional scene interval diagonal matrix, represents the i-th interval, Indicates the number of intervals into which the scene is divided.

[0024] Optionally, the reward function includes: task completion reward, vehicle conflict penalty, time step penalty and timeout penalty.

[0025] Termination conditions include: reaching a specified number of tasks at each loading node, two or more trucks being stuck for more than a certain time step, and exceeding the maximum time step limit.

[0026] Optionally, obtaining a scenario model of an underground mine transportation system in S1, constructing a topology model of the underground mine transportation system according to the scenario model, and constructing an underground mine truck fleet operation problem according to the topology model and the conflict type, including:

[0027] The scenario model of the underground mine transportation system is obtained, the scenario model is simplified to obtain the key elements, and the topological structure model of the underground mine transportation system is constructed according to the key elements; among them, the key elements include loading / unloading nodes, branch roads and main sections, passing chambers, and two-way single-lane driving environment.

[0028] The operation problem of underground mining truck fleet is constructed based on the topological structure model, conflict between missed chambers and conflict between intersection nodes.

[0029] On the other hand, an underground mining truck dispatching device based on a state machine and reinforcement learning is provided, and the device is applied to an underground mining truck dispatching method based on a state machine and reinforcement learning, and the device includes:

[0030] The acquisition module is used to acquire the scenario model of the underground mine transportation system, construct the topological structure model of the underground mine transportation system according to the scenario model, and construct the underground mine truck fleet operation problem according to the topological structure model and the conflict type.

[0031] A building module is used to build a fleet scheduling decision model based on state machine and reinforcement learning according to the operation problem of the underground mining truck fleet; wherein each truck in the underground mining truck fleet is used as an intelligent agent of the fleet scheduling decision model, and the intelligent agent is equipped with a state machine.

[0032] The output module is used to solve the fleet scheduling decision model with the goal of maximizing transportation efficiency, reducing waiting time and reducing conflicts, and obtain the scheduling results of mining trucks in the underground transportation working face with two-way single-lane driving characteristics.

[0033] On the other hand, an underground mining truck dispatching device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the underground mining truck dispatching methods based on state machine and reinforcement learning is implemented.

[0034] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned underground mining truck scheduling methods based on state machine and reinforcement learning.

[0035] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0036] In the present invention, by constructing a topological structure model and analyzing the conflict type, a state machine reinforcement learning method is proposed, and a state machine is equipped for the agent, whose state represents the truck transportation stage. The state machine is deeply coupled with reinforcement learning, and the system state transfer and action transfer are described by relevant formulas. The action and observation space of RL are adjusted, and the reward function and termination conditions are set. The effectiveness of the algorithm is verified through case studies, such as the change in action space dimension, the comparison of different RL algorithms, and the performance analysis of different numbers of trucks and tasks. It has significant advantages over genetic algorithms, and introduces time and energy consumption weights to optimize vehicle scheduling and achieve optimal energy efficiency. The system can improve transportation efficiency, reduce conflicts and reduce energy consumption, and is suitable for scheduling and management of underground metal mine transportation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 It is a flow chart of an underground mining truck dispatching method based on a state machine and reinforcement learning provided by an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a transportation system provided by an embodiment of the present invention;

[0040] Figure 3 is a topological diagram of a working surface of a transportation system provided by an embodiment of the present invention;

[0041] Figure 4 is a state machine state transition flow chart provided by an embodiment of the present invention;

[0042] Figure 5 It is a flow chart of a state machine combined with a Markov decision process provided by an embodiment of the present invention;

[0043] Figure 6 It is a block diagram of an underground mining truck dispatching device based on a state machine and reinforcement learning provided by an embodiment of the present invention;

[0044] Figure 7 It is a structural schematic diagram of an underground mining truck dispatching device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0046] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0047] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0048] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0049] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0050] The embodiment of the present invention provides an underground mining truck dispatching method based on state machine and reinforcement learning, which can be implemented by an underground mining truck dispatching device, which can be a terminal or a server. Figure 1 The flowchart of the underground mining truck scheduling method based on state machine and reinforcement learning is shown in the figure. The processing flow of the method may include the following steps:

[0051] S1. Obtain a scenario model of an underground mine transportation system, construct a topological structure model of the underground mine transportation system according to the scenario model, and construct an underground mine truck fleet operation problem according to the topological structure model and the conflict type.

[0052] In a feasible implementation, a scenario model of an underground mine transportation system is constructed, such as Figure 2 As shown in the figure, it is reasonably simplified and the key elements are extracted, including loading / unloading nodes, branch roads and main roads, passing chambers, and two-way single-lane driving environments, so as to build a topological structure model. In-depth analysis of typical conflict types, including passing chamber conflicts and intersection node conflicts, is conducted. Based on these conflict analyses, the core issues that need to be solved during the operation of the fleet are clarified, such as avoiding vehicle conflicts, reducing delays, and optimizing passing operations to achieve efficient and smooth transportation.

[0053] Specifically, the present invention focuses on the underground metal mine transportation system, which is mainly composed of ore bodies, mining faces, chute shafts, loading-transport-unloading machines, underground mining trucks, main chute shafts, crushers, underground ore bins, lifting equipment, etc. For ease of analysis, the complex transportation system is simplified into a topological structure model, such as Figure 3 , including loading and unloading nodes, branch roads and main sections, passing chambers and two-way single lane driving environment. Main sections such as 1-2-3-4-5-6 are relatively long, ranging from several hundred meters to thousands of meters; branch sections 4-7, 5-8 and 6-9 are relatively short, including loading nodes. Operation nodes include loading nodes (nodes 7, 8, and 9) and unloading nodes (node ​​1), and non-operation nodes include passing chamber nodes (2 and 3) and lane intersection nodes (4, 5, and 6).

[0054] S2. According to the operation problems of underground mining truck fleets, a fleet scheduling decision model based on state machine and reinforcement learning is constructed; wherein each truck in the underground mining truck fleet is used as an intelligent agent of the fleet scheduling decision model, and the intelligent agent is equipped with a state machine.

[0055] The present invention combines the state machine with RL (Reinforcement Learning), and can optimize the scheduling process of vehicles in underground transportation working faces with two-way single-lane driving characteristics.

[0056] The fleet scheduling decision model can quickly generate the optimal driving plan for each truck based on different transportation task volumes and real-time environmental changes, avoiding deadlock problems and ineffective parking of vehicles at non-operating nodes.

[0057] Optionally, the fleet scheduling decision model based on state machine and reinforcement learning is constructed in S2, including:

[0058] S21. Take the stage of truck transportation as the state and design the state transfer trigger.

[0059] In a feasible implementation, each state of the state machine represents a specific stage or task of the truck in the transportation process, including loading, transporting, unloading, waiting, etc.

[0060] Furthermore, transitions between states are determined by specific triggers, including completing a specific task, reaching a specific location, satisfying time or capacity constraints, etc.

[0061] S22: The action space is adjusted according to the truck driving range and the state of the state machine.

[0062] S23. Design an observation space, which includes the horizontal and vertical coordinates of each truck and the state number of the state machine where each truck is located.

[0063] In a feasible implementation, the optimized observation space and action space can effectively improve the training effect and decision-making quality of the intelligent agent, where the observation space includes each state in the state machine model, and the action space is a set of exclusive actions designed for each state.

[0064] S24. Set the reward function and termination conditions.

[0065] This paper proposes a state machine reinforcement learning method and carefully constructs a scenario framework. In this framework, each agent is equipped with a state machine, in which the state accurately represents the specific stage or task of the truck in the transportation process, such as loading, transporting, dumping, waiting, etc. The transition between states is strictly controlled by specific triggers, which include task completion, arrival at a specific location, or meeting time and capacity constraints. In this framework, the state machine is deeply coupled with RL, and a series of formulas such as the system state transition trigger matrix and the finite state matrix are used to accurately describe the system state transition and action transfer process.

[0066] Specifically, the RL action space is adjusted according to the truck's driving range and the state of the state machine, and it is clear that the action space covers multiple operations such as moving forward at different speeds, entering and exiting the wrong car chamber, and stopping and waiting. The meaning of these operations will change dynamically according to the state of the truck in the state machine. Figure 4 , clarify the state machine interval division, state and action, and state transition triggers. The observation space is designed to contain the horizontal and vertical coordinates of each truck and the state number in the state machine where it is located. As the number of trucks participating in the task changes, the dimension of the observation space is adjusted accordingly to more accurately describe the scheduling task, so that the RL algorithm can make wise decisions based on the comprehensive status of all trucks in the system.

[0067] like Figure 5 , set a reward function, which includes multiple parts such as task completion reward, vehicle conflict penalty, time step penalty, timeout penalty, etc., and clarify the termination conditions, such as when each loading node reaches the specified number of tasks, when two or more trucks are stagnant for more than a certain time step, and when the maximum time step limit is exceeded, the corresponding reward or termination operation is triggered respectively. In the present invention, a reward value calculation formula is designed, which comprehensively considers factors such as task completion, number of conflicts, task time and whether it is timeout. The agent adjusts its strategy according to the reward value to maximize the cumulative reward.

[0068] Alternatively, the descriptive formula for the state machine reinforcement learning system can be summarized as follows:

[0069] (1)

[0070] In the formula, represents the state machine reinforcement learning system equation, represents the system state transition trigger matrix, represents the system finite state matrix, Indicates the vehicle position status. represents the observation space of the agent in reinforcement learning, represents the initial state of the vehicle, Represents the initial state of the state machine, Indicated in The corresponding action space vector in the state, represents the system transfer equation vector, Indicates the position of the vehicle, Indicates the empty condition of the vehicle. Indicates the loading condition of the vehicle. Indicates the current vehicle position. Represents the state of the vehicle in the state machine, represents the horizontal coordinate of the vehicle at the initial moment, represents the vehicle ordinate at the initial moment, represents the loading status of the vehicle at the initial moment, Reinforcement learning The observation space of the moment, Indicates time, Indicates the time at which the truck driving status, express The state transition trigger condition at the moment, Indicates that the state machine is The state at any moment, Reinforcement learning The action space at the moment, represents reinforcement learning action sampling, Represents the system state transition equation.

[0071] The trigger formula of the state transition trigger is shown in the following equations (2)-(4):

[0072] (2)

[0073] (3)

[0074] (4)

[0075] In the formula, Indicates that dimensional scene interval diagonal matrix, Is has dimensional state transition trigger column vector represents the i-th interval, Indicates the number of intervals for scene division, through the formula Determine whether the truck is within a certain range. If so, the value is 1, otherwise the value is 0. Divide the scene into n sub-ranges according to the scene node position and lane section, and determine whether the truck is in a certain range, and assign it a corresponding state transfer trigger for the truck's state machine state transition.

[0076] The observation space is designed to contain the horizontal and vertical coordinates of each truck and the state number in the state machine where it is located. As the number of trucks participating in the task changes, the dimensions of the observation space are adjusted accordingly to more accurately describe the scheduling task, enabling the RL algorithm to make intelligent decisions based on the comprehensive status of all trucks in the system.

[0077] S3. With the goal of maximizing transportation efficiency, reducing waiting time and reducing conflicts, the fleet scheduling decision model is solved to obtain the scheduling results of mining trucks in the underground transportation working face with two-way single-lane driving characteristics.

[0078] In one possible implementation, a reinforcement learning algorithm is trained through a Markov decision process to maximize transportation efficiency, reduce waiting time and conflicts, and dynamically adjust the fleet size and operation strategy to cope with the complex underground transportation network to maximize transportation efficiency.

[0079] The present invention verifies the effectiveness of the algorithm through a series of comprehensive and in-depth case studies. When comparing different RL algorithms, the success rate, conflict rate and time control are evaluated from multiple dimensions. When analyzing the performance under different numbers of trucks, it is found that when the number of trucks is less than 7, the success rate steadily increases with the increase of training iterations, and the conflict rate gradually decreases and converges to 0, but as the number of trucks increases, the convergence speed slows down. When it reaches 7, the training performance deteriorates and the task cannot be successfully completed. It is determined that the maximum capacity of the system under a specific topology is 6 trucks. When testing the performance under different numbers of tasks, taking 3 trucks as an example, when the number of loading tasks increases from 2 to 10 and 20, the time to complete all tasks increases linearly, but the operational stability is not affected, indicating that the effectiveness of the method is not affected by the change in the number of tasks. Comparing the present invention with the genetic algorithm, the results show that the present invention has significant advantages in reducing waiting time, starting and stopping times, and total time consumption, such as reducing total time consumption by 22.24%, reducing waiting time by 44.74%, significantly reducing the number of stops, and no need to retrain the model when the number of tasks changes.

[0080] For example, the state machine reinforcement learning method, PPO and A2C algorithms are used to train the model. The scenario is that three mining carts perform two tasks from the unloading node to different loading nodes. The results show that the algorithm success rate of the state machine reinforcement learning method is high and stable, the conflict rate is low, and the time consumption is stable; the PPO algorithm is low at the beginning but improves later, and the conflict rate gradually decreases and fluctuates; the A2C algorithm is poor at the beginning and fluctuates greatly. This shows that the state machine reinforcement learning method performs well in success rate, conflict rate and time control. Gradually increase the number from 3 mining carts, and Figure 3 Training under topological structure. When the number of vehicles is less than 7, the success rate increases with iteration, the conflict rate decreases and converges, and the average time step converges; when the number of vehicles reaches 7, the training performance deteriorates. It shows that the maximum capacity of the three-branch lane transportation network is 6 vehicles. Currently, 3 vehicles can meet the operation. If production needs to be increased or time is reduced, the optimal number of vehicles can be further analyzed.

[0081] Furthermore, time weight and energy consumption weight are introduced to comprehensively consider various resistances encountered by the vehicle during driving, including rolling resistance (related to the gravity and rolling resistance coefficient of the vehicle), air resistance (related to the air resistance coefficient, frontal area and vehicle speed), acceleration resistance (related to the rotational mass conversion formula of a vehicle with a fixed transmission ratio and the vehicle mass), and establish a vehicle power balance relationship to calculate the mechanical energy consumption of the vehicle traveling at a constant speed in any interval. According to the energy efficiency performance of vehicle operation, the vehicle allocation principle is determined, such as reasonably arranging more vehicles to loading nodes far away from the unloading node. As the number of vehicles and the number of tasks at each loading node increase, by comprehensively considering the time steps taken to complete the task and the relationship between the total energy consumption of all vehicles when completing the task and the number of vehicles, it is found that the optimal energy efficiency performance is achieved when the number of vehicles is 6.

[0082] In the embodiment of the present invention, by constructing a topological structure model and analyzing the conflict type, a state machine reinforcement learning method is proposed, and a state machine is equipped for the agent, whose state represents the truck transportation stage. The state machine is deeply coupled with reinforcement learning, and the system state transition and action transition are described by relevant formulas. The action and observation space of RL are adjusted, and the reward function and termination conditions are set. The effectiveness of the algorithm has been verified through case studies, such as the change in the dimension of the action space, the comparison of different RL algorithms, and the performance analysis of different numbers of trucks and tasks. It has significant advantages over genetic algorithms, and introduces time and energy consumption weights to optimize vehicle scheduling and achieve optimal energy efficiency. The system can improve transportation efficiency, reduce conflicts and reduce energy consumption, and is suitable for scheduling and management of underground metal mine transportation systems.

[0083] Figure 6 The block diagram of an underground mining truck dispatching device based on a state machine and reinforcement learning is shown according to an exemplary embodiment. The device is used in an underground mining truck dispatching method based on a state machine and reinforcement learning. Figure 6The device includes an acquisition module 310, a construction module 320 and an output module 330. Among them:

[0084] The acquisition module 310 is used to acquire the scenario model of the underground mine transportation system, construct the topology model of the underground mine transportation system according to the scenario model, and construct the underground mine truck fleet operation problem according to the topology model and the conflict type.

[0085] The construction module 320 is used to construct a fleet scheduling decision model based on a state machine and reinforcement learning according to the operation problem of the underground mining truck fleet; wherein each truck in the underground mining truck fleet is used as an intelligent agent of the fleet scheduling decision model, and the intelligent agent is equipped with a state machine.

[0086] The output module 330 is used to solve the fleet scheduling decision model with the goal of maximizing transportation efficiency, reducing waiting time and reducing conflicts, and obtain the scheduling results of mining trucks in the underground transportation working face with two-way single-lane driving characteristics.

[0087] In the embodiment of the present invention, by constructing a topological structure model and analyzing the conflict type, a state machine reinforcement learning method is proposed, and a state machine is equipped for the agent, whose state represents the truck transportation stage. The state machine is deeply coupled with reinforcement learning, and the system state transition and action transition are described by relevant formulas. The action and observation space of RL are adjusted, and the reward function and termination conditions are set. The effectiveness of the algorithm has been verified through case studies, such as the change in the dimension of the action space, the comparison of different RL algorithms, and the performance analysis of different numbers of trucks and tasks. It has significant advantages over genetic algorithms, and introduces time and energy consumption weights to optimize vehicle scheduling and achieve optimal energy efficiency. The system can improve transportation efficiency, reduce conflicts and reduce energy consumption, and is suitable for scheduling and management of underground metal mine transportation systems.

[0088] Figure 7 is a structural schematic diagram of an underground mining truck dispatching device provided by an embodiment of the present invention, such as Figure 7 As shown, the underground mining truck dispatching equipment may include the above Figure 6 The underground mining truck dispatching device based on state machine and reinforcement learning is shown. Optionally, the underground mining truck dispatching device 410 may include a first processor 2001 .

[0089] Optionally, the underground mining truck dispatching device 410 may further include a memory 2002 and a transceiver 2003 .

[0090] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0091] Combine the following Figure 7The components of the underground mining truck dispatching device 410 are introduced in detail:

[0092] The first processor 2001 is the control center of the underground mining truck dispatching device 410, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0093] Optionally, the first processor 2001 can perform various functions of the underground mining truck dispatching device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0094] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in FIG.

[0095] In a specific implementation, as an embodiment, the underground mining truck dispatching device 410 may also include multiple processors, such as Figure 7 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0096] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0097] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently and access the first processor 2001 through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0098] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0099] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 7 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0100] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0101] It should be noted that Figure 7 The structure of the underground mining truck dispatching device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0102] In addition, the technical effects of the underground mining truck dispatching device 410 can refer to the technical effects of the underground mining truck dispatching method based on state machine and reinforcement learning described in the above method embodiment, which will not be repeated here.

[0103] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0104] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0105] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0106] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0107] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0108] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0109] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0111] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0114] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0115] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for dispatching underground mining trucks based on state machine and reinforcement learning, characterized in that: The method comprises: S1. Obtain a scenario model of an underground mine transportation system, construct a topological structure model of the underground mine transportation system according to the scenario model, and construct an underground mine truck fleet operation problem according to the topological structure model and the conflict type; S2. According to the operation problem of the underground mining truck fleet, a fleet scheduling decision model based on a state machine and reinforcement learning is constructed; wherein each truck in the underground mining truck fleet is used as an intelligent agent of the fleet scheduling decision model, and the intelligent agent is equipped with a state machine; S3. With the goal of maximizing transportation efficiency, reducing waiting time and reducing conflicts, the fleet scheduling decision model is solved to obtain the scheduling results of mining trucks in the underground transportation working face with two-way single-lane driving characteristics.

2. The underground mining truck dispatching method based on state machine and reinforcement learning according to claim 1 is characterized in that: The fleet scheduling decision model constructed in S2 is based on state machine and reinforcement learning, including: Take the truck transportation stage as the state and design the state transition trigger; The action space is adjusted according to the truck driving range and the state of the state machine; Designing an observation space, wherein the observation space includes the horizontal and vertical coordinates of each truck and the state number of the state machine where each truck is located; Set the reward function and termination condition.

3. The underground mining truck dispatching method based on state machine and reinforcement learning according to claim 2 is characterized in that: The state machine reinforcement learning system model in S2 is shown in the following formula (1): （1） In the formula, represents the state machine reinforcement learning system equation, represents the system state transition trigger matrix, represents the system finite state matrix, Indicates the vehicle position status. represents the observation space of the agent in reinforcement learning, represents the initial state of the vehicle, Represents the initial state of the state machine, Indicated in The corresponding action space vector in the state, represents the system transfer equation vector, Indicates the position of the vehicle, Indicates the empty condition of the vehicle. Indicates the loading condition of the vehicle. Indicates the current vehicle position. Represents the state of the vehicle in the state machine, represents the horizontal coordinate of the vehicle at the initial moment, represents the vehicle ordinate at the initial moment, represents the loading status of the vehicle at the initial moment, Reinforcement learning The observation space of the moment, Indicates time, Indicates the time at which the truck driving status, express The state transition trigger condition at the moment, Indicates that the state machine is The state at any moment, Reinforcement learning The action space at the moment, Represents reinforcement learning action sampling.

4. The underground mining truck dispatching method based on state machine and reinforcement learning according to claim 3 is characterized in that: The state transition triggers include a trigger for completing a specific task, a trigger for reaching a specific location, and a trigger for satisfying a time or capacity constraint; The trigger formula of the state transition trigger is shown in the following equations (2)-(4): （2） （3） （4） In the formula, Indicates that dimensional scene interval diagonal matrix, represents the i-th interval, Indicates the number of intervals into which the scene is divided.

5. The underground mining truck dispatching method based on state machine and reinforcement learning according to claim 2 is characterized in that: The reward function includes: task completion reward, vehicle conflict penalty, time step penalty and timeout penalty; The termination conditions include: each loading node reaches a specified number of tasks, two or more trucks are stuck for more than a specific time step, and the maximum time step limit is exceeded.

6. The underground mining truck dispatching method based on state machine and reinforcement learning according to claim 1 is characterized in that: The step of obtaining a scenario model of an underground mine transportation system in S1, constructing a topology model of the underground mine transportation system according to the scenario model, and constructing an underground mine truck fleet operation problem according to the topology model and the conflict type includes: Obtain a scenario model of an underground mine transportation system, simplify the scenario model to obtain key elements, and construct a topological structure model of the underground mine transportation system according to the key elements; wherein the key elements include loading / unloading nodes, branch roads and main road sections, passing chambers, and a two-way single lane driving environment; According to the topological structure model, the conflict between passing vehicles in the chamber and the conflict between intersection nodes, the operation problem of underground mining truck fleet is constructed.

7. An underground mining truck dispatching device based on a state machine and reinforcement learning, the underground mining truck dispatching device based on a state machine and reinforcement learning is used to implement the underground mining truck dispatching method based on a state machine and reinforcement learning as claimed in any one of claims 1 to 6, characterized in that: The device comprises: An acquisition module is used to acquire a scenario model of an underground mine transportation system, construct a topology model of the underground mine transportation system according to the scenario model, and construct an underground mine truck fleet operation problem according to the topology model and the conflict type; A construction module is used to construct a fleet scheduling decision model based on a state machine and reinforcement learning according to the underground mining truck fleet operation problem; wherein each truck in the underground mining truck fleet is used as an intelligent agent of the fleet scheduling decision model, and the intelligent agent is equipped with a state machine; The output module is used to solve the fleet scheduling decision model with the goal of maximizing transportation efficiency, reducing waiting time and reducing conflicts, and obtain the scheduling results of mining trucks in the underground transportation working face with two-way single-lane driving characteristics.

8. The underground mining truck dispatching device based on state machine and reinforcement learning according to claim 7 is characterized in that: The fleet scheduling decision model based on state machine and reinforcement learning is constructed, including: Take the truck transportation stage as the state and design the state transition trigger; The action space is adjusted according to the truck driving range and the state of the state machine; Designing an observation space, wherein the observation space includes the horizontal and vertical coordinates of each truck and the state number of the state machine where each truck is located; Set the reward function and termination condition.

9. An underground mining truck dispatching device, characterized in that: The underground mining truck dispatching equipment comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images