Underground mine truck scheduling method and device based on state machine and reinforcement learning

Abstract

The application provides a kind of underground mine truck scheduling method and device based on state machine and reinforcement learning, it is related to underground mine transportation system scheduling optimization technical field.The method comprises: obtaining the scene model of underground mine transportation system, constructs topological structure model, constructs the underground mine truck fleet operation problem according to topological structure model and conflict type, and then constructs the fleet scheduling decision model based on state machine and reinforcement learning;Wherein, each truck is regarded as the agent of fleet scheduling decision model, and the state machine is equipped for the agent;With the goal of maximizing transportation efficiency, reducing waiting time and reducing conflict, the fleet scheduling decision model is solved, and the scheduling result of mine truck in the underground transportation working face with bidirectional single lane driving characteristics is obtained.The application realizes the intelligent scheduling of underground mine truck fleet by combining state machine and reinforcement learning technology, to improve transportation efficiency, reduce traffic conflict and reduce energy consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underground mine transportation system scheduling optimization, in particular to an underground mine truck scheduling method and device based on a state machine and reinforcement learning. BACKGROUND

[0002] With the in-depth research and development of intelligent mining equipment, advanced technologies such as data acquisition, positioning and navigation, and intelligent and autonomous control of underground mine trucks have been widely applied in mining operations at home and abroad. The introduction of these technologies has greatly improved the intelligent level of mine operation. However, underground mining operations involve the coordinated operation of multiple devices, and efficient and safe production requires coordinated operation between devices. Different stages of underground mining production are connected through transportation and hoisting processes, forming a complete mining production cycle. In this cycle, the underground mine truck fleet, as an important part of the transportation link, is responsible for shuttling between the mine and the mine, and performing ore transfer tasks. This multi-vehicle coordinated transportation system puts higher requirements on scheduling management.

[0003] Existing research on underground mine truck fleet management mainly focuses on two scheduling strategies: vehicle-by-vehicle scheduling strategy and global scheduling strategy. The vehicle-by-vehicle scheduling strategy assigns a new task to a truck only after it has completed the current transportation task and is ready to depart again. This method is similar to the automatic guided vehicle in manufacturing. The goal of this strategy is to find the shortest conflict-free route without modifying the existing routes reserved by other trucks. Although this method can solve the conflict problem to some extent, it is difficult to adapt to changes in loading / unloading nodes and fluctuations in transportation tasks due to the need to build a discrete graph.

[0004] Current methods for scheduling underground mine truck fleets do not fully exploit the potential of automated intelligent equipment, especially before large-load autonomous mine trucks that can accurately respond to operational scheduling instructions and adjust their speed. In recent years, reinforcement learning, as a machine learning method for solving decision-making problems in system-environment interactions, has shown its advantages in scheduling problems. However, existing RL-based methods mainly focus on scheduling problems in open-pit mining operations and within network structures, and have not yet explored their application in the two-way single-channel underground mine transportation scenario. SUMMARY

[0005] To solve the technical problem that existing RL-based methods mainly focus on scheduling problems in open-pit mining operations and within network structures, and have not yet explored their application in the two-way single-channel underground mine transportation scenario, the present application provides an underground mine truck scheduling method and device based on a state machine and reinforcement learning. The technical solution is as follows:

[0006] In one aspect, a state machine and reinforcement learning-based underground mine truck scheduling method is provided. The method is implemented by an underground mine truck scheduling device. The method comprises:

[0007] S1, obtaining a scene model of an underground mine transportation system, constructing a topological structure model of the underground mine transportation system according to the scene model, and constructing an underground mine truck fleet operation problem according to the topological structure model and the conflict type.

[0008] S2, constructing a fleet scheduling decision model based on a state machine and reinforcement learning according to the underground mine truck fleet operation problem; wherein each truck in the underground mine truck fleet is regarded as an agent of the fleet scheduling decision model, and the agent is equipped with a state machine.

[0009] S3, solving the fleet scheduling decision model to obtain a scheduling result of the mine truck in the underground transportation working face with the bidirectional single-lane driving characteristic, with the goal of maximizing the transportation efficiency, reducing the waiting time, and reducing the conflict.

[0010] Optionally, the construction of the fleet scheduling decision model based on the state machine and reinforcement learning in S2 comprises:

[0011] The stage of truck transportation is regarded as a state, and a state transition trigger is designed.

[0012] The action space is adjusted according to the truck driving interval and the state of the state machine.

[0013] An observation space is designed, which includes the horizontal and vertical coordinates of each truck and the state number of the state machine in which each truck is located.

[0014] A reward function and a termination condition are set.

[0015] Optionally, the state machine reinforcement learning system model in S2 is as shown in the following formula (1):

[0016] (1)

[0017] In the formula, represents the state machine reinforcement learning system equation, represents the system state transition trigger matrix, represents the system finite state matrix, represents the vehicle position state, represents the observation space of the agent in reinforcement learning, represents the initial state of the vehicle, represents the initial state of the state machine, represents the corresponding action space vector in the state, represents the system transition equation vector, ​Indicates the vehicle's position, Indicates the vehicle's empty condition. Indicates the loading condition of the vehicle. Indicates the current vehicle position. Represents the state of the vehicle in the state machine, represents the horizontal coordinate of the vehicle at the initial moment, represents the vehicle's ordinate at the initial moment, Indicates the loading status of the vehicle at the initial moment, Reinforcement learning in The observation space of the moment, Indicates time, The truck is at time driving status, express The state transition trigger condition at the moment, Indicates that the state machine is The state at any moment, Reinforcement learning in The action space at each moment, represents reinforcement learning action sampling.

[0018] Optionally, the state transition trigger includes a trigger of completing a specific task, a trigger of arriving at a specific location, or a trigger of satisfying a time or capacity constraint.

[0019] The trigger formula of the state transition trigger is shown in the following equations (2)-(4):

[0020] (2)

[0021] (3)

[0022] (4)

[0023] Where, Indicates that dimensional scene interval diagonal matrix, represents the i-th interval, Indicates the number of intervals into which the scene is divided.

[0024] Optionally, the reward function includes: task completion reward, vehicle conflict penalty, time step penalty and timeout penalty.

[0025] Termination conditions include: reaching a specified number of tasks per loading node, two or more trucks being stuck for more than a certain time step, and exceeding the maximum time step limit.

[0026] Optionally, the scene model of the underground mine transportation system in S1 is acquired, a topological structure model of the underground mine transportation system is constructed according to the scene model, and a mine truck fleet operation problem is constructed according to the topological structure model and the conflict type, including:

[0027] The scene model of the underground mine transportation system is acquired, and key elements are obtained by simplifying the scene model to construct a topological structure model of the underground mine transportation system; wherein the key elements include a loading / unloading node, a branch and a main road section, a passing chamber and a bidirectional single-lane driving environment.

[0028] According to the topological structure model, the passing chamber conflict and the intersection node conflict, a mine truck fleet operation problem is constructed.

[0029] On the other hand, a state machine and reinforcement learning-based underground mine truck scheduling device is provided, which is applied to the state machine and reinforcement learning-based underground mine truck scheduling method, and the device includes:

[0030] The acquisition module is configured to acquire a scene model of an underground mine transportation system, construct a topological structure model of the underground mine transportation system according to the scene model, and construct a mine truck fleet operation problem according to the topological structure model and the conflict type.

[0031] The construction module is configured to construct a fleet scheduling decision model based on a state machine and reinforcement learning according to the mine truck fleet operation problem; wherein each truck in the mine truck fleet is regarded as an agent of the fleet scheduling decision model, and the agent is equipped with a state machine.

[0032] The output module is configured to solve the fleet scheduling decision model to obtain a scheduling result of the mine truck in the underground transportation working face with bidirectional single-lane driving characteristics, with the goal of maximizing transportation efficiency, reducing waiting time and reducing conflicts.

[0033] On the other hand, an underground mine truck scheduling device is provided, which includes a processor and a memory having computer readable instructions stored thereon, wherein the computer readable instructions are executed by the processor to implement any of the above state machine and reinforcement learning-based underground mine truck scheduling methods.

[0034] On the other hand, a computer readable storage medium is provided, which stores at least one instruction, wherein the at least one instruction is loaded and executed by a processor to implement any of the above state machine and reinforcement learning-based underground mine truck scheduling methods.

[0035] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0036] In this paper, a state machine reinforcement learning method is proposed by constructing a topological structure model and analyzing conflict types. The state machine is equipped with a state machine whose states represent the truck transportation stages. The state machine is deeply coupled with reinforcement learning, and relevant formulas are used to describe the system state and action transitions. The action and observation spaces of the RL are adjusted, and the reward function and termination conditions are set. Case studies have verified the effectiveness of the algorithm, such as the changes in the action space dimension, the comparison of different RL algorithms, and the performance analysis of different truck numbers and task numbers. It has significant advantages over genetic algorithms. Time and energy consumption weights are introduced to optimize vehicle scheduling and achieve optimal energy efficiency. This system can improve transportation efficiency, reduce conflicts, and lower energy consumption, and is suitable for scheduling and management of underground metal mine transportation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 This is a flow chart of an underground mining truck scheduling method based on a state machine and reinforcement learning provided by an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a transportation system provided by an embodiment of the present invention;

[0040] Figure 3 This is a topological diagram of the working surface of the transportation system provided by an embodiment of the present invention;

[0041] Figure 4 This is a state machine state transition flow chart provided by an embodiment of the present invention;

[0042] Figure 5 This is a flow chart of a state machine combined with a Markov decision process provided by an embodiment of the present invention;

[0043] Figure 6 This is a block diagram of an underground mining truck dispatching device based on a state machine and reinforcement learning provided by an embodiment of the present invention;

[0044] Figure 7 The present invention is a schematic structural diagram of an underground mining truck dispatching device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0046] In the embodiments of the present application, the words such as "exemplary", "for example", etc. are used to represent an example, illustration, or description. Any embodiment or design scheme described as "exemplary" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "exemplary" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0047] In the embodiments of the present application, "image" and "picture" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "relevant" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0048] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0049] To make the technical problems, technical schemes and advantages to be solved by the present application more clear, the following will be described in detail in conjunction with the drawings and specific embodiments.

[0050] The embodiments of the present application provide a mine truck scheduling method based on a state machine and reinforcement learning, which can be implemented by a mine truck scheduling device. The mine truck scheduling device can be a terminal or a server. Figure 1 As shown in the flowchart of the mine truck scheduling method based on the state machine and reinforcement learning, the processing flow of the method can include the following steps:

[0051] S1, acquire a scene model of an underground mine transportation system, construct a topological structure model of the underground mine transportation system according to the scene model, and construct a mine truck fleet operation problem according to the topological structure model and conflict types.

[0052] In a feasible implementation manner, the scene model of the underground mine transportation system is constructed, as shown in Figure 2 After reasonable simplification, the key elements are extracted, including the loading / unloading nodes, branch and main road sections, car switching chambers and bidirectional single-lane driving environment, to construct the topological structure model. Typical conflict types are analyzed in depth, including car switching chamber conflicts and intersection node conflicts. Based on these conflict analyses, the core problems to be solved in the fleet operation process are clarified, such as avoiding vehicle conflicts, reducing delays and optimizing car switching operations, to achieve efficient and smooth transportation.

[0053] Specifically, the present application focuses on the underground metal mine transportation system, which mainly consists of ore body, mining working face, ore chute, loading-transport-unloading machine, underground mine truck, main ore chute, crusher, underground mine warehouse, hoisting equipment and other parts. For the convenience of analysis, the complex transportation system is simplified into a topological structure model, such as Figure 3 , including ore loading and unloading nodes, branch and main road sections, car switching chamber and bidirectional single lane driving environment. The main road section, such as 1-2-3-4-5-6, is relatively long, up to several hundred meters to thousands of meters; the branch section 4-7, 5-8 and 6-9 is relatively short, containing loading nodes. The operation nodes include ore loading nodes (nodes 7, 8 and 9) and ore unloading nodes (node 1), and the non-operation nodes include car switching chamber nodes (nodes 2 and 3) and lane intersection nodes (nodes 4, 5 and 6).

[0054] S2, according to the underground mine truck fleet operation problem, a fleet scheduling decision model based on state machine and reinforcement learning is constructed; wherein each truck in the underground mine truck fleet is regarded as an agent of the fleet scheduling decision model, and the agent is equipped with a state machine.

[0055] The present application combines state machine and RL (Reinforcement Learning, reinforcement learning) and can optimize the scheduling process of vehicles in the underground transportation working face with bidirectional single lane driving characteristics.

[0056] The fleet scheduling decision model can quickly generate the optimal driving plan of each truck according to different transportation task quantities and real-time environmental changes, avoiding the problem of deadlock and invalid parking of vehicles at non-operation nodes.

[0057] Optionally, the construction of the fleet scheduling decision model based on state machine and reinforcement learning in S2 comprises:

[0058] S21, the stage of truck transportation is taken as a state, and a state transition trigger is designed.

[0059] In a feasible implementation manner, each state of the state machine represents a specific stage or task of the truck in the transportation process, including loading, transportation, unloading, waiting and the like.

[0060] Further, the transition between states is determined by a specific trigger, including completion of a specific task, arrival at a specific location, satisfaction of time or capacity constraints and the like.

[0061] S22, the action space is adjusted according to the truck driving interval and the state of the state machine.

[0062] S23, an observation space is designed, which includes the horizontal and vertical coordinates of each truck and the state number of the state machine in which each truck is located.

[0063] In an implementable embodiment, the optimized observation space and action space can effectively improve the training effect and decision quality of the agent, wherein the observation space includes each state in the state machine model, and the action space is a dedicated action set designed for each state.

[0064] S24, set the reward function and termination condition.

[0065] The state machine reinforcement learning method is proposed, and the scene framework is carefully constructed. In this framework, each agent is equipped with a state machine, and the state accurately represents the specific stage or task of the truck during transportation, such as loading, transportation, dumping, waiting, etc. The transition between states is strictly controlled by specific triggers, including task completion, arrival at a specific location, or meeting time and capacity constraints. Under this framework, the state machine is deeply coupled with RL, and a series of formulas such as system state transition trigger matrix and finite state matrix are used to accurately describe the system state transition and action transition process.

[0066] Specifically, the RL action space is adjusted according to the truck driving interval and state machine state, and the action space covers a variety of operations such as advancing at different speeds, entering and exiting the wrong car chamber, and waiting, and the meaning of these operations will dynamically change according to the state of the truck in the state machine. Figure 4 The state machine interval is divided, the state and action are determined, and the state transition trigger is determined. The observation space is designed to include the horizontal and vertical coordinates of each truck and the state number in the state machine. As the number of trucks participating in the task changes, the observation space dimension is adjusted accordingly to more accurately describe the scheduling task, so that the RL algorithm can make intelligent decisions based on the comprehensive state of all trucks in the system.

[0067] As Figure 5 , the reward function is set, which includes task completion reward, vehicle conflict penalty, time step penalty, overtime penalty, and other parts, and the termination condition is determined, such as when each loading node reaches the specified task quantity, when two or more trucks are stalled for more than a certain time step, when the maximum time step limit is exceeded, the corresponding reward or termination operation is triggered. In this invention, a reward value calculation formula is designed, which considers factors such as task completion, conflict frequency, task time consumption, and whether to exceed the time limit. The agent adjusts its strategy according to the reward value to maximize the cumulative reward.

[0068] Optionally, the descriptive formula of the state machine reinforcement learning system can be summarized as follows:

[0069] (1)

[0070] In the formula, represents the state machine reinforcement learning system equation, a system state transition trigger matrix, a system finite state matrix, a vehicle position state, an observation space of an agent in reinforcement learning, an initial state of a vehicle, an initial state of a state machine, a corresponding action space vector in a state, a system transition equation vector, a position of a vehicle, an empty condition of a vehicle, a loaded condition of a vehicle, a position of a vehicle at a current time, a state of a vehicle in a state machine, a horizontal coordinate of a vehicle at an initial time, a vertical coordinate of a vehicle at an initial time, a loaded condition of a vehicle at an initial time, an observation space of reinforcement learning at a time, a time, a driving state of a truck at a time , a state transition trigger condition at a time, a state of a state machine at a time, an action space of reinforcement learning at a time, reinforcement learning action sampling, a system state transition equation.

[0071] A trigger formula of a state transition trigger is shown in the following formulas (2)-(4):

[0072] (2)

[0073] (3)

[0074] (4)

[0075] In the formulas, denotes a diagonal matrix with a scene interval, is a state transition trigger column vector with a dimension, denotes the i-th interval, denotes the number of intervals of scene division, which is calculated by the formula Determine whether the truck is in a certain interval, if in the interval, the value is 1, otherwise the value is 0; by dividing the scene into n sub-intervals according to the scene node position and the lane section, and determining whether the truck is in a certain interval, and assigning it a corresponding state transition trigger for the state machine state conversion of the truck.

[0076] The observation space is designed to include the horizontal and vertical coordinates of each truck and the state number in its state machine. As the number of trucks participating in the task changes, the dimension of the observation space is adjusted accordingly to more accurately describe the scheduling task, enabling the RL algorithm to make intelligent decisions based on the comprehensive state of all trucks in the system.

[0077] S3, aiming at maximizing transportation efficiency, reducing waiting time and reducing conflicts, the truck fleet scheduling decision model is solved to obtain the scheduling result of the mine truck in the underground transportation working face with the bidirectional single lane driving characteristics.

[0078] In a feasible implementation, the reinforcement learning algorithm is trained through a Markov decision process to maximize transportation efficiency, reduce waiting time and conflicts. The size of the truck fleet and the operation strategy are dynamically adjusted to cope with the complex underground transportation network, and the transportation efficiency is maximized.

[0079] The effectiveness of the algorithm is verified by a series of comprehensive and in-depth case studies. When comparing different RL algorithms, success rate, conflict rate and time control are evaluated from multiple dimensions. When analyzing the performance under different number of trucks, it is found that when the number of trucks is less than 7, the success rate increases steadily with the increase of training iterations, and the conflict rate gradually decreases and converges to 0, but as the number of trucks increases, the convergence speed slows down, and when it reaches 7, the training performance deteriorates, and the task cannot be completed successfully. It is determined that the maximum capacity of the system under the specific topology is 6 trucks. When testing the performance under different number of tasks, taking 3 trucks as an example, when the number of loading tasks increases from 2 to 10 and 20, the time to complete all tasks increases linearly, but the operation stability is not affected, indicating that the effectiveness of the method is not disturbed by the change of the number of tasks. Compared with the genetic algorithm, the results show that the present application has significant advantages in reducing waiting time, number of start-stop times, total time consumption, etc., such as reducing total time consumption by 22.24%, reducing waiting time by 44.74%, significantly reducing the number of stops, and without the need to retrain the model when the number of tasks changes.

[0080] For example, the state machine reinforcement learning method, PPO and A2C algorithms are used to train the model. The scenario is that three mine carts perform two tasks from the unloading node to different loading nodes. The results show that the algorithm success rate of the state machine reinforcement learning method is high and stable, the conflict rate is low, and the time consumption is stable; the PPO algorithm is low at the beginning but improves in the later stage, and the conflict rate gradually decreases and fluctuates; the A2C algorithm is poor at the beginning and fluctuates greatly. This shows that the state machine reinforcement learning method performs well in terms of success rate, conflict rate and time control. Gradually increase the number from 3 mine carts, and Figure 3 Training under this topological structure. When the number of vehicles is less than 7, the success rate increases with iteration, the conflict rate decreases and converges, and the average time step converges. When the number of vehicles reaches 7, the training performance deteriorates. This indicates that the maximum capacity of the three-branch lane transport network is 6 vehicles. Currently, 3 vehicles are sufficient for operation. If production capacity is increased or time is reduced, further analysis of the optimal number of vehicles is required.

[0081] Furthermore, time and energy weightings are introduced to comprehensively account for the various resistances encountered by a vehicle during travel, including rolling resistance (related to the vehicle's gravity and rolling resistance coefficient), air resistance (related to the air resistance coefficient, frontal area, and vehicle speed), and acceleration resistance (related to the rotational mass conversion formula for vehicles equipped with fixed transmission ratios and vehicle mass). This establishes a vehicle power balance equation, which is used to calculate the mechanical energy consumption of a vehicle traveling at a constant speed within any interval. Vehicle allocation principles are determined based on vehicle operating energy efficiency, such as rationally assigning more vehicles to loading nodes farther from unloading nodes. As the number of vehicles and the number of tasks at each loading node increases, the optimal energy efficiency is found when the number of vehicles is six, by comprehensively considering the relationship between the time required to complete a task and the total energy consumption of all vehicles when completing the task, as well as the number of vehicles.

[0082] In an embodiment of the present invention, a state machine reinforcement learning method is proposed by constructing a topological structure model and analyzing the conflict types. The state machine is equipped with a state machine for the intelligent agent, and its state represents the truck transportation stage. The state machine is deeply coupled with reinforcement learning, and the system state transition and action transition are described by relevant formulas. The action and observation space of RL are adjusted, and the reward function and termination conditions are set. The effectiveness of the algorithm has been verified through case studies, such as the change in action space dimension, the comparison of different RL algorithms, and the performance analysis of different numbers of trucks and tasks. It has significant advantages over genetic algorithms. The time and energy consumption weights are introduced to optimize vehicle scheduling and achieve optimal energy efficiency. The system can improve transportation efficiency, reduce conflicts and reduce energy consumption, and is suitable for scheduling and management of underground metal mine transportation systems.

[0083] Figure 6 This is a block diagram of an underground mining truck scheduling device based on a state machine and reinforcement learning according to an exemplary embodiment. The device is used in an underground mining truck scheduling method based on a state machine and reinforcement learning. Figure 6The device comprises an acquisition module 310, a construction module 320 and an output module 330.

[0084] The acquisition module 310 is configured to acquire a scene model of the underground mine transportation system, construct a topological structure model of the underground mine transportation system according to the scene model, and construct a mine truck fleet operation problem according to the topological structure model and the conflict type.

[0085] The construction module 320 is configured to construct a fleet scheduling decision model based on a state machine and reinforcement learning according to the mine truck fleet operation problem, wherein each truck in the mine truck fleet is regarded as an agent of the fleet scheduling decision model, and the agent is equipped with a state machine.

[0086] The output module 330 is configured to solve the fleet scheduling decision model to obtain a scheduling result of the mine truck in the underground transportation working face with the bidirectional single-lane driving characteristic, with the goal of maximizing the transportation efficiency, reducing the waiting time and reducing the conflict.

[0087] In the embodiment of the present application, by constructing the topological structure model, analyzing the conflict type, and proposing the state machine reinforcement learning method, the agent is equipped with the state machine, the state of which represents the truck transportation stage, the state machine is deeply coupled with the reinforcement learning, and the system state transition and action transition are described through related formulas. The action and observation space of RL are adjusted, and the reward function and termination condition are set. Through case study, the effectiveness of the algorithm is verified, such as the change of action space dimension, the comparison of different RL algorithms, the performance analysis of different truck numbers and task numbers, which performs well, and has significant advantages compared with the genetic algorithm. The time and energy consumption weights are introduced to optimize the vehicle arrangement, and the energy efficiency is optimized. The system can improve the transportation efficiency, reduce the conflict and reduce the energy consumption, and is suitable for the scheduling management of the underground metal mine transportation system.

[0088] Figure 7 is a structural schematic diagram of an underground mine truck scheduling device provided by the embodiment of the present application, as Figure 7 shown, the underground mine truck scheduling device can comprise the underground mine truck scheduling device based on the state machine and the reinforcement learning shown in the above Figure 6 Optionally, the underground mine truck scheduling device 410 can comprise the first processor 2001.

[0089] Optionally, the underground mine truck scheduling device 410 can further comprise a memory 2002 and a transceiver 2003.

[0090] The first processor 2001, the memory 2002 and the transceiver 2003 can be connected through a communication bus.

[0091] The following will be described in combination with Figure 7The components of the underground mine truck dispatching device 410 are described in detail as follows:

[0092] The first processor 2001 is the control center of the underground mine truck dispatching device 410, which can be one processor or a plurality of processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0093] Optionally, the first processor 2001 can execute various functions of the underground mine truck dispatching device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0094] In a specific implementation, as an embodiment, the first processor 2001 can include one or more CPUs, such as the CPU0 and CPU1 shown in FIG. 1. Figure 7

[0095] In a specific implementation, as an embodiment, the underground mine truck dispatching device 410 can also include a plurality of processors, such as the first processor 2001 and the second processor 2004 shown in FIG. 1. Each of these processors can be a single-CPU or a multi-CPU. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Figure 7

[0096] The memory 2002 is used to store software programs for implementing the schemes of the present application and is controlled by the first processor 2001 for execution. The specific implementation can refer to the above method embodiments, which will not be described here.

[0097] ​​Optionally, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magneto-optical disk, a magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by the computer, but is not limited to this. The memory 2002 can be integrated with the first processor 2001 or exist independently and be coupled with the first processor 2001 through an interface circuit (not shown in the figure) of the underground mine truck scheduling device 410, and the embodiments of the present application do not make specific limitations here. Figure 7

[0098] The transceiver 2003 is configured to communicate with a network device or a terminal device.

[0099] Optionally, the transceiver 2003 can include a receiver and a transmitter (not shown separately in the figure). The receiver is configured to realize the receiving function, and the transmitter is configured to realize the transmitting function. Figure 7

[0100] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently and be coupled with the first processor 2001 through an interface circuit (not shown in the figure) of the underground mine truck scheduling device 410, and the embodiments of the present application do not make specific limitations here. Figure 7

[0101] It should be noted that the structure of the underground mine truck scheduling device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure identification device can include more or fewer components than the figure, or combine certain components, or different component arrangements. ​

[0102] In addition, the technical effects of the underground mine truck scheduling device 410 can refer to the technical effects of the underground mine truck scheduling method based on the state machine and reinforcement learning described in the above method embodiments, which will not be repeated here.

[0103] ​​​​It is to be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0104] It is also to be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0105] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0106] It should be understood that the term "and / or" herein merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after it are in an "or" relationship, but it can also represent an "and / or" relationship, which can be understood according to the context before and after it.

[0107] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0108] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0109] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0111] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0113] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0114] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0115] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for dispatching underground mining trucks based on state machine and reinforcement learning, characterized in that: The method comprises: S1. Obtain a scenario model of an underground mine transportation system, construct a topology model of the underground mine transportation system based on the scenario model, and construct an underground mine truck fleet operation problem based on the topology model and conflict types; S2. Based on the underground mining truck fleet operation problem, construct a fleet scheduling decision model based on a state machine and reinforcement learning; wherein each truck in the underground mining truck fleet is used as an intelligent agent in the fleet scheduling decision model, and each intelligent agent is equipped with a state machine; S3. Solving the fleet scheduling decision model with the goal of maximizing transportation efficiency, reducing waiting time and reducing conflicts, and obtaining a scheduling result for mining trucks in an underground transportation working face with a two-way single-lane driving characteristic; The fleet scheduling decision model constructed in S2 is based on state machine and reinforcement learning, including: Take the truck transportation stage as the state and design the state transition trigger; The action space is adjusted according to the truck's driving range and the state of the state machine; Design an observation space, wherein the observation space includes the horizontal and vertical coordinates of each truck and the state number of the state machine where each truck is located; Set the reward function and termination conditions; The fleet scheduling decision model based on state machine and reinforcement learning in S2 is shown in the following formula (1): Where M represents the state machine reinforcement learning system equation, A represents the system state transfer trigger matrix, S represents the system finite state matrix, P represents the vehicle position state, Obs represents the observation space of the agent in reinforcement learning, p0 represents the initial state of the vehicle, s0 represents the initial state of the state machine, Act represents the initial state of the vehicle in s t The corresponding action space vector in the state, represents the system transfer equation vector, x represents the position of the vehicle, y represents the empty state of the vehicle, d represents the loaded state of the vehicle, p represents the current position of the vehicle, s represents the state of the vehicle in the state machine, x0 represents the horizontal coordinate of the vehicle at the initial moment, y0 represents the vertical coordinate of the vehicle at the initial moment, d0 represents the loaded state of the vehicle at the initial moment, obs t Represents the observation space of reinforcement learning at time t, t represents time, p t represents the driving state of the truck at time t, a t Indicates the state transition trigger condition at time t, s t Indicates the state of the state machine at time t, act t Represents the action space of reinforcement learning at time t, and sample represents the reinforcement learning action sampling; The state transition triggers include triggers for completing a specific task, reaching a specific location, and satisfying time or capacity constraints; The trigger formula of the state transition trigger is shown in the following equations (2)-(4): F1(A,p t )=f(p t ×Pos)·A=a t (2) Where Pos represents the scene interval diagonal matrix with n dimensions, pos i represents the i-th interval, and n represents the number of intervals into which the scene is divided.

2. The underground mining truck scheduling method based on state machine and reinforcement learning according to claim 1 is characterized in that: The reward function includes: task completion reward, vehicle conflict penalty, time step penalty and timeout penalty; The termination conditions include: each loading node reaches a specified number of tasks, two or more trucks are stuck for more than a specific time step, and the maximum time step limit is exceeded.

3. The underground mining truck scheduling method based on state machine and reinforcement learning according to claim 1 is characterized in that: The step S1 of obtaining a scenario model of the underground mine transportation system, constructing a topology model of the underground mine transportation system according to the scenario model, and constructing an underground mine truck fleet operation problem according to the topology model and the conflict type includes: Obtain a scenario model of an underground mine transportation system, simplify the scenario model to obtain key elements, and construct a topological structure model of the underground mine transportation system based on the key elements; wherein the key elements include loading / unloading nodes, branch roads and main road sections, passing chambers, and a two-way single-lane driving environment; According to the topological structure model, the conflict of passing chambers and the conflict of intersection nodes, the operation problem of underground mining truck fleet is constructed.

4. An underground mining truck dispatching device based on a state machine and reinforcement learning, wherein the underground mining truck dispatching device based on a state machine and reinforcement learning is used to implement the underground mining truck dispatching method based on a state machine and reinforcement learning as described in any one of claims 1 to 3, characterized in that: The device comprises: an acquisition module, configured to acquire a scenario model of an underground mine transportation system, construct a topology model of the underground mine transportation system according to the scenario model, and construct an underground mine truck fleet operation problem according to the topology model and the conflict type; A construction module is used to construct a fleet scheduling decision model based on a state machine and reinforcement learning according to the underground mining truck fleet operation problem; wherein each truck in the underground mining truck fleet is used as an intelligent agent in the fleet scheduling decision model, and the intelligent agent is equipped with a state machine; The output module is used to solve the fleet scheduling decision model with the goal of maximizing transportation efficiency, reducing waiting time and reducing conflicts, and obtain the scheduling results of mining trucks in the underground transportation working face with two-way single-lane driving characteristics.

5. An underground mining truck dispatching device, characterized in that: The underground mining truck dispatching equipment includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 3.

Images