A Dynamic Scheduling System and Method for Unmanned Trucks in Open-Pit Mines Based on Reinforcement Learning

Abstract

This invention discloses a dynamic scheduling system and method for unmanned mining trucks in open-pit mines based on reinforcement learning. The system uses onboard sensors and terminals to perform real-time status assessments of the unmanned mining trucks and feeds this status information back to the scheduling processing center. The center optimizes the scheduling of all trucks that meet the requirements. A scheduling model is constructed based on reinforcement learning, trained using historical data, employs a greedy algorithm to improve the quality of the solution, and utilizes an adaptive reward adjustment mechanism. The scheduling model is repeatedly trained to obtain a better solution. A fault warning module can provide early warnings of potential malfunctions in the unmanned mining trucks. The system uses 5G networks for data transmission and computer technology to achieve real-time optimized scheduling of the mining trucks. This invention enables unmanned dynamic scheduling of unmanned mining trucks in open-pit mines, accurately predicts vehicle malfunctions, reduces labor intensity, improves transportation efficiency, ensures production and transportation safety, and accelerates the intelligentization of mining operations.

Description

Technical Field

[0001] This invention relates to a dynamic scheduling system and method for unmanned trucks, specifically to a dynamic scheduling system and method for unmanned trucks in open-pit mines based on reinforcement learning. Background Technology

[0002] With the development of artificial intelligence and 5G communication technologies, China's open-pit mining industry is gradually becoming intelligent and unmanned. Unmanned truck dispatching in open-pit mines is an important development direction in the mining sector. Through vehicle monitoring, command issuance, and transportation dispatch response, the full-process control and real-time dispatching of unmanned mining trucks is a crucial step in reducing costs for mining companies. However, the truck dispatching management systems currently widely used in open-pit mines have not achieved truly intelligent dispatching; they are more often used as auxiliary tools on-site.

[0003] Currently, many scholars have conducted extensive research on the vehicle scheduling problem in open-pit mines. However, most of these studies are based on ideal scenarios and focus on static scheduling problems under deterministic environmental conditions. In contrast, real-world mine production is a dynamic process, and there is still a lack of in-depth research on dynamic truck scheduling that takes into account uncertain factors. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a dynamic scheduling system and method for unmanned trucks in open-pit mines based on reinforcement learning, which realizes the dynamic scheduling of unmanned trucks in open-pit mines and accelerates the process of intelligent mining.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is: a dynamic scheduling system for unmanned trucks in open-pit mines based on reinforcement learning, comprising on-board sensors, on-board terminals and a dynamic scheduling system, wherein the on-board sensors are connected to the on-board terminals, and the on-board terminals are connected to the dynamic scheduling system through a network;

[0006] The dynamic scheduling system includes a scheduling processing center, a data storage module, a fault early warning module, a communication module, and a visual human-computer interaction platform. The scheduling processing center, data storage module, and fault early warning module are connected to each other in pairs, and the three are connected to the visual human-computer interaction platform through the communication module.

[0007] The scheduling and processing center includes a scheduling and processing host and a reward update module. The reward update module is integrated into the scheduling and processing host. The scheduling and processing host contains a high-precision 3D map and performs dynamic scheduling of mining cards based on reinforcement learning and greedy algorithms. The reward update module adopts an adaptive reward adjustment mechanism. By analyzing model performance indicators and adjusting reward weights, it feeds the updated reward function back to the scheduling and processing host, repeatedly trains the scheduling model, and realizes the system's dynamic scheduling and self-learning functions.

[0008] The data storage module is used to store and analyze historical data, scheduling data, and fault data acquired and generated by the system, and to build a database for the scheduling processing host to learn from.

[0009] The fault warning module is used to predict and diagnose the faults of mining trucks. By analyzing the status information of the mining trucks transmitted by the vehicle terminal, it predicts the possible fault types and occurrence times of the mining trucks, sets alarm thresholds, and when the thresholds are exceeded, the fault warning module will issue an alarm. The dispatch processing host will issue a command to the mining truck to go to the maintenance station for repair. At the same time, the fault warning module will transmit the analyzed real-time vehicle status, specific data and maintenance information to the visual human-machine interaction platform.

[0010] The communication module connects to the 5G network covering the entire open-pit mine, and is used for the transmission of data, information and scheduling commands for the entire system;

[0011] The visual human-computer interaction platform displays a real-time 3D map of the mine and the mine card data input by the system. It also provides a manual intervention mode. In case of an emergency, the operator can set up a temporary solution and make an external voice broadcast.

[0012] Furthermore, the vehicle-mounted sensors include one or more of the following: lidar, IMU, binocular camera, speed sensor, acceleration sensor, temperature sensor, pressure sensor, fuel level sensor, battery level sensor, and GPS; the vehicle-mounted terminal is equipped with a processor and global and local path planning algorithms.

[0013] Furthermore, the vehicle-mounted terminal is equipped with six real-time statuses for mining trucks: empty, loaded, transporting, unloading, faulty, and charging. The specific method for determining these six real-time statuses is based on signal data detected by the vehicle-mounted sensors and combined with the fault warning module:

[0014] No-load condition judgment: The pressure value detected by the pressure sensor is less than the preset no-load threshold, and this state continues for a period of time;

[0015] Loading status judgment: The pressure value detected by the pressure sensor increases at a certain rate, and the speed value detected by the speed sensor is 0;

[0016] Transportation status judgment: The pressure value detected by the pressure sensor is greater than or equal to the preset full load threshold, and the speed value detected by the speed sensor is not 0 or the acceleration sensor detects a value that is not 0.

[0017] Unloading status determination: The pressure value detected by the pressure sensor decreases at a certain rate;

[0018] Fault status determination: determined by the fault early warning module;

[0019] Charging status judgment: The remaining fuel or remaining power detected by the fuel level sensor or power sensor is less than or equal to the set charging value.

[0020] Furthermore, the method for the scheduling processing host to dynamically schedule mining cards is as follows:

[0021] S1: Define the state space S and action space A of the scheduling problem;

[0022] S2: Select the algorithm and gradient to build the reinforcement learning model;

[0023] A reinforcement learning model consists of an agent, an environment, a state, an action, a reward, and a policy; the reward function R(s,a) contains multiple metrics related to the scheduling objective, each with an initial weight ω. i Initialize the reward function:

[0024] R(s,a)=ω1·η+ω2·t avg +ω3·x avg +ω4·q+ω5·y;

[0025] Where η is the transportation efficiency, t avg It is the average waiting time, x avg y is the average driving distance, q is the maintenance cost, and y is the safety index;

[0026] The policy μ is a function that selects action a based on the current state s; the policy function is represented as a neural network:

[0027] a = π(s|θ);

[0028] Where θ is the parameter of the policy network;

[0029] The goal is to maximize long-term cumulative rewards while ensuring efficient operation of the mining cards and reducing maintenance costs. The objective function can be expressed as:

[0030]

[0031] Where π is the strategy, τ is a trajectory, and s t It is the state at time t, a t It is the action at time t, r(s) t ,a t ) is the reward at time t, γ is the discount factor, and T is the time step;

[0032] For a continuous action space, the DDPG algorithm is chosen, and its policy gradient is:

[0033]

[0034] Where, ρ 3It is a state distribution, Q φ (s,a) is the state-action value evaluated by the critic network;

[0035]

[0036]

[0037] Where, θ i and θ μ These are the parameters for the critic network and the actor network, respectively, and α is the learning rate;

[0038] S3: Use historical data to train the model and introduce a greedy algorithm to improve the quality of scheduling decisions;

[0039] First, initialize the DDPG model; second, collect experience; then update the network to stabilize the learning process; repeat the process of collecting experience and updating the network to continuously iterate and optimize the actor network and critic network.

[0040] Updated commentator network:

[0041]

[0042] Where Q(s,a) is the output of the critic network for state s and action a, r is the immediate reward, and γ is the discount factor. It is the output of the target critic network Q′ for the next state s′ and the action a′ chosen by the agent according to the policy μ′;

[0043] Updated actor network:

[0044]

[0045] in, It calculates the policy gradient. It is the gradient with respect to action a, given by the parameters θ of the critic network. Q Decide, The parameter θ is related to the policy μ. μ The gradient;

[0046] Update the target network:

[0047] θ i ′←τθ i +(1-τ)θ i ′;

[0048] Where, θ i These are the parameters of the original network, θ i ′ is the parameter of the target network, and τ is the update rate;

[0049] The model is trained using historical data. At each step of the action selection, a greedy algorithm is introduced to explore randomly with probability ε, i.e., the ε-greedy strategy, to improve the quality of the model solution.

[0050] S4: Deploy the trained model into the actual open-pit mine unmanned truck dynamic scheduling system for real-time scheduling;

[0051] S5: Evaluate the model's performance based on the new weight reward function generated by the reward update module, and repeatedly train the model to iteratively optimize it in order to achieve real-time dynamic scheduling.

[0052] Furthermore, the adaptive reward adjustment method of the reward update module is as follows:

[0053] S1: The reward update module initializes the reward function, assigning an initial weight ω to each metric in the function. i ;

[0054] S2: Start training the scheduling model and use the initialized reward function to guide the learning process;

[0055] S3: Regularly evaluate the model's performance during training, analyze the model's performance on different metrics, and identify which metrics have a significant impact on model performance.

[0056] S4: Adjust the weights ω of the corresponding indicators in the reward function based on the model's performance on different metrics. i The weight update formula is:

[0057]

[0058] Where λ is the learning rate for weight updates, It is the reward function with respect to the weight ω i The gradient;

[0059] S5: Retrain the model using the updated reward function to make the model pay more attention to metrics that have a significant impact on performance;

[0060] S6: Repeat steps S3 to S5 to continuously iterate and optimize the reward function and model performance.

[0061] Furthermore, the specific method of the ε-greedy strategy is as follows:

[0062] In the early stages of training, a large ε value is set, and the ε value is gradually decreased as training progresses. Then, an ε-greedy strategy is applied while training and scheduling the model. The model's performance under different ε values ​​is evaluated periodically, and the ε value is adjusted according to the model's performance under different ε values. After the model training is completed, a greedy algorithm is used to guide decision-making in practical applications.

[0063] A dynamic scheduling method for unmanned trucks in open-pit mines based on reinforcement learning is disclosed. The fault early warning module analyzes and calculates the status information of the trucks, predicts the possible fault types and occurrence times, sets an alarm threshold, and when this threshold is exceeded, the fault early warning module issues an alarm and sends feedback to the scheduling processing center. The scheduling processing center optimizes the scheduling of all trucks that meet the requirements, constructs a scheduling model based on reinforcement learning, trains it using historical data, employs a greedy algorithm to improve the quality of the solution, adopts an adaptive reward adjustment mechanism, and repeatedly trains the scheduling model to obtain a better solution. The data storage module stores and analyzes the data generated by the system to build a database. Finally, the communication module enables data and command transmission between the modules of the dynamic scheduling system, and between the dynamic scheduling system and the unmanned trucks and electric shovels, achieving real-time dynamic scheduling of the unmanned trucks.

[0064] Compared with existing technologies, the beneficial effects of this invention are: it constructs a scheduling model based on reinforcement learning, uses historical data for training, adopts a greedy algorithm to improve the quality of the solution, uses an adaptive reward adjustment mechanism, and repeatedly trains the scheduling model. At the same time, it utilizes 5G network and computer technology to realize the unmanned real-time dynamic scheduling of unmanned trucks in open-pit mines. It can predict vehicle failures, reduce the labor intensity of personnel, improve transportation efficiency, ensure production and transportation safety, and accelerate the intelligentization process of mines. Attached Figure Description

[0065] Figure 1 This is a block diagram of the dynamic scheduling system of the present invention;

[0066] Figure 2 This is a flowchart of the scheduling and processing center workflow of the present invention;

[0067] Figure 3 This is a flowchart of the greedy algorithm used by the scheduling and processing host of this invention;

[0068] Figure 4 This is a flowchart of the dynamic scheduling method for unmanned trucks in open-pit mines according to the present invention. Detailed Implementation

[0069] The invention will now be further described with reference to the accompanying drawings.

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] like Figure 1As shown, the present invention provides a dynamic scheduling system for unmanned trucks in open-pit mines based on reinforcement learning, including on-board sensors, on-board terminals and a dynamic scheduling system. The on-board sensors are connected to the on-board terminals, and the on-board terminals are connected to the dynamic scheduling system through a network.

[0072] The vehicle-mounted sensors and terminals are used to sense and judge the status information of the unmanned mining truck itself, and transmit the data collected by the vehicle-mounted sensors and the status of the mining truck itself to the dynamic scheduling system for processing in real time.

[0073] The dynamic scheduling system includes a scheduling processing center, a data storage module, a fault early warning module, a communication module, and a visual human-computer interaction platform. The scheduling processing center, data storage module, and fault early warning module are connected to each other in pairs, and the three are connected to the visual human-computer interaction platform through the communication module.

[0074] like Figure 2 As shown, the scheduling processing center includes a scheduling processing host and a reward update module, which is integrated into the scheduling processing host. The scheduling processing host contains a high-precision 3D map of the open-pit mine. By constructing a scheduling model, it dynamically schedules the mining trucks based on the status information of the mining trucks fed back by the vehicle terminal, using reinforcement learning and a greedy algorithm. The reward update module adopts an adaptive reward adjustment mechanism. By analyzing the model performance indicators and adjusting the reward weights, it feeds the updated reward function back to the scheduling processing host, repeatedly training the scheduling model to realize the system's dynamic scheduling and self-learning functions.

[0075] The data storage module is used to store and analyze historical data, scheduling data, fault data, etc. acquired and generated by the system, and to build a database for the scheduling processing host to learn from.

[0076] The fault warning module is used to predict and diagnose the faults of mining trucks. By analyzing the status information of the mining trucks transmitted by the vehicle terminal, it predicts the possible fault types and occurrence times of the mining trucks, sets alarm thresholds, and when the thresholds are exceeded, the fault warning module will issue an alarm. The dispatch processing host will issue a command to the mining truck to go to the maintenance station for repair. At the same time, the fault warning module will transmit the analyzed real-time vehicle status, specific data and maintenance information to the visual human-machine interaction platform.

[0077] The communication module connects to the 5G network covering the entire open-pit mine for the transmission of data, information and scheduling commands of the entire system; the 5G network is used for data transmission, and several 5G base stations are built in the open-pit mine, covering the entire mine production work area and scheduling processing center for data transmission.

[0078] The visual human-machine interaction platform is equipped with multiple interfaces and connects to the 5G network. It provides a large touch-screen display that shows the mine's 3D map in real time and data input from the system, including mine truck location, speed, driving route, working time, oil volume, oil temperature, oil pressure, power, voltage, tire pressure, and the status of the electric shovel and crushing station. Users can query data from the data storage module by selecting different function panels. It also provides a manual intervention mode, allowing operators to set temporary solutions and make voice announcements in case of emergencies.

[0079] Specifically, the vehicle-mounted sensors include one or more of the following: lidar, IMU, binocular camera, speed sensor, acceleration sensor, temperature sensor, pressure sensor, fuel level sensor, battery level sensor, and GPS, used to sense the environment and vehicle status, such as people, vehicles, and objects; the vehicle-mounted terminal is equipped with a processor and global and local path planning algorithms, which can realize functions such as environmental perception, self-positioning error of less than 10cm, optimal path selection, and avoidance of people and obstacles, enabling fully unmanned autonomous operation.

[0080] The vehicle-mounted terminal has six real-time statuses for mining trucks: empty, loaded, transporting, unloading, faulty, and charging. The specific method for determining the six real-time statuses of the mining trucks is based on signal data detected by the vehicle-mounted sensors and combined with the fault warning module:

[0081] No-load condition judgment: The pressure value detected by the pressure sensor is less than the preset no-load threshold, and this state lasts for 2 minutes;

[0082] Loading status judgment: The pressure value detected by the pressure sensor increases at a certain rate, and the speed value detected by the speed sensor is 0;

[0083] Transportation status judgment: The pressure value detected by the pressure sensor is greater than or equal to the preset full load threshold, and the speed value detected by the speed sensor is not 0 or the acceleration sensor detects a value that is not 0.

[0084] Unloading status determination: The pressure value detected by the pressure sensor decreases at a certain rate;

[0085] Fault status determination: determined by the fault early warning module;

[0086] Charging status judgment: The remaining fuel or remaining power detected by the fuel level sensor or power sensor is less than or equal to the set charging value.

[0087] Specifically, the scheduling processing host is based on reinforcement learning. It constructs a scheduling model, trains it using historical data and an adaptive reward adjustment mechanism, and employs a greedy algorithm to improve the quality of the results. The method by which the scheduling processing host dynamically schedules mining cards is as follows:

[0088] S1: Define the state space S and action space A of the scheduling problem;

[0089] The state space S includes all information that may affect scheduling decisions, such as the current position and speed of each mining truck, loading status, fault status, current task, remaining power / fuel, road congestion, and the working status of each loading / unloading point, as well as the location and capacity of charging stations / refueling points. The action space A includes all possible scheduling decisions, such as assigning a mining truck to a specific loading / unloading point, adjusting the driving speed of the mining truck, scheduling the mining truck to a repair station when it needs maintenance, or scheduling the mining truck to a charging station / refueling point when it needs charging, and adjusting the digging speed of the electric shovel.

[0090] S2: Select the algorithm and gradient to build the reinforcement learning model;

[0091] A reinforcement learning model typically consists of an agent, an environment, a state, an action, a reward, and a policy. The agent is responsible for selecting an action based on the current state. The environment provides state feedback and reward signals. The reward is the feedback the agent receives from the environment after taking an action, such as completing a ore transport or reducing waiting time. The policy is used to predict the best action based on the current state. The reward function R(s,a) contains multiple metrics related to the scheduling objective, each with an initial weight ω. i Initialize the reward function:

[0092] R(s,a)=ω1·η+ω2·t avg +ω3+x avg +ω4·q+ω5·y;

[0093] Where η is the transportation efficiency, t avg It is the average waiting time, x avg y is the average driving distance, q is the maintenance cost, and y is the safety index;

[0094] The policy μ is a function that selects action a based on the current state s; the policy function is represented as a neural network:

[0095] a = π(s|θ);

[0096] Where θ is the parameter of the policy network;

[0097] The goal is to maximize long-term cumulative rewards while ensuring efficient operation of the mining cards and reducing maintenance costs. The objective function can be expressed as:

[0098]

[0099] Where π is the strategy, τ is a trajectory, and s t It is the state at time t, a t It is the action at time t, r(s)t ,a t ) is the reward at time t, γ is the discount factor, and T is the time step;

[0100] For a continuous action space, the Deep Deterministic Policy Gradient (DDPG) algorithm is chosen. DDPG combines the advantages of deterministic policy gradient (DPG) and deep Q-learning (DQN), including an actor network (for selecting the best action based on the current state), a critic network (for evaluating the values ​​of the current state and action), an experience replay buffer (for storing the agent's interaction experience with the environment), and a target network (for stabilizing the learning process); its policy gradient is:

[0101]

[0102] Where, ρ 3 It is a state distribution, Q φ (s,a) is the state-action value evaluated by the critic network;

[0103]

[0104] Where, θ i and θ μ These are the parameters for the critic network and the actor network, respectively, and 'a' is the learning rate, which determines the magnitude of parameter updates.

[0105] S3: Use historical data to train the model and introduce a greedy algorithm to improve the quality of scheduling decisions;

[0106] First, the DDPG model is initialized, including the actor network, the critic network, and their target network. Simultaneously, the experience replay buffer is initialized. Second, experience is collected as the agent interacts with the environment, collecting data on state, action, reward, and next state, which is stored in the experience replay buffer. Then, the networks are updated by randomly sampling a batch of data from the experience replay buffer to update the actor and critic networks. Next, the parameters of the target network are periodically updated to stabilize the learning process. This process of collecting experience and updating the network is repeated iteratively to optimize the actor and critic networks. Finally, a reward function is used to evaluate the model's performance, and the model is iteratively optimized as needed.

[0107] Updated commentator network:

[0108]

[0109] Where Q(s,a) is the output of the commentator network for state s and action a, r is the immediate reward, which is the reward that the agent receives from the environment immediately after taking an action, and γ is the discount factor used to weigh the value of immediate reward and future reward. The value of γ is usually between 0 and 1. When it is close to 1, it means that the future reward has a greater impact on the current decision, while when it is close to 0, it means that only the immediate reward is considered. The output of the target commentator network Q′ for the next state s′ and the action a′ chosen by the agent according to the policy μ′ represents the expected reward of taking the optimal action in the next state;

[0110] Updated actor network:

[0111]

[0112] in, It calculates the policy gradient, representing the expected cumulative reward with respect to the policy parameters θ. μ gradient, It is the gradient with respect to action a, given by the parameters θ of the critic network. Q The decision is used to measure how quickly the critic network's evaluation of action 'a' changes with state 's'. The parameter θ is related to the policy μ. μ The gradient is used to measure how quickly the probability of a policy choosing action a in a given state s changes with the policy parameters.

[0113] Update the target network:

[0114] θ′ i ←τθ i +(1-τ)θ′ i ′;

[0115] Where, θ i These are the parameters of the original network, θ′ i These are the parameters of the target network, and τ is the update rate;

[0116] The model is trained using historical data. For each state, the model tries different actions and updates its value estimates for different actions based on the rewards from environmental feedback. At the same time, a greedy algorithm is introduced when selecting an action at each step, which explores randomly with a certain probability ε, i.e., the ε-greedy strategy, to improve the quality of the model's solution.

[0117] Specifically, the greedy algorithm used by the scheduling host is an algorithm that makes the best or optimal choice in the current state at each step, hoping to lead to a globally optimal solution. It is often used in conjunction with an ε-greedy strategy, where ε is the exploration probability, i.e., the probability that the agent will randomly choose an action. When ε is 0, the agent chooses actions entirely according to the greedy strategy, always selecting the action with the highest Q-value in the current state. As training progresses, ε gradually decreases, and the agent relies more on the learned strategy than on random exploration. Introducing a greedy algorithm can improve the quality of scheduling decisions, such as... Figure 3 As shown, the specific method of the ε-greedy strategy is as follows:

[0118] First, determine the ε-greedy policy. In the early stages of training, set a large ε value (e.g., 0.9) to allow the agent to randomly select actions in most cases, exploring more of the state-action space. As training progresses, gradually decrease the value of ε (e.g., decrease by 0.1 each iteration until it approaches 0), allowing the agent to rely more on the learned policy. Then, apply the ε-greedy policy simultaneously during the training and scheduling of the model. Regularly evaluate the model's performance at different ε values, including metrics such as success rate, average reward, and variance. Adjust the ε value based on the model's performance at different ε values. If the model's performance starts to decline at a certain ε value, it may be because it is over-relying on the learned policy and ignoring new exploration opportunities. In this case, the ε value can be appropriately increased to allow the model to start exploring again. After the model training is complete, use a greedy algorithm (i.e., ε is close to 0) to guide decision-making in practical applications. At this point, the agent will mainly rely on the learned policy rather than randomly selecting actions.

[0119] S4: Deploy the trained model into the actual open-pit mine unmanned truck dynamic scheduling system for real-time scheduling;

[0120] S5: Evaluate the model's performance based on the new weight reward function generated by the reward update module, and repeatedly train the model to iteratively optimize it in order to achieve real-time dynamic scheduling.

[0121] Specifically, the reward update module employs an adaptive reward adjustment mechanism, which adjusts the reward weights based on model performance and environmental feedback. The adaptive reward adjustment method of the reward update module is as follows:

[0122] S1: The reward update module initializes the reward function, assigning an initial weight ω to each metric in the function. i ;

[0123] S2: Start training the scheduling model and use the initialized reward function to guide the learning process;

[0124] S3: Regularly evaluate the model's performance during training, analyze the model's performance on different metrics, and identify which metrics have a significant impact on model performance.

[0125] S4: Adjust the weights ω of the corresponding indicators in the reward function based on the model's performance on different metrics. i The weight update formula is:

[0126]

[0127] Where λ is the learning rate for weight updates, It is the reward function with respect to the weight ω i The gradient;

[0128] S5: Retrain the model using the updated reward function to make the model pay more attention to metrics that have a greater impact on performance;

[0129] S6: Repeat steps S3 to S5 to continuously iterate and optimize the reward function and model performance.

[0130] In this way, the reward function can be automatically updated based on the model's performance on different metrics, making the model more flexible in adapting to environmental changes and improving scheduling efficiency.

[0131] This invention also provides a dynamic scheduling method for unmanned trucks in open-pit mines based on reinforcement learning, such as... Figure 4 As shown, firstly, the vehicle-mounted sensors and terminals assess the real-time status of the unmanned mining truck and feed this status information back to the dynamic scheduling system. Secondly, the fault warning module in the dynamic scheduling system analyzes and calculates the truck's status information, predicts the possible fault types and occurrence times, sets alarm thresholds, and triggers an alarm when these thresholds are exceeded, feeding back the information to the scheduling processing center. The scheduling processing center optimizes the scheduling of all trucks that meet the requirements, constructs a scheduling model based on reinforcement learning, trains it using historical data, employs a greedy algorithm to improve the quality of the solution, uses an adaptive reward adjustment mechanism, and repeatedly trains the scheduling model to obtain a better solution. Simultaneously, the data storage module stores and analyzes the data generated by the system to build a database. Finally, the communication module facilitates data and scheduling command transmission between the modules of the dynamic scheduling system, and between the dynamic scheduling system and the unmanned mining truck and electric shovel, achieving real-time dynamic scheduling of the unmanned mining truck.

[0132] The specific scheduling method is as follows: First, based on the signal data detected by the vehicle-mounted sensors, if the pressure value detected by the mining truck's pressure sensor is lower than the preset empty threshold and remains so for a period of time, the mining truck is determined to be in an empty state and this information is uploaded to the dynamic scheduling system. The dynamic scheduling system updates the mining truck's status to empty and adds all empty mining trucks to the scheduleable list. The dynamic scheduling system uses a trained reinforcement learning scheduling model to select the optimal action based on the current state and historical experience, sends optimized loading task allocation instructions to the mining trucks, and schedules them to their respective current optimal loading points, determining the loading coordinates of the mining trucks. When the mining truck meets the loading requirements, the dynamic scheduling system updates the mining truck's status to loading. At this point, the mining truck and the electric shovel work together. The dynamic scheduling system can adjust the electric shovel's loading speed according to the real-time status of the mining truck until the mining truck's pressure sensor value is greater than or equal to the preset full-load threshold, and the speed sensor value is not 0 or the acceleration sensor value is not 0. At this point, the dynamic scheduling system updates the mining truck's status to transportation.

[0133] In the dynamic scheduling system, the scheduling processing host uses a reward update module to adaptively adjust the reward mechanism based on the real-time status and historical performance of the mining trucks, thus optimizing the dynamic scheduling system's decisions in real time. For mining trucks in transport status, the dynamic scheduling system issues unloading commands, publishes the optimal unloading point for each truck, and schedules the trucks to unload. After determining the destination, the unmanned mining trucks proceed to the target unloading point according to the path planning algorithm equipped on the onboard terminal. Speed ​​sensors and GPS transmit the truck's movement status in real time. When the truck stops at the designated unloading coordinates, unloading begins, and the dynamic scheduling system updates the truck's status to unloading. As the pressure sensor readings gradually decrease until the empty-load requirement is met, the dynamic scheduling system updates the truck's status to empty. The dynamic scheduling system then initiates a new round of scheduling based on new production demands and the truck's location.

[0134] Furthermore, if the fault warning module predicts a potential malfunction in the mining truck and issues an alarm, the dynamic dispatch system immediately updates the truck's status to faulty and removes it from the list of dispatchable trucks. It then issues a dispatch command to the repair station. The truck receives the command, stops the alarm, completes its current task, and proceeds to the repair station. Upon completion of repairs, the fault warning module uploads completion information, the dynamic dispatch system updates the truck's status to empty, and it is re-added to the dispatchable list. Similarly, when the fuel level sensor or battery level sensor detects that the remaining fuel or battery level is less than or equal to the set charging value, the vehicle terminal transmits this information to the dynamic dispatch system via the network. The dynamic dispatch system then updates the truck's status to charging, removes it from the list of dispatchable trucks, and issues a dispatch command to charge the truck. The truck receives the command, completes its current task, and proceeds to the charging station / refueling point. Once the fuel or battery level is fully charged, the vehicle terminal uploads completion information, the dynamic dispatch system updates the truck's status to empty, and it is re-added to the dispatchable list, ready for the next operation.

[0135] Throughout the process, the data from the dynamic scheduling system and the status information of the mining trucks are displayed in real time on the visual human-computer interaction platform.

[0136] Through this process, the dynamic scheduling system can effectively schedule unmanned mining trucks, improve transportation efficiency in the mining area, reduce empty runs and waiting time, and flexibly adjust scheduling strategies according to real-time production and road conditions to adapt to the dynamic environment of the mining area.

[0137] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0138] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any minor modifications, equivalent substitutions, and improvements made to the above embodiments based on the technical essence of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic scheduling system for unmanned trucks in open-pit mines based on reinforcement learning, characterized in that, It includes vehicle-mounted sensors, vehicle-mounted terminals, and a dynamic dispatching system. The vehicle-mounted sensors are connected to the vehicle-mounted terminals, and the vehicle-mounted terminals are connected to the dynamic dispatching system via a network. The dynamic scheduling system includes a scheduling processing center, a data storage module, a fault early warning module, a communication module, and a visual human-computer interaction platform. The scheduling processing center, data storage module, and fault early warning module are connected to each other in pairs, and the three are connected to the visual human-computer interaction platform through the communication module. The scheduling and processing center includes a scheduling and processing host and a reward update module. The reward update module is integrated into the scheduling and processing host. The scheduling and processing host contains a high-precision 3D map and performs dynamic scheduling of mining cards based on reinforcement learning and greedy algorithms. The reward update module adopts an adaptive reward adjustment mechanism. By analyzing model performance indicators and adjusting reward weights, it feeds the updated reward function back to the scheduling and processing host, repeatedly trains the scheduling model, and realizes the system's dynamic scheduling and self-learning functions. The data storage module is used to store and analyze historical data, scheduling data, and fault data acquired and generated by the system, and to build a database for the scheduling processing host to learn from. The fault warning module is used to predict and diagnose the faults of mining trucks. By analyzing the status information of the mining trucks transmitted by the vehicle terminal, it predicts the possible fault types and occurrence times of the mining trucks, sets alarm thresholds, and when the thresholds are exceeded, the fault warning module will issue an alarm. The dispatch processing host will issue a command to the mining truck to go to the maintenance station for repair. At the same time, the fault warning module will transmit the analyzed real-time vehicle status, specific data and maintenance information to the visual human-machine interaction platform. The communication module connects to the 5G network covering the entire open-pit mine, and is used for the transmission of data, information and scheduling commands for the entire system; The visual human-computer interaction platform displays a real-time 3D map of the mine and the mine card data input by the system. It also provides a manual intervention mode. In case of an emergency, the operator can set up a temporary plan and make a voice broadcast to the outside world. The method by which the scheduling processing host dynamically schedules mining cards is as follows: S1: Define the state space S and action space A of the scheduling problem; S2: Select the algorithm and gradient to build the reinforcement learning model; A reinforcement learning model consists of an agent, environment, state, action, reward, and policy; among which, the reward function... It includes multiple metrics related to the scheduling objective, each with an initial weight. Initialize the reward function: ； in, It's about transportation efficiency. This is the average waiting time. It is the average driving distance. It's the maintenance cost. It is a safety indicator; Strategy It is a function that, based on the current state Select Action The policy function is represented as a neural network: ； in, These are the parameters of the policy network; The goal is to maximize long-term cumulative rewards while ensuring efficient operation of the mining cards and reducing maintenance costs. The objective function can be expressed as: ； in, It's a strategy. It is a trajectory. It is time state, It is time The action, It is time The reward It is a discount factor. It is the time step; For a continuous action space, the DDPG algorithm is chosen, and its policy gradient is: ； in, It is a state distribution. It is the state-action value evaluated by the critic network; ； ； in, and These are the parameters for the critic network and the actor network, respectively. It is the learning rate; S3: Use historical data to train the model and introduce a greedy algorithm to improve the quality of scheduling decisions; First, initialize the DDPG model; second, collect experience; then update the network to stabilize the learning process; repeat the process of collecting experience and updating the network to continuously iterate and optimize the actor network and critic network. Updated commentator network: ； in, It is the network of critics regarding the state and actions The output, It's an instant reward. It is a discount factor. It is the target commentator network For the next state And the agent according to the policy Selected action The output; Updated actor network: ； in, It calculates the policy gradient. It's about actions. The gradient is determined by the parameters of the commentator network. Decide, It's about strategy. parameters The gradient; Update the target network: ； in, These are the parameters of the original network. These are the parameters of the target network. It is the update rate; The model is trained using historical data, and a greedy algorithm is introduced at each step when selecting an action, based on probability. To conduct random exploration, i.e. -Greedy strategy improves the quality of model solutions; S4: Deploy the trained model into the actual open-pit mine unmanned truck dynamic scheduling system for real-time scheduling; S5: Evaluate the model's performance based on the new weight reward function generated by the reward update module, and repeatedly train the model to iteratively optimize it in order to achieve real-time dynamic scheduling; The adaptive reward adjustment method of the reward update module is as follows: S1: The reward update module initializes the reward function, assigning an initial weight to each metric in the function. ; S2: Start training the scheduling model and use the initialized reward function to guide the learning process; S3: Regularly evaluate the model's performance during training, analyze the model's performance on different metrics, and identify which metrics have a significant impact on model performance. S4: Adjust the weights of the corresponding metrics in the reward function based on the model's performance on different metrics. The weight update formula is: ； in, It is the learning rate for weight updates. It is the reward function with respect to the weights The gradient; S5: Retrain the model using the updated reward function to make the model pay more attention to metrics that have a significant impact on performance; S6: Repeat steps S3 to S5 to continuously iterate and optimize the reward function and model performance.

2. The dynamic scheduling system for unmanned trucks in open-pit mines based on reinforcement learning according to claim 1, characterized in that, The vehicle-mounted sensors include one or more of the following: lidar, IMU, binocular camera, speed sensor, acceleration sensor, temperature sensor, pressure sensor, fuel level sensor, battery level sensor, and GPS. The vehicle-mounted terminal has a built-in processor and is equipped with global and local path planning algorithms.

3. The dynamic scheduling system for unmanned trucks in open-pit mines based on reinforcement learning according to claim 2, characterized in that, The vehicle-mounted terminal has six real-time statuses for mining trucks: empty, loaded, transporting, unloading, faulty, and charging. The specific method for determining the six real-time statuses of the mining trucks is based on signal data detected by the vehicle-mounted sensors and combined with the fault warning module: No-load condition judgment: The pressure value detected by the pressure sensor is less than the preset no-load threshold, and this state continues for a period of time; Loading status judgment: The pressure value detected by the pressure sensor increases at a certain rate, and the speed value detected by the speed sensor is 0; Transportation status judgment: The pressure value detected by the pressure sensor is greater than or equal to the preset full load threshold, and the speed value detected by the speed sensor is not 0 or the acceleration sensor detects a value that is not 0. Unloading status determination: The pressure value detected by the pressure sensor decreases at a certain rate; Fault status determination: determined by the fault early warning module; Charging status judgment: The remaining fuel or remaining power detected by the fuel level sensor or power sensor is less than or equal to the set charging value.

4. The dynamic scheduling system for unmanned trucks in open-pit mines based on reinforcement learning according to claim 1, characterized in that, The The specific method of the -greedy strategy is as follows: In the early stages of training, set up a large [system / structure]. The value gradually decreases as training progresses. The value; and then applied while training the scheduling model. -greedy strategy; periodically evaluate the model in different situations. Performance at different values, depending on the model. Performance at this value, adjustment The magnitude of the value; after the model is trained, a greedy algorithm is used to guide decision-making in practical applications.

5. A dynamic scheduling method for unmanned trucks in open-pit mines based on reinforcement learning, characterized in that, The system described in any one of claims 1-4 includes a fault warning module that analyzes and calculates the status information of the mining truck, predicts the possible fault types and occurrence times of the mining truck, sets an alarm threshold, and when the threshold is exceeded, the fault warning module issues an alarm and sends feedback to the scheduling processing center. The scheduling processing center optimizes the scheduling of all mining trucks that meet the requirements, constructs a scheduling model based on reinforcement learning, trains it using historical data, improves the quality of the solution using a greedy algorithm, adopts an adaptive reward adjustment mechanism, and repeatedly trains the scheduling model to obtain a better solution. The data storage module stores and analyzes the data generated by the system and constructs a database. Finally, the communication module enables data and instruction transmission between the modules of the dynamic scheduling system and between the dynamic scheduling system and the unmanned mining truck and electric shovel, realizing real-time dynamic scheduling of the unmanned mining truck.

Images