A Deep Reinforcement Learning-Based Integrated Scheduling Optimization Method for Unmanned Mining Operations

Abstract

This invention discloses a deep reinforcement learning-based comprehensive scheduling optimization method for underground unmanned driving, belonging to the field of underground unmanned driving technology. It includes the following steps: Step S1. Constructing a hierarchical underground unmanned driving scheduling optimization framework; Step S2. Obtaining underground roadway topology maps, vehicle status information, and production task information at the global static scheduling layer; Step S3. Implementing a multi-agent deep reinforcement learning model at the local dynamic collaboration layer; Step S4. Receiving the collaborative decision results from the upper layer at the single-vehicle trajectory planning layer; Step S5. Iteratively executing steps S2 to S4. This deep reinforcement learning-based comprehensive scheduling optimization method for underground unmanned driving, by constructing a hierarchical framework of global static scheduling – local dynamic collaboration – single-vehicle trajectory planning, and employing multi-objective evolutionary algorithms, multi-agent deep reinforcement learning, and model predictive control at each layer, achieves multi-objective collaborative optimization of underground unmanned driving fleets in complex dynamic environments.

Description

Technical Field

[0001] This invention relates to a deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations, belonging to the field of unmanned underground mining technology. Background Technology

[0002] Mine transportation refers to the transportation of materials, personnel, equipment, and ore during the mining process. Mine transport vehicles operate frequently, transport large volumes of goods, and travel long distances, making them an important part of the mining production system. Due to the harsh conditions, dim lighting, and obstructed visibility of transport vehicle drivers in mines, they are considered high-risk work environments, and safety is the primary consideration. With the continuous progress of society, labor costs are becoming increasingly high, and fewer and fewer people are willing to work in the harsh environment of mines. As a result, unmanned driving technology for underground mining has been applied.

[0003] Unmanned underground driving is a core component of smart mine construction. Its scheduling optimization problem is characterized by multiple objectives, strong constraints, and high dynamism. Existing traditional scheduling methods for underground transportation mostly adopt centralized path planning and traffic light control, lacking dynamic coordination mechanisms between vehicles. When traffic density increases, it is easy to cause intersection congestion and roadway deadlock, resulting in a sharp decline in transportation efficiency. Furthermore, existing methods usually simplify multi-objective problems into single-objective weighted summation, making it difficult to obtain a truly Pareto optimal solution set. Moreover, fixed weights cannot adapt to dynamic changes in underground working conditions. When loading point failures, vehicle breakdowns, or roadway temporary closures occur, the pre-set scheduling scheme cannot respond quickly, resulting in poor system robustness. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a deep reinforcement learning-based integrated scheduling optimization method for downhole unmanned driving.

[0005] The technical solution provided by this invention is as follows: A deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations, characterized by comprising the following steps: Step S1. Construct a hierarchical underground unmanned driving scheduling optimization framework, which includes a global static scheduling layer, a local dynamic coordination layer, and a single vehicle trajectory planning layer. Step S2. Obtain the underground roadway topology map, vehicle status information, and production task information in the global static scheduling layer, construct a multi-objective optimization model, and solve it using a multi-objective evolutionary algorithm to generate a globally conflict-free initial task allocation scheme and static path planning results. Step S3. In the local dynamic collaboration layer, based on the multi-agent deep reinforcement learning model, each autonomous vehicle is used as an agent. The local observation information of the vehicle is used as input, and the collaborative decision-making actions of the vehicle in the local area are output to realize the dynamic optimization of multi-vehicle collaboration. Step S4. Receive the upper-layer collaborative decision-making results at the single-vehicle trajectory planning layer, combine the vehicle dynamics model and real-time perception information, use model predictive control method to generate a smooth, safe, and trackable desired trajectory, and send the control command to the vehicle actuator. Step S5. Iteratively execute steps S2 to S4 to achieve multi-objective integrated scheduling optimization of the underground unmanned vehicle fleet.

[0006] Furthermore, in step S1, The aforementioned global static scheduling layer is a server or cloud platform running in the mine scheduling center. It operates on a minute- or hour-level cycle and is responsible for long-term task allocation and global path planning. The inputs to this layer are the underground roadway topology map, vehicle status information, and production task information. The outputs are the task sequence and initial global path for each vehicle. Among them, the vehicle status information includes the location, battery level, and task status of each vehicle; the production task information includes the loading point status, unloading point status, and production plan. The local dynamic collaboration layer is deployed on edge computing nodes in key downhole areas. It operates on a cycle of seconds or hundreds of milliseconds and is responsible for real-time collaborative decision-making in local areas. This layer receives the task sequence and initial global path of each vehicle from the global static scheduling layer and, in conjunction with the local observation information reported by the vehicles in real time, dynamically adjusts the vehicle's path selection and passage sequence to cope with dynamic events such as local congestion and equipment failure. The single-vehicle trajectory planning layer runs on the onboard controller of each autonomous vehicle, executing in millisecond cycles. It is responsible for converting the upper-layer decisions into smooth and safe actual trajectories. This layer receives path selection and passage timing from the local dynamic coordination layer, and combines it with real-time data from onboard sensors to generate specific throttle, brake, and steering commands through model predictive control, which are then sent to the vehicle's actuators.

[0007] Further, in step S2, the construction of the multi-objective optimization model includes: a. Define the task sequence assigned to each vehicle, as well as the departure time and driving route corresponding to each task; b. Optimization Objectives: The multi-objective optimization model aims to minimize total transportation time, minimize total energy consumption, and maximize equipment utilization balance. Minimize total transportation completion time; minimize total energy consumption, with the energy consumption model constructed based on path length, gradient, and load factors; maximize equipment utilization balance, measured by the variance of all vehicle task times, with the objective of minimizing this variance. c. Constraints: These include lane access constraints, vehicle battery power constraints, and loading / unloading point capacity constraints. Lane access constraints: The number of vehicles traveling in opposite directions within the same lane at any given time must not exceed a safety limit, typically modeled using time windows. Vehicle battery power constraints: The total energy consumption of each vehicle completing its assigned task must not exceed its current battery power's safety threshold. Loading / unloading point capacity constraints: At most one vehicle is allowed to enter a loading or unloading point at any given time, taking loading and unloading time into account.

[0008] Furthermore, in step S2, the multi-objective evolutionary algorithm adopts a non-dominated sorting genetic algorithm with an elitist strategy. Through non-dominated sorting and crowding distance calculation, the Pareto optimal solution set is obtained, which is then used by the scheduler to select an execution plan based on the on-site working conditions.

[0009] Furthermore, in step S3, In the local dynamic cooperation layer, multi-agent deep reinforcement learning is used to achieve vehicle cooperation within the region. The specific implementation steps include: A. Multi-agent system modeling: Each autonomous vehicle is an agent; the local observation information of each agent includes: its own position, speed, current battery level, lane occupancy within 50 meters ahead, relative distance and speed of the three neighboring vehicles, traffic light status at the intersection ahead, and its own waiting time; collaborative decision-making actions include desired speed and steering decisions at the intersection, or combinations of discrete actions; B. Reward Function Design: Comprehensively consider reward signals for multiple objectives; give positive rewards for completing a journey or passing through an intersection; penalize stopping or slow driving; give large negative rewards when too close to other vehicles or lane walls; terminate the journey and give a very large negative reward when a collision occurs; penalize according to the rate of change of acceleration or the degree of speed deviation from the economic range; encourage vehicles to take turns passing through intersections to avoid one side waiting for a long time; C. Algorithm Implementation: A multi-agent deep deterministic policy gradient algorithm is adopted, using a centralized training and distributed execution architecture. During the training phase, global information is used to evaluate the joint policy, while during the execution phase, each agent makes independent decisions based solely on its local observations. In the simulation environment, the experience trajectories (state, action, reward, next state) of all agents are stored in an experience replay pool. Each agent has an Actor network and a Critic network. During training, the Critic network can acquire the states and actions of all agents, thereby accurately evaluating the merits of the joint policy and guiding the Actor network updates. After training, each agent outputs collaborative decision actions (desired speed and steering) through its local Actor network based solely on its local observation information. This network is lightweight and can run in real-time on the vehicle controller. D. Conflict Resolution Mechanism: Collaborative decision-making includes a conflict resolution mechanism based on the time window method. Vehicles send passage requests to the local collaboration layer, which allocates passage time windows through a multi-agent deep reinforcement learning model to avoid deadlocks at lanes and intersections. Specifically, in the local dynamic collaboration layer, each edge node maintains a resource (lane segment, intersection) occupancy schedule within its jurisdiction. After the multi-agent deep reinforcement learning model outputs the vehicle's collaborative decision action, the edge node executes the following process: Request Received: The vehicle sends a passage request to the edge node, including its estimated time of entry into the resource and its estimated time of departure; Conflict detection: Edge nodes query whether the resource has been booked by other vehicles within the time period. If there is overlap, a conflict is detected. Time window allocation: If the multi-agent deep reinforcement learning model has implicitly resolved the conflict through negotiation, the edge node can directly approve the request and record the time window; if the conflict still exists, the edge node runs a priority-based conflict resolution algorithm. The priority is determined according to the urgency of the vehicle task and the waiting time. Vehicles with higher priority get the original time window, while vehicles with lower priority are required to adjust their speed or wait and re-apply for a new time window. The new time window is calculated as the earliest available time after the current occupation ends. Feedback and Adjustment: Edge nodes feed back the allocation results (pass / wait / adjust speed) to the vehicle. The vehicle adjusts its behavior based on the feedback. If waiting is required, it slows down or stops until a new time window is obtained.

[0010] Furthermore, in step S4, At the single-vehicle trajectory planning layer, model predictive control is used to generate a fine trajectory. Within each control cycle, based on the current state and the vehicle dynamics model, the control sequence in the future finite time domain is continuously optimized, and obstacle avoidance constraints are constructed using the safety potential field method to generate a trajectory that satisfies the optimal balance of safety, comfort, and energy consumption.

[0011] Furthermore, the specific steps include: Establish a simplified bicycle model or kinematic model to describe the relationship between the vehicle's position, heading angle, velocity, and acceleration; Predictive Model and Optimization Objective: In each control period t, the model predictive control solves the following finite-time optimization problem: ; Where H is the prediction time domain; x is the state vector, including vehicle position, heading angle, and velocity; u is the control input, including acceleration and turning angle; is the reference path point provided by the upper coordination layer; Q is the state error weight matrix; R is the control input weight matrix; and P is the terminal state error weight matrix. Safety constraints, which incorporate obstacle avoidance constraints into optimization problems, are typically constructed using the safety potential field method, defining the potential field function as follows: ; in This is the vehicle's current location; For obstacle index; indicating the first obstacle in the environment. One obstacle; This is the safe distance threshold; Potential field gain coefficient; By incorporating the potential field penalty term into the objective function of MPC or as a hard constraint (requiring the distance to always be greater than the safety threshold), the optimal control sequence is obtained by solving a quadratic programming or nonlinear programming problem, and the first control variable is applied to the vehicle.

[0012] Furthermore, the hierarchical framework in step S1 also includes a cloud-based global optimization layer, an edge region collaboration layer, and a vehicle-side real-time control layer, which correspond to computation and decision-making tasks at different time scales. The cloud runs a multi-objective evolutionary algorithm for long-cycle task planning, the edge runs a multi-agent deep reinforcement learning model for regional collaboration, and the vehicle-side runs a model predictive control for trajectory tracking.

[0013] Furthermore, the method also includes establishing a downhole digital twin model, performing simulation verification and iterative optimization of the scheduling optimization strategy in a virtual environment, and deploying the optimized strategy to the actual downhole unmanned driving system.

[0014] Furthermore, the multi-objective integrated scheduling optimization method simultaneously considers production efficiency, safety, energy consumption, and economic indicators. By adjusting the reward function weights in the multi-objective evolutionary algorithm and the multi-agent deep reinforcement learning model, it achieves preference scheduling under different working conditions.

[0015] The beneficial effects of this invention are: This invention constructs a hierarchical framework consisting of a global static scheduling layer, a local dynamic collaboration layer, and a single-vehicle trajectory planning layer. At each layer, it employs the National Strategy-Independent Gaining Algorithm (NSGA-II), the Multi-Agent Deep Reinforcement Learning Algorithm (MADDPG), and Model Predictive Control (MPC) to achieve multi-objective collaborative optimization of an unmanned underground vehicle fleet in complex dynamic environments. First, at the global static scheduling layer, the Pareto optimal solution set is solved by a multi-objective evolutionary algorithm, which overcomes the shortcomings of traditional single-objective weighted summation that cannot take into account production efficiency (total transportation time), energy consumption (total energy consumption) and equipment balance (equipment utilization balance), and provides a scientific basis for scheduling decisions. Second, at the local dynamic collaboration layer, based on the multi-agent deep deterministic policy gradient algorithm, a centralized training and distributed execution architecture is adopted, enabling each vehicle to make collaborative decision-making actions (desired speed and steering) in real time by relying only on local observation information (its own position, speed, current battery level, occupancy status of the lane ahead, information of nearby vehicles, traffic light status at intersections, and waiting time). This solves the technical problems of strong communication dependence and slow dynamic response in traditional centralized scheduling. At the same time, through multi-agent autonomous negotiation of passage time windows, combined with a priority-based conflict resolution mechanism, the risk of lane deadlock and intersection congestion is eliminated from the root. Third, at the single-vehicle trajectory planning layer, model predictive control that integrates safety potential field constraints is adopted. Based on the vehicle dynamics model, the control sequence in the future finite time domain is continuously optimized to generate a smooth, safe, and dynamically constrained desired trajectory, ensuring the accurate execution of upper-level collaborative decisions. Fourth, through iterative execution of the hierarchical framework and simulation verification of the downhole digital twin model, the scheduling optimization strategy has been continuously iteratively optimized, enabling rapid response to dynamic events such as loading point failures, vehicle breakdowns, and temporary roadway closures, significantly improving the system's robustness. Detailed Implementation

[0016] The specific embodiments of the present invention will be described in detail below: A deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations includes the following steps: Step S1. Construct a hierarchical underground unmanned driving scheduling optimization framework, which includes a global static scheduling layer, a local dynamic coordination layer, and a single vehicle trajectory planning layer.

[0017] The global static scheduling layer is a server or cloud platform running in the mine's scheduling center. It operates on a minute- or hourly cycle and is responsible for long-term task allocation and global path planning. The inputs to this layer are an underground roadway topology map, vehicle status information, and production task information. The outputs are the task sequence and initial global path for each vehicle. Vehicle status information includes the location, battery level, and task status of each vehicle; production task information includes loading point status, unloading point status, and production plan.

[0018] The local dynamic collaboration layer is deployed on edge computing nodes in key underground areas. It operates on a cycle of seconds or hundreds of milliseconds and is responsible for real-time collaborative decision-making in local areas. This layer receives the task sequence and initial global path of each vehicle from the global static scheduling layer and, in conjunction with the local observation information reported by the vehicles in real time, dynamically adjusts the vehicle's path selection and passage timing to cope with dynamic events such as local congestion and equipment failure.

[0019] The single-vehicle trajectory planning layer runs on the onboard controller of each autonomous vehicle, executing in millisecond cycles. It is responsible for converting the upper-level decisions into a smooth and safe actual trajectory. This layer receives path selection and passage timing from the local dynamic coordination layer, and combines it with real-time data from onboard sensors to generate specific throttle, brake, and steering commands through model predictive control, which are then sent to the vehicle's actuators.

[0020] Step S2. Obtain the underground roadway topology map, vehicle status information, and production task information in the global static scheduling layer, construct a multi-objective optimization model, and solve it using a multi-objective evolutionary algorithm to generate a globally conflict-free initial task allocation scheme and static path planning results.

[0021] In the global static scheduling layer, the following operations are performed: A. Data Acquisition: The system retrieves the shift's production task list from the mine production management system, the current online vehicle status information from the vehicle management system, and the underground roadway topology map from the environmental perception system. Vehicle status information includes the location, battery level, and task status of each vehicle; production task information includes loading point status, unloading point status, and production plan; and the underground roadway topology map includes roadway traffic status.

[0022] B. Construction of a multi-objective optimization model a. Define the task sequence assigned to each vehicle, as well as the departure time and driving route corresponding to each task.

[0023] b. Optimization Objectives: The multi-objective optimization model aims to minimize total transportation time, minimize total energy consumption, and maximize equipment utilization balance. Minimize total transportation completion time; minimize total energy consumption, with the energy consumption model constructed based on path length, gradient, and load factors; maximize equipment utilization balance, measured by the variance of all vehicle task times, with the objective of minimizing this variance.

[0024] c. Constraints: These include lane access constraints, vehicle battery power constraints, and loading / unloading point capacity constraints. Lane access constraints: The number of vehicles traveling in opposite directions within the same lane at any given time must not exceed a safety limit, typically modeled using time windows. Vehicle battery power constraints: The total energy consumption of each vehicle completing its assigned task must not exceed its current battery power's safety threshold. Loading / unloading point capacity constraints: At most one vehicle is allowed to enter a loading or unloading point at any given time, taking loading and unloading time into account.

[0025] C. Multi-objective evolutionary algorithm solution: The multi-objective evolutionary algorithm adopts a non-dominated sorting genetic algorithm with an elitist strategy. Through non-dominated sorting and crowding distance calculation, the Pareto optimal solution set is obtained, which is used by the scheduler to select the execution plan according to the on-site working conditions.

[0026] The specific steps include: a. Real number encoding is used, with chromosome length equal to the number of vehicles × the maximum number of tasks. Each gene position represents the task number or empty task performed by the vehicle.

[0027] b. Randomly generate an initial population, with a population size of 100.

[0028] c. Perform Pareto stratification on individuals in the population based on the objective function value.

[0029] d. Calculate the crowding distance between individuals within the same Pareto layer to maintain solution diversity.

[0030] e. Use tournament selection, simulated binary crossover, and polynomial mutation to generate offspring.

[0031] f. Merge the parent and offspring generations, perform non-dominated sorting and crowding selection, and generate a new generation of population.

[0032] g. After 200 iterations, stop and output multiple solutions on the Pareto front. The scheduler can select one of these solutions as the initial global scheduling plan based on the current production priorities (such as choosing the shortest time solution to increase output or choosing the lowest energy consumption solution when power is tight).

[0033] Step S3. In the local dynamic collaboration layer, based on the multi-agent deep reinforcement learning model, each autonomous vehicle is used as an agent. The local observation information of the vehicle is used as input, and the collaborative decision-making actions of the vehicle in the local area are output to realize the dynamic optimization of multi-vehicle collaboration.

[0034] In the local dynamic cooperation layer, multi-agent deep reinforcement learning is used to achieve vehicle cooperation within the region. The specific implementation steps include: A. Multi-Agent System Modeling: Each autonomous vehicle acts as an agent. Each agent's local observation information includes: its own position (coordinates relative to the local map), speed, current battery level, lane occupancy status within 50 meters ahead (perceived through vehicle-to-vehicle communication or roadside unit sensing), relative distance and speed of the three nearest vehicles, traffic light status at the upcoming intersection (red / green / yellow), and its own waiting time. Cooperative decision-making actions include desired speed and steering decisions at the intersection, or combinations of discrete actions.

[0035] B. Reward Function Design: The reward signal comprehensively considers multiple objectives. Positive rewards are given for completing a journey or passing through an intersection; penalties are given for stopping, waiting, or slow driving; large negative rewards are given for getting too close to other vehicles or lane walls; collisions result in termination and a very large negative reward; penalties are given based on the rate of change of acceleration or the degree of speed deviation from the economic range; vehicles are encouraged to alternate at intersections to avoid one side waiting for an extended period.

[0036] C. Algorithm Implementation: A multi-agent deep deterministic policy gradient algorithm is employed, using a centralized training and distributed execution architecture. During training, global information is used to evaluate the joint policy, while during execution, each agent makes independent decisions based solely on its local observations. In the simulation environment, the experience trajectories (state, action, reward, next state) of all agents are stored in an experience replay pool. Each agent has an Actor network (policy network) and a Critic network (evaluation network). During training, the Critic network acquires the states and actions of all agents, accurately evaluating the merits of the joint policy and guiding the Actor network updates. After training, each agent outputs cooperative decision actions (desired speed and steering) through its local Actor network based solely on its local observations. This lightweight network can run in real-time on the vehicle controller.

[0037] D. Conflict Resolution Mechanism: Collaborative decision-making includes a conflict resolution mechanism based on time windows. Vehicles send passage requests to the local collaboration layer, which allocates passage time windows through a multi-agent deep reinforcement learning model to avoid deadlocks at lanes and intersections. Specifically, in the local dynamic collaboration layer, each edge node maintains a resource (lane segment, intersection) occupancy schedule within its jurisdiction. After the multi-agent deep reinforcement learning model outputs the vehicle's collaborative decision action, the edge node executes the following process: Request Received: The vehicle sends a passage request to the edge node, including its expected time of entering the resource and its expected time of leaving.

[0038] Conflict detection: Edge nodes query whether the resource has been booked by other vehicles within the time period. If there is overlap, a conflict is detected.

[0039] Time window allocation: If the multi-agent deep reinforcement learning model has implicitly resolved the conflict through negotiation, the edge node can directly approve the request and record the time window. If the conflict still exists, the edge node runs a priority-based conflict resolution algorithm. The priority is determined according to the urgency of the vehicle task and the waiting time. Vehicles with higher priority get the original time window, while vehicles with lower priority are required to adjust their speed or wait and re-apply for a new time window. The new time window is calculated as the earliest available time after the current occupation ends.

[0040] Feedback and Adjustment: Edge nodes feed back the allocation results (pass / wait / adjust speed) to the vehicle. The vehicle adjusts its behavior based on the feedback. If waiting is required, it slows down or stops until a new time window is obtained.

[0041] Step S4. Receive the upper-layer collaborative decision-making results at the single-vehicle trajectory planning layer, combine the vehicle dynamics model and real-time perception information, use model predictive control method to generate a smooth, safe, and trackable desired trajectory, and send the control command to the vehicle actuator. At the single-vehicle trajectory planning layer, model predictive control (MPC) is used to generate a fine trajectory. Within each control cycle, based on the current state and the vehicle dynamics model, the control sequence within the finite time domain is continuously optimized. Obstacle avoidance constraints are constructed using the safety potential field method to generate a trajectory that satisfies optimal safety, comfort, and energy consumption. Specific steps include: Establish a simplified bicycle model or kinematic model to describe the relationship between the vehicle's position, heading angle, velocity, and acceleration; Predictive Model and Optimization Objective: In each control period t, the model predictive control solves the following finite-time optimization problem: ; Where H is the prediction time domain; x is the state vector, including vehicle position, heading angle, and velocity; u is the control input, including acceleration and turning angle; is the reference path point provided by the upper coordination layer; Q is the state error weight matrix; R is the control input weight matrix; P is the terminal state error weight matrix; and k is the loop index variable in the summation formula.

[0042] Safety constraints, which incorporate obstacle avoidance constraints into optimization problems, are typically constructed using the safety potential field method, defining the potential field function as follows: ; in This is the vehicle's current location; For obstacle index; indicating the first obstacle in the environment. One obstacle; This is the safe distance threshold; Potential field gain coefficient;

[0043] The potential field penalty term is added to the objective function of MPC or used as a hard constraint (requiring the distance to always be greater than the safety threshold). The optimal control sequence is obtained by solving a quadratic programming or nonlinear programming problem, and the first control variable is applied to the vehicle. Step S5. Iteratively execute steps S2 to S4 to achieve multi-objective integrated scheduling optimization of the underground unmanned vehicle fleet.

[0044] The hierarchical framework in step S1 also includes a cloud-based global optimization layer, an edge region collaboration layer, and a vehicle-side real-time control layer.

[0045] The cloud-based global optimization layer, deployed on a high-performance server or cloud platform in the mine's central dispatch center, is responsible for receiving global status data from all vehicles, equipment, and the environment; running multi-objective evolutionary algorithms to generate globally optimal task allocations and initial paths at minute-level intervals; distributing global scheduling results to each edge node; and triggering rescheduling when significant changes occur. The edge region collaboration layer consists of edge computing nodes deployed in key underground locations. Each node is responsible for a local area and has the ability to maintain high-speed, low-latency communication with vehicles within the area, run multi-agent deep reinforcement learning models, make real-time collaborative decisions for vehicles within the area in seconds, perform time window management and conflict resolution, send decision results to vehicles in real time, and continue to work for a period of time based on local cache and historical models when communication is interrupted, thus achieving a certain degree of autonomy. The vehicle-side real-time control layer is deployed on the onboard controller of each autonomous vehicle. It is responsible for running the model predictive control algorithm, generating smooth trajectories and controlling the vehicle in millisecond cycles, running the safety monitoring module to ensure emergency braking in extreme situations, communicating with edge nodes, reporting status and receiving instructions. The three layers are connected by a network, allowing data to flow bidirectionally and forming a closed-loop optimization.

[0046] The method also includes establishing a downhole digital twin model, simulating and iteratively optimizing the scheduling optimization strategy in a virtual environment, and deploying the optimized strategy to the actual downhole unmanned driving system.

[0047] The multi-objective integrated scheduling optimization method considers production efficiency, safety, energy consumption and economic indicators simultaneously. By adjusting the reward function weights in the multi-objective evolutionary algorithm and the multi-agent deep reinforcement learning model, it achieves preference scheduling under different working conditions.

[0048] It should be understood that any parts not described in detail in this specification belong to the prior art. The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements to the technical solutions of the present invention made by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations, characterized in that, It includes the following steps: Step S1. Construct a hierarchical underground unmanned driving scheduling optimization framework, which includes a global static scheduling layer, a local dynamic coordination layer, and a single vehicle trajectory planning layer. Step S2. Obtain the underground roadway topology map, vehicle status information, and production task information in the global static scheduling layer, construct a multi-objective optimization model, and solve it using a multi-objective evolutionary algorithm to generate a globally conflict-free initial task allocation scheme and static path planning results. Step S3. In the local dynamic collaboration layer, based on the multi-agent deep reinforcement learning model, each autonomous vehicle is used as an agent. The local observation information of the vehicle is used as input, and the collaborative decision-making actions of the vehicle in the local area are output to realize the dynamic optimization of multi-vehicle collaboration. Step S4. Receive the upper-layer collaborative decision-making results at the single-vehicle trajectory planning layer, combine the vehicle dynamics model and real-time perception information, use model predictive control method to generate a smooth, safe, and trackable desired trajectory, and send the control command to the vehicle actuator. Step S5. Iteratively execute steps S2 to S4 to achieve multi-objective integrated scheduling optimization of the underground unmanned vehicle fleet.

2. The deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations according to claim 1, characterized in that, In step S1 The aforementioned global static scheduling layer is a server or cloud platform running in the mine scheduling center. It operates on a minute- or hour-level cycle and is responsible for long-term task allocation and global path planning. The inputs to this layer are the underground roadway topology map, vehicle status information, and production task information. The outputs are the task sequence and initial global path for each vehicle. Among them, the vehicle status information includes the location, battery level, and task status of each vehicle; the production task information includes the loading point status, unloading point status, and production plan. The local dynamic collaboration layer is deployed on edge computing nodes in key downhole areas. It operates on a cycle of seconds or hundreds of milliseconds and is responsible for real-time collaborative decision-making in local areas. This layer receives the task sequence and initial global path of each vehicle from the global static scheduling layer and, in conjunction with the local observation information reported by the vehicles in real time, dynamically adjusts the vehicle's path selection and passage sequence to cope with dynamic events such as local congestion and equipment failure. The single-vehicle trajectory planning layer runs on the onboard controller of each autonomous vehicle, executing in millisecond cycles. It is responsible for converting the upper-layer decisions into smooth and safe actual trajectories. This layer receives path selection and passage timing from the local dynamic coordination layer, and combines it with real-time data from onboard sensors to generate specific throttle, brake, and steering commands through model predictive control, which are then sent to the vehicle's actuators.

3. The deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations according to claim 1, characterized in that... In step S2, the construction of the multi-objective optimization model includes: a. Define the task sequence assigned to each vehicle, as well as the departure time and driving route corresponding to each task; b. Optimization Objectives: The multi-objective optimization model aims to minimize total transportation time, minimize total energy consumption, and maximize equipment utilization balance. Minimize total transportation completion time; minimize total energy consumption, with the energy consumption model constructed based on path length, gradient, and load factors; maximize equipment utilization balance, measured by the variance of all vehicle task times, with the objective of minimizing this variance. c. Constraints: These include lane access constraints, vehicle battery power constraints, and loading / unloading point capacity constraints. Lane access constraints: The number of vehicles traveling in opposite directions within the same lane at any given time must not exceed a safety limit, typically modeled using time windows. Vehicle battery power constraints: The total energy consumption of each vehicle completing its assigned task must not exceed its current battery power's safety threshold. Loading / unloading point capacity constraints: At most one vehicle is allowed to enter a loading or unloading point at any given time, taking loading and unloading time into account.

4. The deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations according to claim 1, characterized in that... In step S2, the multi-objective evolutionary algorithm adopts a non-dominated sorting genetic algorithm with an elite strategy. Through non-dominated sorting and crowding distance calculation, the Pareto optimal solution set is obtained, which is then used by the scheduler to select the execution plan according to the on-site working conditions.

5. The deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations according to claim 1, characterized in that, In step S3 In the local dynamic cooperation layer, multi-agent deep reinforcement learning is used to achieve vehicle cooperation within the region. The specific implementation steps include: A. Multi-agent system modeling: Each autonomous vehicle is an agent; the local observation information of each agent includes: its own position, speed, current battery level, lane occupancy within 50 meters ahead, relative distance and speed of the three neighboring vehicles, traffic light status at the intersection ahead, and its own waiting time; collaborative decision-making actions include desired speed and steering decisions at the intersection, or combinations of discrete actions; B. Reward Function Design: Comprehensively consider reward signals for multiple objectives; give positive rewards for completing a journey or passing through an intersection; penalize stopping or slow driving; give large negative rewards when too close to other vehicles or lane walls; terminate the journey and give a very large negative reward when a collision occurs; penalize according to the rate of change of acceleration or the degree of speed deviation from the economic range; encourage vehicles to take turns passing through intersections to avoid one side waiting for a long time; C. Algorithm Implementation: A multi-agent deep deterministic policy gradient algorithm is adopted, using a centralized training and distributed execution architecture. During the training phase, global information is used to evaluate the joint policy, while during the execution phase, each agent makes independent decisions based solely on its local observations. In the simulation environment, the experience trajectories (state, action, reward, next state) of all agents are stored in an experience replay pool. Each agent has an Actor network and a Critic network. During training, the Critic network can acquire the states and actions of all agents, thereby accurately evaluating the merits of the joint policy and guiding the Actor network updates. After training, each agent outputs collaborative decision actions (desired speed and steering) through its local Actor network based solely on its local observation information. This network is lightweight and can run in real-time on the vehicle controller. D. Conflict Resolution Mechanism: Collaborative decision-making includes a conflict resolution mechanism based on the time window method. Vehicles send passage requests to the local collaboration layer, which allocates passage time windows through a multi-agent deep reinforcement learning model to avoid deadlocks at lanes and intersections. Specifically, in the local dynamic collaboration layer, each edge node maintains a resource (lane segment, intersection) occupancy schedule within its jurisdiction. After the multi-agent deep reinforcement learning model outputs the vehicle's collaborative decision action, the edge node executes the following process: Request Received: The vehicle sends a passage request to the edge node, including its estimated time of entry into the resource and its estimated time of departure; Conflict detection: Edge nodes query whether the resource has been booked by other vehicles within the time period. If there is overlap, a conflict is detected. Time window allocation: If the multi-agent deep reinforcement learning model has implicitly resolved the conflict through negotiation, the edge node can directly approve the request and record the time window; if the conflict still exists, the edge node runs a priority-based conflict resolution algorithm. The priority is determined according to the urgency of the vehicle task and the waiting time. Vehicles with higher priority get the original time window, while vehicles with lower priority are required to adjust their speed or wait and re-apply for a new time window. The new time window is calculated as the earliest available time after the current occupation ends. Feedback and Adjustment: Edge nodes feed back the allocation results (pass / wait / adjust speed) to the vehicle. The vehicle adjusts its behavior based on the feedback. If waiting is required, it slows down or stops until a new time window is obtained.

6. The deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations according to claim 1, characterized in that, In step S4 At the single-vehicle trajectory planning layer, model predictive control is used to generate a fine trajectory. Within each control cycle, based on the current state and the vehicle dynamics model, the control sequence in the future finite time domain is continuously optimized, and obstacle avoidance constraints are constructed using the safety potential field method to generate a trajectory that satisfies the optimal balance of safety, comfort, and energy consumption.

7. The deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations according to claim 6, characterized in that, The specific steps include: Establish a simplified bicycle model or kinematic model to describe the relationship between the vehicle's position, heading angle, velocity, and acceleration; Predictive Model and Optimization Objective: In each control period t, the model predictive control solves the following finite-time optimization problem: ； Where H is the prediction time domain; x is the state vector, including vehicle position, heading angle, and velocity; u is the control input, including acceleration and turning angle; is the reference path point provided by the upper coordination layer; Q is the state error weight matrix; R is the control input weight matrix; and P is the terminal state error weight matrix. Safety constraints, which incorporate obstacle avoidance constraints into optimization problems, are typically constructed using the safety potential field method, defining the potential field function as follows: ； in This is the vehicle's current location; For obstacle index; indicating the first obstacle in the environment. One obstacle; This is the safe distance threshold; Potential field gain coefficient; By incorporating the potential field penalty term into the objective function of MPC or as a hard constraint (requiring the distance to always be greater than the safety threshold), the optimal control sequence is obtained by solving a quadratic programming or nonlinear programming problem, and the first control variable is applied to the vehicle.

8. The deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations according to claim 1, characterized in that, The hierarchical framework in step S1 also includes a cloud-based global optimization layer, an edge region collaboration layer, and a vehicle-side real-time control layer, which correspond to computation and decision-making tasks at different time scales. The cloud runs a multi-objective evolutionary algorithm for long-term task planning, the edge runs a multi-agent deep reinforcement learning model for regional collaboration, and the vehicle-side runs a model predictive control for trajectory tracking.

9. The deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations according to claim 1, characterized in that, The method also includes establishing a downhole digital twin model, simulating and iteratively optimizing the scheduling optimization strategy in a virtual environment, and deploying the optimized strategy to the actual downhole unmanned driving system.

10. The deep reinforcement learning-based integrated scheduling optimization method for unmanned underground mining operations according to claim 1, characterized in that, The multi-objective integrated scheduling optimization method considers production efficiency, safety, energy consumption and economic indicators simultaneously. By adjusting the reward function weights in the multi-objective evolutionary algorithm and the multi-agent deep reinforcement learning model, it achieves preference scheduling under different working conditions.

Images