A multi-terminal vehicle dispatching system based on reinforcement learning

Abstract

This invention relates to the field of vehicle dispatching technology, specifically a multi-terminal vehicle dispatching system based on reinforcement learning. The system includes a heterogeneous data fusion module, a resource allocation module, a hierarchical reinforcement learning module, an optimization feedback module, and a human-machine collaborative control module. By collecting data on vehicle operation, work tasks, and environmental monitoring and uniformly encapsulating it into a structured dataset, the upper-level manager model generates a global dispatching instruction set based on the PPO algorithm, while the lower-level worker model outputs specific vehicle control instructions based on multi-agent reinforcement learning. The system also uses the NSGA-II algorithm to evaluate and optimize historical dispatching performance, selecting the Pareto optimal solution set to achieve continuous iteration of the dispatching strategy. The human-machine collaborative control module supports visual display and manual intervention, improving the system's adaptability and controllability in complex work scenarios.

Description

Technical Field

[0001] This invention relates to the field of vehicle dispatching technology, specifically a multi-terminal vehicle dispatching system based on reinforcement learning. Background Technology

[0002] In multi-vehicle collaborative operation scenarios such as large-scale mines, ports, airports, and industrial parks, vehicle scheduling has long been highly valued as a crucial link in ensuring operational continuity and efficient resource utilization. With the rapid development of autonomous driving and intelligent sensing technologies, traditional scheduling methods relying on human experience or static rules have been gradually replaced by systems with intelligent perception and real-time decision-making capabilities. Simultaneously, the integrated application of technologies such as multi-source sensors, industrial IoT, and digital twins provides a rich real-time data foundation for the work site, supporting the transformation of scheduling systems from "manual scheduling" to "intelligent decision-making."

[0003] Reinforcement learning, as an algorithmic framework that has demonstrated outstanding performance in intelligent control and optimal decision-making in recent years, has gradually become one of the important technical paths for intelligent scheduling systems due to its adaptive optimization capabilities in complex dynamic environments. Although some vehicle scheduling systems based on reinforcement learning have achieved initial application results, most systems are still limited to the scheduling and control of single-type vehicles and lack the ability to coordinate scheduling of heterogeneous multi-terminal operation vehicles. In practical application scenarios, the types of operation vehicles are diverse, the tasks are complexly coupled, and the scheduling environment changes frequently, urgently requiring the construction of an intelligent scheduling mechanism that can integrate multi-source data, unify scheduling strategies, and possess dynamic response capabilities. Therefore, how to effectively integrate the operational data and task information of multi-terminal operation vehicles, build a coordinated and efficient scheduling system, and achieve autonomous strategy optimization through reinforcement learning has become a key direction for the development of intelligent vehicle scheduling systems. Summary of the Invention

[0004] This invention provides a multi-terminal vehicle scheduling system based on reinforcement learning. The system adopts a layered architecture, integrates multi-source heterogeneous data, and achieves intelligent scheduling and refined control of various types of operational vehicles through near-end policy optimization and multi-agent collaborative decision-making, thereby improving mining operation efficiency and system intelligence.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A multi-terminal vehicle dispatching system based on reinforcement learning includes:

[0007] The heterogeneous data fusion module is used to collect and uniformly encapsulate vehicle operation data, task data and environmental monitoring data from the mine site, and form a structured dataset through standardized processing via protocols.

[0008] The resource allocation module extracts the boundary and time window information of the work area based on the structured dataset, and generates a map of passable areas and a sequence of work time windows by combining preset priority rules.

[0009] The hierarchical reinforcement learning module includes an upper-level manager model and a lower-level worker model. The upper-level manager model receives the structured dataset, a map of traversable areas, and a sequence of operation time windows, and generates a global scheduling instruction set. The lower-level worker model outputs vehicle control instructions based on the global scheduling instruction set and local state data.

[0010] The feedback module is optimized by using historical execution data from each terminal vehicle as decision samples. Based on the NSGA-II evolutionary algorithm, the strategy is evaluated and the scheduling scheme is reordered. The Pareto optimal solution is selected and transmitted to the upper-level manager model to participate in the global scheduling instruction set update.

[0011] Furthermore, the formation of structured datasets through protocol standardization includes:

[0012] Time synchronization and format cleaning of vehicle operation data, task information and environmental monitoring data;

[0013] Encapsulation is performed according to the preset ISO 15143-3 communication standard protocol;

[0014] The encapsulated data is packaged into a unified structure to build a structured dataset with field consistency and temporal continuity.

[0015] Furthermore, generating a passable area map and a sequence of operation time windows includes:

[0016] Based on the structured dataset, extract the regional boundary information and task start and end time parameters of the task;

[0017] By combining real-time environmental monitoring data from structured datasets, the level of traffic conflict in each work area at different time periods is assessed.

[0018] Based on preset task priority rules and traffic conflict levels, exclusive passage areas and overlapping operation areas are divided for different types of vehicles.

[0019] Construct corresponding operation time window sequences for operation units within each operation area and associate them with specific vehicle types and operation point locations.

[0020] Furthermore, the upper-level manager model is a near-end policy optimization model, generating a global scheduling instruction set including:

[0021] A joint state space containing vehicle operation data, task data, and environmental monitoring data is constructed, and the traversable area map and the operation time window sequence are used as the scheduling constraint boundary of the joint state space.

[0022] Define the task scheduling action space, including task allocation actions, job area division actions, and time window start and end adjustment actions, and build a corresponding strategy network for each action;

[0023] The policy network is optimized using the cutoff objective function of PPO. In each training round, a structured dataset is used as the observation input to calculate the action selection probability ratio and limit the policy update magnitude.

[0024] Based on the optimized strategy output, the matching and allocation relationship between tasks and vehicles, the division of work areas, and the time window are obtained to form a global scheduling instruction set.

[0025] Furthermore, the lower-level worker model is a multi-agent reinforcement learning model, which outputs vehicle control commands based on the global scheduling instruction set and local state data, including:

[0026] For different types of work vehicles, corresponding independent reinforcement learning agents are constructed, and each agent receives vehicle operation data of its assigned vehicle.

[0027] The global scheduling command is assigned to the vehicle as a behavioral constraint for the intelligent agent;

[0028] Define the control action space, including path selection, speed adjustment, job execution triggering, and resource request behavior;

[0029] During training, a centralized training-distributed execution architecture is adopted. By constructing a joint reward function that includes the distance relationship between vehicles and the degree of task overlap, each agent is guided to optimize its strategy under multi-objective constraints.

[0030] During the inference phase, each agent inputs local vehicle operation data and matching allocation relationships from the global scheduling instruction set, work area division information, and time windows into the trained policy network, and outputs vehicle control instructions.

[0031] Furthermore, the Pareto optimal solution set includes:

[0032] Historical trajectory data, resource usage logs, and task completion rate indicators of each terminal vehicle during the execution of control commands are collected to form a scheduling execution sample set;

[0033] Based on preset multi-objective evaluation indicators, including unit transportation energy consumption, task completion time deviation and regional resource utilization rate, a scheduling evaluation matrix is ​​constructed.

[0034] The non-dominated sorting genetic algorithm is used to perform solution set hierarchies and elite selection on the scheduling evaluation matrix, and the output is a set containing multiple Pareto optimal scheduling schemes.

[0035] Furthermore, the upper-level manager model transmits the global scheduling instruction set to the lower-level worker model via the gRPC remote procedure call protocol;

[0036] The lower-level worker model sends the vehicle control commands to the terminal vehicle via the MQTT lightweight message queue transmission protocol.

[0037] Furthermore, the multi-terminal vehicle dispatching system also includes:

[0038] A human-machine collaborative control module is used to integrate the dispatching system with the digital twin platform, construct a three-dimensional visualization interface based on vehicle operating status and dispatching instruction set, and connect to voice commands and manual intervention interfaces for handling sudden anomalies and dispatching adjustments; the human-machine collaborative control module includes:

[0039] The interface program unit is used to collect and synchronize the location information, operation status and scheduling command execution results of each terminal vehicle in real time, and map the above information to the digital twin platform to build a dynamically updated three-dimensional mining area operation interface.

[0040] The visualization unit, based on WebGL technology and vehicle simulation data, presents the vehicle's running trajectory, the occupancy of the work area, and the execution process of the scheduling strategy in real time.

[0041] The voice access unit is used to receive voice input from the dispatcher and, through the integrated voice recognition module, parses the voice commands and converts them into a standardized task request format.

[0042] The manual intervention unit provides a graphical user interface, supports dispatchers to perform manual control operations, and sends adjustment instructions to the vehicle terminal in real time.

[0043] The beneficial effects of this invention are:

[0044] 1. This invention introduces a hierarchical structure with upper and lower layers. The upper-layer manager model generates a global scheduling instruction set based on the PPO algorithm, coordinating task allocation and time window division across various work areas. The lower-layer worker model employs a multi-agent reinforcement learning mechanism, autonomously outputting vehicle control strategies based on local state data and upper-layer instructions. This structure effectively decouples global planning from local execution problems, improving the response speed and collaborative efficiency of the scheduling system in multi-terminal scenarios.

[0045] 2. This invention utilizes historical execution data from each terminal vehicle to construct a multi-objective scheduling evaluation matrix, and combines it with the NSGA-II non-dominated sorting genetic algorithm for strategy evaluation and optimization, generating a Pareto optimal solution set. This mechanism enables systematic analysis and screening of deployed scheduling strategies, achieving an optimal balance between energy consumption, timeliness, and resource utilization, and assisting in the formation of more efficient scheduling strategy update paths. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a schematic diagram of a multi-terminal vehicle dispatching system based on reinforcement learning provided by the present invention;

[0048] Figure 2 This invention provides a hierarchical reinforcement learning framework diagram;

[0049] Figure 3 This invention provides an execution flowchart for a multi-terminal vehicle dispatching system based on reinforcement learning. Detailed Implementation

[0050] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0051] Example 1

[0052] A multi-terminal vehicle dispatching system based on reinforcement learning, such as Figure 1 As shown, it includes:

[0053] The heterogeneous data fusion module is used to collect and uniformly encapsulate vehicle operation data, task data and environmental monitoring data from the mine site, and form a structured dataset through standardized processing via protocols.

[0054] Specifically, vehicle operation data is collected by the on-board controller and mobile terminal, including at least the remaining load, remaining energy consumption and current task identifier; task data is issued by the dispatch center or manually entered, including at least the task location, expected duration and resource requirements; environmental monitoring data is collected by various environmental sensors deployed in the mining area, including at least road congestion density, slope stability and water mist dust index.

[0055] Furthermore, the formation of structured datasets through protocol standardization includes:

[0056] Time synchronization and format cleaning of vehicle operation data, task information and environmental monitoring data;

[0057] Encapsulation is performed according to the preset ISO 15143-3 communication standard protocol;

[0058] The encapsulated data is packaged into a unified structure to build a structured dataset with field consistency and temporal continuity.

[0059] Specifically, all collected data is uniformly calibrated using GPS timestamps. Data packets with time drift are resampled using linear interpolation to ensure the alignment and consistency of vehicle status, task status, and environmental information on the same timeline. For each data source, a field mapping template is used to unify field names and units, remove duplicates and outliers (such as sudden speed changes or sensor null values), and fill in missing items using historical averages. All data encapsulation follows the ISO 15143-3 standard (i.e., AEMP 2.0 protocol). For raw JSON, XML, or CAN message format data, a middleware module is used to uniformly convert it into structured JSON format. Each data entry contains the {source_id, timestamp, data_type, value_unit, data_payload} fields. The cleaned and protocol-encapsulated data is packaged according to the "task-vehicle-environment" three-dimensional tags to construct a unified structure format.

[0060] By standardizing the data according to the protocol to form a structured dataset, it is helpful to unify the format of multi-source data, improve the consistency of data time sequence and parsing efficiency, and ensure that the vehicle dispatching system can efficiently receive, process and utilize on-site data, providing data support for subsequent dispatching decisions.

[0061] The resource allocation module extracts the boundary and time window information of the work area based on the structured dataset, and generates a map of passable areas and a sequence of work time windows by combining preset priority rules.

[0062] Furthermore, generating a passable area map and a sequence of operation time windows includes:

[0063] Based on the structured dataset, extract the regional boundary information and task start and end time parameters of the task;

[0064] By combining real-time environmental monitoring data from structured datasets, the level of traffic conflict in each work area at different time periods is assessed.

[0065] Based on preset task priority rules and traffic conflict levels, exclusive passage areas and overlapping operation areas are divided for different types of vehicles.

[0066] Construct corresponding operation time window sequences for operation units within each operation area and associate them with specific vehicle types and operation point locations.

[0067] Specifically, generating a passable area map and a sequence of operation time windows includes: First, extracting the geographical location, task number, and time parameters of the operation tasks from the structured dataset. Using GPS coordinates and task grouping information, a geometric boundary generation algorithm such as the convex hull method is used to construct the regional boundaries of each task. Combined with the task start and end time fields, a preliminary task timetable is formed. The regional boundaries are encoded in GeoJSON format and stored in a spatial database for subsequent scheduling reference. Next, combining real-time environmental monitoring data from the structured dataset, including slope stability coefficients, traffic density, and dust levels, the monitoring data is aggregated using a sliding window and normalized in terms of dimensions. A weighted scoring function is used to calculate the passability safety level of different areas at different time periods, forming a conflict level matrix for evaluating passability constraints. In one feasible implementation, the conflict levels are divided into: low (0–0.3), medium (0.3–0.6), and high (0.6–1.0) according to the weighted scoring function. The formula for calculating the weighted scoring function is:

[0068] ;

[0069] in, Indicates the time window sequence number. Represents normalized traffic density. This indicates the normalized dust level. This represents the normalized slope stability coefficient. , and The corresponding data weighting coefficients. Based on the conflict level results, according to the preset task priority rules and vehicle type (such as dump trucks, bulldozers, loaders) traffic strategies, each work area is divided into exclusive traffic areas and cross-operation areas within a specific time period. High-priority tasks are assigned to areas with single-vehicle exclusive access, while low-priority tasks allow for cross-scheduling of multiple vehicles, forming a "vehicle-area-time" traffic relationship table. For each work area, based on the task start and end times and conflict level sequence, multiple alternative time windows are generated using the sliding window method. Each window satisfies the following conditions: conflict level ≤ threshold, duration ≥ estimated task duration. The generated time windows are marked as those that can be assigned to specific types of vehicles. For example, one type of time window is marked as follows:

[0070] {

[0071] "region_id": "RG_04",

[0072] "vehicle_type": "DUMP_TRUCK",

[0073] "available_windows": [

[0074] { "start": "09:15", "end": "10:00"},

[0075] { "start": "10:15", "end": "11:00"} ]

[0077] }

[0078] By generating a map of passable areas and a sequence of operation time windows, the accessibility and conflict risks of each operation area at different time periods can be accurately identified, thereby effectively constraining the operation routes and time arrangements of various types of vehicles, improving the safety and feasibility of scheduling strategies, and reducing the probability of path conflicts and resource conflicts.

[0079] Hierarchical reinforcement learning modules, such as Figure 2 As shown, it includes an upper-level manager model and a lower-level worker model. The upper-level manager model receives the structured dataset, a map of passable areas, and a sequence of work time windows, and generates a global scheduling instruction set. The lower-level worker model outputs vehicle control instructions based on the global scheduling instruction set and local status data.

[0080] Furthermore, the upper-level manager model is a near-end policy optimization model, generating a global scheduling instruction set including:

[0081] A joint state space containing vehicle operation data, task data, and environmental monitoring data is constructed, and the traversable area map and the operation time window sequence are used as the scheduling constraint boundary of the joint state space.

[0082] Define the task scheduling action space, including task allocation actions, job area division actions, and time window start and end adjustment actions, and build a corresponding strategy network for each action;

[0083] The policy network is optimized using the cutoff objective function of PPO. In each training round, a structured dataset is used as the observation input to calculate the action selection probability ratio and limit the policy update magnitude.

[0084] Based on the optimized strategy output, the matching and allocation relationship between tasks and vehicles, the division of work areas, and the time window are obtained to form a global scheduling instruction set.

[0085] Specifically, firstly, a joint state space is constructed that includes vehicle operation data, task data, and environmental monitoring data. The traversable area map and job time window sequence generated by the resource allocation module are used as the scheduling constraint boundaries of the state space. The action space for task scheduling is defined. This includes actions such as task allocation, job area division, and job time window start-end fine-tuning, and policy sub-networks are constructed for each of these actions. Each policy network... The output is the probability distribution of actions in the current state, implemented through a Softmax layer:

[0086] ;

[0087] in, Represents the policy network function, This represents the set of parameters for the policy network. During policy optimization, the PPO (Procedure for Policy Optimization) objective function is used to iteratively update each policy network. Each training round uses a structured dataset as the observation input to compute the old policy. With new strategies The probability of the following action is higher than :

[0088] ;

[0089] Then, the shear optimization objective is constructed:

[0090] ;

[0091] in, [ The symbol ] represents the expected value over time, indicating that the expected value of a batch of samples is taken and averaged. Represents the function to be minimized. Represents the shearing function, with restrictions exist[ Within the specified range, to prevent excessively large updates, This represents the shearing parameter, typically set to 0.1 to 0.3. It controls the step size range of the strategy update to prevent drastic changes. The advantage function estimate measures the performance of the current action relative to the average policy and is calculated using the GAE method. After model optimization, the system outputs a global scheduling instruction set based on the policy network, which includes the matching relationship between tasks and vehicles, the work area division scheme, and the adjustment amount of the work time window.

[0092] By introducing a PPO-based upper-level manager model, vehicle status, task requirements, and environmental factors can be comprehensively analyzed from a global perspective. This constructs a joint state space constrained by regional maps and time constraints, enhancing the model's global perception and dynamic optimization capabilities regarding task priorities and resource allocation strategies. This mechanism helps to form a coordinated and comprehensive scheduling instruction set, achieving optimal resource allocation across regions and task types.

[0093] Furthermore, the lower-level worker model is a multi-agent reinforcement learning model, which outputs vehicle control commands based on the global scheduling instruction set and local state data, including:

[0094] For different types of work vehicles, corresponding independent reinforcement learning agents are constructed, and each agent receives vehicle operation data of its assigned vehicle.

[0095] The global scheduling command is assigned to the vehicle as a behavioral constraint for the intelligent agent;

[0096] Define the control action space, including path selection, speed adjustment, job execution triggering, and resource request behavior;

[0097] During training, a centralized training-distributed execution architecture is adopted. By constructing a joint reward function that includes the distance relationship between vehicles and the degree of task overlap, each agent is guided to optimize its strategy under multi-objective constraints.

[0098] During the inference phase, each agent inputs local vehicle operation data and matching allocation relationships from the global scheduling instruction set, work area division information, and time windows into the trained policy network, and outputs vehicle control instructions.

[0099] Specifically, for different types of vehicles in a mining scenario, such as transport vehicles, loaders, and auxiliary vehicles, multiple agents with independent policy networks are constructed. Each agent controls only one vehicle in a work unit. The input information for each agent consists of two sources: one is the local state data of the vehicle; the other is the global scheduling instructions transmitted by the upper-level manager model, specifically the task number assigned to the vehicle, the specified work area information, and the work time window. In the action space of the agent, the following main control behaviors are defined: path selection, used to determine the current vehicle's route; speed adjustment, used to adapt to regional congestion or dynamic changes in task nodes; task execution triggering, used to determine the start and completion times of the task; and resource request behavior, used to initiate resource requests (such as maintenance, refueling, loading, etc.) during task execution. During the training phase, a centralized training and distributed execution (CTDE) method is adopted, where the local states of multiple vehicle agents are jointly input to the central trainer, and a joint reward function is constructed to optimize the agent policies. In one feasible implementation, an example of the joint reward function is as follows:

[0100]

[0101] in, The distance relationship between vehicle intelligent agents can be represented as , The GPS coordinates of the vehicle's intelligent agent. The degree of task overlap between vehicle intelligent agents can be represented as: , This indicates the task area of ​​the vehicle's intelligent agent. This represents the area of ​​intersection of task regions between vehicle intelligent agents. This represents the union area of ​​the task regions among the vehicle agents. During training, the MADDPG algorithm is used to optimize the policies of each agent, enabling them to learn efficient cooperative control behaviors under multi-objective constraints. After training, in the actual deployment and inference phase, each agent runs independently, receiving its vehicle status and scheduling instructions in real time, obtaining the control action sequence through forward propagation, and outputting vehicle control commands.

[0102] By constructing a lower-level worker model as a multi-agent reinforcement learning architecture, the differentiated advantages of different types of work vehicles in local state perception and execution strategies can be fully explored. Under the guidance of global scheduling instructions, each agent collaborates and optimizes within a centralized training-distributed execution framework. Combining the dynamic interaction relationships between vehicles and task overlap, it learns efficient and flexible local control strategies, significantly improving the system's adaptability and execution efficiency in complex work environments, and maximizing the fine-grained scheduling of vehicles and resource utilization at the micro level.

[0103] The feedback module is optimized by using historical execution data from each terminal vehicle as decision samples. Based on the NSGA-II evolutionary algorithm, the strategy is evaluated and the scheduling scheme is reordered. The Pareto optimal solution is selected and transmitted to the upper-level manager model to participate in the global scheduling instruction set update.

[0104] Furthermore, the Pareto optimal solution set includes:

[0105] Historical trajectory data, resource usage logs, and task completion rate indicators of each terminal vehicle during the execution of control commands are collected to form a scheduling execution sample set;

[0106] Based on preset multi-objective evaluation indicators, including unit transportation energy consumption, task completion time deviation and regional resource utilization rate, a scheduling evaluation matrix is ​​constructed.

[0107] The non-dominated sorting genetic algorithm is used to perform solution set hierarchies and elite selection on the scheduling evaluation matrix, and the output is a set containing multiple Pareto optimal scheduling schemes.

[0108] Specifically, the system collects historical operational trajectories, resource usage logs, and task completion rate metrics from each terminal vehicle during the scheduling process to construct a scheduling execution sample set. For each scheduling scheme... The following three multi-objective evaluation functions are defined to assess its performance in terms of energy efficiency, time accuracy, and resource balance: unit transportation energy consumption ,in Representing the scheduling scheme Total energy consumption of vehicles under [specific conditions] Representing the scheduling scheme Total number of tasks completed; Task completion time deviation ,in Indicates the total number of tasks. and They represent the first Actual and planned completion times of each task; regional resource utilization balance. ,in Indicates the total number of task areas. Indicates the task area The frequency of vehicle use This represents the average vehicle usage frequency across all task areas. The three objective functions described above constitute the evaluation vector of the scheduling scheme. After forming the scheduling evaluation matrix, the NSGA-II algorithm is used to perform a non-dominated sorting of the solution set. For two sets of solutions (scheduling schemes)... and If a solution Non-inferior in all objectives And superior to at least one objective Then it is believed Dominate Such comparisons can generate a Pareto front solution set that is not dominated by any other scheme. To improve the diversity of the solution set, crowding distance is introduced as a criterion for preserving solutions, and the calculation formula is as follows:

[0109] ;

[0110] in, Distance indicating crowding level and Indicates the current solution at the th position. The objective values ​​of adjacent solutions in the objective function. and The maximum and minimum values ​​of this objective are defined. Through multiple generations of evolution, selection, crossover, and mutation operations, the final output is a set of scheduling schemes containing multiple Pareto optimal solutions. }

[0111] By constructing a scheduling execution sample set based on historical execution data and introducing multi-objective evaluation indicators, and using a non-dominated sorting genetic algorithm to evaluate the merits of strategies and screen Pareto front solutions, we can effectively discover the set of optimal scheduling schemes that achieve a balance between transportation energy consumption, task timeliness, and resource utilization.

[0112] Furthermore, the upper-level manager model transmits the global scheduling instruction set to the lower-level worker model via the gRPC remote procedure call protocol;

[0113] The lower-level worker model sends the vehicle control commands to the terminal vehicle via the MQTT lightweight message queue transmission protocol.

[0114] By introducing the gRPC remote procedure call protocol and the MQTT lightweight message transmission protocol, an efficient, low-latency, and structured communication mechanism for scheduling and control commands between models and terminals is realized.

[0115] Furthermore, the multi-terminal vehicle dispatching system also includes:

[0116] A human-machine collaborative control module is used to integrate the dispatching system with the digital twin platform, construct a three-dimensional visualization interface based on vehicle operating status and dispatching instruction set, and connect to voice commands and manual intervention interfaces for handling sudden anomalies and dispatching adjustments; the human-machine collaborative control module includes:

[0117] The interface program unit is used to collect and synchronize the location information, operation status and scheduling command execution results of each terminal vehicle in real time, and map the above information to the digital twin platform to build a dynamically updated three-dimensional mining area operation interface.

[0118] The visualization unit, based on WebGL technology and vehicle simulation data, presents the vehicle's running trajectory, the occupancy of the work area, and the execution process of the scheduling strategy in real time.

[0119] The voice access unit is used to receive voice input from the dispatcher and, through the integrated voice recognition module, parses the voice commands and converts them into a standardized task request format.

[0120] The manual intervention unit provides a graphical user interface, supports dispatchers to perform manual control operations, and sends adjustment instructions to the vehicle terminal in real time.

[0121] By introducing a human-machine collaborative control module, deep integration between the scheduling system and the digital twin platform was achieved, constructing a human-machine interaction system that integrates data synchronization, visualization, voice interaction, and manual intervention at the functional level. This module not only improves the visibility, transparency, and controllability of the scheduling process but also enhances the system's ability to respond flexibly to unexpected anomalies.

[0122] Example 2

[0123] At a large open-pit coal mine stripping operation site, the work involved over 20 dump trucks, 5 excavators, and several auxiliary transport vehicles. The work area was complex, with frequent path intersections, and significant variations in terrain and weather conditions. To improve overall operational efficiency and scheduling robustness, the multi-terminal vehicle scheduling system based on reinforcement learning, as described in this invention, was deployed. The specific process is as follows: Figure 3 As shown, its implementation is as follows:

[0124] Data Acquisition and Structured Packaging: Through vehicle-mounted sensing devices installed at the terminals and environmental monitoring nodes deployed in the mining area, environmental data such as vehicle trajectory, work progress, wind speed, and visibility are collected in real time. The system performs time synchronization, format cleaning, and structure unification of the data according to the ISO 15143-3 communication protocol to generate standardized structured datasets.

[0125] Generate work constraint layer: Based on the collected structured dataset, the system extracts the boundary and unloading point information of each excavation point, and combines weather and traffic availability to dynamically construct a work area map and traffic time window, assigning different priority traffic areas to each vehicle type.

[0126] Scheduling strategy generation and distribution: The upper-level manager model is based on the PPO strategy optimization framework. Under the constraints of passable areas and time windows, it outputs a global scheduling instruction set, including vehicle-task matching relationships, area division and start and end times of the operation window, and distributes it to the worker agent model of the corresponding vehicle through the gRPC protocol.

[0127] The worker model executes control commands in a distributed manner: the lower-level multi-agent model outputs control commands such as speed control, path selection, and job triggering through the policy network based on the assigned task commands and local state information, and pushes them to each vehicle controller in real time via the MQTT protocol.

[0128] Feedback optimization and strategy iteration: The system periodically collects trajectory data, task completion records and resource usage of each vehicle during execution, builds a scheduling sample library, and performs multi-objective optimization of task completion time, energy consumption and resource allocation efficiency based on the NSGA-II algorithm, outputting a Pareto optimal scheduling solution set to support the training of the next round of scheduling strategy.

[0129] Human-machine collaboration and anomaly intervention: In the dispatch center, dispatchers can monitor vehicle trajectories and task status in real time through a 3D visualized mining area interface. In the event of emergencies (such as road landslides, equipment failures, etc.), they can issue adjustment commands through voice or graphical interfaces to achieve rapid intervention and system adaptive adjustment.

[0130] Table 1 shows a comparison of scheduling efficiency and resource utilization before and after system deployment within a 30-working-day statistical period. The comparison demonstrates that the reinforcement learning-based multi-terminal vehicle scheduling system described in this invention significantly improves scheduling efficiency and resource utilization in actual open-pit coal mine stripping operations. While reducing average task completion time, lowering vehicle empty-running rate and unit fuel consumption, the system effectively avoids operational conflicts and significantly reduces the frequency of manual intervention, exhibiting high autonomous decision-making capability and scheduling robustness. Furthermore, the optimization feedback mechanism formed by the NSGA-II algorithm enables continuous self-evolution of scheduling strategies, giving the system stronger adaptability and optimization potential in long-term operation, and showcasing the comprehensive advantages of intelligent, efficient, and refined collaborative management.

[0131] Table 1 Comparison of scheduling efficiency and resource utilization before and after system deployment

[0132]

[0133] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0134] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0135] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0137] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0138] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-terminal vehicle dispatching system based on reinforcement learning, characterized in that, include: The heterogeneous data fusion module is used to collect and uniformly encapsulate vehicle operation data, task data and environmental monitoring data from the mine site, and form a structured dataset through standardized processing via protocols. The resource allocation module extracts the boundary and time window information of the work area based on the structured dataset, and generates a map of passable areas and a sequence of work time windows by combining preset priority rules. The hierarchical reinforcement learning module includes an upper-level manager model and a lower-level worker model. The upper-level manager model receives the structured dataset, a map of traversable areas, and a sequence of operation time windows, and generates a global scheduling instruction set. The lower-level worker model outputs vehicle control instructions based on the global scheduling instruction set and local state data. The feedback module is optimized by using historical execution data of each terminal vehicle as decision samples, performing post-policy evaluation and re-ranking of scheduling schemes based on the NSGA-II evolutionary algorithm, selecting the Pareto optimal solution and transmitting it to the upper-level manager model to participate in the global scheduling instruction set update. The generation of a passable area map and a sequence of operation time windows includes: Based on the structured dataset, extract the regional boundary information and task start and end time parameters of the task; By combining real-time environmental monitoring data from structured datasets, the level of traffic conflict in each work area at different time periods is assessed. Based on preset task priority rules and traffic conflict levels, exclusive passage areas and overlapping operation areas are divided for different types of vehicles. Construct corresponding work time window sequences for work units within each work area and associate them with specific vehicle types and work point locations; Among them, the upper-level manager model is a near-end policy optimization model, which generates a global scheduling instruction set including: A joint state space containing vehicle operation data, task data, and environmental monitoring data is constructed, and the traversable area map and the operation time window sequence are used as the scheduling constraint boundary of the joint state space. Define the task scheduling action space, including task allocation actions, job area division actions, and time window start and end adjustment actions, and build a corresponding strategy network for each action; The policy network is optimized using the cutoff objective function of PPO. In each training round, a structured dataset is used as the observation input to calculate the action selection probability ratio and limit the policy update magnitude. Based on the optimized strategy output, the matching and allocation relationship between tasks and vehicles, the division information of work areas and time windows are obtained to form a global scheduling instruction set. The lower-level worker model is a multi-agent reinforcement learning model, which outputs vehicle control commands based on the global scheduling instruction set and local state data, including: For different types of work vehicles, corresponding independent reinforcement learning agents are constructed, and each agent receives vehicle operation data of its assigned vehicle. The global scheduling command is assigned to the vehicle as a behavioral constraint for the intelligent agent; Define the control action space, including path selection, speed adjustment, job execution triggering, and resource request behavior; During training, a centralized training-distributed execution architecture is adopted. By constructing a joint reward function that includes the distance relationship between vehicles and the degree of task overlap, each agent is guided to optimize its strategy under multi-objective constraints. During the inference phase, each agent inputs local vehicle operation data and matching allocation relationships from the global scheduling instruction set, work area division information, and time windows into the trained policy network, and outputs vehicle control instructions.

2. The multi-terminal vehicle dispatching system based on reinforcement learning according to claim 1, characterized in that, Structured datasets are formed through protocol standardization, including: Time synchronization and format cleaning of vehicle operation data, task information and environmental monitoring data; Encapsulated according to the preset ISO 15143-3 communication standard protocol; The encapsulated data is packaged into a unified structure to build a structured dataset with field consistency and temporal continuity.

3. The multi-terminal vehicle dispatching system based on reinforcement learning according to claim 1, characterized in that, The Pareto optimal solution set includes: Historical trajectory data, resource usage logs, and task completion rate indicators of each terminal vehicle during the execution of control commands are collected to form a scheduling execution sample set; Based on preset multi-objective evaluation indicators, including unit transportation energy consumption, task completion time deviation and regional resource utilization rate, a scheduling evaluation matrix is ​​constructed. The non-dominated sorting genetic algorithm is used to perform solution set hierarchies and elite selection on the scheduling evaluation matrix, and the output is a set containing multiple Pareto optimal scheduling schemes.

4. A multi-terminal vehicle dispatching system based on reinforcement learning according to claim 1, characterized in that, The upper-level manager model transmits the global scheduling instruction set to the lower-level worker model via the gRPC remote procedure call protocol; The lower-level worker model sends the vehicle control commands to the terminal vehicle via the MQTT lightweight message queue transmission protocol.

5. A multi-terminal vehicle dispatching system based on reinforcement learning according to claim 1, characterized in that, The multi-terminal vehicle dispatching system also includes: A human-machine collaborative control module is used to integrate the dispatching system with the digital twin platform, construct a three-dimensional visualization interface based on vehicle operating status and dispatching instruction set, and connect to voice commands and manual intervention interfaces for handling sudden anomalies and dispatching adjustments; the human-machine collaborative control module includes: The interface program unit is used to collect and synchronize the location information, operation status and scheduling command execution results of each terminal vehicle in real time, and map the above information to the digital twin platform to build a dynamically updated three-dimensional mining area operation interface. The visualization unit, based on WebGL technology and vehicle simulation data, presents the vehicle's running trajectory, the occupancy of the work area, and the execution process of the scheduling strategy in real time. The voice access unit is used to receive voice input from the dispatcher and, through the integrated voice recognition module, parses the voice commands and converts them into a standardized task request format. The manual intervention unit provides a graphical user interface, supports dispatchers to perform manual control operations, and sends adjustment instructions to the vehicle terminal in real time.

Images