An unmanned decision control method and system for a downhole transport vehicle

Abstract

The application discloses a kind of unmanned decision control method and system of underground transport vehicle, it is related to coal mine underground intelligent auxiliary transport and unmanned control technical field, control method includes as follows: acquisition multi-source sensor data and pre-processing;Build vision-language-action end-to-end decision control model and training, input pre-processing multi-source perception data, vehicle state information, underground scene knowledge, transport task instruction and underground safety rules into the model after training, generate initial vehicle control instruction;Safety check is carried out to initial vehicle control instruction, and when it does not satisfy safety constraint condition, safety clipping or trigger safety protection control is carried out, obtain executable control instruction;Control instruction is issued to underground transport vehicle, corresponding key scene data is recorded;Based on key scene data, model is carried out offline incremental training and light weight update.The application improves the safety control capability of transport vehicle in complex roadway transport environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent auxiliary transportation and unmanned driving control technology in coal mines, and in particular to an unmanned driving decision-making and control method and system for underground transportation vehicles. Background Technology

[0002] Underground transport vehicles in coal mines are an important component of the underground auxiliary transport system, mainly including trackless rubber-tired vehicles, mining transport vehicles, personnel transport vehicles, material transport vehicles, and equipment transfer vehicles. Among them, trackless rubber-tired vehicles have become essential equipment for personnel transportation, material transport, and equipment transfer in underground coal mines due to their maneuverability, high transport efficiency, and strong adaptability to tunnel environments. With the advancement of intelligent coal mine construction, realizing unmanned driving of underground transport vehicles, especially trackless rubber-tired vehicles, is of great significance for reducing safety risks for underground workers, improving auxiliary transport efficiency, and enhancing the level of mine intelligence.

[0003] Currently, existing unmanned driving systems for underground coal mine transport vehicles, especially trackless rubber-tired vehicles, generally adopt a modular architecture, dividing the system into a perception module, a positioning module, a planning module, and a control module. Each module is designed and optimized independently, and information is transmitted through predefined interfaces. While this architecture has advantages such as clear structure and ease of engineering development, it still has significant shortcomings under complex underground working conditions: on the one hand, errors in the preceding perception or positioning stages are propagated along the planning and control links, causing error accumulation; on the other hand, the main reliance on structured result transmission between modules can easily lead to the loss of original perception information and scene context information, affecting the overall decision-making effect of the system. Furthermore, the complex coupling relationships between modules make parameter debugging and system maintenance difficult, hindering the achievement of overall optimization for complex underground transportation scenarios.

[0004] In recent years, end-to-end autonomous driving methods have established a direct mapping from sensor inputs to vehicle control outputs through deep learning models, providing a new technical path for simplifying the architecture of autonomous driving systems. With the development of large language models and multimodal large models, utilizing the multimodal representation, semantic understanding, and logical reasoning capabilities of large models to achieve end-to-end driving decisions has gradually become a research hotspot. However, existing end-to-end large models are mostly geared towards open road scenarios, and their direct application to underground coal mine transport vehicles still suffers from insufficient adaptability. Underground coal mines lack GNSS signals, have poor lighting conditions, and suffer from severe interference from dust, water mist, and vehicle vibration. The narrow tunnels also contain numerous curves, slopes, and areas where vehicles pass each other, placing higher demands on real-time vehicle control, safety constraints, and environmental robustness.

[0005] Furthermore, the control cycle of trackless rubber-tired vehicles in underground mines is relatively short, while large models typically have large parameter scales and high inference computation loads, making it difficult to directly meet the real-time operation requirements of the vehicle-mounted end. Underground environments present high safety risks such as gas, mixed traffic of personnel and vehicles, and limited space for passing, requiring effective constraints on the uncertainty of end-to-end model outputs. Simultaneously, underground driving data collection is costly and difficult to annotate, making it difficult to form large-scale, high-quality training datasets. Therefore, there is an urgent need for an unmanned driving decision-making and control method and system for underground coal mine transport vehicles, especially trackless rubber-tired vehicles, to balance real-time performance, safety, and adaptability to complex underground environments. Summary of the Invention

[0006] This solution addresses the problems and needs raised above by proposing an unmanned driving decision-making and control method and system for underground transport vehicles. Due to the adoption of the following technical features, it can achieve the above-mentioned technical objectives and bring about several other technical effects.

[0007] One objective of this invention is to provide a decision-making and control method for unmanned underground transport vehicles, comprising the following steps: S10: Collect multi-source sensor data from underground transport vehicles and preprocess the multi-source sensor data to obtain standardized multi-source sensing data; S20: Construct and train a vision-language-action end-to-end decision control model. Input the standardized multi-source perception data, vehicle status information, underground scene knowledge, transportation task instructions and underground safety rules into the trained vision-language-action end-to-end decision control model to generate initial vehicle control instructions. S30: Perform a safety verification on the initial vehicle control command based on the safety constraints of the underground transport vehicle, and when the initial vehicle control command does not meet the safety constraints of the underground transport vehicle, perform safety trimming or trigger safety protection control to obtain the final executable control command. S40: The final executable control command is sent to the actuator of the underground transport vehicle chassis, and key scenario data corresponding to manual takeover, safety constraint triggering, abnormal working conditions and control command correction are recorded; based on the key scenario data, the vision-language-action end-to-end decision control model is subjected to offline incremental training and lightweight update.

[0008] In one example of the present invention, the preprocessing of the multi-source sensor data in step S10 includes: The system performs false echo filtering, outlier removal, and multi-frame consistency filtering on the lidar point cloud data; low-light enhancement, defogging, and distortion correction on the camera image data; clutter suppression and target detection on the millimeter-wave radar data; and filtering, outlier removal, and normalization on the inertial measurement unit data and vehicle CAN bus status data. The multi-source sensor data includes at least two of the following: lidar point cloud data installed on the underground transport vehicle, camera image data, millimeter-wave radar data, inertial measurement unit data, and vehicle CAN bus status data.

[0009] In one example of the present invention, in step S20, the vision-language-action end-to-end decision control model includes: a multimodal fusion coding module, a cross-modal semantic projection module, a language reasoning module, and an action decoding module; The multimodal fusion coding module is configured to extract multi-source sensing features from the standardized multi-source sensing data and generate a unified scene representation. The cross-modal semantic projection module is configured to map the unified scene representation to the semantic embedding space of the language reasoning module; The language reasoning module is configured to combine vehicle status information, underground scene knowledge, transportation task instructions, and underground safety rules to generate a driving decision representation. The action decoding module is configured to convert the driving decision representation into vehicle control commands.

[0010] In one example of the present invention, the multimodal fusion coding module is used to convert lidar point cloud data, camera image data and millimeter-wave radar data to a bird's-eye view plane, to obtain point cloud BEV features, image BEV features and radar BEV features respectively, and to perform feature-level fusion of the point cloud BEV features, image BEV features and radar BEV features to obtain a unified BEV fusion feature map.

[0011] In one example of the present invention, step S20, training the vision-language-action end-to-end decision control model, includes the following steps: S21: Construct an underground transport vehicle driving dataset; the dataset includes manual driving data, safety rule correction data, and underground scene image-text pairing data; wherein, the manual driving data includes multi-source sensor data, vehicle status data, and corresponding manual driving control commands; the safety rule correction data includes: safety constraint trigger segments and control command correction segments, with the safety correction control commands serving as the supervisory signal; the underground scene image-text pairing data includes multi-source perception data and its corresponding natural language scene descriptions; the underground transport vehicle driving dataset is divided into a training set and a validation set; S22: First-stage cross-modal semantic alignment training using the training set: Based on the image-text pairing data of the underground scene, cross-modal semantic alignment training is performed on the multimodal fusion coding module and the cross-modal semantic projection module; the distance between the visual embedding and the corresponding text semantic embedding is constrained by the contrastive learning loss, so that the matching visual-text features are close to each other, and the mismatched visual-text features are far apart; S23: Second-stage end-to-end joint training using the training set: Based on data collected from manual driving and safety rule correction data, the vision-language-action end-to-end decision control model is jointly trained. Multi-source perception data, vehicle status data, underground scene knowledge prompts, transportation task instructions, and underground safety rule prompts are input into the vision-language-action end-to-end decision control model. The model outputs longitudinal acceleration, steering angle, and gear instructions through a multi-modal fusion encoding module, a cross-modal semantic projection module, a language reasoning module, and an action decoding module. During training, manual driving control instructions and safety correction control instructions are used as supervision signals to enable the model to learn the mapping relationship from underground multi-source perception inputs to vehicle control outputs under safety constraints. S24: After training, the performance of the vision-language-action end-to-end decision control model is evaluated based on the validation set. The evaluation metrics include longitudinal acceleration prediction error, steering angle prediction error, gear classification accuracy, and safety default rate.

[0012] In one example of the present invention, step S30 specifically includes the following steps: S31: Construct real-time safety constraint boundaries: Based on the initial vehicle control command, the current downhole environment state, the vehicle state, and downhole safety rules, construct the real-time safety constraint boundaries for the current moment; S32: Perform safety and feasibility verification on the initial vehicle control commands. The initial vehicle control command is:

[0013] in, Indicates longitudinal acceleration. Indicates the steering angle. Indicates gear position command; The verification includes: determining whether the longitudinal acceleration exceeds the vehicle's permissible acceleration or braking range; determining whether the steering angle exceeds the maximum steering angle or minimum turning radius constraint; determining whether the gear command is consistent with the current vehicle speed, driving direction, and transportation task status; determining whether the vehicle trajectory predicted based on the initial control command deviates from the passable area; determining whether the distance between the vehicle and personnel, vehicles, obstacles, or lane boundaries is less than the safety threshold; and determining whether the current gas concentration, visibility, and road surface conditions allow the vehicle to continue executing the control command. If the initial vehicle control command meets the real-time safety constraint boundary, proceed to step S35; if the initial vehicle control command exceeds the real-time safety constraint boundary, proceed to step S33. S33: Safety trimming and correction of control commands: When the initial vehicle control command exceeds the real-time safety constraint boundary, it is safely trimmed and corrected; For longitudinal acceleration, when the acceleration output by the model may cause the vehicle to exceed the current maximum permissible speed, it is corrected to a control quantity to maintain a safe speed or decelerate; when the distance to people, vehicles or obstacles in front is less than the minimum safe distance, the longitudinal acceleration is corrected to a braking control quantity. Regarding the steering angle, when the steering angle output by the model may cause the vehicle to approach the roadway boundary, support structure, or deviate from the passable area, the steering angle is limited to a safe steering range to keep the vehicle within the safe passable area. For gear position commands, the gear position output by the model is corrected when it is inconsistent with the current vehicle status or transportation task. S34: Trigger safety protection control strategy: When a high-risk working condition is detected, or the initial vehicle control command cannot meet the safe driving requirements of the underground transport vehicle through safety trimming, the safety protection control strategy is triggered. S35: Generate final executable control instructions: If the initial vehicle control instructions meet the real-time safety constraint boundaries, then the initial vehicle control instructions are used as the final executable control instructions; if the initial vehicle control instructions have safety risks, then the safety correction control instructions obtained in step S33, or the control instructions generated by the safety protection control strategy in step S34, are used as the final executable control instructions.

[0014] In one example of the present invention, in step S30, the safety constraints of the underground transport vehicle include at least one of the following: minimum safe distance, maximum permissible speed, roadway boundary safe distance, personnel and vehicle avoidance rules, oncoming vehicle avoidance rules, curve deceleration rules, slope driving rules, low-speed passage rules in loading and unloading areas, gas concentration threshold, visibility threshold, and vehicle dynamics constraints.

[0015] In one example of the present invention, in step S30, when the initial vehicle control command does not meet the safety constraints of the underground transport vehicle, at least one of the following processes is performed: Limit vehicle longitudinal acceleration or speed, limit vehicle steering angle, correct vehicle gear, trigger deceleration control, emergency braking, parking hold, audible and visual alarms, remote prompts, and manual takeover requests.

[0016] In one example of the present invention, step S40 involves offline incremental training and lightweight updates of the vision-language-action end-to-end decision control model based on the key scene data, specifically including the following steps: S41: Lightweighting the vision-language-action end-to-end decision control model: Considering the constraints of computing power, storage space, power consumption, and real-time control cycle of the on-board computing platform for underground transport vehicles, the vision-language-action end-to-end decision control model is lightweighted. S42: Deploy the lightweight model to the vehicle: Deploy the lightweight model to the vehicle-mounted computing platform of the underground transport vehicle. The vehicle receives the standardized multi-source perception data set D and generates the initial vehicle control command a according to the inference process of step S20. After the safety verification in step S30, the final executable control command is generated. S43: Record key scenario data: During vehicle operation, the on-board unit automatically records key scenario data, including manual takeover segments, safety constraint triggering segments, control command correction segments, abnormal operating condition segments, unreliable perception segments, and segments with low model output confidence. S44: Transmit key scenario data back to the ground training center: Key scenario data is transmitted back to the ground training center through the communication link from underground to the surface; After receiving the data, the ground training center cleans, filters, labels, and classifies the scenarios to form an incremental training sample library; S45: Offline incremental training of the model: Based on the incremental training sample library, the vision-language-action end-to-end decision control model is trained offline. For manual takeover segments, the control instructions after manual takeover are used as the supervision signal; for safety constraint trigger segments, the control instructions after safety verification and correction are used as the reference signal; for abnormal working condition segments, by supplementing scene descriptions and safety rule prompts, the model's understanding and decision-making ability for complex roadway environments, abnormal traffic conditions and high-risk scenarios are improved. S46: Perform safety verification and version management on the incrementally trained model: After incremental training is completed, perform safety verification on the updated model. The safety verification includes scene understanding accuracy, control command rationality, collision risk, speed limit compliance, ability to maintain lane boundaries, ability to avoid personnel and vehicles, ability to handle oncoming traffic scenarios, ability to pass curves and slopes, and safety response capability under abnormal environments. When the updated model passes the preset safety test indicators, a deployable model version is generated. S47: Update the model that has passed security verification to the vehicle terminal: The model version that has passed security verification is sent to the vehicle terminal of the underground transport vehicle through the communication link or OTA. After receiving the update package, the vehicle terminal performs integrity verification and version verification on the model file. After confirming that there are no errors, the model update is completed.

[0017] Another objective of this invention is to provide an unmanned driving decision-making control system for underground transport vehicles, characterized by comprising: The multi-source data acquisition and preprocessing module is configured to acquire multi-source sensor data from underground transport vehicles and preprocess the multi-source sensor data to obtain standardized multi-source sensing data. The end-to-end decision control module is configured to build and train a vision-language-action end-to-end decision control model. It inputs the standardized multi-source perception data, vehicle status information, underground scene knowledge, transportation task instructions and underground safety rules into the trained vision-language-action end-to-end decision control model to generate initial vehicle control instructions. The safety verification and protection control module is configured to perform safety verification on the initial vehicle control command according to the safety constraints of the underground transport vehicle, and to perform safety trimming or trigger safety protection control when the initial vehicle control command does not meet the safety constraints of the underground transport vehicle to obtain the final executable control command. The chassis execution module is configured to send the final executable control commands to the chassis execution mechanism of the underground transport vehicle, and record key scenario data corresponding to manual takeover, safety constraint triggering, abnormal working conditions and control command correction; based on the key scenario data, the vision-language-action end-to-end decision control model is subjected to offline incremental training and lightweight updates.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention integrates multi-source perception, scene understanding, task reasoning, safety rule constraints, and vehicle control into an end-to-end decision-making and control framework without altering the basic structure of the chassis actuators of underground transport vehicles. This reduces error accumulation and information loss in traditional modular autonomous driving architectures. Simultaneously, through safety verification and protection control mechanisms, it prevents abnormal control commands output by the model from directly affecting the vehicle chassis, thereby improving the real-time decision-making, safety control, and environmental adaptability of trackless rubber-tired vehicles and other transport vehicles in complex underground tunnel environments.

[0019] The preferred embodiments of the invention will be described in more detail below with reference to the accompanying drawings, so as to facilitate an understanding of the features and advantages of the invention. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. The drawings are merely illustrative of some embodiments of the present invention and are not intended to limit the scope of the present invention to all embodiments.

[0021] Figure 1 A flowchart of an unmanned driving decision-making and control method for underground transport vehicles according to an embodiment of the present invention; Figure 2This is a flowchart of the vision-language-action end-to-end large model driven decision control for unmanned downhole driving according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the lightweight deployment and offline iterative update process of a model according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The same reference numerals in the drawings represent the same components. It should be noted that the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0023] According to a first aspect of the present invention, an unmanned driving decision-making and control method for underground transport vehicles is provided, such as... Figure 1 As shown, it includes the following steps: S10: Collect multi-source sensor data from underground transport vehicles, and perform time synchronization, spatial calibration, noise reduction and enhancement, and standardization preprocessing on the multi-source sensor data to obtain standardized multi-source sensing data; S20: Construct and train a vision-language-action end-to-end decision control model. Input the standardized multi-source perception data, vehicle status information, underground scene knowledge, transportation task instructions and underground safety rules into the trained vision-language-action end-to-end decision control model to generate initial vehicle control instructions. S30: Perform a safety verification on the initial vehicle control command based on the safety constraints of the underground transport vehicle, and when the initial vehicle control command does not meet the safety constraints of the underground transport vehicle, perform safety trimming or trigger safety protection control to obtain the final executable control command. S40: The final executable control command is sent to the actuator of the underground transport vehicle chassis, and key scenario data corresponding to manual takeover, safety constraint triggering, abnormal working conditions and control command correction are recorded; based on the key scenario data, the vision-language-action end-to-end decision control model is subjected to offline incremental training and lightweight update; wherein, the underground transport vehicle includes at least one of the following: coal mine trackless rubber-tired vehicle, mining transport vehicle, personnel transport vehicle, material transport vehicle and equipment transfer vehicle.

[0024] This control method integrates multi-source perception, scene understanding, task reasoning, safety rule constraints, and vehicle control into an end-to-end decision-making and control framework without altering the basic structure of the chassis actuators of underground transport vehicles. This reduces error accumulation and information loss in traditional modular autonomous driving architectures. Simultaneously, through safety verification and protection control mechanisms, it prevents abnormal control commands output by the model from directly affecting the vehicle chassis, thereby improving the real-time decision-making, safety control, and environmental adaptability of trackless rubber-tired vehicles and other transport vehicles in complex roadway environments in coal mines.

[0025] In one example of the present invention, step S10 specifically includes the following steps: S11: Deploy a multi-source sensor group on the underground transport vehicle. The multi-source sensor group includes a lidar, a camera, a millimeter-wave radar, an inertial measurement unit, and a vehicle CAN bus access module. S12: Using the vehicle coordinate system as a unified reference coordinate system, perform joint extrinsic parameter calibration on the lidar, camera, millimeter-wave radar and inertial measurement unit to obtain the rotation matrix and translation vector of each sensor coordinate system relative to the vehicle coordinate system. S13: Using the on-board computing platform system clock or the main sensor clock as a unified time reference, hardware synchronization, timestamp alignment or interpolation compensation methods are used to align multi-source sensor data with different sampling frequencies to the same time axis. S14: For underground environments with no GNSS, low light, dust and water mist, vehicle vibration and electromagnetic interference, preprocess the lidar point cloud data, camera image data, millimeter-wave radar data, inertial measurement unit data and vehicle CAN bus status data to obtain standardized multi-source sensing data.

[0026] In one example of the present invention, step S10, preprocessing the multi-source sensor data includes: The system performs false echo filtering, outlier removal, and multi-frame consistency filtering on the lidar point cloud data; low-light enhancement, defogging, and distortion correction on the camera image data; clutter suppression and target detection on the millimeter-wave radar data; and filtering, outlier removal, and normalization on the inertial measurement unit data and vehicle CAN bus status data. The multi-source sensor data includes at least two of the following: lidar point cloud data installed on the underground transport vehicle, camera image data, millimeter-wave radar data, inertial measurement unit data, and vehicle CAN bus status data.

[0027] Specifically, the underground transport vehicle is equipped with lidar, cameras, millimeter-wave radar, inertial measurement units, and a vehicle CAN bus access module. The lidar is used to collect 3D point cloud data of the underground tunnel structure, obstacles, personnel, vehicles, and tunnel boundaries; the cameras are used to collect image data of the tunnel, traffic signs, passable areas, and personnel and vehicles; the millimeter-wave radar is used to collect target distance, orientation, and relative speed information in dusty, misty, and low-light environments; and the inertial measurement unit and vehicle CAN bus are used to collect vehicle status information such as vehicle speed, steering wheel angle, longitudinal acceleration, yaw rate, gear status, and chassis status.

[0028] To address the issues of varying sensor installation locations, sampling frequencies, and coordinate systems, the vehicle coordinate system is used as a unified reference coordinate system for joint extrinsic parameter calibration of the LiDAR, camera, millimeter-wave radar, and inertial measurement unit (IMU). All sensor data is then uniformly converted to the vehicle coordinate system. The onboard computing platform system clock or the main sensor clock serves as a unified time reference for time synchronization of multi-source sensor data. Furthermore, considering environments such as no GNSS, low light, dust, water mist, vehicle vibration, and electromagnetic interference underground, the LiDAR point cloud data undergoes false echo filtering, outlier removal, and multi-frame consistency filtering. Camera image data undergoes low-light enhancement, defogging, and distortion correction. Millimeter-wave radar data undergoes clutter suppression and target detection. Inertial measurement unit data and vehicle CAN bus status data undergo outlier removal, filtering, smoothing, and normalization to obtain standardized multi-source sensing data.

[0029] In one example of the present invention, in step S20, as Figure 2 As shown, the vision-language-action end-to-end decision control model includes: a multimodal fusion encoding module, a cross-modal semantic projection module, a language reasoning module, and an action decoding module; The multimodal fusion coding module is configured to extract multi-source sensing features from the standardized multi-source sensing data and generate a unified scene representation. Specifically, the multimodal fusion coding module processes the following steps: Multimodal features are constructed based on a standardized multi-source sensing data set D.

[0030] Denoising LiDAR point cloud data Point cloud feature extraction is performed to obtain information on lane geometry, lane boundaries, obstacle contours, personnel and vehicle positions, and the spatial structure around the vehicle. Preferably, a point cloud feature extraction network can be used to extract local geometric features and global spatial features from the point cloud, and project each point feature onto a bird's-eye view plane in the vehicle coordinate system. Point features falling within the same BEV grid are pooled or aggregated to obtain a point cloud BEV feature map. .in , To standardize the number of grid cells in the vertical and horizontal directions of the BEV plane, This represents the number of feature channels in the point cloud.

[0031] Image feature extraction is performed on the enhanced camera image data I' to obtain visual semantic information such as lane boundaries, personnel and vehicles, traffic signs, loading and unloading areas, passable areas, and environmental risk areas. Preferably, an image feature extraction network can be used to obtain multi-scale image features, and the image features can be projected onto the BEV plane in the vehicle coordinate system through viewpoint transformation to obtain the image BEV feature map. ,in This represents the number of BEV feature channels in the image.

[0032] Detection results of millimeter-wave radar The distance, azimuth, radial velocity, and radar cross section information of each detected target are projected onto the BEV plane in the vehicle coordinate system and rasterized at a preset resolution. For each BEV grid, the target attribute vector falling into that grid is encoded into the feature vector of that grid using a multilayer perceptron. The feature vector of grids where no target is detected is set to zero, resulting in a sparse radar BEV feature map. ,in This represents the number of radar characteristic channels.

[0033] Filtered vehicle status data (Including vehicle speed, steering wheel angle, longitudinal acceleration, yaw rate, gear position, etc.), firstly, each physical quantity is normalized to map it to a uniform numerical range; then, the normalized state vector is input into a multilayer perceptron, and after passing through several fully connected layers and nonlinear activation functions, the state embedding vector is obtained. ,in The state is embedded in the dimension.

[0034] Fusing features from multiple modal BEVs: Feature-level fusion is used to analyze the BEV feature map of the point cloud. Image BEV feature map and radar BEV feature map Feature-level fusion is performed. Preferably, a cross-modal attention mechanism can be employed to enable point cloud features to provide reliable spatial geometry and distance information, image features to provide rich semantic recognition information, and millimeter-wave radar features to supplement target range and velocity information in dusty and misty environments. After multimodal fusion, a unified BEV fusion feature map is obtained. ,in This represents the number of feature channels to be fused.

[0035] Convert the BEV fusion feature map into a visual token sequence: Will The space is divided into several patches according to a preset spatial block size P×P, and then mapped to a visual token sequence through a linear projection layer. ,in The number of tokens is [number]. The visual token sequence serves as input to the subsequent cross-modal semantic projection module.

[0036] The cross-modal semantic projection module is configured to map the unified scene representation to the semantic embedding space of the language reasoning module. This module also maps the fused visual features and vehicle state features to the semantic embedding space of the language reasoning module, enabling multi-source perception information, vehicle state information, underground scene knowledge, transportation task instructions, and underground safety rules to be concatenated and reasoned within the same semantic space. The underground scene knowledge includes information such as roadway type, roadway width, slope, curves, meeting areas, loading / unloading areas, personnel activity areas, speed limit areas, and prohibited areas. The transportation task instructions include information such as vehicle start point, destination, waypoints, task type, loading / unloading tasks, and meeting requirements. The underground safety rules include roadway speed limit rules, curve deceleration rules, slope driving rules, meeting avoidance rules, personnel and vehicle avoidance rules, roadway boundary safety distance rules, gas concentration threshold rules, visibility threshold rules, and vehicle dynamics constraint rules.

[0037] Specifically, the construction and processing of the cross-modal semantic projection module is as follows: Constructing a cross-modal semantic projection module: The cross-modal semantic projection module is used to project visual token sequences. vehicle status embedding The semantic embedding space mapped to the language reasoning module enables multi-source perception features, vehicle status information, underground scene knowledge, transportation task instructions, and underground safety rules to be spliced ​​and reasoned within the same semantic space.

[0038] Since the feature dimensions of visual tokens are usually inconsistent with the word embedding dimensions of the language inference module, a visual projection network is set up to process the visual token sequence. Dimensional transformation and semantic space mapping are performed. The visual projection network may include linear layers, nonlinear activation layers, and normalization layers. After processing by the visual projection network, the projected visual embedding sequence is obtained, still denoted as... Embed vehicle status The input state projection network is mapped to the same semantic space as the visual embedding. Through this process, the vehicle's current motion state can participate in language reasoning along with visual perception information, underground scene knowledge, transportation task instructions, and underground safety rules.

[0039] Constructing downhole scenario knowledge, transportation task instructions, and safety rule prompts: The underground scene knowledge is used to describe the current transportation environment of the vehicle, including information such as roadway type, roadway width, roadway height, slope, curve radius, intersections, passing areas, loading and unloading areas, personnel activity areas, speed limit areas, and restricted areas. The transportation task instruction is used to describe the current transportation task of the vehicle, including information such as origin, destination, waypoints, transportation task type, loading and unloading tasks, passing requirements, priority passage rules, and access requirements for the work area.

[0040] The underground safety rules are used to describe the safety constraints that need to be followed during the model reasoning process, including roadway speed limit rules, curve deceleration rules, slope driving rules, passing rules, personnel and vehicle passing rules, roadway boundary safety distance rules, gas concentration threshold rules, visibility threshold rules, and vehicle dynamics constraint rules.

[0041] The language reasoning module is configured to combine vehicle status information, underground scene knowledge, transportation task instructions, and underground safety rules to generate a driving decision representation; the language reasoning module is used to perform joint semantic reasoning on multi-source perception information, vehicle status information, underground scene knowledge, transportation task instructions, and underground safety rules to generate a hidden state containing driving decision intentions.

[0042] Constructing a multimodal input sequence for the language reasoning module: Embed the output visual sequence Vehicle status embedding The embedded knowledge words for underground scenarios, the embedded instructions for transportation tasks, and the embedded safety rules for underground operations are concatenated in a preset order to form a multimodal input sequence for the language reasoning module.

[0043] To enable the language reasoning module to distinguish input information from different sources, positional encoding and modality identifier embedding are added to the multimodal input sequence. Positional encoding is used to represent the position information of each input token in the sequence, while modality identifier embedding is used to distinguish between visual embeddings, vehicle state embeddings, underground scene knowledge prompts, transportation task instructions, and underground safety rule prompts.

[0044] Output the hidden state of driving decisions using the language reasoning module: The multimodal input sequence is fed into the language inference module. This module can use a pre-trained large language model or a multimodal large model as the inference backbone, and adapts to the underground transport vehicle driving scenario through efficient parameter fine-tuning. The language inference module models the relationships between visual embedding, vehicle state embedding, underground scene knowledge, transport task instructions, and underground safety rules, and outputs a hidden state sequence. ,in, Indicates the length of the input sequence. Indicates the first Each input token corresponds to a hidden state. The hidden state sequence includes the current understanding of the downhole scene, the judgment result of safety constraints, and the driving decision intention.

[0045] To adapt the language reasoning module to the driving scenarios of underground transport vehicles, a parameter-efficient fine-tuning method can be used for training. Specifically, some or all of the backbone parameters of the pre-trained language model are frozen, and only the low-rank adaptation parameters, cross-modal semantic projection module parameters, multimodal fusion encoding module parameters, and action decoding module parameters are trained. This enables the model to learn the understanding of underground roadway structure, transportation rules, safety rule constraints, and driving decision-making reasoning capabilities, while reducing training computation and GPU memory usage.

[0046] The motion decoding module is configured to convert the driving decision representation into vehicle control commands. The motion decoding module converts the hidden state into vehicle control commands, which include at least one of longitudinal acceleration commands, steering angle commands, and gear shift commands. This yields the initial vehicle control commands for the underground transport vehicle.

[0047] The construction and specific processing of the action decoding module are as follows: Build the motion decoding module and generate initial vehicle control commands; The action decoding module maps the output driving decision hidden states to executable vehicle control commands for underground transport vehicles. The action decoding module includes action representation extraction, continuous control quantity regression, and discrete gear classification. An action query token is set at the end of the input sequence of the language inference module to aggregate decision information related to vehicle control. After calculation by the language inference module, the hidden state corresponding to the action query token is extracted from the hidden state sequence H as the action representation.

[0048] A continuous control regression branch is set up to predict the longitudinal acceleration and steering angle of the downhole transport vehicle. The motion representation is input into a regression network consisting of a fully connected layer, a nonlinear activation layer, and an output layer to obtain the continuous control prediction results. ,in, Indicates longitudinal acceleration. This represents the steering angle. To ensure that the output meets vehicle dynamics constraints, range constraints are applied to the regression network output, limiting the longitudinal acceleration and steering angle to the preset maximum acceleration, maximum braking deceleration, and maximum steering angle, respectively.

[0049] Action representation Inputting a classification network consisting of fully connected layers and a softmax function yields the probability distribution of gear categories. ,in, This represents the action feature vector output by the action decoder. For the gear classification network weight matrix, For bias vectors, For the probability distribution of gear categories, This is the final gear command.

[0050] Determine the gear command based on the highest probability category. ,in, This indicates the gear position command, and the gear positions include forward, neutral, and reverse.

[0051] In one example of the present invention, the multimodal fusion coding module is used to convert lidar point cloud data, camera image data and millimeter-wave radar data to a bird's-eye view plane, to obtain point cloud BEV features, image BEV features and radar BEV features respectively, and to perform feature-level fusion of the point cloud BEV features, image BEV features and radar BEV features to obtain a unified BEV fusion feature map.

[0052] The multimodal fusion coding module is used to extract and fuse features from LiDAR point cloud data, camera image data, millimeter-wave radar data, and vehicle status data. Specifically, LiDAR point cloud features are projected onto a bird's-eye view plane to obtain point cloud BEV features; camera image features are transformed onto the bird's-eye view plane to obtain image BEV features; millimeter-wave radar target information is projected onto the bird's-eye view plane to obtain radar BEV features; and vehicle status data such as vehicle speed, steering wheel angle, longitudinal acceleration, yaw rate, and gear position are encoded into vehicle status embeddings. Then, feature-level fusion is performed on the point cloud BEV features, image BEV features, and radar BEV features to obtain a unified BEV fusion feature map, used to characterize the tunnel structure, obstacle distribution, personnel and vehicle positions, passable areas, road surface conditions, and environmental risks in the current underground transportation scenario.

[0053] In one example of the present invention, step S20, training the vision-language-action end-to-end decision control model, includes the following steps: S21: Construct an underground transport vehicle driving dataset; the dataset includes manual driving data, safety rule correction data, and underground scene image-text pairing data; wherein, the manual driving data is collected by the underground transport vehicle during manual driving, including multi-source sensor data, vehicle status data, and corresponding manual driving control commands; the safety rule correction data comes from recorded safety constraint trigger segments and control command correction segments, with the safety correction control command output in step S30 serving as the supervision signal; the underground scene image-text pairing data includes multi-source perception data and its corresponding natural language scene description, used for cross-modal semantic alignment training; the underground transport vehicle driving dataset is divided into a training set and a validation set, wherein the training set is used for training the vision-language-action end-to-end driving decision control model, and the validation set is used for model performance evaluation after training; S22: First-stage cross-modal semantic alignment training using the training set: Based on the image-text pairing data of the underground scene, cross-modal semantic alignment training is performed on the multimodal fusion coding module and the cross-modal semantic projection module; the distance between the visual embedding and the corresponding text semantic embedding is constrained by the contrastive learning loss, so that the matching visual-text features are close to each other, and the mismatched visual-text features are far apart; S23: Second-stage end-to-end joint training using the training set: Based on data collected from manual driving and safety rule correction data, the vision-language-action end-to-end decision control model is jointly trained. Multi-source perception data, vehicle status data, underground scene knowledge prompts, transportation task instructions, and underground safety rule prompts are input into the vision-language-action end-to-end decision control model. The longitudinal acceleration, steering angle, and gear instructions are output through the multi-modal fusion encoding module, cross-modal semantic projection module, language reasoning module, and action decoding module. During the training process, manual driving control instructions and safety correction control instructions are used as supervision signals to enable the model to learn the mapping relationship from underground multi-source perception inputs to vehicle control outputs under safety constraints.

[0054] The end-to-end joint training employs a multi-task joint loss function, which includes continuous control quantity regression loss, gear classification loss, and cross-modal semantic alignment loss. The total loss function can be expressed as:

[0055] in, This represents the regression loss of the continuous control quantity. Indicates gear classification loss, This represents the cross-modal semantic alignment loss. , , These are the weighting coefficients for each loss term.

[0056] S24: After training, the performance of the vision-language-action end-to-end decision control model is evaluated based on the validation set. Evaluation metrics include longitudinal acceleration prediction error, steering angle prediction error, gear classification accuracy, and safety default rate. The safety default rate measures the proportion of control commands output by the model that trigger safety verification or safety pruning in step S30. If the model's control error, gear classification accuracy, and safety default rate on the validation set all meet preset requirements, the current model weights are saved as the base model for model lightweighting and vehicle-mounted deployment in step S40.

[0057] In one example of the present invention, the vehicle control command includes at least one of longitudinal acceleration command, steering angle command, and gear command.

[0058] In one example of the present invention, step S30 specifically includes the following steps: S31: Constructing Real-Time Safety Constraint Boundaries: Based on the initial vehicle control command output in step S20, the current underground environment state, vehicle state, and underground safety rules, construct the real-time safety constraint boundaries for the current moment; the real-time safety constraint boundaries include speed constraints, distance constraints, roadway boundary constraints, environmental risk constraints, and vehicle dynamics constraints, forming a set of safety constraints:

[0059] in, Indicates speed constraint, Indicates distance constraints. Indicates the roadway boundary constraints. This indicates environmental risk constraints. This indicates vehicle dynamics constraints.

[0060] Speed ​​constraints determine the current maximum permissible speed based on roadway type, speed-limited areas, curves, slopes, meeting areas, loading / unloading areas, and visibility conditions; distance constraints determine the minimum safe distance between the vehicle and surrounding targets based on personnel, vehicles, obstacles, roadway boundaries, and the location of support structures; roadway boundary constraints determine the drivable area and lateral safe distance based on roadway width, vehicle width, roadway sidewalls, and the location of support structures; environmental risk constraints determine whether the vehicle needs to be speed-limited, decelerated, or stopped based on gas concentration, dust and water mist, water accumulation, slippery road surface, and visibility conditions; vehicle dynamics constraints determine the physical executable range of control commands based on the vehicle's maximum acceleration, maximum braking deceleration, maximum steering angle, minimum turning radius, and gear status.

[0061] S32: Perform safety and feasibility verification on the initial vehicle control commands. The initial vehicle control command is:

[0062] in, Indicates longitudinal acceleration. Indicates the steering angle. This indicates a gear shift command.

[0063] The verification includes: determining whether the longitudinal acceleration exceeds the vehicle's permissible acceleration or braking range; determining whether the steering angle exceeds the maximum steering angle or minimum turning radius constraint; determining whether the gear command is consistent with the current vehicle speed, driving direction, and transportation task status; determining whether the vehicle trajectory predicted based on the initial control command deviates from the passable area; determining whether the distance between the vehicle and personnel, vehicles, obstacles, or alleyway boundaries is less than the safety threshold; and determining whether the current environmental risks, such as gas concentration, visibility, and road surface conditions, allow the vehicle to continue executing the control command. If the initial vehicle control command meets the real-time safety constraint boundary, proceed to step S35; if the initial vehicle control command exceeds the real-time safety constraint boundary, proceed to step S33. S33: Safety trimming and correction of control commands: When the initial vehicle control command exceeds the real-time safety constraint boundary, it is safely trimmed and corrected; For longitudinal acceleration, when the acceleration output by the model may cause the vehicle to exceed the current maximum permissible speed, it is corrected to a control quantity to maintain a safe speed or decelerate; when the distance to people, vehicles or obstacles in front is less than the minimum safe distance, the longitudinal acceleration is corrected to a braking control quantity.

[0064] Regarding the steering angle, when the steering angle output by the model may cause the vehicle to approach the roadway boundary, support structure, or deviate from the passable area, the steering angle is limited to a safe steering range to keep the vehicle within the safe passable area.

[0065] For gear selection commands, the gear selection is corrected when the model outputs a gear that is inconsistent with the current vehicle status or transportation task. For example, switching to reverse gear is prohibited when the vehicle has not come to a complete stop, and continuing to output forward gears is prohibited when the safe distance in front of the vehicle is insufficient. After safety trimming and correction, the safety correction control command is obtained:

[0066] in, , , These represent the longitudinal acceleration, steering angle, and gear command after safety correction, respectively.

[0067] S34: Trigger safety protection control strategy: When a high-risk working condition is detected, or the initial vehicle control command cannot meet the safe driving requirements of the underground transport vehicle through safety trimming, the safety protection control strategy is triggered. The high-risk operating conditions include: the distance to personnel, vehicles or obstacles in front is less than the emergency safety distance; there is a risk of collision with the vehicle; the vehicle is about to deviate from the drivable area or approach the roadway boundary; the gas concentration exceeds the safety threshold; visibility is severely reduced or dust and water mist interference leads to a decrease in perception reliability; the vehicle chassis is in abnormal condition or the control commands conflict with the vehicle status.

[0068] The safety protection control strategy includes at least one of the following: deceleration control, emergency braking, parking assist, audible and visual alarms, remote prompts, and manual takeover requests. This strategy prevents the vehicle from directly executing dangerous actions when the model outputs abnormal control commands or when the environmental risk is high.

[0069] S35: Generate final executable control instructions: If the initial vehicle control instructions meet the real-time safety constraint boundaries, then the initial vehicle control instructions are used as the final executable control instructions; if the initial vehicle control instructions pose a safety risk, then the safety correction control instructions obtained in step S33, or the control instructions generated by the safety protection control strategy in step S34, are used as the final executable control instructions; the final executable control instructions are expressed as follows:

[0070] in, This indicates the final longitudinal acceleration command. This indicates the final steering angle command. This indicates the final gear selection command.

[0071] Based on the current underground environment, vehicle status, transportation tasks, and underground safety rules, a real-time safety constraint boundary is constructed. This real-time safety constraint boundary includes speed constraints, distance constraints, roadway boundary constraints, environmental risk constraints, and vehicle dynamics constraints. Specifically, the speed constraint determines the current maximum permissible speed based on roadway type, speed-limited areas, curves, slopes, meeting areas, and visibility conditions; the distance constraint determines the minimum safe distance based on the positions of personnel, vehicles, obstacles, and roadway boundaries; the roadway boundary constraint determines the drivable area based on roadway width, vehicle width, and the positions of roadway walls and support structures; the environmental risk constraint determines whether the vehicle needs to be speed-limited, decelerated, or stopped based on gas concentration, dust and water mist, water accumulation, slippery road surfaces, and visibility conditions; and the vehicle dynamics constraint determines the physically executable range of control commands based on the vehicle's maximum acceleration, maximum braking deceleration, maximum steering angle, minimum turning radius, and gear status.

[0072] The initial vehicle control commands undergo safety and feasibility verification. If longitudinal acceleration could cause the vehicle to exceed the current maximum permissible speed, it is corrected to maintain a safe speed or decelerate. If the distance to personnel, vehicles, or obstacles ahead is less than the minimum safe distance, the longitudinal acceleration is corrected to a braking control. If the steering angle could cause the vehicle to approach the roadway boundary, support structure, or deviate from the passable area, the steering angle is limited to a safe steering range. If the gear command is inconsistent with the current vehicle status or transportation task, the gear is corrected. If a high-risk condition is detected, or if the initial vehicle control commands cannot meet the safe driving requirements of the underground transport vehicle through safety trimming, safety protection control strategies such as deceleration control, emergency braking, stop holding, audible and visual alarms, remote prompts, or manual takeover requests are triggered. Through the above processing, the final executable control commands are obtained and sent to the chassis actuators of the underground transport vehicle.

[0073] In one example of the present invention, in step S30, the safety constraints of the underground transport vehicle include at least one of the following: minimum safe distance, maximum permissible speed, roadway boundary safe distance, personnel and vehicle avoidance rules, oncoming vehicle avoidance rules, curve deceleration rules, slope driving rules, low-speed passage rules in loading and unloading areas, gas concentration threshold, visibility threshold, and vehicle dynamics constraints.

[0074] In one example of the present invention, in step S30, when the initial vehicle control command does not meet the safety constraints of the underground transport vehicle, at least one of the following processes is performed: Limit vehicle longitudinal acceleration or speed, limit vehicle steering angle, correct vehicle gear, trigger deceleration control, emergency braking, parking hold, audible and visual alarms, remote prompts, and manual takeover requests.

[0075] In one example of the present invention, in step S40, as Figure 3 As shown, the offline incremental training and lightweight update of the vision-language-action end-to-end decision control model based on the key scene data specifically includes the following steps: Step S40 is used to deploy the vision-language-action end-to-end decision control model built and trained in step S20 to the on-board computing platform of the underground transportation vehicle, and to achieve continuous optimization of the model in the complex underground transportation environment through key scenario data backhaul, offline incremental training, safety verification and on-board updates.

[0076] S41: Lightweighting the vision-language-action end-to-end decision control model: To address the constraints of computing power, storage space, power consumption, and real-time control cycle of the onboard computing platform for underground transport vehicles, a lightweight approach is taken for the end-to-end decision control model of vision-language-action. Specifically, at least one of the following methods can be used to reduce model size and inference latency: knowledge distillation, parameter quantization, operator fusion, and efficient parameter fine-tuning.

[0077] Specifically, knowledge distillation is used to transfer the scene understanding and driving decision-making capabilities of the large model to a lightweight student model; parameter quantization is used to reduce the computational precision of model parameters and intermediate features, thereby reducing memory usage and computational overhead; operator fusion is used to merge adjacent computational operators to improve inference efficiency on the vehicle side; and efficient parameter fine-tuning is used to reduce training and update costs while maintaining the capabilities of the backbone model. After lightweight processing, a lightweight vision-language-action decision control model suitable for deployment on vehicle platforms is obtained.

[0078] S42: Deploy the lightweight model to the vehicle: Deploy the lightweight model to the vehicle-mounted computing platform of the underground transport vehicle. The vehicle receives the standardized multi-source perception data set D output in step S10, and generates the initial vehicle control command a according to the inference process in step S20. After the safety verification in step S30, the final executable control command is generated. During operation, the on-board computing platform monitors model inference latency, memory usage, control cycle stability, and control command output frequency to ensure that the system meets the real-time driving control requirements of underground transport vehicles.

[0079] S43: Record key scenario data: During vehicle operation, the on-board unit automatically records key scenario data, including manual takeover segments, safety constraint triggering segments, control command correction segments, abnormal operating condition segments, unreliable perception segments, and segments with low model output confidence. Each key scenario segment includes multi-source sensor data at the corresponding moment, vehicle status data, downhole scenario knowledge, transportation task instructions, the model's original output control instructions, the final control instructions after safety verification, safety rule trigger types, and manual takeover operation records. This data is used to reflect the model's weaknesses and long-tail conditions in the complex real-world downhole transportation environment.

[0080] S44: Transmit key scenario data back to the ground training center: Key scenario data is transmitted back to the ground training center via the communication link from underground to the ground. In cases where underground communication conditions are limited, data transmission can be completed by uploading in batches, resuming interrupted transmissions, or uploading in batches after the task is completed.

[0081] After receiving the data, the ground training center cleans, filters, labels, and classifies the data into scenarios to form an incremental training sample library. Among them, data from manual takeover and data triggered by safety constraints are used as key samples to enhance the model's ability to learn dangerous scenarios, boundary scenarios, and long-tail conditions.

[0082] S45: Offline incremental training of the model: Based on the incremental training sample library, the vision-language-action end-to-end decision control model is trained offline. For manual takeover segments, the control instructions after manual takeover are used as the supervision signal; for safety constraint trigger segments, the control instructions after safety verification and correction are used as the reference signal; for abnormal working condition segments, by supplementing scene descriptions and safety rule prompts, the model's understanding and decision-making ability for complex roadway environments, abnormal traffic conditions and high-risk scenarios are improved. Through offline incremental training, the model can further learn driving decision-making patterns under changes in the underground transportation environment, vehicle status, road conditions, and operating rules.

[0083] S46: Perform safety verification and version management on the incrementally trained model: After incremental training is completed, perform safety verification on the updated model. The safety verification includes the accuracy of scene understanding, the rationality of control commands, collision risk, speed limit compliance, ability to maintain lane boundaries, ability to avoid personnel and vehicles, ability to handle oncoming traffic scenarios, ability to pass through curves and slopes, and safety response capability under abnormal conditions. A deployable model version is generated only after the updated model passes the preset safety test indicators. For each model version, the source of training data, training time, model parameter version, test results, and applicable vehicle range are recorded to achieve traceable management of model versions.

[0084] S47: Update the model that has passed security verification to the vehicle terminal: The model version that has passed security verification is sent to the vehicle terminal of the underground transport vehicle through the communication link or OTA. After receiving the update package, the vehicle terminal performs integrity verification and version verification on the model file. After confirming that there are no errors, the model update is completed.

[0085] The updated model continues to perform autonomous driving control of the underground transport vehicle according to steps S10 to S30, and continuously records new key scenario data during operation, thus forming a closed-loop iterative mechanism of "on-board operation - data backhaul - offline training - safety verification - on-board update".

[0086] In other words, the final executable control commands are sent to the vehicle chassis actuators: these commands are sent to the underground transport vehicle chassis actuators; the chassis actuators include a drive control unit, a brake control unit, a steering control unit, and a transmission control unit, used to realize vehicle acceleration, deceleration, braking, steering, and gear shifting. Safety events and key scenario data are recorded: during the control command verification process, the onboard computing platform synchronously records safety events and key scenario data; the recorded data includes trigger time, vehicle position, original control command, corrected control command, final executable control command, triggered safety rule type, environmental state, vehicle state, and corresponding multi-source perception data fragments. This data is stored as key scenario data on the onboard terminal and transmitted back to the ground training center in step S40 for offline incremental training, verification, and optimization of the model.

[0087] Cross-modal semantic alignment training is performed based on downhole scene image-text pairing data; end-to-end joint training is performed based on manual driving data and safety rule correction data; wherein, the manual driving data includes multi-source sensor data, vehicle status data and manual driving control commands, and the safety rule correction data includes safety constraint trigger segments and their corresponding safety correction control commands.

[0088] In one example of the present invention, in step S40, the lightweight update of the vision-language-action end-to-end decision control model includes at least one of knowledge distillation, parameter quantization, operator fusion, and efficient parameter fine-tuning.

[0089] According to a second aspect of the present invention, an unmanned driving decision control system for underground transport vehicles includes: The multi-source data acquisition and preprocessing module is configured to acquire multi-source sensor data from underground transport vehicles, and perform time synchronization, spatial calibration, noise reduction and enhancement, and standardization preprocessing on the multi-source sensor data to obtain standardized multi-source sensing data. The end-to-end decision control module is configured to build and train a vision-language-action end-to-end decision control model. It inputs the standardized multi-source perception data, vehicle status information, underground scene knowledge, transportation task instructions and underground safety rules into the trained vision-language-action end-to-end decision control model to generate initial vehicle control instructions. The safety verification and protection control module is configured to perform safety verification on the initial vehicle control command according to the safety constraints of the underground transport vehicle, and to perform safety trimming or trigger safety protection control when the initial vehicle control command does not meet the safety constraints of the underground transport vehicle to obtain the final executable control command. The chassis execution module is configured to issue final executable control commands to the chassis execution mechanism of the underground transport vehicle, and record key scenario data corresponding to manual intervention, safety constraint triggering, abnormal working conditions, and control command corrections. Based on the key scenario data, the module performs offline incremental training and lightweight updates to the vision-language-action end-to-end decision control model. The underground transport vehicle includes at least one of the following: a coal mine trackless rubber-tired vehicle, a mining transport vehicle, a personnel transport vehicle, a material transport vehicle, and an equipment transfer vehicle. In other words, the module controls the underground transport vehicle to perform acceleration, braking, steering, or gear shifting according to the final executable control commands. The data feedback and model update module is configured to record key scenario data corresponding to manual intervention, safety constraint triggering, abnormal working conditions, and control command corrections, and perform offline incremental training and lightweight updates to the vision-language-action end-to-end decision control model based on the key scenario data.

[0090] This control system, without altering the basic structure of the chassis actuators of underground transport vehicles, integrates multi-source perception, scene understanding, task reasoning, safety rule constraints, and vehicle control into an end-to-end decision-making and control framework, reducing error accumulation and information loss in traditional modular autonomous driving architectures. Simultaneously, through safety verification and protection control mechanisms, it prevents abnormal control commands output by the model from directly affecting the vehicle chassis, improving the real-time decision-making, safety control, and environmental adaptability of trackless rubber-tired vehicles and other transport vehicles in complex roadway environments in coal mines.

[0091] It should be noted that the unmanned driving decision control system for underground transport vehicles of the present invention can also perform any of the processing described in the previously described unmanned driving decision control method for underground transport vehicles, and the specific details are not repeated here.

[0092] The foregoing description, with reference to preferred embodiments, details an exemplary implementation of the unmanned driving decision-making and control method and system for underground transport vehicles proposed in this invention. However, those skilled in the art will understand that various modifications and alterations can be made to the above specific embodiments without departing from the concept of this invention, and various combinations can be made to the various technical features and structures proposed in this invention without exceeding the protection scope of this invention.

Claims

1. A decision-making and control method for unmanned underground transport vehicles, characterized in that, Includes the following steps: S10: Collect multi-source sensor data from underground transport vehicles and preprocess the multi-source sensor data to obtain standardized multi-source sensing data; S20: Construct and train a vision-language-action end-to-end decision control model. Input the standardized multi-source perception data, vehicle status information, underground scene knowledge, transportation task instructions and underground safety rules into the trained vision-language-action end-to-end decision control model to generate initial vehicle control instructions. S30: Perform a safety verification on the initial vehicle control command based on the safety constraints of the underground transport vehicle, and when the initial vehicle control command does not meet the safety constraints of the underground transport vehicle, perform safety trimming or trigger safety protection control to obtain the final executable control command. S40: The final executable control command is sent to the actuator of the underground transport vehicle chassis, and key scenario data corresponding to manual takeover, safety constraint triggering, abnormal working conditions and control command correction are recorded; based on the key scenario data, the vision-language-action end-to-end decision control model is subjected to offline incremental training and lightweight update.

2. The unmanned driving decision-making and control method for underground transport vehicles according to claim 1, characterized in that, In step S10, the preprocessing of the multi-source sensor data includes: The system performs false echo filtering, outlier removal, and multi-frame consistency filtering on the lidar point cloud data; low-light enhancement, defogging, and distortion correction on the camera image data; clutter suppression and target detection on the millimeter-wave radar data; and filtering, outlier removal, and normalization on the inertial measurement unit data and vehicle CAN bus status data. The multi-source sensor data includes at least two of the following: lidar point cloud data installed on the underground transport vehicle, camera image data, millimeter-wave radar data, inertial measurement unit data, and vehicle CAN bus status data.

3. The unmanned driving decision-making and control method for underground transport vehicles according to claim 1, characterized in that, In step S20, the vision-language-action end-to-end decision control model includes: a multimodal fusion encoding module, a cross-modal semantic projection module, a language reasoning module, and an action decoding module; The multimodal fusion coding module is configured to extract multi-source sensing features from the standardized multi-source sensing data and generate a unified scene representation. The cross-modal semantic projection module is configured to map the unified scene representation to the semantic embedding space of the language reasoning module; The language reasoning module is configured to combine vehicle status information, underground scene knowledge, transportation task instructions, and underground safety rules to generate a driving decision representation. The action decoding module is configured to convert the driving decision representation into vehicle control commands.

4. The unmanned driving decision-making and control method for underground transport vehicles according to claim 3, characterized in that, The multimodal fusion coding module is used to convert lidar point cloud data, camera image data and millimeter-wave radar data into a bird's-eye view plane, to obtain point cloud BEV features, image BEV features and radar BEV features respectively, and to perform feature-level fusion of the point cloud BEV features, image BEV features and radar BEV features to obtain a unified BEV fusion feature map.

5. The unmanned driving decision-making and control method for underground transport vehicles according to claim 3, characterized in that, In step S20, the vision-language-action end-to-end decision control model is trained, including the following steps: S21: Construct an underground transport vehicle driving dataset; the dataset includes manual driving data, safety rule correction data, and underground scene image-text pairing data; wherein, the manual driving data includes multi-source sensor data, vehicle status data, and corresponding manual driving control commands; the safety rule correction data includes: safety constraint trigger segments and control command correction segments, with the safety correction control commands serving as the supervisory signal; the underground scene image-text pairing data includes multi-source perception data and its corresponding natural language scene descriptions; the underground transport vehicle driving dataset is divided into a training set and a validation set; S22: First-stage cross-modal semantic alignment training using the training set: Based on the image-text pairing data of the underground scene, cross-modal semantic alignment training is performed on the multimodal fusion coding module and the cross-modal semantic projection module; the distance between the visual embedding and the corresponding text semantic embedding is constrained by the contrastive learning loss, so that the matching visual-text features are close to each other, and the mismatched visual-text features are far apart; S23: Second-stage end-to-end joint training using the training set: Based on data collected from manual driving and safety rule correction data, the vision-language-action end-to-end decision control model is jointly trained. Multi-source perception data, vehicle status data, underground scene knowledge prompts, transportation task instructions, and underground safety rule prompts are input into the vision-language-action end-to-end decision control model. The model outputs longitudinal acceleration, steering angle, and gear instructions through a multi-modal fusion encoding module, a cross-modal semantic projection module, a language reasoning module, and an action decoding module. During training, manual driving control instructions and safety correction control instructions are used as supervision signals to enable the model to learn the mapping relationship from underground multi-source perception inputs to vehicle control outputs under safety constraints. S24: After training, the performance of the vision-language-action end-to-end decision control model is evaluated based on the validation set. The evaluation metrics include longitudinal acceleration prediction error, steering angle prediction error, gear classification accuracy, and safety default rate.

6. The unmanned driving decision-making and control method for underground transport vehicles according to claim 1, characterized in that, Step S30 specifically includes the following steps: S31: Construct real-time safety constraint boundaries: Based on the initial vehicle control command, the current downhole environment state, the vehicle state, and downhole safety rules, construct the real-time safety constraint boundaries for the current moment; S32: Perform safety and feasibility verification on the initial vehicle control commands. The initial vehicle control command is: in, Indicates longitudinal acceleration. Indicates the steering angle. Indicates gear position command; The verification includes: determining whether the longitudinal acceleration exceeds the vehicle's permissible acceleration or braking range; determining whether the steering angle exceeds the maximum steering angle or minimum turning radius constraint; determining whether the gear command is consistent with the current vehicle speed, driving direction, and transportation task status; determining whether the vehicle trajectory predicted based on the initial control command deviates from the passable area; determining whether the distance between the vehicle and personnel, vehicles, obstacles, or lane boundaries is less than the safety threshold; and determining whether the current gas concentration, visibility, and road surface conditions allow the vehicle to continue executing the control command. If the initial vehicle control command meets the real-time safety constraint boundary, proceed to step S35; if the initial vehicle control command exceeds the real-time safety constraint boundary, proceed to step S33. S33: Safety trimming and correction of control commands: When the initial vehicle control command exceeds the real-time safety constraint boundary, it is safely trimmed and corrected; For longitudinal acceleration, when the acceleration output by the model may cause the vehicle to exceed the current maximum permissible speed, it is corrected to a control quantity to maintain a safe speed or decelerate; when the distance to people, vehicles or obstacles in front is less than the minimum safe distance, the longitudinal acceleration is corrected to a braking control quantity. Regarding the steering angle, when the steering angle output by the model may cause the vehicle to approach the roadway boundary, support structure, or deviate from the passable area, the steering angle is limited to a safe steering range to keep the vehicle within the safe passable area. For gear position commands, the gear position output by the model is corrected when it is inconsistent with the current vehicle status or transportation task. S34: Trigger safety protection control strategy: When a high-risk working condition is detected, or the initial vehicle control command cannot meet the safe driving requirements of the underground transport vehicle through safety trimming, the safety protection control strategy is triggered. S35: Generate final executable control instructions: If the initial vehicle control instructions meet the real-time safety constraint boundaries, then the initial vehicle control instructions are used as the final executable control instructions; if the initial vehicle control instructions have safety risks, then the safety correction control instructions obtained in step S33, or the control instructions generated by the safety protection control strategy in step S34, are used as the final executable control instructions.

7. The unmanned driving decision-making and control method for underground transport vehicles according to claim 1, characterized in that, In step S30, the safety constraints of the underground transport vehicle include at least one of the following: minimum safe distance, maximum permissible speed, roadway boundary safe distance, personnel and vehicle avoidance rules, passing rules, curve deceleration rules, slope driving rules, low-speed passage rules in loading and unloading areas, gas concentration threshold, visibility threshold, and vehicle dynamics constraints.

8. The unmanned driving decision-making and control method for underground transport vehicles according to claim 1, characterized in that, In step S30, when the initial vehicle control command does not meet the safety constraints of the underground transport vehicle, at least one of the following processes is performed: Limit vehicle longitudinal acceleration or speed, limit vehicle steering angle, correct vehicle gear, trigger deceleration control, emergency braking, parking hold, audible and visual alarms, remote prompts, and manual takeover requests.

9. The unmanned driving decision-making and control method for underground transport vehicles according to claim 1, characterized in that, In step S40, the vision-language-action end-to-end decision control model is subjected to offline incremental training and lightweight updates based on the key scene data, specifically including the following steps: S41: Lightweighting the vision-language-action end-to-end decision control model: Considering the constraints of computing power, storage space, power consumption, and real-time control cycle of the on-board computing platform for underground transport vehicles, the vision-language-action end-to-end decision control model is lightweighted. S42: Deploy the lightweight model to the vehicle: Deploy the lightweight model to the vehicle-mounted computing platform of the underground transport vehicle. The vehicle receives the standardized multi-source perception data set D and generates the initial vehicle control command a according to the inference process of step S20. After the safety verification in step S30, the final executable control command is generated. S43: Record key scenario data: During vehicle operation, the on-board unit automatically records key scenario data, including manual takeover segments, safety constraint triggering segments, control command correction segments, abnormal operating condition segments, unreliable perception segments, and segments with low model output confidence. S44: Transmit key scenario data back to the ground training center: Key scenario data is transmitted back to the ground training center through the communication link from underground to the surface; After receiving the data, the ground training center cleans, filters, labels, and classifies the scenarios to form an incremental training sample library; S45: Offline incremental training of the model: Based on the incremental training sample library, the vision-language-action end-to-end decision control model is trained offline. For manual takeover segments, the control instructions after manual takeover are used as the supervision signal; for safety constraint trigger segments, the control instructions after safety verification and correction are used as the reference signal; for abnormal working condition segments, by supplementing scene descriptions and safety rule prompts, the model's understanding and decision-making ability for complex roadway environments, abnormal traffic conditions and high-risk scenarios are improved. S46: Perform safety verification and version management on the incrementally trained model: After incremental training is completed, perform safety verification on the updated model. The safety verification includes scene understanding accuracy, control command rationality, collision risk, speed limit compliance, ability to maintain lane boundaries, ability to avoid personnel and vehicles, ability to handle oncoming traffic scenarios, ability to pass curves and slopes, and safety response capability under abnormal environments. When the updated model passes the preset safety test indicators, a deployable model version is generated. S47: Update the model that has passed security verification to the vehicle terminal: The model version that has passed security verification is sent to the vehicle terminal of the underground transport vehicle through the communication link or OTA. After receiving the update package, the vehicle terminal performs integrity verification and version verification on the model file. After confirming that there are no errors, the model update is completed.

10. A decision-making and control system for unmanned underground transport vehicles, characterized in that, include: The multi-source data acquisition and preprocessing module is configured to acquire multi-source sensor data from underground transport vehicles and preprocess the multi-source sensor data to obtain standardized multi-source sensing data. The end-to-end decision control module is configured to build and train a vision-language-action end-to-end decision control model. It inputs the standardized multi-source perception data, vehicle status information, underground scene knowledge, transportation task instructions and underground safety rules into the trained vision-language-action end-to-end decision control model to generate initial vehicle control instructions. The safety verification and protection control module is configured to perform safety verification on the initial vehicle control command according to the safety constraints of the underground transport vehicle, and to perform safety trimming or trigger safety protection control when the initial vehicle control command does not meet the safety constraints of the underground transport vehicle to obtain the final executable control command. The chassis execution module is configured to send the final executable control commands to the chassis execution mechanism of the underground transport vehicle, and record key scenario data corresponding to manual takeover, safety constraint triggering, abnormal working conditions and control command correction; based on the key scenario data, the vision-language-action end-to-end decision control model is subjected to offline incremental training and lightweight updates.

Images