Multi-mode interaction regulation and control method and system for mine digital twin ventilation system

Abstract

The invention discloses a multi-mode interactive regulation and control method and system for a mine digital twinning ventilation system, and relates to the crossing field of mine ventilation safety and digital twinning technologies. Ventilation equipment operation, underground environment, roadway geometric parameters and personnel position data are acquired, and corresponding acquisition frequency and precision are set; a three-dimensional digital twinborn model is constructed based on the collected data, and real-time updating is carried out through an OPCUA protocol; building an interactive interface containing a visualization unit, a voice unit, a touch unit and a gesture unit; the ventilation state is analyzed, abnormity is marked, and the state within 24 hours is predicted based on historical data; a user generates a regulation and control instruction through the interaction unit, the regulation and control instruction is converted into a control signal for execution after analysis and verification, and a regulation and control effect is fed back. According to the method, digital twinning and multi-mode interaction are fused, real-time collection of mine ventilation data and accurate model mapping are achieved, and the state evaluation accuracy is improved; a multi-interaction mode adapts to a complex scene, and operation convenience is optimized; real-time analysis and prediction and graded early warning guarantee safety, and intelligent regulation and control and effect evaluation improve the reliability of the system.

Description

Technical Field

[0001] This invention relates to the intersection of mine ventilation safety and digital twin technology, and particularly to a multimodal interactive control method and system for mine digital twin ventilation systems. Background Technology

[0002] Mine ventilation systems are the core guarantee system for safe production in underground mines. Their main functions are to deliver fresh air underground, dilute and expel harmful gases and pollutants such as methane and dust, maintain a suitable temperature and humidity environment underground, and provide a safe working space for workers. With the increase in mining depth, the expansion of mining areas, and the advancement of intelligent mine construction, traditional mine ventilation systems have gradually exposed many shortcomings. Traditional ventilation systems rely heavily on manual inspections and experience-based adjustments, resulting in problems such as delayed data acquisition, limited monitoring dimensions, and slow response times. For example, underground sensors are scattered and mostly use wired transmission methods, leading to low data acquisition frequency and susceptibility to interference from the complex underground environment. This makes it difficult to reflect real-time dynamic changes in the ventilation system. In emergencies such as abnormal methane concentrations or sudden drops in airflow, manual judgment and instruction transmission are time-consuming, easily missing the optimal response time and increasing safety risks.

[0003] The rise of digital twin technology has provided technical support for the upgrading of mine ventilation systems. In recent years, some mines have attempted to introduce digital twin technology to build ventilation system models. However, existing digital twin ventilation systems still suffer from shortcomings such as untimely model updates, limited interaction methods, and poor system coordination. Existing models mostly adopt a periodic update mechanism, which is difficult to adapt to dynamic factors such as changes in underground geological structures and equipment aging. The deviation between the model and the actual system gradually increases, leading to a decrease in the accuracy of analysis and prediction. The interaction methods are mostly based on fixed terminal visual interfaces, which only support mouse and keyboard operations. Due to factors such as wearing protective equipment and limited operating space, underground workers find it difficult to conveniently obtain ventilation data or issue control commands. Furthermore, the lack of multimodal interaction methods such as voice and gestures fails to meet the operational needs of different scenarios.

[0004] Furthermore, the existing ventilation system lacks a unified coordination mechanism for processing and executing control commands. When multiple users send commands through different interaction methods, command conflicts easily occur, leading to equipment malfunctions or system malfunctions. Simultaneously, the evaluation of control effectiveness relies heavily on manual post-event review, lacking real-time data feedback and automated evaluation mechanisms, making it difficult to quickly optimize control strategies. With the increasing intelligence level of mines, higher demands are placed on the real-time performance, interactive flexibility, and control reliability of ventilation systems. There is an urgent need to build an integrated system that combines digital twins, multimodal interaction, and intelligent control to address the pain points of traditional ventilation systems and existing digital twin ventilation systems. Summary of the Invention

[0005] The present invention proposes a multimodal interactive control method and system for a mine digital twin ventilation system to solve the problems mentioned in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal interactive control method for a mine digital twin ventilation system, comprising:

[0007] S1. Multi-source acquisition of mine ventilation data: Sensor devices and positioning terminals are deployed underground in the mine to collect ventilation equipment operation data, including fan speed, power, air pressure, and air volume, using current sensors, voltage sensors, air pressure sensors, and air volume sensors respectively; underground environmental data includes temperature, humidity, gas concentration, carbon monoxide concentration, and dust concentration of the mining face, roadways, and chambers; the gas concentration and carbon monoxide concentration are collected at a frequency of 1 time / 5 seconds, and the temperature, humidity, and dust concentration are collected at a frequency of 1 time / 10 seconds; roadway geometric parameter data are collected using a laser scanner; personnel location and movement trajectory data are collected using UWB positioning technology.

[0008] S2. Construct a 3D digital twin model of the mine ventilation system and achieve real-time twin mapping: Based on the roadway geometric parameter data collected in S1, a 3D geometric model of the roadway is constructed using AutoCAD and 3DMax combined with a mine modeling plugin. At the same time, the 3D model of the ventilation equipment is imported to form a preliminary 3D model of the mine ventilation system. The association mapping relationship between various data collected in S1 and the preliminary 3D model is established. A data interface is built through the OPCUA protocol to receive data transmitted from various sensors and positioning terminals in real time. The data is updated to the 3D digital twin model in real time to achieve real-time synchronization between the model and the actual mine ventilation system.

[0009] S3. Build the interactive interface: The interface includes a visual interaction unit, a voice interaction unit, a touch interaction unit, and a gesture interaction unit. The visual interaction unit supports rotation, scaling, and translation of the 3D digital twin model using a mouse and keyboard, displaying tunnel profiles, internal equipment structures, environmental data heatmaps, and dynamic personnel location annotations. Clicking on model elements allows viewing corresponding detailed data. It also supports voice command input and voice feedback. The gesture interaction unit supports 10 gesture recognition methods.

[0010] S4. Mine Ventilation Status Analysis and Prediction: Users can view real-time data through the interactive interface. The system has a built-in ventilation status analysis algorithm to calculate ventilation resistance, air volume distribution, and air pressure distribution parameters in various areas of the mine. Abnormal data is marked by highlighting and flashing. Based on one year of stored historical ventilation data, the system uses an LSTM neural network model to predict the mine ventilation status for the next 24 hours. The prediction results are displayed on the interface in the form of data tables and line graphs.

[0011] S5. Generate and execute control commands: The user generates control commands through any interactive unit on the interface. The system parses the commands, extracts the control object, control parameters, and control time requirements, and verifies the legality and rationality of the commands. After verification, the control commands are converted into control signals that the equipment can recognize and sent to the target ventilation equipment through the data transmission network. After the equipment executes the control operation, the system collects the equipment operation data and downhole environment data in real time and feeds them back to the three-dimensional digital twin model. If the control effect does not meet expectations, the above command generation and execution process is repeated.

[0012] Furthermore, it also includes optimization steps for calculating mine ventilation resistance. When performing mine ventilation state analysis in S4, an improved ventilation resistance calculation method is adopted. When calculating roadway friction resistance, a roadway surface roughness correction coefficient is introduced, and the corrected roadway friction resistance is calculated using the following formula: Among them, h f1 R represents the corrected roadway friction resistance; f This represents the uncorrected roadway frictional resistance, calculated from the roadway geometry, air density, and roadway friction coefficient; Q represents the airflow through the roadway. It represents the surface roughness correction coefficient of the roadway; at the same time, it establishes a database of local resistance coefficients containing 50 common local resistance components, supporting users to correct and supplement the local resistance coefficients.

[0013] Furthermore, it also includes a multimodal interaction command priority sorting step. After S5 parses and verifies the control commands, if more valid control commands are received simultaneously, the command priorities are calculated and sorted using the following function: In the formula, P represents the priority value of the control command; S represents the urgency coefficient of the problem corresponding to the control command; T represents the time sensitivity coefficient of the control command; U represents the operational complexity coefficient of the control command; α, β, Let α + β + α represent the weights of the problem urgency coefficient, time sensitivity coefficient, and operation complexity coefficient, respectively, and let α + β + ...β + α + β + β + β + α + β + β + β + β + β + β + β + =1, α=0.6, β=0.3, =0.1; execute control commands according to priority values ​​from largest to smallest.

[0014] Furthermore, when constructing a 3D digital twin model of the mine ventilation system in S2, a dynamic update and optimization mechanism for the model is set up. The system has a weekly update cycle, during which the latest roadway geometric parameter data and ventilation equipment update data are automatically collected to incrementally update the model. Triggering update conditions include changes in the mine's geological structure, major modifications to the ventilation system, and data exceeding the model's prediction range for 10 consecutive minutes with a deviation rate exceeding 15%. When any of these conditions are met, the system verifies the data and initiates a real-time model update. During the model update process, a data fusion algorithm is used, assigning weights according to the timeliness of the data: data from the most recent month has a weight of 0.7, data from 1-3 months has a weight of 0.2, and data older than 3 months has a weight of 0.1. After the model update, the data deviation rate is verified.

[0015] Furthermore, during the analysis and prediction of mine ventilation status in S4, detailed steps for abnormal operating condition early warning are set up, classifying abnormal operating conditions into three levels: Level 1, Level 2, and Level 3. For Level 1 abnormalities, the visual interactive interface flashes red across the entire screen to mark the abnormal area and pops up an early warning window. The voice interaction unit plays the early warning information in a high-frequency loop, sends a vibration warning signal to the underground personnel positioning terminal, and triggers an audible and visual alarm at the ground monitoring center. For Level 2 abnormalities, the visual interactive interface flashes orange to mark the abnormal area, the voice interaction unit plays early warning information periodically, and sends text warnings to the personnel positioning terminals around the abnormal area. For Level 3 abnormalities, the visual interactive interface marks the abnormal area in yellow, the voice interaction unit broadcasts the abnormal information when the user queries it, and sends a text prompt to the ground monitoring terminal. Each abnormality level corresponds to a preset emergency handling procedure guide, which clearly defines the operating steps, responsible personnel, and handling time limits.

[0016] Furthermore, after the S5 executes the control operation, a control effect evaluation and optimization step is set. The system collects data at 10 minutes, 30 minutes, 1 hour, and 2 hours after the control, and evaluates the control effect from three dimensions: target parameter compliance rate, parameter stability, and energy consumption change rate. An evaluation report containing tables and charts is generated and displayed on the multimodal interactive interface. If the control effect does not meet expectations, the system analyzes the reasons and provides optimization suggestions. The user adjusts the control command according to the suggestions and re-executes the control operation to evaluate the effect again. At the same time, the process data of each control is stored in the control case database, which supports querying.

[0017] Furthermore, when building a multimodal interactive interface in S3, steps for user collaboration and permission management are set up; permissions are allocated based on a role-based access control model: system administrators have full permissions, enabling them to manage users, allocate permissions, configure system parameters, maintain models, and perform various command operations; security management personnel can view security data for the entire area, view early warning information, and generate control suggestions, but do not have the permission to execute control commands.

[0018] A system for implementing the multimodal interactive control method of the mine digital twin ventilation system, comprising:

[0019] The data acquisition module consists of a sensor device group, a positioning terminal group, and a data acquisition controller. The sensor device group includes a current sensor, voltage sensor, wind pressure sensor, and air volume sensor installed at the fan location; and a temperature and humidity sensor, a gas sensor, a carbon monoxide sensor, and a dust sensor installed underground. The positioning terminal group has a built-in UWB positioning module and a data transmission module, supporting 4G / 5G or mining wireless communication networks. The data acquisition controller connects to each device via wired or wireless means, performing data filtering, noise reduction, and format conversion preprocessing.

[0020] The 3D modeling and twin mapping module includes a modeling unit, a data association unit, a real-time update unit, and a model verification unit. The modeling unit uses AutoCAD and 3DMax combined with a professional mining modeling plugin to construct a 3D geometric model of the tunnel and import a 3D model of the ventilation equipment containing its appearance and internal structure. The data association unit receives data through the OPCUA protocol interface, establishes the association between the data and model elements, and stores it in the database. The real-time update unit updates incrementally at a 1-second cycle. The verification unit compares the model's predicted data with the actual collected data daily, and issues an early warning and provides correction suggestions when the deviation rate is >5%.

[0021] Furthermore, it also includes:

[0022] The interactive interface module consists of a visual interaction unit, a voice interaction unit, a touch interaction unit, a gesture interaction unit, and an interaction management unit. The visual interaction unit is developed based on the Unity or Unreal Engine 3D rendering engine and supports the display of tunnel profiles, internal equipment structures, environmental heat maps, and personnel location markings. The voice interaction unit includes a speech recognition and speech synthesis module, and the speech recognition vocabulary contains 500 mine ventilation professional terms. The touch interaction unit is adapted to mobile terminals and supports click, swipe, and zoom gestures. The interaction management unit coordinates the various interaction units, processes user commands, and provides feedback.

[0023] Ventilation Status Analysis and Prediction Module: Includes a data receiving unit, a status analysis unit, a prediction unit, and an early warning unit; the data receiving unit verifies data integrity and format, and requests retransmission when data is missing; the status analysis unit calculates ventilation resistance, airflow distribution, and air pressure distribution, and marks abnormal parameters; the prediction unit uses an LSTM neural network model to predict the ventilation status for the next 24 hours; the early warning unit initiates early warnings according to the level of abnormality and stores the early warning information.

[0024] The control command generation and execution module includes a command receiving unit, a command parsing and verification unit, a command sorting unit, a command conversion unit, a command execution unit, and an effect feedback unit. The command receiving unit converts different types of commands into a unified format; the command parsing and verification unit verifies the legality and rationality of the commands; the command sorting unit sorts the commands using the priority calculation function of claim 3; the command conversion unit converts the commands into equipment control signals; the command execution unit sends signals through a redundant network; and the effect feedback unit evaluates the control effect and provides feedback.

[0025] The system control module includes a module coordination unit, a parameter configuration unit, a user management unit, a log management unit, and a fault diagnosis unit. The module coordination unit allocates system resources; the parameter configuration unit allows administrators to set system parameters; the user management unit manages accounts and permissions based on the RBAC model; the log management unit stores two years of encrypted logs and supports querying and exporting; the fault diagnosis unit monitors the module status, alarms when faults occur, analyzes the causes and provides handling suggestions, and stores fault information.

[0026] Compared with existing technologies, the beneficial effects of this invention are:

[0027] In terms of data acquisition and model building, multi-source sensors and UWB positioning technology are used to achieve comprehensive real-time acquisition of ventilation equipment operation data, underground environment data, and personnel location data. Combined with dynamic updates and data fusion algorithms, the three-dimensional digital twin model can accurately match the actual ventilation system status, providing reliable data support for subsequent analysis, prediction, and control, avoiding analysis errors caused by model deviations, and improving the accuracy of ventilation status assessment.

[0028] In terms of interactive experience and operational flexibility, the multimodal interactive interface supports four interaction methods: visualization, voice, touch, and gesture. It is adapted to complex underground operation scenarios. For example, when underground workers are wearing protective gloves, they can quickly obtain data and issue instructions through gestures or voice without touching the operating equipment, thus solving the limitations of traditional single interaction methods. At the same time, the interaction management unit can coordinate the work of each interaction unit, handle instruction conflicts, ensure that the interaction process is orderly and reliable, and improve the convenience of operation and the efficiency of instruction execution in different scenarios.

[0029] In terms of control and safety assurance, the ventilation status analysis and prediction module can calculate ventilation parameters in real time and predict future conditions, identify abnormal working conditions in advance, and promptly remind relevant personnel through multi-level and multi-modal early warnings, buying time for emergency response; the multi-modal control command generation and execution module realizes command parsing and verification, priority sorting, reliable transmission and effect evaluation, ensuring accurate execution of control commands, and can optimize strategies based on evaluation results to reduce ineffective control; the system control module coordinates the resource allocation of each module, ensures stable system operation, improves the overall intelligence level of the mine ventilation system, reduces safety risks, and provides strong protection for the safety of underground workers and efficient mine production. Attached Figure Description

[0030] Figure 1 This is a schematic block diagram of the multimodal interactive control system of the mine digital twin ventilation system proposed in this invention;

[0031] Figure 2 This is a schematic block diagram of the multimodal interactive control method for a mine digital twin ventilation system proposed in this invention;

[0032] Figure 3 This is a schematic diagram comparing the control effects of the multimodal interactive control method for the mine digital twin ventilation system proposed in this invention on key parameters of the mine.

[0033] Figure 4 This diagram illustrates the deviation rate between the predicted and actual values ​​of the digital twin model in the multimodal interactive control method for the mine digital twin ventilation system proposed in this invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0036] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0037] Reference Figures 1 to 4 A multimodal interactive control method for a mine digital twin ventilation system includes the following steps:

[0038] S1: Multi-source acquisition of mine ventilation data: Sensor devices and positioning terminals are deployed at key locations underground to collect multi-dimensional data; ventilation equipment operation data, including fan speed, power, air pressure, and air volume, are collected using current sensors, voltage sensors, air pressure sensors, and air volume sensors, with a collection frequency of 1 time / second; underground environmental data, including temperature, humidity, gas concentration, carbon monoxide concentration, and dust concentration in mining faces, roadways, and chambers, are collected with gas and carbon monoxide concentrations collected at a frequency of 1 time / 5 seconds, and temperature, humidity, and dust concentrations collected at a frequency of 1 time / 10 seconds; roadway geometric parameter data are collected using a laser scanner with a scanning accuracy of ±5mm, and a comprehensive scan is conducted every 3 months; personnel location and movement trajectory data are collected using UWB positioning technology with a positioning accuracy of ±30cm, and a positioning frequency of 1 time / 2 seconds.

[0039] S2: Constructing a 3D digital twin model of the mine ventilation system and achieving real-time twin mapping: Based on the roadway geometric parameter data collected in S1, a 3D geometric model of the roadway is constructed using AutoCAD and 3DMax combined with a professional mine modeling plugin. At the same time, the 3D model of the ventilation equipment is imported to form a preliminary 3D model of the mine ventilation system. The association mapping relationship between various data collected in S1 and the preliminary 3D model is established. A data interface is built through the OPCUA protocol to receive data transmitted from various sensors and positioning terminals in real time. The data is updated to the 3D digital twin model in real time, and the model update delay is controlled within 100ms, so as to realize the real-time synchronization between the model and the actual mine ventilation system.

[0040] S3: Building a multimodal interactive interface: The interface includes a visual interaction unit, a voice interaction unit, a touch interaction unit, and a gesture interaction unit; the visual interaction unit supports rotation, scaling, and translation of the 3D digital twin model via mouse and keyboard, and can display tunnel profiles, internal equipment structures, environmental data heat maps, and dynamic annotations of personnel positions. Clicking on model elements allows viewing corresponding detailed data; the voice interaction unit has a voice recognition accuracy of over 95% and a recognition response time of no more than 1 second, supporting voice command input and voice information feedback; the touch interaction unit is adapted to mobile terminals, with a touch operation response time of no more than 500ms; the gesture interaction unit supports more than 10 gesture recognitions, with a recognition accuracy of over 90% and a recognition latency of no more than 300ms.

[0041] S4: Mine Ventilation Status Analysis and Prediction: Users can view real-time data through a multimodal interactive interface. The system has a built-in ventilation status analysis algorithm to calculate parameters such as ventilation resistance, air volume distribution, and air pressure distribution in various areas of the mine. Abnormal data is marked by highlighting and flashing. Based on more than one year of stored historical ventilation data, the system uses an LSTM neural network model to predict the mine ventilation status in the next 24 hours with a prediction accuracy of over 85%. The prediction results are displayed on the interface in the form of data tables and line graphs.

[0042] S5: Generate multimodal control commands and execute control operations: The user generates control commands through any interactive unit of the multimodal interface. The system parses the commands, extracts the control object, control parameters, and control time requirements, and verifies the legality and rationality of the commands. After verification, the control commands are converted into control signals that the equipment can recognize and sent to the target ventilation equipment through the data transmission network. After the equipment executes the control operation, the system collects the equipment operation data and downhole environmental data in real time and feeds them back to the three-dimensional digital twin model. If the control effect does not meet expectations, the above command generation and execution process is repeated.

[0043] This invention also includes an optimization step for calculating mine ventilation resistance. During the mine ventilation status analysis in step S4, an improved ventilation resistance calculation method is used. A roadway surface roughness correction coefficient is introduced when calculating roadway friction resistance. The corrected roadway friction resistance is calculated using the following formula: Among them, h f1 R represents the corrected roadway friction resistance (Pa); f This represents the uncorrected roadway frictional resistance (N·s). 2 / m 8 The air volume (m³) is calculated from the tunnel's geometric parameters, air density, and tunnel friction coefficient; Q represents the air volume passing through the tunnel. 3 / s); \xi represents the roadway surface roughness correction coefficient, with a value range of 0.8-1.2. Simultaneously, a database of local resistance coefficients containing more than 50 common local resistance components is established, supporting users in correcting and supplementing local resistance coefficients.

[0044] This invention also includes a multimodal interaction command priority sorting step. After the control command is parsed and verified in S5, if multiple valid control commands are received simultaneously, the command priorities are calculated and sorted using the following function: In the formula, P represents the priority value of the control command; S represents the urgency coefficient of the problem corresponding to the control command; T represents the time sensitivity coefficient of the control command; U represents the operational complexity coefficient of the control command; α, β, Let α + β + α represent the weights of the problem urgency coefficient, time sensitivity coefficient, and operation complexity coefficient, respectively, and let α + β + ...β + α + β + β + β + α + β + β + β + β + β + β + β + =1, α=0.6, β=0.3, =0.1. Control commands are executed according to priority values ​​from largest to smallest.

[0045] In this invention, when constructing a three-dimensional digital twin model of the mine ventilation system in S2, a dynamic update and optimization mechanism for the model is set up. The system has a weekly update cycle, during which the latest roadway geometric parameter data and ventilation equipment update data are automatically collected to incrementally update the model. Triggering update conditions include changes in the mine's geological structure, major modifications to the ventilation system, and key data exceeding the model's prediction range for 10 consecutive minutes with a deviation rate exceeding 15%. When any of these conditions are met, the system verifies the data and initiates real-time model updates. During the model update process, a data fusion algorithm is used, assigning weights according to the timeliness of the data: data within the last month has a weight of 0.7, data within 1-3 months has a weight of 0.2, and data older than 3 months has a weight of 0.1. After the model update, the data deviation rate is verified, and a deviation rate ≤ 5% is considered a qualified update.

[0046] In this invention, when analyzing and predicting the mine ventilation status in S4, a detailed multimodal early warning process for abnormal operating conditions is set up, dividing the abnormal operating conditions into three levels: Level 1, Level 2, and Level 3. For Level 1 abnormalities, the visual interactive interface flashes red across the entire screen to mark the abnormal area and pops up an early warning window. The voice interaction unit plays the early warning information in a high-frequency loop, sends a vibration warning signal to the underground personnel positioning terminal, and triggers an audible and visual alarm at the ground monitoring center. For Level 2 abnormalities, the visual interactive interface flashes orange to mark the abnormal area, the voice interaction unit plays the early warning information periodically, and sends a text warning to the personnel positioning terminals around the abnormal area. For Level 3 abnormalities, the visual interactive interface marks the abnormal area in yellow, the voice interaction unit broadcasts the abnormal information when the user queries, and sends a text prompt to the ground monitoring terminal. Each level of abnormality corresponds to a preset emergency handling procedure guide, which clearly defines the operating steps, responsible personnel, and processing time limits.

[0047] In this invention, after the S5 performs the control operation, a control effect evaluation and optimization step is set. The system collects data at 10 minutes, 30 minutes, 1 hour, and 2 hours after the control, and evaluates the control effect from three dimensions: target parameter compliance rate, parameter stability, and energy consumption change rate. An evaluation report containing tables and charts is generated and displayed on the multimodal interactive interface. If the control effect does not meet expectations, the system analyzes the reasons and provides optimization suggestions. The user adjusts the control command according to the suggestions and re-executes the control operation to evaluate the effect again. At the same time, the process data of each control is stored in the control case database, which supports multi-dimensional query.

[0048] In this invention, when building a multimodal interactive interface in S3, steps for user collaborative interaction and permission management are set up; a role-based access control model is used to allocate permissions: the system administrator has full permissions and can perform user management, permission allocation, system parameter configuration, model maintenance, and various command operations; safety management personnel can view safety-related data for the entire area, view early warning information, and generate control suggestions, but do not have the permission to execute control commands; ventilation dispatchers can view data for their assigned area, generate and execute daily control commands, and control commands for first-level abnormal working conditions require approval; underground workers can only view environmental data for their area and receive early warning information; the system has a collaborative operation module that supports users to annotate and leave messages on the model, records an unalterable operation log for 2 years, and supports encrypted communication between users.

[0049] This invention also discloses a system for a multimodal interactive control method of a mine digital twin ventilation system, comprising:

[0050] The data acquisition module consists of a sensor device group, a positioning terminal group, and a data acquisition controller. Within the sensor device group, current sensors, voltage sensors, wind pressure sensors, and airflow sensors are installed at the fan. The current sensor has a measurement range of 0-500A, the voltage sensor has a measurement range of 0-10kV, the wind pressure sensor has a measurement range of 0-5000Pa, and the airflow sensor has a measurement range of 0-500m³. 3 / s, with a measurement accuracy of ±1%FS; downhole, temperature and humidity sensors, gas sensors, carbon monoxide sensors, and dust sensors are installed. The temperature and humidity sensors have a measurement range of -20-80℃ and 0-100%RH, with a measurement accuracy of ±0.5℃ for temperature and ±3%RH for humidity. The gas sensor has a measurement range of 0-4%CH4, with a measurement accuracy of ±0.05%CH4. The carbon monoxide sensor has a measurement range of 0-100ppm, with a measurement accuracy of ±1ppm. The dust sensor has a measurement range of 0-1000mg / m³. 3Measurement accuracy ±5%; laser scanner scanning speed 1000 points / second, scanning angle 360° (horizontal) × 90° (vertical); positioning terminal group with built-in UWB positioning module and data transmission module, positioning distance range 0-100m, data transmission supports 4G / 5G or mining wireless communication network, battery life not less than 12 hours; data acquisition controller connects to each device through wired or wireless means, performs data filtering, noise reduction, and format conversion preprocessing, processes not less than 1000 data entries per second, storage capacity not less than 16GB, and can cache 24 hours of data when the network is interrupted;

[0051] The 3D modeling and twin mapping module includes a modeling unit, a data association unit, a real-time update unit, and a model verification unit. The modeling unit uses AutoCAD and 3DMax combined with a professional mining modeling plugin to construct a 3D geometric model of the tunnel with an accuracy of ±5mm, and imports a 3D model of the ventilation equipment containing its appearance and internal structure. The data association unit receives data through the OPCUA protocol interface, establishes the association between the data and model elements, and stores it in the database. The real-time update unit updates incrementally at a 1-second cycle with a latency of ≤100ms. The verification unit compares the model's predicted data with the actual collected data daily, and issues an early warning and provides correction suggestions when the deviation rate is >5%.

[0052] This invention also includes:

[0053] The multimodal interactive interface module consists of a visual interaction unit, a voice interaction unit, a touch interaction unit, a gesture interaction unit, and an interaction management unit. The visual interaction unit is developed based on the Unity or Unreal Engine 3D rendering engine, supporting the display of tunnel profiles, internal equipment structures, environmental heat maps, and personnel location markers. Mouse and keyboard operation response time is ≤100ms, and model display frame rate is ≥30fps. The voice interaction unit includes speech recognition and speech synthesis modules. The speech recognition vocabulary contains over 500 mine ventilation professional terms, with a recognition accuracy of ≥95% and a speech synthesis naturalness score of ≥4.5 out of 5. The touch interaction unit is adapted to mobile terminals, supporting click, swipe, and zoom gestures, with a response time of ≤500ms. The gesture interaction unit uses a camera with a resolution ≥1080P and a frame rate ≥30fps, supporting over 10 gesture recognition methods with an accuracy of ≥90% and a recognition latency of ≤300ms. The interaction management unit coordinates the various interaction units, processes user commands, and provides feedback.

[0054] Ventilation Status Analysis and Prediction Module: Includes a data receiving unit, a status analysis unit, a prediction unit, and an early warning unit; the data receiving unit verifies data integrity and format, and requests retransmission when data is missing; the status analysis unit calculates ventilation resistance, air volume distribution, and air pressure distribution, and marks abnormal parameters; the prediction unit uses an LSTM neural network model to predict the ventilation status for the next 24 hours, with a prediction accuracy of ≥85%; the early warning unit initiates early warnings according to the abnormality level and stores the early warning information.

[0055] The multimodal control command generation and execution module includes a command receiving unit, a command parsing and verification unit, a command sorting unit, a command conversion unit, a command execution unit, and an effect feedback unit. The command receiving unit converts different types of commands into a unified format; the command parsing and verification unit verifies the legality and rationality of the commands; the command sorting unit sorts the commands using the priority calculation function of claim 3; the command conversion unit converts the commands into device control signals; the command execution unit sends signals through a redundant network, with a link switching time ≤ 1 second; and the effect feedback unit evaluates the control effect and provides feedback.

[0056] The system control module includes a module coordination unit, a parameter configuration unit, a user management unit, a log management unit, and a fault diagnosis unit. The module coordination unit allocates system resources; the parameter configuration unit allows administrators to set system parameters; the user management unit manages accounts and permissions based on the RBAC model; the log management unit stores two years of encrypted logs and supports querying and exporting; the fault diagnosis unit monitors the module status, alarms when faults occur, analyzes the causes and provides handling suggestions, and stores fault information.

[0057] Example 1: Multimodal Interactive Control of Digital Twin Ventilation System for a Deep Coal Mine with an Annual Output of 5 Million Tons. This example focuses on a deep coal mine with an annual output of 5 million tons. The mine has a mining depth of 1200m and includes 12 main transport roadways and 8 mining faces. The existing ventilation system includes 16 main ventilation fans, 42 roadway air doors, and 28 ventilation windows. Due to the complex underground geological structure and large fluctuations in gas emission, this invention is needed to achieve precise control of the ventilation system.

[0058] 1. Multi-source acquisition of mine ventilation data: Deploying sensor equipment and positioning terminals at key underground locations: ① Ventilation equipment monitoring: Installing a JLK-400A current sensor (measuring range 0-500A, accuracy ±1%FS), a GY1000 voltage sensor (0-10kV, ±1%FS), an FY2000 wind pressure sensor (0-5000Pa, ±1%FS), and an FL-500 air volume sensor (0-500m³ / h) at the motor of each main ventilation fan. 3 / s, ±1%FS), the sensor is connected to the data acquisition controller through a mining intrinsically safe wired transmission module (transmission rate 1Mbps), the acquisition frequency is set to 1 time / second, and the fan speed (converted from current and voltage data), power, wind pressure and air volume data are acquired in real time; ② Underground environmental monitoring: temperature and humidity sensors (SHT35, -20-80℃ / 0-100%RH, accuracy ±0.5℃ / ±3%RH), gas sensors (KGJ23, 0-4%CH4, ±0.05%CH4), carbon monoxide sensors (GTH1000, 0-100ppm, ±1ppm) and dust sensors (GCG1000, 0-1000mg / m 3 The data acquisition frequency is ±5%, with gas and carbon monoxide sensors transmitting data via a mine-use wireless mesh network (transmission distance ≤500m), and a collection frequency of 1 time / 5 seconds. Temperature, humidity, and dust sensors collect data at a frequency of 1 time / 10 seconds. ③ Roadway geometric parameter acquisition: A FAROFocus S70 laser scanner (scanning speed 1000 points / second, scanning angle 360°×90°, accuracy ±5mm) is used to perform a comprehensive scan of all mine roadways every 3 months, with a focus on scanning the advancing area of ​​the tunnel face (supplementary scan every 15 days) to obtain parameters such as roadway cross-sectional dimensions, direction, and slope. ④ Personnel positioning: KJW127 intrinsically safe UWB positioning terminals (positioning distance 0-100m, accuracy ±30cm) are provided for underground workers. The terminals transmit data via a 4G mine base station, with a positioning frequency of 1 time / 2 seconds, and record personnel location and movement trajectory in real time.

[0059] The data acquisition controller uses the KJD200 intrinsically safe mining controller. It connects to the sensors at the fan via wired connection and receives data from environmental sensors and positioning terminals wirelessly. The controller performs filtering (using Kalman filtering algorithm to remove pulse interference), noise reduction (using moving average filtering to process dust concentration fluctuations), and format conversion (converting to JSON format, including device ID, acquisition time, parameter values, and data checksum fields). The controller has a storage capacity of 32GB and can process 1200 data entries per second. When the mining wireless communication network is interrupted, it can cache 24 hours of data and automatically retransmit it after the network is restored.

[0060] 2. Construction of 3D Digital Twin Model and Real-time Twin Mapping

[0061] Model construction based on the roadway geometric parameter data collected in step 1: ① Modeling tools: AutoCAD 2024 was used to draw a two-dimensional cross-sectional view of the roadway, which was then imported into 3DMax 2024 and combined with the MineDesign mining modeling plugin (which supports parametric roadway modeling) to construct a three-dimensional geometric model of the roadway with an accuracy of ±5mm. Simultaneously, three-dimensional models of ventilation equipment such as fans (modeled at a 1:1 scale according to the actual model, including internal structures such as motors, impellers, and casings), air doors (including drive mechanisms and sealing structures), and air windows were imported to form a preliminary three-dimensional model of the mine ventilation system; ② Data association and mapping: A data interface was built using the KEPServerEXOPCUA server, with the server IP set to 192.168.1.100 and port 4840. Data collected by sensors and positioning terminals was associated with model elements. For example, the air pressure data of fan No. 1 was associated with the air pressure display attribute of the "Fan No. 1" component in the model, and personnel positioning data was associated with the coordinate attributes of the "Personnel Icon" in the model; ③ Real-time updates: The model is set to update weekly. Every Monday at 2:00 AM, the latest tunnel scanning data and equipment maintenance records (such as fan blade replacement information) are automatically collected to incrementally update the model. Triggered update conditions include changes in underground geological structure (such as tunnel deformation exceeding 10cm), major modifications to the ventilation system (such as the addition of air doors), and key data (such as air volume in a certain tunnel) exceeding the model's prediction range for 10 consecutive minutes with a deviation rate >15%. When any of these conditions are met, the system first verifies the validity of the data (by comparing data from adjacent sensors in the same area; if the deviation is ≤5%, it is considered valid) before initiating real-time model updates. During updates, a data fusion algorithm is used, with data from the most recent month having a weight of 0.7, data from 1-3 months having a weight of 0.2, and data from more than 3 months having a weight of 0.1. For example, when calculating the tunnel friction coefficient, scanning data from the most recent month is used first. After the update, the predicted air volume is compared with the actual air volume through the model verification unit. If the deviation rate is ≤5%, it is considered qualified. If the deviation rate is >5%, the system prompts "Model update deviation exceeds the limit, tunnel geometric parameters need to be re-collected".

[0062] 3. Multimodal interactive interface construction

[0063] The interface is developed based on a Windows 10 embedded system and deployed on a ground monitoring center terminal (27-inch 4K monitor) and an intrinsically safe mining tablet (10.1-inch, explosion-proof rating ExdIMb). It includes four main interactive units: ① Visual Interaction Unit: Utilizing the Unity 2022 3D rendering engine, it supports mouse wheel zoom (0.1-10x zoom range), mouse drag rotation (360° without blind spots), and keyboard arrow key panning. Users can select "Tunnel Profile View" (displaying wind speed at any cross-section) via a drop-down menu. The system includes: 1) "Aircraft internal structure" (double-clicking the fan model expands to view the real-time animation of the impeller speed); 2) "Environmental data heat map" (gas concentration is displayed using a red-yellow-green gradient, with red representing ≥1.0% CH4); 3) "Personnel location marking" (different colored icons distinguish different types of personnel; clicking the icon displays name, work group, and current location); 4) Voice interaction unit: integrates iFlytek's mining voice recognition engine, with a vocabulary database containing 500+ mine ventilation professional terms (such as "view the gas concentration at face 3" and "open air door 2 to 80% open"). The system features a 96% accuracy rate for voice recognition, a 0.8-second response time, and supports voice command input (e.g., "Adjust the speed of fan #1 to 1500 r / min") and information feedback (the system will announce "Command received, verifying"). The touch interaction unit is a capacitive touchscreen adapted for underground flatbeds, supporting clicks (selecting devices), swipes (switching interfaces), and zoom gestures (scaling models), with a 400ms response time. The interface layout is simplified to large icons (≥5cm×5cm), and it is suitable for operation while wearing gloves. The gesture interaction unit... The ground terminal connects to a Hikvision DS-2CD3T46WD-L camera (1080P resolution, 30fps), supports 12 gesture recognitions (such as "clenched fist" for emergency stop, "swipe up" for viewing historical data), with a recognition accuracy of 92% and a latency of 250ms; the interaction management unit uses an STM32F407 processor to coordinate the work of various interaction units. For example, when the user sends commands via voice and touch simultaneously, the command received first is executed first, and a voice prompt is given that "touch command has been executed, voice command is invalid".

[0064] Simultaneously, multi-user collaborative interaction and permission management are set up: Based on the RBAC model, four types of roles are assigned: ① System Administrator (1 person): Has full permissions and can add users through the ground terminal (enter name, employee number, contact information, and set initial password), assign permissions (such as granting command execution permissions to ventilation dispatchers), configure system parameters (such as adjusting the first-level gas warning threshold), and maintain the model (manually trigger model updates); ② Safety Management Personnel (3 people): Can view safety data for the entire mine (such as historical curves of gas concentration in each area), view warning information (export warning records in Excel spreadsheets), and generate control suggestions (the system automatically generates text such as "suggest increasing the ventilation volume in roadway 4"), but has no command execution permissions; ③ Ventilation Dispatchers (5 people): Can view data for their assigned area (such as the east wing mining area), generate daily control instructions (such as adjusting the opening degree of ventilation windows), and execute first-level abnormal working condition control instructions, which require approval from the system administrator (approval requests are sent via encrypted communication, and instructions become invalid if not approved within 5 minutes); ④ Underground Workers (50 people): Can only view environmental data of their area (such as the current gas concentration) and receive warning information (pop-up window + vibration alert) through the underground tablet. The system includes a collaborative annotation module, which allows users to annotate the model (e.g., safety managers annotate "No. 2 ventilation door seal aging") and leave messages. The operation log is stored using AES-256 encryption and is retained for 2 years. It supports querying by user account and operation type (e.g., "instruction generation" and "parameter modification"). Users transmit information through a mine-use encrypted communication module (encryption algorithm SM4) to prevent data leakage.

[0065] 4. Ventilation Status Analysis and Prediction

[0066] Users can view real-time data through a multimodal interactive interface: ① Status Analysis: The system has a built-in ventilation resistance calculation algorithm, which uses an improved friction resistance calculation method. The formula is as follows: , where R f Based on the tunnel's geometric parameters (cross-sectional area S, perimeter P) and air density ρ (taken as 1.2 kg / m³), 3 The friction coefficient λ (0.012 for concrete tunnels) is calculated using the following formula: For example, calculate the air volume of East Wing No. 1 tunnel (cross-section 4m×3m, perimeter 14m, air volume Q=80m³). 3 Frictional resistance (r / s, ξ=0.9): First calculate R f =0.012×1.2×14 / (8×(12)^3)=0.012×1.2×14 / (8×1728)=0.2016 / 13824=1.458×10^-5N·s 2 / m 8 Calculate h again f1=1.458×10^-5×80²×0.9=1.458×10^-5×6400×0.9=0.832Pa; The system simultaneously calculates air volume distribution (using nodal wind pressure method) and air pressure distribution, and marks abnormal data (such as gas concentration ≥0.8%CH4) with red highlighting and flashing twice per second; ② Abnormal warning: When a first-level abnormality (gas >1.5%CH4, carbon monoxide >50ppm) occurs, the visual interface flashes red across the entire screen and a warning window pops up (displaying the abnormal location and value), and the voice unit broadcasts "Gas exceeds limit at No. 3 mining face in the east wing, please evacuate" every 5 seconds, sends a vibration warning (frequency 2 times / second) to the underground personnel positioning terminal, and triggers the audible and visual alarm (decibel ≥110dB) at the ground monitoring center; When a second-level abnormality (gas 1.0%-1.5%CH4, carbon monoxide 30-50ppm) occurs, the interface flashes orange, and the voice unit broadcasts every 30 seconds. It broadcasts a warning every second, sending a text alert to terminals within 50m of the abnormal area; when a level 3 anomaly occurs (0.8%-1.0% CH4 methane, 10-30ppm carbon monoxide), it is highlighted in yellow on the interface, and a voice broadcast is given when the user queries, while a text prompt is sent to the ground terminal; each level corresponds to an emergency procedure guide, for example, the level 1 anomaly guide clearly states that "ventilation dispatch personnel must turn on the backup fan within 5 minutes, and underground personnel must evacuate along the disaster avoidance route, responsible person: Zhang XX"; ③ Prediction: Based on the historical ventilation data of the past 18 months (including fan operating parameters and environmental data), an LSTM neural network model (input features: air volume, methane concentration, and temperature of the past 7 days, 64 neurons in the hidden layer, 100 iterations) is used to predict the ventilation status of the next 24 hours. The prediction results are displayed in a table (including time, predicted air volume, and predicted methane concentration) and a line graph (comparing historical data with predicted data), with a prediction accuracy of 88%.

[0067] 5. Generation and execution of multimodal control commands

[0068] ① Command Generation and Verification: Users generate commands through any interactive unit. For example, a ventilation dispatcher says via voice, "Adjust the opening degree of damper #2 to 80%." The system parses and extracts the control object (damper #2, ID: FM-02), control parameters (opening degree 80%), and control time requirement (immediate execution). It verifies legality by querying the equipment database to confirm that FM-02 is registered, verifying the parameters (damper opening degree is allowed within the range of 0-100%, 80% is within the range), and querying user permissions (ventilation dispatchers have daily control permissions). Justification: Analyze the impact of the adjustment on the air volume of the adjacent No. 3 roadway (through model simulation, the air volume fluctuation is ≤5%, which is within the safe range); ② Command sequencing: If three valid commands are received simultaneously: Command A (gas over-limit control of No. 1 fan, S=12, T=9, U=8), Command B (routine adjustment of No. 2 damper, S=6, T=5, U=6), Command C (parameter optimization of No. 3 ventilation window, S=3, T=3, U=4), substituting into the formula P=0.6×S+0.3×T+0.1×U, we get PA=0.6×12+0.3 ×9+0.1×8=7.2+2.7+0.8=10.7, PB=0.6×6+0.3×5+0.1×6=3.6+1.5+0.6=5.7, PC=0.6×3+0.3×3+0.1×4=1.8+0.9+0.4=3.1. Sorting P from largest to smallest as A>B>C, instruction A is executed first; ③ Instruction conversion and execution: The instruction conversion unit converts signals according to the equipment type. For damper No. 2, a PWM pulse signal is used (frequency 50Hz, duty cycle 80% corresponding to 80% opening degree). The data is transmitted via a mining industrial Ethernet (main link). The network adopts a redundant design, automatically switching to the backup link when the main link is interrupted (switching time 0.8 seconds). The instruction execution unit receives the "execution completed" signal from the damper and updates the model in real time. ④ Effect evaluation: Data is collected at 10 minutes, 30 minutes, 1 hour, and 2 hours after the control is implemented to evaluate the target parameter compliance rate (e.g., gas concentration ≤0.8%CH4 is considered compliant), parameter stability (airflow fluctuation range), and energy consumption change rate (fan power change). An evaluation report is generated, as shown in the table below.

[0069]

[0070] 6. System Control

[0071] The module coordination unit utilizes a CPU (Intel Xeon E3-1230), memory (16GB DDR4), and network bandwidth (1000Mbps) to dynamically allocate resources. For example, during 3D model updates, the coordination data acquisition module prioritizes providing tunnel scanning data, suspending non-urgent dust data acquisition (pause duration 30 minutes), and the coordination interactive interface module reduces the model rendering resolution to 2K to minimize resource consumption. The parameter configuration unit allows system administrators to set parameters through a visual interface: data acquisition frequency (e.g., adjusting the gas concentration acquisition frequency to 1 time / 3 seconds), model update cycle (e.g., changing to a regular update every 5 days), early warning threshold (e.g., adjusting the first-level gas early warning threshold to 1.4% CH4), and priority weights (e.g., adjusting α=0.7, β=0.2, γ=0.1). These parameters take effect immediately after verification. The fault diagnosis unit monitors the operating parameters of each module. If it detects that the data acquisition module has "no data output from the sensor," it analyzes the cause (abnormal power supply, loose wiring) and provides suggestions such as "check the sensor power supply voltage (should be 12V), reconnect the cable," while simultaneously issuing an audible and visual alarm and storing the fault information in the fault database.

[0072] Example 2: Multimodal interactive control of digital twin ventilation system for a mine with an annual output of 2 million tons of metal ore. This example is for a copper mine with an annual output of 2 million tons, a mining depth of 800m, 8 main roadways and 5 mining faces underground, and a ventilation system including 8 main ventilation fans and 24 air doors. Due to the high dust concentration of metal ore and the large heat dissipation of underground equipment, this invention is needed to achieve precise temperature control and dust control of the ventilation system.

[0073] 1. Multi-source acquisition of mine ventilation data

[0074] ① Ventilation equipment monitoring: Install current, voltage, wind pressure, and air volume sensors (same as in Example 1) at the main ventilation fan, with a data acquisition frequency of 1 time / second, and transmit the data via mining fiber optic (strong anti-interference capability); ② Underground environment monitoring: Focus on strengthening dust monitoring, install two GCG1000 dust sensors (10m apart) at the mining face, with a data acquisition frequency of 1 time / 5 seconds; temperature and humidity sensors (focusing on temperature monitoring, as local temperatures can reach 40℃ due to equipment heat dissipation) with a data acquisition frequency of 1 time / 10 seconds; and gas sensors (for low gas emission in metal mines, with a range of 0-2% CH4) with a data acquisition frequency of 1 time / 5 seconds; ③ Roadway geometric parameter acquisition: Use the same laser scanner as in Example 1, conducting a comprehensive scan every 2 months, and supplementing the mining face with a scan every 10 days; ④ Personnel positioning: Use the same UWB positioning terminal as in Example 1, with a positioning frequency of 1 time / 2 seconds, and add a "equipment maintenance personnel" positioning label (to distinguish them from ordinary workers).

[0075] 2. Construction of 3D Digital Twin Model and Real-time Twin Mapping

[0076] ① Modeling: Considering the impact of metal ore transportation equipment (such as mine cars) on ventilation, a 3D model of the mine car (modeled according to the actual model) is added to the model, and the interaction between the mine car's driving route and the ventilation airflow is marked; ② Data association: The dust sensor data is associated with the dust concentration attribute of the "mining face" in the model, and the mine car position data is associated with the coordinates of the "mine car icon"; ③ Real-time update: The regular update cycle is 1 week, and the trigger update condition is added as "the mining face advances more than 5m". When data is fused, the weight of the data in the past month is 0.7. The model verification deviation rate is ≤5% to be qualified. If the scanning data deviation is caused by the mine car blocking the view, the prompt "the obstacles in the scanning area need to be cleared and the data is re-collected" is displayed.

[0077] 3. Multimodal interactive interface construction

[0078] ① Visual Interaction: A new "Dust Diffusion Simulation" function has been added, which uses particle effects to display the diffusion path of dust in the tunnel and supports viewing the impact of mine car movement on dust diffusion; ② Voice Interaction: A new "Dust Concentration Query" terminology has been added (e.g., "View dust concentration at mining surface No. 5"), with a recognition accuracy of 95%; ③ Touch Interaction: The explosion-proof rating of the underground flat panel has been upgraded to ExdIMa to adapt to the humid environment of metal mines; ④ Access Control: A new "Equipment Maintenance Personnel" role has been added, which can view ventilation equipment fault data and receive equipment maintenance warnings, but has no control permissions; The collaborative annotation module supports "Equipment Fault Annotation" (e.g., annotating "Bearing temperature of fan No. 1 is too high").

[0079] 4. Ventilation Status Analysis and Prediction

[0080] ① State Analysis: The frictional resistance calculation is the same as in Example 1. For example, calculate the frictional resistance of the No. 5 mining face roadway (cross-section 3m×3m, perimeter 12m, air volume Q=60m³). 3 / s, ξ=1.0), Rf=0.012×1.2×12 / (8×(9)^3)=0.1728 / (8×729)=0.1728 / 5832=2.96×10^-5N·s 2 / m 8 hf1 = 2.96 × 10^-5 × 60² × 1.0 = 2.96 × 10^-5 × 3600 = 0.1066 Pa; ② Anomaly warning: The first-level anomaly adds "temperature > 40℃, dust concentration > 200mg / m³", which triggers the linkage of the underground spray dust suppression device when the warning is issued; ③ Prediction: The LSTM model input feature adds "mining truck driving frequency", and the prediction accuracy reaches 86%.

[0081] 5. Generation and execution of multimodal control commands

[0082] ① Example of an instruction: Equipment maintenance personnel trigger "No. 1 fan temperature query" via gesture ("drawing a circle"), and ventilation dispatch personnel send "Activate the No. 5 mining face dust suppression spray device, and simultaneously adjust the opening degree of No. 3 air door to 70%" via touch control; ② Instruction sequence: If instruction D (dust over-limit control, S=10, T=8, U=7) and instruction E (daily air door adjustment, S=5, T=6, U=5) are received simultaneously, substitute into the formula P=0.6×S+0.3×T+0.1 ×U, we calculate PD = 0.6×10 + 0.3×8 + 0.1×7 = 6 + 2.4 + 0.7 = 9.1, PE = 0.6×5 + 0.3×6 + 0.1×5 = 3 + 1.8 + 0.5 = 5.3. Sorting P from largest to smallest, D > E, so D is prioritized. ③ Effect evaluation: Data is collected 10 minutes, 30 minutes, 1 hour, and 2 hours after the adjustment to evaluate the target parameter compliance rate, parameter stability, and energy consumption change rate, generating an evaluation report as shown in the table below:

[0083]

[0084] 6. System Control

[0085] The module coordination unit prioritizes allocating resources to the "Mine Car-Ventilation Interaction Simulation" during peak mining periods (when mine cars travel frequently); the fault diagnosis unit adds a "Dust Sensor Blockage" fault type, with the suggested solution being "regularly clean the sensor sampling port"; the log management unit adds a "Mine Car Driving Record" field to facilitate analysis of the impact of mine cars on ventilation.

[0086] Data Interpretation

[0087] The evaluation data in both examples conform to the actual operating patterns of the mine: In Example 1 (coal mine), although increasing the speed of fan No. 1 led to an 8% increase in energy consumption, the gas concentration quickly dropped from 1.6% to 0.7%, reflecting the "safety first" control objective and meeting the core needs of coal mines for gas control, thus avoiding safety accidents caused by excessive gas levels; the control of air door No. 2 and air window No. 3 did not increase energy consumption, and the parameter stability was controlled within ±3%, indicating that the control operation was precise and would not cause significant fluctuations in the ventilation system, ensuring a stable underground environment. In Example 2 (metal mine), the activation of the spray device reduced the dust concentration from 220mg / m³ to 90mg / m³, significantly improving the underground working environment and reducing the impact of dust on personnel health; the temperature of air door No. 3 decreased by 4℃ after adjustment, alleviating the high temperature problem caused by equipment heat dissipation and ensuring stable equipment operation and comfortable operation for personnel; the increase in the speed of fan No. 1 reduced the bearing temperature, preventing equipment damage due to high temperature and extending the service life of the equipment. In both embodiments, the target parameter compliance rate was 100%, verifying that the control strategy of the present invention can effectively solve the core pain points of ventilation systems in different types of mines. Moreover, the data did not exceed the physical limits of the equipment (such as the fan speed not exceeding 2000 r / min, and the damper opening degree not exceeding 0-100%) or violate natural laws (such as the dust concentration not dropping suddenly for no reason, and the temperature not fluctuating significantly due to the damper fine adjustment). It is completely in line with the actual operation logic and common sense of mines.

[0088] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal interactive control method for a mine digital twin ventilation system, characterized in that, include: S1. Multi-source acquisition of mine ventilation data: Sensor devices and positioning terminals are deployed underground in the mine to collect ventilation equipment operation data, including fan speed, power, air pressure, and air volume, using current sensors, voltage sensors, air pressure sensors, and air volume sensors respectively; underground environmental data includes temperature, humidity, gas concentration, carbon monoxide concentration, and dust concentration of the mining face, roadways, and chambers; the gas concentration and carbon monoxide concentration are collected at a frequency of 1 time / 5 seconds, and the temperature, humidity, and dust concentration are collected at a frequency of 1 time / 10 seconds; roadway geometric parameter data are collected using a laser scanner; personnel location and movement trajectory data are collected using UWB positioning technology. S2. Construct a 3D digital twin model of the mine ventilation system and achieve real-time twin mapping: Based on the roadway geometric parameter data collected in S1, a 3D geometric model of the roadway is constructed using AutoCAD and 3DMax combined with a mine modeling plugin. At the same time, the 3D model of the ventilation equipment is imported to form a preliminary 3D model of the mine ventilation system. The association mapping relationship between various data collected in S1 and the preliminary 3D model is established. A data interface is built through the OPCUA protocol to receive data transmitted from various sensors and positioning terminals in real time. The data is updated to the 3D digital twin model in real time to achieve real-time synchronization between the model and the actual mine ventilation system. S3. Build the interactive interface: The interface includes a visual interaction unit, a voice interaction unit, a touch interaction unit, and a gesture interaction unit. The visual interaction unit supports rotation, scaling, and translation of the 3D digital twin model using a mouse and keyboard, displaying tunnel profiles, internal equipment structures, environmental data heatmaps, and dynamic personnel location annotations. Clicking on model elements allows viewing corresponding detailed data. It also supports voice command input and voice feedback. The gesture interaction unit supports 10 gesture recognition methods. S4. Mine Ventilation Status Analysis and Prediction: Users can view real-time data through the interactive interface. The system has a built-in ventilation status analysis algorithm to calculate ventilation resistance, air volume distribution, and air pressure distribution parameters in various areas of the mine. Abnormal data is marked by highlighting and flashing. Based on one year of stored historical ventilation data, the system uses an LSTM neural network model to predict the mine ventilation status for the next 24 hours. The prediction results are displayed on the interface in the form of data tables and line graphs. S5. Generate control instructions and execute control operations: Users generate control instructions through any interactive unit on the interactive interface. The system parses the instructions, extracts the control object, control parameters, and control time requirements, and verifies the legality and rationality of the instructions. After successful verification, the control command is converted into a control signal that the equipment can recognize and sent to the target ventilation equipment through the data transmission network. After the equipment performs the control operation, the system collects the equipment operation data and downhole environment data in real time and feeds them back to the three-dimensional digital twin model. If the control effect does not meet expectations, the above command generation and execution process is repeated.

2. The multimodal interactive control method for a mine digital twin ventilation system according to claim 1, characterized in that, It also includes an optimization step for calculating mine ventilation resistance. When performing mine ventilation status analysis in S4, an improved ventilation resistance calculation method is adopted. When calculating roadway friction resistance, a roadway surface roughness correction coefficient is introduced, and the corrected roadway friction resistance is calculated using the following formula: Among them, h f1 R represents the corrected roadway friction resistance; f This represents the uncorrected roadway frictional resistance, calculated from the roadway geometry, air density, and roadway friction coefficient; Q represents the airflow through the roadway. It represents the surface roughness correction coefficient of the roadway; at the same time, it establishes a database of local resistance coefficients containing 50 common local resistance components, supporting users to correct and supplement the local resistance coefficients.

3. The multimodal interactive control method for a mine digital twin ventilation system according to claim 1, characterized in that, It also includes a multimodal interaction command priority sorting step. After S5 parses and verifies the control commands, if more valid control commands are received simultaneously, the command priorities are calculated and sorted using the following function: In the formula, P represents the priority value of the control command; S represents the urgency coefficient of the problem corresponding to the control command; T represents the time sensitivity coefficient of the control command; U represents the operational complexity coefficient of the control command; α, β, Let α + β + α represent the weights of the problem urgency coefficient, time sensitivity coefficient, and operation complexity coefficient, respectively, and let α + β + ...β + α + β + β + β + α + β + β + β + β + β + β + β + =1, α=0.6, β=0.3, =0.1; execute control commands according to priority values ​​from largest to smallest.

4. The multimodal interactive control method for a mine digital twin ventilation system according to claim 1, characterized in that, When constructing a 3D digital twin model of the mine ventilation system in S2, a dynamic update and optimization mechanism for the model is set up. The system has a weekly update cycle, during which the latest roadway geometric parameters and ventilation equipment update data are automatically collected to incrementally update the model. Triggering update conditions include changes in the mine's geological structure, major modifications to the ventilation system, and data exceeding the model's prediction range for 10 consecutive minutes with a deviation rate exceeding 15%. When any of these conditions are met, the system verifies the data and initiates a real-time model update. During the model update process, a data fusion algorithm is used, assigning weights according to the timeliness of the data: data from the most recent month has a weight of 0.7, data from 1-3 months has a weight of 0.2, and data older than 3 months has a weight of 0.

1. After the model update, the data deviation rate is verified.

5. The multimodal interactive control method for a mine digital twin ventilation system according to claim 1, characterized in that, When conducting mine ventilation status analysis and prediction in S4, detailed steps for abnormal operating condition early warning are set up, classifying abnormal operating conditions into three levels: Level 1, Level 2, and Level 3. For Level 1 abnormalities, the visual interactive interface flashes red across the entire screen to mark the abnormal area and pops up an early warning window. The voice interaction unit plays the early warning information in a high-frequency loop, sends a vibration early warning signal to the underground personnel positioning terminal, and triggers an audible and visual alarm at the ground monitoring center. For Level 2 abnormalities, the visual interactive interface flashes orange to mark the abnormal area, the voice interaction unit plays early warning information periodically, and sends text warnings to the personnel positioning terminals around the abnormal area. For Level 3 abnormalities, the visual interactive interface marks the abnormal area in yellow, the voice interaction unit broadcasts the abnormal information when the user queries it, and sends text prompts to the ground monitoring terminal. Each abnormality level corresponds to a preset emergency handling procedure guide, which clearly defines the operating steps, responsible personnel, and handling time limits.

6. The multimodal interactive control method for a mine digital twin ventilation system according to claim 1 is characterized in that, After the S5 system performs a control operation, it sets up steps for evaluating and optimizing the control effect. The system collects data at 10 minutes, 30 minutes, 1 hour, and 2 hours after the control operation, and evaluates the control effect from three dimensions: target parameter compliance rate, parameter stability, and energy consumption change rate. It generates an evaluation report with tables and charts and displays it on the multimodal interactive interface. If the control effect does not meet expectations, the system analyzes the reasons and provides optimization suggestions. The user adjusts the control command according to the suggestions and re-executes the control operation to evaluate the effect again. At the same time, the process data of each control operation is stored in the control case database, which supports querying.

7. The multimodal interactive control method for a mine digital twin ventilation system according to claim 1 is characterized in that, When building a multimodal interactive interface in S3, set up steps for user collaboration and permission management; assign permissions based on a role-based access control model: system administrators have full permissions, including user management, permission allocation, system parameter configuration, model maintenance, and various command operations; security management personnel can view security data for the entire area, view early warning information, and generate control suggestions, but do not have the permission to execute control commands.

8. A system for implementing the multimodal interactive control method of a mine digital twin ventilation system according to any one of claims 1-7, characterized in that, include: The data acquisition module consists of a sensor device group, a positioning terminal group, and a data acquisition controller. The sensor device group includes a current sensor, voltage sensor, wind pressure sensor, and air volume sensor installed at the fan location; and a temperature and humidity sensor, a gas sensor, a carbon monoxide sensor, and a dust sensor installed underground. The positioning terminal group has a built-in UWB positioning module and a data transmission module, supporting 4G / 5G or mining wireless communication networks. The data acquisition controller connects to each device via wired or wireless means, performing data filtering, noise reduction, and format conversion preprocessing. 3D modeling and twin mapping module: includes modeling unit, data association unit, real-time update unit, and model verification unit; The modeling unit uses AutoCAD and 3DMax combined with a professional mining modeling plugin to construct a three-dimensional geometric model of the tunnel and import a three-dimensional model of the ventilation equipment containing its appearance and internal structure. The data association unit receives data through the OPCUA protocol interface, establishes the association between the data and model elements, and stores it in the database. The real-time update unit updates incrementally at a 1-second cycle. The verification unit compares the model prediction data with the actual collected data every day, and issues an early warning and provides correction suggestions when the deviation rate is >5%.

9. The system for multimodal interactive control of a mine digital twin ventilation system according to claim 8, characterized in that, Also includes: The interactive interface module consists of a visual interaction unit, a voice interaction unit, a touch interaction unit, a gesture interaction unit, and an interaction management unit. The visual interaction unit is developed based on the Unity or Unreal Engine 3D rendering engine and supports the display of tunnel profiles, internal equipment structures, environmental heat maps, and personnel location markings. The voice interaction unit includes a speech recognition and speech synthesis module, and the speech recognition vocabulary contains 500 mine ventilation professional terms. The touch interaction unit is adapted to mobile terminals and supports click, swipe, and zoom gestures. The interaction management unit coordinates the various interaction units, processes user commands, and provides feedback.

10. The system for multimodal interactive control of a mine digital twin ventilation system according to claim 8, characterized in that, Also includes: Ventilation Status Analysis and Prediction Module: Includes a data receiving unit, a status analysis unit, a prediction unit, and an early warning unit; the data receiving unit verifies data integrity and format, and requests retransmission when data is missing; the status analysis unit calculates ventilation resistance, airflow distribution, and air pressure distribution, and marks abnormal parameters; The prediction unit uses an LSTM neural network model to predict the ventilation status for the next 24 hours; The early warning unit activates early warnings according to the level of abnormality and stores the early warning information; The control command generation and execution module includes a command receiving unit, a command parsing and verification unit, a command sorting unit, a command conversion unit, a command execution unit, and an effect feedback unit. The command receiving unit converts different types of commands into a unified format; the command parsing and verification unit verifies the legality and rationality of the commands; the command sorting unit sorts the commands using the priority calculation function of claim 3; the command conversion unit converts the commands into equipment control signals; the command execution unit sends signals through a redundant network; and the effect feedback unit evaluates the control effect and provides feedback. The system control module includes a module coordination unit, a parameter configuration unit, a user management unit, a log management unit, and a fault diagnosis unit. The module coordination unit allocates system resources; the parameter configuration unit allows administrators to set system parameters; the user management unit manages accounts and permissions based on the RBAC model; the log management unit stores two years of encrypted logs and supports querying and exporting; the fault diagnosis unit monitors the module status, alarms when faults occur, analyzes the causes and provides handling suggestions, and stores fault information.

Images