A well shaft coal bunker blockage positioning method based on multi-modal perception and digital twinning

Abstract

The present application belongs to the field of intelligent monitoring of mines, and particularly relates to a method for locating blockage of underground coal bunker based on multi-modal perception and digital twinning, comprising inputting multi-modal sensor array deployment parameters and underground working condition data, performing spatio-temporal registration and dynamic weighted fusion processing to obtain multi-modal perception data set under unified spatio-temporal coordinate system, etc. The present application constructs a multi-modal sensor array covering geometric morphology, physical stress, material characteristics and environmental state, ensuring the integrity and reliability of the perception data; utilizes the Doppler velocity characteristics of the fusion data of 3D laser radar and 4D millimeter wave radar to realize separation of dynamic and static point clouds, and identifies the wall-hanging features through sequential imaging and deep neural network classification method, supports identification of hidden blockage precursors such as thin layer wall-hanging and initial arching, and provides accurate blockage point position information for subsequent unblocking operation.

Description

Technical Field

[0001] This invention belongs to the field of intelligent monitoring in mines, and specifically relates to a method for locating blockages in underground coal bunkers based on multimodal perception and digital twins. Background Technology

[0002] As a key hub in the coal storage and transportation system, the coal bunker's internal status perception and blockage location capabilities directly affect the efficiency of blockage removal and operational safety. Currently, the industry mainly uses solutions such as single-point level gauges, three-dimensional lidar, distributed fiber optic grating sensors, and simple stacking of multiple sensors to address the issues of coal bunker status perception and blockage location. Single-point level gauges are installed on the top or side wall of a silo using rotary paddle, weighted, or ultrasonic sensors to measure the height of a single point of material level. Although they are simple in structure, low in cost, and technologically mature, they can only provide silo level information for a limited number of points and cannot obtain the three-dimensional shape of the entire silo or the distribution of material hanging on the wall. Furthermore, they cannot accurately locate the location and shape of blockages. The three-dimensional lidar scanning solution acquires three-dimensional point cloud data of the material surface by rotating lidar installed on the top of the silo. Although it can build a three-dimensional model of the silo and realize the calculation of material level and volume and the identification of material surface shape, the signal attenuation is severe under the conditions of dust and water mist in the well. It cannot penetrate the material to obtain deep information and can only reflect the surface shape. It is not accurate enough to identify the signs of blockage such as thin-layer wall adhesion and initial arching, and it cannot sense the internal stress state. The distributed fiber optic grating sensing scheme measures strain by embedding fiber optic grating sensors in the silo wall to infer the pressure distribution. Although it can obtain the stress distribution of the silo wall in the whole circumference and has strong anti-electromagnetic interference capability, it is complicated to deploy and has high cost. The calculation model relies heavily on prior knowledge of the silo structure and material properties, and cannot directly output the three-dimensional spatial coordinates of the blockage location, nor can it intuitively display the blockage morphology. The simple multi-sensor stacking solution deploys various sensors such as level gauges, pressure sensors, humidity meters, temperature sensors, and cameras in the coal bunker, and each sensor independently collects and displays data. Although the data dimensions are relatively rich, the data from each sensor are independent of each other and lack spatiotemporal registration and deep fusion mechanisms. It fails to collaboratively explore the complex characteristics of blockage precursors from multiple dimensions such as geometric shape, physical stress, material properties, and image recognition. It is difficult to achieve collaborative analysis of multi-source data and cannot build a unified three-dimensional digital twin model. In summary, existing technologies generally suffer from the following common defects: they cannot acquire the three-dimensional morphology and wall distribution of the entire coal bunker; they cannot accurately locate the position and form of blockages; the data from each sensor are independent, lacking spatiotemporal registration and dynamic weighted fusion; they have not constructed a digital twin model that can be updated in real time; they cannot predict the evolution trend of blockages in advance; when blockages occur, they cannot provide key information to support engineering decisions, such as the risk location, development trend, and optimal intervention window; the early warning mechanism relies on fixed thresholds and is not dynamically optimized by combining the collaborative characteristics of multi-source sensing data, resulting in high false alarm and false negative rates, which reduces the reliability and safety of coal bunker operation and maintenance. Summary of the Invention

[0003] (a) Technical problems to be solved To address the shortcomings of the existing technology, this invention proposes a method for locating blockages in underground coal bunkers based on multimodal sensing and digital twins.

[0004] (II) Technical Solution This invention is achieved through the following technical solution: This invention provides a method for locating blockages in underground coal bunkers based on multimodal sensing and digital twins, comprising the following steps: Step S1: Input the deployment parameters of the multimodal sensor array and the downhole working condition data, and after spatiotemporal registration and dynamic weighted fusion processing, obtain the multimodal sensing dataset in a unified spatiotemporal coordinate system; The deployment parameters of the multimodal sensor array include the installation position coordinates, spatial attitude angle, calibration error limit, and communication address of each sensor in the global coordinate system of the coal bunker. The underground operating data includes real-time dust concentration, ambient temperature and humidity, coal flow vibration frequency, and bin tilt angle. Step S2: Input the multimodal sensing dataset, and after dynamic and static point cloud separation and contour feature extraction, obtain the warehouse wall contour feature curve and static three-dimensional point cloud; Step S3: Input the warehouse wall contour feature curve and the multimodal sensing dataset, and output the blockage location result after multi-source feature association and blockage localization processing; The blockage location result is a structured dataset containing the three-dimensional spatial coordinates of the blockage point, the blockage type, the corresponding morphological parameter vector, and the risk level. Step S4: Input the static 3D point cloud and the blockage location results, and after 3D reconstruction and digital twin construction processing, output the 3D digital twin model of the coal bunker, and mark the blockage location and risk level in the model; Step S5: Input the blockage location results and multimodal sensing time series data, perform time series evolution analysis and hierarchical early warning processing, and output the risk warning level and the optimal intervention window; The multimodal sensing time-series data is the serialized representation of the multimodal sensing dataset output in step S1 in the time dimension. Step S6: Input the risk warning level, the optimal intervention window, and the feedback data of the blockage clearing operation. After closed-loop optimization and parameter self-correction processing, output the optimized blockage location model parameters and the hierarchical warning strategy parameters. The feedback data for the unblocking operation includes the changes in the bin wall profile, the changes in fiber optic strain, the pressure response curve, and the determination results of the blockage status changes, which are collected by multimodal sensors and processed in step S1 after the execution of the unblocking intervention by lifting vibration or drilling.

[0005] According to a preferred embodiment of the present invention, step S1 specifically includes the following sub-steps: Step S11: Establish spatial coordinate unification. The measurement data obtained by the i-th sensor in its own coordinate system is transformed into the global coordinate system of the coal bunker by performing rigid body transformation based on the actual installation position and spatial attitude angle of the sensor in the global coordinate system of the coal bunker. Step S12: Establish time synchronization. Using the BeiDou / GPS dual-mode high-precision time synchronization module as a unified time reference, add absolute timestamps to the raw data streams of each sensor, and apply a linear interpolation algorithm to resample the multi-source data with different sampling frequencies to a unified time axis. Step S13: Determine the dynamic fusion weights. Based on the signal-to-noise ratio, data efficiency, and distance to the target area of ​​each sensor at sampling time t, calculate the fusion weight of the i-th sensor, and perform a three-factor weighted fusion of signal-to-noise ratio, data efficiency, and distance on the data of each sensor according to the weights to obtain the multimodal sensing dataset.

[0006] According to a preferred embodiment of the present invention, step S2 specifically includes the following sub-steps: Step S21: Based on the fusion data of 3D lidar and 4D millimeter-wave radar, using the radial velocity information provided by 4D millimeter-wave radar, combined with Doppler velocity analysis and time continuity constraints, perform dynamic and static classification to separate the static three-dimensional point cloud and the dynamic point cloud. Step S22: The static three-dimensional point cloud is layered by height, and circular fitting is applied to the point cloud of each cross section to obtain the distribution of the warehouse wall radius at each height. The warehouse wall contour feature curve is then generated by further partitioning the point cloud by circumferential angle. Step S23: Compare the characteristic curve of the coal bunker wall with the design contour of the coal bunker, perform contour inversion, obtain the wall thickness distribution, and quantify the severity of wall adhesion based on the wall thickness and distribution range.

[0007] According to a preferred embodiment of the present invention, step S3 specifically includes the following sub-steps: Step S31: Input the silo wall contour feature curve and the multimodal perception dataset, and apply sequence image processing and deep neural network classification to identify wall hanging features, stress gradient calculation to identify arching features, multi-source index weighting to identify caking features, and target detection to identify foreign object features, and output four types of blockage precursor features: wall hanging, arching, caking, and foreign objects. Step S32: Based on the spatial distribution patterns and physical correlations of various precursory blockage features, calculate the three-dimensional spatial coordinates of the corresponding blockage points and determine their blockage type and morphological parameters; Step S33: Combining the blockage location results with the multimodal sensing time series data, apply first-order time series smoothing analysis to output the blockage risk level and evolution trend prediction results.

[0008] According to a preferred embodiment of the present invention, step S4 specifically includes the following sub-steps: Step S41: Input the static 3D point cloud and the blockage location result, and apply the dynamic adaptive surface reconstruction algorithm to reconstruct the static 3D point cloud into a closed 3D geometric model; Step S42: Input the three-dimensional geometric model and the blockage location result, perform visualization annotation, map the blockage location, blockage type and risk level to the model, and obtain the three-dimensional digital twin model of the coal bunker; Step S43: Input the three-dimensional digital twin model of the coal bunker and real-time multimodal sensing data to construct a real-time dynamic mapping relationship and realize real-time synchronization of the physical coal bunker and the digital model.

[0009] According to a preferred embodiment of the present invention, step S5 specifically includes the following sub-steps: Step S51: Input the blockage location results and multimodal sensing time series data, and extract the time series features related to blockage risk; Step S52: Based on the aforementioned temporal characteristics, apply a first-order temporal smoothing model to calculate the predicted value of the congestion risk level at the next moment; Step S53: Based on the prediction results and the preset risk level threshold, output the risk warning level and the optimal intervention window.

[0010] According to a preferred embodiment of the present invention, step S6 specifically includes the following sub-steps: Step S61: Input the optimal intervention window for the risk warning level and the feedback data for the blockage dredging operation, and calculate the positioning error warning error and synchronization delay; Step S62: Based on the positioning error, the warning error, and the synchronization delay, calculate the comprehensive error index using a multi-dimensional error weighted fusion method; Step S63: Based on the comprehensive error index, jointly adjust the parameters of the bottleneck location model and the parameters of the hierarchical early warning strategy.

[0011] According to a preferred embodiment of the present invention, the multimodal sensor array includes a 3D lidar, a 4D millimeter-wave radar, a distributed fiber optic strain sensor, a pressure sensor, a microwave moisture meter, and an AI image recognition device.

[0012] According to a preferred embodiment of the present invention, the blockage types include four categories: hanging wall, arching, caking, and foreign objects; the morphological parameter vectors vary according to the different blockage types: hanging wall corresponds to thickness and coverage area, arching corresponds to span and arch height, caking corresponds to area and density, and foreign objects correspond to size and material characteristics.

[0013] According to a preferred embodiment of the present invention, the closed-loop optimization process in step S6 continues to run, and the parameters of the bottleneck location model and the parameters of the hierarchical early warning strategy are fed back to step S1 to update the linear interpolation benchmark of rigid body transformation parameters and the three-factor weighted weight of signal-to-noise ratio, efficiency, and distance.

[0014] (III) Beneficial Effects 1. This invention constructs a multimodal sensor array covering geometric shape, physical stress, material properties and environmental conditions, and combines spatiotemporal registration and dynamic weighted fusion algorithms to achieve full-dimensional perception of the internal state of the coal bunker. It overcomes the limitations of traditional single-point sensors, which have limited information and large blind spots. It supports continuous and stable operation under complex underground working conditions such as dust and water mist, and ensures the integrity and reliability of the perception data. 2. This invention utilizes the Doppler velocity characteristics of fused data from 3D lidar and 4D millimeter-wave radar to achieve separation of dynamic and static point clouds. Based on the static point cloud, it generates the contour feature curve of the warehouse wall and identifies wall-hanging features through sequential image processing and deep neural network classification methods. This supports the identification of hidden blockage precursors such as thin-layer wall hanging and initial arching, providing accurate blockage location information for subsequent unblocking operations. 3. This invention uses a multi-source feature association localization algorithm to integrate contour feature curves, fiber optic strain distribution, pressure data, microwave moisture data and AI image recognition results to construct a blockage feature vector and calculate the three-dimensional spatial coordinates and morphological parameters of the blockage point. This enables differentiated localization of four types of blockages: wall hanging, arching, caking and foreign objects, supporting precise blockage dredging decisions. 4. This invention uses a dynamic adaptive surface reconstruction algorithm to transform discrete point clouds into closed three-dimensional geometric models, and marks the location, type and risk level of blockages in the model to build a digital twin model with real-time mapping capabilities. This enables the synchronization of the physical coal bunker and the digital model, supporting maintenance personnel to intuitively grasp the internal status of the coal bunker and conduct remote collaborative diagnosis. 5. This invention analyzes the congestion evolution trend based on multimodal sensing time-series data, outputs risk warning levels and optimal intervention windows, and continuously updates model parameters and warning strategies by combining a closed-loop optimization mechanism, enabling the system to have adaptive evolution capabilities, promoting the transformation of coal bunker operation and maintenance mode from passive response to proactive prevention, and improving the timeliness and scientific nature of congestion dredging operations. Attached Figure Description

[0015] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the positioning method described in this invention. Detailed Implementation

[0016] Reference Figure 1 As shown, this invention provides a method for locating blockages in underground coal bunkers based on multimodal sensing and digital twins, comprising the following steps: Step S1: Input the deployment parameters of the multimodal sensor array and the downhole working condition data, and after spatiotemporal registration and dynamic weighted fusion processing, obtain the multimodal sensing dataset in a unified spatiotemporal coordinate system; The deployment parameters of the multimodal sensor array include the installation position coordinates, spatial attitude angle, calibration error limit, and communication address of each sensor in the global coordinate system of the coal bunker. The underground operating data includes real-time dust concentration, ambient temperature and humidity, coal flow vibration frequency, and bin tilt angle. Step S2: Input the multimodal sensing dataset, and after dynamic and static point cloud separation and contour feature extraction, obtain the warehouse wall contour feature curve and static three-dimensional point cloud; Step S3: Input the warehouse wall contour feature curve and the multimodal perception dataset, and output the blockage location result after multi-source feature association and blockage location processing; wherein, the blockage location result is a structured data set, including the three-dimensional spatial coordinates of the blockage point, the blockage type, the corresponding morphological parameter vector and the risk level; Step S4: Input the static 3D point cloud and the blockage location results, and after 3D reconstruction and digital twin construction processing, output the 3D digital twin model of the coal bunker, and mark the blockage location and risk level in the model; Step S5: Input the blockage location result and multimodal sensing time series data, and after time series evolution analysis and hierarchical early warning processing, output the risk warning level and the optimal intervention window; wherein, the multimodal sensing time series data is the serialized representation of the multimodal sensing dataset output in step S1 in the time dimension. Step S6: Input the risk warning level, the optimal intervention window, and the feedback data of the blockage clearing operation. After closed-loop optimization and parameter self-correction processing, output the optimized blockage location model parameters and the hierarchical warning strategy parameters. The feedback data for the unblocking operation includes the changes in the bin wall profile, the changes in fiber optic strain, the pressure response curve, and the determination results of the blockage status changes, which are collected by multimodal sensors and processed in step S1 after the execution of the unblocking intervention by lifting vibration or drilling.

[0017] Specifically, step S1 includes the following sub-steps: Step S11: Establish spatial coordinate unification, and unify the measurement data acquired by the i-th sensor in its own coordinate system. Based on the actual installation position of the sensor in the global coordinate system of the underground coal bunker and spatial attitude angle Perform rigid body transformation (by constructing the corresponding rotation matrix) Translation vector ), convert to the global coordinate system of the coal bunker ; Step S12: Establish time synchronization. Using the BeiDou / GPS dual-mode high-precision time synchronization module as the unified time reference, assign absolute timestamps to the raw data streams of each sensor, and apply a linear interpolation algorithm to resample multi-source data with different sampling frequencies to a unified time axis. Specifically, let the sensor sampling period be... Adjacent sampling times are and If the sampling time is t, then the difference at the target time is : ; In addition, a unified time axis sampled to 100 Hz is used to ensure that the time alignment error of multi-source data at any point in time does not exceed 10 milliseconds, providing a unified spatiotemporal reference for subsequent multi-source data fusion.

[0018] Step S13: Determine the dynamic fusion weights. Based on the signal-to-noise ratio, data efficiency, and distance to the target area of ​​each sensor at sampling time t, calculate the fusion weight of sensor i (the fusion weight of the i-th sensor). Perform a three-factor weighted fusion of signal-to-noise ratio, data efficiency, and distance on the data of each sensor according to the weights to obtain the multimodal sensing dataset. Specifically, the fusion weight w of the i-th sensor is... i (t): ; In the formula, Let be the signal-to-noise ratio of the i-th sensor. Let be the data efficiency of the i-th sensor at sampling time t. Let be the distance between the i-th sensor and the target area. Here, j is the sensor distance attenuation coefficient, and j is the summation index variable. As the attenuation factor, The total number of sensors participating in the fusion; Take a multimodal sensing dataset in a unified spatiotemporal coordinate system : ; The sensor's measurement data at sampling time t; The above-mentioned method achieves accurate alignment of multi-source data through spatiotemporal registration, effectively overcoming the limitations of limited information from a single sensor and large perception blind spots. Furthermore, it dynamically integrates sensor signal quality into the fusion weights, enabling the fusion results to adaptively respond to changes in complex downhole working conditions. This effectively solves the problem of poor fusion performance of traditional fixed-weight fusion under harsh conditions such as dust interference.

[0019] Specifically, step S2 includes the following sub-steps: Step S21: Based on the fusion data of 3D LiDAR and 4D millimeter-wave radar, utilize the radial velocity information provided by the 4D millimeter-wave radar (radial velocity... Combining Doppler velocity analysis with temporal continuity constraints (introducing temporal continuity constraints can further improve classification accuracy, and only points classified as static in multiple consecutive frames are confirmed as static points, which can avoid instantaneous noise interference), dynamic-static classification is performed to separate the static 3D point cloud. With dynamic point clouds ; The specific classification rule model is as follows: ; The velocity threshold is determined based on the normal falling velocity of the coal flow within the coal bunker. This is the radial velocity of the target point relative to the radar (the velocity component along the radar's line of sight), in m / s. A positive value indicates that the target is moving away from the radar, while a negative value indicates that the target is moving closer to the radar. Step S22: Layer the static 3D point cloud according to height, layer thickness For each layer of the point cloud, a circular fit is applied, with the center coordinates being (x, y). c (z), y c (z) Obtain the distribution of the warehouse wall radius at each height, with the actual radius being R. actual (z, θ), the fitting objective is to determine the coordinates of the circle center and minimize the sum of the squared distances from each point to the fitted circle. The objective function is: ; k is the index number of the k-th point in the point cloud of that height layer section. , Let be the rectangular coordinates of the k-th point in the horizontal plane. The actual contour radius of the cross section at this height level is the unknown quantity to be solved, representing the radius of the fitted circle, and m is the number of cross section point clouds. By traversing all height levels, the contour feature curve that varies along the height can be obtained. Since there may be uneven wall hanging in the warehouse, the actual radius is different at different circumferential angles θ. Therefore, the cross section point cloud is further divided into angle partitions, and the average radius R(z, θ) of each angle interval is calculated to generate the warehouse wall contour feature curve. Step S23: Compare the characteristic curve of the bin wall with the design contour of the coal bin, perform contour inversion, obtain the thickness distribution of the wall hanging, and quantify the severity of the wall hanging based on the thickness and distribution range of the wall hanging. Wall thickness radius Design radius Subtract the actual radius R actual (z, θ) yields: when A value greater than 0 indicates the presence of wall-mounted debris at that location. To further quantify the severity of the debris buildup, a debris volume index is defined. : ; z represents the coordinate in the height direction of the coal bunker. The shallowest depth is usually determined by the starting height of the top of the straight section of the coal bunker (the highest point where wall adhesion may occur). The deepest depth is usually determined by the bottom end height of the straight section of the coal bunker (the lowest point where wall adhesion may occur). When the wall thickness exceeds the threshold of 0.3m or the wall volume index exceeds the set value, it is judged as significant wall adhesion and an early warning is triggered. These are circumferential angle micro-elements and height direction micro-elements, respectively. Data fusion of 3D lidar and 4D millimeter-wave radar was performed to quantify the contour feature curves, enabling quantitative inversion of the wall position and thickness, and providing accurate geometric parameters for subsequent blockage location. Specifically, step S3 includes the following sub-steps: Step S31: Input the silo wall contour feature curve and the multimodal perception dataset, and apply sequence image processing and deep neural network classification to identify wall hanging features, stress gradient calculation to identify arching features, multi-source index weighting to identify caking features, and target detection to identify foreign object features, and output four types of blockage precursor features: wall hanging, arching, caking, and foreign objects. The wall-hanging feature extraction includes: converting the warehouse wall contour feature curve into two-dimensional image features, using a sequence structure information preservation method to spatially encode the one-dimensional contour data, generating an image representation with geometric topological consistency, inputting the image into a deep neural network model (DenseNet neural network), performing classification and recognition, and outputting the probability of wall-hanging existence and the wall-hanging type determination result. Arch feature extraction includes: inputting fiber optic strain distribution and pressure sensor data to construct a stress gradient field; identifying local maximum points in the stress gradient field and determining them as arch foot positions; and determining the arch span based on the circumferential distance between two arch foot positions. The caking feature extraction includes: acquiring microwave moisture meter data, material level change rate at the inlet and outlet of the bin, and coal feeder current; combining the bin depth with a preset weighting relationship to calculate the caking risk index; when the index exceeds a set threshold, it is determined that there is a risk of caking. Foreign object and coal / water identification includes: acquiring image data from an AI camera, executing a target detection model (YOLO deep learning model), identifying large foreign objects (>30cm) and coal / water, and outputting their category, spatial location, and identification confidence level; Step S32: Based on the spatial distribution patterns and physical correlations of various precursory blockage features, calculate the three-dimensional spatial coordinates of the corresponding blockage points and determine their blockage type and morphological parameters; The calculation of the three-dimensional spatial coordinates of the blockage point includes: constructing multi-source sensing data (blockage feature vector, spatial correlation and cross-validation of contour feature curves, fiber optic strain distribution, pressure sensor data, microwave moisture meter data and AI image recognition results); For wall-mounted blockages, their three-dimensional spatial coordinates are determined based on the location of the maximum wall thickness in the contour feature curve. For arched blockages, their three-dimensional spatial coordinates are determined by combining the location of local maximum values ​​in the stress gradient field with the contour characteristic curve. For caking blockages, their three-dimensional spatial coordinates are inferred based on the material level change trend and moisture distribution characteristics. For complex blockages, the three-dimensional spatial coordinates of multiple blockage points and their spatial relationships are output to form a blockage point distribution map. Step S33: Combining the blockage location results with the multimodal sensing time series data, apply first-order time series smoothing analysis to output the blockage risk level and evolution trend prediction results; The blockage risk level is obtained by weighting the wall adhesion risk component, arching risk component, caking risk component, and foreign object risk component according to a preset weighting relationship. Based on multimodal sensing time-series data, a time-series smoothing model is used to predict the evolution trend of congestion risk level; The deep fusion of the above-mentioned multi-source sensing data (geometric shape, physical stress, material properties, and image recognition) enables the identification of blockage features from a single dimension to a multi-dimensional composite, effectively solving the problem that traditional methods cannot accurately locate the blockage location, with a positioning accuracy of ≤1m.

[0020] Specifically, step S4 includes the following sub-steps: Step S41: Input the static 3D point cloud and the blockage location result, and apply the dynamic adaptive surface reconstruction algorithm to reconstruct the static 3D point cloud into a closed 3D geometric model; the reconstructed 3D geometric model is represented by a set of triangular facets, which can intuitively show the morphological structure inside the coal bunker; The dynamic adaptive surface reconstruction algorithm dynamically adjusts the radius of the rolling sphere based on the point cloud density. The algorithm traverses the point cloud with the rolling sphere, and when there are no other point clouds in the sphere, the points on the sphere are identified as boundary points. A closed three-dimensional geometric model is generated based on all boundary points. This model is composed of a set of triangular facets and is used to characterize the internal spatial structure of the coal bunker. Step S42: Input the three-dimensional geometric model and the blockage location result, perform visualization annotation, map the blockage location, blockage type and risk level to the model, and obtain the three-dimensional digital twin model of the coal bunker; In the three-dimensional geometric model, the location, risk level, and blockage type of the blockage are mapped to visual elements; the visual intensity is determined based on the risk level, the graphic symbol shape is determined based on the blockage type, and the size of the visual elements is determined based on the geometric parameters; a label layer containing the above information is generated and superimposed on the three-dimensional geometric model to achieve an intuitive presentation of the internal state of the coal bunker. The colors gradient from green to red according to the risk level, the icons are selected according to the type of blockage (fan-shaped areas for wall-mounted blocks, arches for arched blocks, and semi-transparent layers for solidified blocks), and the size is scaled according to geometric parameters. Step S43: Input the three-dimensional digital twin model of the coal bunker and real-time multimodal sensing data to construct a real-time dynamic mapping relationship, so as to realize the real-time synchronization of the physical coal bunker and the digital model. The dynamic mapping engine continuously receives real-time sensing data and detects changes in the coal pile surface, wall growth, and stress distribution. When the detected changes exceed a preset threshold, the three-dimensional model is triggered to update. The model update frequency is dynamically adjusted according to the rate of change. When the change is drastic, the update frequency is increased, and when the change is gradual, the update frequency is decreased. The above steps call the dynamic adaptive surface reconstruction algorithm to reconstruct the discrete point cloud into a closed three-dimensional geometric model, and visualize and mark the location and shape of the blockage in the model, realizing the transparent reconstruction of the internal state of the coal bunker and filling the technical gap in the field of coal bunker digital twins. Specifically, step S5 includes the following sub-steps: Step S51: Input the blockage location results and multimodal sensing time series data, and extract the time series features related to blockage risk; The temporal features include the sequence of changes in blockage risk level over time, the growth rate of wall thickness, the fluctuation amplitude of stress gradient, the evolution trend of slab area, and the duration of foreign object identification results. Step S52: Based on the aforementioned temporal characteristics, apply a first-order temporal smoothing model to calculate the predicted value of the congestion risk level at the next moment; The temporal features include the sequence of changes in blockage risk level over time, the growth rate of wall thickness, the fluctuation amplitude of stress gradient, the evolution trend of slab area, and the duration of foreign object identification results. Step S53: Based on the prediction results and the preset risk level threshold, output the risk warning level and the optimal intervention window; The system compares the predicted value with a preset first threshold and a second threshold: if the value is lower than the first threshold, the output level is 0, indicating normal operation; if the value reaches or exceeds the first threshold but does not reach the second threshold, the output level is 1, corresponding to a lifting vibration warning; if the value reaches or exceeds the second threshold, the output level is 2, corresponding to a drilling blockage warning; at the same time, the optimal intervention window is generated by combining the moment when the predicted value first touches the threshold with the response time of the blockage removal equipment, and the warning level, intervention window and blockage location result are pushed to the blockage removal equipment control system. The early warning information includes the three-dimensional spatial coordinates of the blockage point, the type of blockage, the lifting vibration command or the drilling command for clearing the blockage, and the optimal intervention window. This information is generated by the hierarchical early warning module and encapsulated into a structured data packet, which is sent to the blockage clearing equipment control system via industrial Ethernet or wireless communication. After receiving the data packet, the control system parses it and automatically starts the corresponding blockage clearing operation based on the command type and the optimal intervention window, realizing a closed-loop linkage between the perceived data and the on-site execution.

[0021] Specifically, step S6 includes the following sub-steps: Step S61: Input the optimal intervention window for the risk warning level and the feedback data for the blockage dredging operation, and calculate the positioning error warning error and synchronization delay; Among them, the positioning error is obtained by comparing the actual blockage location verified by the blockage clearing operation with the predicted location output in step S3; the early warning error is obtained by comparing the risk prediction value output in step S5 with the actual risk level; and the synchronization delay is obtained by comparing the update time of the digital twin model with the time of change in the physical coal bunker state. Step S62: Based on the positioning error, the warning error, and the synchronization delay, a multi-dimensional error weighted fusion method is applied to calculate a comprehensive error index. This index weights and sums the three types of errors according to a preset weight relationship to quantify the overall performance deviation of the system and serve as a unified optimization target for subsequent parameter correction. Step S63: Based on the comprehensive error index, jointly adjust the parameters of the bottleneck location model and the parameters of the hierarchical early warning strategy. The parameter adjustment process is subject to closed-loop stability constraints to ensure that the correction will not cause drastic fluctuations in system performance. At the same time, combined with the typical working condition parameter memory mechanism, when the current working condition is detected to be similar to the historical best working condition, the corresponding stored parameter combination is called as the initial value to speed up the convergence speed and improve the correction accuracy. The S6 steps described above incorporate positioning accuracy, prediction accuracy, and twin update delay into a unified optimization objective through multi-dimensional error fusion and closed-loop optimization. This enables the system to continuously improve the perception model and early warning strategy based on actual operational results, achieving a closed-loop self-correction of "perception-positioning-early warning-optimization".

[0022] The multimodal sensor array includes a 3D lidar, a 4D millimeter-wave radar, a distributed fiber optic strain sensor, a pressure sensor, a microwave moisture meter, and an AI image recognition device.

[0023] The blockage types include four categories: hanging, arching, compaction, and foreign objects. The morphological parameter vectors vary depending on the type of blockage: hanging corresponds to thickness and coverage area, arching corresponds to span and arch height, compaction corresponds to area and density, and foreign objects correspond to size and material characteristics.

[0024] In step S6, the closed-loop optimization process continues to run, and the parameters of the bottleneck location model and the parameters of the hierarchical early warning strategy are fed back to step S1 to update the linear interpolation benchmark of rigid body transformation parameters and the three-factor weighted weight of signal-to-noise ratio, efficiency, and distance.

[0025] To implement the above method, the present invention provides a computer device, including an edge computing node, an intrinsically safe multimodal data interface for coal mines, a real-time operating system, and functional modules; The edge computing nodes are deployed inside wall-mounted protective boxes in underground coal bunkers. The intrinsically safe multimodal data interface for coal mines is used to connect 3D lidar, 4D millimeter-wave radar, distributed fiber optic strain sensor, pressure sensor, microwave moisture meter and AI image recognition device, and to receive and cache multimodal sensing datasets in a unified spatiotemporal coordinate system. The real-time operating system supports high real-time task scheduling. The functional modules include: Spatiotemporal registration module, dynamic weighted fusion module, dynamic and static point cloud separation module, contour feature extraction module, multi-source feature association module, blockage localization module, digital twin construction module, hierarchical early warning module, and closed-loop optimization module; in, The spatiotemporal registration module is used to perform rigid body transformations and linear interpolation; The dynamic weighted fusion module is used to perform signal-to-noise ratio, efficiency, and distance three-factor weighted fusion; The static / dynamic point cloud separation module is used to perform Doppler velocity analysis and time continuity constraint-based static / dynamic classification. The contour feature extraction module is used to perform hierarchical circular fitting; The multi-source feature association module is used to perform sequence image processing and deep neural network classification, stress gradient calculation, multi-source index weighting, and target detection. The blockage location module is used to calculate the three-dimensional spatial coordinates of the blockage point and determine the blockage type and morphological parameters; The digital twin building block is used to perform dynamic adaptive surface reconstruction and visualization annotation; The graded early warning module is used to perform first-order time series smoothing analysis and risk level threshold determination; The closed-loop optimization module is used to perform multi-dimensional error weighted fusion and joint adjustment of model parameters and strategy parameters; Each module operates collaboratively in the order of the blockage location method described above.

Claims

1. A method for locating blockages in underground coal bunkers based on multimodal sensing and digital twins, characterized in that: Includes the following steps: Step S1: Input the deployment parameters of the multimodal sensor array and the downhole working condition data, and after spatiotemporal registration and dynamic weighted fusion processing, obtain the multimodal sensing dataset in a unified spatiotemporal coordinate system; The deployment parameters of the multimodal sensor array include the installation position coordinates, spatial attitude angle, calibration error limit, and communication address of each sensor in the global coordinate system of the coal bunker. The underground operating data includes real-time dust concentration, ambient temperature and humidity, coal flow vibration frequency, and bin tilt angle. Step S2: Input the multimodal sensing dataset, and after dynamic and static point cloud separation and contour feature extraction processing, obtain the warehouse wall contour feature curve and static three-dimensional point cloud; Step S3: Input the warehouse wall contour feature curve and the multimodal perception dataset, and output the blockage location result after multi-source feature association and blockage location processing; wherein, the blockage location result is a structured data set, including the three-dimensional spatial coordinates of the blockage point, the blockage type, the corresponding morphological parameter vector and the risk level; Step S4: Input the static 3D point cloud and the blockage location results, and after 3D reconstruction and digital twin construction processing, output the 3D digital twin model of the coal bunker, and mark the blockage location and risk level in the model; Step S5: Input the blockage location results and multimodal sensing time series data, perform time series evolution analysis and hierarchical early warning processing, and output the risk warning level and the optimal intervention window; The multimodal sensing time-series data is the serialized representation of the multimodal sensing dataset output in step S1 in the time dimension. Step S6: Input the risk warning level, the optimal intervention window, and the feedback data of the blockage clearing operation. After closed-loop optimization and parameter self-correction processing, output the optimized blockage location model parameters and the hierarchical warning strategy parameters. The feedback data for the unblocking operation includes the changes in the bin wall profile, the changes in fiber optic strain, the pressure response curve, and the determination results of the blockage status changes, which are collected by multimodal sensors and processed in step S1 after the execution of the unblocking intervention by lifting vibration or drilling.

2. The method for locating blockages in underground coal bunkers based on multimodal perception and digital twins according to claim 1, characterized in that: Step S1 specifically includes the following sub-steps: Step S11: Establish spatial coordinate unification. The measurement data obtained by the i-th sensor in its own coordinate system is transformed into the global coordinate system of the coal bunker by performing rigid body transformation based on the actual installation position and spatial attitude angle of the sensor in the global coordinate system of the coal bunker. Step S12: Establish time synchronization. Using the BeiDou / GPS dual-mode high-precision time synchronization module as a unified time reference, add absolute timestamps to the raw data streams of each sensor, and apply a linear interpolation algorithm to resample the multi-source data with different sampling frequencies to a unified time axis. Step S13: Determine the dynamic fusion weights. Based on the signal-to-noise ratio, data efficiency, and distance to the target area of ​​each sensor at sampling time t, calculate the fusion weight of sensor i. Perform three-factor weighted fusion of signal-to-noise ratio, data efficiency, and distance on the data of each sensor according to the weights to obtain the multimodal sensing dataset.

3. The method for locating blockages in underground coal bunkers based on multimodal perception and digital twins according to claim 2, characterized in that: Step S2 specifically includes the following sub-steps: Step S21: Based on the fusion data of 3D lidar and 4D millimeter-wave radar, using the radial velocity information provided by 4D millimeter-wave radar, combined with Doppler velocity analysis and time continuity constraints, perform dynamic and static classification to separate the static three-dimensional point cloud and the dynamic point cloud. Step S22: The static three-dimensional point cloud is layered by height, and circular fitting is applied to the point cloud of each cross section to obtain the distribution of the warehouse wall radius at each height. The warehouse wall contour feature curve is then generated by further partitioning the point cloud by circumferential angle. Step S23: Compare the characteristic curve of the bin wall with the design contour of the coal bin, perform contour inversion, obtain the distribution of wall thickness, and quantify the severity of wall adhesion based on the wall thickness and distribution range.

4. The method for locating blockages in underground coal bunkers based on multimodal perception and digital twins according to claim 1, characterized in that: Step S3 specifically includes the following sub-steps: Step S31: Input the silo wall contour feature curve and the multimodal perception dataset, and apply sequence image processing and deep neural network classification to identify wall hanging features, stress gradient calculation to identify arching features, multi-source index weighting to identify caking features, and target detection to identify foreign object features, and output four types of blockage precursor features: wall hanging, arching, caking, and foreign objects. Step S32: Based on the spatial distribution patterns and physical correlations of various precursory blockage features, calculate the three-dimensional spatial coordinates of the corresponding blockage points and determine their blockage type and morphological parameters; Step S33: Combining the blockage location results with the multimodal sensing time series data, apply first-order time series smoothing analysis to output the blockage risk level and evolution trend prediction results.

5. The method for locating blockages in underground coal bunkers based on multimodal perception and digital twins according to claim 1, characterized in that: Step S4 specifically includes the following sub-steps: Step S41: Input the static 3D point cloud and the blockage location result, and apply the dynamic adaptive surface reconstruction algorithm to reconstruct the static 3D point cloud into a closed 3D geometric model; Step S42: Input the three-dimensional geometric model and the blockage location result, perform visualization annotation, map the blockage location, blockage type and risk level to the model, and obtain the three-dimensional digital twin model of the coal bunker; Step S43: Input the three-dimensional digital twin model of the coal bunker and real-time multimodal sensing data to construct a real-time dynamic mapping relationship and realize real-time synchronization of the physical coal bunker and the digital model.

6. The method for locating blockages in underground coal bunkers based on multimodal sensing and digital twins according to claim 1, characterized in that: Step S5 specifically includes the following sub-steps: Step S51: Input the blockage location results and multimodal sensing time series data, and extract the time series features related to blockage risk; Step S52: Based on the aforementioned temporal characteristics, apply a first-order temporal smoothing model to calculate the predicted value of the congestion risk level at the next moment; Step S53: Based on the prediction results and the preset risk level threshold, output the risk warning level and the optimal intervention window.

7. The method for locating blockages in underground coal bunkers based on multimodal sensing and digital twins according to claim 1, characterized in that: Step S6 specifically includes the following sub-steps: Step S61: Input the optimal intervention window for the risk warning level and the feedback data for the blockage dredging operation, and calculate the positioning error warning error and synchronization delay; Step S62: Based on the positioning error, the warning error, and the synchronization delay, calculate the comprehensive error index using a multi-dimensional error weighted fusion method; Step S63: Based on the comprehensive error index, jointly adjust the parameters of the bottleneck location model and the parameters of the hierarchical early warning strategy.

8. The method for locating blockages in underground coal bunkers based on multimodal sensing and digital twins according to claim 1, characterized in that: The multimodal sensor array includes a 3D lidar, a 4D millimeter-wave radar, a distributed fiber optic strain sensor, a pressure sensor, a microwave moisture meter, and an AI image recognition device.

9. A method for locating blockages in underground coal bunkers based on multimodal sensing and digital twins according to claim 1, characterized in that: The blockage types include four categories: hanging wall, arching, hardening, and foreign objects; the morphological parameter vectors vary according to the blockage type: hanging wall corresponds to thickness and coverage area, arching corresponds to span and arch height, hardening corresponds to area and density, and foreign objects correspond to size and material characteristics.

10. A method for locating blockages in underground coal bunkers based on multimodal sensing and digital twins according to claim 1, characterized in that: In step S6, the closed-loop optimization process continues to run, and the parameters of the bottleneck location model and the parameters of the hierarchical early warning strategy are fed back to step S1 to update the linear interpolation benchmark of rigid body transformation parameters and the three-factor weighted weight of signal-to-noise ratio, efficiency, and distance.

Images