Method and system for detecting roof caving and coal wall caving of coal face

Abstract

The invention relates to a method for detecting roof caving and coal wall caving of a coal face, and the method comprises the steps: introducing a virtual point generation strategy based on environment perception and geometric priori through the fusion of multi-modal data of visible light, infrared and radar, and carrying out the detection of the roof caving and coal wall caving of the coal face under the conditions of visual degradation (such as high dust) and sparse point cloud. The geometric virtual points generated by fitting of the section contour line of the working face are adaptively utilized to replace failed visual guidance, so that accurate physical-level feature alignment is realized, and the problems of perception interruption and feature noise interference caused by visual failure of the coal face in an extreme environment are solved; and meanwhile, by utilizing deep interactive heterogeneous features of a multi-stage cross-modal attention mechanism and combining a multi-dimensional signal joint verification strategy, the recognition precision of the roof caving and wall caving boundary is remarkably enhanced, the false alarm rate is greatly reduced, and high-robustness real-time monitoring in a complex scene of a coal mine is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of coal mine safety monitoring, in particular to a coal mining face roof fall and coal wall spalling detection method and system. BACKGROUND

[0002] The structural safety of the coal mining face is directly related to the production efficiency of the mine and the safety of the operating personnel. In actual production, the periodic weighting caused by the frequent coal cutting of the coal mining machine, the repeated disturbance of the roof by the repeated support of the support, and other factors restrict the coal wall and the roof of the coal mining face. The surrounding rock of the coal mining face is prone to rock spalling, crack development, coal wall bulging, and local roof fall. If it cannot be found and warned in time, it is easy to cause serious safety accidents.

[0003] The existing coal mining face coal wall and roof fall and spalling detection method mainly relies on manual inspection or single sensor monitoring. Manual inspection is not only inefficient, but also has high safety risks, and it is difficult to achieve all-weather continuous coverage. Single sensor monitoring methods, such as using only visible light cameras to detect rock surface cracks or only using infrared cameras to detect temperature anomalies, have obvious limitations in the complex disaster evolution environment of the coal mine. Specifically, the working face is affected by the high-pressure water curtain generated by the coal cutting of the coal mining machine drum, the high-concentration coal dust, and the limited lighting conditions, resulting in a serious decrease in the clarity of visible light images, and even complete failure due to visual degradation; although infrared images have certain smoke and fog penetration ability, they lack texture details and are difficult to clearly define the boundaries of the roof fall area or the spalling body; radar can penetrate dust and perceive small structural deformations, but the point cloud data obtained is extremely sparse and noisy, making it difficult to be directly used for high-precision pixel-level semantic segmentation.

[0004] In addition, the existing multi-modal fusion method often does not fully consider the problem of extreme environmental degradation in the coal mine. In the traditional fusion strategy, it is usually assumed that each modal data is valid input, but when the dust is severely blocked, the sparse point cloud and the noisy visual features are forcibly aligned, which may introduce interference and reduce the detection accuracy. At the same time, the existing deep learning network has high computational complexity when processing multi-modal data, and it is difficult to realize real-time inference on the resource-constrained underground edge computing device.

[0005] Therefore, how to accurately identify and early warn the surrounding rock instability area in the complex underground environment is a technical problem to be solved in the current coal mine safety technology field. SUMMARY

[0006] The coal mining face roof fall and coal wall spalling detection method and system provided by the embodiments of the present application at least solve the problem of low detection accuracy of the coal mining face roof fall and spalling in the presence of high-concentration dust and water mist and poor lighting conditions in the related art.

[0007] In a first aspect, the embodiments of the present application provide a roof fall and coal wall spalling detection method for a coal mining face, the method comprising: obtaining multi-modal data of the coal mining face, the multi-modal data comprising a visible light image, an infrared image and radar point cloud data; projecting the radar point cloud data to an imaging plane of the visible light image to generate a sparse depth projection map, and performing densification processing on the sparse depth projection map using a preset virtual point generation strategy to obtain a dense depth map aligned with pixels of the visible light image; performing feature encoding on the visible light image, the infrared image, the radar point cloud data and the dense depth map respectively by using a pre-constructed multi-modal Transformer fusion network to obtain visual features and radar geometric features; performing deep interaction fusion on the visual features and the radar geometric features using a multi-stage cross-modal attention mechanism to generate unified fusion features containing physical attribute constraints, and performing up-sampling reconstruction on the unified fusion features to output a semantic segmentation map for roof fall and coal wall spalling detection of the coal mining face.

[0008] In some embodiments, the visual features comprise visible light features and infrared features, and the radar geometric features comprise point features and projection features, and the feature encoding on the visible light image, the infrared image and the radar point cloud data respectively comprises: inputting the visible light image and the infrared image into a visual Transformer encoder respectively to perform image slice embedding, position encoding and multi-layer self-attention calculation respectively to obtain the visible light features and the infrared features; inputting the radar point cloud data into a point cloud feature extraction module to extract the point features, and performing lightweight convolution processing on the dense depth map to obtain projection features, and taking the point features and the projection features as the radar geometric features.

[0009] In some embodiments, the image slice embedding, position encoding and multi-layer self-attention calculation to obtain the visible light features and the infrared features respectively comprise: performing grid partitioning processing on the input image, flattening and linearly mapping each partition to a low-dimensional vector to obtain an image slice embedding vector; Learnable position embedding parameters are added to the image slice embedding vector to generate an input sequence that preserves the spatial topological information of the coal wall and roof. In the multi-layer self-attention calculation, the input sequence is transformed linearly to generate a query vector, a key vector, and a value vector. Attention weights are obtained by calculating the dot product similarity between the query vector and the key vector. The value vector is then weighted and summed using the attention weights to obtain visible light features and infrared features containing context-aware information.

[0010] In some embodiments, a multi-stage cross-modal attention mechanism is used to perform deep interactive fusion of the visual features and the radar geometric features, including: A bidirectional cross-attention calculation is performed between the visible light features and the infrared features to complement the texture information and the infrared radiation information caused by rock mass stress, thereby obtaining image fusion features; The image fusion features and the radar geometric features are cross-attention calculated to generate the unified fusion features by utilizing the radar's depth constraints and motion vector information.

[0011] In some embodiments, the sparse depth projection map is densified using a preset virtual point generation strategy, including: Calculate the local contrast and high-frequency components of the visible light image, and the thermal difference distribution characteristics of the infrared image; Based on the local contrast, the high-frequency components, and the thermal difference distribution characteristics, a visual degradation confidence map is constructed. The visual degradation confidence map is used to characterize the effectiveness weight of visual information at each pixel position in an environment where the dust concentration is greater than a first preset threshold or the light intensity is lower than a second preset threshold. Based on the radar point cloud data, the working surface cross-sectional contour line is fitted to construct a prior geometric model of the working surface, and geometric virtual points are generated based on the prior geometric model of the working surface. A dual-source adaptive fusion strategy is adopted to perform densification processing based on the texture features of the visual degradation confidence map, the geometric virtual points, and the visible light image to obtain the dense depth map.

[0012] In some embodiments, fitting the working face cross-sectional profile based on the radar point cloud data to construct a prior geometric model of the working face, and generating geometric virtual points based on the prior geometric model of the working face, includes: Nonlinear spatial fitting is performed on the radar point cloud data to obtain the coal wall cross-sectional profile line representing the current working face cross-sectional shape, and the coal wall cross-sectional profile line is used as the prior geometric model of the working face. Determine the coordinate gap between adjacent real radar sampling points, and use the coal wall cross-sectional profile to calculate the optimal interpolation trajectory that spans the coordinate gap and connects the adjacent real radar sampling points; Along the optimal interpolation trajectory, geometric virtual points are generated whose spatial distribution conforms to the natural curvature of the coal face and roof.

[0013] In some embodiments, a dual-source adaptive fusion strategy is employed to generate the dense depth map based on the visual degradation confidence map, the geometric virtual points, and the texture features of the visible light image, including: A pixel weight mask is generated based on the relationship between the validity weight of each pixel in the visual degradation confidence map and a preset threshold. When the weighted mask indicates that the corresponding position in the visible light image is valid, the texture features of the visible light image are used to guide depth interpolation; When the validity of the corresponding position in the visible light image indicated by the weighted mask is lower than the effective threshold, depth filling is performed based on the depth value of the geometric virtual point to achieve depth map reconstruction in the area where visual information is missing.

[0014] In some embodiments, the method further includes early warning based on multi-physics co-operation: Based on the semantic segmentation graph, a connectivity analysis is performed on the identified areas of roof fall and coal wall spalling in the coal mining face. By combining the depth change rate and instantaneous displacement velocity in the radar point cloud data, and the temperature anomaly characteristics in the infrared image, a comprehensive alarm index for roof collapse is calculated. When the comprehensive alarm index of the roof collapse exceeds the preset total threshold, or when any physical quantity among the depth change rate and instantaneous displacement velocity exceeds the corresponding sub-item safety threshold, a disaster alarm signal of the corresponding level is generated.

[0015] Secondly, embodiments of this application provide a detection system for the instability of surrounding rock in coal mining faces. The system includes: an acquisition module, a processing module, and a generation module, wherein: The acquisition module is used to acquire multimodal data of the coal mining face, including visible light images, infrared images, and radar point cloud data. The computational processing module is used to project the radar point cloud data onto the imaging plane of the visible light image to generate a sparse depth projection map, and to perform a densification process on the sparse depth projection map using a preset virtual point generation strategy to obtain a dense depth map aligned with the pixels of the visible light image. Furthermore, through a pre-constructed multimodal Transformer fusion network, the visible light image, the infrared image, the radar point cloud data, and the dense depth map are respectively feature-encoded to obtain visual features and radar geometric features; The generation module is used to perform deep interactive fusion of the visual features and the radar geometric features using a multi-stage cross-modal attention mechanism to generate unified fused features, upsample and reconstruct the unified fused features, and output a semantic segmentation map to characterize the risk areas of roof collapse and coal wall spalling in the coal mining face.

[0016] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0017] Compared to related technologies, the detection method provided in this application integrates multimodal data from visible light, infrared, and radar, and introduces a virtual point generation strategy based on environmental perception and geometric priors. In scenarios of visual degradation at the coal mining face (such as high dust), it adaptively uses geometric virtual points generated by fitting the contour line of the working face to replace the failed visual guidance, achieving accurate physical-level feature alignment and solving the problems of perception interruption and feature noise interference caused by visual failure at the coal mining face in extreme environments. At the same time, it utilizes a multi-stage cross-modal attention mechanism to deeply interact with heterogeneous features, and combines a multi-dimensional signal and physical attribute joint verification strategy to significantly enhance the recognition accuracy of roof fall and sidewall boundary, greatly reduce the false alarm rate, and achieve highly robust real-time monitoring in complex coal mine scenarios. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a method for detecting roof collapse and coal wall spalling in a coal mining face according to an embodiment of this application; Figure 2 This is a schematic diagram of multimodal feature extraction and fusion according to an embodiment of this application; Figure 3 This is a structural block diagram of a detection system for roof collapse and coal wall spalling in a coal mining face according to an embodiment of this application; Figure 4 This is a schematic diagram of the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0020] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0021] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0022] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] For ease of understanding, some of the terms used in this application are explained below: Surrounding Rock Instability: refers to the redistribution of stress in the coal and rock mass that was originally in equilibrium after the coal face is excavated. When the stress exceeds the ultimate strength of the coal and rock mass, it causes deformation, breakage and failure of the rock strata around the working face.

[0024] Roof fall and rib spalling: refers to a common production disaster in coal mines; roof fall refers to the phenomenon of rock falling from the roof of the working face, while rib spalling refers to the phenomenon of coal face peeling and collapsing under stress.

[0025] Visible light image (RGB): refers to a three-channel color image acquired by a visible light camera, used to provide information on the texture, structure and deformation of the coal face and roof surface.

[0026] Infrared (IR) images provide information on the temperature distribution of the working face rock mass. Since coal and rock masses often undergo stress-induced deformation or instability of the surrounding rock before the working face becomes unstable, and because the infrared band has better penetration ability for smoke and dust than visible light, IR images can provide crucial clues to stress concentration and visual compensation under harsh working conditions.

[0027] Radar point cloud (mm Wave Point Cloud): refers to sparse three-dimensional point set data acquired by radar, which includes physical parameters such as distance, reflection intensity, and velocity, and still has the ability to sense structural deformation in extreme dust-covered environments.

[0028] ViT (Vision Transformer): A visual encoding network based on Transformer that extracts visual features by dividing an image into fixed-size slices and using a self-attention mechanism.

[0029] Multimodal Transformer: A data fusion network for fusing information from different sensors, which achieves deep feature interaction between visual, thermal imaging and radar point clouds through a cross-attention structure.

[0030] Projected & Virtual Points: Projects a 3D point cloud onto the camera plane, or generates compensation points around sparse points to alleviate the problem of an overly sparse point cloud.

[0031] Semantic Segmentation Map: The pixel-level classification map output by this invention is used to mark the boundaries and locations of risk areas such as roof collapse, coal wall spalling, and crack development.

[0032] In coal mining face safety monitoring, traditional detection methods are limited by the harsh underground environment (such as high dust, low illumination, water mist, etc.), often making it difficult to balance real-time performance and accuracy. Therefore, this application provides a method for detecting roof falls and coal wall spalling in coal mining faces. This method integrates data from three modalities: visible light, infrared thermal imaging, and radar. Through a unique virtual point generation strategy and a cross-modal Transformer network, it achieves highly robust intelligent detection.

[0033] Figure 1 This is a flowchart of a method for detecting roof collapse and coal wall spalling in a coal mining face according to an embodiment of this application, as shown below. Figure 1 As shown, the process includes the following steps: S101 acquires multimodal data of the coal mining face, including visible light images, infrared images, and radar point cloud data.

[0034] Specifically, an integrated multi-sensor data acquisition and sensing array is deployed within the coal mining face. This array undergoes precise time-base alignment and spatial geometric calibration, and its specific characteristics are as follows: Visible light images (RGB): provide high-resolution surface texture and spatial structure details to capture visual features of working face support construction, coal wall fracture development, and roof strata delamination.

[0035] Infrared (IR) imagery: provides the radiation temperature distribution of the surrounding rock at the working face. Utilizing the thermal field anomalies generated in the coal and rock mass before stress concentration or deformation, it provides crucial clues to stress evolution and visual enhancement compensation under dust-covered or zero-light conditions. Radar point cloud data: acquired by millimeter-wave radar. Millimeter waves have the ability to penetrate fog and dust, enabling the acquisition of the three-dimensional coordinates, reflection intensity, and deformation displacement vector of the working surface, providing reliable physical measurements for dynamic deformation monitoring.

[0036] To ensure that the data from multiple sources correspond in time sequence, each sensor acquires data synchronously via hardware triggering or Network Time Protocol (PTP).

[0037] This step involves multi-sensor collaborative data acquisition to obtain comprehensive coal face information, including texture, thermal spectrum, and spatial geometry. High-precision time synchronization technology is used to eliminate timing errors, providing high-quality data support for the accurate fusion and feature alignment of subsequent multimodal data.

[0038] S102, the radar point cloud data is projected onto the imaging plane of the visible light image to generate a sparse depth projection map, and the sparse depth projection map is densified using a preset virtual point generation strategy to obtain a dense depth map aligned with the pixels of the visible light image.

[0039] Because the point cloud data acquired by radar is sparse, direct fusion with high-resolution images would lead to missing feature matching. Furthermore, considering the severe visual degradation often present in underground environments (such as high-concentration dust obscuring the view), this step achieves point cloud enhancement and geometric alignment through the following detailed process: S102.1, Generate a sparse depth projection map.

[0040] Based on the pre-calibrated spatial extrinsic matrix between the radar and the camera (composed of rotation matrix R and translation vector T) and the camera's intrinsic matrix K, a homogeneous transformation model from the three-dimensional radar coordinate system to the image pixel coordinate system is established.

[0041] The specific projection process is as follows: (1) Coordinate system transformation: The coordinates (x, y, z) of the three-dimensional radar point cloud are transformed to the camera center coordinate system through the external parameter matrix.

[0042] (2) Perspective projection mapping: Using the camera intrinsic parameters and pinhole imaging model, the points in the camera coordinate system are projected onto the physical image and the corresponding pixel coordinates (u,v) are calculated.

[0043] (3) Multidimensional attribute assignment: In the generated sparse depth projection map, the pixel values ​​of the projection position (u,v) are assigned the corresponding physical attributes such as radial depth, reflection intensity and radial velocity, which serve as the original geometric features for subsequent fusion; the non-target areas that are not covered are masked by assigning zero values, thereby forming a depth projection map with a highly sparse spatial distribution but with precise physical quantities.

[0044] S102.2, use a preset virtual point generation strategy to perform densification processing.

[0045] In conventional image and point cloud fusion, the logic for point cloud enhancement (upsampling) is usually static and fixed. However, the working conditions in underground coal mines are complex: when the dust concentration is extremely high (greater than a first preset threshold) or in an environment with extremely low light (less than a second preset threshold), the RGB and infrared image effects are significantly degraded or even completely ineffective. In such cases, blindly performing pixel alignment will introduce a large amount of noise. The first and second preset thresholds can be determined based on experience in the field combined with real environmental parameters, and their specific values ​​are not limited in this embodiment.

[0046] To address this issue, this application introduces a virtual point generation mechanism based on confidence weighting during the densification stage. Utilizing a prior geometric model of the coal face, it automatically switches to generating virtual points based on the fitting results of the working face cross-sectional contour in visually inaccessible areas, instead of relying solely on images for guidance.

[0047] Specifically, this embodiment employs an adaptive enhancement technique based on environmental degradation perception and geometric prior, including: S1, Construct a Visual Degradation Map: First, the local contrast and high-frequency components of the visible light image are calculated. Dust occlusion leads to reduced local contrast and blurred edge textures. Simultaneously, the thermal difference distribution characteristics of the infrared image are analyzed. Normal coal wall temperature exhibits physical continuity; irregular thermal noise spots are identified as environmental interference.

[0048] Based on the above features, a weighted mask M with the same resolution as the visible light image is generated. The value (0~1) of each pixel in the mask represents the effectiveness weight of the visual information at that location: A value close to 0 indicates a severe lack of visual signal in that area. A value close to 1 indicates a clear field of vision.

[0049] S2, Construct the prior geometric model of the working face and generate virtual geometric points: Non-empty linear spatial fitting is performed using the acquired sparse radar point cloud data. Since the coal face formed after the coal mining machine cuts the coal has a locally flat feature, a cross-sectional contour line representing the current spatial morphology of the coal face is fitted, which can be achieved through a random sampling consensus algorithm.

[0050] Next, the coordinate gaps between adjacent real radar sampling points are identified. Using the working face geometric constraint model, the optimal interpolation trajectory that spans this gap and connects adjacent sampling points is calculated. The geometric virtual points generated along this optimal interpolation trajectory conform to the natural curvature of the coal wall and roof in spatial distribution, avoiding the geometric artifacts and suspension noise generated by traditional linear interpolation, and ensuring that the virtual points conform to the surrounding rock support structure of the coal mining face.

[0051] S3 uses a dual-source adaptive fusion strategy to generate a dense depth map: The reference source for densification processing is dynamically switched based on the weighted mask M: In the pixel region where the visible light image is valid (i.e., greater than a preset threshold, with a clear field of view) indicated by the weighted mask, the texture features (such as edge gradients) of the visible light image are extracted as spatial constraints to regulate the depth interpolation process of virtual points in the sparse depth map. Through the visual edge constraint, it is ensured that the reconstructed depth boundary is strictly aligned with the visual contour of physical targets within the working face (such as hydraulic support sidewalls and coal wall edges).

[0052] In pixel regions where the weighted mask indicates that the visible light image is ineffective (i.e., below the effective threshold, with heavy dust or extremely low illumination), the interference of visual features is shielded, and depth value filling and value regression are performed entirely based on the aforementioned generated geometric virtual points.

[0053] This application leverages the geometric constraints of the cross-section formed by the coal face and hydraulic supports in the coal mining face. It utilizes virtual points generated from prior knowledge of the geometric structure and a visually degraded confidence map to collaboratively drive the reconstruction of a dense depth map. This strategy ensures that the dense depth map conforms to the physical structure of the surrounding rock while adaptively adjusting perception weights to mask invalid visual information. This guarantees that even in extreme environments, it can output geometric information that is strictly consistent with image pixels and possesses high robustness, providing a high-quality spatial benchmark for subsequent deep fusion of multimodal features.

[0054] S103 uses a pre-built multimodal Transformer fusion network to encode features from visible light images, infrared images, radar point cloud data, and dense depth maps, respectively, to obtain visual features and radar geometric features.

[0055] This step leverages the powerful global context modeling capabilities of the Transformer architecture for deep feature mining, specifically including: S103.1 Extract visual features (visible light features and infrared features).

[0056] The visible light image and the infrared image are respectively input into two parallel Visual Transformer (ViT) encoders. The processing procedure is as follows: Image Patch Embedding: The input image is divided into grid-like blocks (such as 16×16 pixel blocks), each block is flattened and then converted into a low-dimensional continuous vector through linear mapping to generate an image patch embedding sequence.

[0057] Positional Encoding: Injecting learnable positional parameters into the embedding sequence to compensate for the Transformer architecture's need to perceive the topology of the image space.

[0058] Self-attention computation: The input sequence is processed through multiple Transformer Encoders. In each encoding layer, a query vector, a key vector, and a value vector are generated through linear transformations. Attention weights are calculated using the dot product similarity between the query and key vectors. Based on this, the values ​​are weighted and aggregated to capture long-distance semantic relationships within the image.

[0059] Ultimately, the output includes visible light features rich in texture details and infrared features characterizing thermal distribution anomalies.

[0060] S103.2, Extract radar geometric features.

[0061] Point path feature extraction: The original radar point cloud (containing coordinates x, y, z, radial velocity, and reflection intensity) is input into the point cloud processing module (such as Point Net++ or Point Transformer). Through local neighborhood aggregation and self-attention mechanisms, point features containing microscopic 3D topology and motion vectors are extracted.

[0062] Projection path feature extraction: Input the dense depth map generated in S102 into a lightweight convolutional network (or a small ViT), and use the translation invariance of the convolution operator to extract projection features that are strongly aligned with visual pixels.

[0063] Feature fusion and dimensionality reduction: Point features and projected features are concatenated to obtain the final radar geometric features. This model retains the accurate physical quantities of the original point cloud while incorporating the structural continuity after densification, enabling it to keenly detect the dynamic instability trends of the coal wall and roof.

[0064] This step employs a parallel architecture of dual-stream visual coding and dual-path geometric coding to map heterogeneous sensor data to a unified high-dimensional feature space. This design preserves the unique physical properties of each modality (such as infrared thermal features) while standardizing the feature dimensions, providing semantically rich and geometrically complementary feature primitives for subsequent deep interaction through cross-modal attention mechanisms.

[0065] S104 utilizes a multi-stage cross-modal attention mechanism to deeply interact and fuse visual features and radar geometric features to generate unified fused features. The unified fused features are then upsampled and reconstructed to output a semantic segmentation map that characterizes the risk areas of roof collapse and coal wall spalling in the coal mining face.

[0066] S104.1, Cross-modal deep interactive fusion. Figure 2 This is a schematic diagram illustrating multimodal feature extraction and fusion according to an embodiment of this application, as shown below. Figure 2 As shown, this embodiment employs a multi-stage fusion strategy, wherein: RGB-IR Bidirectional Semantic Complementary Fusion: This method performs bidirectional cross-attention between visible light and infrared features to achieve semantic alignment and complementarity of heterogeneous features. Attention is calculated using visible light features as the query and infrared features as the key / value pair, and vice versa. This process achieves deep coupling between texture information and infrared radiation information caused by rock stress. For example, under low-light conditions, infrared features can be used to enhance the contour visibility of visible light modes, generating enhanced image fusion features.

[0067] Image-radar geometric constraint fusion: This method interacts with image fusion features and radar geometric features. Using precise depth information and Doppler motion vectors provided by the radar as physical constraints, image features are reweighted. When the radar detects a significant abrupt change in depth or an anomalous velocity in a certain area, the system significantly increases the weight of features in that area in the global fusion, thereby generating unified fusion features that are highly sensitive to structural instability.

[0068] S104.2, Decoding, Reconstruction, and Output.

[0069] Multi-scale feature decoding: The feature input decoder module is unified and fused. The decoder uses a hierarchical upsampling structure (such as transposed convolution) to gradually restore the high-dimensional feature map to the original input resolution.

[0070] Semantic segmentation map output: The output layer generates a pixel-level semantic segmentation map, which is a pixel-level classification map that accurately marks the two-dimensional location and geometric shape of dangerous targets such as coal face and roof collapse areas, cracks, and delamination blocks in the coal mining face.

[0071] Auxiliary decision outputs: Boundary confidence maps and modal attention heatmaps are output simultaneously. The attention map visualizes the contribution of each sensor in the decision-making process, providing downhole operators with an intuitive reference for model interpretability.

[0072] This step utilizes a multi-stage attention mechanism to fuse information from various modalities, achieving deep fusion and complementary advantages of heterogeneous features. This enhances the detection sensitivity for small roof collapse targets in complex backgrounds and outputs an attention heatmap to improve model interpretability, ensuring high accuracy and reliability of segmentation results under insufficient lighting or smoke interference.

[0073] S105 executes a roof collapse alarm decision based on multi-source information fusion.

[0074] To improve early warning accuracy and reduce false alarms caused by sudden sensor noise, this embodiment constructs a multi-dimensional indication-weighted decision model: Morphological connected component analysis: Connected component extraction is performed on the foreground pixels in the semantic segmentation map to determine the geometric area, aspect ratio, and fractal dimension of each independent defect region, etc.

[0075] Multi-dimensional alarm index calculation: For each connected region, the corresponding original radar data and infrared information are jointly retrieved to calculate the depth change rate (deformation), radial instantaneous displacement velocity (instability trend), and local thermal field anomaly index (stress concentration) of that region. A comprehensive alarm index for roof collapse is constructed through weighted summation.

[0076] Tiered alarm and spatiotemporal filtering: The system has a preset tiered threshold system. When the comprehensive index exceeds the set limit, the corresponding alarm signal (low / medium / high) is triggered. The decision logic incorporates temporal continuity verification, requiring that abnormal features have spatiotemporal consistency in continuous multi-frame observations, thereby effectively filtering transient interference and ensuring high confidence of alarm results.

[0077] Through the above steps, this application can overcome the limitations of a single sensor in the underground coal mine environment. Even under conditions such as dust obstruction, insufficient light, and water mist interference, it can still utilize the penetrability of radar, the thermal characteristics of infrared radiation, and the spatial geometric constraints of the working face to adaptively generate virtual points, thereby achieving high-precision and robust detection of the coal mining face and coal wall spalling.

[0078] On the other hand, this application also provides a detection system for roof collapse and coal wall spalling in coal mining faces. Figure 3 This is a structural block diagram of a detection system for roof collapse and coal wall spalling in a coal mining face, according to an embodiment of this application. Figure 3 As shown, the system includes: an acquisition module 30, a processing module 31, and a generation module 32.

[0079] The acquisition module 30 is used to acquire multimodal data of the coal mining face, including visible light images, infrared images, and radar point cloud data. This module can be composed of a multi-sensor array integrated within an explosion-proof enclosure.

[0080] The computation and processing module 31 is used to perform core data processing and feature extraction. First, the radar point cloud data is projected onto the imaging plane of the visible light image to generate a sparse depth projection map. Then, a preset virtual point generation strategy (including logic such as visual degradation evaluation, working surface cross-sectional contour fitting, and dual-source adaptive interpolation) is invoked to densify the sparse depth projection map, resulting in a dense depth map aligned with the pixels of the visible light image. Next, this module runs a pre-built multimodal Transformer fusion network to encode features from the visible light image, infrared image, radar point cloud data, and dense depth map, respectively, to obtain visual features and radar geometric features.

[0081] The generation module 32 is used to perform feature fusion and result output. This module utilizes a multi-stage cross-modal attention mechanism to perform deep interactive fusion of visual features and radar geometric features to generate unified fused features. Finally, the unified fused features are upsampled and reconstructed to output a semantic segmentation map for detecting roof falls and coal wall spalling in coal mining faces, and alarm signals can be further generated based on this semantic segmentation map.

[0082] This system integrates visible light, infrared, and radar data, and introduces a virtual point generation strategy based on environmental perception and geometric priors. It achieves accurate physical-level feature alignment under conditions of visual degradation and sparse point clouds, ensuring perception continuity in extreme environments. By utilizing a multi-stage cross-modal attention mechanism to deeply interact with heterogeneous features, it enhances the recognition accuracy of roof fall and sidewall boundary, and combines a multi-dimensional signal joint verification strategy to significantly reduce the false alarm rate, achieving robust and high-precision real-time safety monitoring in complex coal mining scenarios.

[0083] In one embodiment, Figure 4 This is a schematic diagram of the internal structure of a computer device according to an embodiment of this application, such as... Figure 4 As shown, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 4 As shown, the computer device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides the environment for the operating system, the computer programs are executed by the processor to implement a method for detecting roof collapse and coal wall spalling in coal mining faces, and the database stores data.

[0084] Those skilled in the art will understand that Figure 4 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combinations of certain components, or different component arrangements. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combinations of these technical features do not contradict each other, they should be considered within the scope of this specification.

[0085] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting roof fall and coal wall spalling in coal mining faces, characterized in that, The method includes: Acquire multimodal data of the coal mining face, including visible light images, infrared images, and radar point cloud data; The radar point cloud data is projected onto the imaging plane of the visible light image to generate a sparse depth projection map. The sparse depth projection map is then densified using a preset virtual point generation strategy to obtain a dense depth map that is aligned with the pixels of the visible light image. By using a pre-constructed multimodal Transformer fusion network, the visible light image, the infrared image, the radar point cloud data, and the dense depth map are respectively feature-encoded to obtain visual features and radar geometric features; By utilizing a multi-stage cross-modal attention mechanism, the visual features and the radar geometric features are deeply interacted and fused to generate unified fused features. The unified fused features are then upsampled and reconstructed to output a semantic segmentation map that characterizes the risk areas of roof collapse and coal wall spalling in the coal mining face.

2. The detection method according to claim 1, characterized in that, The visual features include visible light features and infrared features, and the radar geometric features include point features and projection features. Feature encoding is performed on the visible light image, the infrared image, and the radar point cloud data, including: The visible light image and the infrared image are respectively input into a visual Transformer encoder, and image slice embedding, position encoding and multi-layer self-attention calculation are performed respectively to obtain visible light features and infrared features. The radar point cloud data is input into the point cloud feature extraction module to extract the point features. The dense depth map is processed by a lightweight convolutional layer to obtain projection features. The point features and the projection features are used as the radar geometric features.

3. The method according to claim 2, characterized in that, Image slice embedding, position encoding, and multi-layer self-attention calculation are performed separately to obtain visible light and infrared features, including: The input image is divided into grids, and each block is flattened and linearly mapped to a low-dimensional vector to obtain the image slice embedding vector. Learnable position embedding parameters are added to the image slice embedding vector to generate an input sequence that preserves the spatial topological information of the coal wall and roof. In multi-layer self-attention computation, the input sequence is transformed linearly to generate a query vector, a key vector, and a value vector. Attention weights are obtained by calculating the dot product similarity between the query vector and the key vector. The value vector is then weighted and summed using the attention weights to obtain visible light features and infrared features containing context-aware information.

4. The detection method according to claim 2, characterized in that, A multi-stage cross-modal attention mechanism is used to perform deep interactive fusion of the visual features and the radar geometric features, including: A bidirectional cross-attention calculation is performed between the visible light features and the infrared features to complement the texture information and the infrared radiation information caused by rock mass stress, thereby obtaining image fusion features; The image fusion features and the radar geometric features are cross-attention calculated to utilize the radar's depth constraints and motion vector information to constrain the image features and generate the unified fusion features.

5. The detection method according to claim 1, characterized in that, The sparse depth projection map is densified using a preset virtual point generation strategy, including: Calculate the local contrast and high-frequency components of the visible light image, and the thermal difference distribution characteristics of the infrared image; Based on the local contrast, the high-frequency components, and the thermal difference distribution characteristics, a visual degradation confidence map is constructed. The visual degradation confidence map is used to characterize the effective weight of visual information at each pixel position in an environment where the dust concentration is greater than a first preset threshold or the light intensity is lower than a second preset threshold. Based on the radar point cloud data, the working surface cross-sectional contour line is fitted to construct a prior geometric model of the working surface, and geometric virtual points are generated based on the prior geometric model of the working surface. A dual-source adaptive fusion strategy is adopted to perform densification processing based on the texture features of the visual degradation confidence map, the geometric virtual points, and the visible light image to obtain the dense depth map.

6. The detection method according to claim 5, characterized in that, Fitting the working face cross-sectional contour line based on the radar point cloud data to construct a prior geometric model of the working face, and generating geometric virtual points based on the prior geometric model of the working face, includes: Nonlinear spatial fitting is performed on the radar point cloud data to obtain the coal wall cross-sectional profile line representing the current working face cross-sectional shape, and the coal wall cross-sectional profile line is used as the prior geometric model of the working face. Determine the coordinate gap between adjacent real radar sampling points, and use the coal wall cross-sectional profile to calculate the optimal interpolation trajectory that spans the coordinate gap and connects adjacent sampling points; Along the optimal interpolation trajectory, geometric virtual points are generated whose spatial distribution conforms to the natural curvature of the coal face and roof.

7. The detection method according to claim 6, characterized in that, The dense depth map is generated using a dual-source adaptive fusion strategy, including: A pixel weight mask is generated based on the relationship between the validity weight of each pixel in the visual degradation confidence map and a preset threshold. When the weighted mask indicates that the corresponding position in the visible light image is valid, the texture features of the visible light image are used to guide depth interpolation; When the validity of the corresponding position in the visible light image indicated by the weighted mask is lower than the effective threshold, depth filling is performed based on the depth value of the geometric virtual point to achieve depth map reconstruction in the area where visual information is missing.

8. The detection method according to claim 1, characterized in that, The method also includes early warning based on multi-physics joint analysis: Based on the semantic segmentation graph, a connectivity analysis is performed on the identified roof fall and side spalling areas of the coal mining face; By combining the depth change rate and instantaneous displacement velocity in the radar point cloud data, and the temperature anomaly characteristics in the infrared image, a comprehensive alarm index for roof collapse is calculated. When the comprehensive alarm index of the roof collapse exceeds the preset total threshold, or when any physical quantity among the depth change rate and instantaneous displacement velocity exceeds the corresponding sub-item safety threshold, a disaster alarm signal of the corresponding level is generated.

9. A detection system for roof fall and coal wall spalling in a coal mining face, characterized in that, The system includes: an acquisition module, a processing module, and a generation module, wherein: The acquisition module is used to acquire multimodal data of the coal mining face, including visible light images, infrared images, and radar point cloud data. The computational processing module is used to project the radar point cloud data onto the imaging plane of the visible light image to generate a sparse depth projection map, and to perform a densification process on the sparse depth projection map using a preset virtual point generation strategy to obtain a dense depth map aligned with the pixels of the visible light image. Furthermore, through a pre-constructed multimodal Transformer fusion network, the visible light image, the infrared image, the radar point cloud data, and the dense depth map are respectively feature-encoded to obtain visual features and radar geometric features; The generation module is used to perform deep interactive fusion of the visual features and the radar geometric features using a multi-stage cross-modal attention mechanism to generate unified fused features, upsample and reconstruct the unified fused features, and output a semantic segmentation map to characterize the risk areas of roof collapse and coal wall spalling in the coal mining face.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Images