A coal seam structure real-time detection method based on multi-modal data
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHALAI NUOER COAL IND CO LTD
- Filing Date
- 2026-02-10
- Publication Date
- 2026-06-02
Smart Images
Figure CN122135203A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of coal seam structure detection technology, and specifically to a real-time coal seam structure detection method based on multimodal data. Background Technology
[0002] Coal seam structure is a crucial factor affecting coal mining safety and resource utilization. Traditional detection methods, such as ground-penetrating radar (GPR) or acoustic detection, often rely on a single data source, such as geological exploration data or physical monitoring data. Specifically, GPR is a non-destructive detection method used to detect coal seam structure. It utilizes the propagation characteristics of electromagnetic waves in different media. By transmitting and receiving electromagnetic waves, it acquires the reflected signals from underground layers. Based on the time and intensity of the electromagnetic wave reflection, GPR can create a subsurface stratigraphic map, revealing the thickness, distribution, and structural features of the coal seam, such as faults, cavities, or fissures. This method offers advantages such as high spatial resolution and real-time performance, and can detect deeper layers, making it suitable for complex geological environments. However, the accuracy of GPR is significantly affected by factors such as groundwater content and soil mineral composition, and its detection depth is limited.
[0003] Acoustic wave detection obtains information about the coal seam structure by exciting sound waves in the seam and detecting their propagation characteristics. Sound waves travel at different speeds in different media; by measuring the propagation time, reflected waves, and refracted waves, the density, porosity, and fracture characteristics of the coal seam can be inferred. Acoustic wave detection is mainly suitable for shallower coal seams and can effectively detect microstructures such as cracks and cavities. Its advantages include simple equipment and low cost, making it suitable for routine monitoring in coal mines. Its disadvantages include significant susceptibility to environmental noise and limited effectiveness in detecting deep coal seams.
[0004] The inventors of this application have discovered through research that the single data source of traditional coal seam structure detection methods cannot fully reflect the complexity and changes of coal seams, which can easily lead to local or systemic errors in judgment. Summary of the Invention
[0005] To address the technical problem that existing coal seam structure detection methods, relying on a single data source, cannot fully reflect the complexity and changes of the coal seam, and are prone to local or systemic errors in judgment, this invention provides a real-time coal seam structure detection method based on multimodal data. This method can significantly improve detection efficiency and accuracy, and also has an early warning function for the overall working status of the coal seam.
[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:
[0007] A real-time detection method for coal seam structure based on multimodal data includes the following steps:
[0008] S1. Construction of Coal Seam Multimodal Surface Sensing Dataset: Image sequences of exposed coal seam surfaces and point cloud sequences of corresponding regions are collected, and time synchronization and spatial registration are completed to form multimodal sample input under a unified coordinate system; the multimodal samples are used as dataset input, and the ground truth values of cracks and cavities are labeled for each sample, and the ground truth values of spalling and bulging deformation are further labeled, thereby constructing a coal seam multimodal surface sensing dataset for model training;
[0009] S2. Construction of a coal seam hazard feature perception model based on a multi-attention layer: The coal seam multimodal surface perception dataset sample input constructed in step S1 is used as the model input. In the prediction stage, the coal seam surface is structurally layered based on the registration point cloud, and the structural layering unit is used as the basic object for feature extraction. The coal seam hazard feature perception model is trained using the ground truth values of cracks, cavities, spalling, and bulging deformation as model inputs.
[0010] S3. Construction of Coal Seam Hazard Characterization Map and Design of Hazard Analysis Algorithm: Based on the coal seam hazard features detected by the coal seam hazard feature perception model in step S2, a coal seam hazard characterization map is innovatively constructed, and a hazard analysis algorithm is designed to accurately determine the safety status of the coal seam: First, each coal seam hazard feature is used as a node in the graph through node construction, and node features are defined by combining coal seam hazard feature label information and location information. Then, the Euclidean distance and cosine distance between nodes are calculated as the distance feature and similarity feature of the edge, respectively, and finally the coal seam hazard characterization map is constructed.
[0011] Next, based on the Laplace operator, the adjacency matrix of the coal seam is constructed and the weighted Laplace matrix is calculated. The hazard patterns of the coal seam region are analyzed using the matrix eigenvalues. Furthermore, an improved graph clustering algorithm is used to identify high-risk areas of the coal seam, and cut points are analyzed using a depth-first search algorithm to assess the propagation risk of the coal seam. Finally, by combining the Laplace quantitative assessment index of the coal seam and the graph clustering hazard factors of the coal seam hazard characterization map, the hazard score of the coal seam is calculated, and the hazard level of the coal seam is determined according to a set threshold, thus providing a scientific basis for coal mine safety monitoring and risk assessment.
[0012] S4. Model Deployment and Real-time Monitoring Application: The coal seam hazard characteristic perception model constructed in step S2, the coal seam hazard characterization map constructed in step S3, and the hazard analysis algorithm design are deployed to the local computer in the mine. Online inference is performed on the multimodal sample inputs that are collected in real time and registered, and the detection results of cracks, cavities, spalling and bulging deformation, as well as the overall hazard level of the coal seam working face, are output for on-site safety management and operation decision support.
[0013] Furthermore, step S1, the construction of the coal seam multimodal surface sensing dataset, specifically includes:
[0014] S11, Multimodal Data Acquisition:
[0015] Image acquisition devices and laser scanning devices are deployed in the coal seam operation area to be monitored. The image acquisition devices are used to collect image data of the exposed surface of the coal seam, and the laser scanning devices are used to collect three-dimensional point cloud data of the coal seam surface in the corresponding area; based on the acquisition time... As an index, the image acquisition device outputs image frames. The laser scanning device outputs point cloud frames. And record the timestamps of the acquired image data respectively. timestamps of point cloud data This yields the original multimodal input data stream: image sequence. With point cloud sequence ;
[0016] S12. Time synchronization and frame-level pairing:
[0017] The image sequence obtained in step S11 Point cloud sequence and its timestamp , As input, a synchronous triggering strategy is used for time alignment: for each image frame During the preset time interval Select the corresponding point cloud frame inside. As point cloud matching frames To satisfy This results in frame-level pairing; the output is a time-synchronized multimodal data frame. And assign a uniform synchronization timestamp to each frame pair. ;
[0018] S13. Spatial calibration and unified coordinate system registration:
[0019] The multimodal data frame output in step S12 As input, the intrinsic parameters of the image acquisition device are used. and the external parameter transformation relationship between the two devices. Spatial calibration is performed on the image acquisition device and the laser scanning device, wherein... This involves a rigid body transformation from the laser coordinate system to the camera coordinate system; the calibration parameters are used for each multimodal data frame. Spatial registration: Specifically, matching point clouds with frames Transform to a unified coordinate system to obtain the registration point cloud frame and establish image frames. With registration point cloud frames The spatial correspondence is output as a unified spatial data pair for spatial registration. ;
[0020] S14. Multimodal input sample organization for real-time monitoring:
[0021] The multimodal data output in step S13 is used to... As basic input samples, for input samples The hazard characteristics of the coal seams present in the sample are calibrated, and the calibration results include the location information of the hazard characteristics detected in the input sample. and tag information Specifically, location information Therefore, the dangerous characteristics of the coal seam are described in the actual coal seam mine coordinate system. Three-dimensional coordinates: The mine coordinate system is consistent with the actual coordinate system used in the mine during data acquisition to ensure the correspondence between location information and the actual environment; tag information. The smallest bounding rectangle that includes the area where the coal seam’s hazardous features are located, along with the corresponding type label. The type label specifically includes four categories: cracks, cavities, spalling, and bulging deformation.
[0022] S15. Creation of a multimodal surface sensing dataset for coal seams:
[0023] Based on the data processing procedures described in steps S12 to S14, the image sequence is processed. With point cloud sequence Temporal alignment, spatial alignment, and image calibration are performed sequentially to obtain the complete input dataset. and the complete output dataset The input and output datasets are then combined to obtain the coal seam multimodal surface sensing dataset. .
[0024] Furthermore, step S2, based on the coal seam hazard feature perception model constructed using a multi-attention layer, specifically includes:
[0025] S21. Design of Coal Seam Hazard Characteristic Structural Unit Sensing Module:
[0026] Input data Image data in The data is fed into a pre-trained ResNet model to perform preliminary feature extraction and obtain the features. Then the features The data is fed into five cascaded structural refinement sensing units for further feature calculation and processing to obtain the features. The features are then processed through three sequentially connected fully connected layers. Specifically, each structurally refined perceptual unit includes a convolutional layer, a pooling layer, and a fully connected layer connected in sequence.
[0027] Input data Point cloud data First, the data is fed into four sequentially connected fully connected layers for preliminary feature processing to obtain the features. Then the features The data is fed into three sequentially connected adaptive convolutional layers for further processing to obtain the features. The features are then obtained through two consecutively connected average pooling layers. ;
[0028] Features obtained based on the above process and characteristics This invention constructs Each structural hierarchical unit completes the feature and Fusion processing and analysis were performed, and a system including... An expert network of hierarchical units with parallel structures, and The final output feature of each hierarchical structural unit is That is, the first The output features corresponding to each structural hierarchical unit are ;
[0029] S22. Design of the coal seam structure unit feature enhancement processing module:
[0030] Based on the results obtained in step S21 Features This invention designs a coal seam structural unit feature enhancement processing module for... Further comprehensive analysis is performed on these features; in the coal seam structure unit feature deepening processing module, the features are first analyzed through two sequentially connected fully connected layers. The features are obtained by integrating the features. Then the features Features are obtained by feeding the data into two convolutional layers with kernel size of 3*3. Next, the features The features are obtained by feeding the data into two convolutional layers with a kernel size of 5*5. Then the features Features are obtained by feeding the data into two convolutional layers with kernel size of 7*7. Finally, the features Features are obtained by calculating the weights of the features using the Sigmoid activation function. This strengthens the characteristics that are highly correlated with the dangerous features of coal seams;
[0031] S23. Design of Coal Seam Hazard Characteristic Output Module:
[0032] Based on the features obtained during the analysis process in step S22 , , , This invention designs a coal seam hazard feature output module based on a multi-attention fusion and multi-convolution mechanism to obtain the final prediction result of the model; specifically, the features Three sequentially connected adaptive convolutional layers are used to further extract coal seam hazard features. Then, a ReLU activation function is used to achieve non-linear activation, followed by an attention layer and feature extraction. Features are obtained through fusion processing To take into account the multi-scale distribution information of coal seam hazards, and then the features The processed features are fed into three sequentially connected adaptive convolutional layers for further feature extraction, and then passed through an attention layer and feature decomposition layer. Further fusion processing is performed to obtain features. Then the features The data is then fed into three sequentially connected adaptive convolutional layers for further processing, and the processed features are then passed through an attention layer and a feature decomposition layer. The final fusion process is performed to obtain the features. , The detection results of coal seam hazard characteristics were obtained after processing with three adaptive convolutional layers and a maximum pooling layer. Test results Including hazard label information and location information ;
[0033] S24. Model Training: After the above model structure design is completed, the model is trained using the dataset constructed in step S1, employing the Adam optimizer and stochastic gradient descent algorithm to obtain the trained coal seam hazard characteristic perception model.
[0034] Furthermore, the method for constructing the structural layered unit in step S21 is as follows:
[0035] In the constructed hierarchical structural units, features are processed through image branches and point cloud branches respectively. and characteristics Process them separately; in the image branch, features Low-level feature extraction is performed through two sequentially connected convolutional layers, followed by an adaptive convolutional layer to enhance the representation of local features and obtain the final feature. Then, a self-attention mechanism is used to focus on the features. Weighted summaries are applied to focus on key regions in the image, and finally, a fully connected layer is used to extract global information, resulting in a more abstract representation of image features. In point cloud branches, point cloud features First, a fully connected layer performs initial processing to gradually reduce the feature dimension. Then, an adaptive convolutional layer is used to enhance the local detail features of the point cloud. Next, a spatial pyramid pooling layer is used for multi-scale pooling to capture information at different scales. Finally, an average pooling layer reduces the spatial dimension of the features to obtain a unified feature representation. ;
[0036] Features were obtained after processing via a dual-branch path. and characteristics Subsequently, a multi-head self-attention mechanism was designed to focus on features. and characteristics Cross-modal fusion is performed, followed by global pooling to fuse global information, resulting in the fused feature representation. Finally, the features The output features are obtained by feeding the data into two interconnected fully connected layers and a Softmax activation function for final processing. .
[0037] Furthermore, the construction of the coal seam hazard characterization map in step S3 includes:
[0038] (1) Node construction: Based on the coal seam hazard characteristic detection results in step S2. Each coal seam hazard feature is treated as a node in the graph, and its coal seam hazard feature label information is assigned. and location information As this node Features;
[0039] (2) Edge construction: Based on the location information obtained from the detection Calculate the Euclidean distance between the nodes respectively. As the distance edge feature between nodes, and based on the detected label information Calculate the cosine distance between the nodes respectively. As a feature of dangerous similar edges between nodes;
[0040] (3) Based on the above node and edge construction method, the nodes and edges of the coal seam hazard characterization map are constructed sequentially. After the construction is completed, the coal seam hazard characterization map is obtained. .
[0041] Furthermore, in step S3, based on the Laplace operator, the adjacency matrix of the coal seam is constructed and the weighted Laplace matrix is calculated. The analysis of the hazard patterns of the coal seam region using the matrix eigenvalues includes:
[0042] (1) Based on the obtained coal seam hazard characterization map Construct the adjacency matrix of the coal seam Adjacency matrix Each element in Represents a node and nodes The adjacency distance between them, where For the image The total number of nodes in the system;
[0043] (2) Based on adjacency matrix Construct a diagonal matrix To describe the strength of the connection between each node and other nodes. ;
[0044] (3) Calculate the hazard characterization diagram of this coal seam. Weighted Laplace matrix To measure the relationships between nodes and the correlation between coal seam regions, ;
[0045] (4) Subsequently, based on the weighted Laplace matrix Further calculation of all eigenvalues of this matrix is needed, since low eigenvalues correspond to the main structure of the graph. The smallest eigenvalues reflect the main hazardous characteristics of the coal seam region, and these eigenvectors reflect the global hazardous pattern of the coal seam. Therefore, this invention selects the first few eigenvalues from all eigenvalues. The eigenvectors corresponding to the smallest eigenvalues This serves as an indicator for assessing the hazard level of this coal seam, and further calculations are performed separately. vectors Magnitude weighted average and To comprehensively consider the contributions of different feature vectors, and to... As a Laplace quantitative evaluation index for coal seams.
[0046] Furthermore, in step S3, an improved graph clustering algorithm is used to identify high-risk areas of the coal seam, and a depth-first search algorithm is used to analyze cut points and assess the propagation risk of the coal seam, including:
[0047] (1) Based on the obtained coal seam hazard characterization map Define the scope of the domain Then calculate the graph in sequence. All nodes Euclidean distance to other nodes in the graph ,in This represents the total number of coal seam hazard characteristics; subsequently, based on the calculated... and scope of fields If node To the node Euclidean distance Less than Then this node As a node The neighbor nodes, and in turn calculate the distances from each of the remaining nodes in the graph to the node. The Euclidean distance is calculated and compared sequentially with the neighborhood range to obtain the node. The number of neighboring nodes ;
[0048] (2) Calculate the figures sequentially according to the calculation method in process (1) above. The number of neighboring nodes of all nodes After the calculation is completed, the nodes with the most neighboring nodes in the top 50% are retained, and the rest are discarded.
[0049] (3) Repeat the above (2) process three times, and reconnect the remaining nodes after the last iteration to obtain a new enhanced characterization map of coal seam hazard conditions. ;
[0050] (4) The depth-first search algorithm is used to further analyze the cut points of the enhanced characterization map of coal seam hazards in order to find all the cut points in the map and calculate the total number of cut points in the map. The cut point is the graph Removing a node from the middle will cause the graph to be divided into multiple disconnected nodes. Therefore, the number of cut vertices can reflect the fragility of the graph structure.
[0051] (5) The enhanced characterization map of coal seam hazard conditions is calculated sequentially using the Dijkstra algorithm. Propagation risk coefficient of all nodes And the graph is obtained by weighted average. The sum of the transmission risk coefficients ;
[0052] (6) The total number of cut points obtained from (4) above The sum of the transmission risk coefficients obtained in (5) The graph clustering hazard factor of this coal seam hazard characterization map was calculated. .
[0053] Furthermore, in step S3, the coal seam hazard score is calculated by combining the Laplace quantitative assessment index of the coal seam and the graph clustering hazard factor of the coal seam hazard characterization map, and the hazard level of the coal seam is determined according to a set threshold, including:
[0054] Based on the calculated Laplace quantitative evaluation index Clustering risk factors The hazard score of this coal seam was obtained by simultaneous calculation. ,in and This serves as the weighting constant for the hazard score; and based on this hazard score, the coal seam hazard level is determined, with a threshold value set. When the coal seam hazard rating Less than or equal to the threshold When the coal seam is in a normal state, it is determined that the coal seam is in a normal state. Greater than and less than or equal to At that time, the coal seam was determined to be in a low-risk state; when Greater than and less than or equal to At that time, the coal seam was determined to be in a medium-risk state; when Greater than At that time, the coal seam was determined to be in a high-risk state.
[0055] Furthermore, step S4, model deployment and real-time monitoring application, includes:
[0056] S41. Model Deployment: Deploy the coal seam hazard characteristic perception model constructed in step S2, the coal seam hazard situation characterization map constructed in step S3, and the hazard analysis algorithm design to the local computer in the mine.
[0057] S42. Collecting data to be detected inside the mine: Deploying image acquisition devices at the coal seam working face to collect real-time image data of the coal seam. Deploy laser scanning devices to collect real-time point cloud data of coal seams ;
[0058] S43. Risk Event Identification and Comprehensive Risk Assessment: The image data collected in step S42... and point cloud data The data is input into the coal seam hazard feature perception model deployed on the local computer, and the coal seam hazard feature detection results are obtained. Based on this, the test results will be simultaneously transmitted. The data is fed into the designed hazard analysis algorithm to comprehensively determine the real-time operating status of the coal seam, ultimately yielding the coal seam risk assessment level. ;
[0059] S44. Risk Information Feedback: Feedback on coal seam hazard characteristics detected by the model. And the coal seam risk assessment level determined in real time by the algorithm. Promptly report to the safety management personnel at the coal seam working face.
[0060] Compared with existing technologies, the real-time coal seam structure detection method based on multimodal data provided by this invention has the following advantages:
[0061] 1. This invention combines multimodal data and deep learning technology, which can significantly improve the detection accuracy and efficiency of dangerous features of coal seam structure, avoiding the limitations of traditional single data sources.
[0062] 2. Multimodal data acquisition and spatiotemporal registration technology is adopted to ensure the accuracy and consistency of data; a coal seam hazard feature perception model based on multi-attention layers is used to effectively improve the perception capability of hazard features; by constructing a coal seam hazard characterization map and designing a hazard analysis algorithm, the accurate determination of coal mine safety status is achieved, providing a scientific basis for mine safety management.
[0063] 3. An innovative coal seam hazard characterization map was constructed, and a graph clustering algorithm was designed to achieve effective early warning of coal seam conditions, which effectively improved the decision support capability for safe operation in coal mines and reduced the occurrence of coal mine accidents. Attached Figure Description
[0064] Figure 1 This is a schematic diagram of the real-time detection method for coal seam structure based on multimodal data provided by the present invention.
[0065] Figure 2 This is a schematic diagram of the coal seam hazard characteristic perception model provided by the present invention.
[0066] Figure 3 This is a schematic diagram of the hierarchical unit architecture provided by the present invention. Detailed Implementation
[0067] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.
[0068] Please refer to Figure 1 As shown, this invention provides a real-time detection method for coal seam structure based on multimodal data, comprising the following steps:
[0069] S1. Construction of Coal Seam Multimodal Surface Sensing Dataset: Collect image sequences of exposed coal seam surfaces and point cloud sequences of corresponding regions, and complete time synchronization and spatial registration to form multimodal sample input under a unified coordinate system; use the multimodal samples as dataset input, label each sample with crack ground truth, cavity ground truth, and further label spalling ground truth and bulging deformation ground truth, thereby constructing a coal seam multimodal surface sensing dataset for model training, providing data support for the model in step S2;
[0070] S2. Construction of a coal seam hazard feature perception model based on a multi-attention layer: The coal seam multimodal surface perception dataset sample input constructed in step S1 is used as the model input. In the prediction stage, the coal seam surface is structurally layered based on the registration point cloud, and the structural layering unit is used as the basic object for feature extraction. The coal seam hazard feature perception model is trained using the ground truth values of cracks, cavities, spalling, and bulging deformation as model inputs.
[0071] S3. Construction of Coal Seam Hazard Characterization Map and Design of Hazard Analysis Algorithm: Based on the coal seam hazard features detected by the coal seam hazard feature perception model in step S2, a coal seam hazard characterization map is innovatively constructed, and a hazard analysis algorithm is designed to accurately determine the safety status of the coal seam: First, each coal seam hazard feature is used as a node in the graph through node construction, and node features are defined by combining coal seam hazard feature label information and location information. Then, the Euclidean distance and cosine distance between nodes are calculated as the distance feature and similarity feature of the edge, respectively, and finally the coal seam hazard characterization map is constructed.
[0072] Next, based on the Laplace operator, the adjacency matrix of the coal seam is constructed and the weighted Laplace matrix is calculated. The hazard patterns of the coal seam region are analyzed using the matrix eigenvalues. Furthermore, an improved graph clustering algorithm is used to identify high-risk areas of the coal seam, and cut points are analyzed using a depth-first search algorithm to assess the propagation risk of the coal seam. Finally, by combining the Laplace quantitative assessment index of the coal seam and the graph clustering hazard factors of the coal seam hazard characterization map, the hazard score of the coal seam is calculated, and the hazard level of the coal seam is determined according to a set threshold, thus providing a scientific basis for coal mine safety monitoring and risk assessment.
[0073] S4. Model Deployment and Real-time Monitoring Application: The coal seam hazard characteristic perception model constructed in step S2, the coal seam hazard characterization map constructed in step S3, and the hazard analysis algorithm design are deployed to the local computer in the mine. Online inference is performed on the multimodal sample inputs that are collected in real time and registered, and the detection results of cracks, cavities, spalling and bulging deformation, as well as the overall hazard level of the coal seam working face, are output for on-site safety management and operation decision support.
[0074] As a specific embodiment, due to differences in underground working environments, coal seam exposure conditions, and equipment layout locations in different coal mines, in order to ensure the integrity and consistency of the input data required for subsequent structural unit construction and multimodal joint modeling, this invention collects multimodal sensing data of the coal seam surface and performs time synchronization and spatial registration of image data and point cloud data. Accordingly, step S1, the construction of the coal seam multimodal surface sensing dataset, specifically includes:
[0075] S11, Multimodal Data Acquisition:
[0076] Image acquisition devices and laser scanning devices are deployed in the coal seam operation area to be monitored. The image acquisition devices are used to collect image data of the exposed surface of the coal seam, and the laser scanning devices are used to collect three-dimensional point cloud data of the coal seam surface in the corresponding area; based on the acquisition time... As an index, the image acquisition device outputs image frames. The laser scanning device outputs point cloud frames. And record the timestamps of the acquired image data respectively. timestamps of point cloud data This yields the original multimodal input data stream: image sequence. With point cloud sequence .
[0077] S12. Time synchronization and frame-level pairing:
[0078] The image sequence obtained in step S11 Point cloud sequence and its timestamp , As input, a synchronous triggering strategy is used for time alignment: for each image frame During the preset time interval Select the corresponding point cloud frame inside. As point cloud matching frames To satisfy This results in frame-level pairing; the output is a time-synchronized multimodal data frame. And assign a uniform synchronization timestamp to each frame pair. .
[0079] S13. Spatial calibration and unified coordinate system registration:
[0080] The multimodal data frame output in step S12 As input, the intrinsic parameters of the image acquisition device are used. and the external parameter transformation relationship between the two devices. Spatial calibration is performed on the image acquisition device and the laser scanning device, wherein... This involves a rigid body transformation from the laser coordinate system to the camera coordinate system; the calibration parameters are used for each multimodal data frame. Spatial registration: Specifically, matching point clouds with frames Transform to a unified coordinate system to obtain the registration point cloud frame and establish image frames. With registration point cloud frames The spatial correspondence is output as a unified spatial data pair for spatial registration. .
[0081] S14. Multimodal input sample organization for real-time monitoring:
[0082] To meet the real-time processing requirements of subsequent structural unit construction and adaptive refinement, the multimodal data output in step S13 is used to... As the basic input sample, its input sample The hazard characteristics of the coal seams present in the sample are calibrated, and the calibration results include the location information of the hazard characteristics detected in the input sample. and tag information Specifically, location information Therefore, the dangerous characteristics of the coal seam are described in the actual coal seam mine coordinate system. Three-dimensional coordinates: The mine coordinate system is consistent with the actual coordinate system used in the mine during data acquisition to ensure the correspondence between location information and the actual environment; tag information. The smallest bounding rectangle that includes the area where the coal seam’s hazardous features are located, along with the corresponding type label, is defined as follows: cracks, cavities, spalling, and bulging deformation.
[0083] S15. Creation of a multimodal surface sensing dataset for coal seams:
[0084] Based on the data processing procedures described in steps S12 to S14, the image sequence is processed. With point cloud sequence Temporal alignment, spatial alignment, and image calibration are performed sequentially to obtain the complete input dataset. and the complete output dataset The input and output datasets are then combined to obtain the coal seam multimodal surface sensing dataset. .
[0085] As a specific embodiment, to further improve the accuracy and efficiency of detecting coal seam hazard features, this invention performs independent and joint analysis on input image data and input point cloud data, and uses a multi-attention layer perception mechanism in the designed coal seam hazard feature perception model to further target different types of hazard features present in the coal seam, thereby further improving the robustness of coal seam hazard feature perception. Please refer to [link / reference] for details. Figure 2 As shown, step S2, based on the coal seam hazard feature perception model with multiple attention layers, specifically includes:
[0086] S21. Design of Coal Seam Hazardous Feature Structural Unit Perception Module: This invention designs a coal seam hazardous feature structural unit perception module to perform more refined unit differentiation on the initial input data, thereby further improving the accuracy of perceiving minute defect features of coal seams; specifically:
[0087] Input data Image data in The data is fed into a pre-trained ResNet model to perform preliminary feature extraction and obtain the features. Then the features The data is fed into five cascaded structural refinement sensing units for further feature calculation and processing to obtain the features. The features are then processed through three sequentially connected fully connected layers. Specifically, each structurally refined perceptual unit includes a convolutional layer, a pooling layer, and a fully connected layer connected in sequence.
[0088] Input data Point cloud data First, the data is fed into four sequentially connected fully connected layers for preliminary feature processing to obtain the features. Then the features The data is fed into three sequentially connected adaptive convolutional layers for further processing to obtain the features. The features are then obtained through two consecutively connected average pooling layers. Compared to conventional convolutional layers, the kernel parameters of adaptive convolutional layers can be adaptively adjusted according to the input data, thus achieving better feature extraction efficiency for various types of coal seam hazard characteristics.
[0089] Features obtained based on the above process and characteristics This invention constructs Each structural hierarchical unit completes the feature and Fusion processing and analysis were performed, and a system including... An expert network of hierarchical units with parallel structures, and The final output feature of each hierarchical structural unit is That is, the first The output features corresponding to each structural hierarchical unit are .
[0090] S22. Design of the coal seam structure unit feature enhancement processing module:
[0091] Based on the results obtained in step S21 Features This invention designs a coal seam structural unit feature enhancement processing module for... Further comprehensive analysis is performed on these features; in the coal seam structure unit feature deepening processing module, the features are first analyzed through two sequentially connected fully connected layers. The features are obtained by integrating the features. Then the features Features are obtained by feeding the data into two convolutional layers with kernel size of 3*3. Next, the features The features are obtained by feeding the data into two convolutional layers with a kernel size of 5*5. Then the features Features are obtained by feeding the data into two convolutional layers with kernel size of 7*7. This approach utilizes multi-scale convolution to extract information at different scales, capturing features from local to global perspectives, thereby enhancing the model's sensitivity to multi-scale coal seam hazard characteristics. Finally, the features are... Features are obtained by calculating the weights of the features using the Sigmoid activation function. This strengthens the characteristics that are highly correlated with the dangerous features of coal seams.
[0092] S23. Design of Coal Seam Hazard Characteristic Output Module:
[0093] Based on the features obtained during the analysis process in step S22 , , , This invention designs a coal seam hazard feature output module based on a multi-attention fusion and multi-convolution mechanism to obtain the final prediction result of the model; specifically, the features Three sequentially connected adaptive convolutional layers are used to further extract coal seam hazard features. Then, a ReLU activation function is used to achieve non-linear activation, followed by an attention layer and feature extraction. Features are obtained through fusion processing To take into account the multi-scale distribution information of coal seam hazards, and then the features The processed features are fed into three sequentially connected adaptive convolutional layers for further feature extraction, and then passed through an attention layer and feature decomposition layer. Further fusion processing is performed to obtain features. Then the features The data is then fed into three sequentially connected adaptive convolutional layers for further processing, and the processed features are then passed through an attention layer and a feature decomposition layer. The final fusion process is performed to obtain the features. , The detection results of coal seam hazard characteristics were obtained after processing with three adaptive convolutional layers and a maximum pooling layer. Test results Including hazard label information and location information .
[0094] S24. Model Training: After the above model structure design is completed, the model is trained using the dataset constructed in step S1, employing the Adam optimizer and stochastic gradient descent algorithm to obtain the trained coal seam hazard characteristic perception model.
[0095] For a specific embodiment, please refer to Figure 3 As shown, the method for constructing the structural layered unit in step S21 is as follows:
[0096] In the constructed hierarchical structural units, features are processed through image branches and point cloud branches respectively. and characteristics Process them separately; in the image branch, features Low-level feature extraction is performed through two sequentially connected convolutional layers, followed by an adaptive convolutional layer to enhance the representation of local features and obtain the final feature. Then, a self-attention mechanism is used to focus on the features. Weighted summaries are applied to focus on key regions in the image, and finally, a fully connected layer is used to extract global information, resulting in a more abstract representation of image features. In point cloud branches, point cloud features First, a fully connected layer performs initial processing to gradually reduce the feature dimension. Then, an adaptive convolutional layer is used to enhance the local detail features of the point cloud. Next, a spatial pyramid pooling layer is used for multi-scale pooling to capture information at different scales. Finally, an average pooling layer reduces the spatial dimension of the features to obtain a unified feature representation. ;
[0097] Features were obtained after processing via a dual-branch path. and characteristics Subsequently, a multi-head self-attention mechanism was designed to focus on features. and characteristics Cross-modal fusion is performed, followed by global pooling to fuse global information, resulting in the fused feature representation. Finally, the features The output features are obtained by feeding the data into two interconnected fully connected layers and a Softmax activation function for final processing. .
[0098] Based on the coal seam hazard characteristics detected in step S2, this invention constructs a coal seam hazard characterization map based on the coal seam hazard characteristics and designs a coal seam hazard analysis algorithm, thereby further accurately determining the overall safety status of the coal seam based on the coal seam hazard detected in step S2.
[0099] As a specific example, during actual testing, the coal seam exhibits a common... Different risk characteristics Based on the hazard feature detection results in step S2 above, this invention completes the construction of a coal seam hazard characterization map, thereby providing a data foundation for accurately predicting coal seam hazards. Specifically, the construction of the coal seam hazard characterization map in step S3 includes:
[0100] (1) Node construction: Based on the coal seam hazard characteristic detection results in step S2. Each coal seam hazard feature is treated as a node in the graph, and its coal seam hazard feature label information is assigned. and location information As this node Features;
[0101] (2) Edge construction: After the node construction is completed, in order to further consider the spatial adjacency relationship between coal seam hazard features, the edge construction is based on the detected location information. Calculate the Euclidean distance between the nodes respectively. As the distance edge feature between nodes, and based on the detected label information Calculate the cosine distance between the nodes respectively. As a feature of dangerous similar edges between nodes;
[0102] (3) Based on the above node and edge construction method, the nodes and edges of the coal seam hazard characterization map are constructed sequentially. After the construction is completed, the coal seam hazard characterization map is obtained. .
[0103] In a specific embodiment, step S3 involves constructing the adjacency matrix of the coal seam based on the Laplace operator and calculating the weighted Laplace matrix. The analysis of the hazard patterns of the coal seam region using the matrix eigenvalues includes:
[0104] (1) Based on the above-obtained coal seam hazard characterization diagram Construct the adjacency matrix of the coal seam Adjacency matrix Each element in Represents a node and nodes The adjacency distance between them, where For the image The total number of nodes in the system; the specific calculation method is as follows: ,in The adjacency matrix weight constant;
[0105] (2) Subsequently, in order to further measure the importance of coal seam nodes in the entire region, based on the adjacency matrix A diagonal matrix was constructed. To describe the strength of the connection between each node and other nodes. ;
[0106] (3) Calculate the hazard characterization diagram of this coal seam. Weighted Laplace matrix To measure the relationships between nodes and the correlation between coal seam regions, In constructing this Laplace matrix, the present invention uses... The construction method introduces matrix normalization processing, which can better capture the distribution relationship between coal seam regions;
[0107] (4) Subsequently, based on the weighted Laplace matrix This invention further calculates all eigenvalues of this matrix. Since low eigenvalues correspond to the main structure of the graph, the preceding... The smallest eigenvalues reflect the main hazardous characteristics of the coal seam region, and these eigenvectors reflect the global hazardous pattern of the coal seam. Therefore, this invention selects the first few eigenvalues from all eigenvalues. The eigenvectors corresponding to the smallest eigenvalues This serves as an indicator for assessing the hazard level of this coal seam, and further calculations are performed separately. vectors Magnitude weighted average and To comprehensively consider the contributions of different feature vectors, and to... As a Laplace quantitative evaluation index for coal seams.
[0108] As a specific embodiment, step S3 employs an improved graph clustering algorithm to identify high-risk areas of the coal seam, and uses a depth-first search algorithm to analyze cut points and assess the propagation risk of the coal seam, including:
[0109] (1) Based on the above-obtained coal seam hazard characterization diagram Define the scope of the domain Then calculate the graph in sequence. All nodes Euclidean distance to other nodes in the graph ,in This represents the total number of coal seam hazard characteristics; subsequently, based on the calculated... and scope of fields If node To the node Euclidean distance Less than Then this node As a node The neighbor nodes, and in turn calculate the distances from each of the remaining nodes in the graph to the node. The Euclidean distance is calculated and compared sequentially with the neighborhood range to obtain the node. The number of neighboring nodes ;
[0110] (2) Calculate the figures sequentially according to the calculation method in process (1) above. The number of neighboring nodes of all nodes After the calculation is completed, the nodes with the most neighboring nodes in the top 50% are retained, and the rest are discarded.
[0111] (3) Repeat the above (2) process three times, and reconnect the remaining nodes after the last iteration to obtain a new enhanced characterization map of coal seam hazard conditions. ;
[0112] (4) The depth-first search algorithm is used to further analyze the cut points of the enhanced characterization map of coal seam hazards in order to find all the cut points in the map and calculate the total number of cut points in the map. The cut point is the graph Removing a node will cause the graph to be divided into multiple disconnected nodes. Therefore, the number of cut vertices can reflect the fragility of the graph structure. The more cut vertices there are, the more high-risk areas there are in the coal seam, because the removal of these vertices will cause a significant disruption to connectivity.
[0113] (5) The enhanced characterization map of coal seam hazard conditions is calculated sequentially using the Dijkstra algorithm. Propagation risk coefficient of all nodes And the graph is obtained by weighted average. The sum of the transmission risk coefficients ;
[0114] (6) The total number of cut points obtained from (4) above The sum of the transmission risk coefficients obtained in (5) The graph clustering hazard factor of this coal seam hazard characterization map was calculated. .
[0115] In a specific embodiment, step S3, which combines the Laplace quantitative assessment index of the coal seam and the graph clustering hazard factor of the coal seam hazard characterization map to calculate the hazard score of the coal seam and determine the hazard level of the coal seam based on a set threshold, includes:
[0116] Based on the above calculations, the Laplace quantitative evaluation index is obtained. Clustering risk factors The hazard score of this coal seam was obtained by simultaneous calculation. ,in and This serves as the weighting constant for the hazard score; and based on this hazard score, the coal seam hazard level is determined, with a threshold value set. When the coal seam hazard rating Less than or equal to the threshold When the coal seam is in a normal state, it is determined that the coal seam is in a normal state. Greater than and less than or equal to At that time, the coal seam was determined to be in a low-risk state; when Greater than and less than or equal to At that time, the coal seam was determined to be in a medium-risk state; when Greater than At that time, the coal seam was determined to be in a high-risk state.
[0117] As a specific embodiment, to achieve real-time hazard feature detection and risk warning of coal seam working faces in coal mining sites, this invention deploys a model and an algorithm. The trained coal seam hazard feature perception model and hazard analysis algorithm are deployed to the local computer in the mine. Combined with on-site coal seam image data and point cloud data, a comprehensive risk assessment of the coal seam condition is achieved. Specifically, step S4, model deployment and real-time monitoring application, includes:
[0118] S41. Model Deployment: Deploy the coal seam hazard characteristic perception model constructed in step S2, the coal seam hazard situation characterization map constructed in step S3, and the hazard analysis algorithm design to the local computer in the mine.
[0119] S42. Collecting data to be detected inside the mine: Deploying image acquisition devices at the coal seam working face to collect real-time image data of the coal seam. Deploy laser scanning devices to collect real-time point cloud data of coal seams ;
[0120] S43. Risk Event Identification and Comprehensive Risk Assessment: The image data collected in step S42... and point cloud data The data is input into the coal seam hazard feature perception model deployed on the local computer, and the coal seam hazard feature detection results are obtained. Based on this, the test results will be simultaneously transmitted. The data is fed into the designed hazard analysis algorithm to comprehensively determine the real-time operating status of the coal seam, ultimately yielding the coal seam risk assessment level. (Including normal status, low-risk status, medium-risk status and high-risk status).
[0121] S44. Risk Information Feedback: Feedback on coal seam hazard characteristics detected by the model. And the coal seam risk assessment level determined in real time by the algorithm. Prompt feedback should be given to safety management personnel at the coal seam working face to ensure that they have an accurate understanding of the real-time working status of the coal seam and increase the safety of coal seam mining operations.
[0122] Compared with existing technologies, the real-time coal seam structure detection method based on multimodal data provided by this invention has the following advantages:
[0123] 1. This invention combines multimodal data and deep learning technology, which can significantly improve the detection accuracy and efficiency of dangerous features of coal seam structure, avoiding the limitations of traditional single data sources.
[0124] 2. Multimodal data acquisition and spatiotemporal registration technology is adopted to ensure the accuracy and consistency of data; a coal seam hazard feature perception model based on multi-attention layers is used to effectively improve the perception capability of hazard features; by constructing a coal seam hazard characterization map and designing a hazard analysis algorithm, the accurate determination of coal mine safety status is achieved, providing a scientific basis for mine safety management.
[0125] 3. An innovative coal seam hazard characterization map was constructed, and a graph clustering algorithm was designed to achieve effective early warning of coal seam conditions, which effectively improved the decision support capability for safe operation in coal mines and reduced the occurrence of coal mine accidents.
[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for real-time detection of coal seam structure based on multimodal data, characterized in that, Includes the following steps: S1. Construction of Coal Seam Multimodal Surface Sensing Dataset: Collect image sequences of exposed coal seam surfaces and point cloud sequences of corresponding regions, and complete time synchronization and spatial registration to form multimodal sample input under a unified coordinate system; Using the multimodal samples as the dataset input, the ground truth values of cracks and cavities are labeled for each sample, and the ground truth values of spalling and bulging deformation are further labeled, thereby constructing a coal seam multimodal surface perception dataset for model training. S2. Construction of a coal seam hazard feature perception model based on a multi-attention layer: The coal seam multimodal surface perception dataset sample input constructed in step S1 is used as the model input. In the prediction stage, the coal seam surface is structurally layered based on the registration point cloud, and the structural layering unit is used as the basic object for feature extraction. The coal seam hazard feature perception model is trained using the ground truth values of cracks, cavities, spalling, and bulging deformation as model inputs. S3. Construction of Coal Seam Hazard Characterization Map and Design of Hazard Analysis Algorithm: Based on the coal seam hazard features detected by the coal seam hazard feature perception model in step S2, a coal seam hazard characterization map is innovatively constructed, and a hazard analysis algorithm is designed to accurately determine the safety status of the coal seam: First, each coal seam hazard feature is used as a node in the graph through node construction, and node features are defined by combining coal seam hazard feature label information and location information. Then, the Euclidean distance and cosine distance between nodes are calculated as the distance feature and similarity feature of the edge, respectively, and finally the coal seam hazard characterization map is constructed. Next, based on the Laplace operator, the adjacency matrix of the coal seam is constructed and the weighted Laplace matrix is calculated. The hazard patterns of the coal seam region are analyzed using the matrix eigenvalues. Furthermore, an improved graph clustering algorithm is used to identify high-risk areas of the coal seam, and cut points are analyzed using a depth-first search algorithm to assess the propagation risk of the coal seam. Finally, by combining the Laplace quantitative assessment index of the coal seam and the graph clustering hazard factors of the coal seam hazard characterization map, the hazard score of the coal seam is calculated, and the hazard level of the coal seam is determined according to a set threshold, thus providing a scientific basis for coal mine safety monitoring and risk assessment. S4. Model Deployment and Real-time Monitoring Application: The coal seam hazard characteristic perception model constructed in step S2, the coal seam hazard characterization map constructed in step S3, and the hazard analysis algorithm design are deployed to the local computer in the mine. Online inference is performed on the multimodal sample inputs that are collected in real time and registered, and the detection results of cracks, cavities, spalling and bulging deformation, as well as the overall hazard level of the coal seam working face, are output for on-site safety management and operation decision support.
2. The real-time detection method for coal seam structure based on multimodal data according to claim 1, characterized in that, The specific steps of constructing the coal seam multimodal surface sensing dataset in step S1 include: S11, Multimodal Data Acquisition: Image acquisition devices and laser scanning devices are deployed in the coal seam operation area to be monitored. The image acquisition devices are used to collect image data of the exposed surface of the coal seam, and the laser scanning devices are used to collect three-dimensional point cloud data of the coal seam surface in the corresponding area; based on the acquisition time... As an index, the image acquisition device outputs image frames. The laser scanning device outputs point cloud frames. And record the timestamps of the acquired image data respectively. timestamps of point cloud data This yields the original multimodal input data stream: image sequence. With point cloud sequence ; S12. Time synchronization and frame-level pairing: The image sequence obtained in step S11 Point cloud sequence and its timestamp , As input, a synchronous triggering strategy is used for time alignment: for each image frame During the preset time interval Select the corresponding point cloud frame inside. As point cloud matching frames To satisfy This results in frame-level pairing; the output is a time-synchronized multimodal data frame. And assign a uniform synchronization timestamp to each frame pair. ; S13. Spatial calibration and unified coordinate system registration: The multimodal data frame output in step S12 As input, the intrinsic parameters of the image acquisition device are used. and the external parameter transformation relationship between the two devices. Spatial calibration is performed on the image acquisition device and the laser scanning device, wherein... This involves a rigid body transformation from the laser coordinate system to the camera coordinate system; the calibration parameters are used for each multimodal data frame. Spatial registration: Specifically, matching point clouds with frames Transform to a unified coordinate system to obtain the registration point cloud frame and establish image frames. With registration point cloud frames The spatial correspondence is output as a unified spatial data pair for spatial registration. ; S14. Multimodal input sample organization for real-time monitoring: The multimodal data output in step S13 is used to... As basic input samples, for input samples The hazard characteristics of the coal seams present in the sample are calibrated, and the calibration results include the location information of the hazard characteristics detected in the input sample. and tag information Specifically, location information Therefore, the dangerous characteristics of the coal seam are described in the actual coal seam mine coordinate system. Three-dimensional coordinates: The mine coordinate system is consistent with the actual coordinate system used in the mine during data acquisition to ensure the correspondence between location information and the actual environment; tag information. The smallest bounding rectangle that includes the area where the coal seam’s hazardous features are located, along with the corresponding type label. The type label specifically includes four categories: cracks, cavities, spalling, and bulging deformation. S15. Creation of a multimodal surface sensing dataset for coal seams: Based on the data processing procedures described in steps S12 to S14, the image sequence is processed. With point cloud sequence Temporal alignment, spatial alignment, and image calibration are performed sequentially to obtain the complete input dataset. and the complete output dataset The input and output datasets are then combined to obtain the coal seam multimodal surface sensing dataset. .
3. The real-time detection method for coal seam structure based on multimodal data according to claim 1, characterized in that, Step S2, based on the coal seam hazard feature perception model with multiple attention layers, specifically includes the following: S21. Design of Coal Seam Hazard Characteristic Structural Unit Sensing Module: Input data Image data in The data is fed into a pre-trained ResNet model to perform preliminary feature extraction and obtain the features. Then the features The data is fed into five cascaded structural refinement sensing units for further feature calculation and processing to obtain the features. The features are then processed through three sequentially connected fully connected layers. Specifically, each structurally refined perceptual unit includes a convolutional layer, a pooling layer, and a fully connected layer connected in sequence. Input data Point cloud data First, the data is fed into four sequentially connected fully connected layers for preliminary feature processing to obtain the features. Then the features The data is fed into three sequentially connected adaptive convolutional layers for further processing to obtain the features. The features are then obtained through two consecutively connected average pooling layers. ; Features obtained based on the above process and characteristics This invention constructs Each structural hierarchical unit completes the feature and Fusion processing and analysis were performed, and a system including... An expert network of hierarchical units with parallel structures, and The final output feature of each hierarchical structural unit is That is, the first The output features corresponding to each structural hierarchical unit are ; S22. Design of the coal seam structure unit feature enhancement processing module: Based on the results obtained in step S21 Features This invention designs a coal seam structural unit feature enhancement processing module for... Further comprehensive analysis is performed on these features; in the coal seam structure unit feature deepening processing module, the features are first analyzed through two sequentially connected fully connected layers. The features are obtained by integrating the features. Then the features Features are obtained by feeding the data into two convolutional layers with kernel size of 3*3. Next, the features The features are obtained by feeding the data into two convolutional layers with a kernel size of 5*5. Then the features Features are obtained by feeding the data into two convolutional layers with kernel size of 7*7. Finally, the features Features are obtained by calculating the weights of the features using the Sigmoid activation function. This strengthens the characteristics that are highly correlated with the dangerous features of coal seams; S23. Design of Coal Seam Hazard Characteristic Output Module: Based on the features obtained during the analysis process in step S22 , , , This invention designs a coal seam hazard feature output module based on a multi-attention fusion and multi-convolution mechanism to obtain the final prediction result of the model; specifically, the features Three sequentially connected adaptive convolutional layers are used to further extract coal seam hazard features. Then, a ReLU activation function is used to achieve non-linear activation, followed by an attention layer and feature extraction. Features are obtained through fusion processing To take into account the multi-scale distribution information of coal seam hazards, and then the features The processed features are fed into three sequentially connected adaptive convolutional layers for further feature extraction, and then passed through an attention layer and feature decomposition layer. Further fusion processing is performed to obtain features. Then the features The data is then fed into three sequentially connected adaptive convolutional layers for further processing, and the processed features are then passed through an attention layer and a feature decomposition layer. The final fusion process is performed to obtain the features. , The detection results of coal seam hazard characteristics were obtained after processing with three adaptive convolutional layers and a maximum pooling layer. Test results Including hazard label information and location information ; S24. Model Training: After the above model structure design is completed, the model is trained using the dataset constructed in step S1, employing the Adam optimizer and stochastic gradient descent algorithm to obtain the trained coal seam hazard characteristic perception model.
4. The real-time detection method for coal seam structure based on multimodal data according to claim 3, characterized in that, The method for constructing the structural layered unit in step S21 is as follows: In the constructed hierarchical structural units, features are processed through image branches and point cloud branches respectively. and characteristics Process them separately; in the image branch, features Low-level feature extraction is performed through two sequentially connected convolutional layers, followed by an adaptive convolutional layer to enhance the representation of local features and obtain the final feature. Then, a self-attention mechanism is used to focus on the features. Weighted summaries are applied to focus on key regions in the image, and finally, a fully connected layer is used to extract global information, resulting in a more abstract representation of image features. In point cloud branches, point cloud features First, a fully connected layer performs initial processing to gradually reduce the feature dimension. Then, an adaptive convolutional layer is used to enhance the local detail features of the point cloud. Next, a spatial pyramid pooling layer is used for multi-scale pooling to capture information at different scales. Finally, an average pooling layer reduces the spatial dimension of the features to obtain a unified feature representation. ; Features were obtained after processing via a dual-branch path. and characteristics Subsequently, a multi-head self-attention mechanism was designed to focus on features. and characteristics Cross-modal fusion is performed, followed by global pooling to fuse global information, resulting in the fused feature representation. Finally, the features The output features are obtained by feeding the data into two interconnected fully connected layers and a Softmax activation function for final processing. .
5. The real-time detection method for coal seam structure based on multimodal data according to claim 1, characterized in that, The construction of the coal seam hazard characterization map in step S3 includes: (1) Node construction: Based on the coal seam hazard characteristic detection results in step S2. Each coal seam hazard feature is treated as a node in the graph, and its coal seam hazard feature label information is assigned. and location information As this node Features; (2) Edge construction: Based on the location information obtained from the detection Calculate the Euclidean distance between the nodes respectively. As the distance edge feature between nodes, and based on the detected label information Calculate the cosine distance between the nodes respectively. As a feature of dangerous similar edges between nodes; (3) Based on the above node and edge construction method, the nodes and edges of the coal seam hazard characterization map are constructed sequentially. After the construction is completed, the coal seam hazard characterization map is obtained. .
6. The real-time detection method for coal seam structure based on multimodal data according to claim 1, characterized in that, In step S3, based on the Laplace operator, an adjacency matrix of the coal seam is constructed and a weighted Laplace matrix is calculated. The hazard mode of the coal seam region is analyzed using the matrix eigenvalues, including: (1) Based on the obtained coal seam hazard characterization map Construct the adjacency matrix of the coal seam Adjacency matrix Each element in Represents a node and nodes The adjacency distance between them, where For the image The total number of nodes in the system; (2) Based on adjacency matrix Construct a diagonal matrix To describe the strength of the connection between each node and other nodes. ; (3) Calculate the hazard characterization diagram of this coal seam. Weighted Laplace matrix To measure the relationships between nodes and the correlation between coal seam regions, ; (4) Subsequently, based on the weighted Laplace matrix Further calculation of all eigenvalues of this matrix is needed, since low eigenvalues correspond to the main structure of the graph. The smallest eigenvalues reflect the main hazardous characteristics of the coal seam region, and these eigenvectors reflect the global hazardous pattern of the coal seam. Therefore, this invention selects the first few eigenvalues from all eigenvalues. The eigenvectors corresponding to the smallest eigenvalues This serves as an indicator for assessing the hazard level of this coal seam, and further calculations are performed separately. vectors Magnitude weighted average and To comprehensively consider the contributions of different feature vectors, and to... As a Laplace quantitative evaluation index for coal seams.
7. The real-time detection method for coal seam structure based on multimodal data according to claim 1, characterized in that, In step S3, an improved graph clustering algorithm is used to identify high-risk areas of the coal seam, and a depth-first search algorithm is used to analyze cut points and assess the propagation risk of the coal seam, including: (1) Based on the obtained coal seam hazard characterization map Define the scope of the domain Then calculate the graph in sequence. All nodes Euclidean distance to other nodes in the graph ,in This represents the total number of coal seam hazard characteristics; subsequently, based on the calculated... and scope of fields If node To the node Euclidean distance Less than Then this node As a node The neighbor nodes, and in turn calculate the distances from each of the remaining nodes in the graph to the node. The Euclidean distance is calculated and compared sequentially with the neighborhood range to obtain the node. The number of neighboring nodes ; (2) Calculate the figures sequentially according to the calculation method in process (1) above. The number of neighboring nodes of all nodes After the calculation is completed, the nodes with the most neighboring nodes in the top 50% are retained, and the rest are discarded. (3) Repeat the above (2) process three times, and reconnect the remaining nodes after the last iteration to obtain a new enhanced characterization map of coal seam hazard conditions. ; (4) The depth-first search algorithm is used to further analyze the cut points of the enhanced characterization map of coal seam hazards in order to find all the cut points in the map and calculate the total number of cut points in the map. The cut point is the graph Removing a node from the middle will cause the graph to be divided into multiple disconnected nodes. Therefore, the number of cut vertices can reflect the fragility of the graph structure. (5) The enhanced characterization map of coal seam hazard conditions is calculated sequentially using the Dijkstra algorithm. Propagation risk coefficient of all nodes And the graph is obtained by weighted average. The sum of the transmission risk coefficients ; (6) The total number of cut points obtained from (4) above The sum of the transmission risk coefficients obtained in (5) The graph clustering hazard factor of this coal seam hazard characterization map was calculated. .
8. The real-time detection method for coal seam structure based on multimodal data according to claim 1, characterized in that, In step S3, the coal seam hazard score is calculated by combining the Laplace quantitative assessment index and the graph clustering hazard factor of the coal seam hazard characterization map, and the hazard level of the coal seam is determined according to a set threshold, including: Based on the calculated Laplace quantitative evaluation index Clustering risk factors The hazard score of this coal seam was obtained by simultaneous calculation. ,in and This serves as the weighting constant for the hazard score; and based on this hazard score, the coal seam hazard level is determined, with a threshold value set. When the coal seam hazard rating Less than or equal to the threshold When the coal seam is in a normal state, it is determined that the coal seam is in a normal state. Greater than and less than or equal to At that time, the coal seam was determined to be in a low-risk state; when Greater than and less than or equal to At that time, the coal seam was determined to be in a medium-risk state; when Greater than At that time, the coal seam was determined to be in a high-risk state.
9. The real-time detection method for coal seam structure based on multimodal data according to claim 1, characterized in that, The step S4, model deployment and real-time monitoring application, includes: S41. Model Deployment: Deploy the coal seam hazard characteristic perception model constructed in step S2, the coal seam hazard situation characterization map constructed in step S3, and the hazard analysis algorithm design to the local computer in the mine. S42. Collecting data to be detected inside the mine: Deploying image acquisition devices at the coal seam working face to collect real-time image data of the coal seam. Deploy laser scanning devices to collect real-time point cloud data of coal seams ; S43. Risk Event Identification and Comprehensive Risk Assessment: The image data collected in step S42... and point cloud data The data is input into the coal seam hazard feature perception model deployed on the local computer, and the coal seam hazard feature detection results are obtained. Based on this, the test results will be simultaneously transmitted. The data is fed into the designed hazard analysis algorithm to comprehensively determine the real-time operating status of the coal seam, ultimately yielding the coal seam risk assessment level. ; S44. Risk Information Feedback: Feedback on coal seam hazard characteristics detected by the model. And the coal seam risk assessment level determined in real time by the algorithm. Promptly report to the safety management personnel at the coal seam working face.