Coal and gas outburst early warning method and system based on multi-modal feature fusion

Abstract

The invention relates to the technical field of outburst early warning, and provides a coal and gas outburst early warning method and system based on multi-modal fusion. The method comprises the following steps of: processing obtained multi-modal data of specified yield at a specified monitoring point of a driving working face during normal mining and when an outburst omen appears, so as to obtain different time scale characteristics and different space scale characteristics of the multi-modal data; and constructing a bidirectional cross attention mechanism based on different time scale features and different spatial scale features, fusing the time scale features and the spatial scale features to form fused features, carrying out fusion splicing on the fused features to obtain fused splicing features, and carrying out early warning on coal and gas outburst.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of outburst early warning, and in particular relates to a coal and gas outburst early warning method and system based on multi-modal feature fusion. BACKGROUND

[0002] Coal and gas outburst (referred to as "outburst" for short) is a dynamic phenomenon in which coal bodies rush to the working space under the joint action of rapid desorption of gas due to local stress concentration leading to coal body breakage and instability during underground mining of coal mines, and mainly occurs at the heading face, which can cause suffocation of personnel, burying of people in flowing coal, and destruction of shaft and roadway facilities, thereby bringing great threat to underground operating personnel.

[0003] A perfect monitoring and early warning system is one of the fundamental guarantees for outburst prevention and control. In view of the existence of different degrees of audible and inaudible precursors in the outburst process, such as coal shooting, abnormal structure sound, gas fluctuation, coal wall sweating, air odor, layering disorder, top drilling, sticking, and coal wall bulging, these precursors involve multi-modal data such as sound, video, image, and odor, and using multi-modal data for outburst early warning is an effective method to improve the accuracy of early warning.

[0004] However, the multi-modal data composed of parameters from different monitoring systems have significant differences in type, format, and features, and the correlation between parameters is weak, and generally presents nonlinear dynamic changes, which makes it difficult to effectively capture the internal spatiotemporal regularity and limits the availability and accuracy of outburst early warning.

[0005] Therefore, there is an urgent need to provide a technical solution to the above-mentioned deficiencies of the prior art. SUMMARY

[0006] The purpose of the present application is to provide a coal and gas outburst early warning method and system based on multi-modal feature fusion to solve or alleviate the problems existing in the prior art.

[0007] In order to achieve the above-mentioned purpose, the present application provides the following technical solution: The present application provides a coal and gas outburst early warning method based on multi-modal feature fusion, comprising: acquiring features of multi-modal data of specified parameters of a specified monitoring point of a heading face at different time scales and features of multi-modal data of specified parameters of the specified monitoring point at different spatial scales during normal mining and when an outburst precursor appears and different spatial scales ; wherein, is the specified monitoring point, is a time series, is the type of modal data; based on the features at different time scales and the features at different spatial scales and different spatial scales A bidirectional cross-attention mechanism is constructed to fuse the features and the features to form fused features and the fused features , and the fused features and the fused features are fused and spliced to obtain fused spliced features , so as to perform early warning on coal and gas outburst.

[0008] Preferably, the obtained multi-modal data is subjected to spatiotemporal synchronization processing to obtain original data ; In response to one and only one modal abnormal fluctuation of the same specified monitoring point and the same time period, the original data is subjected to multi-modal collaborative denoising to obtain modal data ; The modal data is subjected to normalization processing, and features of different time scales of the modal data obtained through normalization processing are extracted through feature extraction ; wherein the features of different time scales at least include trend features, mutation features, fluctuation features and frequency domain features of different time scales; The modal data is subjected to spatial feature extraction based on a graph convolutional neural network to generate features of different spatial scales ; wherein, is an embedded feature representing local spatial correlation.

[0009] Preferably, according to the formula: The original data is subjected to multi-modal collaborative denoising to obtain modal data ; In the formula, represents the number of decomposition layers during wavelet decomposition, and is at least 1, is the maximum number of decomposition layers during wavelet decomposition, is the wavelet decomposition of the original data ; is a low-frequency approximation coefficient of the decomposition layer of the wavelet decomposition process, is a high-frequency detail coefficient of the decomposition layer of the wavelet decomposition process, is a threshold value of the decomposition layer , is a soft threshold value, The high-frequency detail coefficient after soft threshold processing.

[0010] Preferably, according to the formula: The modal data is normalized to obtain the modal data ; in the formula, is the average value of the multi-modal collaborative denoising modal data . is the standard deviation of the multi-modal collaborative denoising modal data .

[0011] Preferably, according to the formula: The trend features of different time scales of the modal data are extracted; in the formula, respectively, the maximum value , the minimum value , the mean value and the variance of the modal data at the time scale .

[0012] Preferably, according to the formula: The mutation features of different time scales of the modal data are extracted; in the formula, respectively, the jump amplitude , the first-order difference , the cumulative increment , and the continuous growth duration of the modal data at the time scale . is an indicator function, indicating the cumulative duration of the continuous growth period; is the time difference from time to time .

[0013] Preferably, according to the formula: The fluctuation features of different time scales of the modal data are extracted; in the formula, respectively, the range of the modal data at the time scale Standard deviation , mean square deviation Coefficient of variation .

[0014] Preferably, according to the formula: Extract modal data Frequency domain characteristics at different time scales; In the formula, They are time scales Lower modal data main frequency Energy entropy transient rate Energy ratio .

[0015] Preferably, the fused and stitched features are obtained through a multilayer perceptron. Mapped to risk value Furthermore, it provides early warnings for coal and gas outbursts based on preset early warning rules.

[0016] This embodiment also provides a coal and gas outburst early warning system based on multimodal feature fusion, which uses any of the above-described coal and gas outburst early warning methods based on multimodal feature fusion to provide early warning of coal and gas outbursts. The system includes: The feature processing unit is configured to acquire the features of multimodal data with specified parameters at designated monitoring points on the tunneling face during normal mining and when outburst signs appear, at different time scales. and characteristics at different spatial scales ;in, For the first A designated monitoring point, It is a time series. The type of modal data; The integrated early warning unit is configured based on features at different time scales. and characteristics at different spatial scales Construct a bidirectional cross-attention mechanism for features and characteristics To achieve fusion and form fusion characteristics and fusion features And will integrate features and fusion features By performing fusion and splicing, the fusion and splicing features are obtained. This is to provide early warning of coal and gas outbursts.

[0017] Beneficial effects: The method and system for coal and gas outburst early warning based on multi-modal fusion provided by the embodiments of the present application process the multi-modal data of the specified monitoring point of the tunneling working face obtained at the specified yield in normal mining and when an outburst precursor appears, to obtain the features of different time scales and the features of different spatial scales of the multi-modal data and different spatial scales Then, the features of different time scales and the features of different spatial scales are constructed, the features and the features are fused to form fused features and fused features , and the fused features and the fused features are fused and spliced to obtain fused and spliced features , and the coal and gas outburst is early warned.

[0018] Therefore, the problems of difficult multi-modal data fusion, difficulty in effectively capturing intrinsic features, and difficulty in significantly improving the accuracy of outburst early warning are effectively overcome in the existing coal and gas outburst early warning. The explicit spatiotemporal features of different time scales (instantaneous, short-term, and long-term) and spatial scale feature dimensions are mined from the multi-modal data collected by the sensor, and the implicit spatiotemporal features are mined by using the graph convolutional neural network. Finally, the multi-modal spatiotemporal data of cross-modal and cross-scale mined is deeply fused based on the cross-attention fusion algorithm, the perception ability of weak and nonlinear precursor information in the outburst process is enhanced, the coal and gas outburst risk is comprehensively perceived, the outburst risk discrimination model based on the multi-layer perception machine is constructed, the coal and gas outburst risk fusion early warning is realized, and the usability and accuracy of the coal and gas outburst early warning are improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, the illustrative embodiments of the present application and the explanations thereof serve to explain the present application, and do not constitute an improper limitation on the present application. Among them: Figure 1 a flowchart of a coal and gas outburst early warning method based on multi-modal fusion provided according to some embodiments of the present application; Figure 2 a monitoring arrangement schematic diagram of a multi-modal sensor in a roadway and a coal body provided according to some embodiments of the present application; Figure 3 a multi-modal data fusion schematic diagram provided according to some embodiments of the present application; Figure 4 a structure schematic diagram of a coal and gas outburst early warning system based on multi-modal fusion provided according to some embodiments of the present application. Detailed Implementation

[0020] The present application will now be described in detail with reference to the accompanying drawings and embodiments. Various examples are provided by way of explanation and not by way of limitation. In fact, those skilled in the art will understand that modifications and variations can be made to the present application without departing from the scope or spirit of the present application. For example, a feature shown or described as part of one embodiment may be used in another embodiment to produce yet another embodiment. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0021] In coal mine safety production, a sound monitoring and early warning system is one of the fundamental guarantees for the prevention and control of coal and gas outbursts. Outbursts are accompanied by varying degrees of audible and silent warning signs, such as: coal bursts, abnormal noises from structures, fluctuating gas levels, coal wall condensation, unusual odors in the air, disordered stratification, drill bit jacking, drill bit jamming, and coal wall bulging. Therefore, comprehensive monitoring of parameters such as gas, stress, temperature, microseismic activity (acoustic emission), electromagnetic radiation, and odor is crucial. However, the multimodal data generated by these parameters from different monitoring systems exhibit significant differences in type, format, and characteristics. Furthermore, the weak correlation between these parameters and their generally nonlinear dynamic changes make it difficult to effectively capture their inherent spatiotemporal patterns, thus limiting the usability and accuracy of outburst early warnings. Therefore, in-depth exploration of the spatiotemporal characteristics of multimodal data and the development and construction of coal and gas outburst early warning methods and systems based on multimodal data fusion are of great theoretical significance and urgent engineering need for overcoming existing early warning bottlenecks, achieving precise disaster prevention and control, and ensuring safe coal mine production.

[0022] Based on this, this embodiment provides a coal and gas outburst early warning method based on multimodal data fusion, overcoming the problems of difficulty in multimodal data fusion, difficulty in effectively capturing intrinsic features, and difficulty in significantly improving the accuracy of outburst early warning in existing coal and gas outburst early warning methods. Figures 1 to 3 As shown, this coal and gas outburst early warning method based on multimodal feature fusion includes: Step S101: Obtain multimodal data of specified parameters at designated monitoring points on the tunneling face during normal mining and when outburst signs appear, at different time scales. and characteristics at different spatial scales .

[0023] In this embodiment, a gas sensor, an infrared temperature sensor, a fiber optic stress sensor, a fiber optic microseismic sensor, an electromagnetic radiation sensor, and an olfactory sensor are deployed at a designated monitoring point of the tunneling working face to obtain multi-modal data such as gas emission, stress, temperature, microseismic, electromagnetic radiation, and odor molecules of the specified parameters during normal mining and when a premonitory sign of outburst occurs.

[0024] Then, feature mining is performed on the multi-source heterogeneous data stream collected by the sensor in different time scales (instantaneous, short-term, and long-term) and spatial scale feature dimensions. Specifically, spatiotemporal synchronization processing is performed on the obtained multi-modal data to obtain original data . Wherein, is the th designated monitoring point, that is, the spatial position coordinate; is the time series, is the type of modal data. Here, it is defined that corresponds to the data being gas emission; corresponds to the data being stress; corresponds to the data being temperature; corresponds to the data being microseismic; corresponds to the data being electromagnetic radiation; corresponds to the data being odor molecules.

[0025] If there is only one modal abnormal fluctuation at the same designated monitoring point and the same time period, it is considered to be noise, and multi-modal collaborative noise reduction is performed on the original data to obtain modal data , removing environmental interference. Specifically, according to the formula: Multi-modal collaborative noise reduction is performed on the original data to obtain modal data . In the formula, represents the decomposition level during wavelet decomposition, which is at least 1, is the maximum decomposition level during wavelet decomposition, is the wavelet decomposition of the original data ; is the low-frequency approximation coefficient of the decomposition level of the wavelet decomposition process, is the high-frequency detail coefficient of the decomposition level of the wavelet decomposition process, is the threshold value of the decomposition level , is the soft threshold value, is the high-frequency detail coefficient after soft threshold processing, indicates the low-frequency approximation coefficient and high frequency detail coefficients Wavelet reconstruction is performed. In this way, low frequency approximation coefficients and high frequency detail coefficients are obtained through wavelet decomposition, and the high frequency detail coefficients are filtered through threshold processing. It should be noted that the low frequency approximation coefficients are defined as a sequence / array obtained through wavelet decomposition that is less than or equal to the cutoff frequency of the filter, and the high frequency detail coefficients are defined as a sequence / array obtained through wavelet decomposition that is greater than the cutoff frequency of the filter.

[0026] Then, the multi-modal collaborative denoised modal data is normalized, specifically according to the formula: The modal data is normalized to eliminate the dimensional effect, and the obtained modal data ; in the formula, is the average value of the multi-modal collaborative denoised modal data , and is the standard deviation of the multi-modal collaborative denoised modal data .

[0027] In this embodiment, the time scale is divided into the short-term ( seconds), short-term ( minutes), medium-term ( hours), and long-term ( days), and the modal data is normalized to obtain the modal data Different time scale features are extracted. The features include at least trend features (maximum value ( ), minimum value ( ), mean value ( ), variance ( )), mutation features (jump amplitude ( ), first-order difference ( ), cumulative increment ( ), continuous growth time ( )), fluctuation features (range ( ), standard deviation ( ), mean square error ( ), coefficient of variation ( ), and frequency domain features (dominant frequency ( ), energy entropy ( ), transient rate ( ), energy ratio ( ).

[0028] In a specific example, according to the formula: Extract modal data The trend characteristics at different time scales; where, Separate time scales Lower modal data The maximum value ( ), minimum value ( ), mean ( ) and variance ( ).

[0029] According to the formula: Extract modal data The abrupt change characteristics at different time scales; where, They are time scales Lower modal data The amplitude of the jump ( ), first-order difference ( ), cumulative increment ( ), duration of sustained growth ( ); This is an indicator function representing the cumulative duration of a continuously increasing time period; For time Time The time difference.

[0030] According to the formula: Extract modal data The fluctuation characteristics at different time scales; where, Representing time scales Lower modal data range Standard deviation , mean square deviation Coefficient of variation .

[0031] According to the formula: Extract modal data Frequency domain characteristics at different time scales; where, They are time scales Lower modal data main frequency Energy entropy transient rate Energy ratio ; representing modal data performing fast Fourier transform.

[0032] Meanwhile, based on the graph convolutional neural network, spatial feature extraction is performed on the modal data to generate features of different spatial scales . Specifically, an asymmetric adjacency graph is constructed based on the spatial positions of the sensors in the roadway and the coal body (designated monitoring points), a heterogeneous graph structure is established by fusing physical distance and modal coupling, and the graph convolutional neural network is used to perform spatial feature extraction on the modal data to generate embedded features representing local spatial correlation .

[0033] Therefore, by deploying gas sensors, infrared temperature sensors, optical fiber stress sensors, optical fiber microseismic sensors, electromagnetic radiation sensors, and olfactory sensors at the heading face, a multi-modal perception network covering gas emission, mining stress, temperature, acoustic emission / microseismic signal, electromagnetic radiation, and odor molecules is constructed. Then, the multi-modal data collected by the sensors are used to mine explicit spatiotemporal features in different time scales (instantaneous, short-term, and long-term) and spatial scales, and the implicit spatiotemporal features are mined using the graph convolutional neural network.

[0034] In step S102, based on the features of different time scales and the features of different spatial scales, a bidirectional cross-attention mechanism is constructed to fuse the features and the features to form fused features and fused features , and the fused features and the fused features are fused and spliced to obtain fused and spliced features for early warning of coal and gas outburst.

[0035] In this embodiment, based on the symmetric cross structure, the time features are taken as the query (Query), the spatial features are taken as the Key and Value, and the spatial features are taken as the Query and the time features are taken as the Key and Value in the reverse direction, a bidirectional cross-attention mechanism is constructed to fuse the features and the features to form fused features and fused features . Specifically, according to the formula: the features and the features are fused to form fused features and fused features wherein, is a cross-attention operation, represents a hidden layer feature, respectively represent a time feature , a spatial feature extracted from the query vector; respectively represent a spatial feature , a time feature extracted from the key vector; respectively represent a spatial feature , a time feature extracted from the value vector; is a dimension of the key vector, represents an activation function, respectively represent a weight matrix of the query, the key and the value.

[0036] Finally, a multi-layer perception is constructed to map the fused and concatenated features to a risk value . Specifically, according to the formula: determines the risk value of the coal and gas outburst risk ; wherein, respectively represent a weight of the first hidden layer and a weight of the second hidden layer in the multi-layer perception; respectively represent a first bias and a second bias in the multi-layer perception. , both represent an activation function.

[0037] Finally, according to the defined early warning rules, the coal and gas outburst risk is early warned. When the risk value , it is a safety level; when the risk value , it is a yellow early warning; when the risk value , it is an orange early warning; and when the risk value , it is a red early warning. Here, the risk early warning levels of the yellow early warning, the orange early warning and the red early warning are sequentially improved.

[0038] Therefore, a cross-attention fusion algorithm is used to deeply fuse multimodal and multiscale spatiotemporal data obtained from mining, enhancing the perception of weak and nonlinear precursory information during coal and gas outbursts. A multilayer perceptron-based outburst risk identification model is constructed to achieve fusion-based early warning of coal and gas outburst risks. Furthermore, by mapping the fused features to risk values, the risk values ​​are used to comprehensively and multidimensionally identify coal and gas outburst risks as safe, yellow, orange, or red, offering flexibility, reliability, and a high degree of intelligence. This approach can be widely applied in coal and gas outburst mines and is suitable for early warning of coal and gas outbursts, especially in tunneling faces.

[0039] This embodiment integrates multimodal data of different types, formats, and characteristics to sensitively capture the nonlinear dynamic changes in the coal and gas outburst process, enabling comprehensive perception of coal and gas outburst risks. It effectively overcomes the problems of difficulty in multimodal data integration, difficulty in effectively capturing intrinsic features, and difficulty in significantly improving the accuracy of outburst warnings in existing coal and gas outburst early warning systems.

[0040] By analyzing multimodal data collected from sensors, explicit spatiotemporal features at different time scales (instantaneous, short-term, and long-term) and spatial scales are mined. Simultaneously, implicit spatiotemporal features are mined using graph convolutional neural networks. Finally, a cross-attention fusion algorithm is used to deeply fuse the mined cross-modal and cross-scale multimodal spatiotemporal data, enhancing the perception of weak and nonlinear precursory information during coal and gas outbursts. This enables comprehensive perception of coal and gas outburst risks and the construction of an outburst risk identification model based on a multilayer perceptron, achieving fusion-based early warning for coal and gas outburst risks and improving the availability and accuracy of early warning systems.

[0041] like Figure 4 As shown, this embodiment also provides a coal and gas outburst early warning system based on multimodal feature fusion. The system employs any of the above embodiments' coal and gas outburst early warning methods based on multimodal feature fusion to provide early warning of coal and gas outbursts. The system includes: The feature processing unit is configured to acquire the features of multimodal data with specified parameters at designated monitoring points on the tunneling face during normal mining and when outburst signs appear, at different time scales. and characteristics at different spatial scales ;in, For the first A designated monitoring point, It is a time series. The type of modal data; The integrated early warning unit is configured based on features at different time scales. and characteristics at different spatial scales Construct a bidirectional cross-attention mechanism for features and features perform fusion to form a fusion feature and fusion features , and fuse the fusion features and fusion features perform fusion splicing to obtain a fusion splicing feature to early warn coal and gas outburst.

[0042] The coal and gas outburst early warning system based on multi-modal feature fusion of the embodiment can realize the steps and processes of the coal and gas outburst early warning method based on multi-modal feature fusion of any one of the above embodiments, and achieve the same technical effects, which will not be described here.

[0043] In the description of the present application, the terms “one embodiment”, “some embodiments”, “example”, “specific example”, or “some examples” mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0044] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for early warning of coal and gas outbursts based on multimodal feature fusion, characterized in that, include: Acquire multimodal data of specified parameters at designated monitoring points on the tunneling face during normal mining operations and when outburst warning signs appear, at different time scales. and characteristics at different spatial scales ;in, For the first A designated monitoring point, It is a time series. The type of modal data; Features based on different time scales and characteristics at different spatial scales Construct a bidirectional cross-attention mechanism for features and characteristics To achieve fusion and form fusion characteristics and fusion features And will integrate features and fusion features By performing fusion and splicing, the fusion and splicing features are obtained. This is to provide early warning of coal and gas outbursts.

2. The method according to claim 1, characterized in that, The acquired multimodal data is subjected to spatiotemporal synchronization processing to obtain the raw data. ; In response to a single modal anomalous fluctuation at the same specified monitoring point within the same time period, the raw data... Multimodal collaborative noise reduction is performed to obtain modal data. ; Modal data Normalization is performed, and the normalized modal data is obtained through feature extraction. Features of different time scales Among them, the characteristics of different time scales It should include at least trend characteristics, mutation characteristics, fluctuation characteristics, and frequency domain characteristics at different time scales; Modal data based on graph convolutional neural networks Spatial feature extraction is performed to generate features at different spatial scales. ;in, Embedded features are used to represent local spatial relationships.

3. The method according to claim 2, characterized in that, According to the formula: For raw data Multimodal collaborative noise reduction is performed to obtain modal data. ; In the formula, This indicates the number of decomposition levels in wavelet decomposition, with a minimum of 1. This represents the maximum number of decomposition levels during wavelet decomposition. For the raw data Perform wavelet decomposition; The number of decomposition levels in the wavelet decomposition process The low-frequency approximation coefficients, The number of decomposition levels in the wavelet decomposition process High-frequency detail coefficients, Number of decomposition layers The threshold, Soft threshold, These are the high-frequency detail coefficients after soft thresholding.

4. The method according to claim 2, characterized in that, According to the formula: Modal data Normalization is performed to obtain modal data In the formula, Modal data after multimodal collaborative noise reduction The average value, Modal data after multimodal collaborative noise reduction The standard deviation.

5. The method according to claim 2, characterized in that, According to the formula: Extract modal data Trend characteristics at different time scales; In the formula, Separate time scales Lower modal data maximum value Minimum value mean and variance .

6. The method according to claim 2, characterized in that, According to the formula: Extract modal data The characteristics of mutations at different time scales; In the formula, They are time scales Lower modal data jump amplitude First-order difference Cumulative increment Duration of continuous growth ; This is an indicator function representing the cumulative duration of a continuously increasing time period; For time Time The time difference.

7. The method according to claim 2, characterized in that, According to the formula: Extract modal data Fluctuation characteristics at different time scales; In the formula, Representing time scales Lower modal data range Standard deviation , mean square deviation Coefficient of variation .

8. The method according to claim 2, characterized in that, According to the formula: Extract modal data Frequency domain characteristics at different time scales; In the formula, They are time scales Lower modal data main frequency Energy entropy transient rate Energy ratio .

9. The method according to claim 1, characterized in that, Fusing and stitching features through a multilayer perceptron Mapped to risk value Furthermore, it provides early warnings for coal and gas outbursts based on preset early warning rules.

10. A coal and gas outburst early warning system based on multimodal feature fusion, characterized in that, The coal and gas outburst early warning method based on multimodal feature fusion as described in any one of claims 1-9 is used to provide early warning of coal and gas outbursts. The system includes: The feature processing unit is configured to acquire the features of multimodal data with specified parameters at designated monitoring points on the tunneling face during normal mining and when outburst signs appear, at different time scales. and characteristics at different spatial scales ;in, For the first A designated monitoring point, It is a time series. The type of modal data; The integrated early warning unit is configured based on features at different time scales. and characteristics at different spatial scales Construct a bidirectional cross-attention mechanism for features and characteristics To achieve fusion and form fusion characteristics and fusion features And will integrate features and fusion features By performing fusion and splicing, the fusion and splicing features are obtained. This is to provide early warning of coal and gas outbursts.

Images