Mine safety monitoring method based on image processing

Abstract

The application relates to a mine safety monitoring method based on image processing, which comprises the following steps: acquiring monitoring data of a target area of a mine; extracting multi-modal features of the monitoring data, performing feature splicing on the multi-modal features and spatial features to obtain a joint feature vector; performing frequency domain weighting processing on the joint feature vector to obtain a first feature vector, performing modal interaction on visible light features and infrared features in the joint feature vector to obtain a second feature vector, and performing cross fusion on the second feature vector and the first feature vector to obtain a multi-modal fusion feature vector; inputting the multi-modal fusion feature vector into a pre-trained target prediction model to obtain confidence scores of multiple safety state categories, and performing a preset level alarm according to a preset safety threshold and the confidence scores of the safety state categories, so that the target area of the mine can be comprehensively monitored, and strong hidden safety hazards can be discovered and warned in a timely manner.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mine safety monitoring, and particularly relates to a mine safety monitoring method based on image processing. BACKGROUND

[0002] With the development of science and technology and the application of artificial intelligence, the mining method of a mine is changed from shaft mining to intelligent mining, and the intelligent mining using intelligent technology and equipment significantly improves the efficiency of mine mining. However, it is well known that the environment of a mine is complex and changeable, and various accidents such as roof collapse, gas leakage and equipment failure may occur at any time during the mining process. In order to ensure safety production, it is necessary to monitor the mining of a mine and ensure that safety hazards are found in time.

[0003] The traditional mine safety monitoring generally adopts an artificial inspection method or a single sensor monitoring method, but these methods have monitoring loopholes, and not only have a limited monitoring coverage, but also cannot find strong hidden safety hazards in time.

[0004] Therefore, it is necessary to provide a new technical scheme to improve one or more problems in the above-mentioned scheme.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The purpose of the embodiments of the present application is to provide a mine safety monitoring method based on image processing, which comprehensively monitors the target area of a mine, finds strong hidden safety hazards in time and gives an early warning.

[0007] According to the embodiments of the present application, a mine safety monitoring method based on image processing is provided, comprising:

[0008] obtaining monitoring data of a target area of a mine, the monitoring data comprising visible light image data, infrared image data, radar point cloud data and millimeter wave radar data;

[0009] extracting visible light features of the visible light image data through a feature extraction encoder, extracting infrared features of the infrared image data through an infrared feature extraction network, extracting point cloud features of the radar point cloud data through three-dimensional sparse convolution, and extracting radar features of the millimeter wave radar data through constant false alarm detection, the visible light features, the infrared features, the point cloud features and the radar features constituting multi-modal features, and performing feature splicing on the multi-modal features and spatial features to obtain a joint feature vector;

[0010] The joint feature vector is subjected to frequency domain weighting processing to obtain a first feature vector, and the visible light feature and the infrared feature in the joint feature vector are extracted to interact in modalities to obtain a second feature vector, and the second feature vector and the first feature vector are cross-fused to obtain a multi-modal fusion feature vector aligned in frequency domain and space domain;

[0011] The multi-modal fusion feature vector is input into a pre-trained target prediction model to obtain confidence scores of a plurality of safety state categories, and a preset level alarm is performed according to a preset safety threshold and the confidence scores of the safety state categories;

[0012] The training of the target prediction model includes:

[0013] The training sample corresponding to the multi-modal feature and the multi-modal fusion feature vector is obtained, and the effective training sample is constructed by using the training sample;

[0014] The effective training sample is input into a pre-constructed student model and a teacher model, the student model is subjected to knowledge distillation training by using the teacher model, and the trained student model is used as the target prediction model.

[0015] In an embodiment of the present application, the visible light feature of the visible light image data is extracted by the feature extraction encoder, including:

[0016] The original high-frequency signal of the visible light image data is subjected to enhancement processing to obtain a high-frequency enhanced signal;

[0017] The visible light image data is reconstructed based on the high-frequency enhanced signal and the original low-frequency signal of the visible light image data to obtain the visible light image data after interference elimination;

[0018] The visible light image data after interference elimination is input into the feature extraction encoder to extract the visible light feature.

[0019] In an embodiment of the present application, the infrared image data includes long-wave infrared data and short-wave infrared data, and the infrared feature of the infrared image data is extracted by the infrared feature extraction network, including:

[0020] The long-wave infrared data and the short-wave infrared data are input into the infrared feature extraction network;

[0021] The infrared feature extraction network is used to extract the temperature rise feature of the target region from the long-wave infrared data, extract the contour feature of the monitored target in the target region from the short-wave infrared data, and perform weighted fusion processing on the temperature rise feature and the contour feature to obtain the infrared feature.

[0022] In an embodiment of the present application, the radar features of the millimeter wave radar data extracted through the constant false alarm rate detection include:

[0023] The millimeter wave radar data is subjected to constant false alarm rate detection to identify a moving target in the target area, and the millimeter wave radar data corresponding to the moving target is subjected to signal processing to extract the radar features.

[0024] In an embodiment of the present application, the frequency domain weighting processing of the joint feature vector to obtain a first feature vector includes:

[0025] The joint feature vector is subjected to residual convolution processing to obtain a preliminary feature vector, and the preliminary feature vector includes preliminarily interacted multi-modal features and spatial features;

[0026] The preliminarily interacted multi-modal features in the preliminary feature vector are subjected to frequency domain weighting processing to obtain multi-modal weighted features;

[0027] Based on a multi-head attention mechanism, the multi-modal weighted features are fused with the preliminarily interacted spatial features in the preliminary feature vector to obtain the first feature vector.

[0028] In an embodiment of the present application, the frequency domain weighting processing of the preliminarily interacted multi-modal features in the preliminary feature vector to obtain multi-modal weighted features includes:

[0029] The preliminarily interacted multi-modal features in the preliminary feature vector are extracted and subjected to frequency domain conversion processing to obtain frequency domain features corresponding to the multi-modal features;

[0030] The frequency domain features corresponding to the multi-modal features are subjected to normalization processing to determine attention weight parameters corresponding to each frequency band of the frequency domain features;

[0031] The frequency domain features are weighted using the attention weight parameters to obtain the multi-modal weighted features.

[0032] In an embodiment of the present application, the extraction of the visible light features and the infrared features in the joint feature vector for modal interaction to obtain a second feature vector includes:

[0033] The visible light features and the infrared features in the joint feature vector are extracted, and the visible light features and the infrared features are subjected to convolution processing respectively to obtain preliminary visible light features and preliminary infrared features;

[0034] The preliminary visible light features and the preliminary infrared features are fused by weighting to obtain interaction features;

[0035] The internal dependency of the interaction feature is calculated based on a self-attention mechanism to obtain an enhanced feature;

[0036] The enhanced feature is subjected to a nonlinear transformation to obtain the second feature vector.

[0037] In an embodiment of the present application, the plurality of safety state categories includes a first abnormal state category, a second abnormal state category, and a third abnormal state category.

[0038] The preset level alarm is executed according to the preset safety threshold and the confidence score of each safety state category, including:

[0039] If the confidence score of the first abnormal state category is greater than the preset safety threshold, a first preset level alarm is triggered.

[0040] If the confidence score of the second abnormal state category is greater than the preset safety threshold, a second preset level alarm is triggered.

[0041] If the confidence score of the third abnormal state category is greater than the preset safety threshold, a third preset level alarm is triggered.

[0042] In an embodiment of the present application, the training sample corresponding to the multi-modal feature and the multi-modal fusion feature vector is obtained, and the effective training sample is constructed by using the training sample, including:

[0043] The training sample of the multi-modal fusion feature vector is obtained, a causal diagram is constructed by taking the training sample of the multi-modal fusion feature vector as a variable, and a real abnormal feature is determined according to the causal diagram.

[0044] The training sample of the multi-modal feature is obtained, and the training sample of the multi-modal feature is matched with the real abnormal feature in similarity to screen the effective training sample.

[0045] In an embodiment of the present application, the student model is subjected to knowledge distillation training by using the teacher model, including:

[0046] The student feature output by the student model is mapped to the teacher feature dimension, and a feature distillation loss is calculated according to the teacher feature output by the teacher model and the student feature mapped to the teacher feature dimension.

[0047] A prediction distillation loss is calculated according to the prediction probability distribution output by the student model and the prediction probability distribution output by the teacher model.

[0048] The total distillation loss is calculated according to the feature distillation loss and the prediction distillation loss, and the student model is trained by minimizing the total distillation loss.

[0049] The technical solutions provided in the application can have the following beneficial effects:

[0050] In the embodiments of the application, the monitoring data fuses visible light image data, infrared image data, radar point cloud data and millimeter wave radar data, expands the monitoring coverage range, and realizes comprehensive monitoring of the target area of the mine; the multi-modal joint feature vector is obtained by feature extraction and splicing of the monitoring data, and the multi-modal joint feature vector contains multi-modal perception data that can be used to identify strong hidden safety hazards; further, the different modal features in the joint feature vector are processed by frequency domain weighting or modal interaction, and then cross fusion is performed, so that the different modal features are fully complementary, which enables the target prediction model to fully perceive complex safety hazards and strong hidden safety hazards from the multi-modal fusion feature vector when predicting the confidence score, and thus ensures the accuracy of the prediction and the timeliness of the early warning.

[0051] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application. It is apparent that the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.

[0053] Figure 1 A step flow chart of a mine safety monitoring method based on image processing in an exemplary embodiment of the application is shown;

[0054] Figure 2 A training step flow chart of a target prediction model in an exemplary embodiment of the application is shown;

[0055] Figure 3 A simplified schematic diagram of a frequency-space modal alignment network in an exemplary embodiment of the application is shown;

[0056] Figure 4 A simplified schematic diagram of another frequency-space modal alignment network in an exemplary embodiment of the application is shown. DETAILED DESCRIPTION

[0057] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations in any suitable manner.

[0058] Moreover, the drawings are not necessarily to scale. Like reference numerals can be used to denote like parts throughout the various illustrations. Some of the blocks in the drawings can be functional blocks that can employ software, hardware, firmware, or a combination of software, hardware, and / or firmware to achieve the specified functions. Some of these blocks can be implemented as software modules or code that operates in one or more processors or microcontrollers.

[0059] A mine safety monitoring method based on image processing is provided in the present example implementation, which can include the following steps, as shown in FIG. 1. Figure 1

[0060] Step S101: Obtain monitoring data of a target area of a mine, the monitoring data including visible light image data, infrared image data, radar point cloud data, and millimeter wave radar data.

[0061] The target area of the mine includes a roadway, a working face, and a transportation channel. The visible light image data, the infrared image data, the radar point cloud data, and the millimeter wave radar data are all marked with a timestamp.

[0062] Step S102: Extract visible light features of the visible light image data through a feature extraction encoder, extract infrared features of the infrared image data through an infrared feature extraction network, extract point cloud features of the radar point cloud data through three-dimensional sparse convolution, and extract radar features of the millimeter wave radar data through constant false alarm detection. The visible light features, the infrared features, the point cloud features, and the radar features constitute multi-modal features. The multi-modal features are spliced with spatial features to obtain a joint feature vector.

[0063] Step S103: Perform frequency domain weighting processing on the joint feature vector to obtain a first feature vector, and extract the visible light features and the infrared features in the joint feature vector to perform modal interaction to obtain a second feature vector. The second feature vector is cross-fused with the first feature vector to obtain a multi-modal fusion feature vector that is aligned in the frequency domain and the spatial domain.

[0064] ​Step S104: input the multi-modal fusion feature vector into the pre-trained target prediction model to obtain confidence scores of a plurality of safety state categories, and perform a preset level alarm according to a preset safety threshold and the confidence scores of the safety state categories.

[0065] Further, referring to FIG. 1, the training of the target prediction model can include the following steps. Figure 2

[0066] Step S201: obtain training samples corresponding to the multi-modal features and the multi-modal fusion feature vector, and construct effective training samples by using the training samples.

[0067] Step S202: input the effective training samples into the pre-constructed student model and teacher model, perform knowledge distillation training on the student model by using the teacher model, and take the trained student model as the target prediction model.

[0068] In the embodiments of the present application, the monitoring data fuses visible light image data, infrared image data, radar point cloud data and millimeter wave radar data, expands the monitoring coverage, and realizes comprehensive monitoring of the target area of the mine; the multi-modal joint feature vector is obtained by feature extraction and splicing of the monitoring data, and the multi-modal joint feature vector contains multi-modal perception data capable of identifying strong hidden safety hazards; further, the different modal features in the joint feature vector are processed by frequency domain weighting or modal interaction, and then cross fusion is performed, so that the different modal features are fully complementary, which enables the target prediction model to fully perceive complex safety hazards and strong hidden safety hazards from the multi-modal fusion feature vector when predicting the confidence score, and further ensures the accuracy of the prediction and the timeliness of the warning.

[0069] Next, each step of the above method in the present example embodiment will be described in more detail.

[0070] In one embodiment, after obtaining the monitoring data of the target area of the mine in step S101, the following step S101L is further included.

[0071] Step S101L: performing anomaly labeling on the monitoring data to obtain labeled monitoring data. It should be noted that the labeled monitoring data includes labeled training samples and labeled test samples.

[0072] Specifically, in step S101L, the open source data labeling tool is used to perform anomaly labeling on the monitoring data, and all samples in the monitoring data are labeled with real class labels, which define 10 safety state categories that can be recognized by the present method, including: normal working condition, roof collapse, roof deformation, crack, gas leakage, gas accumulation, abnormal temperature rise of gas, equipment overheating, water seepage and illegal intrusion.​

[0073] In an embodiment, the labeled training samples are subjected to data augmentation processing. For example, the labeled visible light image data is subjected to image enhancement using a low-light image enhancement network. For example, the labeled infrared image data is subjected to Gaussian blur processing and fogging processing. For example, the labeled radar point cloud data is subjected to random down-sampling processing and rotation and translation processing. For example, perturbations are added to the Doppler dimension of the millimeter wave radar data to achieve data augmentation processing of the labeled millimeter wave radar data. Of course, the manner in which the labeled training samples are subjected to data augmentation processing is not limited to the above-described manner. It should be noted that the labeled and data-augmented training samples are used for training of the target prediction model.

[0074] In an embodiment, the visible light features of the visible light image data are extracted by a feature extraction encoder, including the following steps S102A1 to S102A3.

[0075] Step S102A1: The original high-frequency signal of the visible light image data is subjected to enhancement processing to obtain a high-frequency enhanced signal.

[0076] Step S102A2: The visible light image data is reconstructed based on the high-frequency enhanced signal and the original low-frequency signal of the visible light image data to obtain de-interfered visible light image data.

[0077] Step S102A3: The de-interfered visible light image data is input to the feature extraction encoder to extract the visible light features.

[0078] It should be noted that by steps S102A1 to S102A3, the visible light image data is subjected to high-frequency enhancement and reconstruction, which can suppress environmental interference such as dust and strengthen the detail expression capability in the visible light image. The visible light features extracted therefrom can accurately reflect the subtle textures and color changes of the roof deformation, cracks, and water seepage in the target region.

[0079] Specifically, in step S102A1, the original high-frequency signal of the visible light image data is subjected to enhancement processing, including the following steps:

[0080] The visible light image data is subjected to Fourier frequency domain decomposition to obtain the original high-frequency signal and the original low-frequency signal of the visible light image data, wherein the original high-frequency signal represents details or noise in the visible light image data; the original high-frequency signal is subjected to enhancement processing to suppress high-frequency noise caused by dust, and the high-frequency enhanced signal obtained by the enhancement processing.

[0081] Specifically, in step S102A2, the high-frequency enhanced signal obtained in step S102A1 is componentally superimposed with the original low-frequency signal in the frequency domain to obtain a frequency domain reconstruction signal; the frequency domain reconstruction signal is subjected to two-dimensional inverse Fourier transform to obtain the visible light image data after interference removal.

[0082] Specifically, in step S102A3, the visible light image data after interference removal is subjected to feature extraction by using the trained feature extraction encoder. The trained feature extraction encoder includes a multi-convolution layer group and a global average pooling layer, the multi-convolution layer group is used to perform multi-layer convolution processing on the visible light image data after interference removal to obtain a high-level semantic feature map, and the global average pooling layer is used to perform spatial dimension compression processing on the high-level semantic feature map to obtain visible light features.

[0083] It should be noted that the training of the feature extraction encoder in step S102A3 includes:

[0084] The training sample based on the visible light image data is used to construct positive sample pairs and negative sample pairs. For example, for each original training sample, two different data augmentation transformations are applied to generate corresponding two enhanced views. For each original training sample, its two enhanced views constitute a positive sample pair; and for all enhanced views obtained after data augmentation transformation of all training samples, any enhanced view and the remaining multiple enhanced views other than its positive sample constitute a negative sample pair.

[0085] The positive sample pairs and the negative sample pairs are input into the feature extraction encoder, and the feature extraction encoder is trained by using a contrastive loss function. The similarity of the positive sample pairs is maximized and the similarity of the negative sample pairs is minimized by using the contrastive loss function, so as to realize the training and optimization of the feature extraction encoder. The contrastive loss function is represented as:

[0086] (1)

[0087] wherein, represents the contrastive loss function, represents a feature vector obtained by processing the i th enhanced view by using the feature extraction encoder, i represents the index of the enhanced view, and 1≤i≤2N; represents a feature vector obtained by processing the positive sample of the i th enhanced view by using the feature extraction encoder, j represents the index of the enhanced view, and 1≤j≤2N, and constitute a positive sample pair; represents a feature vector obtained by processing the enhanced view other than u≠i by using the feature extraction encoder, u represents the index of the enhanced view, and 1≤u≤2N, when u≠i and u≠j, and constitute a negative sample pair; denotes a cosine similarity between and denotes a cosine similarity between and denotes a temperature parameter, denotes a number of training samples, denotes a number of enhanced views.

[0088] In one embodiment, the infrared image data includes long-wave infrared data and short-wave infrared data, and the infrared feature extraction network is used to extract infrared features of the infrared image data, including the following steps S102B1 and S102B2.

[0089] Step S102B1: input the long-wave infrared data and the short-wave infrared data into the infrared feature extraction network. The infrared feature extraction network adopts a dual-branch convolutional neural network structure, including a first convolutional neural network branch and a second convolutional neural network branch.

[0090] Step S102B2: use the first convolutional neural network branch of the infrared feature extraction network to extract the temperature rise features of the target region from the long-wave infrared data, and use the second convolutional neural network branch of the infrared feature extraction network to extract the contour features of the monitored target in the target region from the short-wave infrared data; perform weighted fusion processing on the extracted temperature rise features and contour features to obtain the infrared features.

[0091] In step S102B2, the monitored target can be understood as a device or a person in the target region, and the temperature rise features of the target region extracted from the long-wave infrared data include but are not limited to: gas temperature rise, device temperature rise.

[0092] It should be noted that the first convolutional neural network branch is used to extract the temperature rise features of the target region from the long-wave infrared data, including the following processing: performing multi-level convolution and pooling processing on the long-wave infrared data to obtain high-level temperature field features; performing spatial dimension compression processing on the high-level temperature field features to obtain a global feature vector; performing linear transformation on the global feature vector to obtain a predicted temperature distribution; performing difference operation on the predicted temperature distribution and a reference temperature of a preset normal working condition to obtain the temperature rise features.

[0093] ​​It should be noted that the second convolutional neural network branch is used to extract the contour features of the monitored target in the target region from the short-wave infrared data, including the following processing: performing multi-level convolution and pooling processing on the short-wave infrared data to obtain semantic-level contour features; performing multi-level upsampling and convolution processing on the semantic-level contour features to generate a high-resolution feature map; performing channel dimension reduction processing on the high-resolution feature map to obtain a single-channel feature; performing nonlinear mapping processing on the single-channel feature to output a contour confidence map; performing thresholding processing on the contour confidence map to obtain a binary contour map, and performing contour tracking on the binary contour map to identify a contour set; filtering a main contour representing the monitored target from the contour set according to the contour area and the contour closure degree; and performing feature quantization processing on the main contour to obtain contour features for subsequent weighted fusion.

[0094] In one embodiment, before extracting the temperature rise features of the target region from the long-wave infrared data using the first convolutional neural network branch, the following steps are further included:

[0095] The training sample of the long-wave infrared data is used as the input of the first convolutional neural network branch, and the first convolutional neural network branch is trained and optimized using a physical loss function.

[0096] It should be noted that, in order to ensure that the first convolutional neural network branch meets the physical law during the training and optimization process, the embodiment constructs a physical loss function according to the Fourier heat conduction equation.

[0097] The Fourier heat conduction equation is used to simulate the theoretical temperature rise in the gas accumulation scene, and the Fourier heat conduction equation is expressed as:

[0098] (2)

[0099] wherein, represents the position in the target region at time , the temperature distribution, represents the thermal diffusion coefficient, represents the Laplacian operator, which is used to describe the second-order derivative change of temperature in the spatial dimension; represents the heat source term, which reflects the heat source intensity of position at time .

[0100] The physical loss function constructed according to the Fourier heat conduction equation is:

[0101] (3)

[0102] wherein, represents the position predicted by the first convolutional neural network branch at time a temperature distribution of the current time, represents a physical loss function.

[0103] It needs to be explained that the long-wave infrared data is input into the first convolutional neural network branch, and the first convolutional neural network branch predicts and outputs a temperature distribution based on the input long-wave infrared data , based on the predicted temperature distribution , the physical loss is calculated by using the physical loss function. In the training and optimization process of the first convolutional neural network branch, the degree to which the temperature distribution predicted by the first convolutional neural network branch deviates from the physical law is quantified by the physical loss. By minimizing the physical loss, the first convolutional neural network branch is obtained.

[0104] Specifically, in step S102B2, the temperature rise feature and the contour feature are weighted and fused, including: first, mapping the temperature rise feature corresponding to the long-wave infrared data and the contour feature corresponding to the short-wave infrared data to the same dimension, splicing the temperature rise feature and the contour feature mapped to the same dimension to form an infrared feature dataset; second, performing dimension reduction processing on the infrared feature dataset by principal component analysis, and retaining two principal components after dimension reduction processing, the two principal components being a first principal component and a second principal component, the first principal component reflecting temperature rise information, and the second principal component reflecting contour information; and finally, weighting and fusing the first principal component and the second principal component to obtain an infrared feature. The weight corresponding to the first principal component can be set to 0.6, and the weight corresponding to the second principal component can be set to 0.4.

[0105] The above-mentioned weighting and fusion of the temperature rise feature and the contour feature can obtain an infrared feature containing a gas temperature rise feature and a device temperature rise feature, so that the infrared feature can lock the heat object in the target area and the temperature rise abnormal behavior of the heat object, such as gas accumulation, gas leakage, abnormal temperature rise of gas, and device overheating, etc.

[0106] In one embodiment, in the step of extracting the point cloud feature of the radar point cloud data by three-dimensional sparse convolution, the feature extraction is performed by formula (4):

[0107] (4)

[0108] wherein, represents the radar point cloud data, represents a three-dimensional sparse convolution operation, represents the point cloud feature.

[0109] It needs to be explained that the point cloud feature can be understood as a structure deformation feature, which can reflect the structure deformation in the target area, such as roof collapse, structure deformation caused by water seepage, etc.

[0110] In one embodiment, radar features of the millimeter wave radar data are extracted by constant false alarm detection, including the following steps:

[0111] The millimeter wave radar data is subjected to constant false alarm detection to identify a moving target in the target area, and the millimeter wave radar data corresponding to the moving target is subjected to signal processing to extract radar features. The moving target is a moving person or a vibrating device.

[0112] It should be noted that the radar features include a person movement speed feature, a device vibration acceleration feature, and a rule violation intrusion distance feature. The person movement speed feature and the device vibration acceleration feature are combined with the point cloud features, which can more accurately reflect whether a roof collapse occurs in the target area. The combination of the person movement speed feature and the device vibration acceleration feature with the infrared features is also more conducive to accurately locking abnormal conditions such as gas leakage, gas accumulation, and device overheating.

[0113] It should also be noted that the rule violation intrusion distance can be understood as the distance between the person or device detected by the millimeter wave radar and the pre-set construction safety boundary, which can be understood as a geometric object defined in a global coordinate system. The rule violation intrusion distance feature can reflect whether a rule violation intrusion occurs in the target area.

[0114] For example, the millimeter wave radar data corresponding to the moving person is obtained, the complex sequence of the millimeter wave radar data across multiple consecutive pulses in the same distance unit is subjected to fast Fourier transform to obtain a velocity spectrum, and the velocity coordinate corresponding to the peak value in the velocity spectrum is obtained to obtain the person movement speed feature.

[0115] For example, the millimeter wave radar data corresponding to the vibrating device is subjected to short-time Fourier transform to obtain a time-frequency spectrum, the vibration frequency and vibration amplitude of the vibrating device are obtained from the time-frequency spectrum, and the acceleration peak value is determined according to the vibration frequency and vibration amplitude to obtain the device vibration acceleration feature.

[0116] In an example, millimeter wave radar data corresponding to the moving target is acquired, a sampling sequence of the millimeter wave radar data in a same pulse cycle is subjected to fast Fourier transform to obtain a distance spectrum, a radial distance of the moving target relative to the radar is acquired from the distance spectrum, a direction of arrival estimation algorithm is used to process the millimeter wave radar data to obtain a horizontal azimuth angle and a pitch angle of the moving target, the radial distance, the horizontal azimuth angle and the pitch angle of the moving target obtained are clustered to obtain a position state representation of the moving target, the position state representation is subjected to coordinate conversion to obtain a coordinate representation of the moving target in a global coordinate system, and a distance between the moving target and a construction safety boundary is determined according to the construction safety boundary preset in the global coordinate system and the coordinate representation of the moving target in the global coordinate system. It should be explained that the direction of arrival estimation algorithm is a conventional technical means for direction of arrival estimation in the prior art, and will not be described here.

[0117] It should be explained that the multi-modal features extracted from the monitoring data include 256-dimensional visible light features, 128-dimensional infrared features, 64-dimensional point cloud features and 64-dimensional radar features. It should be noted that the training samples corresponding to the multi-modal features are used for training of the target prediction model, and the test samples corresponding to the multi-modal features are used for prediction of the target prediction model.

[0118] It should be explained that the spatial features in step S102 are extracted from the infrared feature extraction network. Specifically, a long-wave infrared branch feature map corresponding to long-wave infrared data and a short-wave infrared branch feature map corresponding to short-wave infrared data are acquired from the infrared feature extraction network, the long-wave infrared branch feature map and the short-wave infrared branch feature map are spliced along the channel dimension to obtain a fused spatial feature map, a global average pooling operation is performed on the fused spatial feature map to obtain an initial spatial feature, and the initial spatial feature is normalized to obtain the spatial feature.

[0119] In an example, the step of splicing the multi-modal features and the spatial features to obtain the joint feature vector in step S102 can include the following steps: splicing the multi-modal features extracted from the monitoring data along the channel dimension to obtain a complete multi-modal feature vector; performing mapping processing on the spatial features to obtain a standard spatial feature vector, the dimension of the standard spatial feature vector being the same as that of the complete multi-modal feature vector; and splicing the complete multi-modal feature vector and the standard spatial feature vector along the channel dimension to obtain the joint feature vector.

[0120] It should be explained that the joint feature vector is input into the frequency-space modal alignment network, and the joint feature vector is processed by the frequency-space modal alignment network to obtain a multi-modal fusion feature vector aligned in the frequency domain and the spatial domain.

[0121] Reference Figure 3 andFigure 4 As shown, the frequency-space modal alignment network comprises at least one alignment processing module, and the alignment processing module comprises a cross-domain encoder, a dual-modal collaborative encoder, and a frequency-space alignment unit.

[0122] In an example, the cross-domain encoder comprises a residual dilation convolution layer, a frequency domain attention layer, a multi-head mutual attention layer, a full connection layer, an instance normalization layer, and a regularization layer. The dual-modal collaborative encoder comprises a dual-channel convolution layer, a modal interaction gate, and a self-attention enhancement layer. The frequency-space alignment unit comprises a normalization layer, a convolution layer, a cross-attention layer, a concatenation layer, and a full connection layer.

[0123] It is also necessary to explain that by inputting the joint feature vector into the cross-domain encoder of the alignment processing module, the joint feature vector is processed by the cross-domain encoder, and a first feature vector is output; by inputting the joint feature vector into the dual-modal collaborative encoder of the alignment processing module, the joint feature vector is processed by the dual-modal collaborative encoder, and a second feature vector is output; by inputting the first feature vector output by the cross-domain encoder and the second feature vector output by the dual-modal collaborative encoder into the frequency-space alignment unit, the second feature vector is cross-fused with the first feature vector by the frequency-space alignment unit, and a multi-modal fusion feature vector aligned in the frequency domain and the spatial domain is output.

[0124] In one embodiment, in step S103, the joint feature vector is subjected to frequency domain weighting processing to obtain a first feature vector, comprising steps S103A1 to S103A3.

[0125] Step S103A1: performing residual convolution processing on the joint feature vector to obtain a preliminary feature vector, the preliminary feature vector comprising preliminarily interacted multi-modal features and spatial features.

[0126] Step S103A2: performing frequency domain weighting processing on the preliminarily interacted multi-modal features in the preliminary feature vector to obtain multi-modal weighted features.

[0127] Step S103A3: based on a multi-head mutual attention mechanism, fusing the multi-modal weighted features with the preliminarily interacted spatial features in the preliminary feature vector to obtain the first feature vector.

[0128] Specifically, in step S103A1, the residual convolution processing is performed on the joint feature vector by the residual dilation convolution layer of the cross-domain encoder, and multi-scale context features are extracted by the residual convolution processing of the joint feature vector to obtain the preliminary feature vector.

[0129] In one embodiment, the frequency domain attention layer of the cross-domain encoder is used to perform frequency domain weighting processing on the preliminarily interacted multi-modal features in the preliminary feature vector.

[0130] Furthermore, step S103A2 performs frequency domain weighting on the multimodal features of the preliminary interaction in the preliminary feature vector to obtain multimodal weighted features, including steps S103A21 to S103A23.

[0131] Step S103A21: Extract the multimodal features of the initial interaction from the initial feature vector and perform frequency domain transformation on them to obtain the frequency domain features corresponding to the multimodal features.

[0132] Step S103A22: Normalize the frequency domain features corresponding to the multimodal features to determine the attention weight parameters corresponding to each frequency band of the frequency domain features.

[0133] Step S103A23: Use attention weight parameters to weight the frequency domain features to obtain multimodal weighted features.

[0134] Specifically, in step S103A22, the attention weight parameters can be determined using formula (5):

[0135] (5)

[0136] in, The attention weight parameters corresponding to each frequency band represent the frequency domain features. This represents the frequency domain features corresponding to multimodal features. and Indicates learnable parameters, This represents the normalized exponential function.

[0137] Specifically, in step S103A23, the frequency domain features are weighted using formula (6):

[0138] (6)

[0139] in, The first characteristic representing the frequency domain Each frequency band This represents multimodal weighted features.

[0140] Through the above step S103A2, the multimodal features of the initial interaction in the preliminary feature vector are subjected to frequency domain weighting to achieve feature optimization.

[0141] Specifically, in step S103A3, based on the multi-head mutual attention mechanism, the multimodal weighted features are fused with the spatial features of the initial interaction in the initial feature vector to obtain the first feature vector, including:

[0142] The multi-modal weighted features are linearly projected by a multi-head self-attention layer of the cross-domain encoder to generate a query matrix, and the spatial features of the preliminary interaction are linearly projected to generate a key matrix and a value matrix.

[0143] The multi-head self-attention layer includes a plurality of attention heads, and based on the query matrix and the key matrix, the feature output factor of each attention head is determined as: ; wherein m represents the index of the attention head, represents the feature output factor of the mth attention head, represents the query matrix of the mth attention head, represents the key matrix of the mth attention head represents the transpose of the mth attention head, m represents the dimension of the mth attention head, represents a normalized exponential function.

[0144] Based on the output factor of each attention head and the value matrix, the feature output of each attention head is calculated as: represents the feature output of the mth attention head, represents the value matrix of the mth attention head. The feature outputs of the plurality of attention heads are integrated by a fully connected layer of the cross-domain encoder to obtain and output a feature ; the feature is normalized by an instance normalization layer of the cross-domain encoder, the normalized feature is regularized by a regularization layer, and finally the regularized feature is nonlinearly transformed by an activation function of the cross-domain encoder to obtain a first feature vector.

[0145] In one embodiment, in step S103, the visible light features and the infrared features in the joint feature vector are extracted for modal interaction to obtain a second feature vector, including the following steps S103B1 to S103B4.

[0146] Step S103B1: Extracting the visible light features and the infrared features in the joint feature vector, and respectively performing convolution processing on the visible light features and the infrared features to obtain preliminary visible light features and preliminary infrared features.

[0147] Step S103B2: Weighted fusion of the preliminary visible light features and the preliminary infrared features to obtain interaction features.

[0148] Step S103B3: Based on the self-attention mechanism, the internal dependency relationship of the interaction features is calculated to obtain enhanced features.

[0149] Step S103B4: Nonlinear transformation of the enhanced features to obtain a second feature vector. ​​

[0150] Specifically, in step S103B1, the dual-channel convolutional layer of the dual-modal collaborative encoder is used to respectively perform convolutional processing on the visible light feature and the infrared feature, to obtain a preliminary visible light feature and a preliminary infrared feature .

[0151] Specifically, the preliminary visible light feature and the preliminary infrared feature output by the dual-channel convolutional layer are input into the modal interaction gate of the dual-modal collaborative encoder, and the modal interaction gate is used to perform weighted fusion on the preliminary visible light feature and the preliminary infrared feature.

[0152] In step S103B2, the interaction weight is calculated by formula (7):

[0153] (7)

[0154] wherein, and denote learnable parameters, denotes the interaction weight, denotes the preliminary visible light feature, denotes the preliminary infrared feature, ( ) denotes an s-shaped curve function.

[0155] Based on the interaction weight, the preliminary visible light feature and the preliminary infrared feature are weighted fused by formula (8):

[0156] (8)

[0157] wherein, denotes the interaction feature.

[0158] Specifically, the interaction feature output by the modal interaction gate is input into the self-attention enhancement layer of the dual-modal collaborative encoder, and the self-attention enhancement layer is used to calculate the internal dependency relationship of the interaction feature, and output an enhanced feature .

[0159] It should be explained that in step S103B3, the internal dependency relationship of the interaction feature is calculated based on the self-attention mechanism, to obtain the enhanced feature , including: performing linear projection transformation on the interaction feature to respectively generate a query matrix, a key matrix and a value matrix; performing matrix dot product operation on the query matrix and the key matrix to obtain an original correlation degree; performing normalization processing on the original correlation degree to obtain an internal dependency weight matrix; performing weighted summation on the value matrix by using the internal dependency weight matrix to obtain a preliminary enhanced feature; performing residual connection on the preliminary enhanced feature and the original interaction feature, and performing layer normalization processing on the feature obtained by the residual connection to obtain the enhanced feature .

[0160] Specifically, in step S103B4, the enhanced feature is subjected to nonlinear transformation processing by using the activation function of the dual-modal collaborative encoder to obtain a second feature vector.

[0161] In one embodiment, in step S103, the second feature vector and the first feature vector are cross-fused by using the frequency-space alignment unit to obtain a frequency and space aligned multi-modal fusion feature vector, including the following steps S103C1 to S103C3.

[0162] Step S103C1: The second feature vector and the first feature vector are normalized by the normalization layer of the frequency-space alignment unit.

[0163] Step S103C2: Determine the target dimension:

[0164] (9)

[0165] wherein, denotes the dimension of the first feature vector, denotes the dimension of the second feature vector, denotes a dimension adjustment parameter, denotes the minimum value between and , denotes a rounding operation, denotes the target dimension.

[0166] Step S103C3: The normalized second feature vector and the first feature vector are processed by the convolution operation of the convolution layer of the frequency-space alignment unit to unify the dimensions to the target dimension; the second feature vector and the first feature vector after dimension unification are cross-fused to obtain a frequency and space aligned multi-modal fusion feature vector.

[0167] It should be noted that regarding the cross-fusion of the second feature vector and the first feature vector, the following manner can be adopted: first, the first feature vector and the second feature vector are subjected to linear transformation processing by the cross-attention layer of the frequency-space alignment unit to obtain a query vector, a key vector and a value vector; the first feature vector is taken as the query vector, and the second feature vector is taken as the key vector and the value vector to calculate a first attention output; the second feature vector is taken as the query vector, and the first feature vector is taken as the key vector and the value vector to calculate a second attention output; then, the first attention output and the second attention output are spliced by the splicing layer of the frequency-space alignment unit to obtain a spliced feature vector; finally, the spliced feature vector is mapped by the full connection layer of the frequency-space alignment unit to obtain a frequency and space aligned multi-modal fusion feature vector.​

[0168] In one embodiment, reference Figure 4 As shown, the frequency-spatial modal alignment network may include three sequentially connected alignment processing modules. The output of the cross-domain encoder in the previous alignment processing module serves as the input of the cross-domain encoder in the next alignment processing module, and the output of the bimodal co-encoder in the previous alignment processing module serves as the input of the bimodal co-encoder in the next alignment processing module. Furthermore, each alignment processing module is connected to a corresponding decoding module, and the output of the frequency-spatial alignment unit of each alignment processing module serves as the input of the decoding module connected to it. For the decoding modules connected to the three alignment processing modules, the output of the decoding module connected to the previous alignment processing module serves as the input of the decoding module connected to the next alignment processing module. The output of the decoding module connected to the last alignment processing module is a frequency- and spatially aligned multimodal fusion feature vector. This frequency- and spatially aligned multimodal fusion feature vector is input into the target prediction model, which then predicts confidence scores for multiple security state categories.

[0169] In one embodiment, in step S104, the multimodal fusion feature vector is input into a pre-trained target prediction model to obtain confidence scores for multiple security status categories. This can be achieved through the following steps:

[0170] The target prediction model utilizes a fully connected layer to align multimodal fusion feature vectors in the frequency and spatial domains, mapping them from a high-dimensional feature space to multidimensional logits values ​​corresponding to multiple security state categories. These logits values ​​are the original values ​​output by the fully connected layer. The activation function of the target prediction model then transforms each logits value into a probability distribution ranging from [0,1]. The sum of the probability distributions corresponding to the multidimensional logits values ​​is 1.

[0171] The probability distribution corresponding to the above multidimensional logits values ​​is the confidence score of multiple safety status categories, reflecting the confidence level of the target prediction model in judging the safety status category.

[0172] In one embodiment, the multiple security status categories in step S104 include a first-class abnormal status category, a second-class abnormal status category, and a third-class abnormal status category.

[0173] Execute preset level alarms based on preset security thresholds and confidence scores for each security status category, including the following situations:

[0174] If the confidence score of the first abnormal state category is greater than the preset safety threshold, a first preset level alarm is triggered; if the confidence score of the second abnormal state category is greater than the preset safety threshold, a second preset level alarm is triggered; if the confidence score of the third abnormal state category is greater than the preset safety threshold, a third preset level alarm is triggered.

[0175] It should be noted that the first abnormal state category matches the water seepage in the plurality of safety state categories; the second abnormal state category matches the roof deformation, cracks, and equipment overheating in the plurality of safety state categories; and the third abnormal state category matches the roof collapse, gas leakage, gas accumulation, abnormal gas temperature rise, and illegal intrusion in the plurality of safety state categories.

[0176] It should be further noted that when the confidence score corresponding to any abnormality in the second abnormal state category is greater than the preset safety threshold, the second preset level alarm is triggered; and when the confidence score corresponding to any abnormality in the third abnormal state category is greater than the preset safety threshold, the third preset level alarm is triggered.

[0177] Optionally, the preset safety threshold is 0.8. Of course, the preset safety threshold can be set to different values for abnormal state categories of different dangerous degrees.

[0178] It should be noted that the plurality of safety state categories in step S104 also includes a normal working condition. When the confidence scores corresponding to the first abnormal state category, the second abnormal state category, and the third abnormal state category predicted by the target prediction model are all less than the preset safety threshold, the safety state of the target region of the mine is a normal working condition, and no alarm is triggered.

[0179] It should be noted that the monitoring data used in steps S102 to S104 is the labeled test sample obtained in step S101.

[0180] In one embodiment, in step S201, training samples corresponding to the multi-modal feature and the multi-modal fusion feature vector are obtained, and effective training samples are constructed using the training samples, including the following steps S2011 and S2012.

[0181] Step S2011: Obtain the training sample of the multi-modal fusion feature vector, construct a causal diagram using the training sample of the multi-modal fusion feature vector as a variable, and determine the real abnormal feature according to the causal diagram.

[0182] Step S2012: Obtain the training sample of the multi-modal feature, and perform similarity matching between the training sample of the multi-modal feature and the real abnormal feature to obtain effective training samples. The effective training samples are labeled with real category labels.

[0183] It should be noted that, in step S2012, obtaining training samples for multimodal features can be achieved by referring to step S102, which involves feature extraction from the labeled and data-augmented training samples to obtain training samples for multimodal features. The specific process will not be elaborated further. Similarly, in step S2011, obtaining training samples for multimodal fusion feature vectors can be achieved by referring to steps S102 and S103, which involve processing the labeled and data-augmented training samples to obtain training samples for multimodal fusion feature vectors. The specific process will not be elaborated further.

[0184] It should be noted that the construction of the causal graph in step S2011 can use the Peter-Clark algorithm. The Peter-Clark algorithm is a constraint-based causal graph learning algorithm that constructs the causal graph through conditional independence testing.

[0185] Specifically, in step S2011, the Peter-Clark algorithm is used to construct a causal graph using the training samples of the multimodal fusion feature vector as variables, and the true abnormal features are determined based on the causal graph, including the following steps S20111 to S20113.

[0186] Step S20111: Construct an undirected graph, which includes multiple nodes; where each node represents a variable in the training sample of the multimodal fusion feature vector, and the variable can be specifically understood as visible light features, infrared features, point cloud features and radar features extracted from the training sample.

[0187] The set of variables represented by multiple nodes is represented as ;in, This represents the total number of variables.

[0188] Step S20112: Calculate the conditional mutual information of any two variables in the undirected graph, determine whether the two variables are conditionally independent based on the conditional mutual information, and determine the direction of the edge between the two variables to obtain the causal graph.

[0189] Step S20113: Determine the causal relationships between variables in the cause-effect diagram to separate the true abnormal features and the interference factor features.

[0190] Specifically, in step S20112, for any two variables in an undirected graph... and and subsets of variables Determining the relationship between two variables by calculating conditional mutual information. and Whether they are conditionally independent. The formula for calculating conditional mutual information is:

[0191] (10)

[0192] in,i , j denotes the index of the variable, 1 < i ≤ | V |, i , 1 < j ≤ | V |, , denotes the i-th variable, i denotes the j-th variable, denotes the conditional mutual information, j denotes the joint probability distribution of variables , denotes the joint probability distribution of variables , , ; denotes the conditional probability distribution of variables given variables and ; denotes the conditional probability distribution of variables given variables ; denotes the conditional probability distribution of variables given variables ; denotes the specific value of variable , denotes the specific value of variable , denotes a specific combination of values of variables in the variable subset . It is noted that if , then variables and are conditionally independent given the variable subset , and the edge between variables and is removed.

[0193] Specifically, in step S20112, for each undirected edge in the undirected graph, if there exists a variable such that variables and are conditionally independent given variable , and variables and , variables and are all independent, then the direction of the edge between variables and is determined as . denotes the undirected edge between variables and .

[0194] ​It should be further explained that the causal diagram obtained through the above step S20112 can clearly present the causal relationship between the variables, so that in step S20113, the causal relationship between the variables in the diagram can be determined according to the causal diagram, and the real abnormal feature and the interference factor feature are separated. The real abnormal feature is a feature that has a direct causal relationship with the real classification label; the interference factor feature is a feature that has no direct causal relationship with the classification label, and the interference factor feature will affect the prediction confidence score accuracy of the model, which belongs to environmental interference.

[0195] It should be further explained that in step S2012, the training samples of the multi-modal feature and the real abnormal feature are matched in similarity, which can be performed in the following manner: normalizing the training samples of the multi-modal feature and the real abnormal feature, determining the weight of each modal feature according to the causal contribution degree of each modal feature to the abnormality in the causal diagram, calculating the similarity between the training samples of the multi-modal feature and the real abnormal feature using the weight corresponding to each modal feature, and screening the training samples of the multi-modal feature with a similarity greater than a preset similarity threshold as effective training samples.

[0196] It should be noted that the teacher model and the student model both adopt a Transformer framework including an embedding layer, a multi-head self-attention layer, a feedforward network layer and a full connection layer. The Transformer framework is a deep neural network model based on a self-attention mechanism. The effective training samples are input into the embedding layer, and the embedding layer is used to map and position encode the effective training samples to obtain and output a feature map; the feature map output by the embedding layer is input into the multi-head self-attention layer, and the multi-head self-attention layer is used to weight and aggregate the feature map to capture the complex dependencies of the multi-modal features, to obtain and output an enhanced feature sequence; the enhanced feature sequence output by the multi-head self-attention layer is input into the feedforward network layer, and the feedforward network layer is used to perform nonlinear transformation processing on the enhanced feature sequence to obtain and output a nonlinearly enhanced feature sequence; the feature sequence output by the feedforward network layer is input into the full connection layer, and the full connection layer is used to map the nonlinearly enhanced feature sequence to multi-dimensional logits values corresponding to the plurality of safety state categories. Finally, the multi-dimensional logits values are converted into a probability distribution with a value range of [0, 1] through the activation functions of the teacher model and the student model.

[0197] It should be further explained that in the training process, the effective training samples are respectively input into the pre-constructed student model and teacher model, the multi-head self-attention layer of the teacher model outputs teacher features , and the multi-head self-attention layer of the student model outputs student features , wherein, represents the total number of feature channels of the teacher model, represents the total number of feature channels of the student model, represents a real number, a height of a feature map, a width of a feature map. The student feature output by the student model is mapped to the teacher feature dimension by a 1x1 convolution operation, and the mapping process is ; wherein, denotes a 1x1 convolution operation, denotes the student feature mapped to the teacher feature dimension.

[0198] Further, the knowledge distillation training of the student model by the teacher model in step S202 includes steps S2021 to S2023.

[0199] Step S2021: performing convolution processing on the student feature output by the student model to map the student feature output by the student model to the teacher feature dimension, to obtain the student feature mapped to the teacher feature dimension ; and calculating a feature distillation loss according to the teacher feature output by the teacher model and the student feature mapped to the teacher feature dimension:

[0200] (11)

[0201] wherein, denotes the feature distillation loss, denotes the total number of feature channels of the teacher model, c denotes the index of the feature channel, and 1≤c≤ , denotes a height of a feature map, h denotes the index in the height dimension of the feature map, and 1≤h≤ , denotes a width of a feature map, w denotes the index in the width dimension of the feature map, and 1≤w≤W, denotes the student feature mapped to the teacher feature dimension the feature value at position , denotes the feature value of the teacher feature at position .

[0202] Step S2022: calculating a prediction distillation loss according to the prediction probability distribution output by the student model and the prediction probability distribution output by the teacher model:

[0203] (12)

[0204] wherein, , the prediction probability distribution output by the student model, the prediction probability distribution output by the teacher model, used to separate the prediction probability distribution in the formula, G denotes the total number of the plurality of safety state categories, G takes a value of 10, and g denotes the index of the safety state category, 1 a predicted probability of the student model for the safety state category corresponding to the gth index, a predicted probability of the teacher model for the safety state category corresponding to the gth index, denotes the relative entropy, used to measure the difference between two predicted probability distributions and .

[0205] Step S2023: Calculate the total distillation loss according to the feature distillation loss and the prediction distillation loss, and train the student model by minimizing the total distillation loss.

[0206] wherein the expression of the total distillation loss is:

[0207] (13)

[0208] wherein, denotes the total distillation loss; denotes a weight parameter, used to balance the contribution of the feature distillation loss and the prediction distillation loss.

[0209] It should be noted that the predicted probability distribution output by the student model and the predicted probability distribution output by the teacher model are the probability distributions obtained by converting the logits values corresponding to the plurality of safety state categories.

[0210] It should be noted that, in order to improve the anti-interference ability of the target prediction model in a complex downhole environment, a preset adversarial perturbation can be added to the effective training sample during the model training process to generate an adversarial sample used to mislead the model prediction. During the model training, the effective training sample and the adversarial sample are both input into the student model and the teacher model to realize the improvement of the robustness of the target prediction model. The generation formula of the adversarial sample is:

[0211] (14)

[0212] wherein, denotes the effective training sample, denotes the adversarial sample; denotes a perturbation amplitude hyperparameter, > 0; is the classification loss of the student model for the effective training sample and the true class label y, denotes the gradient of the classification loss with respect to the effective training sample ; denotes a sign function.

[0213] It needs to be explained that in the model training process, the classification loss and the total distillation loss are taken as the total loss for model training, and the trained student model is obtained by minimizing the total loss.

[0214] It needs to be explained that based on the trained target prediction model, a lightweight computing node is constructed on the mine site, and a high-efficiency model architecture adapted to the local environment is mounted. The edge node of the high-efficiency model architecture receives real-time monitoring data through industrial Ethernet, predicts the confidence scores of multiple safety state categories according to the above steps S101 to S104, and performs the corresponding preset level of alarm operation according to the preset safety threshold and the predicted confidence scores of each safety state category.

[0215] In order to show the effect of the mine safety monitoring method based on image processing proposed in the present application, the following experimental verification results are given to prove it.

[0216] This experiment conducted a thirty-day field monitoring in the mining area of the mine, covering key areas such as roadway turning sections, core working faces, and transportation channels. The test scenarios of this experiment include normal lighting environment, low light dim environment, smoke dust environment, and complex mixed environment. Dual-light infrared thermal imager equipment, millimeter wave radar sensors, and laser radar sensors are used to synchronously collect monitoring data of the target area of the mine.

[0217] This experiment evaluates the method proposed in the present application from two aspects of multi-modal feature extraction effect and monitoring effect.

[0218] (1) The evaluation indexes of multi-modal feature extraction effect include feature confusion degree, abnormal recognition accuracy, and micro-feature detection rate. Among them, the feature confusion degree refers to the average cosine similarity between the features corresponding to different safety state categories. The lower the value of this feature confusion degree, the stronger the ability of the extracted multi-modal features to distinguish different hidden dangers. The abnormal recognition accuracy refers to the correct recognition rate of abnormal states in complex environments. The micro-feature detection rate refers to the detection proportion of micro-features such as cracks and water seepage.

[0219] The evaluation results of multi-modal feature extraction effect are shown in Table 1 below.

[0220] Table 1 Evaluation results of multi-modal feature extraction effect

[0221]

[0222] The normal lighting environment has the least interference on the monitoring data, the low-light dim environment, the smoke dust environment and the complex mixed environment all have different degrees of interference on the monitoring data. As can be seen from Table 1, in the normal lighting environment, the feature confusion degree is as low as 0.21, the abnormal recognition accuracy is 96.3%, and the micro feature detection rate is 92.5%, which shows that the extracted multi-modal features can fully show the precise feature distinguishing ability of normal working condition and various abnormalities. In the low-light dim environment, the three evaluation indexes fluctuate slightly, which shows that the details of the extracted visible light features are lost due to insufficient light, but the extracted infrared features and radar features play a complementary role, effectively suppressing the performance decline. In the smoke dust environment, the feature confusion degree reaches 0.35, the feature overlap is the most serious, the micro feature detection rate is 83.7%, but the abnormal recognition accuracy can still reach 89.6%, which shows that the extracted radar features retain the relevant features for identifying moving targets, and the extracted infrared features retain the features for capturing temperature rise, thereby ensuring that the core abnormalities are not missed. In the complex mixed environment, the feature confusion degree reaches 0.32, the abnormal recognition accuracy reaches 90.5%, and the micro feature detection rate reaches 85.1%, the three evaluation indexes reverse the smoke dust environment, which shows that each modality of the extracted multi-modal features can dynamically adapt to the interference type.

[0223] (2) In terms of monitoring effect, the present experiment compares the monitoring method one based on a single visible light model (hereinafter referred to as method one), the monitoring method two based on a traditional feature fusion model (hereinafter referred to as method two) and the method proposed in the present application, taking accuracy, false positive rate and response delay as evaluation indexes.

[0224] Among them, the accuracy refers to the proportion of abnormal samples correctly predicted by the model in the total abnormal samples. The false positive rate refers to the proportion of normal working conditions misjudged as abnormal states. The response delay refers to the average time from obtaining the monitoring data to the target prediction model outputting the prediction result. The evaluation results of the monitoring effect are shown in Table 2 as follows.

[0225] Table 2 Evaluation results of monitoring effect

[0226]

[0227] As can be seen from Table 2, the single visible light model is the worst in accuracy and has the highest false positive rate; the traditional feature fusion model is better than the single visible light model in both accuracy and false positive rate, but has the highest response delay; the target prediction model of the present application performs best in both accuracy and false positive rate, and has a lower response delay than the traditional feature fusion model. This shows that the single visible light model is limited by single modal data, and although it has the lowest response delay, its precision and reliability cannot meet the needs of complex scenarios; the traditional feature fusion model, although it improves the accuracy and reduces the false positive rate to a certain extent through multi-modal splicing, causes an increase in computational load and a high response delay due to the failure to solve the problem of alignment between the frequency domain and the spatial domain among the modes; and the method proposed in the present application reduces the response delay while improving the accuracy and reducing the false positive rate.

[0228] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope of the application being indicated by the following claims.

Claims

1. A mine safety monitoring method based on image processing, characterized in that, include: Acquire monitoring data of the target area in the mine, including visible light image data, infrared image data, radar point cloud data, and millimeter-wave radar data; Visible light features of the visible light image data are extracted by a feature extraction encoder, infrared features of the infrared image data are extracted by an infrared feature extraction network, point cloud features of the radar point cloud data are extracted by a three-dimensional sparse convolution, and radar features of the millimeter-wave radar data are extracted by constant false alarm rate detection. The visible light features, infrared features, point cloud features, and radar features constitute multimodal features. The multimodal features and spatial features are concatenated to obtain a joint feature vector. The joint feature vector is subjected to frequency domain weighting to obtain a first feature vector. The visible light feature and the infrared feature are extracted from the joint feature vector and modal interaction is performed to obtain a second feature vector. The second feature vector is cross-fused with the first feature vector to obtain a multimodal fusion feature vector aligned in the frequency domain and spatial domain. The step of performing frequency domain weighting on the joint feature vector to obtain the first feature vector includes: The joint feature vector is subjected to residual convolution to obtain a preliminary feature vector, which includes preliminary interactive multimodal features and spatial features; The multimodal features of the initial interactions in the preliminary feature vector are subjected to frequency domain weighting to obtain multimodal weighted features; Based on the multi-head mutual attention mechanism, the multimodal weighted features are fused with the spatial features of the preliminary interaction in the preliminary feature vector to obtain the first feature vector; The step of extracting the visible light features and the infrared features from the joint feature vector and performing modal interaction to obtain the second feature vector includes: The visible light features and the infrared features are extracted from the joint feature vector, and the visible light features and the infrared features are convolved respectively to obtain preliminary visible light features and preliminary infrared features; The preliminary visible light features and the preliminary infrared features are weighted and fused to obtain the interactive features; The internal dependencies of the interaction features are calculated based on the self-attention mechanism to obtain enhanced features; The enhanced features are then subjected to a nonlinear transformation to obtain the second feature vector; The multimodal fusion feature vector is input into a pre-trained target prediction model to obtain confidence scores for multiple security status categories. A preset level alarm is then executed based on a preset security threshold and the confidence scores for each security status category. The training of the target prediction model includes: Obtain the training samples corresponding to the multimodal features and the multimodal fusion feature vector, and use the training samples to construct effective training samples; The effective training samples are input into the pre-built student model and teacher model. The student model is trained by knowledge distillation using the teacher model. The trained student model is then used as the target prediction model.

2. The mine safety monitoring method based on image processing according to claim 1, characterized in that, The step of extracting visible light features from the visible light image data using a feature extraction encoder includes: The original high-frequency signal of the visible light image data is enhanced to obtain a high-frequency enhanced signal; The visible light image data is reconstructed based on the high-frequency enhanced signal and the original low-frequency signal of the visible light image data to obtain the interference-free visible light image data. The interference-free visible light image data is input into the feature extraction encoder to extract the visible light features.

3. The mine safety monitoring method based on image processing according to claim 1, characterized in that, The infrared image data includes long-wave infrared data and short-wave infrared data. Extracting infrared features from the infrared image data using an infrared feature extraction network includes: The long-wave infrared data and the short-wave infrared data are input into the infrared feature extraction network; The infrared feature extraction network is used to extract the temperature rise features of the target area from the long-wave infrared data and the contour features of the monitored target in the target area from the short-wave infrared data. The temperature rise features and the contour features are then weighted and fused to obtain the infrared features.

4. The mine safety monitoring method based on image processing according to claim 1, characterized in that, The extraction of radar features from the millimeter-wave radar data through constant false alarm rate (CFAR) detection includes: Constant false alarm rate (CFAR) detection is performed on the millimeter-wave radar data to identify moving targets within the target area, and signal processing is performed on the millimeter-wave radar data corresponding to the moving targets to extract the radar features.

5. The mine safety monitoring method based on image processing according to claim 1, characterized in that, The step of performing frequency domain weighting on the multimodal features of the initial interactions in the initial feature vector to obtain multimodal weighted features includes: The initial interaction multimodal features are extracted from the initial feature vector and subjected to frequency domain transformation to obtain the frequency domain features corresponding to the multimodal features; The frequency domain features corresponding to the multimodal features are normalized to determine the attention weight parameters corresponding to each frequency band of the frequency domain features. The frequency domain features are weighted using the attention weight parameters to obtain the multimodal weighted features.

6. The mine safety monitoring method based on image processing according to claim 1, characterized in that, The multiple security status categories include a first-class abnormal status category, a second-class abnormal status category, and a third-class abnormal status category; The step of executing a preset level alarm based on a preset security threshold and a confidence score for each of the security status categories includes: If the confidence score of the aforementioned abnormal state category is greater than the preset security threshold, then a first preset level alarm is triggered. If the confidence score of the two types of abnormal states is greater than the preset security threshold, then a second preset level alarm is triggered. If the confidence score of the three abnormal status categories is greater than the preset security threshold, a third preset level alarm will be triggered.

7. The mine safety monitoring method based on image processing according to claim 1, characterized in that, The step of obtaining the training samples corresponding to the multimodal features and the multimodal fusion feature vector, and constructing effective training samples using the training samples, includes: Obtain training samples of the multimodal fusion feature vector, construct a causal graph using the training samples of the multimodal fusion feature vector as variables, and determine the true abnormal features based on the causal graph; The training samples of the multimodal features are obtained, and the training samples of the multimodal features are matched with the real abnormal features to obtain the effective training samples.

8. The mine safety monitoring method based on image processing according to claim 1, characterized in that, The step of using the teacher model to train the student model through knowledge distillation includes: The student features output by the student model are mapped to the teacher feature dimension, and the feature distillation loss is calculated based on the teacher features output by the teacher model and the student features mapped to the teacher feature dimension. Calculate the predictive distillation loss based on the predicted probability distributions output by the student model and the teacher model. The total distillation loss is calculated based on the characteristic distillation loss and the predicted distillation loss, and the student model is trained by minimizing the total distillation loss.

Images