Early warning method for coal mine equipment fault

Abstract

The invention provides a coal mine equipment fault early warning method, which comprises the following steps that: image data and time sequence data in the operation process of coal mine equipment are acquired, and time information is marked on the image data and the time sequence data; the image data and the time sequence data are preprocessed, preprocessing comprises standardization processing and fragmentation alignment, and an image tensor and a time sequence matrix are correspondingly obtained after preprocessing; cNN coding processing is carried out on the preprocessed image tensor, and spatial features are extracted; carrying out LSTM coding processing on the preprocessed time sequence matrix, and extracting time sequence features; fusing the spatial features and the time sequence features to generate fused features; inputting the fusion features into the trained identification model, and outputting the current fault type and risk level of the coal mine equipment; and inputting into a three-dimensional coal mine simulation model. The problems that in the prior art, data modals of coal mine equipment are diversified, time synchronization is difficult, model adaptability is insufficient, and a risk response mechanism is lacked are solved.

Description

Technical Field

[0001] This invention relates to the field of coal mining, and more particularly to a method for early warning of equipment failures in coal mines. Background Technology

[0002] As a core component of the coal production system, the operational safety and stability of coal mine equipment directly affect mine production efficiency and the safety of workers. With the development of intelligent mines and digital mine technologies, a large amount of operational data related to coal mine equipment is collected in real time, including multi-source heterogeneous data such as image data, vibration signals, electrical parameters, temperature, and pressure. This data contains crucial information such as equipment operating status, fault symptoms, and environmental interference. How to effectively extract and integrate fault characteristics from multi-source data has become a key issue in the development of intelligent early warning technology for coal mine equipment. Summary of the Invention

[0003] Based on the above problems, this invention proposes an early warning method for coal mine equipment failures. This invention solves the problems existing in the prior art, such as diverse data modalities in coal mine equipment, difficulties in time synchronization, insufficient model adaptability, and the lack of risk response mechanisms. By constructing an end-to-end data acquisition, heterogeneous feature extraction, modality fusion, risk classification, and visualization linkage process, this invention achieves accurate identification and efficient early warning of coal mine equipment failures.

[0004] This invention proposes an early warning method for coal mine equipment failures, comprising: Image data and time-series data are collected during the operation of coal mine equipment. Both image data and time-series data are labeled with time information. Image data and time series data are preprocessed, including standardization and fragment alignment, resulting in image tensors and time series matrices. The preprocessed image tensor is encoded using CNN to extract spatial features; The preprocessed time series matrix is ​​encoded using LSTM to extract time series features; Spatial and temporal features are fused using an attention mechanism to generate fused features; The fused features are input into the trained recognition model, which outputs the current fault type and risk level of the coal mine equipment. Input the current fault type and risk level of the coal mine equipment into the three-dimensional coal mine simulation model.

[0005] In addition, time-series data should include at least: equipment vibration data, electrical data, and temperature data.

[0006] In addition, standardization processing includes: scaling, grayscale conversion, and normalization of image data; Denoising, interpolation, and format standardization are performed on time-series data. Fragmentation alignment includes: resampling image data and time series data according to a fixed time window, processing the image data into an image tensor, and processing the time series data into a time series matrix, forming an input package of "image tensor + time series matrix" with completely consistent length and interval in the time dimension.

[0007] Furthermore, the preprocessed image tensor is subjected to CNN encoding to extract spatial features, including: The image tensor is taken as input, and after multi-layer convolution, batch normalization, ReLU activation and max pooling, the output feature map is processed. The feature map is globally average pooled and compressed into a fixed-length image embedding vector; The visual representation vector of the current time window is formed by weighted summation through self-attention. That is, spatial characteristics.

[0008] In addition, visual representation vectors These are abnormal signals in the appearance of coal mining equipment. Abnormal signals include at least: belt misalignment, drum deformation, and shell damage.

[0009] In addition, the preprocessed time series matrix is ​​encoded using LSTM to extract time series features, including: The time series matrix is ​​used as input data and fed into a two-layer LSTM model; The hidden state at the last time step of the output of a two-layer LSTM model As a semantic embedding representation of time series Time series semantic embedding representation This is a characteristic representation of the time series matrix.

[0010] Furthermore, if the coal mine equipment has multiple parallel input channels, multiple... Concatenation or weighting can be used as a semantic embedding representation of time series; The characteristics represented are the dynamic trends and abnormal evolution patterns of the time series matrix, including at least: temperature rise trend, current fluctuation and shock vibration.

[0011] Furthermore, by fusing spatial and temporal features through an attention mechanism, fused features are generated, including: Fusion features after using attention mechanism Represented as: ,in, , , This is the visual representation vector corresponding to the spatial features. This refers to the semantic embedding of time series corresponding to time series features; After fusing spatial and temporal features through an attention mechanism, the outputs of multiple attention heads are concatenated and projected into a feature vector of a unified dimension. :

[0012] in For the number of attention heads, It is a linear mapping matrix.

[0013] Furthermore, by inputting the fused features into the trained recognition model, the output of the current fault type and risk level of the coal mine equipment includes: The recognition model uses a standard fully connected neural network, with the fused feature vector as input. The output is a multi-class vector. , where C is the total number of fault types, and the multi-classification vector corresponds to the original prediction score of each fault type; The multi-component vector is fed into the softmax function to convert the original prediction scores of various faults into probability distributions of various faults, and the fault type with the highest confidence in the probability distribution is selected as the result.

[0014] In addition, inputting the current fault type and risk level of coal mine equipment into the 3D coal mine simulation model includes: The 3D coal mine simulation model marks the fault location on the coal mine equipment according to the current fault type, and automatically pushes maintenance suggestions according to the risk level; By jointly modeling the current time window results with historical results, the system can record and infer the fault evolution process.

[0015] This invention addresses the problems in existing technologies, such as diverse data modalities in coal mine equipment, difficulties in time synchronization, insufficient model adaptability, and the lack of risk response mechanisms. By constructing an end-to-end data acquisition, heterogeneous feature extraction, modality fusion, risk classification, and visualization linkage process, this invention achieves accurate identification and efficient early warning of coal mine equipment faults. Attached Figure Description

[0016] Figure 1 A flowchart illustrating an embodiment of the present invention for an early warning method for coal mine equipment failure; Figure 2 A block diagram of an early warning system for coal mine equipment failure provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to specific embodiments and accompanying drawings. This description is intended only to illustrate the specific embodiments of the invention and does not constitute any limitation on the invention. The scope of protection of the invention is defined by the claims.

[0018] Reference Figure 1 This invention proposes an early warning method for coal mine equipment failures, comprising: Step S001: Collect image data and time-series data during the operation of coal mine equipment. Both image data and time-series data are labeled with time information. Step S002: Preprocess the image data and time series data. The preprocessing includes: normalization and fragmentation alignment. After preprocessing, the image tensor and time series matrix are obtained. Step S003: Perform CNN encoding on the preprocessed image tensor to extract spatial features; Step S004: Perform LSTM encoding on the preprocessed time series matrix to extract time series features; Step S005: Spatial features and temporal features are fused through an attention mechanism to generate fused features; Step S006: Input the fused features into the trained recognition model and output the current fault type and risk level of the coal mine equipment; Step S007: Input the current fault type and risk level of the coal mine equipment into the three-dimensional coal mine simulation model.

[0019] Existing technologies have explored combining Convolutional Neural Networks (CNNs) and Long Short-Term Memory Networks (LSTMs) for fault feature extraction and status identification of equipment. CNNs excel at extracting local spatial features from images, while LSTMs are capable of processing dynamic evolution trends in time-series data. Combining these two methods for fault diagnosis under a single modality has yielded some success. However, these methods are only applicable to specific types of data input, such as single time-series electrical signals or image data, and are ill-suited to the complex needs of collaborative processing of multi-source heterogeneous data from coal mine equipment. Furthermore, existing technologies prioritize model training accuracy while neglecting practical issues such as inconsistent sampling frequencies between modalities, timestamp misalignment, and inconsistent feature semantics. They also lack visual interaction mechanisms and response strategies for interfacing with actual field systems.

[0020] In coal mine equipment operation scenarios, multi-source heterogeneous data exhibits typical characteristics such as large differences in sampling frequency, asynchronous acquisition time, high missing rate, and strong noise interference, making it difficult for traditional neural network models to effectively model them uniformly. Furthermore, due to the diverse types of coal mine equipment, complex operating conditions, and varying fault risk levels, unified fault identification models often lack sensitivity to different equipment states, resulting in persistently high false alarm and false negative rates. Therefore, there is an urgent need to propose a fault early warning method and system that is highly robust, fusion-capable, and possesses risk classification capabilities for multi-source heterogeneous coal mine data, and can be linked with a digital twin system to achieve dynamic response, addressing the technical challenges of existing technologies such as single modeling, insufficient fusion capabilities, and inability to respond to actual operating conditions.

[0021] This invention proposes an early warning method for coal mine equipment failures. In step S001, image data and time-series data during the operation of the coal mine equipment are collected, and both the image data and time-series data are labeled with time information. Optionally, the following data collection devices may be installed at key locations in coal mine equipment (such as belt conveyors, hydraulic supports, electric drums, etc.): Industrial camera module: Installed on the equipment casing or above moving parts, it is responsible for acquiring image data to identify equipment appearance abnormalities (such as belt cracks, bracket misalignment, roller deformation, etc.). The image sampling frequency is set to acquire 1 frame every 10 seconds, and the image resolution is 1280×720 pixels.

[0022] Triaxial accelerometer: Installed on the bearing housing of the electric drum or the cylinder body of the hydraulic cylinder, it collects vibration data during equipment operation. The sampling frequency is 1000Hz, and it is used to identify dynamic signal characteristics such as periodic mechanical impact and bearing wear.

[0023] Temperature sensor: Installed inside the equipment motor, hydraulic pipeline or control cabinet, it collects thermal status data with a sampling period of 1 second, and is used to analyze overheating risk and long-term temperature rise trend.

[0024] Current and voltage sensors: connected to the power supply circuit of the control system to monitor the electrical working status. The sampling frequency is 10Hz, and they are used to analyze electrical anomalies such as overload and undervoltage.

[0025] Optionally, all of the above-mentioned acquisition devices are connected to field edge acquisition nodes. The nodes are equipped with industrial-grade data acquisition cards, embedded edge processing chips, and unified high-precision time modules (such as GPS clocks or IEEE 1588 PTP modules) to ensure that multiple types of data are labeled with a unified timestamp and to achieve horizontal alignment of multi-source data.

[0026] Image data is acquired from an industrial camera module, and time-series data includes at least: equipment vibration data, electrical data, and temperature data. Equipment vibration data is acquired from a triaxial accelerometer, electrical data from current and voltage sensors, and temperature data from a temperature sensor.

[0027] In step S002, the image data and time series data are preprocessed. The preprocessing includes normalization and fragmentation alignment. After preprocessing, the image tensor and time series matrix are obtained. Industrial camera images are primarily used to detect equipment anomalies (such as belt misalignment, component deformation, and bracket cracks). However, the raw images suffer from background interference and scattered feature targets. Therefore, image data preprocessing is necessary. Optionally, image data can be scaled, converted to grayscale, or normalized.

[0028] Optionally, grayscale processing: convert the RGB image to a single-channel grayscale image to reduce redundancy; Size scaling: Unify image size to 224×224 pixels to match the input requirements of convolutional neural networks; Key region extraction: Use a pre-trained object detection model (such as YOLOv5) to select key structures (rollers, supports, belts), retaining only the working parts of the equipment; YOLOv5 is a deep learning-based object detection algorithm, YOLO (You Only Look Once).

[0029] Image enhancement (optional): Apply lightweight image enhancement operations (rotation, brightness perturbation, etc.) to improve model robustness and enhance sample diversity.

[0030] When preprocessing time series data, optionally, the time series data can be denoised, imputed, and formatted.

[0031] For equipment vibration data in time-series data, which serves as an important representation of mechanical condition, noise reduction is necessary, specifically including: Bandpass filtering: Use a bandpass filter in the 10Hz–500Hz frequency band to remove low-frequency trend terms and high-frequency spike noise; Envelope demodulation (optional): Use Hilbert transform to extract the envelope signal for bearing-related fault scenarios; Optionally, feature extraction, downsampling, and segmentation can also be performed here: with a sampling frequency of 1kHz and a time window of 30 seconds, there are a total of 30,000 data points, which are compressed into a one-dimensional vector input model with a length of 3000 using the sliding window mean method.

[0032] Optionally, the time series data of slow variables such as electrical data and temperature data in the time series data are processed. For the time series data of slow variables with sampling frequency below 1Hz, such as temperature data, current and voltage, the missing segments are filled by linear interpolation and the z-score (standard score) is used for standardization to eliminate the difference in dimensions. Optionally, the data from each channel (referring to equipment vibration data, electrical data, and temperature data) are spliced ​​into a two-dimensional time series tensor with a size of 30×N (N is the number of sensing channels, such as 4).

[0033] Optionally, fragmentation alignment includes: resampling the image data and time-series data according to a fixed time window, processing the image data into an image tensor, and processing the time-series data into a time-series matrix to form an input package of "image tensor + time-series matrix" with completely consistent length and interval in the time dimension.

[0034] To ensure the time synchronization of image data and time-series data, the image data and time-series data are resampled according to a fixed time window. To accommodate the subsequent modeling structure based on time windows, this step uses a fixed window sliding mechanism for data integration: optionally, the fixed time window length is set to 30 seconds; Within each fixed time window, image data, equipment vibration data, temperature data, and electrical data are extracted and aligned according to the labeled time information (which can be timestamps) to form a dataset for the corresponding time period. Optionally, 2–4 frames of image data are retained within each fixed time window, with a size of (batch, ...). , 224,224, 1); Optionally, each fixed time window of the vibration data corresponds to a 3000-dimensional feature vector; Optionally, for temperature and electrical data: the corresponding time series matrix has a size of (30, ); All data were assigned the same time window number and device number to facilitate matching, fusion, and label alignment in subsequent models. This process not only cleaned and standardized the input of multi-source heterogeneous data but also significantly reduced redundant information, improving the efficiency and accuracy of feature modeling.

[0035] The above steps, through the coordinated deployment of multiple types of sensors and the construction of a unified time reference system, have enabled efficient acquisition and multimodal synchronization of coal mine equipment operation data. This has solved the problems of heterogeneous data sources, inconsistent sampling, and incomparable time in the underground environment, and provided a standardized data foundation for subsequent decoupling modeling and fault identification.

[0036] In step S003, the preprocessed image tensor is subjected to CNN encoding to extract spatial features; optionally, the image tensor is used as input, and after multi-layer convolution, batch normalization, ReLU activation and max pooling, a feature map is output. The feature map is globally average pooled and compressed into a fixed-length image embedding vector; The visual representation vector of the current time window is formed by weighted summation through self-attention. That is, spatial characteristics.

[0037] Optionally, the image modeling pathway takes keyframe image tensors as input, and uses a lightweight CNN architecture (such as MobileNet or ResNet-18) as the network structure. The input is a grayscale image with a size of 224×224×1. After 3 to 5 layers of convolution, batch normalization, ReLU activation and max pooling, the output is a feature map. The feature maps are subjected to global average pooling (GAP) and compressed into fixed-length image embedding vectors (e.g., 128-dimensional). Multiple image embeddings can be retained for each time window, and they are weighted and summed through self-attention to form the visual representation vector of the current time window. .

[0038] This pathway focuses on extracting abnormal signals from the appearance features of coal mining equipment, such as belt misalignment, drum deformation, and shell damage. These features are clearly manifested in the spatial structure and are suitable for CNN modeling. CNN stands for Convolutional Neural Network.

[0039] In step S004, the preprocessed time series matrix is ​​subjected to LSTM encoding to extract time series features; Optionally, the time series matrix is ​​used as input data and fed into a two-layer LSTM model; the hidden state of the last time step is output by the two-layer LSTM model. As a semantic embedding representation of time series Time series semantic embedding representation This is a characteristic representation of the time series matrix.

[0040] Optionally, if the coal mining equipment has multiple parallel input channels, multiple... Concatenation or weighting can be used as a semantic embedding representation of time series; The characteristics represented are the dynamic trends and abnormal evolution patterns of the time series matrix, including at least: temperature rise trend, current fluctuation and shock vibration.

[0041] Optionally, the time series modeling path takes the time series matrix as input, and the processing flow is as follows: The equipment vibration data (one-dimensional vector) is concatenated with the slow variable time series (temperature data and electrical data) to form a unified multi-channel time series matrix input; the input size is... Where T is the time step (e.g., 30), and C is the number of sensor channels (e.g., 4-6); the network adopts a two-layer LSTM structure, with the number of hidden layer units set to 128; the final output is the hidden state at the last time step. As a semantic embedding representation of time series ; It is also known as a time-series characteristic; If the coal mine equipment has multiple channels for parallel input, multiple... Concatenate or weight the data as a time-series embedded output.

[0042] This pathway is mainly designed for time-sensitive data such as electrical, vibration, and thermal data, extracting their dynamic trends and abnormal evolution patterns, such as temperature rise trends, current fluctuations, and impact vibrations. It is suitable for modeling early, weak fault characteristics.

[0043] After completing the collection and format unification of heterogeneous data, to fully explore the structural features and semantic information of image data and time-series data, independent deep neural network modeling pathways were adopted for different modalities. Specifically, a convolutional neural network (CNN) was used to process image data to extract spatial structure and appearance anomaly features; a long short-term memory network (LSTM) was used to process time-series data to capture equipment state evolution trends and anomaly precursors. This decoupled modeling approach not only conforms to the operating mechanism of coal mine equipment itself, but also differs fundamentally from the single modeling approach of "using only LSTM to model vibration data" in existing technologies.

[0044] In step S005, spatial features and temporal features are fused through an attention mechanism to generate fused features; The core settings of the attention mechanism are: Query, Key, and Value: The Query uses image features. That is, Q = (Focus on "What information does the image need to extract from the time series"); Key and Value: Using time series features That is, K = V = (Time-series features serve as the "key" for the query and the "value" provided). The output of the i-th attention head (i = 1, 2, ..., h, where h is the number of attention heads) is: = (Q, K, V); Therefore, fusion features Represented as: This is the visual representation vector corresponding to the spatial features. This refers to the semantic embedding of time series corresponding to time series features; Spatial and temporal features are fused through an attention mechanism, and the outputs of multiple attention heads are concatenated and projected into a feature vector of a unified dimension. :

[0045] in For the number of attention heads, This is a linear mapping matrix. The purpose of the linear mapping matrix is ​​to perform a linear transformation on the concatenated features, further integrate multi-head information, and finally obtain a fused feature with a unified dimension.

[0046] Concat represents a concatenation operation, which concatenates the output vectors of h attention heads according to their dimensions to obtain an intermediate feature vector of dimension d.

[0047] The core idea is to integrate the different modal correlation information captured by multiple attention heads (for example, some heads focus on local detail correlations, while others focus on global trend correlations) into a comprehensive feature through "stitching + linear projection". This not only preserves the diversity of the multi-head mechanism, but also ensures the dimensionality of the output feature, providing a more comprehensive fusion representation for subsequent tasks (such as prediction, classification, etc.).

[0048] In step S006, the fused features are input into the trained recognition model, and the current fault type and risk level of the coal mine equipment are output. Optionally, the recognition model employs a standard fully connected neural network, with the input being the fused feature vector. The output is a multi-class vector. , where C is the total number of fault types, and the multi-classification vector corresponds to the original prediction score of each fault type; The multi-component vector is fed into the softmax function to convert the original prediction scores of various faults into probability distributions of various faults, and the fault type with the highest confidence in the probability distribution is selected as the result.

[0049] After completing the feature fusion of the cross-modal attention mechanism, this step uses the fused feature vectors. As input, the system performs fault type classification, health status assessment, and risk level output. It not only outputs specific fault types but also introduces a risk level classification mechanism specific to coal mine scenarios. Furthermore, it utilizes a digital twin to visualize fault conditions and generate response strategies, enhancing the system's intelligent decision-making and operational guidance capabilities.

[0050] The identification model, i.e., the fault classification and discrimination model, uses a standard fully connected neural network (MLP), with the fused feature vector as input. ; The output is a multi-class vector. Where C represents the total number of fault types, such as bearing damage, overheating, and belt misalignment in coal mining equipment; Use the softmax function to output various probabilities:

[0051] Optionally, the cross-entropy loss function is used to train the recognition model, and the class with the highest confidence is output as the current diagnostic result in actual deployment.

[0052] In step S007, the current fault type and risk level of the coal mine equipment are input into the three-dimensional coal mine simulation model.

[0053] To improve the operability of the model output and the response efficiency in the mining area, a risk level mechanism is introduced based on the fault type classification, as follows: Based on factors such as the type of classification results (e.g., critical components / non-critical components), confidence threshold, and the number of persistent anomalies within a time window, three risk levels are defined: High risk (red alert): There is a high probability of critical component failure (such as drum overheating or bracket breakage); Medium risk (orange alert): Non-critical components or intermittent abnormalities (such as slight belt misalignment); Low risk (blue alert): Early signs of slight vibration or temperature drift; continued observation is required. Risk levels are used to guide subsequent maintenance scheduling priorities and alarm strategies.

[0054] Based on the aforementioned judgment results and risk levels, this method, in conjunction with a three-dimensional coal mine simulation model (e.g., a mining area digital twin platform), achieves the following functions: Dynamic highlighting of 3D virtual models: Locate the corresponding coal mine equipment in the 3D coal mine simulation model and display it in red / orange / blue according to the risk level; Virtual maintenance suggestion generation: Automatically push maintenance suggestions based on risk level, such as "bearing needs to be replaced" or "suggest shutdown for inspection". Anomaly evolution trajectory recording: Jointly model the current time window results with historical results to achieve the recording and reasoning of the fault evolution process.

[0055] This step not only enables intelligent identification of coal mine equipment failure types, but also visualizes risk levels and maintenance recommendations, ultimately completing a closed-loop intelligent early warning process.

[0056] This invention addresses the problems in existing technologies, such as diverse data modalities in coal mine equipment, difficulties in time synchronization, insufficient model adaptability, and the lack of risk response mechanisms. By constructing an end-to-end data acquisition, heterogeneous feature extraction, modality fusion, risk classification, and visualization linkage process, this invention achieves accurate identification and efficient early warning of coal mine equipment faults.

[0057] This invention has significant technical advantages and practical value in the following aspects: First, the multi-source data acquisition and time synchronization mechanism designed in this invention supports the collaborative acquisition and alignment of image modalities and time-series sensing modalities. In response to problems such as inconsistent sampling frequencies and timestamp misalignment of multi-modal data in coal mines, a unified time window alignment strategy is proposed, which effectively improves the quality of data structuring and temporal consistency, laying the foundation for subsequent modeling.

[0058] Secondly, the decoupled neural network modeling method proposed in this invention optimizes image feature extraction and temporal feature modeling through convolutional neural networks (CNN) and long short-term memory networks (LSTM), respectively, and combines them with a unified modality embedding layer for semantic alignment, thereby achieving unified feature expression under heterogeneous modalities and effectively improving the model's ability to perceive the state of complex coal mine equipment.

[0059] Furthermore, this invention introduces a feature fusion module based on a cross-modal attention mechanism, utilizes a multi-head attention structure to extract key information fragments from each modality, and generates dynamic weights to achieve adaptive semantic fusion, thereby improving the robustness of fault symptom identification in complex environments and reducing the risk of false alarms and false negatives.

[0060] Furthermore, this invention constructs a risk assessment matrix by combining equipment importance level, historical failure probability, and current operating load, and generates multi-level risk level results based on fusion features and confidence mapping, clarifying alarm response strategies and effectively supporting the hierarchical management and differentiated maintenance of coal mine equipment.

[0061] Finally, this invention maps the fault identification results to a digital twin system, enabling the marking of fault locations and the overlay of risk heat maps in the equipment's 3D model, and supports historical status retrospection and remote operation and maintenance linkage, thereby enhancing the system's visualization monitoring capabilities and on-site response efficiency.

[0062] In summary, this invention achieves unified modeling, risk identification, and response scheduling of multi-source heterogeneous data in the complex operating environment of coal mines. It has significant advantages such as strong integration, accurate identification, timely response, and flexible deployment, and has good prospects for promotion and application.

[0063] In one embodiment, the timing data includes at least: equipment vibration data, electrical data, and temperature data.

[0064] The following data collection devices should be installed at key locations in coal mine equipment (such as belt conveyors, hydraulic supports, electric drums, etc.): Industrial camera module: Installed on the equipment casing or above moving parts, it is responsible for acquiring image data to identify equipment appearance abnormalities (such as belt cracks, bracket misalignment, roller deformation, etc.). The image sampling frequency is set to acquire 1 frame every 10 seconds, and the image resolution is 1280×720 pixels.

[0065] Triaxial accelerometer: Installed on the bearing housing of the electric drum or the cylinder body of the hydraulic cylinder, it collects vibration data during equipment operation. The sampling frequency is 1000Hz, and it is used to identify dynamic signal characteristics such as periodic mechanical impact and bearing wear.

[0066] Temperature sensor: Installed inside the equipment motor, hydraulic pipeline or control cabinet, it collects thermal status data with a sampling period of 1 second, and is used to analyze overheating risk and long-term temperature rise trend.

[0067] Current and voltage sensors: connected to the power supply circuit of the control system to monitor the electrical working status. The sampling frequency is 10Hz, and they are used to analyze electrical anomalies such as overload and undervoltage.

[0068] Optionally, all of the above-mentioned acquisition devices are connected to field edge acquisition nodes. The nodes are equipped with industrial-grade data acquisition cards, embedded edge processing chips, and unified high-precision time modules (such as GPS clocks or IEEE 1588 PTP modules) to ensure that multiple types of data are labeled with a unified timestamp and to achieve horizontal alignment of multi-source data.

[0069] Image data is acquired from industrial camera modules, and time-series data includes at least: equipment vibration data, electrical data, and temperature data. Equipment vibration data is acquired from a triaxial accelerometer, electrical data from current and voltage sensors, and temperature data from a temperature sensor. This data acquisition provides the foundation for subsequent data processing.

[0070] In one embodiment, the standardization process includes: scaling, grayscale conversion, and normalization of the image data; Denoising, interpolation, and format standardization are performed on time-series data. Fragmentation alignment includes: resampling image data and time series data according to a fixed time window, processing the image data into an image tensor, and processing the time series data into a time series matrix, forming an input package of "image tensor + time series matrix" with completely consistent length and interval in the time dimension.

[0071] Industrial camera images are primarily used to detect equipment anomalies (such as belt misalignment, component deformation, and bracket cracks). However, the raw images suffer from background interference and scattered feature targets. Therefore, image data preprocessing is necessary. Optionally, image data can be scaled, converted to grayscale, or normalized.

[0072] Optionally, grayscale processing: convert the RGB image to a single-channel grayscale image to reduce redundancy; Size scaling: Unify image size to 224×224 pixels to match the input requirements of convolutional neural networks; Key region extraction: Use a pre-trained object detection model (such as YOLOv5) to select key structures (rollers, supports, belts), retaining only the working parts of the equipment; YOLOv5 is a deep learning-based object detection algorithm, YOLO (You Only Look Once).

[0073] Image enhancement (optional): Apply lightweight image enhancement operations (rotation, brightness perturbation, etc.) to improve model robustness and enhance sample diversity.

[0074] Optionally, for image data, 2–4 representative keyframes are extracted within each fixed time window; Optionally, for equipment vibration data, a fixed-length sequence can be uniformly truncated according to the fixed time window length and sampling frequency. For example, if the equipment vibration data sampling rate is 1000Hz, then 30,000 data points can be collected in each fixed time window, and can be further downsampled at a ratio of 1:10 to a 3000-dimensional vector input model. All fixed time windows are numbered and cached to correspond one-to-one with subsequent feature extraction and fusion processes.

[0075] For slow variables with sampling frequencies below 1 Hz (such as temperature, current, and voltage), linear interpolation is uniformly used to fill in the missing segments, and z-score standardization is used to eliminate dimensional differences.

[0076] In one embodiment, the preprocessed image tensor is subjected to CNN encoding to extract spatial features, including: The image tensor is taken as input, and after multi-layer convolution, batch normalization, ReLU activation and max pooling, the output feature map is processed. The feature map is globally average pooled and compressed into a fixed-length image embedding vector; The visual representation vector of the current time window is formed by weighted summation through self-attention. That is, spatial characteristics.

[0077] Optionally, the image modeling pathway takes keyframe image tensors as input, and uses a lightweight CNN architecture (such as MobileNet or ResNet-18) as the network structure. The input is a grayscale image with a size of 224×224×1. After 3 to 5 layers of convolution, batch normalization, ReLU activation and max pooling, the output is a feature map. The feature maps are subjected to global average pooling (GAP) and compressed into fixed-length image embedding vectors (e.g., 128-dimensional). Multiple image embeddings can be retained for each time window, and they are weighted and summed through self-attention to form the visual representation vector of the current time window. .

[0078] This pathway focuses on extracting abnormal signals from the appearance features of coal mining equipment, such as belt misalignment, drum deformation, and shell damage. These features are clearly manifested in the spatial structure and are suitable for CNN modeling. CNN stands for Convolutional Neural Network.

[0079] In one embodiment, the visual representation vector These are abnormal signals in the appearance of coal mining equipment. Abnormal signals include at least: belt misalignment, drum deformation, and shell damage.

[0080] These abnormal signals are important indicators of problems with coal mine equipment, so it is necessary to extract the characteristics of these abnormal signals.

[0081] In one embodiment, the preprocessed time series matrix is ​​subjected to LSTM encoding to extract time series features, including: The time series matrix is ​​used as input data and fed into a two-layer LSTM model; The hidden state at the last time step of the output of a two-layer LSTM model As a semantic embedding representation of time series Time series semantic embedding representation This is a characteristic representation of the time series matrix.

[0082] The time series modeling approach takes a time series matrix as input, and the processing flow is as follows: The equipment vibration data (one-dimensional vector) is concatenated with the slow variable time series (temperature data and electrical data) to form a unified multi-channel time series matrix input; the input size is... Where T is the time step (e.g., 30), and C is the number of sensor channels (e.g., 4-6); the network adopts a two-layer LSTM structure, with the number of hidden layer units set to 128; the final output is the hidden state at the last time step. As a semantic embedding representation of time series ; It is also known as a time-series characteristic; If the coal mine equipment has multiple channels for parallel input, multiple... Concatenate or weight the data as a time-series embedded output.

[0083] This pathway is mainly designed for time-sensitive data such as electrical, vibration, and thermal data, extracting their dynamic trends and abnormal evolution patterns, such as temperature rise trends, current fluctuations, and impact vibrations. It is suitable for modeling early, weak fault characteristics.

[0084] In one embodiment, if the coal mining equipment has multiple parallel input channels, multiple... Concatenation or weighting can be used as a semantic embedding representation of time series; The characteristics represented are the dynamic trends and anomalous evolution patterns of the time series matrix, including at least: temperature rise trends, current fluctuations, and shock vibrations. This is achieved by using multiple... Concatenation or weighting, as a semantic embedding representation of time series, enables the simultaneous feature extraction of multiple parallel input channels.

[0085] In one embodiment, spatial features and temporal features are fused through an attention mechanism to generate fused features, including: By fusing spatial and temporal features through an attention mechanism, the generated fused features include: Fusion features after using attention mechanism Represented as: This is the visual representation vector corresponding to the spatial features. This refers to the semantic embedding of time series corresponding to time series features; After fusing spatial and temporal features through an attention mechanism, the outputs of multiple attention heads are concatenated and projected into a feature vector of a unified dimension. :

[0086] in For the number of attention heads, It is a linear mapping matrix.

[0087] The core settings of the attention mechanism are: Query, Key, and Value: The Query uses image features. That is, Q = (Focus on "What information does the image need to extract from the time series"); Key and Value: Using time series features That is, K = V = (Time-series features serve as the "key" for the query and the "value" provided). The output of the i-th attention head (i = 1, 2, ..., h, where h is the number of attention heads) is: = (Q, K, V); Therefore, fusion features Represented as: This is the visual representation vector corresponding to the spatial features. This refers to the semantic embedding of time series corresponding to time series features; Spatial and temporal features are fused through an attention mechanism, and the outputs of multiple attention heads are concatenated and projected into a feature vector of a unified dimension. :

[0088] in For the number of attention heads, This is a linear mapping matrix. The purpose of the linear mapping matrix is ​​to perform a linear transformation on the concatenated features, further integrate multi-head information, and finally obtain a fused feature with a unified dimension.

[0089] Concat represents a concatenation operation, which concatenates the output vectors of h attention heads according to their dimensions to obtain an intermediate feature vector of dimension d.

[0090] The core idea is to integrate the different modal correlation information captured by multiple attention heads (for example, some heads focus on local detail correlations, while others focus on global trend correlations) into a comprehensive feature through "stitching + linear projection". This not only preserves the diversity of the multi-head mechanism, but also ensures the dimensionality of the output feature, providing a more comprehensive fusion representation for subsequent tasks (such as prediction, classification, etc.).

[0091] The attention mechanism enables semantic co-modeling between images and temporal features, and can dynamically allocate modal attention intensity according to the operating status within different time windows. For issues such as occlusion in some images of coal mine sites and missing data from some sensors, the attention mechanism can achieve information redundancy compensation and improve robustness. Compared with the fixed-structure feature splicing method used in comparison files, this method has modal weight adaptability and semantic aggregation capability, making it more practical in multi-source data fusion scenarios.

[0092] In one embodiment, the fused features are input into the trained recognition model, and the output of the current fault type and risk level of the coal mine equipment includes: The recognition model uses a standard fully connected neural network, with the fused feature vector as input. The output is a multi-class vector. , where C is the total number of fault types, and the multi-classification vector corresponds to the original prediction score of each fault type; The multi-component vector is fed into the softmax function to convert the original prediction scores of various faults into probability distributions of various faults, and the fault type with the highest confidence in the probability distribution is selected as the result.

[0093] After completing the feature fusion of the cross-modal attention mechanism, this step uses the fused feature vectors. As input, the system performs fault type classification, health status assessment, and risk level output. It not only outputs specific fault types but also introduces a risk level classification mechanism specific to coal mine scenarios. Furthermore, it utilizes a digital twin to visualize fault conditions and generate response strategies, enhancing the system's intelligent decision-making and operational guidance capabilities.

[0094] The identification model, i.e., the fault classification and discrimination model, uses a standard fully connected neural network (MLP), with the fused feature vector as input. ; The output is a multi-class vector. Where C represents the total number of fault types, such as bearing damage, overheating, and belt misalignment in coal mining equipment; Use the softmax function to output various probabilities:

[0095] Optionally, the cross-entropy loss function is used to train the recognition model, and the class with the highest confidence is output as the current diagnostic result in actual deployment.

[0096] In one embodiment, inputting the current fault type and risk level of coal mine equipment into the three-dimensional coal mine simulation model includes: The 3D coal mine simulation model marks the fault location on the coal mine equipment according to the current fault type, and automatically pushes maintenance suggestions according to the risk level; By jointly modeling the current time window results with historical results, the system can record and infer the fault evolution process.

[0097] To improve the operability of the model output and the response efficiency in the mining area, a risk level mechanism is introduced based on the fault type classification, as follows: Based on factors such as the type of classification results (e.g., critical components / non-critical components), confidence threshold, and the number of persistent anomalies within a time window, three risk levels are defined: High risk (red alert): There is a high probability of critical component failure (such as drum overheating or bracket breakage); Medium risk (orange alert): Non-critical components or intermittent abnormalities (such as slight belt misalignment); Low risk (blue alert): Early signs of slight vibration or temperature drift; continued observation is required. Risk levels are used to guide subsequent maintenance scheduling priorities and alarm strategies.

[0098] Based on the aforementioned judgment results and risk levels, this method, in conjunction with a three-dimensional coal mine simulation model (e.g., a mining area digital twin platform), achieves the following functions: Dynamic highlighting of 3D virtual models: Locate the corresponding coal mine equipment in the 3D coal mine simulation model and display it in red / orange / blue according to the risk level; Virtual maintenance suggestion generation: Automatically push maintenance suggestions based on risk level, such as "bearing needs to be replaced" or "suggest shutdown for inspection". Anomaly evolution trajectory recording: Jointly model the current time window results with historical results to achieve the recording and reasoning of the fault evolution process.

[0099] This step not only enables intelligent identification of coal mine equipment failure types, but also visualizes risk levels and maintenance recommendations, ultimately completing a closed-loop intelligent early warning process.

[0100] Reference Figure 2 This embodiment provides an early warning system for coal mine equipment faults based on multi-source heterogeneous data. It enables real-time perception of the operating status of coal mine equipment, multi-modal feature extraction, intelligent identification of fault risks, and visual response. This system corresponds one-to-one with the methods described above, and its structural composition, functional division, and data interaction logic are as follows: I. System Overall Architecture like Figure 2 As shown, the system adopts a distributed multi-module collaborative structure and mainly includes the following functional modules: 1. Multi-source data acquisition module; 2. Data preprocessing and synchronization module; 3. Image feature extraction module (CNN branch); 4. Temporal Feature Extraction Module (LSTM Branch); 5. Cross-modal feature fusion module; 6. Fault classification and risk assessment module; 7. Digital twin visualization module; 8. Central control and communication scheduling module.

[0101] The above modules interact with each other through an internal communication bus, and are all subject to unified logic scheduling and status management by a central control module.

[0102] II. System Module Composition and Functions 1. Multi-source data acquisition module This module is deployed at the field equipment operating points and mainly includes: Image acquisition unit: Used for periodically acquiring images of the target device, supporting grayscale / infrared / visible light capture; Vibration sensor array: installed in key locations such as motors, bearings, and gearboxes; Slow-variable sensors such as electrical / temperature / pressure sensors; Timestamp recorder: Enables unified time-series identification for various types of data.

[0103] The collected data is uploaded to the edge computing node via industrial bus or wireless transmission.

[0104] 2. Data Preprocessing and Synchronization Module Image data is processed by scaling, grayscale conversion, normalization, etc. Denoising, imputation, and standardization are performed on time series data. Resampling is performed uniformly according to a fixed time window (e.g., every 30 seconds) to form a consistent "image tensor + time series matrix" input package.

[0105] 3. Image Feature Extraction Module (CNN Branch) Extracting spatial features from images using lightweight CNN networks (such as MobileNet); Embedding vector of output device appearance image Used for subsequent fusion processing; It can support model fine-tuning to adapt to the appearance characteristics of different coal mining equipment.

[0106] 4. Temporal Feature Extraction Module (LSTM Branch) Input structured sensor time series; Dynamic evolution features are extracted based on a two-layer LSTM structure, and the embedded vector is output. ; It supports sliding the state memory window to adapt to different operating rhythms.

[0107] 5. Cross-modal feature fusion module Input image features With time series characteristics This forms the input matrix; A multi-head attention mechanism is used to model inter-modal dependencies and generate fused feature vectors. ; The fused features are used to support fault identification and risk scoring.

[0108] 6. Fault Classification and Risk Assessment Module MLP network architecture receives fusion characteristics ; Output the corresponding fault type label and its confidence level; Based on rules such as equipment importance, confidence threshold, and duration of anomalies, a risk level (high / medium / low) is output for use in response strategies.

[0109] 7. Digital Twin Visualization Module The system interfaces with the mining area's 3D digital twin platform; Highlight the current warning device in the virtual model and label its risk level; Push visual operation and maintenance suggestions to assist engineers in remote diagnosis and decision-making.

[0110] 8. Central Control and Communication Scheduling Module Unified scheduling of data acquisition, modeling and inference, visualization and rendering operations; It has the capabilities of remote updates, edge inference deployment, and breakpoint resume; Provides standard API interfaces for interfacing with third-party systems (such as SCADA).

[0111] This system can be deployed at key nodes such as main conveyor belt systems, fully mechanized mining equipment, and motor drive units in coal mines. Data acquisition terminals and edge nodes are installed on-site, connecting to a central server via 5G / industrial WiFi to achieve continuous monitoring and intelligent early warning of equipment status. In engineering applications, this system significantly reduces blind spots in traditional manual inspections, shortens fault response time, and enhances management transparency and traceability through digital twin modeling.

[0112] The cross-modal attention mechanism, risk level output, and digital twin interaction function in the system constitute important creative features that distinguish it from the comparison documents; The modules are connected through standard interface protocols, which provides good modularity and scalability.

[0113] The multi-source data acquisition module is used to collect various types of sensor data, such as images, vibrations, electrical data, and temperature data, during equipment operation and to timestamp them. The data preprocessing and synchronization module is used to perform unified format standardization and fragment alignment on multi-source heterogeneous data. The image feature extraction module is used to perform CNN encoding on image data and extract spatial features; The temporal feature extraction module is used to perform LSTM encoding on temporal data and extract temporal features; The cross-modal fusion module is used to fuse image and temporal features through an attention mechanism to generate a fused feature representation; The fault classification and risk identification module is used to output the current fault type and risk level of the device based on fused features. The digital twin visualization module is used to map the recognition results to a 3D coal mine model, enabling alarm response, equipment location, and operation and maintenance assistance.

[0114] This embodiment acquires multi-source data, including images and time series data, and performs time synchronization and preprocessing. It extracts image and time series features using convolutional neural networks and long short-term memory networks, respectively. It uses a cross-modal attention mechanism to fuse heterogeneous features and generate a fused representation. It constructs a risk identification model and outputs multi-level early warning results. Finally, it maps the fault information to a digital twin platform for visualization.

[0115] Reference Figure 3 The present invention also proposes a hardware structure diagram of an electronic device, comprising: At least one processor 301; and, A memory 302 communicatively connected to at least one of the processors 301; wherein, The memory 302 stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the aforementioned early warning method for coal mine equipment failures.

[0116] Take processor 301 as an example.

[0117] The electronic device is preferably a controller. The electronic device may also include an input device 303 and a display device 304.

[0118] The processor 301, memory 302, input device 303 and display device 304 can be connected by a bus or other means. The figure shows an example of connection by bus.

[0119] The memory 302, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the early warning method for coal mine equipment faults in the embodiments of this application. Figure 1 The method flow is shown. The processor 301 executes various functional applications and data processing by running non-volatile software programs, instructions, and modules stored in the memory 302, thereby realizing the early warning method for coal mine equipment failures in the above embodiments.

[0120] The memory 302 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the early warning method for coal mine equipment failures. Furthermore, the memory 302 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 302 may optionally include memory remotely located relative to the processor 301, and these remote memories may be connected via a network to the apparatus performing the early warning method for coal mine equipment failures. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] Input device 303 can receive user clicks and generate signal inputs related to user settings and function control for early warning methods for coal mine equipment malfunctions. Display device 304 may include display devices such as a display screen.

[0122] One or more modules are stored in the memory 302, and when run by one or more processors 301, they execute the early warning method for coal mine equipment failures in any of the above method embodiments.

[0123] This embodiment acquires multi-source data, including images and time series data, and performs time synchronization and preprocessing. It extracts image and time series features using convolutional neural networks and long short-term memory networks, respectively. It uses a cross-modal attention mechanism to fuse heterogeneous features and generate a fused representation. It constructs a risk identification model and outputs multi-level early warning results. Finally, it maps the fault information to a digital twin platform for visualization.

[0124] One embodiment of the present invention provides a storage medium that stores computer instructions, which, when executed by a computer, are used to perform all the steps of the aforementioned early warning method for coal mine equipment failures.

[0125] In the context of this disclosure, a storage medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The storage medium can be a machine-readable signal medium or a machine-readable storage medium. Optionally, the storage medium can be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc ROM (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0126] This embodiment acquires multi-source data, including images and time series data, and performs time synchronization and preprocessing. It extracts image and time series features using convolutional neural networks and long short-term memory networks, respectively. It uses a cross-modal attention mechanism to fuse heterogeneous features and generate a fused representation. It constructs a risk identification model and outputs multi-level early warning results. Finally, it maps the fault information to a digital twin platform for visualization.

[0127] The above description is merely the principle and preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several other modifications can be made based on the principle of the present invention, and these modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for early warning of equipment failure in coal mines, characterized in that, include: Image data and time-series data are collected during the operation of coal mine equipment. Both image data and time-series data are labeled with time information. Image data and time series data are preprocessed, including standardization and fragment alignment, resulting in image tensors and time series matrices. The preprocessed image tensor is encoded using CNN to extract spatial features; The preprocessed time series matrix is ​​encoded using LSTM to extract time series features; Spatial and temporal features are fused using an attention mechanism to generate fused features; The fused features are input into the trained recognition model, which outputs the current fault type and risk level of the coal mine equipment. Input the current fault type and risk level of the coal mine equipment into the three-dimensional coal mine simulation model.

2. The early warning method for coal mine equipment failure according to claim 1, characterized in that, The time-series data should include at least: equipment vibration data, electrical data, and temperature data.

3. The early warning method for coal mine equipment failure according to claim 1, characterized in that, Standardization processing includes: scaling, grayscale conversion, and normalization of image data; Denoising, interpolation, and format standardization are performed on time-series data. Fragmentation alignment includes: resampling image data and time series data according to a fixed time window, processing the image data into an image tensor, and processing the time series data into a time series matrix, forming an input package of "image tensor + time series matrix" with completely consistent length and interval in the time dimension.

4. The early warning method for coal mine equipment failure according to claim 1, characterized in that, The preprocessed image tensor is encoded using CNN to extract spatial features, including: The image tensor is taken as input, and after multi-layer convolution, batch normalization, ReLU activation and max pooling, the output feature map is processed. The feature map is globally average pooled and compressed into a fixed-length image embedding vector; The visual representation vector of the current time window is formed by weighted summation through self-attention. That is, spatial characteristics.

5. The early warning method for coal mine equipment failure according to claim 4, characterized in that, Visual representation vector These are abnormal signals in the appearance of coal mining equipment. Abnormal signals include at least: belt misalignment, drum deformation, and shell damage.

6. The early warning method for coal mine equipment failure according to claim 1, characterized in that, The preprocessed time series matrix is ​​encoded using LSTM to extract time series features, including: The time series matrix is ​​used as input data and fed into a two-layer LSTM model; The hidden state at the last time step of the output of a two-layer LSTM model As a semantic embedding representation of time series Time series semantic embedding representation This is a characteristic representation of the time series matrix.

7. The early warning method for coal mine equipment failure according to claim 6, characterized in that, If the coal mine equipment has multiple parallel input channels, multiple... Concatenation or weighting can be used as a semantic embedding representation of time series; The characteristics represented are the dynamic trends and abnormal evolution patterns of the time series matrix, including at least: temperature rise trend, current fluctuation and shock vibration.

8. The early warning method for coal mine equipment failure according to claim 7, characterized in that, By fusing spatial and temporal features through an attention mechanism, the generated fused features include: Fusion features after using attention mechanism Represented as: ,in, , , This is the visual representation vector corresponding to the spatial features. This refers to the semantic embedding of time series corresponding to time series features; After fusing spatial and temporal features through an attention mechanism, the outputs of multiple attention heads are concatenated and projected into a feature vector of a unified dimension. : in For the number of attention heads, It is a linear mapping matrix.

9. The early warning method for coal mine equipment failure according to claim 8, characterized in that, The fused features are input into the trained recognition model, which outputs the current fault type and risk level of the coal mine equipment, including: The recognition model uses a standard fully connected neural network, with the fused feature vector as input. The output is a multi-class vector. , where C is the total number of fault types, and the multi-classification vector corresponds to the original prediction score of each fault type; The multi-component vector is fed into the softmax function to convert the original prediction scores of various faults into probability distributions of various faults, and the fault type with the highest confidence in the probability distribution is selected as the result.

10. The early warning method for coal mine equipment failure according to any one of claims 1-9, characterized in that, Inputting the current fault type and risk level of coal mine equipment into the 3D coal mine simulation model includes: The 3D coal mine simulation model marks the fault location on the coal mine equipment according to the current fault type, and automatically pushes maintenance suggestions according to the risk level; By jointly modeling the current time window results with historical results, the system can record and infer the fault evolution process.

Images