Conveyor drive system fault diagnosis method based on multi-modal feature fusion

Abstract

The application discloses a conveyor drive system fault diagnosis method based on multi-modal feature fusion. The method synchronously collects multi-source signals such as current, vibration and sound, and constructs a multi-branch neural network model integrating a cross-modal attention mechanism. The model can dynamically learn the interaction and weight of different signal features, realize deep fusion diagnosis, and locate key fault features combined with an explainable analysis technology. The system also adopts an online incremental learning mechanism, so that the model can continuously learn new fault samples and avoid forgetting historical knowledge. The application has the advantages that through deep fusion of multi-modal information, the diagnosis accuracy and reliability of early and complex faults are significantly improved; meanwhile, the decision-making process is explainable and can be updated adaptively, effectively prolonging the practical life cycle of an intelligent operation and maintenance system.

Description

Technical Field

[0001] This invention belongs to the field of conveyor fault detection, specifically relating to a fault diagnosis method for conveyor drive systems based on multimodal feature fusion. Background Technology

[0002] CN121612592A discloses a "Fault Diagnosis Method for Gearbox Bearings in Belt Conveyors of Coal-fired Power Plants." This patented technical solution is based on an order tracking analysis method for instantaneous frequency estimation without a tachometer to extract fault features from gearbox bearings. Specifically, it includes: obtaining instantaneous frequency ridges through the time-frequency distribution of vibration signals to estimate shaft frequency; improving the accuracy of instantaneous frequency ridges through an improved multi-source ridge fusion algorithm; and then using a computational order tracking (COT) algorithm to extract the order information of bearing fault features, adaptively addressing the fault diagnosis problem of belt conveyor drive gearbox bearings. The main problem is that this method relies only on a single vibration signal mode. Although ridge fusion improves feature extraction accuracy, it lacks complementary verification from multi-source heterogeneous information such as current, temperature, and sound, making it difficult to handle the coupling of multiple faults under complex operating conditions. Furthermore, it has limited sensitivity to early, weak faults, and the single information dimension leads to incomplete fault feature extraction.

[0003] In his article "A Review of Fault Diagnosis Methods for Belt Conveyor Systems," Professor Yang Chunyu systematically summarizes two main categories of diagnostic methods: knowledge-driven and data-driven. Existing measurement methods primarily collect equipment status data by deploying multiple sensors such as vibration, temperature, pressure, sound, speed, and images, and then analyze and diagnose using algorithms such as expert systems, fault trees, and support vector machines (SVM). The shortcomings of these technical solutions include: 1) Lack of multimodal information fusion mechanisms: Existing methods often only utilize single-modal data, lacking an effective multimodal information fusion mechanism, resulting in limited information dimensions and incomplete fault feature extraction; 2) Difficulty in diagnosing coupled faults: Multiple faults are often interconnected and coupled, making it difficult for existing single-fault diagnosis methods to effectively isolate and identify complex faults; 3) Insufficient model adaptability and interpretability: Traditional data-driven methods have poor adaptability to multiple fault types and lack a deep understanding of fault mechanisms, requiring improvement in model robustness and interpretability. Summary of the Invention

[0004] This invention proposes a fault diagnosis method for conveyor drive systems based on multimodal feature fusion, which solves the problems of traditional fault diagnosis relying on a single signal, incomplete feature extraction, and inability to adaptively update the model, resulting in low recognition rate and poor interpretability for early and complex faults.

[0005] The specific technical solution of this invention is as follows: A fault diagnosis method for a conveyor drive system based on multimodal feature fusion, characterized by comprising the following steps:

[0006] Step S1: Synchronously collect multi-source heterogeneous monitoring data of the conveyor in operation. The multi-source heterogeneous monitoring data includes at least: the three-phase current signal of the drive motor, the vibration signal of the drive system, and the acoustic signal of the drive system operating environment. Proceed to step S2.

[0007] Step S2: Perform spatiotemporal alignment and preprocessing on the multi-source heterogeneous monitoring data, and extract the modal features, including current feature vector, vibration feature vector and acoustic feature vector, to construct a high-dimensional fault feature fingerprint database, and then proceed to step S3.

[0008] Step S3: Construct a multimodal feature fusion diagnostic model. The multimodal feature fusion diagnostic model adopts a multi-branch hybrid neural network architecture that includes a cross-modal attention mechanism. Input the high-dimensional fault feature fingerprint database into the pre-trained multimodal feature fusion diagnostic model, dynamically learn and extract the interaction weights and deep fusion features of different modal features under specific fault modes, and proceed to step S4.

[0009] Step S4: Generate fault diagnosis results based on the interaction weights and deep fusion features of different modal features output by the model under specific fault modes. The fault diagnosis results include at least the equipment health status, fault type, and fault occurrence probability.

[0010] Compared with the prior art, the significant advantages of this invention are:

[0011] (1) By fusing multimodal data of current, vibration and acoustics and cross-modal attention mechanism, the limitations of single signal are overcome and the accuracy of early and complex fault identification is significantly improved.

[0012] (2) Introduce explainable artificial intelligence technology, reverse attribution to locate key features and root causes of failures, make the "black box" judgment transparent, and enhance the trust and decision-making basis of operation and maintenance.

[0013] (3) Incremental learning and anti-forgetting mechanisms are adopted to enable the model to continuously learn new failure modes without losing historical knowledge, thus greatly extending the effective life cycle of the system. Attached Figure Description

[0014] Figure 1 This diagram illustrates a fault diagnosis method for a conveyor drive system based on multimodal feature fusion, as described in this invention.

[0015] Figure 2 This is a diagram of a fault diagnosis system for a conveyor drive system based on multimodal feature fusion, according to the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0017] To make the structure and operation of the present invention easier to understand, the present invention will be further described below with reference to the accompanying drawings.

[0018] Combination Figure 2 A conveyor intelligent fault diagnosis system for implementing the above method includes:

[0019] Multi-source data acquisition module: used to synchronously acquire multi-source heterogeneous monitoring data during the operation of the conveyor, including current transformers, vibration acceleration sensors and acoustic sensors;

[0020] Feature engineering and processing module: used to perform time-scale alignment and noise reduction preprocessing on the multi-source heterogeneous monitoring data, and extract the current feature vector, vibration feature vector and acoustic feature vector;

[0021] Multimodal fusion diagnostic module: Deployed with the aforementioned multimodal feature fusion diagnostic model, used to receive feature vectors of each modality and output fault type and probability;

[0022] Interpretability and Root Cause Analysis Module: Combining interpretability algorithms, it performs reverse attribution of diagnostic decisions and outputs key contribution features;

[0023] Human-computer interaction and adaptive update module: used to visualize diagnostic results, early warning information and interpretable reports, and to trigger incremental learning of the model based on user feedback or new samples.

[0024] The sensors in the multi-source data acquisition module achieve high-precision time synchronization through the Precise Time Protocol (PTP) based on IEEE 1588 or a unified hardware-triggered clock, so as to ensure that multimodal data can be correlated and analyzed under the same time reference.

[0025] The system is deployed using a cloud-edge collaborative architecture. The edge computing gateway deploys the multi-source data acquisition module and the feature engineering and processing module to perform real-time data preprocessing and lightweight status monitoring. The cloud server deploys the multimodal fusion diagnostic module, the interpretability and root cause analysis module, and the model update framework to perform complex cross-modal fusion inference and incremental learning training tasks.

[0026] Combination Figure 1 A fault diagnosis method for a conveyor drive system based on multimodal feature fusion includes the following steps:

[0027] Step S1: Synchronously collect multi-source heterogeneous monitoring data of the conveyor in operation. The multi-source heterogeneous monitoring data includes at least: the three-phase current signal of the drive motor, the vibration signal of the drive system, and the acoustic signal of the drive system operating environment. Proceed to step S2.

[0028] Step S2: Spatiotemporal alignment and preprocessing of multi-source heterogeneous monitoring data are performed, and modal features, including current feature vector, vibration feature vector and acoustic feature vector, are extracted respectively to construct a high-dimensional fault feature fingerprint database.

[0029] The extracted current feature vector includes: the effective value, peak value, and waveform factor in the time domain, as well as the fundamental amplitude, harmonic amplitude, total harmonic distortion rate, and current harmonic impedance characteristics in the frequency domain.

[0030] The extracted vibration feature vector includes: the root mean square value and kurtosis in the time domain, and the amplitude features of the characteristic frequencies of the mechanical components and their vibration modulation sideband components obtained by envelope demodulation analysis in the frequency domain.

[0031] The extracted acoustic feature vector includes: Mel frequency cepstral coefficients (MFCC) extracted based on the acoustic signal and the cumulative energy features of the acoustic emission signal reflecting the friction state.

[0032] Proceed to step S3.

[0033] Step S3: Construct a multimodal feature fusion diagnostic model. The multimodal feature fusion diagnostic model adopts a multi-branch hybrid neural network architecture that includes a cross-modal attention mechanism. Input the high-dimensional fault feature fingerprint database into the pre-trained multimodal feature fusion diagnostic model, dynamically learn and extract the interaction weights and deep fusion features of different modal features under specific fault modes.

[0034] The multi-branch hybrid neural network architecture specifically includes:

[0035] The first branch network uses a Long Short-Term Memory (LSTM) network structure to process the current feature vector and extract the current time-dependent features.

[0036] The second branch network adopts a one-dimensional convolutional neural network (1D-CNN) structure to process the vibration feature vector and extract local spatial features of vibration.

[0037] The third branch network: adopts a Transformer encoder structure to process the acoustic feature vectors and extract global acoustic context features.

[0038] Cross-modal attention fusion layer: Receives the outputs of the first, second, and third branch networks, dynamically generates the fusion weights of each modality feature by calculating the correlation matrix between feature vectors of different modalities, realizes the weighted concatenation and nonlinear fusion of features, and outputs them to the fully connected classification layer.

[0039] Proceed to step S4.

[0040] Step S4: Generate fault diagnosis results based on the interaction weights and deep fusion features of different modal features output by the model under specific fault modes. The fault diagnosis results include at least the equipment health status, fault type, and fault occurrence probability.

[0041] Step S5: Perform fault interpretability analysis and root cause localization on the fault diagnosis results, as follows:

[0042] The interpretable artificial intelligence algorithm SHAP is introduced to perform reverse attribution analysis on the fault diagnosis results and calculate the marginal contribution of each modal feature extracted in step S2; a fault diagnosis interpretability report is generated and output. The report quantitatively indicates the top N key features that contribute the most to the current fault diagnosis conclusion and their respective modalities, so as to achieve preliminary location of the root cause of the fault.

[0043] Step S6: Adaptively update the multimodal feature fusion diagnostic model, as follows:

[0044] An online incremental learning framework is constructed. When unseen fault mode samples or manually verified misjudged correction samples are obtained, these samples are used as incremental datasets and input into the multimodal feature fusion diagnostic model for fine-tuning training. During the training process, the Elastic Weight Consolidation (EWC) algorithm is introduced. By calculating the Fisher information matrix of each modal feature extracted in step S2, the update magnitude of historically important parameters is limited. This prevents catastrophic forgetting of existing fault knowledge while learning new fault features, and enables the modal feature fusion diagnostic model to be updated adaptively.

[0045] During fine-tuning training, the goal of the model is to minimize the total loss.

[0046] Total loss = Cost of making a mistake on the new task + λ × penalty term, represented by the magnitude of change × importance:

[0047]

[0048] Where EWC represents the total model loss, This represents the magnitude of change, used to measure the extent of parameter variation; the greater the change, the greater the magnitude of the variation. The larger the value, the better. Represents an important metric; the more important it is, the better. The larger the value.

[0049] If the parameters are important, i.e. Large, even with a large range of change Even if the parameters are very small, the product can still be quite large, leading to a considerable penalty; if the parameters are unimportant, i.e. Small, even if the range of change is small The product may be very large, but the penalty is negligible.

[0050] Realizing the aforementioned system faces a series of intertwined technical and engineering challenges. The primary difficulty lies in the "alignment" and "fusion" of multimodal data. The physical meanings, sampling frequencies, and noise backgrounds of current, vibration, and acoustic signals vary significantly. Even with high-precision time synchronization methods like IEEE 1588, robust algorithms are needed for time-scale alignment, denoising, and feature scale normalization to ensure the effectiveness of subsequent correlation analysis. A deeper challenge lies in semantic fusion at the feature level: designing efficient network architectures (such as cross-modal attention) to dynamically mine and weight complementary and correlated information between different modalities, while suppressing redundant or conflicting information, is crucial for improving model performance. Secondly, the system's continuous learning capability in dynamic environments faces a balancing challenge. Online incremental learning aims to adapt the model to new data, but it is highly susceptible to "catastrophic forgetting," which can impair its ability to diagnose learned fault modes. Although algorithms such as elastic weight consolidation are introduced to constrain update magnitudes by calculating parameter importance, precise evaluation of parameter importance and setting reasonable constraint strengths to achieve the optimal balance between "remembering old knowledge" and "learning new skills" requires meticulous tuning. Finally, there is a huge gap between laboratory models and industrial implementation. Cloud-edge collaborative architectures need to complete high-quality feature extraction with limited edge computing resources, which places high demands on algorithm lightweighting. How to connect the "feature importance" obtained from interpretability analysis with the experience and knowledge of maintenance personnel and transform it into feasible maintenance work orders is the "last mile" for the ultimate realization of system value. This involves the integration of interdisciplinary knowledge and in-depth optimization of human-computer interaction design.

[0051] The multimodal feature fusion diagnostic model described in this invention is a customized implementation based on the PyTorch framework. The input feature dimension is determined based on the output of the aforementioned feature engineering module, and the specific parameter configuration is shown in the table below:

[0052]

[0053] The core significance of this invention lies in constructing a closed-loop diagnostic system that integrates multi-dimensional perception and intelligent decision-making, fundamentally changing the passive mode of traditional conveyor operation and maintenance that relies on single signals and periodic inspections. By simultaneously collecting three types of heterogeneous data—current, vibration, and acoustics—the system achieves cross-sensing and joint analysis of the electrical characteristics, mechanical state, and acoustic environment of the drive system. It can capture early, subtle fault signs and accurately identify complex faults, significantly improving the timeliness of fault warnings and the reliability of diagnostic conclusions. The key lies in its intelligent design of the entire process: "feature extraction-fusion diagnosis-interpretable analysis-online evolution." The multimodal fusion diagnostic model, leveraging a cross-modal attention mechanism, dynamically learns the contribution weights of each sensor signal under different fault modes, thereby extracting more discriminative fusion features. The introduction of SHAP interpretability analysis transforms the model's "black box" decisions into understandable key feature reports, providing maintenance personnel with direct fault location clues and significantly shortening troubleshooting time. Furthermore, the system's cloud-edge collaborative architecture and online incremental learning mechanism resolve the core contradictions in practical deployment: real-time, low-latency preprocessing and monitoring are achieved at the edge, while complex model training and updates are carried out in the cloud. The incremental learning framework enables the system to continuously learn from new fault samples or human feedback, constantly adapting to changes such as equipment aging and new fault modes, achieving self-evolution of diagnostic capabilities, and providing a feasible and sustainable technical paradigm for the digital transformation of predictive maintenance of industrial equipment.

Claims

1. A conveyor drive system fault diagnosis method based on multi-modal feature fusion, characterized in that, Includes the following steps: Step S1: Synchronously collect multi-source heterogeneous monitoring data of the conveyor in operation. The multi-source heterogeneous monitoring data includes at least: the three-phase current signal of the drive motor, the vibration signal of the drive system, and the acoustic signal of the drive system operating environment. Proceed to step S2. Step S2: Perform spatiotemporal alignment and preprocessing on the multi-source heterogeneous monitoring data, and extract the modal features, including current feature vector, vibration feature vector and acoustic feature vector, and then construct a high-dimensional fault feature fingerprint database, and proceed to step S3; Step S3: Construct a multimodal feature fusion diagnostic model. The multimodal feature fusion diagnostic model adopts a multi-branch hybrid neural network architecture that includes a cross-modal attention mechanism. Input the high-dimensional fault feature fingerprint database into the pre-trained multimodal feature fusion diagnostic model, dynamically learn and extract the interaction weights and deep fusion features of different modal features under specific fault modes, and proceed to step S4. Step S4: Generate fault diagnosis results based on the interaction weights and deep fusion features of different modal features output by the model under specific fault modes. The fault diagnosis results include at least the equipment health status, fault type, and fault occurrence probability.

2. The method of claim 1, wherein, In step S2, the multi-source heterogeneous monitoring data is spatiotemporally aligned and preprocessed, and the modal features are extracted respectively, as follows: The extracted current feature vector includes: the effective value, peak value, and waveform factor in the time domain, as well as the fundamental amplitude, harmonic amplitude, total harmonic distortion rate, and current harmonic impedance characteristics in the frequency domain. The extracted vibration feature vector includes: the root mean square value and kurtosis in the time domain, and the amplitude features of the characteristic frequencies of the mechanical components and their vibration modulation sideband components obtained by envelope demodulation analysis in the frequency domain; The extracted acoustic feature vector includes: Mel frequency cepstral coefficients (MFCC) extracted based on the acoustic signal and the cumulative energy features of the acoustic emission signal reflecting the friction state.

3. The method of claim 2, wherein, In step S3, the multi-branch hybrid neural network architecture specifically includes: The first branch network adopts a Long Short-Term Memory (LSTM) network structure to process the current feature vector and extract the current time-dependent features. The second branch network adopts a one-dimensional convolutional neural network (1D-CNN) structure to process vibration feature vectors and extract local features of vibration space. The third branch network: adopts a Transformer encoder structure to process acoustic feature vectors and extract global acoustic context features; Cross-modal attention fusion layer: Receives the outputs of the first, second, and third branch networks, dynamically generates the fusion weights of each modality feature by calculating the correlation matrix between feature vectors of different modalities, realizes the weighted concatenation and nonlinear fusion of features, and outputs them to the fully connected classification layer.

4. The method of claim 3, wherein, It also includes step S5, which performs fault interpretability analysis and root cause localization on the fault diagnosis results: The interpretable artificial intelligence algorithm SHAP is introduced to perform reverse attribution analysis on the fault diagnosis results and calculate the marginal contribution of each modal feature extracted in step S2. A fault diagnosis interpretability report is generated and output. The report quantitatively indicates the top N key features that contribute the most to the current fault diagnosis conclusion and their respective modes, so as to achieve preliminary location of the root cause of the fault.

5. The method of claim 4, wherein, It also includes step S6, which adaptively updates the multimodal feature fusion diagnostic model: An online incremental learning framework is constructed. When unseen fault mode samples or manually verified misjudged correction samples are obtained, these samples are used as incremental datasets and input into the multimodal feature fusion diagnostic model for fine-tuning training. During the training process, an elastic weight consolidation algorithm is introduced. By calculating the Fisher information matrix of each modal feature extracted in step S2, the update range of historical important parameters is limited. This prevents catastrophic forgetting of existing fault knowledge while learning new fault features, and enables the modal feature fusion diagnostic model to be updated adaptively.

6. The method of claim 5, wherein, During fine-tuning training, the goal of the model is to minimize the total loss; Total loss = Cost of making a mistake on the new task + λ × penalty term, represented by the magnitude of change × importance:

7. wherein, EWC denotes the total loss of the model, represents the change amplitude, which measures the amplitude of the parameter change, the larger the change, the larger the value of represent an important metric, the more important, the larger the value; If the parameters are important, i.e. The larger the value, the greater the magnitude of change. Even if the parameters are very small, the product can still be quite large, leading to a considerable penalty; if the parameters are unimportant, i.e. The smaller the value, the smaller the range of change. The product may be very large, but the penalty is negligible.

8. A conveyor intelligent fault diagnosis system for implementing the method of any one of claims 1-6, characterized in that, include: Multi-source data acquisition module: used to synchronously acquire multi-source heterogeneous monitoring data during the operation of the conveyor, including current transformers, vibration acceleration sensors and acoustic sensors; Feature engineering and processing module: used to perform time-scale alignment and noise reduction preprocessing on the multi-source heterogeneous monitoring data, and extract the current feature vector, vibration feature vector and acoustic feature vector; Multimodal fusion diagnostic module: Deployed with the aforementioned multimodal feature fusion diagnostic model, used to receive feature vectors of each modality and output fault type and probability; Interpretability and Root Cause Analysis Module: Combining interpretability algorithms, it performs reverse attribution of diagnostic decisions and outputs key contribution features; Human-computer interaction and adaptive update module: used to visualize diagnostic results, early warning information and interpretable reports, and to trigger incremental learning of the model based on user feedback or new samples.

9. The system according to claim 7, characterized in that: The sensors in the multi-source data acquisition module are synchronized with high precision through the Precision Time Protocol PTP based on IEEE 1588 or a unified hardware-triggered clock to ensure that multimodal data can be correlated and analyzed under the same time reference.

10. The system of claim 8, wherein: The system is deployed using a cloud-edge collaborative architecture. The edge computing gateway deploys the multi-source data acquisition module and the feature engineering and processing module to perform real-time data preprocessing and lightweight status monitoring. The cloud server deploys the multimodal fusion diagnostic module, the interpretability and root cause analysis module, and the model update framework to perform complex cross-modal fusion inference and incremental learning training tasks.

Images