A fault detection method and system based on self-attention and neural differential equation

Abstract

The present invention provides a fault detection method and system based on self-attention and neural differential equations, which belongs to the technical field of data analysis and machine learning. It solves the technical problem of efficiently identifying and detecting minor faults in dynamic and complex systems. Its technical solution is: including the following steps: 1) using the interquartile range IQR method to pre-process the collected sequence data to identify and eliminate outliers; 2) capturing long-distance dependencies in the sequence data; 3) constructing a neural ordinary differential equation model to analyze the extracted features; 4) performing a parameter search process based on a Bayesian statistical model to seek the optimal parameter combination. The beneficial effects of the present invention are: the present invention provides a novel, accurate and efficient detection technology solution for minor faults in dynamic and complex systems.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis and machine learning, and in particular to a fault detection method and system based on self-attention and neural differential equations. Background Art

[0002] In the highly automated and intelligent contemporary industrial environment, the stable operation of equipment and systems plays a vital role in ensuring production safety, improving efficiency and reducing costs. With the development of technology, systems are becoming more and more complex, which makes fault detection and diagnosis more difficult, especially the increasing need to accurately identify minor faults in the early stages. Minor faults, if not discovered and handled in time, may lead to system performance degradation and even cause major accidents. Therefore, developing a method that can accurately and promptly detect minor faults has great practical application value and social significance.

[0003] In time series data, the signals of minor faults often need to capture long-term data dependencies, while traditional models such as autoregressive models are limited in their effectiveness in dealing with long series dependencies. The operating environment and conditions of industrial systems are constantly changing, which requires fault detection methods to be able to adapt to these changes and achieve dynamic monitoring.

[0004] In recent years, with the rapid development of deep learning technology, especially the breakthroughs in natural language processing, some advanced technologies have been proposed and gradually applied to the processing of sequence data, such as the Transformer model and its self-attention mechanism, which have demonstrated a strong ability to handle long-distance dependency problems. In addition, neural differential equations, as an emerging deep learning framework, provide a continuous perspective for modeling dynamic systems, which has potential advantages for understanding and predicting the behavior of complex systems.

[0005] Nevertheless, applying these advanced technologies to the detection and diagnosis of minor faults still faces challenges such as adapting the model to the specific application environment, optimizing model parameters to achieve the best performance, and improving model interpretability. Therefore, a new fault detection method that comprehensively utilizes the advantages of self-attention mechanism and neural differential equations is needed to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a fault detection method and system based on self-attention and neural differential equations, focusing on the real-time detection and analysis of small faults in highly dynamic and complex systems.

[0007] The inventive concept of the present invention is as follows: the present invention aims to efficiently identify and analyze minor fault data, and is applicable to highly complex and dynamically changing system environments; the method uses the interquartile range method to identify and remove outliers in the data preparation and preprocessing stages to ensure a high degree of purification of data quality and provide accurate basic data for subsequent analysis. In terms of data feature extraction, the method of the present invention innovatively establishes a Transformer model, which effectively captures long-distance dependencies in sequence data through a self-attention mechanism, and can significantly improve the recognition accuracy and efficiency of fault features; the present invention constructs a neural ordinary differential equation model, and combines cross-validation technology for model training, thereby achieving high-sensitivity detection of minor faults under different operating conditions. Finally, a parameter search process is performed based on a Bayesian statistical model to efficiently seek the optimal parameter combination. The present invention provides a novel, accurate and efficient detection technology solution for minor faults in dynamic and complex systems, and has broad application prospects.

[0008] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a fault detection method based on self-attention and neural differential equation, comprising:

[0009] Data preparation and preprocessing, including IQR outlier processing: First, the interquartile range method is used to identify and remove outliers in the data. This step ensures the data quality of subsequent analysis and lays the foundation for high-accuracy fault detection. The data preparation and preprocessing also includes data standardization: standardizing the processed data to ensure comparability between different dimensional features.

[0010] Feature extraction, which includes the application of self-attention mechanism: the self-attention mechanism in the Transformer model is established to capture long-distance dependencies in the sequence data. This step effectively extracts key features in the time series data, especially small change signals, by calculating the attention scores between each position in the sequence. The feature extraction also includes deep feature learning of time series: using the features extracted by the self-attention mechanism as input, the deep features of the time series are further refined and learned through a deep learning network.

[0011] The fault detection model includes a neural differential equation model: a neural ordinary differential equation model is constructed to describe the continuous changes in the dynamic behavior of the system. The model can capture the subtle dynamic changes caused by minor faults, providing a new perspective for fault detection. The fault detection model also includes a cross-validation technology: a cross-validation technology is used to train and verify the neural differential equation model, ensuring the generalization ability and stability of the model.

[0012] Parameter optimization, which includes a Bayesian optimization strategy: performing parameter search based on a Bayesian statistical model to scientifically find the optimal model parameter combination. This step effectively shortens the optimization time and improves model performance by constructing a probability model of the parameters and using Bayesian inference to guide the search process.

[0013] Fault detection implementation includes early identification of minor faults: through the high sensitivity of the model to minor changes, early identification and early warning of minor faults are achieved. Fault detection implementation also includes real-time monitoring and diagnosis: the trained model is deployed in a real-time monitoring system to continuously monitor and diagnose faults in the industrial system.

[0014] The following steps are involved:

[0015] 5) The collected sequence data were preprocessed using the interquartile range (IQR) method to identify and remove outliers;

[0016] 6) Establish a Transformer model based on the self-attention mechanism to extract features from the preprocessed data to capture long-distance dependencies in the sequence data;

[0017] 7) Construct a neural ordinary differential equation model to analyze the extracted features, and combine the cross-validation technique to train the model to achieve high-sensitivity detection of minor faults under different operating conditions;

[0018] 8) Perform parameter search process based on Bayesian statistical model to efficiently seek the optimal parameter combination and improve the accuracy of fault detection.

[0019] As a further optimization scheme of the fault detection method based on self-attention and neural differential equation provided by the present invention, in the step 1), the data preprocessing step further includes a data normalization process to ensure that different feature dimensions are comparable in subsequent analysis.

[0020] As a further optimization scheme of the fault detection method based on self-attention and neural differential equations provided by the present invention, in the step 2), the Transformer model of the self-attention mechanism can automatically adjust the attention weight to focus on analyzing data features that are more relevant to fault detection.

[0021] As a further optimization scheme of the fault detection method based on self-attention and neural differential equations provided by the present invention, in the step 3), the neural differential equation model models the data flow in the manner of a continuous-time dynamic system to more accurately simulate and predict the system behavior.

[0022] As a further optimization scheme of the fault detection method based on self-attention and neural differential equations provided by the present invention, in the step 4), the Bayesian optimization algorithm constructs a probability model of the parameters and updates the model using prior knowledge and experimental data to effectively find the optimal parameter combination.

[0023] In order to better achieve the purpose of the present invention, the present invention also provides a fault detection system based on self-attention and neural differential equation, comprising:

[0024] 5) Data acquisition module, which is equipped with a variety of sensors for real-time capture of industrial system operating parameters. This module is used to perform real-time data acquisition, including but not limited to monitoring of indicators such as temperature, vibration, torque and sound;

[0025] 6) Data preprocessing module, which is equipped with a processor for preliminary cleaning, filtering and outlier removal of the collected raw sensor data, as well as performing data standardization and normalization to improve data quality and prepare for feature extraction;

[0026] 7) Algorithm control module, which is equipped with a Transformer model based on self-attention mechanism to extract key features from preprocessed data and build a neural differential equation model to simulate and analyze the dynamic changes of the system over time and identify potential minor faults;

[0027] 8) A parameter tuning module configured to optimize the parameters of the neural differential equation model using a Bayesian optimization algorithm to achieve more accurate fault detection.

[0028] As a further optimization scheme of the self-attention and neural differential equation fault detection system provided by the present invention, the data acquisition module is further configured to utilize multi-dimensional sensors to capture detailed data of the machine's operating status and related information in the environment, thereby providing real-time, multi-angle data support for fault diagnosis.

[0029] As a further optimization scheme of the fault detection system based on self-attention and neural differential equations provided by the present invention, the data preprocessing module is further configured to apply an algorithm to perform denoising and data conversion processing on the collected data to provide reliable and accurate data for fault detection.

[0030] As a further optimization scheme of the fault detection system based on self-attention and neural differential equations provided by the present invention, the algorithm control module is further configured to combine the self-attention mechanism with the neural differential equations, analyze the processed data and identify abnormal behavior patterns, so as to monitor minor faults in real time and respond immediately.

[0031] As a further optimization scheme for the self-attention and neural differential equation fault detection system provided by the present invention, the parameter tuning module is further configured to use the Bayesian optimization algorithm to fine-tune the parameters in the algorithm control module according to the fault diagnosis results, so as to optimize the performance of the system and improve the accuracy and efficiency of fault detection.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention uses the self-attention mechanism to deeply mine the long-distance dependencies in the sequence data, and can more accurately capture the subtle signal changes caused by minor faults. Compared with traditional methods, this method can significantly reduce false positives and false negatives, and improve the accuracy of fault detection. Especially when processing complex time series data, this method can effectively identify those minor abnormal patterns that are easily overlooked, providing strong technical support for early fault warning and timely maintenance.

[0034] 2. In the present invention, the neural differential equation is the core of the model, which provides a new perspective for describing and understanding the dynamic behavior of the system. Compared with the traditional discrete model, NODE, with its continuous characteristics, enables the model to adapt more naturally to slight changes in the system behavior, and enhances the adaptability and stability of the model under different working conditions and changing environments. This is especially important for industrial applications where the operating environment changes frequently or the system parameters are constantly adjusted.

[0035] 3. Although deep learning models are often criticized as "black box" models, this invention improves the transparency and interpretability of the model's decision-making process by introducing two visualization-friendly technologies: self-attention mechanism and neural differential equations. In particular, the introduction of the self-attention mechanism enables researchers and engineers to intuitively observe which data features are given higher weights when the model makes fault judgments, which helps to further analyze the causes of faults and optimize detection strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0037] Figure 1 This is a system framework diagram of a fault detection method based on self-attention and neural differential equations of the present invention.

[0038] Figure 2 This is a data flow diagram of a fault detection method based on self-attention and neural differential equations of the present invention.

[0039] Figure 3A schematic diagram of a model algorithm of a fault detection method based on self-attention and neural differential equations of the present invention.

[0040] Figure 4 This is a fault detection architecture diagram of a fault detection method based on self-attention and neural differential equations in the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0042] Example 1

[0043] like Figures 1 to 4 As shown, the technical solution provided by this embodiment is: a fault detection method based on self-attention and neural differential equations, including a data processing terminal, which is respectively connected to a data acquisition module, a preprocessing module, an algorithm control module and a parameter tuning module.

[0044] The acquisition module in this embodiment is used to obtain sensor data information in the industrial process, such as sound, vibration, pressure, temperature, etc. These sensors can be installed in various links of the industrial process to monitor and obtain the operating status and data in the industrial process. The sound sensor connected to the acquisition module is used to measure and monitor the machine noise level, abnormal changes in the sound of the equipment operation, or specific sound wave signals in the environment, and the temperature sensor connected to the acquisition module is used to measure and monitor the temperature of machine components such as motors, bearings or other mechanical parts during operation to ensure that they are within the safe and efficient working range. The sensor can accurately monitor and obtain the operating status and data in the industrial process, providing real-time and high-precision input for subsequent data analysis. In addition, the control algorithm and deep learning model are configured in the algorithm control module, and machine learning and deep learning can be used to realize data detection automation, fault analysis, fault diagnosis and other functions. These models can make decisions and adjustments based on feedback and learning data.

[0045] like Figure 1 As shown, the data acquisition module in this embodiment is responsible for real-time collection of multi-dimensional sensor data required for the machine's operating status, including but not limited to temperature, vibration frequency, sound waveform, and operating pressure. The module is equipped with various sensors and is located at the key link of the industrial process. The sensors can accurately monitor and obtain the operating status and data conditions in the industrial process, providing real-time and high-precision input for subsequent data analysis.

[0046] like Figure 1As shown, the preprocessing module includes software filtering, interquartile range method and data standardization. The main function of this module is to perform preliminary analysis and processing on the raw data collected from industrial process sensors, ensure data quality and provide accurate input for subsequent fault detection. This module uses advanced signal processing technology to filter out noise and statistical methods such as interquartile range to identify and remove outliers. The data is standardized by the Z score method to ensure the consistency and comparability of the data, thereby effectively reducing the complexity of subsequent calculations and improving the operating efficiency of the system.

[0047] like Figure 1 As shown, the parameter tuning module includes a Bayesian optimization algorithm, a probability model and cross-validation. The module continuously optimizes and adjusts the parameter settings in the algorithm module by using Bayesian optimization techniques. These techniques can adjust hyperparameters such as learning rate, number of network layers and number of hidden units based on the performance feedback of the algorithm to seek the best model performance. The performance of the parameters can be detected using cross-validation. The design of the parameter tuning module allows the system to automatically learn and adapt to new operating environments, continuously improve the accuracy of fault detection and the reliability of prediction, and ultimately achieve effective control of minor faults in complex systems.

[0048] like Figure 2 As shown in the figure, a Transformer model based on the self-attention mechanism is used to extract features from the data. The model combines the self-attention mechanism with the multi-head attention structure to model the complex relationships between time points in the time series data, thereby effectively capturing long-term dependencies. In the feature extraction process, the position encoding of the time series is combined to retain the order information of the time points. After the deep network structure of the model, a high-level representation vector containing the details and features of the time series is generated. This method not only enhances the accuracy of time series data analysis, but also improves the reliability of fault prediction, providing an efficient fault detection technology solution for industrial automation systems.

[0049] like Figure 3 As shown, the feature sequence extracted from the Transformer model is input into the Neural Ordinary Differential Equation model (NODE). This embodiment uses the unique characteristics of the NODE model to treat time as a continuous dimension to model the dynamic changes of the system. During the model training process, the NODE model gradually fine-tunes the parameters through an optimization process to minimize the difference between the predicted fault state and the actual fault state. This optimization process uses a loss function to evaluate the accuracy of fault prediction. Through this method, the system can accurately predict industrial machine faults and adaptively adjust over time to ensure high-precision fault detection performance throughout the machine's operating cycle.

[0050] The entire system uses efficient data processing capabilities and advanced algorithm models to achieve real-time monitoring, rapid diagnosis and automated processing of minor faults in industrial machines, greatly improving the stability and safety of industrial production. This system can not only reduce downtime caused by faults, but also continuously optimize performance in long-term operation and adapt to various complex and changing working environments.

[0051] Working principle: The specific implementation of this embodiment provides a fault detection method based on self-attention mechanism and neural ordinary differential equations. The method is specially designed for highly complex and dynamically changing industrial environments, aiming to improve the detection sensitivity and diagnostic accuracy of minor faults. First, real-time data is collected through a multidimensional sensor system (including temperature, pressure, vibration and sound sensors) installed at key locations of the industrial system. The acquisition frequency and data transmission adopt wireless protocols to upload to the central processing unit in real time. The preprocessing module runs on a high-performance microprocessor and is loaded with specially developed signal processing software. The software uses high-pass and low-pass filter algorithms to remove electromagnetic interference and sensor noise. These sensor data are filtered by preliminary software to remove noise and invalid data points caused by sensor failure or temporary external interference. Then, the interquartile range method is applied to further eliminate outliers in the data. The first quartile (Q1) and the third quartile (Q3) of each parameter data set are calculated, and then the threshold range of the outlier is defined. It is usually set to data points exceeding Q1-1.5×IQR, Q3+1.5×IQR, where

[0052] IQR=Q3-Q1

[0053] After outlier processing, the data were processed using the Z-score standardization method.

[0054]

[0055] Where x is the original data point, μ is the mean, and σ is the standard deviation to ensure that the data from different sensors are at the same level, laying the foundation for subsequent analysis.

[0056] For the preprocessed data, the Transformer model based on the self-attention mechanism is used for feature extraction. The model models the complex relationship between each time point through the self-attention mechanism, and automatically adjusts the weights to focus on those data features that are critical to fault prediction, effectively capturing the long-term dependencies in time series data. When the model is initialized, the number of self-attention layers, the number of heads, and the dimension of the hidden layer are adjusted according to the data characteristics and fault detection requirements. The attention score matrix is ​​constructed by calculating the association weights between different time points, and the input data is weighted by these scores, and then combined with position encoding to retain the position information in the time series, and finally a high-level feature representation is generated.

[0057] The extracted features are input into the neural differential equation model (NODE), and the characteristics of the NODE model are used to introduce time as a continuous variable to simulate the dynamic changes of the system over time. The NODE model continuously adjusts the model parameters by minimizing the difference between the predicted results and the actual fault state. The difference calculation formula is

[0058] L=||Y pred -Y actual || 2

[0059] where Y pred and Y actual Represent the predicted and actual fault states respectively.

[0060] To improve the accuracy of fault detection, the Bayesian optimization algorithm is used to fine-tune the NODE model parameters. Bayesian optimization calculates the posterior probability of each point in the parameter space and selects the parameter combination that is most likely to improve the model performance for testing based on existing experimental results and prior knowledge. The optimization process continues, and each round of iteration updates the probability model based on new performance feedback. This not only enhances the accuracy of parameter selection, but also significantly improves the model's ability to adapt to fault detection under different operating conditions. After each experiment, the prior distribution of the parameters is updated to accurately reflect which parameter combinations are likely to lead to better performance until the optimal parameter combination is found.

[0061] Finally, in order to achieve real-time monitoring and fault diagnosis, a user-friendly interface is designed to display the real-time data and status information of the industrial system, and the trained fault detection model is integrated into the monitoring system. The system dynamically updates the status indication and fault alarm on the interface according to the model analysis results, and provides operators with customizable alarm trigger conditions and fault diagnosis suggestions, so as to achieve rapid response when potential faults occur.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fault detection method based on self-attention and neural differential equations, characterized in that: The following steps are involved: Step 1): The collected sequence data were preprocessed using the interquartile range (IQR) method to identify and remove outliers; In step 1), the data preprocessing step further includes a data normalization process to achieve comparability of different feature dimensions in subsequent analysis; Step 2): Establish a Transformer model based on the self-attention mechanism to extract features from the preprocessed data to capture long-distance dependencies in the sequence data; In the step 2), the Transformer model of the self-attention mechanism automatically adjusts the attention weight to focus on analyzing the data features related to fault detection; Step 3): Construct a neural ordinary differential equation model to analyze the extracted features, and combine the cross-validation technique to train the model to achieve high-sensitivity detection of minor faults under different operating conditions; Step 4): Perform parameter search based on the Bayesian statistical model to find the optimal parameter combination; In step 4), the Bayesian statistical model constructs a probability model of parameters and updates the model using prior knowledge and experimental data to find the optimal parameter combination.

2. The fault detection method based on self-attention and neural differential equation according to claim 1, characterized in that: In the step 3), the neural differential equation model uses a continuous-time dynamic system to model data flow for simulating and predicting system behavior.

3. A fault detection system based on self-attention and neural differential equations, applying the fault detection method based on self-attention and neural differential equations as described in any one of claims 1-2, characterized in that: include: Data acquisition module, equipped with multiple sensors for real-time capture of industrial system operating parameters. This module is used to perform real-time data acquisition, including monitoring of temperature, vibration, torque and sound indicators; The data preprocessing module is equipped with a processor for preliminary cleaning, filtering and outlier removal of the collected raw sensor data, as well as performing data standardization and normalization to improve data quality and prepare for feature extraction; The algorithm control module is equipped with a Transformer model based on the self-attention mechanism to extract key features from the preprocessed data and build a neural differential equation model to simulate and analyze the dynamic changes of the system over time and identify potential minor faults; A parameter tuning module is configured to optimize the parameters of the neural differential equation model by applying a Bayesian optimization algorithm to achieve accurate fault detection.

4. The self-attention and neural differential equation based fault detection system according to claim 3, characterized in that: The data acquisition module is configured to use multi-dimensional sensors to capture detailed data on the machine's operating status and related information in the environment, providing real-time, multi-angle data support for fault diagnosis.

5. The self-attention and neural differential equation based fault detection system according to claim 3, characterized in that: The data preprocessing module is configured to apply an algorithm to perform denoising and data conversion processing on the collected data to provide accurate data for fault detection.

6. The self-attention and neural differential equation based fault detection system according to claim 3, characterized in that: The algorithm control module is further configured to combine the self-attention mechanism with the neural differential equation to analyze the processed data and identify abnormal behavior patterns, monitor minor faults in real time and respond immediately.

7. The self-attention and neural differential equation based fault detection system according to claim 3, characterized in that: The parameter tuning module is further configured to use a Bayesian optimization algorithm to fine-tune the parameters in the algorithm control module according to the fault diagnosis results, so as to optimize the performance of the system and improve the accuracy and efficiency of fault detection.

Images