A device failure prediction system based on multi-modal deep learning

Abstract

The application belongs to the technical field of equipment fault prediction, and discloses an equipment fault prediction system based on a multi-modal deep learning, which is characterized by comprising a picture processing model, a fusion module, a TranAD model and a report output module; the picture processing model comprises an encoder, a generator, a first decoder and a second decoder; the encoder extracts a picture feature vector of an external input picture and inputs the picture feature vector into the generator to reconstruct a picture; the first decoder extracts a first decoding vector from the external input picture; and the second decoder extracts a second decoding vector from the reconstructed picture; the fusion module fuses the picture feature vector and an external time sequence vector and constructs a feature matrix to input the feature matrix into the TranAD model; the TranAD model predicts feature data in a next time window based on a feature matrix in a current time window; and the report output module calculates an error and outputs an error report for fault prediction, wherein the greater the error is, the higher the probability of a fault risk of the equipment is. Through the above method, the accuracy of fault prediction can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of equipment failure analysis, and more particularly relates to an equipment failure prediction system based on multi-modal deep learning. BACKGROUND

[0002] Electrical equipment is diverse in types and complex in operation mechanism, and has a high failure rate, which is easy to cause significant losses. Therefore, predictive maintenance of electrical equipment, that is, predicting the health status of the equipment through equipment data before failure occurs and carrying out targeted maintenance, is a key support and important guarantee for building a new power system.

[0003] The prior art can be mainly divided into two categories: model-based failure prediction and data-driven failure prediction. The model-based failure prediction technology requires that the mathematical model of the equipment is known, the degree of wear of the key components is evaluated by calculating the functional damage, and the cumulative effect of failure in use of the components is evaluated within the effective service life. The data-driven failure prediction technology does not require prior knowledge (mathematical model and expert experience) of the equipment, and is based on the collected equipment operation data to mine the implicit information therein through various data analysis processing methods for prediction operation.

[0004] However, the prediction accuracy of the model-based failure prediction is not high at present. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides an equipment failure prediction system based on multi-modal deep learning, which aims to improve the accuracy of equipment failure prediction.

[0006] To achieve the above-mentioned purpose, the present application provides an equipment failure prediction system based on multi-modal deep learning, which comprises:

[0007] a picture processing model comprising an encoder, a generator, a first decoder and a second decoder, the encoder being configured to extract a picture feature vector of an external input picture, the generator being configured to reconstruct a picture based on the picture feature vector, the first decoder being configured to extract a first decoding vector from the external input picture, and the second decoder being configured to extract a second decoding vector from the reconstructed picture;

[0008] a fusion module configured to fuse M1 external time sequence vectors and M2 picture feature vectors to obtain a fusion vector, and input column vectors in a (M1+M2+1)*N-dimensional feature matrix composed of the fusion vector, the M1 external time sequence vectors and the M2 picture feature vectors into a TranAD model in sequence; N is the number of elements in each vector, and the external time sequence vector is a digital signal collected;

[0009] a TranAD model configured to predict feature data in a next time window based on a feature matrix in a current time window;

[0010] a report output module configured to calculate a first error between the external input picture and the reconstructed picture, a second error between the first decoded vector and the second decoded vector, and a third error between feature data of the input TranAD model and feature data predicted by the TranAD model at the same time and the same position, and output an error report for fault prediction, wherein the greater the error, the higher the probability of the device having a fault risk;

[0011] wherein the picture processing model is obtained by training pictures of a normal device to make the sum of the first error and the second error converge, and the TranAD model is obtained by training feature data of the normal device to make the third error converge.

[0012] Optionally, the external input picture and the external time sequence vector are both encrypted signals obtained by homomorphic encryption of original sampling data; and a polynomial activation function is used instead of a nonlinear activation function in the activation function layer of the TranAD model.

[0013] Optionally, the device further comprises a collection module, wherein the collection module comprises:

[0014] a digital signal type sensor configured to collect a digital signal of the device;

[0015] an image signal type sensor configured to collect a picture of the device.

[0016] Optionally, the digital signal type sensor is a vibration sensor or a sound sensor; and the image signal type sensor comprises a camera.

[0017] Optionally, the collection module uses a non-blocking mode to send data collection instructions to each sensor through serial communication, so that each sensor performs a collection action at the same time.

[0018] Optionally, the digital signal is stored in an hdf5 file format, and the picture is stored in a jpg file format.

[0019] Optionally, the stored digital signal and picture are transmitted to the picture processing model and the fusion module after data alignment.

[0020] Optionally, the data input into the TranAD model is normalized data.

[0021] Optionally, N is 100-1000.

[0022] Optionally, adjacent time windows are different by one sampling step.

[0023] Overall, compared with the prior art, the above technical solutions conceived by the present application mainly have the following beneficial effects:

[0024] 1. The device fault prediction system based on multi-modal deep learning provided by the present application, on the one hand, extracts picture features through a picture processing model, fuses the picture features with external time sequence vectors, and inputs the data fused by the current time window into a TranAD model to predict the data of the next time window. Since the picture and time sequence signals are fused, the prediction accuracy can be improved through multi-modal deep learning. On the other hand, since the picture processing model and the TranAD model are both obtained by training on normal data, if the input data is normal data, the prediction result will tend to be close to the measured data. If the prediction result has a large error with the measured data, it indicates that the measured data is abnormal. In this way, the size of the fault risk can be judged based on the error size between the prediction result and the measured data. Furthermore, the present application also designs the picture processing model to calculate the error between the input picture and the reconstructed picture and the error between the input picture feature vector and the reconstructed picture feature vector. Similarly, the larger the error is, the higher the picture abnormal risk is. Combined with the three errors, more basis can be provided for fault prediction, thereby improving the accuracy of fault prediction.

[0025] 2. Further, by homomorphic encryption of the data, the homomorphic encrypted data is close to the original data, which can be directly input into the model for deep learning. The encrypted data can be directly used for model training, which can improve the security of the entire data transmission. Moreover, by improving the TranAD model, the BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a system block diagram of the device fault prediction system based on multi-modal deep learning in an embodiment of the present application;

[0027] Figure 2 is an analysis and comparison diagram of the predicted value and the input value of the TranAD model in an embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0029] As Figure 1 shown is a system block diagram of the device fault prediction system based on multi-modal deep learning in an embodiment of the present application, which at least includes:

[0030] The picture processing model comprises an encoder, a generator, a first decoder and a second decoder, the encoder is used for extracting a picture feature vector of an external input picture, the generator is used for reconstructing a picture based on the picture feature vector, the first decoder is used for feature extraction on the external input picture to obtain a first decoding vector, and the second decoder is used for feature extraction on the reconstructed picture to obtain a second decoding vector;

[0031] The fusion module is used for fusing M1 external time sequence vectors and M2 picture feature vectors to obtain a fusion vector, and sequentially inputting column vectors in a (M1+M2+1)*N-dimensional feature matrix composed of the fusion vector, the M1 external time sequence vectors and the M2 picture feature vectors into the TranAD model; N is the number of elements in each vector, and the external time sequence vector is a digital signal collected;

[0032] The TranAD model is used for predicting feature data in a next time window based on a feature matrix in a current time window;

[0033] The report output module is used for calculating a first error between the external input picture and the reconstructed picture, a second error between the first decoding vector and the second decoding vector, and a third error between feature data input into the TranAD model at the same time and at the same position and feature data predicted by the TranAD model, and outputting an error report for fault prediction, and the greater the error is, the higher the probability of a fault risk of the equipment is;

[0034] The picture processing model is obtained by training pictures of a normal equipment to make a sum of the first error and the second error converge, and the TranAD model is obtained by training feature data of the normal equipment to make the third error converge.

[0035] Further, the system further comprises a collection module, which is divided into a digital signal type sensor and an image signal type sensor, the digital signal type sensor is used for collecting digital signals of the equipment, and the image signal type sensor is used for collecting pictures of the equipment.

[0036] Specifically, the collection module can be a system using a CFET framework for serial communication and finite state machine control, and communicating with sensors through a Modbus protocol. The finite state machine includes states of a system kernel and conditions for switching between different states, i.e., a whole process of system preparation, asynchronous sending of collected signals, polling of a lower computer and obtaining of data, obtaining of a report by an edge platform and uploading of data, and returning to an initial state, and state jump conditions. After the collection module configures internal sensor system sampling rate, sampling length, channel number and other parameters, the collection module starts to collect, analyze and report data.

[0037] Specifically, the digital signal type sensor includes a vibration sensor and a sound sensor; the image signal type sensor includes a camera. For example, several vibration sensors, microphones, and dual-spectrum cameras can be installed on the device, and the acquisition module sends data acquisition instructions to each sensor through serial communication in a non-blocking manner. After each sensor returns a signal indicating that the instruction has been successfully received, it starts collecting data, wherein the digital signal is stored in an hdf5 file format, and the picture is stored in a jpg file format. Further, the stored digital signal and picture are transmitted to the picture processing model and the fusion module after data alignment, for example, the sensor data is saved to the SD card of the lower computer of the industrial computer system. The host computer of the industrial computer system reads the data of each sensor in turn to obtain aligned multi-modal data.

[0038] In an embodiment, in order to improve the security of system data transmission, the aligned multi-modal data is subjected to homomorphic encryption processing, and the encrypted data is uploaded to the server for analysis. At this time, the server approximates and replaces the non-linear activation function layer of the current model with a polynomial activation function, performs model training and fault prediction, thereby reducing time overhead.

[0039] Specifically, the construction process of the picture processing model is as follows:

[0040] (1) Extract the encrypted data in the server, including picture / video modal input. For video modal input, convert it to picture modal by frame interception, which adopts random interception;

[0041] (2) Construct an encoder. After the original picture data passes through the convolutional neural network, a picture feature vector is obtained, which is usually between 100 and 1000 dimensions;

[0042] (3) Construct a generator, which can generate a picture using the feature vector generated by the encoder;

[0043] (4) Train the encoder and the generator using normal picture data, so that the reconstruction error between the generated picture and the original picture is as small as possible;

[0044] (5) Construct a first decoder. After the original picture data passes through the convolutional neural network, an original picture decoding vector is obtained;

[0045] (6) Construct a second decoder. After the reconstructed picture data passes through the convolutional neural network, a reconstructed picture decoding vector is obtained;

[0046] (6) Train the two decoders using normal picture data, so that the error between the vectors generated by the two encoders is as small as possible.

[0047] The signal processing process of the fusion module is as follows:

[0048] (1) Dimension expansion of the vector generated via the encoder to the number of input sequence timestamps, resulting in a new 1*N vector, where N is the number of timestamps, representing picture information;

[0049] (2) Splicing of the M1 digital sequences representing digital modality input (external timing vector) and the M2 picture feature vectors representing picture information, resulting in an (M1+M2)*N matrix, where N is the number of timestamps, representing device-aligned multi-modal information;

[0050] (3) Construction of a fully connected layer for feature fusion of the digital modality input and the vector representing picture information, resulting in a new 1*N vector, which is a multi-modal fusion vector, spliced into the original modality to obtain a new (M1+M2+1)*N matrix, which is input into the TranAD model.

[0051] Specifically, the column vectors of the multi-modal information matrix are taken as the input of the TranAD model, and the output of the TranAD model is taken as the predicted value of the column vectors of the multi-modal information matrix at the next time. By comparing the predicted value and the true value of the column vectors of the multi-modal information matrix, a suitable TranAD model is obtained by minimizing the prediction error.

[0052] In application, the workflow of the entire system is as follows:

[0053] (1) The data acquisition module acquires data, including picture / video modality input and one or more digital modality input;

[0054] (2) The pictures are respectively input into the encoder, generator, and decoder obtained in the model training part, and the errors A between the original pictures and the real pictures and the errors B between the original picture decoding vectors and the generated picture decoding vectors are used to preliminarily determine whether the device is faulty.

[0055] (4) Dimension expansion of the vector generated via the encoder to the number of input sequence timestamps, resulting in a new 1*N vector, where N is the number of timestamps, representing picture information.

[0056] (5) Splicing of the digital sequences representing digital modality input and the vectors representing picture information, resulting in an (M1+M2)*N matrix.

[0057] (6) Feature fusion of the digital modality input and the vector representing picture information through a fully connected layer, resulting in a new 1*N vector, which is a multi-modal fusion vector, spliced into the original modality to obtain a new (M1+M2+1)*N matrix,

[0058] (7) The column vector of the multi-modal information matrix is taken as the input of the TranAD model, and the output of the TranAD model is regarded as the predicted value of the column vector of the multi-modal information matrix at the next moment. The predicted value of the column vector of the multi-modal information matrix is compared with the true value. Wherein the predicted value is to generate normal signal change at the next moment, the greater the gap between the predicted value and the true value, the more the real data does not conform to the normal signal rule, and then it is considered that the device is more likely to have an abnormal situation in the future. For example, the current input is a data matrix composed of data collected at t1 and t2 moments, and the output is data at t2 and t3 moments. By comparing the error between the input data at t2 and the predicted data, it is identified whether the input t2 data is abnormal, and then it is judged whether the device is abnormal.

[0059] As shown in Figure 2 The analysis comparison chart of the predicted value and the input value of the TranAD model in an embodiment is shown, wherein Value is the input value and the predicted value, and Anomaly Score represents the error score between the input value and the predicted value. It can be seen from the chart that there is data with a large error score, and it can be determined that the system has a risk of failure and needs to be checked for risk.

[0060] Overall, the TranAD multi-modal model is constructed to extract and fuse features of various modal data containing pictures, thereby reducing the prediction time and data processing difficulty, enhancing the fault prediction accuracy, and improving the device generalization. Further, the homomorphic encryption processing is performed to directly calculate on the encrypted data without decryption. This means that the encrypted data can be processed and analyzed in the cloud without exposing the original data, thereby reducing the risk of interception of data in the transmission process.

[0061] The technical features of the above-described embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the description. It should be noted that "in an embodiment of the present application", "for example", "further", etc. are intended to illustrate the present application, but not to limit the present application.

[0062] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application.

Claims

1. A device fault prediction system based on multimodal deep learning, characterized in that: include: The image processing model includes an encoder, a generator, a first decoder, and a second decoder, wherein the encoder is used to extract a picture feature vector of an external input picture, the generator is used to reconstruct a picture based on the picture feature vector, the first decoder is used to extract features of the external input picture to obtain a first decoding vector, and the second decoder is used to extract features of the reconstructed picture to obtain a second decoding vector; A fusion module is configured to fuse M1 external time series vectors and M2 image feature vectors to obtain a fused vector, and sequentially input the column vectors of the (M1+M2+1)*N-dimensional feature matrix composed of the fused vector, the M1 external time series vector, and the M2 image feature vectors into the TranAD model; N is the number of elements in each vector, and the external time series vector is the collected digital signal; The TranAD model is used to predict the feature data in the next time window based on the feature matrix in the current time window; a report output module, configured to calculate a first error between the external input image and the reconstructed image, a second error between the first decoding vector and the second decoding vector, and a third error between feature data of the input TranAD model at the same time and position and feature data predicted by the TranAD model, and output an error report for fault prediction, wherein a larger error indicates a higher probability of a fault risk of the device; The image processing model is obtained by training using images of a normal device to converge the sum of the first error and the second error, and the TranAD model is obtained by training using feature data of a normal device to converge the third error.

2. The equipment failure prediction system based on multimodal deep learning according to claim 1, characterized in that: The external input image and the external time series vector are both encrypted signals obtained by homomorphically encrypting the original sampled data; the activation function layer in the TranAD model uses a polynomial activation function instead of a nonlinear activation function.

3. The equipment failure prediction system based on multimodal deep learning according to claim 1, characterized in that: It also includes a collection module, which includes: Digital signal sensors are used to collect digital signals from devices; Image signal sensors are used to capture images of devices.

4. The equipment failure prediction system based on multimodal deep learning according to claim 3, characterized in that: The digital signal sensors include vibration sensors and sound sensors; the image signal sensors include cameras.

5. The equipment failure prediction system based on multimodal deep learning according to claim 3, characterized in that: The acquisition module uses a non-blocking method to send data acquisition instructions to each sensor through serial communication, so that each sensor performs acquisition actions at the same time.

6. The equipment failure prediction system based on multimodal deep learning according to claim 3, characterized in that: The digital signal is stored in hdf5 file format, and the picture is stored in jpg file format.

7. The equipment failure prediction system based on multimodal deep learning according to claim 6, characterized in that: The stored digital signals and images are passed to the image processing model and fusion module after data alignment.

8. The equipment failure prediction system based on multimodal deep learning according to claim 1, characterized in that: The data input into the TranAD model is normalized data.

9. The equipment failure prediction system based on multimodal deep learning according to claim 1, characterized in that: The value of N is 100 to 1000.

10. The equipment failure prediction system based on multimodal deep learning according to claim 1, characterized in that: The difference between adjacent time windows is one sampling step.

Images