Trustworthy Mechanical Fault Diagnosis Method, System, Device and Medium Based on Multimodal Data Imputation and Fusion

Abstract

A reliable mechanical fault diagnosis method, system, device and medium based on multi-modal data interpolation and fusion disclosed in this application relates to the field of electrical digital data processing. The method includes: respectively intercepting vibration signals and current signals from vibration data and current data according to the operating cycle of the mechanical equipment to be diagnosed; using the trained L2 spatio-temporal generative adversarial interpolation network as the data interpolation model and the trained multi-input single-output autoencoder as the data fusion model; inputting the vibration signals and current signals into the data interpolation model, obtaining the interpolated data of the vibration signals and the interpolated data of the current signals, and then inputting the obtained interpolated data into the data fusion model to obtain the fused data; constructing a fault diagnosis model and inputting the fused data into the fault diagnosis model to obtain the fault diagnosis result of the mechanical equipment to be diagnosed. This application can achieve reliable fault diagnosis of mechanical equipment by considering the long-term dependence relationship of data and comprehensively using the data interpolation and fusion method of multi-modal information.

Description

Technical Field

[0001] The present application relates to the field of electrical digital data processing, and in particular, to a trusted mechanical fault diagnosis method, system, device and medium based on multi-modal data imputation and fusion. Background Art

[0002] In engineering practical applications, the integrity and accuracy of sensing data are crucial for the operation monitoring and performance prediction of mechanical equipment systems. However, problems such as sensor failures, communication interruptions, and environmental interferences can easily lead to abnormal sensor data acquisition, among which the problem of missing sensor data is particularly prominent. Missing data refers to the situation where some data is missing or lost, making the monitored sensor data incomplete, which may further affect the state monitoring, fault diagnosis, and prediction results of the mechanical equipment system, thereby reducing the reliability and performance of the equipment. Therefore, effective data processing and imputation methods are needed to address the problem of missing sensor data during the monitoring of mechanical equipment, improve the integrity and reliability of sensor detection data, and ensure the safe and stable operation of the equipment system.

[0003] Common data processing methods include deleting missing values, interpolation methods, etc. Deleting missing values means directly deleting the missing values in the data. This method is applicable to the situation where the amount of missing data is small and the impact is small after deleting the missing values. The interpolation method estimates the missing data through known data, and this method is applicable to the situation where there is a certain correlation between the missing data. Machine learning methods such as K-nearest neighbor, random forest, etc. use the statistical characteristics of time series in a complex way to achieve data imputation.

[0004] Deep learning methods such as TCN (Temporal Convolutional Network), GAIN (Generative Adversarial Imputation Network), etc. extract the features of the data and model the missing values, thereby achieving the imputation of the missing values. Taking the generative adversarial imputation network as an example, by inputting the missing data, noise vector, and mask vector into the generator, the generator outputs the imputed data, and then inputting the imputed data and the hint mechanism into the discriminator, the discriminator outputs the probability that each component of the imputed data is estimated to be true. After adversarial training, the generator can achieve high-quality imputation of the missing data. By introducing additional information for the missing data, the generative adversarial imputation network can achieve data imputation.

[0005] Data fusion methods such as data-level fusion methods, feature-level fusion methods, and decision-level fusion methods can comprehensively utilize diverse information sources, improve the comprehensive ability and effect of data analysis, and thus provide more comprehensive support for fault diagnosis and prediction.

[0006] Based on the above description, although data imputation methods have achieved certain effects, they still have limitations. Directly deleting missing values will, on the one hand, lead to a reduction in the amount of data, and on the other hand, important data information will be lost, thus affecting the accuracy of the analysis results. Although the interpolation method can fill in missing values, it cannot consider the potential correlations and temporal dependencies between data. Machine learning methods use the statistical characteristics of time series to impute data, but when the proportion of missing data is large, machine learning methods still cannot effectively capture the distribution of data. The generative adversarial imputation network cannot consider the long-term dependencies of time series data, resulting in low-quality imputed data. In addition, data-level fusion methods have deficiencies such as data redundancy, increased dimensions, and low computational efficiency, which further lead to the inability to achieve reliable fault diagnosis of mechanical equipment. Summary of the Invention

[0007] The purpose of this application is to provide a reliable mechanical fault diagnosis method, system, device, and medium based on multi-modal data imputation and fusion. By considering the long-term dependencies of data and comprehensively using the data imputation and fusion method of multi-modal information, reliable fault diagnosis of mechanical equipment can be achieved.

[0008] To achieve the above purpose, the following solutions are provided in this application.

[0009] In the first aspect, this application provides a reliable mechanical fault diagnosis method based on multi-modal data imputation and fusion, including:

[0010] Obtain the vibration data and current data of the mechanical equipment to be diagnosed.

[0011] Intercept vibration signals from the vibration data and current signals from the current data according to the operating cycle of the mechanical equipment to be diagnosed.

[0012] Construct a data imputation model and a data fusion model. The data imputation model is a trained L2 spatio-temporal generative adversarial imputation network. The data in the dataset used for training the L2 spatio-temporal generative adversarial imputation network is obtained by simulating continuous loss of historical operation data under different mechanical health states. The historical operation data includes historical vibration data and historical current data. The data fusion model is a trained multi-input single-output autoencoder. The multi-input single-output autoencoder includes multiple encoders and one decoder.

[0013] Input the vibration signal and the current signal into the data imputation model respectively to obtain vibration signal imputation data and current signal imputation data.

[0014] Input the vibration signal imputation data and the current signal imputation data into the data fusion model to obtain fusion data.

[0015] Construct a fault diagnosis model, and input the fused data into the fault diagnosis model to obtain the fault diagnosis result of the mechanical equipment to be diagnosed.

[0016] Optionally, constructing the data imputation model and the data fusion model includes:

[0017] Intercept historical vibration signals and historical current signals from the historical vibration data and the historical current data respectively according to the operation cycle of the mechanical equipment.

[0018] Perform continuous value loss simulation on the historical vibration signal and the historical current signal respectively to obtain a historical abnormal vibration signal and a historical abnormal current signal.

[0019] Construct a first training set and a first test set based on the historical abnormal vibration signal and the historical abnormal current signal.

[0020] Construct an L2 spatio-temporal generative adversarial imputation network. The L2 spatio-temporal generative adversarial imputation network is a generative adversarial imputation network including an L2 regularization function and a spatio-temporal feature extraction module.

[0021] Train the L2 spatio-temporal generative adversarial imputation network with the first training set. When the first set condition is reached, regard the trained L2 spatio-temporal generative adversarial imputation network as the data imputation model.

[0022] Input the first test set into the data imputation model to obtain imputation data of the historical vibration signal and imputation data of the historical current signal.

[0023] Construct a second training set based on the imputation data of the historical vibration signal and the imputation data of the historical current signal.

[0024] Construct a multi-input single-output autoencoder.

[0025] Train the multi-input single-output autoencoder with the second training set. When the second set condition is reached, regard the trained multi-input single-output autoencoder as the data fusion model.

[0026] Optionally, constructing the L2 spatio-temporal generative adversarial imputation network specifically includes:

[0027] Add a multi-head attention mechanism to the residual block of the temporal convolutional network to form a multi-head attention residual block.

[0028] Connect multiple multi-head attention residual blocks in series to form the spatio-temporal feature extraction module.

[0029] Feed the spatio-temporal feature extraction module into the generator of the generative adversarial imputation network to obtain an improved generator.

[0030] Combine the L2 regularization function and the adversarial loss function of the original generator in the generative adversarial imputation network to obtain the loss function of the improved generator, so as to complete the construction of the L2 spatio-temporal generative adversarial imputation network.

[0031] Optionally, the first set condition is that the improved generator and the discriminator reach Nash equilibrium.

[0032] Optionally, the loss function of the improved generator is expressed as:

[0033] L G = αL adv_G + γL 2-norm .

[0034] In the formula, L adv_G represents the adversarial loss function, L 2-norm represents the L2 regularization function, α represents the weight coefficient of the adversarial loss function, γ represents the weight coefficient of the L2 regularization function, and L G represents the loss function of the improved generator.

[0035] Optionally, input the vibration signal imputation data and the current signal imputation data into the data fusion model to obtain fusion data, specifically including:

[0036] Input the vibration signal imputation data and the current signal imputation data into the encoder respectively to obtain vibration data features and current data features.

[0037] Fuse the vibration data features and the current data features to obtain fusion features.

[0038] The decoder restores the dimension of the fusion features to the dimension of the vibration signal imputation data and the current signal imputation data to obtain the fusion data.

[0039] Optionally, construct a fault diagnosis model, specifically including:

[0040] Construct a second test set based on the historical vibration signal imputation data and the historical current signal imputation data.

[0041] Input the second test set into the data fusion model to obtain historical fusion data.

[0042] Construct a third training set and a third test set based on the historical fusion data.

[0043] Construct a convolutional neural network model, and use the third training set and the third test set to train and test the convolutional neural network model respectively. When the third set condition is reached, use the trained convolutional neural network model as the fault diagnosis model.

[0044] In a second aspect, the present application provides a trustworthy mechanical fault diagnosis system based on multi-modal data interpolation and fusion. The trustworthy mechanical fault diagnosis system based on multi-modal data interpolation and fusion is used to implement the steps of the above-mentioned trustworthy mechanical fault diagnosis method based on multi-modal data interpolation and fusion. The system includes:

[0045] A data acquisition module, configured to acquire vibration data and current data of the mechanical equipment to be diagnosed.

[0046] A signal intercepting module, configured to intercept a vibration signal from the vibration data and an electric current signal from the current data according to the operating cycle of the mechanical equipment to be diagnosed.

[0047] An interpolation and fusion model construction module, configured to construct a data interpolation model and a data fusion model. The data interpolation model is a trained L2 spatio-temporal generative adversarial interpolation network. The data used in the training of the L2 spatio-temporal generative adversarial interpolation network is obtained by simulating continuous loss of historical operation data under different mechanical health states. The historical operation data includes historical vibration data and historical current data. The data fusion model is a trained multi-input single-output autoencoder. The multi-input single-output autoencoder includes a plurality of encoders and a decoder.

[0048] A data interpolation module, configured to respectively input the vibration signal and the current signal into the data interpolation model to obtain vibration signal interpolation data and current signal interpolation data.

[0049] A data fusion module, configured to input the vibration signal interpolation data and the current signal interpolation data into the data fusion model to obtain fusion data.

[0050] A fault diagnosis module, configured to construct a fault diagnosis model and input the fusion data into the fault diagnosis model to obtain a fault diagnosis result of the mechanical equipment to be diagnosed.

[0051] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the computer program to implement the steps of the above-mentioned trustworthy mechanical fault diagnosis method based on multi-modal data interpolation and fusion.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned trustworthy mechanical fault diagnosis method based on multi-modal data interpolation and fusion are implemented.

[0053] Fifth aspect, the present application provides a computer program product, including a computer program which, when executed by a processor, implements the steps of the above-mentioned reliable mechanical fault diagnosis method based on multi-modal data interpolation and fusion.

[0054] According to the specific embodiments provided by the present application, the following technical effects are disclosed in the present application:

[0055] The present application provides a reliable mechanical fault diagnosis method, system, device and medium based on multi-modal data interpolation and fusion. By using the trained L2 spatio-temporal generative adversarial interpolation network as the data interpolation model, and the data in the dataset used for training the L2 spatio-temporal generative adversarial interpolation network is obtained by simulating continuous loss of historical operation data under different mechanical health states, considering the long-term dependence relationship of time series data, the problem of being unable to effectively capture the data distribution is solved, and the effective improvement of the quality of interpolated data is realized; by using a data fusion model to perform interpolation data fusion, and using a fault diagnosis model to obtain the fault diagnosis result of the mechanical equipment to be diagnosed based on the fused data, the problems of data redundancy, dimension increase, low calculation efficiency, etc. existing in the existing data fusion methods are solved, and the reliable fault diagnosis of mechanical equipment is realized. Description of the Drawings

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0057] Figure 1 It is an application environment diagram of a reliable mechanical fault diagnosis method based on multi-modal data interpolation and fusion in an embodiment of the present application.

[0058] Figure 2 It is a schematic flowchart of a reliable mechanical fault diagnosis method based on multi-modal data interpolation and fusion provided by an embodiment of the present application.

[0059] Figure 3 It is a schematic implementation flowchart of a reliable mechanical fault diagnosis method based on multi-modal data interpolation and fusion provided by an embodiment of the present application.

[0060] Figure 4 It is an overall structure diagram of L2-TSGAIN provided by another embodiment of the present application.

[0061] Figure 5 It is an overall structure diagram of TSFEM provided by an embodiment of the present application.

[0062] Figure 6This is the overall structure diagram of MISO-AE provided by an embodiment of the present application.

[0063] Figure 7 This is the overall structure diagram of CNN provided by an embodiment of the present application.

[0064] Figure 8 This is the schematic diagram of the overall implementation process of a trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion provided by another embodiment of the present application.

[0065] Figure 9 This is the schematic diagram of the structure of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0066] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0067] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the drawings and specific implementation manners.

[0068] The trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, placed on the cloud or other servers. The terminal 102 can send the vibration data and motor current data of the mechanical equipment to be diagnosed to the server 104. After receiving the vibration data and motor current data of the mechanical equipment to be diagnosed, for the vibration data and motor current data of the mechanical equipment to be diagnosed, the server 104 intercepts the vibration signal from the vibration data and the current signal from the current data according to the operating cycle of the mechanical equipment to be diagnosed. The vibration signal and the current signal are respectively input into the constructed data interpolation model to obtain the vibration signal interpolation data and the current signal interpolation data. The vibration signal interpolation data and the current signal interpolation data are input into the constructed data fusion model to obtain the fusion data. The fusion data is input into the constructed fault diagnosis model to obtain the fault diagnosis result of the mechanical equipment to be diagnosed. The server 104 can feedback the obtained fault diagnosis result of the mechanical equipment to the terminal 102. In addition, in some embodiments, the reliable mechanical fault diagnosis method based on multi-modal data interpolation and fusion can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly perform fault diagnosis on the vibration data and motor current data of the mechanical equipment to be diagnosed, or the server 104 can obtain the vibration data and motor current data of the mechanical equipment to be diagnosed from the data storage system and perform fault diagnosis on the vibration data and motor current data of the mechanical equipment to be diagnosed.

[0069] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0070] In an exemplary embodiment, as Figure 2 shown, a reliable mechanical fault diagnosis method based on multi-modal data interpolation and fusion is provided. This method is executed by a computer device, and can be specifically executed by a computer device such as a terminal or a server alone, or jointly executed by a terminal and a server. In the embodiments of the present application, taking this method applied to Figure 1 the server 104 in

[0071] Step 200: Obtain the vibration data and current data of the mechanical equipment to be diagnosed. For example, obtain the vibration data and current data of the planetary transmission system to be diagnosed.

[0072] Step 201: Intercept the vibration signal from the vibration data and the current signal from the current data according to the operating cycle of the mechanical equipment to be diagnosed.

[0073] Step 202: Construct a data imputation model and a data fusion model. The data imputation model is a trained L2 spatio-temporal generative adversarial imputation network. The data in the dataset used for training the L2 spatio-temporal generative adversarial imputation network is obtained by simulating consecutive losses of historical operation data under different mechanical health states. For example, 10%, 20%, and 30% of the overall sample points of the vibration signal and the current signal are randomly set to 0 to simulate the abnormal sensor data with missing data. In this simulation process, the missing parts of the abnormal data are consecutive missing. The historical operation data includes historical vibration data and historical current data. The data fusion model is a trained multi-input single-output autoencoder. The multi-input single-output autoencoder includes multiple encoders and one decoder.

[0074] Step 203: Input the vibration signal and the current signal into the data imputation model respectively to obtain the imputed data of the vibration signal and the imputed data of the current signal. For example, input the vibration data and current data of the planetary transmission system to be diagnosed into the trained L2-TSGAIN (i.e., the data imputation model) to obtain the imputed vibration data and the imputed current data.

[0075] Step 204: Input the imputed data of the vibration signal and the imputed data of the current signal into the data fusion model to obtain the fused data.

[0076] Step 205: Construct a fault diagnosis model and input the fused data into the fault diagnosis model to obtain the fault diagnosis result of the mechanical equipment to be diagnosed.

[0077] Implementing the above steps 200 to 205, the present application uses the trained L2 spatio-temporal generative adversarial imputation network as the data imputation model. Moreover, the data in the dataset used for training the L2 spatio-temporal generative adversarial imputation network is obtained by simulating consecutive losses of historical operation data under different mechanical health states, considering the long-term dependence relationship of time series data, solving the problem of being unable to effectively capture the data distribution, and effectively improving the quality of the imputed data. By using the data fusion model for imputed data fusion and using the fault diagnosis model to obtain the fault diagnosis result of the mechanical equipment to be diagnosed based on the fused data, the problems of data redundancy, dimension increase, low calculation efficiency, etc. existing in the existing data fusion methods are solved, and the reliable fault diagnosis of the mechanical equipment is realized.

[0078] In another exemplary embodiment of the present application, taking the acquisition of historical vibration data and historical motor current data under different health states of a planetary transmission system as an example, the specific construction processes of the data interpolation model and the data fusion model in this step are described as follows Figure 3 As shown specifically

[0079] (1) Intercept historical vibration signals and historical current signals from the historical vibration data and historical current data respectively according to the operation cycle of the mechanical equipment

[0080] For example, vibration data and current sensor data under different health states are obtained through acceleration sensors and motor current clamps installed on the planetary transmission system, and the vibration data and current sensor data under different health states are intercepted according to the operation cycle of the planetary transmission system to obtain a number of historical vibration signals and historical current signals Among them represents a single historical vibration signal represents a single current signal, i = 1, 2,..., n

[0081] Based on this, a fault data set can be obtained. The fault data set mainly includes vibration signal and current signal data of the planetary transmission system, and these data can reflect different health states of the planetary transmission system. Different health states of the planetary transmission system include normal state, inner race fault of the input shaft bearing, outer race fault of the input shaft bearing, sun gear fault, ring gear fault, etc

[0082] (2) Perform continuous value loss simulation on the historical vibration signals and historical current signals respectively to obtain historical abnormal vibration signals and historical abnormal current signals

[0083] For example, 10%, 20%, and 30% of the overall sample points of the historical vibration signal X v and the historical current signal X c are randomly set to 0 to simulate the abnormal situation of sensor data with continuous data loss

[0084] Based on the descriptions in steps (1) and (2), in this embodiment, abnormal data under different health states is collected by sensors installed on the planetary transmission system

[0085] (3) Construct a first training set and a first test set based on the historical abnormal vibration signals and historical abnormal current signals

[0086] Based on the descriptions in steps (1) and (2) above, the training set and test set of the finally obtained historical abnormal vibration signals are respectively denoted as The training set and test set of the historical abnormal current signals are denoted as Among them Denote a single historical abnormal vibration signal in the training set, Denote a single historical abnormal vibration signal in the test set, Denote a single historical abnormal current signal in the training set, Denote a single historical abnormal current signal in the test set, i = 1, 2, ..., n.

[0087] (4) Construct the L2 Temporal-spatial Generative Adversarial Imputation Network (L2-TSGAIN). The L2 Temporal-spatial Generative Adversarial Imputation Network is a generative adversarial imputation network including an L2 regularization function and a temporal-spatial feature extraction module. Among them, the structure of the L2 Temporal-spatial Generative Adversarial Imputation Network is as Figure 4 shown. The generator consists of two one-dimensional convolutional layers, a temporal-spatial feature extraction module, and two fully connected layers. The discriminator consists of two convolutional layers, two adaptive average pooling layers, and two fully connected layers.

[0088] In the actual application process, the construction process of L2-TSGAIN can be described as:

[0089] (4-1) Introduce the Temporal Convolutional Network (TCN) to focus on the long-term dependence relationship of time series data. The expression of the dilated causal convolution in TCN is:

[0090]

[0091] where w[j] represents the weight of the kernel, x[i + j·d] is the input sequence, d is the dilation factor controlling the kernel spacing, y[i] represents the feature extracted by the convolutional layer, j represents the position encoding of the feature extracted by the convolutional kernel once, j = 1, 2, ..., k, k represents the convolutional kernel size, i represents the position encoding of the input sequence, i = 1, 2, ..., n, and n represents the total number of data.

[0092] By increasing the dilation factor d, L2-TSGAIN can capture the more distant dependence relationship of the data.

[0093] (4-2) At the same time, introduce the Multi-head Attention Mechanism (MHA) to consider the spatial information between each sample point. For the single-head attention mechanism, there is:

[0094]

[0095] In the formula, Q, K, and V represent the query matrix, key matrix, and value matrix of the input sequence respectively. represents the scaling factor, and Softmax(*) is the activation function. Attention(*) represents the data features extracted by the single-head attention mechanism.

[0096] For the multi-head attention mechanism, we have:

[0097] MHA(x) = Concat(head1, head2,..., head h )W O .

[0098] In the formula, head p represents the head, p represents the index of the head, p = 1, 2,..., h, where h represents the total number of indices, and W Q , W K and W V are the weight matrices of Q, K, and V respectively, represents the weight matrix of the query vector of the p-th head, represents the weight matrix of the key vector of the p-th head, represents the weight matrix of the value vector of the p-th head, and W O is the linear projection of the multi-head concatenation. MHA(x) represents the global features extracted by the multi-head attention mechanism, and Concat(*) represents the concatenation operation.

[0099] (4-3) Add MHA to the residual block of TCN to form a multi-head attention residual block (MHAR). Connect three MHARs in series to form a temporal-spatial feature extraction module (TSFEM), and feed it into the generator of the adversarial imputation network (GAIN). Among them, the specific structure of TSFEM is as Figure 5 shown.

[0100] (4-4) Combine the L2 regularization function with the adversarial loss function of the generator to set the improved generator loss function, and we have:

[0101] L G = αL adv_G + γL 2-norm .

[0102]

[0103] In the formula, L adv_G represents the adversarial loss function, and L 2-normThe L2 regularization function is denoted as, α represents the weight coefficient of the adversarial loss function, γ represents the weight coefficient of the L2 regularization function, and L G represents the loss function of the improved generator, D is the discriminator, is the imputed data, and w i is the weight coefficient of the generator, and λ is the weight coefficient used to control the regularization strength.

[0104] (5) Train the L2 spatio-temporal generative adversarial imputation network using the first training set. When the first set condition is reached, use the trained L2 spatio-temporal generative adversarial imputation network as the data imputation model.

[0105] After the L2-TSGAIN is constructed, use the abnormal vibration signal or abnormal current signal as the input and the imputed data as the output, and use the training set and the training set to train the L2-TSGAIN respectively to obtain the trained L2-TSGAIN model.

[0106] In the actual training process, for the generator, first define the mask vector M to describe whether the data at each time point in the original data is missing, and its shape is the same as the missing data matrix. Among them, the elements of the mask vector M are expressed as:

[0107]

[0108] where M ij is the element of the mask vector M, is the component in the abnormal data matrix, N represents the sample length, and L represents the number of samples.

[0109] Secondly, define the noise vector Z to follow a Gaussian distribution, and the shapes of M, Z and the abnormal data are the same.

[0110] Then, input M and Z into the generator of the L2-TSGAIN, and the generator outputs the synthetic data

[0111] In the formula, G represents the generator, and ⊙ represents element-wise multiplication.

[0112] Finally, the missing part of the abnormal data is replaced by the sample points at the same positions in the synthetic data to obtain the imputed data

[0113]

[0114] For the discriminator, define the random variable B = (B1, B2,..., Bn ) ∈ {0, 1} n The hint mechanism H is defined as:

[0115] H = B ⊙ M + 0.5(1 - B).

[0116] In the formula, B n represents a random number, i = 1, 2,..., n.

[0117] The input of the discriminator is and H, and the output is the estimated matrix P, which represents the probability that each component of the imputed data is predicted to be true, expressed as

[0118] After adversarial training, the generator and the discriminator reach the Nash equilibrium, that is, the losses of the generator and the discriminator finally converge to a certain value. At this time, it can be considered that the generator has fully learned the data distribution of the original samples and obtained good data imputation effects. Save the parameters of L2 - TSGAIN for subsequent acquisition of imputed data.

[0119] (6) Input the first test set into the data imputation model to obtain the imputed data of the historical vibration signal and the imputed data of the historical current signal. That is, input the test set and the test set into L2 - TSGAIN to obtain the historical imputed vibration signals and historical imputed current signals in different health states.

[0120] In the actual application process, the imputed data obtained by inputting the training set and into the trained L2 - TSGAIN is and The imputed data obtained by inputting the test set and into the trained L2 - TSGAIN is and

[0121] (7) Construct a second training set based on the imputed data of the historical vibration signal and the imputed data of the historical current signal.

[0122] Based on the above description, the imputed data for data fusion is the imputed data obtained in step (6) and Divide the imputed data and to obtain the training data and the test data and

[0123] (8) Construct a Multi-input Single-output Autoencoder (MISO-AE). For example, the MISO-AE has two encoders and one decoder, and the two encoders are respectively used to receive vibration and current data. The specific structure of the MISO-AE is as Figure 6 shown, and each encoder is composed of two fully connected layers, and the decoder structure is the same as that of the encoder.

[0124] (9) Train the multi-input single-output autoencoder with the second training set until the second set condition is reached, and then use the trained multi-input single-output autoencoder as the data fusion model. Among them, use the training data set and to train the MISO-AE to obtain the trained MISO-AE.

[0125] Based on the above description, in the actual application process, the process of inputting the vibration signal interpolation data and the current signal interpolation data into the data fusion model to obtain the fusion data can be described as follows:

[0126] (1) The vibration signal interpolation data and the current signal interpolation data are respectively input into the encoder to obtain the vibration data features and the current data features. For example, based on the autoencoder, two encoders are set to extract the features of the vibration data and the current data respectively, that is, there is F q =ψ(X q ). Among them, ψ(·) represents the encoding process of the MISO-AE, and F q represents the deep features extracted by the q-th encoder, where q is the number of encoders, and q = 1, 2.

[0127] (2) Fuse the vibration data features and the current data features to obtain the fusion features.

[0128] For example, fuse the deep features of different modal data extracted, and the fusion processing formula is expressed as:

[0129] F = Concat(F1, F2).

[0130] Among them, F represents the fusion features, Concat(*) represents the concatenation operation, F1 represents the deep features extracted by the first encoder, and F2 represents the deep features extracted by the second encoder.

[0131] (3) The decoder restores the dimension of the fusion features to the dimension of the vibration signal interpolation data and the current signal interpolation data to obtain the fusion data.

[0132] For example, the decoder restores the dimension of the fused features to the dimension of the input data to obtain the fused data. The fused data is represented as: X' = ζ(F). Wherein, X′ is the fused data, and ζ(·) represents the decoding process.

[0133] After constructing the trained MISO-AE (i.e., the data fusion model), the test data is input into the MISO-AE to obtain fused samples of different health states. Among them, the fused data obtained by inputting the training data into the trained MISO-AE is The test data is input into the trained MISO-AE, and the fused data obtained is

[0134] In another exemplary embodiment of the present application, after constructing a CNN (Convolutional Neural Network) fault classifier, the training set of the fused data is used to train the CNN fault classifier to obtain a trained CNN fault classifier.

[0135] Specifically:

[0136] (1) Construct a second test set based on the imputed data of the historical vibration signal and the imputed data of the historical current signal.

[0137] (2) Input the second test set into the data fusion model to obtain historical fused data.

[0138] (3) Construct a third training set and a third test set based on the historical fused data. Among them, the data set division in the fault diagnosis stage is to divide the data into a training set and a test set

[0139] (4) Construct a convolutional neural network model (Convolutional Neural Network, CNN), and use the third training set and the third test set to train and test the convolutional neural network model respectively. When the third set condition is reached, the trained convolutional neural network model is used as the fault diagnosis model. For example, the structure of the constructed convolutional neural network model is as Figure 7 composed of four one-dimensional convolutional layers, four max pooling layers and two fully connected layers.

[0140] In the actual application process, first input the training set into the CNN fault classifier for training, and then input the test set into the CNN fault classifier. The fault classifier will analyze the diagnosis result to obtain the test set The different health states included.

[0141] In summary, the reliable mechanical fault diagnosis method based on multi-modal data imputation and fusion provided by this application, from the overall processes of model construction and fault detection as Figure 8 shown, can be generally described as:

[0142] Step 1: Abnormal data acquisition.

[0143] Step 2: Data imputation strategy based on L2-TSGAIN.

[0144] Step 3: Data fusion strategy based on MISO-AE.

[0145] Step 4: Intelligent fault diagnosis based on CNN.

[0146] This application first designs an L2-TSGAIN that introduces TSFEM and L2 regularization function, inputs abnormal data of different modalities into L2-TSGAIN respectively to achieve imputation of abnormal data. Then designs a MISO-AE with a multi-encoder structure, inputs the imputed data of different modalities into MISO-AE to achieve fusion of multi-modal data. Finally constructs a CNN fault classifier, inputs the fused data into CNN to achieve fault diagnosis. This application fully considers the impact of abnormal missing of sensor data on fault diagnosis in industrial scenarios, and based on the L2-TSGAIN data imputation strategy and MISO-AE data fusion strategy, provides a new method for state monitoring and fault diagnosis of mechanical equipment, and provides an important technical support for the safe and reliable operation of mechanical equipment.

[0147] In another exemplary embodiment of the present application, as shown in Table 1, Method 1 in Table 1 indicates that only GAIN is used to impute abnormal data, and then a CNN fault classifier is used to diagnose the imputed data. Method 2 indicates that GAIN introduced with TSFEM is used for data imputation, and then a CNN fault classifier is used to diagnose the imputed data. Method 3 indicates that GAIN introduced with TSFEM and the L2 regularization function is used for data imputation, and then a CNN fault classifier is used to diagnose the imputed data. The proposed method means that the present application uses GAIN introduced with TSFEM and the L2 regularization function for data imputation, then uses MISO-AE for data fusion, and finally uses a CNN fault classifier to diagnose the fused data. In this example, the fault diagnosis results of the diagnosis method based on L2-TSGAIN data imputation and MISO-AE data fusion for the planetary transmission system test data show that the introduced TSFEM, L2 regularization function, and MISO-AE data fusion method can all improve the diagnosis accuracy, and the highest diagnosis accuracy reaches 99.68%. This reflects the effectiveness of the designed L2-TSGAIN data imputation method and MISO-AE data fusion method in improving the diagnosis accuracy.

[0148] Table 1 Results table of missing rates of various methods

[0149]

[0150] As shown in Table 2, KNN in Table 2 represents the K-nearest neighbor imputation method, GAIN represents the generative adversarial imputation network, STA-GAN represents the spatio-temporal attention generative adversarial network, and ESM-GAIN represents the selective multi-generator integrated generative adversarial imputation network. Three quantitative indicators, namely the mean absolute error MAE, mean square error MSE, and root mean square error RMSE, are used to evaluate the effectiveness of different data imputation methods. The comparison results show that the method proposed in the present application achieves the best data imputation effect under the condition of missing sensor data.

[0151] Table 2 Data imputation effect table of various methods

[0152]

[0153] As shown in Table 3, the diagnosis results of the fault diagnosis method based on L2-TSGAIN data imputation and MISO-AE data fusion for the planetary transmission system test data show that, compared with traditional methods and existing advanced methods, the proposed method has the best results in four indicators: accuracy, precision, recall, and F1 score, and the highest accuracy reaches 99.68%. Under three missing ratios of 10%, 20%, and 30%, this method can still achieve a diagnosis accuracy of more than 95%. This reflects the effectiveness of the multi-modal data imputation and fusion method for trustworthy mechanical fault diagnosis in the present application.

[0154] Table 3 Diagnostic Results Table of Planetary Transmission System Test Data

[0155]

[0156] Based on the above description, compared with the prior art, the method provided by this application has the following advantages.

[0157] 1. The multi-modal data imputation and fusion method proposed in this application can extract the features of abnormal data in time and space and generate imputed data, and can fuse the imputed data of different modalities, providing an intelligent fault diagnosis method in the case of data missing.

[0158] 2. The data imputation method based on L2-TSGAIN designed in this application constructs a TSFEM by introducing TCN and MHA to extract data features from the time and space perspectives, and at the same time integrates the L2 regularization function into the generator loss function to avoid overfitting. The designed L2-TSGAIN can fully learn the abnormal data sample distribution and realize data imputation.

[0159] 3. The data fusion method based on MISO-AE designed in this application realizes data fusion by deploying multiple encoders to extract the features of different modality data respectively.

[0160] 4. This application can perform condition monitoring and fault diagnosis on mechanical equipment in the case of data missing, can learn the distribution of abnormal data and impute the abnormal data, can comprehensively utilize the imputed data of different modalities to provide more powerful data support for fault diagnosis, and can provide technical support for the condition monitoring, fault diagnosis and health management of many mechanical equipment.

[0161] Furthermore, based on the same inventive concept, the embodiment of this application also provides a reliable mechanical fault diagnosis system for implementing the above-mentioned multi-modal data imputation and fusion. The implementation solution provided by this system to solve the problem is similar to the implementation solution recorded in the above method. Therefore, the specific limitations in one or more embodiments of the reliable mechanical fault diagnosis system based on multi-modal data imputation and fusion provided below can refer to the limitations on the reliable mechanical fault diagnosis method based on multi-modal data imputation and fusion in the above text, and will not be repeated here.

[0162] In an exemplary embodiment, a reliable mechanical fault diagnosis system based on multi-modal data imputation and fusion provided by this application includes:

[0163] A data acquisition module, configured to acquire vibration data and current data of the mechanical equipment to be diagnosed.

[0164] A signal truncation module, which is used to truncate vibration signals from vibration data and current signals from current data according to the operating cycle of the mechanical equipment to be diagnosed.

[0165] An interpolation fusion model construction module, which is used to construct a data interpolation model and a data fusion model. The data interpolation model is a trained L2 spatio-temporal generative adversarial interpolation network. The data in the dataset used for training the L2 spatio-temporal generative adversarial interpolation network is obtained by simulating continuous loss of historical operation data under different mechanical health states. The historical operation data includes historical vibration data and historical current data. The data fusion model is a trained multi-input single-output autoencoder. The multi-input single-output autoencoder includes multiple encoders and one decoder.

[0166] A data interpolation module, which is used to input the vibration signal and the current signal into the data interpolation model respectively to obtain vibration signal interpolation data and current signal interpolation data.

[0167] A data fusion module, which is used to input the vibration signal interpolation data and the current signal interpolation data into the data fusion model to obtain fusion data.

[0168] A fault diagnosis module, which is used to construct a fault diagnosis model and input the fusion data into the fault diagnosis model to obtain the fault diagnosis result of the mechanical equipment to be diagnosed. [[ID=1q]]

[0169] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as Figure 9 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video tag processing data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it realizes a trusted mechanical fault diagnosis method based on multi-modal data interpolation and fusion.

[0170] Those skilled in the art can understand, Figure 9The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0171] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0172] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0174] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0175] The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logics, data processing logics based on quantum computing, etc., without limitation.

[0176] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0177] Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion, characterized in that, The reliable mechanical fault diagnosis method based on multimodal data imputation and fusion includes: Obtain the vibration data and current data of the mechanical equipment to be diagnosed; Intercept vibration signals from the vibration data and current signals from the current data according to the operating cycle of the mechanical equipment to be diagnosed; Construct a data imputation model and a data fusion model; the data imputation model is a trained L2 spatio-temporal generative adversarial imputation network; the data in the dataset used for training the L2 spatio-temporal generative adversarial imputation network is obtained by simulating the loss of continuous values from historical operation data under different mechanical health states; the historical operation data includes historical vibration data and historical current data; the data fusion model is a trained multi-input single-output autoencoder; the multi-input single-output autoencoder includes multiple encoders and one decoder; the L2 spatio-temporal generative adversarial imputation network is a generative adversarial imputation network including an L2 regularization function and a spatio-temporal feature extraction module; Input the vibration signal and the current signal into the data imputation model respectively to obtain vibration signal imputation data and current signal imputation data; Input the vibration signal imputation data and the current signal imputation data into the data fusion model to obtain fusion data; Construct a fault diagnosis model and input the fusion data into the fault diagnosis model to obtain the fault diagnosis result of the mechanical equipment to be diagnosed.

2. The trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion according to claim 1, wherein Constructing the data imputation model and the data fusion model includes: Intercept historical vibration signals and historical current signals from the historical vibration data and the historical current data respectively according to the operating cycle of the mechanical equipment; Simulate the loss of continuous values for the historical vibration signal and the historical current signal respectively to obtain historical abnormal vibration signals and historical abnormal current signals; Construct a first training set and a first test set based on the historical abnormal vibration signals and the historical abnormal current signals; Construct an L2 spatio-temporal generative adversarial imputation network; Train the L2 spatio-temporal generative adversarial imputation network with the first training set until a first set condition is reached, and use the trained L2 spatio-temporal generative adversarial imputation network as the data imputation model; Input the first test set into the data imputation model to obtain historical vibration signal imputation data and historical current signal imputation data; Construct a second training set based on the historical vibration signal imputation data and the historical current signal imputation data; Construct a multi-input single-output autoencoder; Train the multi-input single-output autoencoder with the second training set until a second set condition is reached, and use the trained multi-input single-output autoencoder as the data fusion model.

3. The trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion according to claim 2, wherein, Constructing the L2 spatio-temporal generative adversarial imputation network specifically includes: Add a multi-head attention mechanism to the residual block of the temporal convolutional network to form a multi-head attention residual block; Connect multiple multi-head attention residual blocks in series to form the spatio-temporal feature extraction module; Feed the spatio-temporal feature extraction module into the generator of the generative adversarial imputation network to obtain an improved generator; Merge the L2 regularization function and the adversarial loss function of the original generator in the generative adversarial imputation network to obtain the loss function of the improved generator, so as to complete the construction of the L2 spatio-temporal generative adversarial imputation network.

4. The trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion according to claim 3, characterized in that, The first set condition is that the improved generator and the discriminator reach Nash equilibrium.

5. The reliable mechanical fault diagnosis method based on multi-modal data imputation and fusion according to claim 3, characterized in that The loss function of the improved generator is expressed as: L G = αL adv_G + γL 2-norm ; Where, L adv_G represents the adversarial loss function, L 2-norm represents the L2 regularization function, α represents the weight coefficient of the adversarial loss function, γ represents the weight coefficient of the L2 regularization function, and L G represents the loss function of the improved generator.

6. The trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion according to claim 1, wherein Input the vibration signal imputation data and the current signal imputation data into the data fusion model to obtain fusion data, specifically including: The vibration signal imputation data and the current signal imputation data are respectively input into the encoder to obtain vibration data features and current data features; Fuse the vibration data features and the current data features to obtain fusion features; The decoder restores the dimension of the fusion features to the dimensions of the vibration signal imputation data and the current signal imputation data to obtain the fusion data.

7. The trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion according to claim 2, characterized in that, Construct a fault diagnosis model, specifically including: Construct a second test set based on the historical vibration signal imputation data and the historical current signal imputation data; Input the second test set into the data fusion model to obtain historical fusion data; Construct a third training set and a third test set based on the historical fusion data; Construct a convolutional neural network model, and use the third training set and the third test set to train and test the convolutional neural network model respectively. When the third set condition is reached, the trained convolutional neural network model is used as the fault diagnosis model.

8. A trustworthy mechanical fault diagnosis system based on multi-modal data imputation and fusion, characterized in that, The trustworthy mechanical fault diagnosis system based on multi-modal data imputation and fusion is used to implement the trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion as described in any one of claims 1-7; the system includes: A data acquisition module for acquiring vibration data and current data of the mechanical equipment to be diagnosed; A signal intercepting module for intercepting a vibration signal from the vibration data and a current signal from the current data according to the operating cycle of the mechanical equipment to be diagnosed; An imputation fusion model construction module for constructing a data imputation model and a data fusion model; the data imputation model is a trained L2 spatio-temporal generative adversarial imputation network; the data in the dataset used for training the L2 spatio-temporal generative adversarial imputation network is obtained by simulating the loss of continuous values from the historical operation data under different mechanical health states; the historical operation data includes historical vibration data and historical current data; the data fusion model is a trained multi-input single-output autoencoder; the multi-input single-output autoencoder includes multiple encoders and one decoder; the L2 spatio-temporal generative adversarial imputation network is a generative adversarial imputation network including an L2 regularization function and a spatio-temporal feature extraction module; A data imputation module for respectively inputting the vibration signal and the current signal into the data imputation model to obtain vibration signal imputation data and current signal imputation data; A data fusion module for inputting the vibration signal imputation data and the current signal imputation data into the data fusion model to obtain fusion data; A fault diagnosis module for constructing a fault diagnosis model and inputting the fusion data into the fault diagnosis model to obtain a fault diagnosis result of the mechanical equipment to be diagnosed.

9. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the trustworthy mechanical fault diagnosis method based on multi-modal data imputation and fusion according to any one of claims 1-7.

Images