Multimodal data fusion method for fault diagnosis of industrial Internet equipment

Through the multimodal data fusion method, using technologies such as MvDA and CNN, the problem of low fault diagnosis accuracy of a single sensor in complex environments is solved, and the accuracy and robustness of fault diagnosis are achieved. It is suitable for industrial Internet equipment Troubleshooting.

CN114707551BActive Publication Date: 2024-11-19NORTHWEST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210354875.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-06
Publication Date
2024-11-19
Estimated Expiration
2042-04-06

AI Technical Summary

Technical Problem

When existing industrial IoT devices use a single sensor for fault diagnosis in complex industrial environments, they are susceptible to external interference and cause failure, resulting in low diagnostic accuracy and weak robustness.

Method used

Multimodal data fusion method is adopted to establish a multimodal data fusion model through multi-view discriminant analysis (MvDA), and solve it using Ratio Trace iterative algorithm, combining wavelet image features and convolutional neural network (CNN) to extract sensor data features and through PCA reduces dimensionality, projecting into public spaces for classification and prediction.

Benefits of technology

It improves the accuracy and robustness of fault diagnosis, enhances the equipment fault diagnosis capabilities in complex industrial environments, reduces dependence on a single sensor, and improves the stability and reliability of the diagnostic system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114707551B_ABST
    Figure CN114707551B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal data fusion method for fault diagnosis of industrial Internet equipment. The method is based on the equipment fault data set, and judges the fault of the equipment by fusing two types of sensor data into a common space, and uses different signal processing methods to process sensor data with different signal stability. This method can not only improve the accuracy of fault diagnosis, but also enhance the robustness of the system after a certain sensor fails. Its innovation lies in: 1) A multimodal data fusion algorithm based on multi-view discriminant analysis (MvDA) is proposed. An iterative algorithm is used in the algorithm to replace the singular value decomposition part in the original MvDA to obtain the global optimal solution, thereby improving the projection accuracy and diagnostic robustness. 2) For stationary signals, a 1DCNN algorithm that can quickly extract features is used, and for non-stationary signals, wavelet transform two-dimensional images are used for feature extraction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial Internet, relates to a method for predicting faults of large equipment, and in particular to a multimodal data fusion method for fault diagnosis of industrial Internet equipment. Background Art

[0002] In the actual industrial Internet, it is very common to use sensor data to monitor the status of equipment. For example, using sensors such as humidity can obtain the current working environment of the equipment. These sensors of different types can usually describe the operation of the same equipment from different angles. If the condition of the equipment can be diagnosed by combining sensors from multiple angles, the diagnostic accuracy of the equipment can be improved, which will have great value for the maintenance of industrial equipment. Most of the existing industrial Internet of Things devices use a single sensor to judge the use status of the equipment, but in the complex and changeable actual industrial environment, a single sensor may fail due to some external interference. In actual industrial production, multiple sensors are often deployed to monitor equipment. Combining multiple sensor data can not only reduce the risk of diagnostic failure due to the failure of a single sensor, but also integrate the judgment of each sensor, improve the diagnostic accuracy, and reduce the negative impact of relying on a single sensor on the fault diagnosis of large equipment. Summary of the invention

[0003] In view of the technical problems existing in existing detection methods such as low data utilization, weak robustness and low accuracy, the purpose of the present invention is to provide a multimodal data fusion method for industrial Internet equipment fault diagnosis.

[0004] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0005] A multimodal data fusion method for industrial Internet equipment fault diagnosis, characterized by comprising the following steps:

[0006] Step 1: collect the fault data of large equipment, perform data preprocessing, and use it as training samples;

[0007] Step 2: for the large equipment fault training samples obtained after sorting and classification in step 1, multimodal data fusion is performed from different angles for the data collected by sensors with different data types or characteristics, and the feature matrix and principal component matrix of different sensor data are extracted;

[0008] Step 3: For all sensors, a multimodal data fusion model is established by using a multimodal data fusion method. First, a mapping matrix for solving the multimodal data fusion model is defined, and then an iterative algorithm based on the Ratio Trace method is used to obtain the global optimal solution of the multimodal data fusion model.

[0009] In step 4, the multimodal data fusion model obtained in step 3 is used to project the sensor features into a new space and classify and predict the data using a classifier.

[0010] Compared with the prior art, the multimodal data fusion method for industrial Internet equipment fault diagnosis of the present invention has the following technical effects:

[0011] (I) A multimodal data fusion equipment fault diagnosis model was established based on MvDA, and an iterative algorithm based on the Ratio Trace method was used to solve the trace error problem. The multimodal data fusion equipment fault diagnosis model can refine the sensor data feature processing, thereby better diagnosing faults;

[0012] (II) The method of learning wavelet image features replaces the original method of directly learning features of non-stationary signals, which improves the fusion accuracy. Experimental results show that this model not only improves the accuracy after fusion, but also improves the robustness of the fault diagnosis model compared with other existing fault diagnosis algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is the sampling method when segmenting the vibration signal.

[0014] Figure 2 This is the wavelet transform time-frequency diagram of the fan end fault.

[0015] Figure 3 The training time and accuracy corresponding to different types of CNNs and different types of data.

[0016] Figure 4 This is the time-frequency domain signal diagram of the outer race fault at the drive end without adding noise.

[0017] Figure 5 Add noise to the time-frequency domain signal of the outer race fault at the drive end.

[0018] Figure 6 A comparison of commonly used public space projection methods.

[0019] Figure 7 The following is a comparison chart of the accuracy of using PCA and not using PCA.

[0020] Figure 8 It is a system model of multimodal data fusion equipment fault detection model. DETAILED DESCRIPTION

[0021] Most of the existing large-scale industrial IoT equipment use a single sensor to judge the use status of the equipment. However, in the complex and changeable actual industrial environment, a single sensor may fail due to some external interference, which will affect the fault detection system of the large equipment and even cause it to fail. Therefore, based on the idea of ​​multimodal data fusion, the applicant explores a multimodal data fusion method for fault diagnosis of industrial Internet equipment.

[0022] The applicant established a multi-modal data fusion equipment fault detection model based on multi-view discriminant analysis (MvDA). By analyzing the characteristics of different sensor data, the optimal features for fault diagnosis of each sensor were obtained. The different feature data were mapped on the same plane through the fusion model, and the iterative algorithm of the Ratio trace method was used to obtain the spatial optimal solution. The fused model was used to diagnose the equipment fault condition.

[0023] It should be noted that, in the following embodiments, all algorithms involved, unless otherwise specified, all adopt algorithms known in the prior art.

[0024] In the following examples:

[0025] SVM algorithm refers to support vector machine algorithm.

[0026] The SDIAE algorithm refers to an automatic encoder based on stacked discriminant information (English: Stacked Discriminant Information-Based Auto-Encoder).

[0027] The VAEGAN-DRA algorithm refers to the Variational Autoencoding Generative Adversarial Networks With Deep RegretAnalysis.

[0028] SNN algorithm refers to the spiking neural network (English: Spiking Neural Network).

[0029] The SIRCNN algorithm refers to the Stacked Inverted Residual Convolution Neural Network (English: Stacked Inverted Residual Convolution Neural Network).

[0030] The following are specific embodiments given by the inventor.

[0031] This embodiment provides a multimodal data fusion method for industrial Internet equipment fault diagnosis, including the following steps:

[0032] Step 1: collect the fault data of large equipment, perform data preprocessing, and use it as training samples;

[0033] The specific implementation of step 1 includes the following sub-steps:

[0034] Step 1.1, based on the collected large equipment failure samples, classify the failure data and mark each type of failure data;

[0035] Specifically in this embodiment, the experiment uses the bearing data set of the University of the Western Reserve in the United States, which has been widely used for bearing model fault research. The bearing data platform is mainly composed of a motor, a torque sensor, a power detector, an electronic control valve, and the type of bearing to be tested. This experiment uses the drive end bearing acceleration vibration sensor data and the fan end bearing acceleration vibration sensor data. The sampling frequency is 12kHz. The failure type of the transmission bearing can be divided into: inner ring failure, outer ring failure, and rotating body failure according to the components. The damage of this data set is a single-point damage caused by spark machining. Since the wear part is relatively fixed, the damage point has a direct effect on the vibration response of the bearing system compared to the different positions of the bearing load zone. The fault damage length is 0.1778, 0.3556, and 0.5334 mm. Each fault type will produce the above three sizes of faults, so each fault will have three different damage conditions. The bearing will produce a total of 10 different conditions, including normal state and nine fault states.

[0036] Step 1.2, for the vibration signal data, a resampling method is selected to expand the number of samples in the data set, and after obtaining the expanded data set, the data of each fault sample is normalized;

[0037] Specifically in this embodiment, each sample has 121556 data, including more periodic data, and the acquisition window is set to 1024 data. This experiment uses resampling to expand the data, and the step length between the next sample and the previous sample is kept at 256. Figure 1 As shown. Using the above window size and step size can greatly expand the number of samples of different types of faults. Each sample data contains 2-3 cycles of data of one type of fault. After obtaining the expanded data set, due to the need to use the convolutional network, the data of each fault sample needs to be normalized. After obtaining the expanded data, each fault type is randomly selected, and there are 400 data of each type. The ratio of the training set to the test set is 7:3.

[0038] Step 1.3, for the expanded data set obtained in step 1.2, adding noise to the sensor data to simulate the behavior of sensor failure, the noise adding method used in the present invention is additive Gaussian noise.

[0039] Specifically in this embodiment, noise is added to the driver sensor data to simulate the behavior of sensor failure. The noise addition method used in this experiment is additive Gaussian noise. The size of the noise is usually measured by the signal-to-noise ratio. The signal-to-noise ratio formula is as follows:

[0040]

[0041] Where: P noise is the noise power; P signal is the useful power of the signal; is the variance of each sample, which is used to approximately represent the power of the signal. For standard Gaussian noise, the signal power is 1;

[0042] The noise to be added to each sample signal is:

[0043]

[0044] Where randn() is the function for generating standard noise, and length() is the length of the signal;

[0045] It should be noted that, for the convenience of experimental comparison, all the noises applied in this embodiment are at the driver end. The signal at the fan end is more unstable, which will have a greater impact on the results, while the excellent data at the driver end makes the diagnosis success rate very high, so the data with noise applied at the driver end can better illustrate the robustness of the method described in this embodiment.

[0046] Advantages of step 1: This step expands the number of samples in the original data set, meeting the data volume requirements of the model. The expanded data set is completely taken from the original data set, and no additional irrelevant information is introduced. The normalized data set is more suitable for learning and training. Adding Gaussian noise can simulate sensor failure, which helps improve the generalization and robustness of the model.

[0047] Step 2: for the large equipment fault training samples obtained after sorting and classification in step 1, multimodal data fusion is performed from different angles for the data collected by sensors with different data types or characteristics, and the feature matrix and principal component matrix of different sensor data are extracted;

[0048] The specific implementation of step 2 includes the following sub-steps:

[0049] Step 2.1: For the expanded data set obtained in step 1.3, wavelet analysis is used to reduce the noise of the signal whose characteristics show strong instability due to noise, and then it is converted into a wavelet time-frequency graph for learning. For the two-dimensional image of the wavelet time-frequency graph, the following convolutional network structure is used for learning:

[0050] k. Image input, parameters are set to: 64×64×3 image, 'zerocenter' normalization;

[0051] l. Convolution layer, the parameters are set as follows: number of convolution kernels is 8, convolution kernel size is 5×5, stride is [1,1], and t padding is “same”;

[0052] m. Batch processing layer, the parameters are set to: batch normalization;

[0053] n. Activation function layer, the parameters are set to: use ReLU activation function;

[0054] o. Pooling layer, the parameters are set as follows: pooling mode is "max pooling", pooling domain size is 2×2, stride is [2, 2], and t is filled with "0";

[0055] p. Convolution layer, the parameters are set as follows: the number of convolution kernels is 16, the convolution kernel size is 5×5, the stride is [1,1], and the t padding is “same”;

[0056] q. Batch processing layer, the parameters are set to: batch normalization;

[0057] r. Activation function layer, the parameters are set to: use ReLU activation function;

[0058] s. Pooling layer, the parameters are set as follows: pooling mode is "max pooling", pooling domain size is 2×2, stride is [2, 2], and t is filled with "0";

[0059] t. Fully connected layer, the parameters are set to: 4096 fully connected layers;

[0060] Through the CNN architecture, the feature matrix of the sensor is extracted as X ijk =(x ijk1 ,x ijk2 ,...,x ijkz ), where z is the number of features extracted;

[0061] Step 2.2: For one-dimensional signals with good quality, the convolutional neural network is directly applied for learning. The structure of the convolutional neural network is as follows:

[0062] g. Vibration input, parameter settings: 1×1024, 'zerocenter' normalization;

[0063] h. Convolution layer, the parameters are set as follows: number of convolution kernels is 8, convolution kernel size is 1×1024, stride is [1,1], and t padding is “same”;

[0064] i. Pooling layer, the parameters are set as follows: pooling mode is "max pooling", pooling domain size is 1×512, stride is [2, 2], and t is filled with "0";

[0065] j. Convolution layer, the parameters are set as follows: the number of convolution kernels is 16, the convolution kernel size is 1×512, the stride is [1,1], and the t padding is “same”;

[0066] k. Pooling layer, the parameters are set as follows: pooling mode is "max pooling", pooling domain size is 1×512, stride is [2, 2], and t is filled with "0";

[0067] l. Fully connected layer, the parameters are set to: 4096 fully connected layers;

[0068] Through the CNN architecture, the feature matrix of the sensor is extracted as X ijk =(x ijk1 ,x ijk2 ,...,x ijkz ), where z is the number of features extracted;

[0069] Step 2.3, use principal component analysis (PCA) to process each sensor matrix. This method projects the data feature matrix onto a set of mutually orthogonal matrices, then calculates the variance between the projected data, and evaluates the importance of the feature matrix based on the variance. For sensor feature data, there are:

[0070] X ijk =[x ij k1 ... x ij k z ] (Formula 3)

[0071] Where: X ijk is a matrix composed of sensor features. Next, solve X ijk The standardized matrix is ​​denoted as

[0072]

[0073] In the formula: M and Y are two variables; K is the total number of samples; is the mean value of M; is the mean of Y;

[0074] calculate The eigenvectors of all eigenvectors form a matrix XC , construct the principal component matrix.

[0075] Specifically in this embodiment, the average accuracy of the driver-end data on the one-dimensional convolutional neural network is 94.48% under the one-dimensional convolutional network architecture, and the average accuracy under the two-dimensional convolutional network architecture is 97.14%; while the average accuracy learned by the fan end is 84.01% and 86.67% respectively. The diagnostic performance of the fan-end time-frequency signal on the convolutional network is not obvious. The reason is that the recognition of the fan-end time-frequency signal in the low-frequency band in the data set does not have an advantage, and the signal in the high-frequency band is not obvious. For this reason, a method that can analyze the fan-end frequency signal is selected. There is a huge difference in the feature level between the signals obtained by the short-time Fourier transform and the wavelet transform. The Fourier transform is difficult to adapt to the problem of rapid and slow changes in the high and low frequency bands, and the wavelet transform has great advantages in processing high and low frequency band transformations. In order to solve the problem of low diagnostic accuracy of the fan-end signal image, the fan-end data is processed using wavelet transform to convert it into a wavelet image with prominent high-frequency features. The converted wavelet image is as follows Figure 2 As shown, from left to right and from top to bottom are respectively normal state, inner ring fault damage length 0.1778mm, inner ring fault damage length 0.3556mm, inner ring fault damage length 0.5334mm, rolling element fault damage length 0.1778mm, rolling element fault damage length 0.3556mm, rolling element fault damage length 0.5334mm, outer ring fault damage length 0.1778mm, outer ring fault damage length 0.3556mm, outer ring fault damage length 0.5334mm.

[0076] Therefore, different processing methods are used for different data. The resolution of the wavelet transform frequency map of the compressor fan end fault is 64×64, and the wavelet image is trained using the two-dimensional convolutional network described in step 2.1. The one-dimensional convolutional network structure described in step 2.2 is adopted, and the drive end data is input into the one-dimensional convolutional structure for feature learning. The bearing fault is classified using a fully connected layer of size 10 through the Softmax function.

[0077] like Figure 3 As shown, the advantages of step 2 are: for the fan-end data, the method of converting the wavelet time-frequency diagram can fully highlight the high-frequency fault characteristics when the rolling bearing fault occurs. The quality of the drive-end data is high, and the use of a two-dimensional convolutional network cannot bring about a significant improvement in accuracy. Therefore, in this embodiment, a one-dimensional convolution is used to extract features from the drive-end data. Step 2 better balances the training time and accuracy, and improves the usability of the method in a real industrial environment.

[0078] Step 3: For all sensors, a multimodal data fusion model is established by a multimodal data fusion method. First, a mapping matrix for solving the multimodal data fusion model is defined, and then an iterative algorithm based on the Ratio Trace method is used to obtain the global optimal solution of the multimodal data fusion model.

[0079] The specific implementation of step 3 includes the following sub-steps:

[0080] Step 3.1, for each sample, define:

[0081]

[0082] Where: D is the data set of equipment failure; d ijk is the kth sample of the jth sensor in the i-th fault; I is the type of fault; J is the total number of sensors; q o is the characteristic dimension of sensor o; k ij is the number of samples of the i-th fault of the j-th sensor;

[0083] It should be noted that, in this embodiment, J = 2;

[0084] Step 3.2: For all sensors, a multimodal data fusion model can be established by multimodal data fusion (MvDA), and the mapping matrix for solving the multimodal data fusion model is defined as:

[0085]

[0086] Where: are the optimal mapping matrices of different sensors; L in is the intra-class scatter matrix of the projection in the common space; L out Between-class scatter matrix;

[0087] The two scatter matrices can be represented as follows:

[0088]

[0089]

[0090] Where: is the average matrix of the i-th type fault samples in all sensors; is the number of samples of all sensors of the i-th fault; is the average matrix of all samples in the public space; K is the number of all fault samples, F ijk is d ijk The sample values ​​corresponding to the samples mapped to the public space;

[0091] For the convenience of calculation, the inter-class scattering matrix L out and the intra-class scattering matrix L in can be written as:

[0092] L out =S T WS (Formula 9)

[0093] L in =S T VS (Equation 10)

[0094] Where: S T is the set of transposed matrices of all sensor mappings, S is the set of all sensor mapping matrices, S = {S1, S2, ..., S J}; W and V are both about The block matrix is ​​defined as follows:

[0095]

[0096]

[0097] Where: is the average matrix of all samples of the i-th fault in the j-th sensor; for The transpose of k ij is the number of samples of the i-th fault of the j-th sensor; is the number of samples of all sensors of the i-th fault; is the average matrix of all samples of the i-th fault in the m-th sensor; for The transpose of k im is the number of samples of the i-th fault of the m-th sensor; is the number of samples of all sensors of the i-th fault;

[0098] Step 3.3, Equation 4 in step 3.2 needs to solve the linear transformation view of J sensors. By converting the objective function into a trace ratio problem, the solution of the problem can be simplified. The trace ratio solution of the objective function of Equation 4 can be written as:

[0099]

[0100] The above objective function can be regarded as a trace ratio problem. It is difficult for singular value decomposition to obtain the global optimal solution of the multimodal data fusion model. In order to solve this problem, an iterative algorithm based on the Ratio Trace method is used to solve the trace ratio problem.

[0101] Step 3.4: First, in order to maintain the solvability of the global structure, add the orthogonal constraint ST S = E. Where E is the identity matrix, and let Then formula 13 can be written as:

[0102]

[0103] It is known that formula 14 satisfies Since the model has processed the sensor data using the PCA principal component analysis method, there is no The space is 0, then the following formula can be used to convert

[0104] In the formula: Θ=[θ1,θ2,θ3,...,θ C ],θ c >0, c=1,2,...,C; C is The number of positive singular values;

[0105] Let S = QB, d is the rank of S, and the objective function can be expressed as:

[0106]

[0107] Among them, W (u) =B T W.B.,

[0108] By singular value decomposition, we consider that in Equation 16, even if there is a closed-form solution, there may still be a large error with the optimal solution;

[0109] Step 3.5, transform equation 16 into a trace difference problem:

[0110]

[0111] where λ n is the projection matrix B from the previous step n-1 The calculated trajectory ratio, the specific algorithm steps are:

[0112] a) B 0 Initialize to an arbitrary column orthogonal matrix;

[0113] b) Starting from n=1, 2, ..., N, by calculating the matrix B n-1 Calculate θ n :

[0114]

[0115] c) The trace difference problem is constructed as:

[0116]

[0117] d) Use eigenvalue decomposition method to solve the trace difference problem:

[0118]

[0119] Where: for The rth largest eigenvalue in ; for The corresponding eigenvector;

[0120] In order to maintain orthogonality, reconstruct the projection matrix, set d is the rank of the low-dimensional feature, let V (b) =B n (B n ) T V (u) B n (B n ) T , perform singular value decomposition, we have:

[0121] V (b) =B n Θ b (B n ) T (Formula 21)

[0122] e) If yes Then exit the loop;

[0123] Output B = B n .

[0124] Specifically in this embodiment, based on the data feature matrix of the driver end and the fan end obtained in step 2, a multimodal data fusion method is adopted to comprehensively consider the data of the driver end and the fan end, and the data of the fan end and the driver end are projected into the public space after PCA dimension reduction and classified using SVM.

[0125] It should be noted that this embodiment tests the influence of different penalty factors (Penalty Factor) on the support vector machine. When the penalty factor is greater than 0.1, it tends to be stable. Therefore, the penalty factors of other performance analysis experiments in this embodiment are set to 0.1.

[0126] Considering that there are more uncertain factors affecting the sensor data in a real industrial environment, in order to verify the robustness of the method proposed in the present invention, standard Gaussian noise is added to the driver data. Figure 4 and Figure 5The time-frequency domain signal diagrams of the outer ring fault data of the drive end before and after adding noise are shown respectively. The experimental results show that the multimodal data fusion model can improve the accuracy of fault diagnosis. And the effect is much better than directly splicing features for fusion classification. The experimental comparison algorithms include algorithms that have achieved good results in bearing fault diagnosis in recent years. Each method was run ten times, and the final accuracy was averaged. The algorithms compared were SDIAE algorithm, VAEGAN-DRA algorithm, SVM algorithm, SNN algorithm and SIRCNN algorithm. The results are shown in Table 1. The data used for multimodal data fusion in Table 1 is the data of the drive end plus the fan end.

[0127] Table 1: Comparison of algorithm accuracy

[0128]

[0129] This embodiment is also compared with the commonly used public space projection methods, all of which use SVM as the classifier. Figure 6 shown.

[0130] It should be noted that PCA was used after each feature extraction to retain more than 95% of the principal components. The purpose of using PCA for dimensionality reduction here is:

[0131] (1) Convert high-dimensional features to low-dimensional features, so it is necessary to reduce the dimensionality of the high-dimensional data to simplify the data and eliminate unnecessary redundant information;

[0132] (2) Speed ​​up the plane projection and shorten the operation time.

[0133] Figure 7 The impact of adding a PCA dimensionality reduction step to the two-end dataset is shown. After PCA dimensionality reduction, the multi-sensor fusion framework method has a lower time complexity, usually achieving the best results in 3-4 iterations.

[0134] Step 4: Use the multimodal data fusion model obtained in step 3 to project the sensor features into a new space and use the classifier to classify and predict the data. Figure 8 shown.

[0135] The specific implementation of step 4 includes the following sub-steps:

[0136] Step 4.1, the multimodal data fusion model projects the sensor features into a space, and then uses SVM (support vector machine) as a classifier;

[0137] Step 4.2, select different kernel functions for the SVM classifier in step 4.1, where:

[0138] a) Linear kernels are generally used in linearly separable situations. They have fewer parameters and faster speeds, so they have excellent performance for linearly separable data.

[0139] b) Polynomial kernel function can directly reflect low-dimensional input gaps to higher-dimensional feature gaps. However, if the polynomial kernel function has too many parameters, the complexity of the calculation will tend to infinity due to the increase in the number of features.

[0140] c) The Gaussian kernel function (RFB) is a kernel function with strong local representation and a kernel variable with strong local representation, which shows excellent performance in both large and small samples;

[0141] d) The Sigmoid function approximates the support vector machine to a multi-layer neural network;

[0142] In step 4.3, the same processing steps are implemented according to the clustering algorithm to complete the fault type diagnosis task.

[0143] It should be noted that the above embodiments are preferred examples of the present invention, and the present invention is not limited to the above embodiments. Any additions or equivalent changes made to the technical features based on the technical solution of this application shall fall within the protection scope of the present invention.

Claims

1. A multimodal data fusion method for industrial Internet equipment fault diagnosis, characterized in that: The following steps are involved: Step 1: collect the fault data of large equipment, perform data preprocessing, and use it as training samples; Step 2: for the large equipment fault training samples obtained after sorting and classification in step 1, multimodal data fusion is performed from different angles for the data collected by sensors with different data types or characteristics, and the feature matrix and principal component matrix of different sensor data are extracted; Step 3: For all sensors, a multimodal data fusion model is established by a multimodal data fusion method. First, a mapping matrix for solving the multimodal data fusion model is defined, and then an iterative algorithm based on the Ratio Trace method is used to obtain the global optimal solution of the multimodal data fusion model. The specific steps include the following: Step 3.1, for each sample, define: Where, is the data set of equipment failure, d ijk is the kth sample of the jth sensor in the i-th fault, I is the type of fault, J is the total number of sensors, q o is the characteristic dimension of sensor o, k ij is the number of samples of the i-th fault of the j-th sensor; Step 3.2: For all sensors, a multimodal data fusion model can be established by multimodal data fusion MvDA, and the mapping matrix for solving the multimodal data fusion model is defined as: In the formula, They are the optimal mapping matrices of different sensors, L in is the intra-class scatter matrix of the projection in the common space, L out Between-class scatter matrix; The two scatter matrices are expressed by the following equations 5 and 6: In the formula, is the average matrix of the i-th type fault samples in all sensors; is the number of samples of all sensors of the i-th fault; is the average matrix of all samples in the common space; K is the number of all fault samples, F ijk is d ijk The sample values ​​corresponding to the samples mapped to the public space; For the convenience of calculation, the inter-class scattering matrix L out and the intra-class scattering matrix L in is written as: L out =S T WS(Formula 7) L in =S T VS (Formula 8) Where: S T is the set of transposed matrices of all sensor mappings, S is the set of all sensor mapping matrices, S = {S1, S2, ..., S J }; W and V are both about The block matrix is ​​defined as follows: In the formula, is the average matrix of all samples of the i-th fault in the j-th sensor, for The transpose of k ij is the number of samples of the i-th fault of the j-th sensor; is the number of samples of all sensors of the i-th fault; is the average matrix of all samples of the i-th fault in the m-th sensor; for The transpose of k im is the number of samples of the i-th fault of the m-th sensor; is the number of samples of all sensors of the i-th fault; Step 3.3, Equation 4 in step 3.2 needs to solve the linear transformation view of J sensors. By converting the objective function into a trace ratio problem, the solution of the problem can be simplified. The trace ratio solution of the objective function of Equation 4 can be written as: The above objective function can be regarded as a trace ratio problem. It is difficult for singular value decomposition to obtain the global optimal solution of the multimodal data fusion model. In order to solve this problem, an iterative algorithm based on the Ratio Trace method is used to solve the trace ratio problem. Step 3.4: First, in order to maintain the solvability of the global structure, add the orthogonal constraint S T S = E; where E is the identity matrix, and let Then formula 11 can be written as: It is known that formula 12 satisfies Since the model has processed the sensor data using the PCA principal component analysis method, there is no The space is 0, then the following formula can be used to convert In the formula: Θ=[θ1,θ2,θ3,...,θ C ],θ c >0, c=1,2,...,C; C is The number of positive singular values; Let S = QB, d is the rank of S, and the objective function can be expressed as: Among them, W (u) =B T W.B., Taking into account the singular value decomposition, there is a problem in Equation 14: even if there is a closed-form solution, there may still be a large error with the optimal solution; Step 3.5, transform equation 14 into a trace difference problem: where λ n is the projection matrix B from the previous step n-1 The calculated trajectory ratio, the specific algorithm steps are: a) B 0 Initialize to an arbitrary column orthogonal matrix; b) Starting from n=1, 2, ..., N, by calculating the matrix B n-1 Calculate θ n : c) The trace difference problem is constructed as: d) Use eigenvalue decomposition method to solve the trace difference problem: In the formula, for The rth largest eigenvalue in for The corresponding eigenvector; In order to maintain orthogonality, reconstruct the projection matrix, set d is the rank of the low-dimensional feature, let V (b) =B n (B n ) T V (u) B n (B n ) T , perform singular value decomposition, we have: V (b) =B n Θ b (B n ) T (Formula 19) e) If yes Then exit the loop; Output B = B n ; In step 4, the multimodal data fusion model obtained in step 3 is used to project the sensor features into a new space and classify and predict the data using a classifier.

2. The method according to claim 1, characterized in that The specific implementation of step 1 includes the following sub-steps: Step 1.1, based on the collected large equipment failure samples, classify the failure data and mark each type of failure data; Step 1.2, for the vibration signal data, a resampling method is selected to expand the number of samples in the data set, and after obtaining the expanded data set, the data of each fault sample is normalized; Step 1.3, for the expanded data set obtained in step 1.2, add noise to the sensor data to simulate the behavior of sensor failure, and the noise adding method used is additive Gaussian noise.

3. The method according to claim 1, characterized in that The specific implementation of step 2 includes the following sub-steps: Step 2.1: For the expanded data set obtained in step 1.3, wavelet analysis is used to reduce the noise of the signal whose characteristics show strong instability due to noise, and then it is converted into a wavelet time-frequency graph for learning. For the two-dimensional image of the wavelet time-frequency graph, the following convolutional network structure is used for learning: a. Image input, parameters are set to: 64×64×3 image, 'zerocenter' normalization; b. Convolution layer, the parameters are set as follows: the number of convolution kernels is 8, the convolution kernel size is 5×5, the stride is [1, 1], and t padding is "same"; c. Batch processing layer, the parameters are set to: batch normalization; d. Activation function layer, the parameters are set as: using ReLU activation function; e. Pooling layer, the parameters are set as follows: pooling mode is "max pooling", pooling domain size is 2×2, stride is [2, 2], and t is filled with "0"; f. Convolution layer, the parameters are set as follows: the number of convolution kernels is 16, the convolution kernel size is 5×5, the stride is [1,1], and the t padding is "same"; g. Batch processing layer, the parameters are set to: batch normalization; h. Activation function layer, the parameters are set to: use ReLU activation function; i. Pooling layer, the parameters are set as follows: pooling mode is "max pooling", pooling domain size is 2×2, stride is [2, 2], and t is filled with "0"; j. Fully connected layer, the parameters are set to: 4096 fully connected layers; Through the CNN architecture, the feature matrix of the sensor is extracted as X ijk =(x ijk1 ,x ijk2 ,...,x ijkz ), where z is the number of features extracted; Step 2.2: For one-dimensional signals with good quality, the convolutional neural network is directly applied for learning. The structure of the convolutional neural network is as follows: a. Vibration signal input, parameter settings: 1×1024, 'zerocenter' normalization; b. Convolution layer, the parameters are set as follows: number of convolution kernels is 8, convolution kernel size is 1×1024, stride is [1, 1], and t padding is "same"; c. Pooling layer, the parameters are set as follows: pooling mode is "max pooling", pooling domain size is 1×512, stride is [2, 2], and t filling is "0"; d. Convolution layer, the parameters are set as follows: the number of convolution kernels is 16, the convolution kernel size is 1×512, the stride is [1, 1], and t padding is "same"; e. Pooling layer, the parameters are set as follows: pooling mode is "max pooling", pooling domain size is 1×512, stride is [2, 2], and t filling is "0"; f. Fully connected layer, the parameters are set to: 4096 fully connected layers; Through the CNN architecture, the feature matrix of the sensor is extracted as X ijk =(x ijk1 ,x ijk2 ,...,x ijkz ), where z is the number of features extracted; Step 2.3, use the principal component analysis method to process each sensor matrix. The method projects the data feature matrix onto a set of mutually orthogonal matrices, then calculates the variance between the projected data, and evaluates the importance of the feature matrix based on the variance. For the sensor feature data, there are: X ijk =[x i j k1 ... x i j k z ] (Formula 1) Where: X ijk is a matrix composed of sensor features. Next, solve X ijk The standardized matrix is ​​denoted as In the formula, M and Y are two variables, K is the total number of samples, is the mean value of M, is the mean of Y; calculate The eigenvectors of all eigenvectors form a matrix X C , construct the principal component matrix.

4. The method according to claim 1, characterized in that The specific implementation of step 4 includes the following sub-steps: Step 4.1, the multimodal data fusion model projects the sensor features into a space, and then uses SVM as a classifier; Step 4.2, select different kernel functions for the SVM classifier in step 4.1, where: a) Linear kernels are generally used in linearly separable situations. They have fewer parameters and faster speeds, so they have excellent performance for linearly separable data. b) The polynomial kernel function can directly reflect the low-dimensional input gap to the higher-dimensional feature gap. However, if the polynomial kernel function has too many parameters, the complexity of the calculation will tend to infinity due to the increase in the number of features. c) The Gaussian kernel function is a kernel function with strong local representation, and is a kernel variable with strong local representation, which shows excellent performance in both large and small samples; d) The Sigmoid function approximates the support vector machine to a multi-layer neural network; In step 4.3, the same processing steps are implemented according to the clustering algorithm to complete the fault type diagnosis task.