Intelligent fault diagnosis method, device and equipment based on deep learning

Abstract

The application relates to an intelligent fault diagnosis method, device and equipment based on deep learning. The method comprises the following steps: acquiring signal data, multichannel data and multimodal data of a target device or component; periodically sampling the signal data and the multichannel data to construct an intelligent fault diagnosis model; inputting the data set generated by sampling into a multichannel data fusion layer through an input layer to splice and fuse the data in the channel dimension; extracting features of a shallow layer according to a hidden layer, inputting deep layer features obtained and multimodal data into a multimodal data fusion layer to fuse the data, classifying faults according to a classification function in a connection layer, and setting a loss function according to a classification prediction result and a label of a corresponding sample in the data set; training the intelligent fault diagnosis model by using the loss function, and realizing intelligent fault diagnosis according to the trained intelligent fault diagnosis model. The method can improve the intelligent device fault diagnosis accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to an intelligent fault diagnosis method, device and equipment based on deep learning. BACKGROUND

[0002] In the production process of modern industry, the damage and failure of production equipment or components often occur, and the working conditions are complex and harsh. Compared with the traditional production process, the frequency of such failures has increased significantly. Therefore, in order to reduce production risks, timely find and eliminate faults, and avoid further losses, fault diagnosis technology is proposed and widely applied in the production process. Traditional fault diagnosis is mainly based on signal processing technology. The fault diagnosis technology in this stage finds fault features from the analysis of relevant signal data of target equipment or components in operation to realize fault diagnosis, such as wavelet analysis, Bayesian decision, time series model diagnosis based on probability statistics, and fault tree analysis. However, traditional fault diagnosis technology often requires a large amount of prior knowledge and is difficult to handle large-scale complex data. Therefore, some data-driven methods have emerged, such as statistical machine learning-based diagnosis methods. These methods, such as support vector machines, random forests, and decision trees, can automatically extract fault features from signal data to a certain extent and have been verified on a certain scale and complexity of data. However, these methods are difficult to automatically process high-dimensional features with certain depth and complexity in the data, and have poor scalability and adaptability. With the continuous improvement of hardware performance in recent years, deep learning, a data-driven method with strong fitting ability, has gradually entered people's field of vision.

[0003] Deep learning uses multi-layer neural networks for learning and representation, aiming to make machines have analysis and learning ability like people by simulating the working way of human brain neural networks. Through training of a large amount of data, it adjusts the weight parameters of the neural network model to mine the deep rules and features in the data and fit the inherent rules, and further generalizes to other data. Since deep learning does not require much prior knowledge, has good scalability and adaptability, and has strong fitting ability to automatically extract features from data and analyze them, it has been favored in the field of intelligent fault diagnosis.

[0004] However, deep learning often assumes that there is the same distribution in the training data and the data to be generalized, that is, the training data and the data to be generalized have the same inherent law, although the law cannot be explicitly represented by a formula, but through the fitting ability of deep learning, the law can be approximated with infinite precision, so that the trained model can also have good performance on the data to be generalized. However, in modern industrial production processes, various working conditions are interwoven and very complex, and different working conditions may have different distribution laws, and it is often very costly to obtain fault data under a certain working condition, therefore, transfer learning is introduced for cross-condition diagnosis.

[0005] Transfer learning can transfer the knowledge learned on the source domain data to the target domain data by comparing the source domain data and the target domain data. Through transfer learning, a neural network model can be trained on only complete source domain data and unlabeled target domain data, and the model can have good performance on both the source domain and the target domain, which makes transfer learning widely used in cross-condition diagnosis. However, transfer learning requires both source domain and target domain data when training the model, and if there is no target domain data, transfer learning may fail, and in modern industrial production processes, it is very costly or technically difficult to collect data under certain harsh production conditions.

[0006] In addition, most of the previous deep learning-based fault diagnosis techniques ignore the periodicity of the target device operation, resulting in low fault diagnosis accuracy. SUMMARY

[0007] Therefore, it is necessary to provide a deep learning-based intelligent fault diagnosis method, device and equipment capable of improving the accuracy of intelligent device fault diagnosis.

[0008] A deep learning-based intelligent fault diagnosis method, the method comprising:

[0009] Obtaining signal data, multi-channel data and multi-modal data of a target device or component; the signal data is a time series signal sampled by a sensor from the target device at a certain frequency within a certain time; the multi-channel data is data collected by other sensors for other components related to the target device or component; and the multi-modal data is other modal data captured by the sensor related to the target device or component;

[0010] Periodically sliding sampling the signal data and the multi-channel data to obtain a data set generated by sampling;

[0011] Constructing an intelligent fault diagnosis model; the intelligent fault diagnosis model comprises an input layer, a multi-channel data fusion layer, a hidden layer, a multi-modal data fusion layer and a fully connected layer;

[0012] The data set generated by sampling is input to the multi-channel data fusion layer through the input layer to splice and fuse in the channel dimension, to obtain shallow features; deep feature extraction is performed on the shallow features according to the hidden layer, and the deep features obtained and the multi-modal data are input to the multi-modal data fusion layer for data fusion, to obtain fused features; the connected layer is used to classify the fused features according to a classification function, to obtain a classification prediction result.

[0013] A loss function is set according to the classification prediction result and the label of the corresponding sample in the data set; the intelligent fault diagnosis model is trained by using the loss function, to obtain a trained intelligent fault diagnosis model; and the intelligent fault diagnosis is realized according to the trained intelligent fault diagnosis model.

[0014] An intelligent fault diagnosis device based on deep learning, comprising:

[0015] A data acquisition module is configured to acquire signal data, multi-channel data and multi-modal data of a target device or component; the signal data is a time series signal sampled by a sensor from the target device at a certain frequency within a certain time; the multi-channel data is data collected by other sensors for other components related to the target device or component; and the multi-modal data is other modal data captured by the sensor and related to the target device or component.

[0016] A data sampling and model construction module is configured to periodically and slidingly sample the signal data and the multi-channel data, to obtain a data set generated by sampling; and construct an intelligent fault diagnosis model; the intelligent fault diagnosis model comprises an input layer, a multi-channel data fusion layer, a hidden layer, a multi-modal data fusion layer and a fully connected layer.

[0017] A fault classification and prediction module is configured to input the data set generated by sampling to the multi-channel data fusion layer through the input layer to splice and fuse in the channel dimension, to obtain shallow features; perform deep feature extraction on the shallow features according to the hidden layer, input the deep features obtained and the multi-modal data to the multi-modal data fusion layer for data fusion, to obtain fused features; and perform fault classification on the fused features according to a classification function in the connected layer, to obtain a classification prediction result.

[0018] A model training and fault diagnosis module is configured to set a loss function according to the classification prediction result and the label of the corresponding sample in the data set; train the intelligent fault diagnosis model by using the loss function, to obtain a trained intelligent fault diagnosis model; and realize intelligent fault diagnosis according to the trained intelligent fault diagnosis model.

[0019] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor realizes the following steps when executing the computer program:

[0020] Obtaining signal data, multi-channel data and multi-modal data of the target device or component; the signal data is a time series signal sampled by a sensor from the target device at a certain frequency within a certain time; the multi-channel data is data collected by other sensors for other components related to the target device or component; the multi-modal data is other modal data captured by the sensor related to the target device or component;

[0021] Periodically sliding sampling the signal data and the multi-channel data to obtain a sampled data set;

[0022] Constructing an intelligent fault diagnosis model; the intelligent fault diagnosis model includes an input layer, a multi-channel data fusion layer, a hidden layer, a multi-modal data fusion layer and a fully connected layer;

[0023] The sampled data set is input into the multi-channel data fusion layer through the input layer for splicing and fusion in the channel dimension to obtain shallow features; the hidden layer extracts deep features from the shallow features, and the obtained deep features and the multi-modal data are input into the multi-modal data fusion layer for data fusion to obtain fused features; the fully connected layer classifies the fused features according to a classification function to obtain a classification prediction result;

[0024] Setting a loss function according to the classification prediction result and the label of the corresponding sample in the data set; training the intelligent fault diagnosis model by using the loss function to obtain a trained intelligent fault diagnosis model; and realizing intelligent fault diagnosis according to the trained intelligent fault diagnosis model.

[0025] The intelligent fault diagnosis method, device and equipment based on deep learning, the application adopts signal data, multi-channel data and multi-modal data of a target device or component as input, wherein the periodicity of fault features is contained, the periodicity is used to determine the sample length in the data set, the model parameter amount is reduced while ensuring the best diagnostic performance, so as to realize a light model. The sensors configured on the intelligent device not only collect relevant signal data of the target device or component, but also collect other data indirectly related to the target device or component. These data usually contain auxiliary information that can improve the diagnostic accuracy. These information can also be used to improve the performance of fault diagnosis. The signal data and multi-channel data are periodically sampled, an intelligent fault diagnosis model is constructed, the generated data set is sent to a multi-channel data fusion layer for shallow feature extraction, the signal data and multi-channel data are spliced in the channel dimension, then a convolutional neural network is used and a two-dimensional convolution is introduced to extract shallow fusion features, by applying the two-dimensional convolution, the convolution kernel can span multiple channels, the field of view of the convolution kernel is enhanced, so that the data of multiple channels are integrated for feature extraction, the extracted shallow features are sent to a hidden layer to extract deep features, an attention mechanism is applied to the multi-modal data to highlight the useful part and suppress the useless part, and the useful features are spliced with the deep features extracted by the hidden layer to realize multi-modal data fusion. The data is sent to a full connection layer for classification, and the probability value of each fault type to which the sample belongs is output. The training loss is calculated and back propagation is performed to adjust the weight to realize model training. The intelligent fault diagnosis based on the trained intelligent fault diagnosis model can greatly improve the intelligent fault diagnosis accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 A flowchart of an embodiment of an intelligent fault diagnosis method based on deep learning;

[0027] Figure 2 A framework diagram of an embodiment of an intelligent fault diagnosis model;

[0028] Figure 3 A schematic diagram of a periodic sliding sampling method in an embodiment;

[0029] Figure 4 A schematic diagram of a multi-channel data fusion method in another embodiment;

[0030] Figure 5 A schematic diagram of a multi-modal data fusion method in an embodiment;

[0031] Figure 6 A schematic diagram of the model accuracy change with or without a multi-channel data fusion method in an embodiment;

[0032] Figure 7 Fig. 2 is a diagram showing the change in model accuracy with or without multi-modal data fusion method in an embodiment;

[0033] Figure 8 Fig. 3 is a diagram showing the performance index effect of the present application on two public data sets in an embodiment; (a) is a diagram showing the test set accuracy effect; (b) is a diagram showing the confusion matrix effect;

[0034] Figure 9 Fig. 4 is a structural block diagram of a deep learning-based intelligent fault diagnosis device in an embodiment;

[0035] Figure 10 Fig. 5 is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0037] In an embodiment, as shown in Fig. 1, a deep learning-based intelligent fault diagnosis method is provided, comprising the following steps: Figure 1

[0038] Step 102, acquiring signal data, multi-channel data and multi-modal data of the target device or component; the signal data is a time series signal sampled by a sensor from the target device at a certain frequency within a certain time; the multi-channel data is data collected by other sensors for other components related to the target device or component; and the multi-modal data is other modal data captured by the sensor related to the target device or component.

[0039] The present application utilizes three kinds of data for input, which are: signal data, multi-channel data and multi-modal data of the target device or component. Among them, the signal data is a time series signal sampled by a sensor from the target device at a certain frequency within a certain time, which is a direct data sampling for the target device, and can be described as follows:

[0040]

[0041] Among them, X is the sampled signal data, X0, X1, …, X N-1 are data points in the signal data. The multi-channel data is data collected by other sensors for other components related to the target device or component, such as the shell or motor current connected to the target device or component, etc. These data have the same structure and data format as the signal data, and can be described as follows: ​

[0042]

[0043] wherein X mc is the multi-channel data, C is the number of channels, is the data point of the multi-channel data. The multi-modal data is other modal data captured by the sensor related to the target device or component, such as load, temperature, etc., which may have different structural forms and data formats from the above two kinds of data, which can be described as follows:

[0044]

[0045] wherein X mm is the multi-modal data, is other modal data sampled by different sensors, which may have different data forms and structural forms.

[0046] The periodicity reflected in the signal data as a fault feature can be utilized to determine the sample length in the data set, which can reduce the model parameter amount while ensuring the best diagnostic performance, so as to realize a lighter model. Furthermore, in modern industrial production processes, a large number of sensors are often deployed in the workshop, which not only collect signal data related to the operation of the target device or component, but also collect other data indirectly related to the target device or component, which usually contains auxiliary information that can help improve the diagnostic accuracy. These information can be utilized to improve the performance of fault diagnosis.

[0047] Step 104, periodically sliding sampling the signal data and the multi-channel data to obtain a data set generated by sampling; constructing an intelligent fault diagnosis model; the intelligent fault diagnosis model includes an input layer, a multi-channel data fusion layer, a hidden layer, a multi-modal data fusion layer and a fully connected layer.

[0048] As Figure 2 shown is the overall flow framework of the present application, including an input layer, a multi-channel data fusion layer, a hidden layer, a multi-modal data fusion layer and an output layer. In order to make full use of the data collected by the multi-sensor, the signal data, the multi-channel data and the multi-modal data of the target device or component are used for intelligent fault diagnosis of the target device or component. First, the signal data and the multi-channel data are periodically sliding sampled, as shown in Figure 3 The number of sampling points of the target device or component in one operating cycle is calculated first:

[0049]

[0050] Wherein, n is the sampling point number of the target device or component in one running cycle, f is the sampling frequency of the sensor, and RPM is the cycle number of the target device or component running per minute. Subsequently, the sample length of the to-be-generated data set is determined according to the sampling point number of the target device or component in one cycle:

[0051] L = a x n, a e (0, 1]

[0052] Wherein, L is the sample length of the to-be-generated data set, n is the sampling point number of the target device or component in one running cycle, and a is a scaling factor. And the sliding step is:

[0053]

[0054] Wherein, stride is the sliding step, and L is the sample length of the to-be-generated data set. For a piece of signal data or data of a channel in multi-channel data The data set generated by sampling is:

[0055]

[0056] Wherein, X pss is the generated data set, and X[a, b) represents a continuous sub-interval data [X a , X a+1 , …, X b-1 ] in the data X. The whole sampling process is as shown in Figure 3 The data set generated by sampling by this method can not only fully utilize the information of the signal data and the multi-channel data to generate the data set, but also can ensure that each sample has sufficient fault information and the sample length is as short as possible, thereby reducing the model parameters and speeding up the training and inference.

[0057] In step 106, the data set generated by sampling is input to the input layer, the multi-channel data fusion layer is spliced and fused in the channel dimension to obtain shallow features, the hidden layer is used to extract deep features from the shallow features, the obtained deep features and the multi-modal data are input to the multi-modal data fusion layer for data fusion to obtain fused features, and the classification function is used in the connection layer to classify the fused features to obtain a classification prediction result.

[0058] The signal data and the multi-channel data sampled by the cycle sliding are input to the multi-channel data fusion layer for data fusion. First, the signal data is spliced with the multi-channel data in the channel dimension as the 0th channel to obtain:

[0059]

[0060] Wherein, Concat C (X, Xmc This indicates that signal data and multi-channel data are concatenated along the channel dimension. The first part is signal data, and the other part is multi-channel data. This data is then fed into a Convolutional Neural Network (CNN), where two-dimensional convolution is applied. The number of channels C+1 is considered the height, and the sample length L is considered the width. A convolution kernel with a height and width of (C+1)×1 is used for shallow feature extraction, resulting in the following output:

[0061]

[0062] Where k0,k1,…,k C These are the kernel components, which are the trainable parameters in a convolutional neural network. The process is as follows: Figure 4 As shown, by applying two-dimensional convolution, the convolution kernel can span multiple channels, enhancing its field of view and thus integrating data from multiple channels for feature extraction.

[0063] The extracted shallow features are fed into hidden layers to extract deeper features. The hidden layers are implemented using a fully convolutional neural network (WCNN) with residual connections added. These residual connections only transform the input channels, with a kernel length and stride of 1. Residual connections prevent performance degradation as the network deepens, enhancing its scalability and adaptability. A residual connection and its associated WCNN layer constitute a residual convolutional block, and the hidden layer consists of several residual convolutional blocks. Each convolutional layer in the WCNN is followed by a normalization layer and an activation function. The normalization layer uses Instance Normalization: it calculates the mean μ and variance σ along each channel, then normalizes the data. Furthermore, scaling and translation (y = γx′ + β) are incorporated. Through normalization, the output of each layer returns to a fixed mean and variance. This reduces the dependence of each neural network layer on the output of the previous layer, enhancing the model's robustness and accelerating convergence. The activation function is implemented using the GELU function.

[0064]

[0065] in, Furthermore, there is a downsampling layer after the activation functions of the first and last convolutional layers. It is implemented using a convolutional neural network. Using a convolutional neural network for downsampling helps to preserve information without directly discarding it. The kernel length and stride are both 2.

[0066] After that, the deep features extracted from the hidden layer and other modal data are sent to the multi-modal data fusion layer to fuse the deep features with other modal data. First, the deep features and other modal data are sent to the multi-layer perception to obtain the similarity weight:

[0067]

[0068] wherein fea is the deep feature extracted through the hidden layer, is the other modal data, is the similarity weight of the extracted deep feature and other modal data. After that, it is normalized by softmax:

[0069]

[0070] wherein a i is the normalized weight corresponding to the other modal data, and e is a natural constant. After that, by multiplying the weight with the corresponding multi-modal data, the attention value can be obtained:

[0071]

[0072] wherein Attention i is the attention value corresponding to the other modal data. By using the attention mechanism, the content in the other modal data that is helpful for the diagnostic performance can be highlighted, while the content that is not helpful for the diagnostic result is suppressed, so that the model can focus more on the truly effective part of the data. Finally, the above attention value is spliced with the deep feature to obtain the feature after the fusion of the multi-modal data, that is, the input of the fully connected classification network:

[0073] Concat(fea,Attrntion1,…,Attention N′ )

[0074] wherein fea is the deep feature extracted by the hidden layer, and the whole process is shown in Figure 5 .

[0075] Finally, the above fused feature data is sent to the fully connected layer for classification, considering the classification function wherein y is the prediction result, F(·) is the function fitted by the network, and x is the input of the classification network, wherein the prediction result is a probability distribution, and the probabilities that the sample data belongs to fault types 1, 2, …, n, respectively, are taken as the classification result, that is:

[0076] ​​

[0077] where i = 1, 2, …, n.

[0078] In step 108, a loss function is set according to the classification prediction result and the label of the corresponding sample in the data set; the intelligent fault diagnosis model is trained by using the loss function to obtain a trained intelligent fault diagnosis model; and the intelligent fault diagnosis is realized according to the trained intelligent fault diagnosis model.

[0079] The data is sent into the full connection classification network for classification, and the probability value of each fault type to which the sample belongs is output. The training loss is calculated and back propagation is performed to adjust the weight until all the training data is trained, which is an epoch. After several epochs, when the loss no longer decreases, the model reaches a convergent state, at which time the training is completed. The intelligent fault diagnosis realized according to the trained intelligent fault diagnosis model can greatly improve the intelligent fault diagnosis accuracy.

[0080] In the above intelligent fault diagnosis method based on deep learning, the signal data, multi-channel data and multi-modal data of the target device or component are used as inputs, which contain periodicity of fault features. The periodicity is used to determine the sample length in the data set, which can reduce the model parameter amount while ensuring the best diagnostic performance, so as to realize a lighter model. The sensors configured on the intelligent device not only collect the relevant signal data of the target device or component, but also collect other data indirectly related to the target device or component. These data usually contain auxiliary information that can help improve the diagnostic accuracy. These information can also be used to improve the performance of fault diagnosis. Then the signal data and multi-channel data are periodically sampled to construct an intelligent fault diagnosis model. The generated data set is sent into a multi-channel data fusion layer to extract shallow layer features. The signal data and multi-channel data are spliced in the channel dimension. Then a convolutional neural network is used and a two-dimensional convolution is introduced to extract shallow layer fusion features. By applying the two-dimensional convolution, the convolution kernel can span multiple channels to enhance the field of view of the convolution kernel, so as to extract features from multiple channels. The extracted shallow layer features are sent into a hidden layer to extract deep layer features. An attention mechanism is applied to the multi-modal data to highlight the useful part and suppress the useless part. The highlighted useful features are spliced with the deep layer features extracted by the hidden layer to realize multi-modal data fusion. The data is sent into a full connection layer for classification, and the probability value of each fault type to which the sample belongs is output. The training loss is calculated and back propagation is performed to adjust the weight to realize model training. The intelligent fault diagnosis realized according to the trained intelligent fault diagnosis model can greatly improve the intelligent fault diagnosis accuracy.

[0081] In one embodiment, the signal data and the multi-channel data are periodically and slidingly sampled to obtain a sampling generated data set, including:

[0082] The number of sampling points of the target device or component in one running cycle is calculated as:

[0083]

[0084] Wherein, n is the number of sampling points of the target device or component in one running cycle, f is the sampling frequency of the sensor, and RPM is the number of cycles per minute of the target device or component running;

[0085] The sample length of the data set to be generated is calculated according to the number of sampling points as:

[0086] L = a x n, a e (0, 1]

[0087] Wherein, L is the sample length of the data set to be generated, n is the number of sampling points, and a is the scaling factor;

[0088] The sliding step is determined using the sample length as:

[0089]

[0090] The signal data and the multi-channel data are periodically and slidingly sampled according to the sliding step to obtain a sampling generated data set as:

[0091]

[0092] Wherein, X pss [a, b) represents a continuous sub-interval data [X a , X a+1 ,…, X b-1 ] in the data X.

[0093] In one embodiment, the sampling generated data set is input to the multi-channel data fusion layer through the input layer to be spliced and fused in the channel dimension to obtain shallow features, including:

[0094] The sampling generated data set is input to the multi-channel data fusion layer through the input layer to be spliced with the multi-channel data in the channel dimension as the 0th channel, and the spliced data is obtained as:

[0095]

[0096] Wherein, Concat C (X, X mc ) represents splicing the signal data and the multi-channel data in the channel dimension, is the signal data, and the other parts are the multi-channel data;

[0097] The channel number C+1 is regarded as high and the sample length L is regarded as wide by using the convolutional neural network, and a convolution kernel with high and wide of (C+1) x 1 is applied to extract the shallow layer feature, and the shallow layer feature is obtained as follows:

[0098]

[0099] wherein k0, k1, …, k C are convolution kernel components, that is, parameters that can be trained in the convolutional neural network.

[0100] In one of the embodiments, the hidden layer includes a plurality of residual convolution blocks; the residual convolution block is composed of a residual connection layer and a convolutional neural network layer included in the residual connection layer; and the deep layer feature extraction on the shallow layer feature by the hidden layer includes:

[0101] The channel transformation is performed on the shallow layer feature by the residual connection layer, and then the normalization and activation and down-sampling are performed after the feature extraction by the convolutional layer in the convolutional neural network to obtain the deep layer feature; and the activation function used in the activation process is as follows:

[0102]

[0103] wherein, x represents the data output by the previous layer of neural network.

[0104] In one of the embodiments, the deep layer feature obtained and the multi-modal data are input into a multi-modal data fusion layer to perform data fusion, and the fused feature is obtained, including:

[0105] The deep layer feature and the multi-modal data are input into the multi-layer perception to obtain the similarity weight as follows:

[0106]

[0107] wherein fea is the deep layer feature extracted by the hidden layer, is the multi-modal data.

[0108] In one of the embodiments, the similarity weight is normalized, and the normalized weight is obtained as follows:

[0109]

[0110] The normalized weight and the corresponding multi-modal data are multiplied to obtain the attention value of the corresponding multi-modal data as follows:

[0111]

[0112] In one of the embodiments, the attention values and deep features corresponding to the multi-modal data are spliced to obtain the fused features:

[0113] Concat(fea,Attrntion1,…,Attention N′ ).

[0114] In one of the embodiments, the loss function is set according to the classification prediction result and the label of the corresponding sample in the data set:

[0115]

[0116] Where y i is the probability value of the probability label position i in the one-hot form, is the probability value of the prediction result position i.

[0117] In specific embodiments, to evaluate the performance of the present application, the present application uses the CWRU data set and the JNU data set for simulation and performance test. The Adam optimizer is used, and the learning rate is set to 0.01. The two data sets are iterated for 100 epochs. Figure 6 The precision changes of the model on the test set during the training process are given in two cases of the multi-channel data fusion method and the non-multi-channel data fusion method. The model using the classic one-dimensional convolution for processing can be seen. During the entire training process, the model with the multi-channel data fusion method is often higher than the model without the multi-channel data fusion method, and it has been maintained throughout the training period, which shows the effectiveness of the proposed multi-channel data fusion method. In addition, Figure 7 The precision of the model with the multi-modal data fusion method and the model without the multi-modal data fusion method on the test set during the training process is given. It can be seen that between the two corresponding methods on the corresponding data set, the model with the multi-modal data fusion method has a higher precision on the test set during the training process than the model without the multi-modal data fusion method. And this advantage is gradually determined until the final model loss converges, and the model training is completed, so it can be proved that the proposed multi-modal data fusion method is effective. In addition, it should be noted that the working conditions of the data set used in training and the working conditions of the data set used in testing are not consistent, and the training results reflect that the model trained by the present application can be well generalized to the data of the target working condition, which shows that the invariant domain attribute existing in multiple domains is found through the multi-modal data fusion method, which enables the present application to overcome the defects of transfer learning and generalize the model from the source domain data to the target domain data in the absence of target domain data.

[0118] Finally,Figure 8 The performance of the present application on two public datasets is demonstrated, including the accuracy change on the test set during the training process and the confusion matrix, as shown in Figure 8 (a) and Figure 8 (b), through the demonstration and explanation of these performance indicators, it can be proved that the proposed method performs well in actual situations, and compared with other intelligent fault diagnosis methods, the scheme proposed in the present application is better.

[0119] It should be understood that, although Figure 1 The steps in the flowchart of the method are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in the method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or sub-steps or stages of other steps.

[0120] In one embodiment, as shown in Figure 9 A deep learning-based intelligent fault diagnosis device is provided, comprising a data acquisition module 902, a data sampling and model construction module 904, a data sampling and model construction module 906, and a model training and fault diagnosis module 908, wherein:

[0121] The data acquisition module 902 is configured to acquire signal data, multi-channel data, and multi-modal data of a target device or component; the signal data is a time series signal sampled by a sensor from the target device at a certain frequency within a certain time; the multi-channel data is data collected by other sensors for other components related to the target device or component; and the multi-modal data is other modal data captured by the sensor related to the target device or component;

[0122] The data sampling and model construction module 904 is configured to periodically slide sample the signal data and multi-channel data to obtain a data set generated by sampling, and construct an intelligent fault diagnosis model; the intelligent fault diagnosis model comprises an input layer, a multi-channel data fusion layer, a hidden layer, a multi-modal data fusion layer, and a fully connected layer;

[0123] The fault classification prediction module 906 is configured to input the data set generated by the sampling into the input layer, splice and fuse the multi-channel data in the channel dimension through the multi-channel data fusion layer to obtain shallow features, perform deep feature extraction on the shallow features according to the hidden layer, input the obtained deep features and multi-modal data into the multi-modal data fusion layer to perform data fusion, and obtain fused features; and perform fault classification on the fused features according to a classification function in the connection layer to obtain a classification prediction result.

[0124] The model training and fault diagnosis module 908 is configured to set a loss function according to the classification prediction result and a label of a corresponding sample in the data set, train the intelligent fault diagnosis model by using the loss function to obtain a trained intelligent fault diagnosis model, and perform intelligent fault diagnosis according to the trained intelligent fault diagnosis model.

[0125] The specific limitations of the intelligent fault diagnosis apparatus based on deep learning can be referred to the limitations of the intelligent fault diagnosis method based on deep learning in the foregoing, and will not be described herein. Each module in the intelligent fault diagnosis apparatus based on deep learning can be realized by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0126] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in FIG. 8. Figure 10 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an intelligent fault diagnosis method based on deep learning. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball, or touchpad arranged on the shell of the computer device, or can be an external keyboard, touchpad, or mouse, etc.

[0127] Those skilled in the art can understand that Figure 10The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0129] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0130] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A deep learning-based intelligent fault diagnosis method, characterized in that, The method includes: Acquire signal data, multi-channel data, and multi-modal data of the target device or component; the signal data is a time-series signal sampled directly from the target device by the sensor at a certain frequency within a certain time; the multi-channel data is data collected by other sensors from other components related to the target device or component; the multi-modal data is data of other modes related to the target device or component captured by the sensor. Periodic sliding sampling is performed on the signal data and multi-channel data to obtain the sampled dataset; A smart fault diagnosis model is constructed; the smart fault diagnosis model includes an input layer, a multi-channel data fusion layer, a hidden layer, a multi-modal data fusion layer, and a fully connected layer; The dataset generated by the sampling is input into a multi-channel data fusion layer through an input layer for concatenation and fusion along the channel dimension to obtain shallow features. Deep features are extracted from the shallow features based on the hidden layer. The obtained deep features and multimodal data are then input into the multimodal data fusion layer for data fusion to obtain fused features. Fault classification is performed on the fused features in the connection layer according to a classification function to obtain a classification prediction result. A loss function is set based on the classification prediction results and the labels of the corresponding samples in the dataset; the intelligent fault diagnosis model is trained using the loss function to obtain a trained intelligent fault diagnosis model; intelligent fault diagnosis is implemented based on the trained intelligent fault diagnosis model. The obtained deep features and multimodal data are input into a multimodal data fusion layer for data fusion to obtain fused features, including: The deep features and multimodal data are fed into a multilayer perceptron to calculate the similarity weights: Here, fea represents the deep features extracted through the hidden layer. For multimodal data; The similarity weights are normalized to obtain the normalized weights: Multiplying the normalized weights by the corresponding multimodal data yields the attention value for the corresponding multimodal data: The attention values ​​and deep features of the corresponding multimodal data are concatenated to obtain the fused features: Concat(fea,Attrntion1,…,Attention N′ )。 2. The method according to claim 1, characterized in that, Periodic sliding sampling is performed on the signal data and multi-channel data to obtain a sampled dataset, including: The number of sampling points for the target device or component within one operating cycle is calculated as follows: Where n is the number of sampling points of the target device or component in one operating cycle, f is the sampling frequency of the sensor, and RPM is the number of cycles per minute of the target device or component. The sample length of the dataset to be generated is calculated based on the number of sampling points: L = α × n, α ∈ (0, 1) Where L is the sample length of the dataset to be generated, n is the number of sampling points, and α is the scaling factor; The sliding step size is determined using the sample length as follows: Based on the stated sliding step size, periodic sliding sampling is performed on the signal data and multi-channel data to obtain the sampled dataset as follows: Among them, X pss For the generated dataset, X[a,b) represents a continuous sub-interval of data [X] in the data X. a ,X a+1 ,…,X b-1 ].

3. The method according to claim 1, characterized in that, The dataset generated by the sampling is input into a multi-channel data fusion layer through an input layer for concatenation and fusion along the channel dimension to obtain shallow features, including: The dataset generated by the sampling is input into the multi-channel data fusion layer through the input layer. It is then concatenated with the multi-channel data along the channel dimension, resulting in the following concatenated data: Among them, Concat C (X,X mc This indicates that signal data and multi-channel data are concatenated along the channel dimension. The data consists of signal data, while the other parts are multi-channel data. Using a convolutional neural network, the number of channels C+1 is considered as the height, and the sample length L is considered as the width. A convolutional kernel with a height and width of (C+1)×1 is applied to extract shallow fusion features, resulting in the following shallow features: Where k0,k1,…,k C These are the kernel components, which are the trainable parameters in a convolutional neural network.

4. The method according to claim 1, characterized in that, The hidden layer includes multiple residual convolutional blocks; each residual convolutional block consists of a residual connection layer and its included convolutional neural network layers; deep feature extraction is performed on the shallow features based on the hidden layer, including: The shallow features are transformed using the residual connection layer, and then normalized, activated, and downsampled after feature extraction using the convolutional layers in the convolutional neural network to obtain the deep features. The activation function used in the activation process is: in, x represents the data output by the previous layer of the neural network.

5. The method according to claim 1, characterized in that, Based on the classification prediction results and the labels of the corresponding samples in the dataset, the loss function is set as follows: Among them, y i Let be the probability value at position i, expressed in one-hot form. Let i be the probability value at position i in the prediction result.

6. A deep learning-based intelligent fault diagnosis device, characterized in that, The device includes: The data acquisition module is used to acquire signal data, multi-channel data, and multi-modal data of the target device or component; the signal data is a time-series signal sampled directly from the target device by the sensor at a certain frequency within a certain time; the multi-channel data is data collected by other sensors from other components related to the target device or component; the multi-modal data is other modal data related to the target device or component captured by the sensor. The data sampling and model building module is used to perform periodic sliding sampling on the signal data and multi-channel data to obtain the sampled dataset; and to build an intelligent fault diagnosis model; the intelligent fault diagnosis model includes an input layer, a multi-channel data fusion layer, a hidden layer, a multimodal data fusion layer, and a fully connected layer; The fault classification and prediction module is used to input the sampled dataset into a multi-channel data fusion layer via an input layer for concatenation and fusion along the channel dimension to obtain shallow features; extract deep features from the shallow features based on the hidden layer; input the obtained deep features and multimodal data into the multimodal data fusion layer for data fusion to obtain fused features; perform fault classification on the fused features in the connection layer according to a classification function to obtain a classification prediction result; input the obtained deep features and multimodal data into the multimodal data fusion layer for data fusion to obtain fused features, including: The deep features and multimodal data are fed into a multilayer perceptron to calculate the similarity weights: Here, fea represents the deep features extracted through the hidden layer. For multimodal data; The similarity weights are normalized to obtain the normalized weights: Multiplying the normalized weights by the corresponding multimodal data yields the attention value for the corresponding multimodal data: The attention values ​​and deep features of the corresponding multimodal data are concatenated to obtain the fused features: Concat(fea,Attrntion1,…,Attention N′ ); The model training and fault diagnosis module is used to set a loss function based on the classification prediction results and the labels of the corresponding samples in the dataset; to train the intelligent fault diagnosis model using the loss function to obtain a trained intelligent fault diagnosis model; and to implement intelligent fault diagnosis based on the trained intelligent fault diagnosis model.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

Images