Multi-modal fusion bearing fault diagnosis method and system

Abstract

The invention belongs to the technical field of bearing fault diagnosis, and particularly relates to a multi-modal fusion bearing fault diagnosis method and system. The method comprises the following steps: firstly, acquiring a vibration signal of a bearing to be diagnosed, converting the vibration signal into a two-dimensional time-frequency image with time-frequency information, and then remodeling the two-dimensional time-frequency image into one-dimensional data; inputting the vibration signal and the one-dimensional data into a fault diagnosis model to obtain the fault type of the bearing to be diagnosed; wherein the fault diagnosis model is obtained by training a deep learning model by using training data. The method fully extracts the characteristics of the input signal, improves the precision of bearing fault diagnosis, and solves a problem that the bearing fault diagnosis effect is not good because only one-dimensional time domain signals are concerned in the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of bearing fault diagnosis, and in particular relates to a bearing fault diagnosis method and system based on multi-modal fusion. Background Art

[0002] With the continuous improvement of modern industrial rotating machinery manufacturing, the health of rolling bearings, core components of various rotating machinery, is crucial to the reliability and stability of the entire equipment. Rolling bearings are highly susceptible to failure due to long-term operation under high loads, fatigue, and high temperatures. Therefore, timely and effective bearing fault diagnosis is of great significance.

[0003] In recent years, many scholars have conducted extensive research on bearing fault diagnosis based on vibration signals. Many studies have focused solely on one-dimensional time-domain signals or two-dimensional time-frequency signals. For example, a Chinese invention patent application with application publication number CN119939338A and application publication date January 2, 2025, discloses a rolling bearing fault diagnosis method based on deep learning. The method first collects rolling bearing vibration signals and uses an overlapping sampling method to divide the collected rolling bearing vibration signals into several rolling bearing vibration signal subsamples of equal length. Then, a signal denoising algorithm based on CPO-ICEEMDAN is established to decompose the rolling bearing vibration signal subsamples into several IMF components. The key IMF components are screened to obtain a reconstructed subsample signal after denoising. A K-nearest neighbor graph is constructed for the reconstructed subsample signal after denoising. A fault diagnosis dataset is established based on the K-nearest neighbor graph, and the fault diagnosis dataset is divided into a training set, a validation set, and a test set. The established GAT fault diagnosis model is then trained, validated, and tested accordingly. The rolling bearing vibration data to be diagnosed is then input into the trained GAT fault diagnosis model to output the diagnosed fault information. That is, this method only focuses on one-dimensional time domain signals, and this method has poor effect on bearing fault diagnosis. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-modal fusion bearing fault diagnosis method and system to solve the problem of poor bearing fault diagnosis effect caused by only focusing on one-dimensional time domain signals in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a multimodal fusion bearing fault diagnosis method, comprising the following steps: 1) obtaining the vibration signal of the bearing to be diagnosed, converting the vibration signal into a two-dimensional time-frequency image with time-frequency information, and then reshaping the two-dimensional time-frequency image into one-dimensional data; 2) inputting the vibration signal and the one-dimensional data into a fault diagnosis model to obtain the fault type of the bearing to be diagnosed; wherein, the fault diagnosis model is obtained by training a deep learning model using training data.

[0006] Furthermore, the deep learning model includes TCN, SE attention mechanism and classification layer; TCN is used to perform deep-level feature extraction on the input of the deep learning model, the SE attention mechanism is used to capture key features of the input of the deep learning model and dynamically adjust channel weights, and the classification layer is used to perform fault classification based on the output of the SE attention mechanism and the output of TCN.

[0007] Furthermore, the hyperparameters in the TCN are optimized using a meta-heuristic optimization algorithm.

[0008] Furthermore, the meta-heuristic optimization algorithm is a butterfly optimization algorithm.

[0009] Furthermore, the vibration signal is converted into the two-dimensional time-frequency image through continuous wavelet transform.

[0010] Furthermore, the wavelet mother function of the continuous wavelet transform is a complex Morlet wavelet.

[0011] Furthermore, the vibration signal is a signal obtained by cutting and then enhancing the original collected vibration signal.

[0012] Furthermore, the enhancement includes overlapping sampling processing.

[0013] Further, the fault types include roller fault, combined fault, healthy, inner race fault, and outer race fault.

[0014] In order to solve the above technical problems, the present invention also provides a multi-modal fusion bearing fault diagnosis system, including a processor, which is used to execute a computer program to implement the steps of the above-mentioned method.

[0015] The beneficial effects are as follows: This invention is an improved invention creation, which provides a method for bearing fault diagnosis using two modal data, namely the bearing vibration signal and the two-dimensional time-frequency image obtained by converting the vibration signal, so as to fully extract the characteristics of the input signal and improve the accuracy of bearing fault diagnosis. In particular, in order to achieve the unification of modal dimensions, the two-dimensional time-frequency image is also reshaped into one-dimensional data during the specific processing. That is, the vibration signal and the one-dimensional data are specifically input into the fault diagnosis model, so that the fault diagnosis model can effectively process the signal characteristics, improve the fault type diagnosis effect of the model, and improve the generalization ability of the fault diagnosis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the fault diagnosis method of the present invention;

[0017] Figure 2 It is a structural diagram of the fault diagnosis model of the present invention;

[0018] Figure 3 It is a continuous wavelet transform time-frequency diagram of the CWRU data set state after CWT of the present invention;

[0019] Figure 4 It is a continuous wavelet transform time-frequency diagram of the SEU data set state after CWT of the present invention;

[0020] Figure 5 This is a graph showing the classification accuracy changes of the multimodal fusion BOA_SETCN model of the present invention on the Case Western Reserve University dataset;

[0021] Figure 6 This is a graph showing the classification accuracy changes of the multimodal fusion BOA_SETCN model of the present invention on the Southeast University bearing dataset;

[0022] Figure 7 This is a confusion matrix diagram of the multimodal fusion BOA_SETCN model of the present invention on the Case Western Reserve University dataset;

[0023] Figure 8 This is the confusion matrix diagram of the multimodal fusion BOA_SETCN model of the present invention on the Southeast University bearing dataset;

[0024] Figure 9 This is a comparison chart of the classification accuracy of the proposed model and the SETCN comparison model on the CWRU dataset;

[0025] Figure 10 This is a comparison chart of the classification accuracy of the model of the present invention and the SETCN comparison model on the SEU dataset. DETAILED DESCRIPTION

[0026] The present invention utilizes a multimodal fusion bearing fault diagnosis method. Its core approach is to utilize two modal data for bearing fault diagnosis: bearing vibration data and one-dimensional data obtained by converting the bearing vibration data into a two-dimensional time-frequency image and then reshaping the two-dimensional time-frequency image. This improves bearing fault diagnosis performance. To further clarify the objectives, technical solutions, and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and examples.

[0027] An implementation method of a multi-modal fusion bearing fault diagnosis method:

[0028] The present invention provides a multi-modal fusion bearing fault diagnosis method, the overall process of which is as follows: Figure 1 The specific process is as follows:

[0029] Step 1: Obtain the original collected vibration signal data under different health conditions of the bearing.

[0030] The bearing may be a rolling element bearing. The different health states may include rolling element failure, combined failure, healthy, inner race failure, and outer race failure. Furthermore, the originally collected vibration signal data is a one-dimensional vibration signal.

[0031] As an optimized processing method, the raw vibration signal data can be preprocessed. This is the data of the first mode. This preprocessing involves segmenting the raw bearing vibration signal data so that each segment has the same sample length, specifically set to 784. Overlapping sampling is then used for data enhancement. Overlapping sampling creates overlap between adjacent sampling windows, fully capturing information in the vibration time series and minimizing information loss.

[0032] Step 2: Convert the vibration signal data obtained in step 1 into a two-dimensional time-frequency image, and reshape the two-dimensional time-frequency image to obtain data of the second mode.

[0033] In this embodiment, the pre-processed one-dimensional vibration signal is converted into a two-dimensional time-frequency image with time-frequency information through continuous wavelet transform (CWT), and then the two-dimensional time-frequency image is reshaped into one-dimensional data. Specifically, the two-dimensional time-frequency image can be converted into a NumPy array using Python and flattened into a one-dimensional vector. The purpose of data reshaping here is because the data needs to be input into the fault diagnosis model later, and the data input into the fault diagnosis model also has a one-dimensional vibration signal, so the two-dimensional time-frequency image needs to be reshaped. The specific formula for processing using continuous wavelet transform is:

[0034]

[0035] Where, is the result of continuous wavelet transform; x(t) is the input signal; is the selected wavelet mother function (also called wavelet basis function); is the complex conjugate wavelet mother function; a is the scale factor; b is the translation factor; t is the time. As an optimized processing method, the selected wavelet mother function is the complex Morlet wavelet, and its expression is:

[0036]

[0037] Where, f c is the center frequency, f b is the bandwidth frequency. Here you can set the center frequency f c =3Hz, bandwidth frequency f b =3Hz. Of course, other wavelet mother functions in the prior art can also be selected.

[0038] Step three: Use the two modal data of the bearing in different health states as training data to train the deep learning model to obtain a fault diagnosis model.

[0039] In this embodiment, the overall architecture of the deep learning model used is as follows: Figure 2 As shown, it includes TCN (temporal convolutional network), SE attention mechanism and classification layer.

[0040] TCN is used to extract deep features from the input of deep learning models, such as Figure 2 As shown in the lower half of the "SETCN", it includes causal convolution, void convolution and residual connection, where the one-dimensional void causal convolution calculation formula is as follows:

[0041]

[0042] Where F(s) is the output of the one-dimensional dilated causal convolution; x is the input sequence; f is the convolution kernel; d is the dilation coefficient; k is the convolution kernel length; i is the number of network layers; s is the number of time steps. In this embodiment, the dilated convolution kernel size k=3 is set, and the dilated convolution factor d is 1, 2, or 4. The calculation formula for the residual connection is:

[0043] o=Activation(x+F(x)) (4)

[0044] Where o is the output of the residual connection; x is the input; F(x) is the output of the residual module after calculation; Activation is the activation function.

[0045] The SE attention mechanism is used to capture key features of the input of the deep learning model and dynamically adjust the channel weights. Figure 2 As shown in the upper part of the "SETCN", its calculation process includes the following four processes: mapping, compression, excitation and dot product:

[0046] ① Mapping. The following formula (4) is the mapping process from X to U, F tr The essence of is the convolution operation.

[0047] U=F tr (X) (5)

[0048] Where X∈R H'×W'×C' , U∈R H×W×C , H', W', C' represent the height, width and number of channels of X respectively, and H, W, C represent the height, width and number of channels of U respectively.

[0049] ② Compression. The following formula (5) is a mathematical description of the compression operation.

[0050]

[0051] Where U represents the c-th channel feature map after the convolution operation; Z represents the c-th channel feature map after the squeeze operation.

[0052] ③ Excitation. The following formula (6) is a mathematical description of the excitation operation.

[0053] S=F ex (Z)=σ[w2δ(w1Z)] (7)

[0054] Where S is the generated weight; w1 and w2 represent two fully connected operations; σ is the Relu activation function; δ is the Sigmoid activation function.

[0055] ④ Dot product. Perform the dot product operation on the corresponding channel using the following formula by combining the weight value S with the mapping result U from the first step.

[0056]

[0057] Where, Represents the result of the dot product operation.

[0058] The classification layer consists of a fully connected layer and a softmax layer connected in sequence, which is used to classify faults based on the output of the SE attention mechanism and the output of the TCN. The softmax classifier is applied to the output z of the fully connected layer to obtain the probability distribution of each category:

[0059]

[0060] Where, is the probability that the predicted image belongs to the i-th category; exp(z i ) is the exponential function of the score of category i output by the model in the last fully connected layer; Represents the sum of the indexation scores for all categories, total number of category C.

[0061] In addition, as a better processing method, the hyperparameters in the above TCN are optimized using a meta-heuristic optimization algorithm, including filters, kernel_size (convolution kernel size), and strides (stride, which controls the sliding step size of the convolution kernel), to obtain the optimal hyperparameter combination to improve model performance. In addition, the butterfly optimization algorithm (BOA) can be used. The execution process of the BOA optimization algorithm is as follows:

[0062] ① Initialization stage. BOA randomly generates a group of butterfly individuals (i.e., hyperparameter combinations), each butterfly position x 0 =(x 1 ,x 2 ,...,x N), each individual represents a set of possible hyperparameter value combinations. Calculate the initial fitness value f0 and record the best position g 0 And the optimal fitness value f g The calculation formula of fragrance size f is as follows:

[0063] f=cI a (10)

[0064] Where I is the stimulus intensity. The higher the fitness value, the larger the I value. c is the perception operator, with a value of 0.01. a is 0.1.

[0065] ② Position update. During the foraging process, the butterfly will undergo global search and local search. The transition between global search and local search is defined by the switching probability p. The value of the switching probability p is generally random between [0,1]. When p≥r, the butterfly enters the global search phase, otherwise, it enters the local search phase. During the global search process, the butterfly individual moves towards the current global optimal position g. t Movement is given by:

[0066] x i,t+1 =x i,t +f1×(r1 2 ×g t -x i,t ) (11)

[0067] Where x i,t represents the position vector of the i-th butterfly in the t-th iteration; r1 is a random number between [0,1], representing a global random walk; f i Indicates the scent size of the i-th butterfly. During the local search process, the individual butterflies move randomly, and the local search update is given by the following formula:

[0068] x i,t+1 =x i,t +f2×(r2 2 ×x j,t -x k,t ) (12)

[0069] Where x k,t and x j,t represents the position vector of the kth and jth butterflies in the tth iteration; r2 is a random number between [0,1], representing a local random walk.

[0070] ③Fitness evaluation and update of the optimal solution. The updated hyperparameter combination is re-evaluated through the objective function. If the fitness of a butterfly individual is better than the current optimal solution, the optimal solution (optimal position best_pos and optimal fitness best_fitness) is updated.

[0071] ④Termination phase: When the maximum number of iterations (max_iter) is reached, the algorithm stops and returns the optimal hyperparameter combination (best_pos).

[0072] It should be noted that the optimal deep learning model (i.e., using BOA to Figure 2 The model that uses the architecture network for hyperparameter optimization is called the multimodal fusion BOA-SETCN model, while the model that does not use BOA for hyperparameter optimization is called the multimodal fusion SETCN model.

[0073] Step 4: Obtain the vibration signal of the bearing to be diagnosed and perform preprocessing on it as described in step 1. After preprocessing, the vibration signal is converted into a two-dimensional time-frequency image with time-frequency information, and the two-dimensional time-frequency image is reshaped into one-dimensional data; then, the preprocessed vibration signal and one-dimensional data are input into the fault diagnosis model obtained in step 3 to obtain the fault type of the bearing to be diagnosed.

[0074] At this point, the entire method is implemented.

[0075] The proposed multimodal fusion BOA_SETCN model is tested on two common public datasets in the field of bearing fault diagnosis. The SETCN model is used as a comparison model. The experimental results verify the effectiveness and advancement of the proposed model. The two datasets are the Case Western Reserve University (CWRU) dataset and the Southeast University Bearing Dataset (SEU). The two datasets are described as follows:

[0076] Dataset from Case Western Reserve University in the United States. This dataset contains bearing vibration data from different fault modes and working conditions. The bearing test bench at Case Western Reserve University includes a 1.5KW (2 horsepower) motor, a torque sensor, a power tester, and an electronic controller. The dataset contains bearing vibration signal data from four different fault conditions. The fault types are: normal state, inner ring fault, outer ring fault, and rolling element fault, and include faults under different drive ends, different sampling frequencies, different loads, and different diameters. This embodiment selects nine types of faults and normal states, a total of ten types of data, from the drive end data with a motor load of 0W, a bearing speed of 1797r / min, and a sampling frequency of 12KHz, according to different damage locations and different damage diameters. The description of the experimental dataset and the classification labels are shown in Table 1.

[0077] Table 1 CWRU dataset description and classification labels

[0078]

[0079] Southeast University bearing data set. The Southeast University test bench consists of a motor, motor controller, reduction gearbox, planetary gearbox, brake, and brake controller. This implementation uses a bearing data set with a speed load of 30Hz (1800rpm) and a load of 2V (7.32Nm). This data set contains data under five conditions: rolling element fault (Ball fault), inner ring fault (Inner ring fault), outer ring fault (Outer ring fault), combination fault (Combination fault on both inner ring and outer ring), and health state (Health woring state). The SEU fault category description is shown in Table 2.

[0080] Table 2 Description of SEU fault data categories

[0081]

[0082] Figure 3 is the continuous wavelet transform time-frequency diagram of the CWRU dataset state after CWT, Figure 4 It is the continuous wavelet transform time-frequency diagram of the SEU data set state after CWT, where ball, comb, health, inner, and outer represent the continuous wavelet transform time-frequency diagrams of rolling element fault data, composite fault data, health status data, inner race fault data, and outer race fault data, respectively. Figure 5 This is a graph showing the classification accuracy changes of the multimodal fusion BOA_SETCN model on the Case Western Reserve University dataset. Figure 6 The following is a graph showing the changes in the classification accuracy of the multimodal fusion BOA_SETCN model on the Southeast University bearing dataset. From these two figures, it can be seen that the classification accuracy of the BOA_SETCN model on both datasets is relatively high. Figure 7 This is the confusion matrix diagram of the multimodal fusion BOA_SETCN model on the Case Western Reserve University dataset in the United States. Figure 8 This is the confusion matrix diagram of the multimodal fusion BOA_SETCN model of the present invention on the Southeast University bearing dataset. From these two figures, it can be seen that the classification accuracy of the BOA_SETCN model on both datasets is relatively high. Figure 9 This is a comparison chart of the classification accuracy of the model of the present invention and the SETCN comparison model on the CWRU dataset. Figure 10 The following are comparison charts of the classification accuracy of the model of the present invention and the SETCN comparison model on the SEU dataset. From these two figures, it can be seen that with the increase of training rounds, the accuracy of the BOA_SETCN model is higher than that of the SETCN model.

[0083] An embodiment of a bearing fault diagnosis system based on multimodal fusion:

[0084] An embodiment of a multimodal fusion bearing fault diagnosis system according to the present invention includes a memory, a processor, an internal bus, and a computer program stored in the memory. The processor and memory communicate and exchange data with each other via the internal bus. Furthermore, as a preferred processing method, the system also includes a communication interface, a display screen, and an input device.

[0085] Among them, the processor is used to provide computing and control capabilities, execute the computer program to implement the steps of the method described in the embodiment of the multimodal fusion bearing fault diagnosis method of the present invention, and can be a processing device such as a microprocessor MCU, a programmable logic device FPGA. The memory can be a non-volatile storage medium or an internal memory. The non-volatile storage medium stores an operating system and a computer program, and the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the system is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the system can be a liquid crystal display screen or an electronic ink display screen. The input device of the system can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the system housing, or an external keyboard, touchpad or mouse.

[0086] In summary, the present invention addresses the problem that existing bearing fault diagnosis methods only use one-dimensional vibration signals or only use two-dimensional time-frequency signals as the input of the diagnosis model, making it difficult for the diagnosis model to effectively process signal features, resulting in poor diagnostic effect of the model. The present invention provides a multimodal fusion bearing fault diagnosis method, which extracts data from two modes of the bearing, namely the vibration signal, and the one-dimensional data obtained by first performing time-frequency conversion on the vibration signal and then reshaping the time-frequency conversion result, so as to fully extract the features of the input signal, improve the classification effect of the fault diagnosis model, and improve its generalization ability. Moreover, the present invention also proposes a new deep learning model, including TCN, SE attention mechanism and the final classification layer. The model can effectively capture the features of the input signal and improve the model performance; and the BOA algorithm is used to optimize the hyperparameters in TCN, which can further improve the performance of the model.

[0087] While specific embodiments have been described above, the present invention is not limited to the described embodiments. The fundamental concept of the present invention lies in the aforementioned basic scheme. Based on the teachings of the present invention, those skilled in the art can devise various variations of models, formulas, and parameters without inventive effort. Changes, modifications, substitutions, and variations to the embodiments without departing from the principles and spirit of the present invention remain within the scope of protection of the present invention.

Claims

1. A bearing fault diagnosis method based on multimodal fusion, characterized in that: The steps include: 1) Obtaining the vibration signal of the bearing to be diagnosed, converting the vibration signal into a two-dimensional time-frequency image with time-frequency information, and then reshaping the two-dimensional time-frequency image into one-dimensional data; 2) Inputting the vibration signal and the one-dimensional data into a fault diagnosis model to obtain the fault type of the bearing to be diagnosed; wherein the fault diagnosis model is obtained by training a deep learning model using training data.

2. The bearing fault diagnosis method based on multimodal fusion according to claim 1, characterized in that: The deep learning model includes TCN, SE attention mechanism and classification layer; TCN is used to perform deep feature extraction on the input of the deep learning model, the SE attention mechanism is used to capture key features of the input of the deep learning model and dynamically adjust channel weights, and the classification layer is used to classify faults based on the output of the SE attention mechanism and the output of the TCN.

3. The bearing fault diagnosis method based on multimodal fusion according to claim 2, characterized in that: The hyperparameters in the TCN are obtained by optimizing using a meta-heuristic optimization algorithm.

4. The bearing fault diagnosis method based on multimodal fusion according to claim 3 is characterized in that: The meta-heuristic optimization algorithm is the butterfly optimization algorithm.

5. The bearing fault diagnosis method based on multimodal fusion according to claim 1, characterized in that: The vibration signal is converted into the two-dimensional time-frequency image through continuous wavelet transform.

6. The bearing fault diagnosis method based on multimodal fusion according to claim 5, characterized in that: The wavelet mother function of the continuous wavelet transform is a complex Morlet wavelet.

7. The bearing fault diagnosis method based on multimodal fusion according to any one of claims 1 to 6, characterized in that: The vibration signal is a signal obtained by cutting and then enhancing the original collected vibration signal.

8. The bearing fault diagnosis method based on multimodal fusion according to claim 7, characterized in that: The enhancement includes overlap sampling processing.

9. The bearing fault diagnosis method based on multimodal fusion according to any one of claims 1 to 6, characterized in that: Fault types include roller fault, combined fault, healthy, inner race fault, and outer race fault.

10. A multi-modal fusion bearing fault diagnosis system, comprising a processor, characterized in that: The processor is configured to execute a computer program to implement the steps of the method according to any one of claims 1 to 9.

Images