Multi-modal bearing fault diagnosis method based on dynamic fusion feature-improved DS evidence theory

Abstract

The invention discloses a multi-modal bearing fault diagnosis method based on a dynamic fusion feature-improved DS evidence theory. According to the method, by introducing time domain, frequency domain and time-frequency domain data, three sub-models of a convolutional neural network, a long and short term memory network and a random forest are constructed for extracting and processing bearing fault features of different modes. The invention provides a feature-level dynamic fusion strategy, and features from different modes are gradually optimized, so that the performance of each sub-model in a fault classification process is improved. The method comprises the following steps of: obtaining a preliminary fault classification result of each sub-model through different feature level fusion methods, fusing diagnosis information of each sub-model by adopting an improved Dempster Shafer (DS) evidence theory, correcting evidence from two dimensions of fault types and model performance, and finally obtaining an accurate fault diagnosis result. The method has higher diagnosis accuracy and stronger robustness in a complex fault diagnosis scene.

Description

Technical Field

[0001] This invention relates to multimodal fault diagnosis, particularly for bearing fault diagnosis in mechanical systems. Specifically, this invention proposes a multimodal bearing fault diagnosis method based on feature-level dynamic fusion and decision-level improvement of Dempster-Shafer evidence theory. This method significantly improves the accuracy and robustness of fault diagnosis by fusing feature information from different sensors and data sources, especially demonstrating superior performance in complex mechanical fault detection tasks. Background Technology

[0002] Bearings, as key components in modern mechanical systems, play a vital role in supporting and transmitting mechanical motion. Their proper operation is crucial for the stability and reliability of mechanical equipment. However, due to long-term operation, overload, high temperatures, and other harsh working conditions, bearings often experience various types of failures, such as rolling element damage, inner and outer ring wear, and lubrication failure. Bearing failures not only lead to equipment downtime and increased maintenance costs but can also trigger serious mechanical failures or production accidents. Therefore, specific methods are susceptible to signal noise interference and cannot effectively capture the correlation between multi-source data. Compared to traditional single-data-source methods, multimodal methods can provide richer information, making timely and accurate bearing fault diagnosis an important means of ensuring the safe operation of mechanical equipment.

[0003] Traditional bearing fault diagnosis methods typically rely on single signal analysis techniques, such as frequency domain analysis, time domain analysis, and time-frequency domain analysis based on vibration signals. These methods identify fault characteristics by extracting and analyzing features from bearing vibration signals. However, traditional single-signal analysis methods have limitations when dealing with complex and variable environments and multimodal data. To achieve more accurate and comprehensive fault diagnosis, the key to multimodal fault diagnosis lies in effectively fusing data features from different modes to extract the most representative fault characteristics.

[0004] Dempster-Shafer Evidence Theory (DS Evidence Theory) is a mathematical framework for handling uncertainty and conflicting information, providing effective decision support in multimodal fault diagnosis. The core idea of ​​DS Evidence Theory is to treat information from different sources as evidence and synthesize this evidence to arrive at a final decision. In fault diagnosis tasks, each sub-model provides evidence through its diagnostic results on specific modal data, and these pieces of evidence are then fused into a comprehensive decision. However, traditional Dempster-Shafer Evidence Theory assumes that the evidence from all sub-models has the same credibility and importance when dealing with evidence from different sub-models, which may not hold true in practical applications. Because the performance of different sub-models can vary significantly across different fault types, a method is needed to measure and adjust the evidence weight of each sub-model to improve overall diagnostic performance.

[0005] To address the issue of poor accuracy and robustness of existing single-mode bearing fault diagnosis methods in complex fault scenarios, this invention proposes a multimodal bearing fault diagnosis method based on feature-level dynamic fusion and improved Dempster-Shafer evidence theory. Summary of the Invention

[0006] This invention provides a multimodal bearing fault diagnosis method that combines a feature-level dynamic fusion strategy with a decision-level improved Dempster-Shafer evidence theory. The method proceeds through four main steps:

[0007] Step 1: Dataset Preparation. The dataset used in the experiment is the Bearing Dataset (CWRU) from Case Western Reserve University in the United States. The proposed method is validated using the CWRU dataset, and then two different DS evidence theories are compared to verify the superiority of the improved DS evidence theory in multimodal data decision-level fusion. Currently, the bearing dataset from Case Western Reserve University is a representative dataset for fault diagnosis experiments. The bearings in the dataset operate under four loads (0hp, 1hp, 2hp, and 3hp) and four speeds (1797rpm, 1772rpm, 1750rpm, and 1730rpm). There are four single-point faults in the bearings: 0.007, 0.014, 0.021, and 0.028 inches (1 inch = 25.4mm). The data samples in this paper are from bearing vibration signals at a speed of 1730 r / min, and this paper selects data samples under 10 bearing operating conditions, subtly distinguishing the degree and type of different faults. This paper selects three different fault levels (0.007, 0.014, and 0.021 inches) and three different fault types (inner ring, outer ring, and ball bearing) for analysis, as well as normal bearing data. The experiment is conducted by dividing the dataset into a 60% training set, a 15% validation set, and a 25% test set, which is essentially a ten-class recognition problem.

[0008] Step Two: Sub-model Construction. Three models—Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM), and Random Forest (RF)—are selected as diagnostic sub-models for processing bearing data in the time domain, frequency domain, and time-frequency domain. A Convolutional Neural Network (CNN) is a deep learning model specifically designed to process data with network structures. The convolution operation is the core of a CNN; input data is processed through a convolution kernel, which performs sliding operations on local regions to calculate local features. The mathematical expression for the convolution operation is shown below:

[0009]

[0010] Where I(m,n) is the convolution kernel used to extract local features, K(im,jn) is the original two-dimensional data, (im,jn) indicates that the input data is calculated after being slid to a certain position by the convolution kernel, and S is the convolution result.

[0011] After the convolution operation, an activation function, such as ReLU (Rectified Linear Unit), is usually applied, and its mathematical expression is shown in Equation 2:

[0012] f(x)=max(0,x), (2)

[0013] Here, x is the input value, f(x) is the output value of the activation function, negative numbers are mapped to zero, and positive numbers remain unchanged.

[0014] Then, pooling is performed. Pooling is used for downsampling, thereby reducing computation and the number of parameters. Common pooling operations include max pooling and average pooling. The mathematical expression for max pooling is shown below:

[0015] Max Pooling(x) = max i,j (x i,j (3)

[0016] Where, x i，j It is the input value within the pooling region, and max represents the max pooling operation, which selects the maximum value within the pooling region as the output.

[0017] The mathematical expression for average pooling is shown in Formula 4:

[0018]

[0019] Where, x i，j is the input value within the pooling region, and h represents the size of the pooling window.

[0020] Through the convolutional operations described above, convolutional neural networks (CNNs) can capture local features of the input data. CNNs typically employ a hierarchical structure, including shallow feature extractors, deep feature extractors, and a classifier. The shallow feature extractor's one-dimensional convolutional layer has 8 filters, a kernel size of 32, and a stride of 1. After ReLU activation, the max-pooling layer has a pooling window of 2 and a stride of 2, and finally passes through a batch normalization layer and a flattening layer. The deep feature extractor's one-dimensional convolutional layer has 16 filters, a kernel size of 64, and a stride of 1. After ReLU activation, the max-pooling layer has a pooling window of 2 and a stride of 2. After batch normalization, the one-dimensional convolutional layer has 32 filters, a kernel size of 3, and after ReLU activation, the max-pooling layer has a pooling window of 2 and a stride of 2, and finally passes through a batch normalization layer and a flattening layer. The classifier's first fully connected layer has 256 neurons with the ReLU activation function, and the Dropout layer has a value of 0.3. The second fully connected layer has 128 neurons with the ReLU activation function, and the Dropout layer has a value of 0.3. The final fully connected layer has 64 neurons and uses the Softmax function.

[0021] Long Short-Term Memory (LSTM) networks are deep learning models used for processing sequential data. They consist of multiple LSTM units, each containing an input gate, a forget gate, and an output gate to control the flow of information and state updates. The core mechanism of LSTM lies in its unique unit state and gating mechanism, which enables it to effectively capture dependencies over long time spans. Specifically, the forget gate determines how much information in the current unit state Ct-1 is retained, as shown in Equation 5:

[0022] f t =σ(W f ·[h t-1 ,x t ]+b f ), (5)

[0023] Where σ represents the sigmoid activation function, W f and b f These are the weight matrix and the bias term, h. t-1 It is the hidden state of the previous time step, x t It is the input at the current time step, f t It is the output value of the forget gate.

[0024] The input gate determines the current input x. t How much information is saved to cell state C? t In the formula, the mathematical expression is shown in Formula 6-7:

[0025] i t =σ(W i ·[h t-1 ,x t ]+b i (6)

[0026]

[0027] Where σ represents the sigmoid activation function, tanh represents the hyperbolic tangent activation function, and W i W c and b i b c These are the weight matrix and the bias term, h. t-1 It is the hidden state of the previous time step, x t This is the input for the current time step. It is the candidate memory unit value, i t It is the output value of the input gate.

[0028] Before entering the output gate, the LSTM updates the current cell state by combining the outputs of the forget gate and the input gate, as shown in Equation 8:

[0029]

[0030] Among them, f t It is the output value of the forget gate, i t It is the output value of the input gate. It is a candidate memory unit value.

[0031] Finally, the information enters the output gate to determine how much information from the current cell state Ct is output to the hidden state ht, as shown in Equation 9-10:

[0032] o t =σ(W o ·[h t-1 ,x t ]+b o ), (9)

[0033] h t =o t *tanh(C t ), (10)

[0034] Where σ represents the sigmoid activation function, tanh represents the hyperbolic tangent activation function, and W o and b o These are the weight matrix and the bias term, h. t-1 It is the hidden state of the previous time step, x t It is the input for the current time step, h t It is the hidden state of the current time step, O t It is the output value of the output gate, which determines the output of the current cell state.

[0035] LSTM is trained layer by layer. First, the lowest-level LSTM units are trained. Then, the outputs of their hidden layers are used as inputs to the upper-level LSTM units, stacking these layers to form a deep structure. LSTM networks can extract and learn abstract features from the temporal dependencies of sequential data, helping to discover high-level representations in the data. The structure consists of an input layer, a bimodal LSTM encoder, a feature fusion module, fully connected layers, and an output layer. The input layer accepts two modal inputs obtained using Fast Fourier Transform (FFT) and Short-Time Fourier Transform (SFT). The bimodal LSTM encoder contains two parallel LSTM branches, each using two LSTM layers with 128 hidden units.

[0036] Random forests improve model accuracy and stability by combining the predictions of multiple decision trees. First, the dataset undergoes bootstrap sampling, where n samples are randomly drawn with replacement from the original dataset to form a new subset. Then, decision trees are trained, one on each subset. Each decision tree randomly selects a subset of features to determine the optimal split point when splitting at a node. Finally, an overall ensemble prediction is performed. For fault classification tasks, the final prediction of the random forest is the majority vote of all decision tree predictions, as shown in Equation 11.

[0037]

[0038] in, is the prediction result of the i-th decision tree, and B is the number of decision trees.

[0039] Suppose that feature j has an importance of I in the i-th tree. (i) j The mathematical expression for overall importance is shown in Formula 12:

[0040]

[0041] Among them, I (i) j Let represent the importance of feature j in the i-th tree, and B be the number of decision trees.

[0042] This random forest model contains 8 decision trees, each with a maximum depth of 3, to prevent overfitting.

[0043] Step 3: During feature-level fusion, dynamic weight adjustment can be applied to the concatenated features. By dynamically assigning weights based on the diagnostic accuracy or reliability of each modality, the degree (shallow or deep) and speed of each feature's participation in the fusion process are influenced. In the fusion of the three features across the three models, dynamic weights are introduced into the network to adjust the weight of each feature according to different levels (shallow or deep) and the speed of feature participation in the fusion process (feature update speed). A detailed dynamic adaptive fusion flowchart is shown below. Figure 1 As shown in Equation 13, specifically, given a representation of each modality feature, their dynamic weights are calculated using an attention network.

[0044] w i =σ(W i ·F i ), (13)

[0045] Among them, F iLet Wi represent the feature representation of the i-th feature (e.g., CNN, LSTM, RF), where Wi is the learned weight matrix, σ is the activation function, and w i This refers to the weights of the modal features, and the dynamically adjusted weights w. i This indicates the level of participation of each feature at the current moment.

[0046] The depth at which features are adjusted determines their level of influence within the network. Some features may contribute significantly in the initial feature extraction stage (shallow layer), while others may have a greater impact in deeper decision-making stages. We can adjust the participation level of each feature through different hierarchical structures. The formula for adjusting the weights of features at different levels is shown in Equation 14:

[0047]

[0048] Among them, W i shallow and W i deep These represent the weight matrices for shallow and deep features, respectively.

[0049] Feature fusion speed reflects the rate at which features are weighted and fused in the network. In some cases, feature fusion may require a high speed to adapt to changes in data, while in others, the fusion rate can be slower. The feature fusion speed can be adjusted by introducing a time step t. Assume that at time step t, the feature fusion rate r... i (t) The mathematical expression is shown in Formula 15:

[0050]

[0051] Where α is the adjustment coefficient, w i (t) is the dynamically adjusted weight, (t+1) -1 This indicates the feature fusion speed adjustment process, r i (t) is the fusion speed of the i-th feature at time step t.

[0052] The final fusion feature is a weighted fusion result of all modal features after dynamic adjustment. The final contribution of each modal feature is dynamically adjusted based on its participation level in shallow and deep layers and the changes over time. The mathematical expression for the final fusion feature is shown in Equation 16:

[0053]

[0054] Among them, w i shallow w i deep The product of Fi and r represents the weighted representation of shallow and deep features, respectively. i(t) is the feature fusion rate adjusted according to time step t.

[0055] Step 4: Decision-level fusion and improvement of Dempster-Shafer evidence theory. This paper proposes a dual-weighted confidence improvement method based on model performance and fault classification accuracy. The model performance weights are obtained using the confusion matrix of the fault diagnosis model. Assuming the fault classification problem is an n-class problem, the confusion matrix is ​​as follows:

[0056]

[0057] Where H is the confusion matrix of diagnostic results, a ij The number of samples of type i diagnosed as type j faults.

[0058] Based on the confusion matrix of the above diagnostic results, x can be calculated respectively. i and y j x i Let y be the total number of samples in class i. j The total number of samples diagnosed as class j is given by the formula shown in Equation 18-19:

[0059]

[0060] P can be calculated using the confusion matrix of the diagnostic results and the total number of the two types of samples mentioned above. sum and P expectation , where P sum P represents the overall accuracy of fault diagnosis. expectation The formula for the expected accuracy of fault diagnosis is shown in Equation 20-21:

[0061]

[0062] The overall fault diagnosis accuracy P calculated above sum And the expected accuracy of fault diagnosis P expectation The expression for the model similarity coefficient Q can be defined as shown in Formula 22:

[0063]

[0064] The model similarity coefficient Q calculated by the above formula is used to measure the similarity of diagnostic results after different sub-models process different modal data. When the confusion matrices of the diagnostic results between sub-models differ too much, it indicates that some sub-models have a greater possibility of overfitting, and the value of the model similarity coefficient Q will decrease. By superimposing the confusion matrices of the two sub-models, the model similarity coefficient Q between the two models can be calculated. ijThen, Q represents the model similarity coefficient between the i-th piece of evidence and the j-th piece of evidence, where each sub-model corresponds to one piece of evidence. By calculating the model similarity between different models, we can determine the model performance weight, one of the two weights used to improve the traditional DS theory, as shown in Formula 23.

[0065]

[0066] Furthermore, considering the different performance of the same or different models on different fault types, the fault classification accuracy of each fault type is obtained through the confusion matrix of the diagnostic results. Then, the fault classification accuracy weight, one of the two weights used to improve the traditional DS theory, is calculated, as shown in Formula 24:

[0067]

[0068] Where accuracy(i,j) is the accuracy of the i-th model in diagnosing the j-th type of fault, w i，j The fault classification accuracy weights are assigned to the i-th model for diagnosing the j-th type of fault.

[0069] The original evidence is improved based on the model performance weights and fault classification accuracy weights calculated above, as shown in Formula 25:

[0070]

[0071] In summary, the multimodal data bearing fault diagnosis process based on the improved DS evidence theory is as follows: Figure 2 As shown.

[0072] The beneficial effects of this invention are as follows:

[0073] (1) Improve the accuracy of fault diagnosis: Through dynamic fusion strategy, data from different modalities can be effectively integrated to enhance the system’s ability to identify complex faults.

[0074] (2) Enhanced robustness: The improved Dempster-Shafer evidence theory can effectively solve the diagnostic error caused by model overfitting and improve the overall diagnostic performance, especially in complex and noisy environments.

[0075] (3) Optimize intermodal fusion: dynamically adjust the weighting of each modal feature so that the model can better adapt to different fault types and improve the diagnostic effect of bearing faults under different working conditions.

[0076] (4) Wide applicability: The method of the present invention is not only applicable to bearing fault diagnosis, but can also be extended to other mechanical systems or fault detection tasks involving multimodal sensor fusion. Attached Figure Description

[0077] Figure 1 Flowchart of dynamic feature fusion

[0078] Figure 2 Flowchart for Multimodal Data Bearing Fault Diagnosis

[0079] Figure 3 A comparison chart of the number of correct and incorrect samples in each category of traditional DS and improved DS evidence theories at the decision-level.

[0080] Figure 4 A comparison chart of the number of correct and incorrect samples in each category of the traditional DS evidence theory at the decision level, showcasing the fusion of static, progressive, and dynamic characteristics.

[0081] Figure 5 A comparison chart of the number of correct and incorrect samples in each category of the decision-level improved DS evidence theory, which integrates static, progressive, and dynamic features. Detailed Implementation

[0082] This invention provides a multimodal bearing fault diagnosis method that combines a feature-level dynamic fusion strategy with a decision-level improved Dempster-Shafer evidence theory. The method proceeds through three main steps:

[0083] Step 1: Sub-model construction. Three models—Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM), and Random Forest (RF)—are selected as diagnostic sub-models for processing bearing data in the time domain, frequency domain, and time-frequency domain. A Convolutional Neural Network (CNN) is a deep learning model specifically designed to process data with network structures. The convolution operation is the core of a CNN; input data is processed through a convolution kernel, which performs sliding operations on local regions to calculate local features. The mathematical expression for the convolution operation is shown below:

[0084]

[0085] Where I(m,n) is the convolution kernel used to extract local features, K(im,jn) is the original two-dimensional data, (im,jn) indicates that the input data is calculated after being slid to a certain position by the convolution kernel, and S is the convolution result.

[0086] After the convolution operation, an activation function, such as ReLU (Rectified Linear Unit), is usually applied, and its mathematical expression is shown in Equation 2:

[0087] f(x)=max(0,x), (2)

[0088] Here, x is the input value, f(x) is the output value of the activation function, negative numbers are mapped to zero, and positive numbers remain unchanged.

[0089] Then, pooling is performed. Pooling is used for downsampling, thereby reducing computation and the number of parameters. Common pooling operations include max pooling and average pooling. The mathematical expression for max pooling is shown below:

[0090] Max Pooling(x) = max i,j (x i,j (3)

[0091] Where, x i，j It is the input value within the pooling region, and max represents the max pooling operation, which selects the maximum value within the pooling region as the output.

[0092] The mathematical expression for average pooling is shown in Formula 4:

[0093]

[0094] Where, x i，j is the input value within the pooling region, and h represents the size of the pooling window.

[0095] Through the above convolution operations, convolutional neural networks can capture local features of input data. Furthermore, convolutional neural networks typically employ a hierarchical structure, including convolutional layers, activation function layers, and pooling layers, which helps to extract and learn abstract features of data layer by layer.

[0096] Long Short-Term Memory (LSTM) networks are deep learning models used for processing sequential data. They consist of multiple LSTM units, each containing an input gate, a forget gate, and an output gate to control the flow of information and state updates. The core mechanism of LSTM lies in its unique unit state and gating mechanism, which enables it to effectively capture dependencies over long time spans. Specifically, the forget gate determines how much information in the current unit state Ct-1 is retained, as shown in Equation 5.

[0097] f t =σ(W f ·[h t-1 ,x t ]+b f ), (5)

[0098] Where σ represents the sigmoid activation function, W f and b f These are the weight matrix and the bias term, h. t-1 It is the hidden state of the previous time step, x t It is the input at the current time step, f t It is the output value of the forget gate.

[0099] The input gate determines the current input x. tHow much information is saved to cell state C? t In the formula, the mathematical expression is shown in Formula 6-7:

[0100] i t =σ(W i ·[h t-1 ,x t ]+b i (6)

[0101]

[0102] Where σ represents the sigmoid activation function, tanh represents the hyperbolic tangent activation function, and W i W c and b i b c These are the weight matrix and the bias term, h. t-1 It is the hidden state of the previous time step, x t This is the input for the current time step. It is the candidate memory unit value, i t It is the output value of the input gate.

[0103] Before entering the output gate, the LSTM updates the current cell state by combining the outputs of the forget gate and the input gate, as shown in Equation 8:

[0104]

[0105] Among them, f t It is the output value of the forget gate, i t It is the output value of the input gate. It is a candidate memory unit value.

[0106] Finally, the information enters the output gate to determine how much information from the current cell state Ct is output to the hidden state ht, as shown in Equation 9-10:

[0107] o t =σ(W o ·[h t-1 ,x t ]+b o ), (9)

[0108] h t =o t *tanh(C t ), (10)

[0109] Where σ represents the sigmoid activation function, tanh represents the hyperbolic tangent activation function, and W o and b o These are the weight matrix and the bias term, h.t-1 It is the hidden state of the previous time step, x t It is the input for the current time step, h t It is the hidden state of the current time step, O t It is the output value of the output gate, which determines the output of the current cell state.

[0110] LSTM is trained layer by layer. First, the lowest-level LSTM unit is trained, and then the output of its hidden layers is used as the input to the upper-level LSTM units, stacking these layers to form a deep structure. LSTM networks can extract and learn abstract features from the temporal dependencies of sequence data layer by layer, helping to discover high-level representations in the data.

[0111] Random forests improve model accuracy and stability by combining the predictions of multiple decision trees. First, the dataset undergoes bootstrap sampling, where n samples are randomly drawn with replacement from the original dataset to form a new subset. Then, decision trees are trained, one on each subset. Each decision tree randomly selects a subset of features to determine the optimal split point when splitting at a node. Finally, an overall ensemble prediction is performed. For fault classification tasks, the final prediction of the random forest is the majority vote of all decision tree predictions, as shown in Equation 11.

[0112]

[0113] in, is the prediction result of the i-th decision tree, and B is the number of decision trees.

[0114] Suppose that feature j has an importance of I in the i-th tree. (i) j The mathematical expression for overall importance is shown in Formula 12:

[0115]

[0116] Among them, I (i) j Let represent the importance of feature j in the i-th tree, and B be the number of decision trees.

[0117] Step Two: During feature-level fusion, dynamic weight adjustment can be applied to the concatenated features. By dynamically assigning weights based on the diagnostic accuracy or reliability of each modality, the degree (shallow or deep) and speed of each feature's participation in the fusion process are influenced. In the fusion of the three features across the three models, dynamic weights are introduced into the network to adjust the weights of each feature according to different levels (shallow or deep) and the speed of feature participation in the fusion process (feature update rate). A detailed dynamic adaptive fusion flowchart is shown below. Figure 1 As shown in Equation 13, specifically, given a representation of each modality feature, their dynamic weights are calculated using an attention network.

[0118] w i =σ(W i ·F i ), (13)

[0119] Among them, F i Let Wi represent the feature representation of the i-th feature (e.g., CNN, LSTM, RF), where Wi is the learned weight matrix, σ is the activation function, and w i This refers to the weights of the modal features, and the dynamically adjusted weights w. i This indicates the level of participation of each feature at the current moment.

[0120] The depth at which features are adjusted determines their level of influence within the network. Some features may contribute significantly in the initial feature extraction stage (shallow layer), while others may have a greater impact in deeper decision-making stages. We can adjust the participation level of each feature through different hierarchical structures. The formula for adjusting the weights of features at different levels is shown in Equation 14:

[0121]

[0122] Among them, W i shallow and W i deep These represent the weight matrices for shallow and deep features, respectively.

[0123] Feature fusion speed reflects the rate at which features are weighted and fused in the network. In some cases, feature fusion may require a high speed to adapt to changes in data, while in others, the fusion rate can be slower. The feature fusion speed can be adjusted by introducing a time step t. Assume that at time step t, the feature fusion rate r... i (t) The mathematical expression is shown in Formula 15:

[0124]

[0125] Where α is the adjustment coefficient, w i (t) is the dynamically adjusted weight, (t+1) -1 This indicates the feature fusion speed adjustment process, r i (t) is the fusion speed of the i-th feature at time step t.

[0126] The final fusion feature is a weighted fusion result of all modal features after dynamic adjustment. The final contribution of each modal feature is dynamically adjusted based on its participation level in shallow and deep layers and the changes over time. The mathematical expression for the final fusion feature is shown in Equation 16:

[0127]

[0128] Among them, w i shallow w i deep The product of Fi and r represents the weighted representation of shallow and deep features, respectively. i (t) is the feature fusion rate adjusted according to time step t.

[0129] Step 3: Decision-level fusion and improvement of Dempster-Shafer evidence theory. This paper proposes a dual-weighted confidence improvement method based on model performance and fault classification accuracy. The model performance weights are obtained using the confusion matrix of the fault diagnosis model. Assuming the fault classification problem is an n-class problem, the confusion matrix is ​​as follows:

[0130]

[0131] Where H is the confusion matrix of diagnostic results, a ij The number of samples of type i diagnosed as type j faults.

[0132] Based on the confusion matrix of the above diagnostic results, x can be calculated respectively. i and y j x i Let y be the total number of samples in class i. j The total number of samples diagnosed as class j is given by the formula shown in Equation 18-19:

[0133]

[0134] P can be calculated using the confusion matrix of the diagnostic results and the total number of the two types of samples mentioned above. sum and P expectation , where P sum P represents the overall accuracy of fault diagnosis. expectation The formula for the expected accuracy of fault diagnosis is shown in Equation 20-21:

[0135]

[0136] The overall fault diagnosis accuracy P calculated above sum And the expected accuracy of fault diagnosis P expectation The expression for the model similarity coefficient Q can be defined as shown in Formula 22:

[0137]

[0138] The model similarity coefficient Q calculated by the above formula is used to measure the similarity of diagnostic results after different sub-models process different modal data. When the confusion matrices of the diagnostic results between sub-models differ too much, it indicates that some sub-models have a greater possibility of overfitting, and the value of the model similarity coefficient Q will decrease. By superimposing the confusion matrices of the two sub-models, the model similarity coefficient Q between the two models can be calculated. ij Then, Q represents the model similarity coefficient between the i-th piece of evidence and the j-th piece of evidence, where each sub-model corresponds to one piece of evidence. By calculating the model similarity between different models, we can determine the model performance weight, one of the two weights used to improve the traditional DS theory, as shown in Formula 23.

[0139]

[0140] Furthermore, considering the different performance of the same or different models on different fault types, the fault classification accuracy of each fault type is obtained through the confusion matrix of the diagnostic results. Then, the fault classification accuracy weight, one of the two weights used to improve the traditional DS theory, is calculated, as shown in Formula 24:

[0141]

[0142] Where accuracy(i,j) is the accuracy of the i-th model in diagnosing the j-th type of fault, w i，j The fault classification accuracy weights are assigned to the i-th model for diagnosing the j-th type of fault.

[0143] The original evidence is improved based on the model performance weights and fault classification accuracy weights calculated above, as shown in Formula 25:

[0144]

[0145] In summary, the multimodal data bearing fault diagnosis process based on the improved DS evidence theory is as follows: Figure 2 As shown.

[0146] To further clarify the technical solution of the present invention, a detailed description is provided below through an embodiment. Assume we used a bearing failure dataset provided by Case Western Reserve University (CWRU) for our experiment. The dataset contains bearing vibration signals under different load and speed conditions, and the signals include three modes: time domain, frequency domain, and time-frequency domain. The specific steps are as follows:

[0147] Step 1: Constructing sub-models. First, we use Convolutional Neural Network (CNN) to process time-domain and time-frequency domain data, Long Short-Term Memory Network (LSTM) to process frequency domain data, and Random Forest (RF) to process features of all modalities, which can filter out the two modalities that different models are best at.

[0148] Step 2: Feature-level dynamic fusion is achieved by introducing a dynamic fusion strategy, dynamically adjusting the fusion weights at each time step based on the performance of different modalities. These sub-models extract features from the input signals of their respective preferred modalities. Features from different modalities are then fused using weighted averaging and dynamic fusion. The fusion strategy is optimized based on the diagnostic performance of each modality. Finally, different feature fusion methods and strategies are used to select the best-performing feature-level fusion method. Feature fusion methods include concat fusion, weighted fusion, and average fusion. Feature fusion strategies include static fusion, progressive fusion, and dynamic fusion. Experimental results are shown in Table 1 below.

[0149] Accuracy / % CNN LSTM RF SF-CF 0.9366 0.9500 0.9233 SF-WF 0.9033 0.9233 0.9133 SF-AF 0.9033 0.9400 0.9366 PF-CF 0.9300 0.9500 0.8967 PF-WF 0.9267 0.9533 0.8900 PF-AF 0.9333 0.8933 0.8833 DF-CF 0.9300 0.9367 0.9400 DF-WF 0.9666 0.9667 0.9366 DF-AF 0.9100 0.9700 0.9600

[0150] Table 1: Comparison Results of Experiments with Different Feature Fusion Methods and Strategies. Through Table 1, we obtained the fault diagnosis confusion matrix of each model after performing fault diagnosis through different feature fusion methods and steps, and the best performance of each model in each same feature fusion strategy and different feature fusion methods.

[0151] Step 3: After feature fusion at the decision level, the diagnostic results from different sub-models are fused using an improved Dempster-Shafer evidence theory. This method calculates the performance weight of each sub-model and dynamically adjusts the evidence weights, considering its diagnostic accuracy for different fault types, to obtain the final fault diagnosis result. The experimental results are shown in Table 2 below.

[0152] <![CDATA[λ P / %]]> λr / % <![CDATA[λ F1 ]]> σ DT-DS 91.86 91.79 91.82 0.08 DI-DS 92.32 92.02 92.17 0.08 SF-DT-DS 93.96 93.43 93.69 0.07 PF-DT-DS 91.52 89.72 90.61 0.10 DF-DT-DS 96.32 95.82 96.07 0.04 SF-DI-DS 94.78 94.22 94.50 0.06 PF-DI-DS 93.11 92.79 92.95 0.07 DF-DI-DS 96.62 96.56 96.59 0.03

[0153] Table 2: Comparative Analysis of Traditional and Improved DS Theories in Static, Gradual, and Dynamic Fusion Characteristics – Decision-Level Fusion Methods

[0154] After performing decision-level fusion on the three confusion matrices obtained from each feature fusion strategy, Table 2 shows that when using traditional DS evidence theory fusion, dynamic fusion outperforms static and progressive fusion methods, while also exhibiting a lower conflict coefficient, ensuring the model maintains good robustness while achieving a high evaluation index. Overall, Table 2 indicates that the feature-decision-level fusion method using dynamic fusion and improved DS evidence theory performs best, achieving the highest evaluation index while minimizing the conflict coefficient.

[0155] Step 4: Train and validate on the CWRU bearing fault dataset. Experimental results are as follows: Figure 3 Figure 4 Figure 5 As shown, the improved DS evidence theory has a significantly higher number of correct samples in each category than the traditional DS evidence theory, fully demonstrating the superiority of the improved method. Under the dynamic fusion strategy, the number of correct samples in each category is significantly higher than that of the static and progressive fusion strategies. This indicates that the dynamic fusion strategy can more fully mine and utilize the complementary information between different modalities when processing multimodal data, thereby achieving higher diagnostic accuracy and lower conflict coefficients. Furthermore, the improved DS evidence theory using dynamic fusion feature decision level also significantly outperforms the traditional DS evidence theory using dynamic fusion feature decision level in terms of the number of correct samples in each category. This further verifies the significant advantages of combining the dynamic fusion strategy with the improved DS evidence theory in solving multimodal fault diagnosis problems.

Claims

1. A multimodal bearing fault diagnosis method based on dynamic fusion features and improved DS evidence theory, characterized in that, It includes the following four steps: Step 1: Dataset preparation. Three different sizes, three different fault levels, and three different fault types (inner ring, outer ring, and ball bearing) were selected for analysis, along with normal bearing data. The dataset was divided into training, validation, and test sets for experiments. Essentially, this is a ten-class recognition problem. Step 2: Sub-model construction. Three models—Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM), and Random Forest (RF)—are selected as diagnostic sub-models for processing bearing data in the time domain, frequency domain, and time-frequency domain. The mathematical expression for the convolution operation is shown below: Where I(m,n) is the convolution kernel used to extract local features, K(im,jn) is the original two-dimensional data, (im,jn) means that the input data is calculated after being slid to a certain position by the convolution kernel, and S is the convolution result; After the convolution operation, an activation function is applied, the mathematical expression of which is shown in Equation 2: f(x)=max(0,x), (2) Where x is the input value, f(x) is the output value of the activation function, negative numbers are mapped to zero, and positive numbers remain unchanged; Then, a pooling operation is performed. This pooling operation is used for downsampling. The mathematical expression for max pooling is shown below: Max Pooling(x)=max i,j (x i,j ), (3) Where, x i，j is the input value within the pooling region, and max represents the max pooling operation, which selects the maximum value in the pooling region as the output. The mathematical expression for average pooling is shown in Formula 4: Where, x i，j is the input value within the pooling region, and h represents the size of the pooling window; Convolutional neural networks employ a hierarchical structure, including shallow feature extractors, deep feature extractors, and a classifier. The shallow feature extractor's one-dimensional convolutional layer has 8 filters, a kernel size of 32, and a stride of 1. After ReLU activation, the max-pooling layer has a pooling window of 2 and a stride of 2, followed by batch normalization and flattening layers. The deep feature extractor's one-dimensional convolutional layer has 16 filters, a kernel size of 64, and a stride of 1. After ReLU activation, the max-pooling layer has a pooling window of 2 and a stride of [missing information].

2. After batch normalization, the one-dimensional convolutional layer has 32 filters and a kernel size of 3. After ReLU activation, the pooling window of the max pooling layer is 2 and the stride is 2. Finally, it passes through a batch normalization layer and a flattening layer. The first fully connected layer of the classifier has 256 neurons with ReLU activation and a dropout layer with a value of 0.

3. The second fully connected layer has 128 neurons with ReLU activation and a dropout layer with a value of 0.

3. The final fully connected layer has 64 neurons and passes through a softmax function. Long Short-Term Memory (LSTM) networks are deep learning models used for processing sequence data. They consist of multiple LSTM units, each containing an input gate, a forget gate, and an output gate. The forget gate determines how much information from the current unit state Ct-1 is retained, as shown in Equation 5. f t =σ(W f ·[h t-1 ,x t ]+b f ), (5) Where σ represents the sigmoid activation function, W f and b f These are the weight matrix and the bias term, h. t-1 It is the hidden state of the previous time step, x t It is the input at the current time step, f t It is the output value of the forget gate; The input gate determines the current input x. t How much information is saved to cell state C? t In this context, the mathematical expression is shown in formula 6-7: i t =σ(W i ·[h t-1 ,x t ]+b i ), (6) Where σ represents the sigmoid activation function, tanh represents the hyperbolic tangent activation function, and W i W c and b i b c These are the weight matrix and the bias term, h. t-1 It is the hidden state of the previous time step, x t This is the input for the current time step. It is the candidate memory unit value, i t It is the output value of the input gate; Before entering the output gate, the LSTM updates the current cell state by combining the outputs of the forget gate and the input gate, as shown in Equation 8: Among them, f t It is the output value of the forget gate, i t It is the output value of the input gate. These are candidate memory unit values; Finally, the information enters the output gate to determine how much information from the current cell state Ct is output to the hidden state ht, as shown in Equation 9-10: the t =σ(W o ·[h t-1 ,x t ]+b o ), (9) h t = no t *fish(C) t ), (10) Where σ represents the sigmoid activation function, tanh represents the hyperbolic tangent activation function, and W o and b o These are the weight matrix and the bias term, h. t-1 It is the hidden state of the previous time step, x t It is the input for the current time step, h t It is the hidden state at the current time step, O t The output value of the output gate determines the output of the current cell state; LSTM is trained layer by layer. First, the bottom-level LSTM unit is trained, and then the output of its hidden layer is used as the input of the upper-level LSTM unit. The layers are stacked to form a deep structure. The structure consists of an input layer, a dual-modal LSTM encoder, a feature fusion module, a fully connected layer, and an output layer. The input layer first accepts two modal inputs obtained by Fast Fourier Transform and Short-Time Fourier Transform. The dual-modal LSTM encoder contains two parallel LSTM branches, each of which uses 2 layers of LSTM with 128 hidden units. Random forests first perform bootstrap sampling on the dataset, that is, randomly drawing n samples with replacement from the original dataset to form a new subset; then, they train decision trees, training one decision tree on each subset. When splitting a node, each decision tree randomly selects a subset of features to determine the optimal split point; finally, they perform overall ensemble prediction. For fault classification tasks, the final prediction of random forests is the majority vote of all the predictions from the decision trees, as shown in Equation 11: in, is the prediction result of the i-th decision tree, and B is the number of decision trees; Suppose that feature j has an importance of I in the i-th tree. (i) j The mathematical expression for overall importance is shown in Formula 12: Among them, I (i) j Let J be the importance of feature j in the i-th tree, and B be the number of decision trees; This random forest model contains 8 decision trees, each with a maximum depth of 3; Step 3: Given the representation of each modality feature, calculate their dynamic weights using an attention network, as shown in Equation 13: w i =σ(W i ·F i ), (13) Among them, F i Let Wi represent the feature representation of the i-th feature (e.g., CNN, LSTM, RF), where Wi is the learned weight matrix, σ is the activation function, and w i This refers to the weights of the modal features, and the dynamically adjusted weights w. i This indicates the level of participation of each feature at the current moment; The formula for adjusting the weights of features at different levels is shown in Equation 14: Among them, W i shallow and W i deep These represent the weight matrices for shallow and deep features, respectively. Feature fusion speed reflects the rate at which features are weighted and fused in the network. In some cases, feature fusion may require a higher speed to adapt to changes in data, while in other cases, the fusion rate can be slower. The feature fusion speed can be adjusted by introducing a time step t. Assuming that at time step t, the feature fusion rate r... i (t) The mathematical expression is shown in Formula 15: Where α is the adjustment coefficient, w i (t) is the dynamically adjusted weight, (t+1) -1 This indicates the feature fusion speed adjustment process, r i (t) is the fusion speed of the i-th feature at time step t; The final fusion feature is a weighted fusion result of all modal features after dynamic adjustment; the final contribution of each modal feature will be dynamically adjusted according to its participation in shallow and deep layers and the changes in time step; the mathematical expression of the final fusion feature is shown in Equation 16: Among them, w i shallow w i deep The product of Fi and r represents the weighted representation of shallow and deep features, respectively. i (t) is the feature fusion rate adjusted according to time step t; Step 4: Set the fault classification problem as an n-class problem, and the confusion matrix is ​​as follows: Where H is the confusion matrix of diagnostic results, a ij The number of samples of type i diagnosed as type j faults; Based on the confusion matrix of the above diagnostic results, x is calculated respectively. i and y j x i Let y be the total number of samples in class i. j The total number of samples diagnosed as class j is given by the formula shown in Equation 18-19: P can be calculated using the confusion matrix of the diagnostic results and the total number of the two types of samples mentioned above. sum and P expectation , where P sum P represents the overall accuracy of fault diagnosis. expectation The formula for the expected accuracy of fault diagnosis is shown in Equation 20-21: The overall fault diagnosis accuracy P calculated above sum And the expected accuracy of fault diagnosis P expectation The expression for the model similarity coefficient Q can be defined as shown in Formula 22: The model similarity coefficient Q calculated by the above formula is used to measure the similarity of diagnostic results after different sub-models process different modal data. ij Then, Q is the model similarity coefficient between the i-th piece of evidence and the j-th piece of evidence, where each sub-model corresponds to one piece of evidence. By calculating the model similarity between different models as described above, we can calculate the model performance weight, one of the two weights used to improve the traditional DS theory, as shown in Formula 23. Furthermore, considering the different performance of the same or different models on different fault types, the fault classification accuracy of each fault type is obtained through the confusion matrix of the diagnostic results. Then, the fault classification accuracy weight, one of the two weights used to improve the traditional DS theory, is calculated, as shown in Formula 24: Where accuracy(i,j) is the accuracy of the i-th model in diagnosing the j-th type of fault, w i，j The fault classification accuracy weights for the i-th model in diagnosing the j-th type of fault; The original evidence is improved based on the model performance weights and fault classification accuracy weights calculated above, as shown in Formula 25:

Images