Rolling bearing diagnosis system and method based on Mama multi-modal fusion

Abstract

The invention discloses a rolling bearing diagnosis system and method based on Mama multi-modal fusion, and the method comprises the steps: S1, obtaining the vibration data of a rolling bearing, carrying out the preprocessing, and outputting a data set; s2, establishing a rolling bearing diagnosis system based on Mamba multi-modal fusion; s3, processing the long-sequence vibration signal by using a time sequence branch, extracting a nonlinear dynamic characteristic and a global time sequence dependency relationship, and outputting a time sequence characteristic; s4, processing the time-frequency image by using image branches, and extracting multi-dimensional time-frequency features through bidirectional state space modeling of a time axis and a frequency axis; and S5, fusing the time sequence features output by the time sequence branch and the time frequency features output by the image branch through a feature fusion module, and outputting a rolling bearing fault classification result. By adopting the system and the method, the efficiency and the accuracy of fault diagnosis of the rolling bearing can be effectively improved, and a practical solution is provided for real-time bearing fault diagnosis in an industrial environment.

Description

Technical Field

[0001] This invention relates to the field of rolling bearing fault diagnosis technology, and in particular to a rolling bearing diagnostic system and method based on Mamba multimodal fusion. Background Technology

[0002] The health of rolling bearings is directly related to the safety and availability of high-speed rotating equipment such as aircraft engines, wind turbines, and high-speed trains. Delayed or misdiagnosed fault detection can easily lead to unplanned downtime and high maintenance costs. Therefore, achieving real-time, accurate, and robust diagnosis of long-term vibration signals under complex operating conditions remains a cutting-edge engineering challenge that urgently needs to be overcome.

[0003] Existing diagnostic methods are mainly divided into two categories: model-based and data-driven. The former relies on domain expertise to extract physical features, which is easily limited in scenarios with changing operating conditions or unknown failure modes. The latter uses machine learning to automatically learn discriminative features, reducing reliance on professional knowledge, but still has significant technical shortcomings. In the field of deep learning, while models such as CNN and LSTM perform well in specific scenarios, the Transformer architecture suffers from O(L) latency. 2 The quadratic complexity of multimodal data leads to a multiplied computational burden in multimodal scenarios, making it difficult to meet the requirements of millisecond-level real-time response. Multimodal data generally suffers from structural heterogeneity, asynchronous sampling, and signal-to-noise ratio differences. Existing fixed-weight fusion strategies are susceptible to interference from low-quality modes, resulting in insufficient fusion efficiency and reliability. At the same time, traditional linear methods are unable to capture the weak nonlinear characteristics of early faults, resulting in insufficient sensitivity for weak fault detection, which severely restricts their diagnostic performance in industrial scenarios.

[0004] Therefore, there is an urgent need to propose an innovative method to break through the existing technical bottlenecks, simultaneously improve the diagnostic accuracy and computational efficiency of the model, and provide a practical and feasible solution for real-time fault diagnosis of rolling bearings in industrial scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a rolling bearing diagnostic system and method based on Mamba multimodal fusion. By combining the linear complexity of the Mamba model, adaptive multimodal fusion, and chaos enhancement, the efficiency and accuracy of rolling bearing fault diagnosis can be effectively improved. While maintaining high precision, it can achieve millisecond-level real-time inference, providing a practical solution for real-time bearing fault diagnosis in industrial environments.

[0006] To achieve the above objectives, the present invention provides a rolling bearing diagnostic system based on Mamba multimodal fusion, including a time series branch, an image branch, and a feature fusion module; The time series branch is used to process the preprocessed long-sequence vibration signal, extract the nonlinear dynamic features and global time series dependencies in the signal, and output the time series features. The image branch is used to process the preprocessed time-frequency image converted from the vibration signal, and extracts multi-dimensional time-frequency features through bidirectional state space modeling of the time axis and frequency axis; The feature fusion module is used to receive and adaptively fuse the time-series features output by the time-series branch with the time-frequency features output by the image branch, and output the rolling bearing fault classification result.

[0007] A rolling bearing diagnostic method based on Mamba multimodal fusion is also provided, the steps of which include: S1. Acquire rolling bearing vibration data, preprocess the vibration data, and output a dataset containing long-sequence vibration signals and time-frequency images; S2. Establish a rolling bearing diagnostic system based on Mamba multimodal fusion as described above; S3. Use time series branching to preprocess the long sequence vibration signal, extract the nonlinear dynamic features and global time series dependencies in the signal, and output the time series features. S4. The time-frequency image converted from the vibration signal after image branching is preprocessed and multi-dimensional time-frequency features are extracted by bidirectional state space modeling of the time axis and frequency axis. S5. The feature fusion module receives and adaptively fuses the time-series features output by the time-series branch with the time-frequency features output by the image branch, and outputs the rolling bearing fault classification result.

[0008] Therefore, the rolling bearing diagnostic system and method based on Mamba multimodal fusion described above, as used in this invention, has the following beneficial effects: (1) Improved inference efficiency and deployability: The linear complexity Mamba model is used to optimize long sequence signal processing, and a simplified model structure is used to achieve efficient real-time inference and adapt to the deployment needs of industrial sites.

[0009] (2) Enhanced fault detection sensitivity: Through dual-branch modeling and feature enhancement mechanism, the signal timing, time frequency and nonlinear dynamic information are fully explored, which greatly improves the ability to identify early weak faults.

[0010] (3) Optimal diagnostic performance and fusion efficiency: The dynamic low-rank multi-head attention mechanism achieves efficient fusion of cross-modal features and outperforms existing methods on multiple datasets, balancing diagnostic accuracy and running efficiency.

[0011] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0012] Figure 1 This is a flowchart of a rolling bearing diagnostic method based on Mamba multimodal fusion according to an embodiment of the present invention; Figure 2 This is a diagram illustrating the overall architecture of the Mamba TFVisionChaos model according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the conversion of time-series data to time-frequency images according to an embodiment of the present invention. Figure 4 This is a confusion matrix diagram of the CWRU dataset in Embodiment 1 of the present invention; Figure 5 This is a confusion matrix diagram of the XJTU-SY dataset in Embodiment 3 of the present invention; Figure 6 This is a confusion matrix diagram of the HIT dataset in Embodiment 4 of the present invention. Detailed Implementation

[0013] Example like Figure 1 As shown, this invention discloses a rolling bearing diagnostic method based on Mamba multimodal fusion, the steps of which include: S1. Acquire rolling bearing vibration data and preprocess the vibration data, including: S11. Select four publicly available datasets for rolling bearing fault diagnosis: CWRU dataset, JNU dataset, XJTU-SY dataset, and HIT dataset. Divide the datasets at the level of raw signals. Distribute the vibration signals corresponding to different working conditions or different acquisition files to non-overlapping training, validation, and test sets according to a preset ratio to ensure uniform data distribution and verify the model's generalization ability.

[0014] S12. Perform uniform length truncation on each original vibration signal and in-sample normalization on each data subset to eliminate the influence of dimensions. The formula is: in, This is the original long sequence of vibration signals. The signal time-series average. The standard deviation of the signal timing. To prevent the minimum value of denominator 0 and ; S13. After completing the signal-level data partitioning, the long-sequence vibration signal in the normalized data subset is segmented using a sliding window strategy. The sliding window step size is set to 2048, and a time series sample of fixed length is constructed. S14. Generate a time-frequency image using time-series samples as input data for the image branch; specifically, use Short-Time Fourier Transform (STFT) to generate the time-frequency image. The continuous form of the STFT is defined as: ; Among them, For analysis window functions of finite length; the corresponding spectrum (energy density) is defined as: ; In the discrete implementation, let the sampling frequency be... Window length Jump size Window function . No. Frame number Frequency The STFT coefficient is: ; An adaptive short-time fractional Fourier transform can be optionally employed to improve time-frequency focusing and resolution; S15. Input the training set, validation set and test set containing long-sequence vibration signals and time-frequency images into the S2 system respectively, and use them in sequence for training the learnable parameters of each module of the system, hyperparameter tuning and model validity verification during training, and final evaluation of fault classification accuracy and generalization ability.

[0015] S2. Establish a rolling bearing diagnostic system based on Mamba multimodal fusion (named MambaTFVisionChaos). The overall model architecture is as follows: Figure 2 As shown. It includes time series branch, image branch, and feature fusion module; The time series branch is used to process the preprocessed long-sequence vibration signal, extract the nonlinear dynamic features and global time-series dependencies in the signal, and output the time-series features. like Figure 3 As shown, the time series branch includes an input normalization layer, a multi-scale convolutional layer, a feature enhancement module, a state space scanning module, and an output layer, which are connected in series. The input normalization layer receives and processes the preprocessed long-sequence vibration signal and outputs the processed signal to the multi-scale convolutional layer. The multi-scale convolutional layer captures the local transient patterns of the signal, the feature enhancement module extracts the nonlinear dynamic features of the signal, the state space scanning module realizes global temporal dependency modeling, and the output layer outputs robust temporal features.

[0016] The image branch is used to process the time-frequency image converted from the vibration signal after preprocessing. It extracts multi-dimensional time-frequency features through bidirectional state space modeling of the time axis and frequency axis. The image branch includes a downsampling layer, a dual-axis state space block module, and a global pooling layer. The downsampling layer receives the preprocessed time-frequency image and performs downsampling processing, outputting the processed features to the dual-axis state space block module. The dual-axis state space block module achieves cross-dimensional feature extraction and fusion through local axial encoding, recursive modeling of dual-axis independent state spaces, structured channel mixing, and residual fusion, outputting the fused features to the global pooling layer. The global pooling layer outputs multi-dimensional time-frequency features through approximately butterfly-shaped structured channel interaction and global average pooling. The dual-axis state space block module includes TFSSM Block1, which is connected to TFSSM Block2. TFSSM Block2 performs depth separable downsampling and is then connected to TFSSM Block3. TFSSM Block3 is used for depth feature refinement, and depth separable downsampling is used to reduce spatial dimensions while retaining channel information.

[0017] The feature fusion module is used to receive and adaptively fuse the time-series features output from the time-series branch and the time-frequency features output from the image branch, and output the rolling bearing fault classification result. The feature fusion module includes a low-rank projection layer, a cross-attention calculation layer, and an output layer. The low-rank projection layer receives temporal features and multi-dimensional time-frequency features, reduces the projection complexity through low-rank parameterization, and outputs the processed features to the cross-attention calculation layer. The cross-attention calculation layer uses multi-dimensional time-frequency features as queries and temporal features as keys and values ​​to complete the interactive calculation of dual features. After feature weighted fusion, RMS normalization, amplitude clipping, and random deactivation, the rolling bearing fault category distribution is finally output through the linear classification head of the output layer.

[0018] S3. Use time series branching to preprocess the long sequence vibration signal, extract the nonlinear dynamic features and global time series dependencies in the signal, and output the time series features. S3 specifically includes: S31. Receive the preprocessed long-sequence vibration signal through the input normalization layer of the time series branch. First, in-sample standardization is performed, then the data is mapped to values ​​using value embeddings and learnable location embeddings. 3D space (where, ), to obtain initial features Downsampling is achieved through 1D average pooling, with the downsampling factor... (Enabled when value is greater than 1), initial features Pass it into a multi-scale convolutional layer.

[0019] S32. Initial features output from S31 are processed by multi-scale convolutional layers with time-series branches. Perform multi-scale depthwise separable convolutions to capture local transient patterns; Initial features of S31 output from multi-scale convolutional layers A fusion of parallel multi-scale depthwise convolution and 1×1 pointwise convolution is employed, with the kernel length selected accordingly. The convolutional kernels are used to perform depthwise convolutions in parallel, obtaining depthwise convolution output features corresponding to different scales. The formula for parallel depthwise convolution is: ; ; in, The kernel length is 1. The normalized long-sequence vibration signal characteristics output by S31 The depthwise convolution outputs features at a single scale. For depthwise convolution operations, Channel splicing operation for output features at different scales. This represents a 1×1 point convolution operation. The multi-scale convolutional fusion features are used as input to S33.

[0020] S33. The feature enhancement module of time series branching is used to perform causal delay embedding enhancement mechanism on the output of S32 to extract nonlinear dynamic features; S33 specifically includes: S331, Given the delay order and step length Construct a delay stack : ; in, For a moment The characteristic of this is that causality is ensured through forward zero-padding; S332. Calculate the change metric and construct an attractor-gated signal based on the differences between adjacent frames. The formula is: ; ; in, For a moment Measure of change for Activation function The learnable coefficient, The learnable bias (-2.5) is initially set to a negative value. , These are the learnable parameters for the feature enhancement module; S333, via embedded functions Delay stacking Mapping to multi-scale convolutional fusion features Then, through the attractor-gated signal Control the injection strength of the mapping result, and compare the injected features with the original time step. Features Superposition yields enhanced features. The formula is: .

[0021] S34. The state-space scanning module of the time series branch performs a selective state-space scan on the output of S33 to realize global time-series dependency modeling, and finally the output layer outputs the time-series features. S34 specifically includes: Selective state-space scanning is achieved by employing stable parameterization and logarithmic field centralization. Diagonal stable parameterization is performed on each hidden channel to construct a diagonal stable state matrix in which the real parts of all eigenvalues ​​are negative. The formula is: ; in, For logarithmic domain stable parameters; according to Generate nonnegative bounded step size Step size parameter through or After the function transformation, the state is truncated to the interval formed by 0 and the preset maximum step size. The state update adopts the logarithmic field-centered prefix sum form; the original cumulative state parameters are solved by prefix sum operation. The formula is: ; For the original cumulative state parameters Centralized processing is performed to obtain centralized cumulative state parameters. The formula is: ; based on Calculate the logarithmic field rescaling factor The formula is: ; Combined with input drive signal With state weight factor The inverse operation result is used to calculate the hidden state through element-wise multiplication. The formula is: ; in, B For read / write parameters; Hidden state Read and write parametersC Read the status information and combine it with the jump item. Supplementing input drive features and synchronizing input drive signals Hidden state and state space scan results Amplitude clipping is performed, and the final output is the state space scan result. The formula is: ; State space scan results at all time points After integrating the temporal dimensions and processing by the output layer, robust temporal features are output. .

[0022] S4. The time-frequency image converted from the vibration signal after image branching is preprocessed and multi-dimensional time-frequency features are extracted by bidirectional state space modeling of the time axis and frequency axis. S4 specifically includes: S41. The input time-frequency image is processed through the downsampling layer of the image branch. Perform a downsampling Stem operation consisting of three convolutions: a 5×5 stride 2-convolution (3→ The passage, in which ) + BN + SiLU, 3×3 stride 2-depth convolution (number of groups = +BN+SiLU, 1×1 point convolution is used for channel integration, outputting downsampled features. .

[0023] S42. Through the dual-axis state space block module of the image branch, local axial encoding, stable axial recursion, structured channel mixing, normalization, and residual fusion are executed sequentially according to the structured computation pipeline, specifically including: Local axial encoding: First, block application and Depth-wise convolution captures local correlations along the frequency axis and the time axis, respectively. , Then, batch normalization and SiLU activation are performed; subsequently, stable 1D spatial recursion covers long-range dependencies on both axes: ; Ensure stable parameterization. , , Depend on Limitations, scaling factor Amplitude clipping suppresses numerical instability. Stable axial recursion involves independent state-space recursive modeling along both the frequency and time axes, specifically: For frequency axis recursion, define the frequency axis state vector. , frequency axis state vector Let the initial value be set as the zero vector, and the formula is: ; in, , The dimension of the state vector; Based on the exponential decay mechanism, a state update rule is used to calculate the current frequency axis state vector by combining the step size parameter, input features, and projection parameters. The formula is: ; in, The output features of the S41 downsampling Stem operation; The recursive initial output of the frequency axis is obtained by performing an inner product operation between the current state vector and the read parameters. The formula is: ; Numerical clipping is performed on the initial output, and the product of the input features and the bias parameters is superimposed to obtain the final frequency axis recursive features. The formula is: ; in, These are bias parameters; For timeline recursion, define the timeline state vector. , time axis state vector The initial value is set as the zero vector, and the formula is: ; in, ; Employing state update, output calculation, and numerical pruning logic that is completely consistent with the frequency axis recursion, feature extraction along the time axis is completed, and the time axis state vector at the current moment is calculated. The formula is: ; The initial output of the timeline recursion is obtained by performing an inner product operation between the current state vector and the read parameters. The formula is: ; After performing numerical cropping on the initial output, the product of the input features and the bias parameters is superimposed to obtain the final time-axis recursive features. The formula is: ; Define the diagonal stability matrix respectively Step size parameters Input projection parameters and read parameters The formula is: ; ; ; ; in, , This is the numerical stability threshold. This is the scaling factor. To crop the upper and lower threshold values, The minimum threshold for the step size. Maximum threshold for step size ( ), For the step size parameter in the logarithmic field, , , Corresponding to step size parameters Input projection parameters Read parameters Convolution extraction operation; The implementation achieves O(ClogC) complexity through multi-stage grouping in 1×1 layers and inter-stage ReLU, with zero-padding and output pruning for non-power-of-2 channels; finally, Batch Normalization (BN) and optional Dropout are executed sequentially. Residual addition and SiLU activation are used to complete the processing of the dual-axis state space block module. S43, Frequency axis recursive generation Timeline recursively generated The two are connected to achieve channel interaction through an approximately butterfly-shaped structured channel blending process. The feature map is projected onto 32 channels via a 1×1 convolution, producing... Global average pooling is performed using a global pooling layer in the image branch to obtain multi-dimensional time-frequency features of the global spatial representation. (in ).

[0024] S5. The feature fusion module receives and adaptively fuses the time-series features output from the time-series branch with the time-frequency features output from the image branch, and outputs the rolling bearing fault classification result. S5 specifically includes: S51. Through the low-rank projection layer of the feature fusion module, the query matrix is ​​processed. Key matrix Value matrix The linear projection uses low-rank parameterization, and the maximum rank is set to . The projection matrix is ​​represented as a decomposition basis. and Introducing learnable weights Processed using the softmax function Obtain weights ,based on Construct a diagonal matrix The decomposition basis is dynamically weighted, and the formula is: ; ; ; S52, For each attention head (default) ), which are query matrices respectively. Key matrix Value matrix Corresponding structural decomposition basis , , Based on the dynamic low-rank weighting rule described in S51, all decomposition bases are processed to obtain the query projection. Key projection Value projection Reduce projection complexity from Reduce to ; S53. Through the cross-attention calculation layer of the feature fusion module, using time-frequency features... As a query, time-series features Using these as keys and values, multi-head cross-attention is calculated to obtain the attention output. The calculation formula is: ; in, Hiding dimensions for attention ; S54, Based on time-frequency characteristics Temporal feature mean and attention output The gating coefficient is calculated using a multilayer perceptron. And then mapped by the activation function, and then through the gating coefficient Time-frequency characteristics With attention output results Weighted fusion is performed to obtain fusion features. The formula is: ; ; in, Temporal characteristics The mean of time-series features is obtained by calculating the global mean; S55. Perform RMS normalization, amplitude pruning, and random deactivation processing (Dropout) sequentially on the fused features. Then, the fault category distribution is output through the linear classification head of the output layer.

[0025] The diagnostic performance of the model is evaluated. Specifically, model performance is evaluated based on the confusion matrix, using accuracy and AUC (Area Under the Receiver Operating Characteristic Curve). The calculation formulas are as follows: ; Wherein, TP represents true positives (the number of samples that are actually faulty but are correctly predicted as faulty), TN represents true negatives (the number of samples that are actually normal but are correctly predicted as normal), FP represents false positives (the number of samples that are actually normal but are incorrectly predicted as faulty, i.e., false alarms), and FN represents false negatives (the number of samples that are actually faulty but are incorrectly predicted as normal, i.e., false misses).

[0026] Instance analysis based on four public datasets for rolling bearing fault diagnosis from different regions and operating conditions: Example 1 Experimental Results on the CWRU Dataset: Experimental results on the Case Western Reserve University (CWRU) 10-class dataset show that the method of this invention achieves 100% test set accuracy and 100% AUC. The model contains 334,664 parameters, has a model size of 1.2763MB, a computational load of 1,825,876,160 FLOPs (1.826 GFLOPs), an average latency of 17.476 ms / sample, a throughput of 3,353.22 samples / second, a peak GPU memory usage of 481.834MB, and an efficiency ratio of 1.836. The model achieves excellent inference efficiency while maintaining high accuracy, as shown in Table 1. The confusion matrix of the CWRU dataset is shown below. Figure 4 As shown.

[0027] Table 1. Performance results of the CWRU dataset model

[0028] Compared with existing methods, the method of this invention achieves state-of-the-art performance on the CWRU dataset, outperforming other methods such as VibrMamba (99.77%), as shown in Table 2.

[0029] Table 2. Comparison Experiment Results of CWRU Dataset

[0030] Example 2 Experimental results on the XJTU-SY dataset: Experimental results on the Xian Jiaotong University (XJTU-SY) 4-class dataset show that the method of this invention achieves 100% test set accuracy and 100% AUC, with no misclassifications. The model contains 334,370 parameters, has a model size of 1.2755MB, a computational load of 0.2313 GFLOPs, an average latency of 110.47 milliseconds / sample, a throughput of 557.05 samples / second, a peak GPU memory usage of 832.25MB, and an efficiency ratio of 2.4078, as shown in Table 3. The confusion matrix of the XJTU-SY dataset is as follows. Figure 5 As shown.

[0031] Table 3 Performance results of the XJTU-SY dataset model

[0032] Compared with existing methods, the method of this invention achieves state-of-the-art performance on the XJTU-SY dataset, outperforming VibrMamba (99.95%), Multi-signal Fusion+IDCGAN+ Methods such as CBAM ResNet (99.64%) and MTF+GAT (98.23%) are shown in Table 4.

[0033] Table 4. Comparison Experiment Results of XJTU-SY Dataset

[0034] Example 3 Experimental Results on the HIT Dataset: Experimental results on the Harbin Institute of Technology (HIT) single-channel 3-class dataset show that the method of this invention achieves a test set accuracy of 97.26% and an AUC of 99.86%. The model contains 334,321 parameters, has a model size of 1.2753MB, a computational load of 0.2313 GFLOPs, an average latency of 87.37 milliseconds / sample, a throughput of 732.54 samples / second, a peak GPU memory usage of 832.24MB, and an efficiency ratio of 3.1664, as shown in Table 5. The confusion matrix of the HIT dataset is as follows. Figure 6 As shown.

[0035] Table 5 Performance Results of the HIT Dataset Model

[0036] Compared with existing methods, the method of this invention significantly outperforms existing methods on the HIT dataset, including WDCNN (85.05%), DRSN-CW (87.9%), VibrMamba (87.63%), LMAFCNN (93.12%), etc., as shown in Table 6.

[0037] Table 6. Comparison Experiment Results of HIT Dataset

[0038] Therefore, the present invention adopts the above-mentioned rolling bearing diagnostic system and method based on Mamba multimodal fusion, which combines the linear complexity Mamba model, adaptive multimodal fusion and chaos enhancement, helps to improve the efficiency and accuracy of rolling bearing fault diagnosis, and achieves millisecond-level real-time inference while maintaining high precision, providing a practical solution for real-time bearing fault diagnosis in industrial environments.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A rolling bearing diagnostic system based on Mamba multimodal fusion, characterized in that, Includes time series branching, image branching, and feature fusion modules; The time series branch is used to process the preprocessed long-sequence vibration signal, extract the nonlinear dynamic features and global time series dependencies in the signal, and output the time series features. The image branch is used to process the preprocessed time-frequency image converted from the vibration signal, and extracts multi-dimensional time-frequency features through bidirectional state space modeling of the time axis and frequency axis; The feature fusion module is used to receive and adaptively fuse the time-series features output by the time-series branch with the time-frequency features output by the image branch, and output the rolling bearing fault classification result.

2. The rolling bearing diagnostic system based on Mamba multimodal fusion according to claim 1, characterized in that: The time series branch includes an input normalization layer, a multi-scale convolutional layer, a feature enhancement module, a state space scanning module, and an output layer, which are connected in series. The input normalization layer receives the preprocessed long-sequence vibration signal and performs normalization processing to obtain initial features, which are then output to the multi-scale convolutional layer. The multi-scale convolutional layer captures the local transient patterns of the signal through multi-scale parallel convolution. The feature enhancement module strengthens and extracts the nonlinear dynamic features of the signal. The state space scanning module realizes global temporal dependency modeling, and the output layer outputs the temporal features. The image branch includes a downsampling layer, a dual-axis state space block module, and a global pooling layer. The downsampling layer receives the preprocessed time-frequency image and performs downsampling processing, outputting the processed features to the dual-axis state space block module. The dual-axis state space block module achieves cross-dimensional feature extraction and integration through local axial encoding, recursive modeling of dual-axis independent state spaces, channel mixing, and residual fusion, outputting the integrated features to the global pooling layer. The global pooling layer performs global pooling processing on the input features, extracts global spatial features, and outputs multi-dimensional time-frequency features. The dual-axis state space block module includes TFSSM Block1, which is connected to TFSSM Block2. TFSSM Block2 is connected to TFSSM Block3 after performing depth separable downsampling. TFSSM Block3 is used for depth feature refinement, and depth separable downsampling is used to reduce spatial dimensions and retain channel information. The feature fusion module includes a low-rank projection layer, a cross-attention calculation layer, and an output layer. The low-rank projection layer receives the temporal features and the multi-dimensional time-frequency features, reduces the projection computation complexity through low-rank parameterization, and outputs the processed features to the cross-attention calculation layer. The cross-attention calculation layer uses the multi-dimensional time-frequency features as queries and the temporal features as keys and values ​​to complete the bimodal feature interaction calculation. After feature weighted fusion and normalization, amplitude clipping, and random deactivation, the rolling bearing fault category distribution is finally output through the classification head of the output layer.

3. A rolling bearing diagnostic method based on Mamba multimodal fusion, characterized in that the steps are as follows: include: S1. Acquire rolling bearing vibration data, preprocess the vibration data, and output a dataset containing long-sequence vibration signals and time-frequency images; S2. Establish a rolling bearing diagnostic system based on Mamba multimodal fusion as described in any one of claims 1-2; S3. Use time series branching to preprocess the long sequence vibration signal, extract the nonlinear dynamic features and global time series dependencies in the signal, and output the time series features. S4. The time-frequency image converted from the vibration signal after image branching is preprocessed and multi-dimensional time-frequency features are extracted by bidirectional state space modeling of the time axis and frequency axis. S5. The feature fusion module receives and adaptively fuses the time-series features output by the time-series branch with the time-frequency features output by the image branch, and outputs the rolling bearing fault classification result.

4. The rolling bearing diagnostic method based on Mamba multimodal fusion according to claim 3, characterized in that, The preprocessing of vibration data in S1 includes: S11. At the level of raw vibration signals, the dataset is divided according to preset rules, and vibration signals corresponding to different working conditions or different acquisition sources are assigned to non-overlapping training sets, validation sets and test sets. S12. Perform length normalization processing on each original vibration signal, and then perform normalization processing on each data subset. S13. After completing the signal level division, the normalized long sequence vibration signal is segmented using a segmentation strategy to construct time-series samples of fixed length. S14. Generate a time-frequency image based on time-series samples using a time-frequency conversion algorithm, and use it as input data for the time-frequency image branch; S15. The training set, validation set, and test set, which contain long-sequence vibration signals and time-frequency images, are used for training the system's learnable parameters in S2, for hyperparameter tuning and model validity verification during training, and for evaluating the final fault classification accuracy and generalization ability.

5. The rolling bearing diagnostic method based on Mamba multimodal fusion according to claim 3, characterized in that, S3 specifically includes: S31. The long-sequence vibration signal is normalized by the input normalization layer of the temporal branch, and the processed signal is fed into the multi-scale convolutional layer. S32. Perform multi-scale parallel convolution on the output of S31 through multi-scale convolutional layers to capture local transient patterns of the signal. S33. Use the feature enhancement module to perform feature enhancement processing on the output of S32 to extract the nonlinear dynamic features of the signal; S34. The state space scanning module performs a state space scan on the output of S33 to achieve global temporal dependency modeling, and finally outputs the temporal features by the output layer.

6. The rolling bearing diagnostic method based on Mamba multimodal fusion according to claim 5, characterized in that: S32 specifically includes: The multi-scale convolutional layer employs a fusion of parallel multi-scale depthwise convolution and 1×1 pointwise convolution. Multiple convolutional kernels of different lengths are selected and performed in parallel to execute depthwise convolution, yielding depthwise convolution output features corresponding to different scales. The formula is as follows: ； ； in, The kernel length is 1. The normalized long-sequence vibration signal characteristics output by S31 The depthwise convolution outputs features for a single scale. For depthwise convolution operations, Channel splicing operation for output features at different scales. This represents a 1×1 point convolution operation. The multi-scale convolutional fusion features are used as input to S33.

7. The rolling bearing diagnostic method based on Mamba multimodal fusion according to claim 5, characterized in that, S33 specifically includes: S331, Given the delay order and step length Construct a delay stack : ； in, For a moment The characteristic of this is that causality is ensured through forward zero-padding; S332. Calculate the change metric and construct an attractor-gated signal based on the differences between adjacent frames. The formula is: ； ； in, For a moment Measure of change for Activation function The learnable coefficient, As a learnable bias, it is initially set to a negative value. , These are the learnable parameters for the feature enhancement module; S333, via embedded functions Delay coordinate vector Mapping to multi-scale convolutional fusion features Then, through the attractor-gated signal Control the injection strength of the mapping result, and compare the injected features with the original time step. Features Superposition yields enhanced features. The formula is: 。 8. The rolling bearing diagnostic method based on Mamba multimodal fusion according to claim 5, characterized in that: S34 specifically includes: Selective state-space scanning is achieved by employing stable parameterization and logarithmic field centralization. Diagonal stable parameterization is performed on each hidden channel to construct a diagonal stable state matrix in which the real parts of all eigenvalues ​​are negative. The formula is: ； in, For logarithmic domain stable parameters; Generate nonnegative bounded step size The step size parameter through or After the function transformation, the state is truncated to the interval formed by 0 and the preset maximum step size. The state update adopts the logarithmic field-centered prefix sum form; the original cumulative state parameters are solved by prefix sum operation. The formula is: ； For the original cumulative state parameters Centralized processing is performed to obtain centralized cumulative state parameters. The formula is: ； based on Calculate the logarithmic domain rescaling factor The formula is: ； Combined with input drive signal With state weight factor The inverse operation result is used to calculate the hidden state through element-wise multiplication. The formula is: ； in, B For read / write parameters; Hidden state Read and write parameters C Read the status information and combine it with the jump item. Supplementing input drive features and synchronizing input drive signals Hidden state and state space scan results Amplitude clipping is performed, and the final output is the state space scan result. The formula is: ； State space scan results at all time points After integrating the temporal dimensions and processing by the output layer, robust temporal features are output. .

9. The rolling bearing diagnostic method based on Mamba multimodal fusion according to claim 3, characterized in that: S4 specifically includes: S41. The input time-frequency image is processed through the downsampling layer of the image branch. Perform a downsampling STEM operation consisting of three convolutional steps to output downsampling features. ; S42. The dual-axis state space block module of the image branch sequentially performs local axial encoding, stable axial recursion, structured channel mixing, normalization and residual fusion. The stable axial recursion is carried out independently along the frequency axis and the time axis, respectively, including state space recursive modeling, specifically including: For frequency axis recursion, define the frequency axis state vector. , frequency axis state vector The initial value is set as the zero vector, and the formula is: ； in, , The dimension of the state vector; Based on the exponential decay mechanism, a state update rule is used to calculate the current frequency axis state vector by combining the step size parameter, input features, and projection parameters. The formula is: ； in, The output features of the S41 downsampling Stem operation; The recursive initial output of the frequency axis is obtained by performing an inner product operation between the current state vector and the read parameters. The formula is: ； Numerical clipping is performed on the initial output, and the product of the input features and the bias parameters is superimposed to obtain the final frequency axis recursive features. The formula is: ； in, These are bias parameters; For timeline recursion, define the timeline state vector. , time axis state vector The initial value is set as the zero vector, and the formula is: ； in, ; Based on the state update rule of exponential decay mechanism, the current time axis state vector is calculated by combining step size parameter, input features and projection parameter. The formula is: ； The initial output of the timeline recursion is obtained by performing an inner product operation between the current state vector and the read parameters. The formula is: ； After performing numerical cropping on the initial output, the product of the input features and the bias parameters is superimposed to obtain the final time-axis recursive features. The formula is: ； Define the diagonal stability matrix respectively Step size parameters Input projection parameters and read parameters The formula is: ； ； ； ； in, , This is the numerical stability threshold. This is the scaling factor. To crop the upper and lower threshold values, The minimum threshold for the step size. The maximum threshold for step size. For the step size parameter in the logarithmic field, , , Corresponding to step size parameters Input projection parameters Read parameters Convolution extraction operation; S43. Channel interaction is achieved through an approximate butterfly-shaped structured channel blending method. Global average pooling is performed using the global pooling layer of the image branch to obtain multi-dimensional time-frequency features of the global spatial representation.

10. The rolling bearing diagnostic method based on Mamba multimodal fusion according to claim 3, characterized in that: S5 specifically includes: S51. Through the low-rank projection layer of the feature fusion module, the query matrix is ​​processed. Key matrix Value matrix The linear projection uses low-rank parameterization, and the maximum rank is set to be . The projection matrix is ​​represented as a decomposition basis. and The product form is introduced, and learnable weights are used. Processed using the softmax function Obtain weights ,based on Construct a diagonal matrix The decomposition basis is dynamically weighted, and the formula is: ； ； ； S52. For each attention head, the query matrix is... Key matrix Value matrix Corresponding structural decomposition basis , , Based on the dynamic low-rank weighting rule described in S51, all decomposition bases are processed to obtain the query projection. Key projection Value projection ; S53. Through the cross-attention calculation layer of the feature fusion module, using time-frequency features... As a query, time-series features Using these as keys and values, multi-head cross-attention is calculated to obtain the attention output. The calculation formula is: ； in, Hiding dimensions for attention; S54, Based on time-frequency characteristics Temporal feature mean and attention output The gating coefficient is calculated using a multilayer perceptron. And then mapped by the activation function, and then through the gating coefficient Time-frequency characteristics With attention output results Weighted fusion is performed to obtain fusion features. The formula is: ； ； in, Temporal characteristics The mean of time-series features is obtained by calculating the global mean; S55. Normalize, prune, and randomly deactivate the fused features in sequence, and then output the fault category distribution through the classification head of the output layer.

Images