Bearing diagnosis method and system based on multi-modal and multi-scale fusion network

Abstract

The bearing diagnosis method and system based on the multi-modal and multi-scale fusion network are characterized in that the feature extraction based on multiple modal signals and the multi-dimensional information fusion and the diagnosis means with the enhanced feature extraction capability are provided, so that the bearing degradation information is comprehensively represented, the health condition of the bearing is accurately calculated, the diagnosis data has the comparability and the interaction, the multi-modal original data is fully utilized, and the design purpose that the multiple data fusion advantages can be combined is achieved. The bearing diagnosis method comprises the following implementation steps: step 1, acquiring multi-source heterogeneous signal data sets of multiple fault types under different working conditions; step 2, constructing a data-level feature vector matrix based on multi-modal data fusion; step 3, generating a multi-scale CWT feature extraction grayscale image; step 4, generating a multi-scale STFT feature extraction grayscale image; step 5, generating a new time-frequency feature image set Image; step 6, constructing a VIT model; and step 7, obtaining a classification result of the bearing fault type condition.

Description

Technical Field

[0001] This invention relates to a bearing fault diagnosis method and system based on multimodal data fusion and multi-scale graph attention fusion network, belonging to the field of deep learning and fault diagnosis technology. Background Technology

[0002] Bearings are an essential and core structure of transmission devices. Common failures caused by bearing failure under load include fatigue shedding, wear, and fracture, with these failures often concentrated between the inner and outer rings. Bearings are more likely to fail under conditions of excessive load over a long period, poor lubrication, and improper installation. Rolling bearings have poor impact resistance and are more prone to failure due to factors such as excessive speed and corrosion.

[0003] Due to the characteristics of the motor structure and usage, the bearings of existing vertical motors inevitably bear axial pressure generated by the rotor's own weight. Therefore, timely fault diagnosis of the bearings and the development of appropriate maintenance strategies are essential to effectively reduce unnecessary expenses and improve maintenance efficiency. Typically, based on monitoring information, timely detection of anomalies or evaluation of equipment health status, and prediction of equipment failure trends, the development of intelligent fault monitoring algorithm systems for practical application in industry has immense practical significance for improving efficiency and reducing costs in operation and maintenance.

[0004] Existing technologies typically employ single-mode vibration signals to handle fault diagnosis and classification based on the non-stationary and nonlinear characteristics of real-world operating conditions. However, such methods have low accuracy in extracting representative key feature information, making it difficult to reasonably and realistically diagnose the fault type and its degree in practical scenarios.

[0005] In view of the above, this patent application is hereby filed. Summary of the Invention

[0006] The bearing diagnosis method and system based on multimodal and multiscale fusion networks described in this invention aims to solve the problems existing in the prior art by proposing a diagnostic method based on feature extraction of multiple modal signals, fusion of multidimensional information, and enhanced feature extraction capabilities. This enables a comprehensive characterization of bearing degradation information and accurate calculation of the bearing's health status, thereby achieving the design objectives of making diagnostic data comparable, interactive, fully utilizing multimodal raw data, and combining the advantages of multiple data fusion.

[0007] To achieve the above-mentioned objectives, the bearing diagnosis method based on a multimodal and multi-scale fusion network includes the following implementation steps:

[0008] Step 1: Obtain a dataset of multi-source heterogeneous signals under different operating conditions and various fault types;

[0009] Step 2: Construct a data-level feature vector matrix based on multimodal data fusion;

[0010] Step 3: Generate grayscale images with multi-scale CWT feature extraction;

[0011] Step 4: Generate a grayscale image with multi-scale STFT feature extraction;

[0012] Step 5: Generate a new time-frequency feature image set Image;

[0013] Step 6: Build the VIT model;

[0014] Step 7: Obtain the classification results of bearing failure types;

[0015] The fused time-frequency feature image set Image is divided into a training set and a validation set, and then imported into a preset fault classification model to obtain the classification results of the health or corresponding fault type status.

[0016] Furthermore, in step 1, the sensor integrated monitoring module is embedded in the bearing to collect signals of N modes in real time; the signals of N modes include multi-source signals under different fault conditions, and the types of multi-source signals include but are not limited to the vibration, temperature, and speed of the bearing, and the voltage, current, and power of the motor operating state.

[0017] Furthermore, in step 2, feature extraction and information fusion preprocessing are performed on multiple modal signals to obtain feature vectors for each modal signal. Each feature vector contains features describing different modal information. After splicing and averaging the features of different modalities, data-level fusion is performed on vibration, temperature, and rotational speed as the main sequences according to modal dimensions to construct a feature vector matrix.

[0018] Furthermore, in step 3, the data-level fused feature vector matrix is ​​used to obtain features at different frequency ranges or time scales by selecting different wavelet functions or scale parameters, and the features at each scale are calculated to generate a grayscale image with multi-scale CWT feature extraction.

[0019] Furthermore, in step 4, the data-level fusion feature vector matrix A1 is subjected to multi-scale STFT feature extraction, and the corresponding spectral energy or other statistical features are calculated to generate a grayscale image with multi-scale STFT feature extraction.

[0020] The basic calculation formula for STFT is as follows:

[0021]

[0022] In the formula, x(t) is the time-domain signal, and h(t-ω) is the window function.

[0023] Furthermore, in step 5, the grayscale images of feature vectors extracted at different scales are weighted and fused, and different weights are assigned according to the importance of different scales; the above images are stitched together to form a longer feature vector and then normalized to generate a new time-frequency feature image set Image.

[0024] Multi-scale fusion of features from STFT grayscale images and wavelet transform images is performed. First, feature representations at different scales are extracted for each feature. Then, the features at different scales are stitched together and fused. Finally, they are superimposed to form a multi-channel image and constitute a grayscale image dataset that fuses the features of both.

[0025] Furthermore, in step 6, the VIT model includes an input layer, a multi-scale image patch partitioning layer, an image patch embedding layer, a position coding layer, a Transformer encoder layer, an adaptive pooling layer, an MLP layer, and an output layer.

[0026] Constructing a fault classification VIT model based on a multi-scale graph attention fusion network architecture includes the following steps:

[0027] S601. Input data preprocessing;

[0028] For the input graph-structured data, embedding operations are performed using different graph embedding sizes to obtain multiple local feature representations of different sizes. Each graph embedding size focuses on different local features of the graph structure. Each image patch is mapped to a low-dimensional feature vector space. This embedding layer converts the pixel information of each image patch into a fixed-length vector representation. The embedding layer is typically a fully connected layer, where the parameters are learnable.

[0029] S602. Input the preprocessed image data through an embedding layer (such as...) Figure 6 Mapped to a low-dimensional feature vector embedding space;

[0030] For image data, local features are extracted using image patch mapping sizes of different sizes (16*16 / 12*12 / 8*8), and each image patch is mapped to the embedded feature space and flattened into a vector.

[0031] The formula for calculating graph embedding mapping is as follows:

[0032]

[0033] Where, k i u-i,v-j Z represents graph embeddings of different sizes and spatial locations. i This represents the local graph features extracted by embedding operations of different sizes;

[0034] S603. Location coding;

[0035] Positional information is introduced, and a weighted average of the input vectors of different graph embeddings is performed using a positional encoding form of sine and cosine coding. Sequence information is introduced into the self-attention mechanism so that the model can perceive the relative positional relationships of different positions in the input sequence.

[0036] S604. Construct a multi-layer Transformer encoder;

[0037] S605. Loss Function and Optimization Algorithm;

[0038] The processed results are used to calculate the loss using the cross-entropy loss function. The Adam optimization algorithm is selected, and the network parameters are updated based on the gradient.

[0039] Furthermore, in step 7, the different types of datasets obtained in step 5 are divided into training and validation sets in a 4:1 ratio. First, the training sets constructed after two different preprocessing steps are input into the multi-scale graph attention fusion network architecture established in step 6. Then, the weight values ​​in the fault detection model are trained using the backpropagation algorithm. The validation set is used to judge the training effect of the bearing defect detection model to obtain a fully trained mechanical defect detection model. Finally, the test set data is used to verify the reliability of the model.

[0040] The data to be detected is collected based on a multi-scale graph attention fusion network architecture. The preprocessing process in step 1 is repeated and the data is input into the trained bearing defect multi-channel data detection model to detect the defects of the bearing during operation. The defect conditions are labeled by the results of fuzzy comprehensive evaluation theory, which are output by two feature extraction networks, characterize the mechanical defect conditions, and are fused by decision-making.

[0041] Based on the aforementioned bearing diagnosis method, this application also proposes a bearing diagnosis system based on a multimodal and multi-scale fusion network. This system includes a raw signal acquisition module, a multi-source data processing module, a multi-scale feature fusion module, a multi-scale feature extraction module, and a diagnosis classification module. Among these,

[0042] The raw signal acquisition module is used to acquire signals of N modes of rolling bearings, including data information such as rolling bearing sound, vibration, temperature and rotational speed;

[0043] The multi-source data processing module is used to preprocess N modal signals. It can perform corresponding traditional signal processing analysis on the raw acquired data, and then perform data-level fusion of the processed data according to the modal dimension to construct a vector matrix containing feature parameters, denoted as the data-level feature vector matrix A1.

[0044] The multi-scale feature fusion module performs continuous wavelet transform and STFT transform on the initially constructed parametric feature vector matrix and fuses the features; it performs weighted fusion on the grayscale images of the feature vectors extracted at different scales, assigning different weights according to the importance of different scales; it continues to stitch the above images together to form a longer feature vector, which is then normalized, and finally generates a new time-frequency feature image set Image.

[0045] The multi-scale feature extraction module is used for the input image. First, it is converted into image patches through an embedding layer and divided into embedding representations of different scales according to certain rules. For each scale, the ViT model assigns one or more attention heads. Each attention head learns different weights and spatial relationships to perform self-attention computation on the input embedding representation. Each attention head generates an attention matrix, representing the attention weights between different locations. These matrices are then used to generate the final multi-scale attention representation in the form of attention convergence.

[0046] The diagnostic classification module is used in the ViT model to map image features to the classification task space and generate classification labels through MLP and softmax functions. MLP extracts abstract feature representations by stacking multiple fully connected layers and activation functions. The softmax function normalizes the output of MLP to obtain the probability prediction result of each category for classification decision-making. The above operations enable the ViT model to learn the mapping relationship from input image to output label during training and to provide classification prediction for input image during the inference stage.

[0047] Applying the above-mentioned bearing diagnosis method and system, this application proposes a host computer industrial control device for realizing bearing fault diagnosis, which includes a processor and a memory. The memory is used to store computer-executable programs, and the processor reads the computer-executable programs from the memory and executes them. Additionally, a computer-readable storage medium is provided for storing the analysis software of the industrial control multi-channel acquisition system based on the above-mentioned bearing diagnosis method using multimodal and multi-scale fusion networks.

[0048] In summary, the bearing diagnosis method and system based on multimodal and multiscale fusion networks proposed in this application have the following advantages and beneficial effects:

[0049] 1. This invention proposes a bearing fault diagnosis method based on multimodal feature-level fusion and multi-scale graph attention fusion network. It can utilize different types of sensors to acquire multi-source heterogeneous original fault data such as sound, vibration, temperature, and speed during the operation of a simulated vertical bearing under the control speed of a frequency converter. Then, information selection and data processing are performed to construct a parametric feature matrix. After wavelet transform at different scales and STFT transform with different window functions of different lengths, feature fusion enhancement and normalization are performed to achieve comprehensive health and fault classification data that characterizes the actual operating condition of the bearing. Finally, an improved ViT image fault classification network based on multi-scale graph attention fusion is constructed to establish a multi-classification data model with optimal performance for training. This solves the defect of poor accuracy of single-signal fault classification under real working conditions in existing technologies and significantly optimizes the robustness and generalization ability of existing deep learning methods.

[0050] 2. This application proposes a preprocessing method that includes feature extraction and multi-dimensional information fusion enhancement for various modal signals such as vibration, temperature, speed, voltage, current, power of motor operating state, and water flow rate of valve speed control device. This method effectively fuses the feature vectors of each modal signal obtained after feature extraction according to modal dimension, thereby significantly improving the feature extraction capability of the feature extraction network.

[0051] 3. The multimodal data fusion process proposed in this application can fully utilize the original multimodal information and combine the advantages of various data fusion methods. It can integrate the advantages and characteristics of data-level and feature-level information. By organically combining fusion strategies, it achieves better performance in terms of efficiency and stability. At the same time, it makes full use of the data preprocessing results in the initial stage to retain the feature information used for bearing monitoring and diagnosis to the greatest extent, and achieves higher accuracy and stability in subsequent feature extraction and diagnostic analysis. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a bearing diagnostic method based on a multimodal and multiscale fusion network.

[0053] Figure 2 This is a schematic diagram of multimodal data fusion;

[0054] Figure 3 This is a flowchart illustrating the feature-level fusion process;

[0055] Figure 4 This is a diagram of a bearing diagnostic system based on a multimodal and multiscale fusion network;

[0056] Figure 5 This is a schematic diagram of the ViT network structure based on a multi-scale graph attention fusion network;

[0057] Figure 6This is a diagram of the operational structure based on the self-attention layer. Detailed Implementation

[0058] The technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described solutions are some embodiments of this application, but not all embodiments. Based on the embodiments proposed in this application, those skilled in the art can make improvements or modifications without creative effort, and all such improvements and modifications should fall within the protection scope of the appended claims.

[0059] Example 1, as Figures 1 to 6 As shown, this application proposes a bearing diagnostic method based on a multimodal and multi-scale fusion network, including the following implementation steps:

[0060] Step 1: Obtain a dataset of multi-source heterogeneous signals under different operating conditions and various fault types;

[0061] The sensor integrated monitoring module is embedded in the bearing to collect signals of N modes in real time. The signals of N modes include multi-source signals under different fault conditions. The types of multi-source signals include, but are not limited to, the vibration, temperature, and speed of the bearing, and the voltage, current, and power of the motor during operation.

[0062] In this embodiment, different bearing speeds are preferred for different operating conditions;

[0063] Further optimization of the scheme: the multi-source heterogeneous signal acquisition of various fault types under different speeds includes two channels of vibration signal, one channel each of bearing operating temperature difference and sound, and one channel each of bearing speed, voltage, current and power exported from the PLC main control board, for a total of eight channels of signal.

[0064] Further optimization of the scheme involves selecting data acquisition equipment based on the following criteria: for vibration signal acquisition equipment, a suitable vibration sensor should be selected according to the requirements of signal frequency range, accuracy, and sensitivity; a suitable measurement point location should be selected according to the structural characteristics and vibration properties of the object being measured; and a suitable sampling frequency should be selected according to the vibration frequency range of the object being measured. Simultaneously, interference with the measurement results should be avoided during the acquisition of vibration signals (e.g., temperature, humidity, electromagnetic interference). For temperature signals, the average temperature of the outer ring can be calculated from the outer ring temperatures obtained from multiple evenly distributed monitoring locations, and then the difference between the outer and inner ring temperatures can be calculated to obtain the inner and outer ring temperature difference signal.

[0065] Step 2: Construct a data-level feature vector matrix based on multimodal data fusion;

[0066] like Figure 2 As shown, multiple modal signals are first subjected to denoising, smoothing, and standardization before statistical feature extraction is performed to obtain the feature vector of each modal signal.

[0067] Each feature vector can contain features describing different modal information. After concatenating and normalizing the features of different modalities, the multimodal information with different data lengths is aligned by interpolation. Then, the vibration, temperature, and rotation speed are used as the main sequences to perform data-level fusion according to the modal dimension and construct a feature vector matrix, denoted as A1.

[0068] Further optimization of the scheme involves concatenating the one-dimensional time sequence scalars of each channel into a multi-dimensional array of length T and form (T, 8) for the features of different modalities, thereby obtaining the data after data-level fusion;

[0069] In a further optimized approach, the aforementioned information fusion preprocessing includes preprocessing the observable signals and measured data to obtain multi-channel signals as follows:

[0070]

[0071] Where n is the number of channels after preprocessing (8 in this invention), and T is the number of signal sampling points; X i，j This represents the j-th sampling point of the i-th channel;

[0072] Step 3: Generate grayscale images with multi-scale CWT feature extraction;

[0073] like Figure 3 As shown, the data-level fusion feature vector matrix A1 is used to obtain features at different frequency ranges or time scales by selecting different wavelet functions or scale parameters, and the amplitude, energy and other features at each scale are calculated to generate a grayscale image with multi-scale CWT feature extraction.

[0074] Specifically, for the aforementioned data-level fusion feature vector matrix A1, wavelet basis functions are selected to first perform data cleaning, compression and other operations to extract effective features and reduce the dimensionality of the data. Then, the data is integrated into multi-channel data after information processing and subjected to wavelet transform.

[0075] The basic operational formulas for wavelet transform are as follows:

[0076]

[0077] In the formula, f(t) is the time-domain signal, ψ(x) is the wavelet basis function, a is the positioning frequency, and b is the positioning time;

[0078] Wavelet transform is used to process multi-channel, multi-source heterogeneous signals. For the data-level fusion feature vector matrix A1, wavelet transform is used to obtain the wavelet coefficient matrix. The rows of the matrix represent different scales, and the columns represent different shifts. CWT is based on a set of mother wavelets (or basic wavelets) that change with different scales. The parameterized feature vector is decomposed, and the signal is decomposed into wavelet coefficients of different scales and frequencies. The original time-domain and frequency-domain feature parameters are converted into feature parameters related to the wavelet coefficients, and the feature vector matrix B1 is constructed. After being converted to three channels, it is mapped to a two-dimensional grayscale image.

[0079] Further optimization involves filtering the wavelet coefficients at each scale using low-pass and high-pass filters, then downsampling the signal to a lower resolution scale. This filtering and downsampling process is repeated until the desired scale level is reached. Finally, the features represented by these coefficients are weighted and fused, and a two-dimensional matrix transformation is performed. Then, wavelet basis functions are applied to the horizontal and vertical directions of the continuous wavelet transform for translation / scaling operations, thereby obtaining a grayscale image with multi-scale CWT feature extraction.

[0080] When using different wavelet basis functions in the wavelet transform process described above, we are essentially calculating the inner product of the wavelet function and the signal at different locations and scales when performing a wavelet transform on a signal. If the wavelet function and the signal have the same "shape" or the same trend of change at a certain location, their inner product (i.e., wavelet coefficients) will be relatively large. Therefore, using different wavelet basis functions to perform wavelet transform on the same data segment will produce different inner product results. Thus, multiple different wavelet basis functions are used to complement each other in representing local feature changes in the time-frequency domain.

[0081] For rolling bearings, localized failure characteristics are caused by surface fatigue, cracks, or damage, and the main types include:

[0082] ① Rolling element failure

[0083] High-frequency components: Rolling element failures typically produce significant harmonic components in the high-frequency range. These frequency components correspond to impacts or contacts between the rolling element and other components.

[0084] Periodic characteristics: Rolling element failures usually manifest as periodic pulse signals, the frequency of which is related to the rotational speed and number of rolling elements.

[0085] Energy Concentration: Rolling element failure can cause energy to concentrate in a specific frequency band, so a significant increase in vibration energy can be observed in the fault frequency band.

[0086] ② Cage failure

[0087] Low-frequency components: Cage failures typically produce more noticeable components in the low-frequency range, which correspond to the inherent vibration modes of the cage.

[0088] Non-periodic characteristics: Cage failures typically manifest as non-periodic vibration signals, whose vibration characteristics may vary over time or be related to external excitations.

[0089] Uniform energy distribution: Cage failures typically result in energy distribution over a wider frequency band, so an increase in energy may be observed across the entire spectrum.

[0090] ③Inner and outer ring faults

[0091] Mid- to high-frequency components: Faults in the inner and outer rings typically produce noticeable components in the mid- to high-frequency range, which correspond to the structural characteristics of the inner or outer ring.

[0092] Periodic or non-periodic characteristics: Faults in the inner and outer races may manifest as periodic or non-periodic vibration signals, depending on the nature and severity of the fault.

[0093] Energy distribution is relatively concentrated: inner and outer ring failures usually result in vibration energy being relatively concentrated in a specific frequency band, but compared to rolling element failures, their energy distribution may be wider.

[0094] Therefore, the selection of wavelet basis functions in wavelet transform processing is as follows:

[0095] For fault characteristics where the frequency components of local faults are usually concentrated within a specific frequency band, the Daubechies wavelet is selected. It has wavelet basis functions with compact support and good frequency response, providing better time and frequency localization characteristics.

[0096] For signals with oscillatory characteristics or mid-to-high frequency inner and outer ring fault characteristics, the Complex Morlet wavelet is selected. It is a complex wavelet basis function composed of Gaussian and complex exponential functions, and is particularly suitable for analyzing non-stationary signals and signals with oscillatory characteristics.

[0097] For pulse signals that typically represent periodic rolling element faults, there are obvious harmonic components. The Meyer wavelet, a quasi-continuous wavelet basis function, is chosen because it has a smooth frequency response and good time-frequency localization characteristics.

[0098] In a further optimized scheme, the wavelet transform processing procedure described above includes the following steps:

[0099] S301 randomly samples segments of the fault heterogeneous signal at a rate of 10%, and applies multiple different wavelet basis functions to the segments in sequence. The wavelet transform of the single wavelet basis function is then performed on the heterogeneous signal at different scales of 256, 128...8, and the reconstruction error is calculated.

[0100] Among them, a lower reconstruction error means that the wavelet basis function at the current scale can more accurately reconstruct the original signal, so this scale is selected.

[0101] S302 calculates and selects wavelet basis functions of different scales to perform multi-scale wavelet transform, and obtains a series of wavelet subbands of multi-source heterogeneous signals with different wavelet basis function inner products.

[0102] S302 calculates the energy of each wavelet subband under different wavelet basis function selections and uses the square of the absolute value of the wavelet coefficients to represent the energy;

[0103] S303 selects wavelet coefficient amplitude weighted fusion to fuse the energy of different wavelet subbands, so as to integrate information at different scales and make the time-frequency spectrum more representative and comprehensive.

[0104] The multi-scale wavelet subband energy wavelet coefficient amplitude weighted fusion is specifically as follows:

[0105] X df1 =WS Daubechies (X df )*a+WS Meyer (X df )*b+WS Complex Morlet (X df )*c

[0106] Among them, wavelet subband processing is abbreviated as WS,X df For the preprocessed multimodal raw heterogeneous data, X df1 These are the CWT modal signal characteristics processed by wavelet basis functions at different scales, where abc is the wavelet basis amplitude weighting weight.

[0107] S304 plots a time-frequency spectrum with time on the horizontal axis and frequency on the vertical axis based on the energy of the fused wavelet subbands.

[0108] Energy levels can be represented using color or brightness, thus illustrating the energy distribution of multi-source heterogeneous signals at different times and frequencies. The subbands of the fused wavelets are mapped to a time-frequency spectrum X. DF1 ;

[0109] Step 4: Generate a grayscale image with multi-scale STFT feature extraction;

[0110] Multi-scale STFT feature extraction is performed on the data-level fused feature vector matrix A1, and the corresponding spectral energy or other statistical features are calculated to generate a grayscale image of multi-scale STFT feature extraction.

[0111] The basic calculation formula for STFT is as follows:

[0112]

[0113] In the formula, x(t) is the time-domain signal, and h(t-ω) is the window function;

[0114] For the aforementioned data-level fusion feature vector matrix A1, different window sizes (time scales) and window overlap degrees are used to calculate spectral features at different scales. Specifically, selecting a longer window can capture information at low frequencies or long time scales, while selecting a shorter window can extract information at high frequencies or short time scales.

[0115] Further optimization was carried out by selecting multi-scale STFT feature extraction. After comparing and optimizing the feature extraction at each scale, the Hanning window was selected, and the window overlap was set to 75%.

[0116] Further optimization of the scheme, the multi-scale STFT feature extraction process includes the following steps:

[0117] S401. Define different window sizes: Select a series of different window sizes in sequence according to the scale of 128, 64, ... 8 to generate different time scales in the time domain;

[0118] S402. Perform multi-scale STFT transformation on the data-level fused feature vector matrix A1: For each window size, apply the window function to each time step of the feature vector matrix A1, and then use Fast Fourier Transform (FFT) to transform the signal to the frequency domain to generate spectral information at different scales;

[0119] S403. Calculate spectral energy or other statistical characteristics: For the spectrum at each scale, spectral energy or other statistical characteristics, such as mean, variance, maximum, etc., can be calculated. These characteristics can provide information about the frequency content of the signal at different scales;

[0120] S404. The spectral reconstruction error is calculated by inversely transforming the STFT results under different window lengths back to the time domain. A smaller spectral reconstruction error indicates better frequency domain characterization, allowing for the selection of the optimal window length setting.

[0121] S405. Generate a grayscale image of STFT feature extraction with optimal window length settings: visualize its spectral features as a grayscale image. Spectral energy or other feature values ​​can be mapped to grayscale levels to form a grayscale image;

[0122] X df2 =STFT d1 (X df )

[0123] X df2 It is the optimal representation of the modal signal characteristics of STFT with different window functions of different scales after comparison of spectral reconstruction error;

[0124] The optimal representation of the frequency domain signal obtained by the above multi-scale STFT is visualized and mapped to a time-frequency spectrum X. DF2 ;

[0125] Step 5: Generate a new time-frequency feature image set Image;

[0126] The grayscale images of feature vectors extracted at different scales are weighted and fused, and different weights are assigned according to the importance of different scales; the above images are then stitched together to form a longer feature vector and then normalized to generate a new time-frequency feature image set Image.

[0127] Specifically, multi-scale fusion is performed on the grayscale image features mapped by STFT and wavelet transform. First, feature representations at different scales are extracted for each feature. Then, the features at different scales are stitched together and fused. Finally, they are superimposed to form a multi-channel image and constitute a grayscale image dataset that fuses the features of both.

[0128] Further optimization of the scheme, the process of generating the time-frequency feature image set Image includes the following steps:

[0129] S501. Define importance weights at different scales: Based on application requirements and domain knowledge, analyze the training data to observe the variance, mean, and other statistics of the feature vectors at different scales, and determine the weights based on their changes and their contribution to fault diagnosis.

[0130] S502. Multiply each feature image by the corresponding weight: For each feature image at each scale, multiply it by the corresponding weight to weight the importance of that scale in the overall features to represent the feature information;

[0131] S503. Stitching weighted feature images: The weighted feature images are stitched together in chronological order to form a longer feature vector, thereby capturing more comprehensive time-frequency features;

[0132] S504. Normalize the feature vector by performing weighted feature-level fusion of the grayscale features representing different time-frequency characteristics mentioned above; normalize the generated longer feature vector to ensure that the feature values ​​are within a certain range, so as to avoid the influence of numerical differences of features at different scales on the final result; the normalization formula is as follows:

[0133] X DF =cat(X) Df1 *A + X Df2 *B)

[0134] Where, X Df1 X Df2 These represent the CWT and SFT processes mapped to time-frequency images, respectively. X DF This represents the features of the spliced ​​and fused two-dimensional image, and cat(·) indicates splicing along the feature dimension.

[0135] S505. Generate a new time-frequency feature image set Image: Reorganize the normalized feature vectors into a time-frequency feature image set Image, where each image represents a time step of the overall features; By weighted fusion of feature images at different scales, a new time-frequency feature image set is generated to comprehensively utilize feature information at different scales, improve the expressive power and robustness of features, and thus better describe the time-frequency features of the signal;

[0136] Step 6: Build the VIT model;

[0137] The VIT model consists of an input layer, a multi-scale image patch partitioning layer, an image patch embedding layer, a positional encoding layer, a Transformer encoder layer (each encoder layer contains two sub-layers: a multi-head self-attention mechanism and a feedforward neural network), an adaptive pooling layer, an MLP layer, and an output layer.

[0138] Based on the algorithm principle of attention mechanism, a fault classification VIT (Vision Transformer) model based on multi-scale graph attention fusion network (Transformer) architecture is constructed. The construction process includes the following steps:

[0139] S601. Input data preprocessing;

[0140] For the input graph structure data, different graph embeddings are used for feature extraction to obtain multiple feature representations; each graph embedding may focus on different local or global features of the graph structure.

[0141] S602. Pass the preprocessed input data through an embedding layer (such as...) Figure 6 (as shown) is converted into an embedding vector;

[0142] For image data, features are extracted using 2D image embedding of different sizes (16*16 / 12*12 / 8*8), and each image patch is mapped to the embedding feature space and flattened into a vector.

[0143] The formula for calculating graph embedding mapping is as follows:

[0144]

[0145] Among them, K i u-i，v-j These represent graph embeddings of different sizes and spatial locations, respectively, and Zi represents the graph features extracted from graph embeddings of different sizes;

[0146] S603. Location coding;

[0147] Positional information is introduced, and a weighted average of the input vectors of different graph embeddings is performed using a positional encoding form of sine and cosine coding. Sequence information is introduced into the self-attention mechanism so that the model can perceive the relative positional relationships of different positions in the input sequence.

[0148] S604. Construct a multi-layer Transformer encoder;

[0149] Constructing a multi-layer Transformer encoder (containing a self-attention sublayer and a fully connected sublayer) involves the following process;

[0150] 1) Self-attention sublayers and layer normalization;

[0151] In the self-attention sublayer, for each feature representation extracted from the graph embedding, it can be mapped into query, key, and value form using a linear mapping (e.g., a fully connected layer). This is to transform the feature representation into a form suitable for computation by the attention mechanism, yielding the self-attention output. The output is then residually concatenated with the input, and layer normalization is performed.

[0152] Self-attention mechanisms generally employ a query-key-value model. For an input H = (h1, h2, ..., hn), it is multiplied by three matrices Wq, Wk, and Wv to obtain the query Q, key K, and value V, as detailed below:

[0153] Q = H·W k

[0154] K = H·W k

[0155] V = H·W k

[0156] 2) Feedforward neural network sublayer;

[0157] In the fully connected sublayers, the GELU nonlinear transformation function is used for nonlinear mapping. The output of each layer serves as the input to the next layer, forming a sequence. Dropout is used in the last layer for regularization to prevent overfitting. Residual connections are introduced to enhance information flow, combined with layer normalization operations.

[0158] 3) Multi-head self-attention quantum layer;

[0159] In the input encoding stage, positional encoding characteristics are assigned to multi-source information. The positional encoding is added to the word embedding vector to form the final input matrix. The features extracted from each graph embedding are treated as a query vector Q and a key vector K. A Self-Attention mechanism is used to calculate the similarity between each query vector Q and all key vectors K, obtaining attention scores (weight coefficients), which are then normalized using the Softmax function. This yields the weights corresponding to each key vector K, which are used to calculate the weighted sum.

[0160] For each query, key, and value extracted from a graph embedding, a multi-head attention mechanism can be used to compute the correlations and weights between features. This attention mechanism can be used to weightedly fuse features extracted from different graph embeddings. The correlations between features are dynamically learned based on the input features, and the feature weights are adaptively adjusted for weighted summation. Layer standardization is then performed to obtain the fused feature representation. This improves the understanding and modeling ability of graph structures, thereby obtaining more discriminative feature representations.

[0161] The operating mechanism of multi-head self-attention sublayers under different graph embedding inputs:

[0162] For input X with different graph embeddings, we define multiple different sets of W. q W k W v Perform linear transformations to generate different Q, K, and V values ​​respectively.

[0163] Corresponding to the number of multi-head attention, with 8 groups of W q W k W v Taking this as an example, performing self-attention operations on each of them yields Z0-Z7.

[0164] Before outputting to the next layer, we need to concat the corresponding multi-head self-attention outputs Z0-Z7 of different graph embeddings together, multiply them by matrix W0 to perform a linear transformation to reduce the dimensionality, and obtain Z.

[0165] 4) Output layer;

[0166] The feature vectors obtained from the last or multiple layers of the encoder can be used for various downstream tasks. Depending on the specific task, appropriate output layers, such as softmax classifiers, can be added.

[0167] S605. Loss Function and Optimization Algorithm;

[0168] The cross-entropy loss function is used to calculate the loss on the processed results. The Adam optimization algorithm is selected, and the network parameters are updated based on the gradient.

[0169] By building a VIT model, a multi-scale attention mechanism can be implemented, thereby effectively processing multimodal data and obtaining better feature representation and performance;

[0170] A simplified diagram of the ViT model is shown below:

[0171] L_vit=softmax(MLP(Trans_Encoder N (PE((PatchEmbed((X DFi ) patch_size_i ) N )))

[0172] In the ViT model, X DFi These are CWT and STFT fused grayscale images that have undergone three-channel transformation respectively. `patch_size_i` represents the image embedding operations of different sizes (16*16 / 12*12 / 8*8). `PatchEmbed(·)` is a function that segments the input image (x) into image patches and converts them into embedding vectors through linear projection. `Trans_Encoder(·)` is an N-layer Transformer encoder function used to encode the image patch sequence. A multi-scale attention mechanism is introduced in the self-attention sublayer of the Transformer encoder, enabling the ViT model to model and extract input features at different scales, thus obtaining a more comprehensive feature representation. `MLP(·)` is a multilayer perceptron function composed of fully connected layers and activation functions, which maps the transformed representation to the output layer of the classification task.

[0173] Step 7: Obtain the classification results of bearing failure types;

[0174] The fused time-frequency feature image set Image is divided into a training set and a validation set, and then imported into a preset fault classification model to obtain the classification results of the health or corresponding fault type status.

[0175] The different types of datasets obtained in step 5 above are divided into training set and validation set in a 4:1 ratio. First, the training sets of the two different preprocessed datasets are input into the multi-scale graph attention fusion network architecture established in step 6. Then, the weight values ​​in the fault detection model are trained using the backpropagation algorithm. The validation set is used to judge the training effect of the bearing defect detection model in order to obtain a fully trained mechanical defect detection model. Finally, the test set data is used to verify the reliability of the model.

[0176] The data to be detected is collected based on a multi-scale graph attention fusion network architecture. The preprocessing process in step 1 is repeated and the data is input into the trained bearing defect multi-channel data detection model to detect the defects of the bearing during operation. The defect conditions are labeled by the results of fuzzy comprehensive evaluation theory, which are output by two feature extraction networks, characterize the mechanical defect conditions, and are fused by decision-making.

[0177] To implement the aforementioned bearing diagnostic method based on multimodal and multiscale fusion networks, this application also proposes the following bearing diagnostic system based on multimodal and multiscale fusion networks, such as... Figure 4 As shown, the system includes:

[0178] The module comprises a raw signal acquisition module, a multi-source data processing module, a multi-scale feature fusion module, a multi-scale feature extraction module, and a diagnostic classification module, as detailed below:

[0179] The raw signal acquisition module is used to acquire signals of N modes of rolling bearings, including data information such as rolling bearing sound, vibration, temperature and rotational speed;

[0180] The multi-source data processing module is used to preprocess N modal signals. It can perform corresponding traditional signal processing analysis on the raw acquired data, and then perform data-level fusion of the processed data according to the modal dimension to construct a vector matrix containing feature parameters, denoted as the data-level feature vector matrix A1.

[0181] The multi-scale feature fusion module performs continuous wavelet transform and STFT transform on the initially constructed parametric feature vector matrix and fuses the features; it performs weighted fusion on the grayscale images of the feature vectors extracted at different scales, assigning different weights according to the importance of different scales; it continues to stitch the above images together to form a longer feature vector, which is then normalized, and finally generates a new time-frequency feature image set Image.

[0182] The multi-scale feature extraction module is used for the input image. First, it is converted into image patches through an embedding layer and divided into embedding representations of different scales according to certain rules. For each scale, the ViT model assigns one or more attention heads. Each attention head learns different weights and spatial relationships to perform self-attention computation on the input embedding representation. Each attention head generates an attention matrix, representing the attention weights between different locations. These matrices are then used to generate the final multi-scale attention representation in the form of attention convergence.

[0183] The diagnostic classification module is used in the ViT model to map image features to the classification task space and generate classification labels using MLP and softmax functions;

[0184] MLP extracts abstract feature representations by stacking multiple fully connected layers and activation functions; the softmax function normalizes the output of MLP to obtain the probability prediction result for each category, so as to make classification decisions; the above operations enable the ViT model to learn the mapping relationship from input image to output label during training and to make classification predictions for input image during the inference stage.

[0185] To realize the bearing diagnosis method and system based on multimodal and multiscale fusion network, this application also proposes the following upper computer industrial control equipment for implementing bearing fault diagnosis and analysis, which includes a processor and a memory;

[0186] The memory is used to store computer-executable programs; the processor reads the computer-executable programs from the memory and executes them;

[0187] And a computer-readable storage medium for storing the analysis software of the industrial control multi-channel acquisition system based on the above-mentioned bearing diagnostic method based on multimodal and multi-scale fusion network.

[0188] Although the present invention has been shown and described in the above embodiments, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A bearing diagnostic method based on a multimodal and multi-scale fusion network, characterized in that: The implementation steps include the following: Step 1: Obtain a dataset of multi-source heterogeneous signals under different operating conditions and various fault types; Step 2: Construct a data-level feature vector matrix based on multimodal data fusion; Multiple modal signals are preprocessed by feature extraction and information fusion to obtain feature vectors for each modal signal. Each feature vector contains features describing different modal information. The features of different modal signals are concatenated and averaged. Then, vibration, temperature, and rotational speed are used as the main sequences to perform data-level fusion according to modal dimensions and construct a feature vector matrix. Step 3: Generate grayscale images with multi-scale CWT feature extraction; The data-level fused feature vector matrix obtains features at different frequency ranges or time scales by selecting different wavelet functions or scale parameters, and calculates the features at each scale to generate a grayscale image with multi-scale CWT feature extraction. Step 4: Generate a grayscale image with multi-scale STFT feature extraction; Step 5: Generate a new time-frequency feature image set Image; The grayscale images of feature vectors extracted at different scales are weighted and fused, and different weights are assigned according to the importance of different scales; the above images are then stitched together to form a longer feature vector and then normalized to generate a new time-frequency feature image set Image. Multi-scale fusion of features from STFT grayscale images and wavelet transform images is performed. First, feature representations at different scales are extracted for each feature. Then, the features at different scales are stitched together and fused. Finally, they are superimposed to form a multi-channel image and constitute a grayscale image dataset that fuses the features of both. Step 6: Build the VIT model; The VIT model consists of an input layer, a multi-scale image patch partitioning layer, an image patch embedding layer, a position coding layer, a Transformer encoder layer, an adaptive pooling layer, an MLP layer, and an output layer. Constructing a fault classification VIT model based on a multi-scale graph attention fusion network architecture includes the following steps: S601. Input data preprocessing; For the input graph structure data, different graph embeddings are used for feature extraction to obtain multiple feature representations; each graph embedding may focus on different local or global features of the graph structure. S602. Convert the preprocessed input data into an embedding vector through the embedding layer; For image data, features are extracted using 2D image embedding of different sizes (16*16 / 12*12 / 8*8), and each image patch is mapped to the embedded feature space and flattened into a vector. The formula for calculating graph embedding mapping is as follows: in, Represents the graph features extracted from graph embeddings of different sizes; Image embeddings representing different sizes and spatial locations; S603. Location coding; Positional information is introduced, and a weighted average of the input vectors of different graph embeddings is performed using a positional encoding form of sine and cosine coding. Sequence information is introduced into the self-attention mechanism so that the model can perceive the relative positional relationships of different positions in the input sequence. S604. Construct a multi-layer Transformer encoder; S605. Loss Function and Optimization Algorithm; The cross-entropy loss function is used to calculate the loss on the processed results, and the Adam optimization algorithm is selected to update the network parameters based on the gradient. Step 7: Obtain the classification results of bearing failure types; The fused time-frequency feature image set Image is divided into a training set and a validation set, and then imported into a preset fault classification model to obtain the classification results of the health or corresponding fault type status.

2. The bearing diagnosis method based on multimodal and multiscale fusion network according to claim 1, characterized in that: In step 1, the sensor integrated monitoring module is embedded into the bearing to collect signals of N modes in real time. The signals of N modes include multi-source signals under different fault conditions. The types of multi-source signals include, but are not limited to, the vibration, temperature, and speed of the bearing, and the voltage, current, and power of the motor under operating conditions.

3. The bearing diagnosis method based on multimodal and multiscale fusion network according to claim 1, characterized in that: In step 4, the data-level fusion feature vector matrix A1 is subjected to multi-scale STFT feature extraction, and the corresponding spectral energy or other statistical features are calculated to generate a grayscale image of multi-scale STFT feature extraction. The basic calculation formula for STFT is as follows. In the formula, x(t) is the time-domain signal, and h(t-ω) is the window function.

4. The bearing diagnosis method based on multimodal and multiscale fusion network according to claim 1, characterized in that: In step 7, the different types of datasets obtained in step 5 are divided into training and validation sets in a 4:1 ratio. First, the training sets of the two different preprocessed datasets are input into the multi-scale graph attention fusion network architecture established in step 6. Then, the weight values ​​in the fault detection model are trained using the backpropagation algorithm. The validation set is used to judge the training effect of the bearing defect detection model to obtain a fully trained mechanical defect detection model. Finally, the test set data is used to verify the reliability of the model. The data to be detected is collected based on a multi-scale graph attention fusion network architecture. The preprocessing process in step 1 is repeated and the data is input into the trained bearing defect multi-channel data detection model to detect the defects of the bearing during operation. The defect conditions are labeled by the results of fuzzy comprehensive evaluation theory, which are output by two feature extraction networks, characterize the mechanical defect conditions, and are fused by decision-making.

5. The bearing diagnostic system based on the bearing diagnostic method according to any one of claims 1 to 4, characterized in that: It includes a raw signal acquisition module, a multi-source data processing module, a multi-scale feature fusion module, a multi-scale feature extraction module, and a diagnostic classification module; The raw signal acquisition module is used to acquire signals of N modes of rolling bearings, including data information such as rolling bearing sound, vibration, temperature and rotational speed; The multi-source data processing module is used to preprocess N modal signals. It can perform corresponding traditional signal processing analysis on the raw acquired data, and then perform data-level fusion of the processed data according to the modal dimension to construct a vector matrix containing feature parameters, denoted as the data-level feature vector matrix A1. The multi-scale feature fusion module performs continuous wavelet transform and STFT transform on the initially constructed parametric feature vector matrix and fuses the features; it performs weighted fusion on the grayscale images of the feature vectors extracted at different scales, assigning different weights according to the importance of different scales; it continues to stitch the above images together to form a longer feature vector, which is then normalized, and finally generates a new time-frequency feature image set Image. The multi-scale feature extraction module is used for the input image. First, it is converted into image patches through an embedding layer and divided into embedding representations of different scales according to certain rules. For each scale, the ViT model assigns one or more attention heads. Each attention head learns different weights and spatial relationships to perform self-attention computation on the input embedding representation. Each attention head generates an attention matrix, representing the attention weights between different locations. These matrices are then used to generate the final multi-scale attention representation in the form of attention convergence. The diagnostic classification module is used in the ViT model to map image features to the classification task space and generate classification labels through MLP and softmax functions; MLP extracts abstract feature representations by stacking multiple fully connected layers and activation functions; The softmax function normalizes the output of the MLP to obtain the probability prediction results for each category, so as to make classification decisions; The above operations enable the ViT model to learn the mapping relationship from the input image to the output label during the training process, and to provide a classification prediction for the input image during the inference phase.

6. A host computer industrial control device that applies the bearing diagnostic system based on a multimodal and multi-scale fusion network as described in claim 5, characterized in that: It includes a processor and a memory, the memory being used to store computer-executable programs, and the processor reading the computer-executable programs from the memory and executing them; And a computer-readable storage medium for storing the analysis software of the industrial control multi-channel acquisition system based on the above-mentioned bearing diagnostic method based on multimodal and multi-scale fusion network.

Images