A fault diagnosis method for workshop equipment multi-modal data fusion

Abstract

The application discloses a kind of fault diagnosis methods of workshop equipment multimodal data fusion, including obtaining the signal data of different modal of workshop equipment;The signal data of different modal is preprocessed, and the signal data of different modal after preprocessing is obtained;The signal data of different modal after preprocessing is input into the fault identification model trained, and fault identification processing is carried out, and the fault diagnosis result of workshop equipment is obtained;The application can be combined with the fault identification processing of multimodal signal data of workshop equipment, and the fault diagnosis result of workshop equipment is obtained, to improve the accuracy of workshop equipment fault diagnosis.

Description

Technical Field

[0001] This invention relates to the field of fault diagnosis technology, and in particular to a fault diagnosis method for workshop equipment using multimodal data fusion. Background Technology

[0002] As the physical center of an enterprise, the workshop is not only the specific executor of the manufacturing process but also the hub where manufacturing data generates value. However, frequent workshop disturbances can ultimately lead to scheduling plan failures, low scheduling execution rates, and significant production stability issues. Taking a domestic aerospace manufacturing workshop as an example, the average daily disturbance exceeds 40 times, with equipment-related disturbances (including equipment failures and processing defects caused by equipment degradation) accounting for over 35%. Equipment failure has become the number one disturbance factor affecting continuous production in the workshop. However, fault diagnosis and predictive maintenance of workshop equipment face many challenges, such as numerous complex mechanical fault sources, limitations in sensor application, and the limitations of single-modal diagnosis. Previous studies have mostly relied on direct data fusion or linear connections to utilize the characteristics of different modes. However, misalignment and heterogeneity between signals can easily lead to the loss of important information between modes or overfitting. Summary of the Invention

[0003] The purpose of this invention is to provide a fault diagnosis method based on multimodal data fusion for workshop equipment. This method performs fault identification processing on multimodal signal data of workshop equipment based on a fault identification model to obtain fault diagnosis results for the workshop equipment, thereby improving the accuracy of fault diagnosis for workshop equipment.

[0004] To achieve the above objectives, the present invention employs the following technical solution:

[0005] This invention provides a fault diagnosis method for workshop equipment based on multimodal data fusion, comprising:

[0006] Acquire signal data from different modes of workshop equipment;

[0007] The signal data of the different modes are preprocessed to obtain preprocessed signal data of the different modes; the preprocessing includes normalization and splicing of the data.

[0008] The preprocessed signal data of different modes are input into the trained fault identification model for fault identification processing to obtain the fault diagnosis results of the workshop equipment. The fault identification model includes a feature extraction module, a multimodal feature fusion module, a global average pooling layer, a normalization layer, and a fully connected layer.

[0009] Optionally, the steps for obtaining the trained fault identification model include:

[0010] Acquire signal datasets of different modes under normal operating conditions and fault conditions of workshop equipment, and preprocess the signal datasets of different modes to obtain preprocessed signal datasets of different modes;

[0011] The preprocessed signal datasets of different modes are divided into a sample set, a training set, and a validation set.

[0012] Device status labels are applied to the preprocessed signal data of different modes to generate labels for the sample set;

[0013] The data from the training set are sequentially input into the feature extraction module, multimodal feature fusion module, global average pooling layer, normalization layer, and fully connected layer of the initial fault identification model for training, to obtain the fault diagnosis results of the training set.

[0014] Fault diagnosis is performed based on the training set, and the optimal model parameters are determined by minimizing the cross-entropy loss function, thereby training a fault identification model.

[0015] Optionally, the feature extraction module includes a first MDRSN submodule, a second MDRSN submodule, a third MDRSN submodule, and a fourth MDRSN submodule connected in sequence; each of the first MDRSN submodule, the second MDRSN submodule, the third MDRSN submodule, and the fourth MDRSN submodule includes multiple sequentially connected MRSBU units.

[0016] The multimodal feature fusion module includes a first MFF submodule, a second MFF submodule, a third MFF submodule, and a fourth MFF submodule that are sequentially connected, and the first MDRSN submodule is connected to the first MFF submodule, the second MDRSN submodule is connected to the second MFF submodule, the third MDRSN submodule is connected to the third MFF submodule, and the fourth MDRSN submodule is connected to the fourth MFF submodule.

[0017] Optionally, the signal data of the different modes are preprocessed to obtain preprocessed signal data of the different modes, including:

[0018] Normalize the signal data of different modes to obtain normalized signal data of different modes;

[0019] Normalized signal data of different modes are concatenated along the channel dimension to obtain preprocessed signal data of different modes.

[0020] Optionally, the preprocessed signal data of different modes is input into a trained fault identification model for fault identification processing to obtain fault diagnosis results for workshop equipment, including:

[0021] The preprocessed signal data of different modes are input into the first MDRSN submodule for the first feature extraction to obtain the first extraction result;

[0022] The first extraction result is input into the second MDRSN submodule for a second feature extraction to obtain the second extraction result;

[0023] The second extraction result is input into the third MDRSN submodule for a third feature extraction, resulting in the third extraction result.

[0024] The third extraction result is input into the fourth MDRSN submodule for the fourth feature extraction, resulting in the fourth extraction result;

[0025] The first extraction result is input into an MFF submodule for LCAT processing to obtain the first processing result; the LCAT processing includes multimodal attention on the first extraction result to extract complementary information between multimodal signals;

[0026] The second extraction result and the first processing result are input into the second MFF submodule to perform the first weighted feature fusion to obtain the first fusion result;

[0027] The third extraction result and the first fusion result are input into the third MFF submodule to perform a second weighted feature fusion to obtain the second fusion result.

[0028] The fourth extraction result and the second fusion result are input into the fourth MFF submodule to perform a third weighted feature fusion to obtain the third fusion result.

[0029] The third fusion result is input into the global average pooling layer, normalization layer, and fully connected layer to output the fault diagnosis results of the workshop equipment.

[0030] Optionally, the MRSBU unit includes a group convolutional layer, a multi-scale depth-separable convolutional layer, and a soft-thresholding layer. The group convolutional layer reduces the channel dimension of the input features. After the channel dimension is reduced, the input features are entered into the multi-scale depth-separable convolutional layer to extract features in the receptive field of view at different scales. The extracted multi-scale features are then concatenated on the reduced channel dimension. The concatenated features are then subjected to soft-threshold adaptive denoising. The result of the adaptive denoising is then residually connected to the input features to output the result processed by the MRSBU unit.

[0031] Optionally, the group convolutional layers reduce the channel dimension of the input features, including:

[0032] The MRSBU unit uses group convolution to process the input feature data. The data is divided into G groups according to the channel, and each group contains data of the same mode.

[0033] Each modality group is processed using a convolutional kernel of size 1, and the number of channels is changed to the number of output channels. The formula is as follows:

[0034]

[0035] in, This indicates the input of the MRSBU unit. This represents the output after channel dimension reduction via group convolutional layers. and These represent the time dimensions of the input and output, respectively. This indicates the channel dimension of the MRSBU unit input. This indicates the channel dimension of the MRSBU unit output, and * indicates the convolution operation. denoted by 1, G represents the number of modes in the input features, and Concat represents the channel dimension concatenation operation; N represents the number of convolutional kernels in the convolutional layer group; g represents the g-th mode in the mode group (1, G); a and b are loop variables in the formula.

[0036] Optionally, after the channel dimension is reduced, it enters a multi-scale depthwise separable convolution to extract features at different scales of the receptive field of view, and then concatenates the extracted multi-scale features on the reduced channel dimension, including:

[0037] After reducing the channel dimension, data of the same modality are stored in the same data set. For each set, N convolutional kernels with a size of [missing value] are performed. Separable convolutions extract features from receptive fields of view at different scales, yielding... The features extracted by these N convolutional kernels are concatenated along the reduced channel dimension to obtain the output of a multi-scale depth-separable convolutional layer. The formula is as follows:

[0038]

[0039]

[0040] in, Indicates the size is The weights of separable convolutions, Indicates the first Each mode has a weight of The output of separable convolution, This represents the output of a multi-scale depth-separable convolutional layer. This represents the size of the time dimension; n is an integer variable. .

[0041] Optionally, the second, third, and fourth MFF submodules each include: a first input layer, an LCAT layer, a first input layer, a convolutional pooling layer, a normalized convolutional layer, a WFF layer, and an IRMLP layer; the processing includes:

[0042] The input to the first input layer is processed by the LCAT layer to perform multimodal attention on the extracted results, extracting complementary information between multimodal signals and outputting the first branch result. The input to the second input layer is first processed by a convolutional pooling layer to obtain the convolutional pooling result, and then by a normalized convolutional layer to obtain the second branch result. The first branch result and the second branch result are simultaneously processed by WFF weighted feature fusion and IRMLP inverse residual processing, and the residuals are concatenated with the convolutional pooling result to output the fusion result of the MFF submodule. The extraction results of the second, third, and fourth MFF submodules, respectively, are connected to the output of the MDRSN submodule as the input to the first input layer. The output of the previous MFF submodule is used as the input to the second input layer.

[0043] Optionally, the LCAT layer is a lightweight cross-modal attention module, which is formed by connecting cross-modal self-attention modules in series along the channel direction;

[0044] The LCAT layer is configured as follows:

[0045] Suppose that the multimodal data input to the LCAT layer has a total of G modes. and All are one of the G modes, that is This represents the current subject mode vector. Represented as Provide a reference mode vector; After being convolved with a scale of 1, the matrix is ​​converted into K and V matrices, which are then processed using... and Indicate; will After being convolved with a scale of 1, it is converted into G Q matrices, and then... The formulas are expressed as follows:

[0046]

[0047]

[0048]

[0049]

[0050] in , and These represent the weights of the 1×1 convolution kernels used to generate the query, key, and value matrices, respectively; that is, the weights of the 1×1 convolution kernels used to generate the Q, K, and V matrices. This indicates the Softmax normalization operation. This represents the current subject mode vector. Represented as Provide a reference mode vector. and This indicates the current main feature matrix The matrix obtained by convolution, Indicates from the reference matrix The matrix obtained by convolution, This refers to the size of the time dimension; C represents the LCAT layer channel dimension.

[0051] G modal vectors After cross-modal scaled dot product attention processing, G are generated. Vectors, and G of them The vectors are concatenated along the channel dimension to obtain ;

[0052] In cross-modal scaled dot product attention processing, generate The formula is as follows:

[0053]

[0054]

[0055]

[0056] in, It is a ReLU activation function. is the weight of the convolution kernel with size 1; t represents the time parameter; m and n' represent the loop parameters in the channel dimension C of the LCAT layer, respectively;

[0057] Will The convolution reduces the vector to the current main modality vector. The same dimensions, yield and will Perform residual processing to obtain the current main mode vector. After cross-modal attention processing The formula is as follows:

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] in, and Refers to learnable weights;

[0064] Based on the cross-modal attention processing described above, G values ​​are obtained. and these G By concatenating along the channel dimension, we obtain the output after passing through the LCAT layer. The formula is as follows:

[0065]

[0066] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0067] (1) The present invention identifies faults in workshop equipment based on a fault identification model, which improves the accuracy of fault diagnosis compared to diagnostic methods that use only a single mode signal.

[0068] (2) This invention improves the Deep Residual Shrink Network (DRSN) by introducing the idea of ​​multi-scale group convolution, so that it can achieve better multi-modal signal feature extraction and noise reduction when the fault signal categories are different. Attached Figure Description

[0069] Figure 1 The diagram shown is a flowchart of a fault diagnosis method in one embodiment of the present invention.

[0070] Figure 2 The diagram shown is a schematic diagram of the fault identification model structure in one embodiment of the present invention;

[0071] Figure 3 The diagram shown is a schematic representation of the processing flow of the MRSBU unit in one embodiment of the present invention.

[0072] Figure 4 The diagram shown is a schematic representation of the processing flow of the MFF submodule in one embodiment of the present invention.

[0073] Figure 5 The diagram shown is a schematic representation of the processing flow of the LCAT layer in one embodiment of the present invention.

[0074] Figure 6 The diagram shown is a schematic representation of the processing flow of the WFF layer in one embodiment of the present invention.

[0075] Figure 7The figure shown is a visualization diagram of the fault diagnosis result t-sne in one embodiment of the present invention; Detailed Implementation

[0076] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0077] Example 1

[0078] like Figure 1 As shown in the figure, this invention provides a fault diagnosis method for workshop equipment based on multimodal data fusion, comprising the following steps:

[0079] S01: Acquire signal data of different modes of workshop equipment;

[0080] S02: Preprocess the signal data of the different modes to obtain preprocessed signal data of the different modes; the preprocessing includes normalization and splicing of the data.

[0081] S03: Input the preprocessed signal data of different modes into the trained fault identification model, perform fault identification processing, and obtain the fault diagnosis results of the workshop equipment; the fault identification model includes a feature extraction module, a multimodal feature fusion module, a global average pooling layer, a normalization layer, and a fully connected layer.

[0082] Specifically, the signal data of different modes of the workshop equipment can be obtained through a microphone sensor, accelerometer and current transformer of model BS1MPA201, which can provide signal data of different modes such as sound, vibration or motor current.

[0083] Specifically, the fault identification model used in this invention is the FMDS (Fusion of Multimodal Data using an Improved Deep Residual Shrinkage Network) network model.

[0084] This invention improves the accuracy of fault diagnosis by using a fault identification model to identify faults in multimodal signal data of workshop equipment, compared to diagnostic methods that only use single-mode signals.

[0085] In this embodiment, the step of obtaining the trained fault identification model in step S03 specifically includes:

[0086] Acquire signal datasets of different modes under normal operating conditions and fault conditions of workshop equipment, and preprocess the signal datasets of different modes to obtain preprocessed signal datasets of different modes;

[0087] The preprocessed signal datasets of different modes are divided into a sample set, a training set, and a validation set.

[0088] Device status labels are applied to the preprocessed signal data of different modes to generate labels for the sample set;

[0089] The data from the training set are sequentially input into the feature extraction module, multimodal feature fusion module, global average pooling layer, normalization layer, and fully connected layer of the initial fault identification model for training, to obtain the fault diagnosis results of the training set.

[0090] Fault diagnosis is performed based on the training set, and the optimal model parameters are determined by minimizing the cross-entropy loss function, thereby training a fault identification model.

[0091] like Figure 2 As shown, in this embodiment, the fault identification model includes a feature extraction module, a multimodal feature fusion module, a global average pooling layer, a normalization layer, and a fully connected layer.

[0092] The feature extraction module includes a first MDRSN (Multi-scale Deep Residual Shrinking Network) submodule, a second MDRSN submodule, a third MDRSN submodule, and a fourth MDRSN submodule. Each of the first, second, third, and fourth MDRSN submodules includes multiple sequentially connected MRSBU (Residual Shrinkage Building Unit) units. Specifically, the number of MRSBU units connected in series in the first, second, third, and fourth MDRSN submodules are 3, 4, 6, and 2, respectively.

[0093] The multimodal feature fusion module includes a first MFF (Multimodal Feature Fusion, multimodal feature fusion submodule, second MFF submodule, third MFF submodule and fourth MFF submodule;

[0094] The first MDRSN submodule, the second MDRSN submodule, the third MDRSN submodule, and the fourth MDRSN submodule are sequentially connected, as are the first MFF submodule, the second MFF submodule, the third MFF submodule, and the fourth MFF submodule. Furthermore, the first MDRSN submodule is connected to the first MFF submodule, the second MDRSN submodule is connected to the second MFF submodule, the third MDRSN submodule is connected to the third MFF submodule, and the fourth MDRSN submodule is connected to the fourth MFF submodule.

[0095] In this embodiment, the signal data of the different modes are preprocessed to obtain preprocessed signal data of the different modes, including:

[0096] Normalize the signal data of different modes to obtain normalized signal data of different modes;

[0097] Normalized signal data of different modes are concatenated along the channel dimension (Concat processing) to obtain preprocessed signal data of different modes.

[0098] In this embodiment, the preprocessed signal data of different modes are input into a trained fault identification model for fault identification processing to obtain fault diagnosis results for workshop equipment, including:

[0099] The preprocessed signal data of different modes are input into the first MDRSN submodule for the first feature extraction to obtain the first extraction result;

[0100] The first extraction result is input into the second MDRSN submodule for a second feature extraction to obtain the second extraction result;

[0101] The second extraction result is input into the third MDRSN submodule for a third feature extraction, resulting in the third extraction result.

[0102] The third extraction result is input into the fourth MDRSN submodule for the fourth feature extraction, resulting in the fourth extraction result;

[0103] The first extraction result is input into an MFF submodule for LCAT processing to obtain the first processing result; the LCAT processing includes multimodal attention on the first extraction result to extract complementary information between multimodal signals;

[0104] The second extraction result and the first processing result are input into the second MFF submodule to perform the first weighted feature fusion to obtain the first fusion result;

[0105] The third extraction result and the first fusion result are input into the third MFF submodule to perform a second weighted feature fusion to obtain the second fusion result.

[0106] The fourth extraction result and the second fusion result are input into the fourth MFF submodule to perform a third weighted feature fusion to obtain the third fusion result.

[0107] The third fusion result is input into the global average pooling layer, normalization layer, and fully connected layer to output the fault diagnosis results of the workshop equipment.

[0108] In this embodiment, the MRSBU unit includes a group convolutional layer, a multi-scale depth-separable convolutional layer, and a soft thresholding layer. The group convolutional layer reduces the channel dimension of the input features. After the channel dimension is reduced, the input features are fed into the multi-scale depth-separable convolutional layer to extract features in the receptive field of view at different scales. The extracted multi-scale features are then concatenated on the reduced channel dimension. The concatenated features are then subjected to soft thresholding adaptive denoising. The result of the adaptive denoising is then residually concatenated with the input features to output the result processed by the MRSBU unit.

[0109] Specifically, the MRSBU unit, while preserving the soft thresholding of the Depth Residual Shrinking Network (DRSN), uses group convolution and multi-scale depthwise separable convolution for multimodal feature extraction, such as... Figure 3 As shown;

[0110] (1) The MRSBU unit uses group convolution to process the input feature data. The data is divided into G groups according to the channel, and each group contains data of the same mode.

[0111] Each modality group is processed using a convolutional kernel of size 1, and the number of channels is changed to the number of output channels. The formula is as follows:

[0112]

[0113] in, This indicates the input of the MRSBU unit. This represents the output after channel dimension reduction via group convolutional layers. and These represent the time dimensions of the input and output, respectively. This indicates the channel dimension of the MRSBU unit input. This indicates the channel dimension of the MRSBU unit output, and * indicates the convolution operation. denoted by 1, G represents the number of modes in the input features, and Concat represents the channel dimension concatenation operation; N represents the number of convolutional kernels in the convolutional layer group; g represents the g-th mode in the mode group (1, G); a and b are loop variables in the formula.

[0114] (2) After reducing the channel dimension, the same set of data stores data of the same modality. For each set, N convolutional kernels with a size of N are performed. Separable convolutions extract features from receptive fields of view at different scales, yielding... The features extracted by these N convolutional kernels are concatenated along the reduced channel dimension to obtain the output of a multi-scale depth-separable convolutional layer. The formula is as follows:

[0115]

[0116]

[0117] in, Indicates the size is The weights of separable convolutions, Indicates the first Each mode has a weight of The output of separable convolution, This represents the output of a multi-scale depth-separable convolutional layer. This represents the size of the time dimension; n is an integer variable. .

[0118] (3) Finally, the output result is obtained through soft threshold adaptive noise reduction and residual connection, as shown below:

[0119]

[0120]

[0121] in, It is a scaling parameter. The output of multimodal, multi-scale, depthwise separable convolution. It is a threshold. The output of the MRSBU unit;

[0122] In this embodiment, the second MFF submodule, the third MFF submodule, and the fourth MFF submodule all include a first input layer, an LCAT layer, a second input layer, a convolutional pooling layer (Conv, Avg pool), a normalized convolutional layer (LN, Conv), a WFF layer, and an IRMLP layer.

[0123] The processing includes:

[0124] The input of the first input layer is processed by the LCAT (Lightweight Cross-modal Attention Transformer) layer to perform multimodal attention on the extracted results, extracting complementary information between multimodal signals and outputting the first branch result. The input of the second input layer is first processed by a convolutional pooling layer to obtain the convolutional pooling result, and then by a normalized convolutional layer to obtain the second branch result. The first branch result and the second branch result are then processed by WFF (Weighted Feature Fusion) and IRMLP (Inverted Residual Multi-Layer Perceptron) respectively, and the residuals are connected to the convolutional pooling result to output the fusion result of this MFF submodule.

[0125] For the second, third, and fourth MFF submodules, the extraction result output by the corresponding MDRSN submodule is used as the input of the first input layer; the fusion result output by the previous MFF submodule is used as the input of the second input layer.

[0126] Specifically, such as Figure 4 As shown, the features extracted from the 4-layer MDRSN sub-modules are sequentially input into the 4-layer MFF sub-modules from shallow to deep layers for layer-by-layer fusion. The MFF sub-modules are as follows: Figure 3 As shown;

[0127] After the features extracted by the current layer MDRSN submodule are input into the MFF submodule, the lightweight cross-modal attention module LCAT (LCAT layer) is first used to perform multimodal attention on the extracted features to extract complementary information between multimodal signals. Then, the extracted information is fused with the information extracted by the previous layer in the same way using WFF weighted feature fusion. The inverse residual MLP is used to process the preliminary fusion result of this layer and output it as the input for the next layer fusion.

[0128] Since the first-level MFF submodule does not have the output of the previous-level MFF submodule as input, the result of the LCAT layer processing of the data passed from the MDRSN submodule is used instead of the output of the MFF submodule.

[0129] Specifically, the LCAT layer (Lightweight Cross-Modal Attention Module, LCAT) is composed of cross-modal self-attention (SA) modules connected in series along the channel direction, such as... Figure 5 As shown; unlike traditional multi-head attention mechanisms, to adapt to tasks involving mutual attention between multiple modalities of a one-dimensional signal, the LCAT layer makes the following settings: Assume the signal has G modalities, This is a feature matrix of a subject that requires cross-modal attention. Let G be the reference feature matrix containing all G modes, where include ; for the feature matrix that is currently the main subject After being convolved with a scale of 1, the matrix is ​​converted into K and V matrices, which are then processed using... and It means that, among them ; After being convolved with a scale of 1, it is converted into G Q matrices, and then... It means that, among them To reduce the number of model parameters, this invention also simplifies the Q-matrix by reducing the number of channels in all Q-matrixes to 1. The specific process is as follows:

[0130]

[0131]

[0132]

[0133]

[0134] in, , and These represent the weights of the 1×1 convolution kernels used to generate the query (Q), key (K), and value (V) matrices, respectively. This indicates the Softmax normalization operation. This represents the current subject mode vector. Represented as Provide a reference mode vector. and This indicates the current main feature matrix The matrix obtained by convolution, Indicates from the reference matrix The matrix obtained by convolution, This refers to the size of the time dimension; C represents the LCAT layer channel dimension.

[0135] Each mode All of them will be processed by scale dot-product attention on the G Q matrices, so that the current modality It has the ability to perform cross-modal attention to all G modalities; then, the cross-modal attention is used to obtain feature vectors. Perform convolution with a scale of 1 kernel to generate The process is as follows:

[0136]

[0137]

[0138]

[0139] in, It is a ReLU activation function. is the weight of the convolution kernel with size 1; t represents the time parameter; m and n' represent the loop parameters in the channel dimension C of the LCAT layer, respectively;

[0140] G modal vectors After cross-modal scaled dot product attention processing, G are generated. Vectors, and G of them The vectors are concatenated along the channel dimension to obtain ;

[0141] Next The convolution reduces the vector to the current main modality vector. The same dimensions, yield and will Perform residual processing to obtain the current main mode vector. After cross-modal attention processing The formula is as follows:

[0142]

[0143]

[0144]

[0145]

[0146]

[0147] in, and Refers to learnable weights;

[0148] Each mode Perform the same cross-modal attention operation as above on all of them to obtain G values. Concatenate these G vectors along the channel dimension to obtain the vectors that have undergone LCAT cross-modal attention. The process is as follows:

[0149]

[0150] In this embodiment, for the second, third, and fourth MFF submodules, the second branch of the MFF submodule is processed as follows: the output of the upper-level MFF submodule... When entering the MFF submodule of this layer, the results after processing by the group convolutional layer and the average pooling layer are... As a residual term following the MFF submodule of this layer; The result after processing by the normalization layer and the ReLU activation function layer As the second branch result of the MFF submodule, it serves as one of the input items before WFF fusion; specifically, it is represented as follows:

[0151]

[0152]

[0153]

[0154] in, This is the output of the previous MFF submodule. The weights of the convolution kernel are of scale 1. for The channel dimension, This refers to the dimension output by the MFF submodule at this layer. Refers to the result after group convolution , Indicates the average pooling layer. This indicates the ReLU activation function layer. This indicates the BatchNormalization layer;

[0155] The output of the previous MFF submodule, after processing, is combined with the current layer's LCAT cross-modal focus module. Features can be fused using weighted feature fusion (WFF). The WFF processing procedure is as follows: Figure 6 As shown, the calculation process is as follows:

[0156]

[0157]

[0158]

[0159] in, Represents the weights of the convolution. and They are respectively represented as and Weighted weights, express The result after processing This indicates the output of this level of MDRSN.

[0160] Specifically, the final step in MFF layer fusion is the processing of the inverse residual multilayer perceptron (IRMLP) and residual connections, as follows:

[0161]

[0162]

[0163] in, This indicates the output of the MFF, and '&' indicates the '&'th MFF submodule. and These represent convolution weights of size 1 and 3, respectively. represents the second branch result after transformation of the input of the second input layer of MFF, and x represents the input of the IRMLP layer;

[0164] Example 2

[0165] The present invention will be further described below with reference to specific embodiments.

[0166] The fault identification model was trained using the bearing dataset from Paderborn University (PU), Germany. The process is as follows:

[0167] Step S1: Select the bearing dataset from Paderborn University (PU), Germany to test the performance of the model, as shown in Table 1;

[0168] The samples were real faults generated by accelerated life testing in the PU dataset. Vibration and current signals were selected and divided into 14 categories according to the damage type. The experiment was conducted under the conditions of N = 1500 rpm, M = 0.7 Nm, and F = 1000 N.

[0169] Table 1 PU Dataset

[0170]

[0171] In Table 1, F and P represent fatigue damage and plastic deformation, respectively; OR and IR represent outer ring failure and inner ring failure, respectively; S, R, and M represent single damage, repeated damage, and multiple damage, respectively; L1, L2, and L3 are used to measure damage level; S and D represent single-point damage and distributed damage, respectively.

[0172] Step S2: Normalize the one-dimensional modal data to [-1, 1], add Gaussian white noise to the original signal, and finally add labels corresponding to the device status information.

[0173] The original data is divided into training, validation, and test sets, with no overlap between them. The initial sample ratio is 2:1:1. To simulate a low-sample environment, the training set is reduced to 20% of its original size, while the validation and test sets remain unchanged. The final ratio of training, validation, and test sets is 2:5:5. Before inputting the data into the model, the data from different modalities are organized into a list.

[0174] Step S3: Input data from different modalities into the feature extraction module and the multimodal feature fusion module for feature extraction and layer-by-layer fusion. For specific steps, please refer to Example 1.

[0175] Step S4: Output the prediction results by performing operations such as global average pooling, normalization, and full connection on the features fused layer by layer.

[0176] Step S5: The number of iterations is 50. The cross-entropy loss function is used to compare the model's prediction results with the true labels, and the cross-entropy between the predicted probability distribution and the true label distribution is calculated.

[0177] The Adam optimizer is used to train and update the optimal model parameters. The regularization coefficient is set to 0.01, and the initial learning rate is 0.01. If the validation metric does not improve within 5 iterations, the scheduler will reduce the learning rate. The lower bound of the learning rate is set to... .

[0178] Step S6: Use F1-score and false alarm rate (FPR) to evaluate the fault diagnosis performance of the proposed method, specifically defined as follows:

[0179]

[0180]

[0181]

[0182]

[0183] TP, FP, FN, and TN represent true positive, false positive, false negative, and true negative, respectively; Precision represents precision rate; and Recall represents recall rate.

[0184] Step S7: To demonstrate the superiority of the invention, the fault identification model of the present invention is compared with other mainstream models, and the average value of 5 experimental results is taken. The results are shown in Table 2.

[0185] Table 2 Diagnostic performance of different models

[0186]

[0187] SNR indicates noise level, F1 refers to F1-score, and FPR refers to false alarm rate.

[0188] Step S9: To more intuitively demonstrate the classification and recognition effect of the fault identification model of the present invention, t-SNE visualization is performed on the data with noise added at a signal-to-noise ratio of -6dB, as shown below. Figure 7 As shown;

[0189] Experimental results show that the present invention has superior diagnostic capabilities in real equipment data with severe noise pollution and a small sample size.

[0190] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A fault diagnosis method based on multimodal data fusion for workshop equipment, characterized in that, include: Acquire signal data from different modes of workshop equipment; The signal data of the different modes are preprocessed to obtain preprocessed signal data of the different modes; The preprocessing includes data normalization and data concatenation. The preprocessed signal data of different modes are input into the trained fault identification model for fault identification processing to obtain the fault diagnosis results of the workshop equipment. The fault identification model includes a feature extraction module, a multimodal feature fusion module, a global average pooling layer, a normalization layer, and a fully connected layer; The feature extraction module includes a first multi-scale deep residual shrinkage network submodule, a second multi-scale deep residual shrinkage network submodule, a third multi-scale deep residual shrinkage network submodule, and a fourth multi-scale deep residual shrinkage network submodule, all of which are sequentially connected. The multimodal feature fusion module includes a first multimodal feature fusion submodule, a second multimodal feature fusion submodule, a third multimodal feature fusion submodule, and a fourth multimodal feature fusion submodule connected in sequence. The first multi-scale depth residual shrinkage network submodule is connected to the first multimodal feature fusion submodule, the second multi-scale depth residual shrinkage network submodule is connected to the second multimodal feature fusion submodule, the third multi-scale depth residual shrinkage network submodule is connected to the third multimodal feature fusion submodule, and the fourth multi-scale depth residual shrinkage network submodule is connected to the fourth multimodal feature fusion submodule. The preprocessed signal data of different modes are input into the trained fault identification model for fault identification processing to obtain the fault diagnosis results of the workshop equipment, including: The preprocessed signal data of different modalities are input into the first multi-scale depth residual shrinkage network submodule for the first feature extraction, and the first extraction result is obtained. The first extraction result is input into the second multi-scale depth residual shrinkage network submodule for a second feature extraction to obtain the second extraction result; The second extraction result is input into the third multi-scale depth residual shrinkage network submodule for a third feature extraction, resulting in the third extraction result. The third extraction result is input into the fourth multi-scale depth residual shrinkage network submodule for the fourth feature extraction, resulting in the fourth extraction result. The first extraction result is input into a multimodal feature fusion submodule for LCAT processing to obtain the first processing result; the LCAT processing includes multimodal attention on the first extraction result to extract complementary information between multimodal signals; The second extraction result and the first processing result are input into the second multimodal feature fusion submodule to perform the first weighted feature fusion to obtain the first fusion result; The third extraction result and the first fusion result are input into the third multimodal feature fusion submodule to perform a second weighted feature fusion to obtain the second fusion result; The fourth extraction result and the second fusion result are input into the fourth multimodal feature fusion submodule to perform a third weighted feature fusion to obtain the third fusion result; The third fusion result is input into the global average pooling layer, normalization layer, and fully connected layer to output the fault diagnosis results of the workshop equipment; The MRSBU unit includes a group convolutional layer, a multi-scale depth-separable convolutional layer, and a soft-thresholding layer. The group convolutional layer reduces the channel dimension of the input features. After the channel dimension is reduced, the input features are fed into the multi-scale depth-separable convolutional layer to extract features in the receptive field of view at different scales. The extracted multi-scale features are then concatenated on the reduced channel dimension. The concatenated features are then subjected to soft-threshold adaptive denoising. The result of the adaptive denoising is then residually concatenated with the input features to output the result processed by the MRSBU unit.

2. The fault diagnosis method for workshop equipment based on multimodal data fusion according to claim 1, characterized in that, The steps for obtaining the trained fault identification model include: Acquire signal datasets of different modes under normal operating conditions and fault conditions of workshop equipment, and preprocess the signal datasets of different modes to obtain preprocessed signal datasets of different modes; The preprocessed signal datasets of different modes are divided into a sample set, a training set, and a validation set. Device status labels are applied to the preprocessed signal data of different modes to generate labels for the sample set; The data from the training set are sequentially input into the feature extraction module, multimodal feature fusion module, global average pooling layer, normalization layer, and fully connected layer of the initial fault identification model for training, to obtain the fault diagnosis results of the training set. Fault diagnosis is performed based on the training set, and the optimal model parameters are determined by minimizing the cross-entropy loss function, thereby training a fault identification model.

3. The fault diagnosis method for workshop equipment based on multimodal data fusion according to claim 1, characterized in that, The signal data of the different modes are preprocessed to obtain preprocessed signal data of the different modes, including: Normalize the signal data of different modes to obtain normalized signal data of different modes; Normalized signal data of different modes are concatenated along the channel dimension to obtain preprocessed signal data of different modes.

4. The fault diagnosis method for workshop equipment based on multimodal data fusion according to claim 1, characterized in that, The group of convolutional layers reduces the channel dimension of the input features, including: The MRSBU unit uses group convolution to process the input feature data. The data is divided into G groups according to the channel, and each group contains data of the same mode. Each modality group is processed using a convolutional kernel of size 1, and the number of channels is changed to the number of output channels. The formula is as follows: ； in, This indicates the input of the MRSBU unit. This represents the output after channel dimension reduction via group convolutional layers. and These represent the time dimensions of the input and output, respectively. This indicates the channel dimension of the MRSBU unit input. This indicates the channel dimension of the MRSBU unit output, and * indicates the convolution operation. denoted by 1, G represents the number of modes in the input features, and Concat represents the channel dimension concatenation operation; N represents the number of convolutional kernels in the group of convolutional layers; g represents the g-th mode in the mode group (1, G); a and b are loop variables in the formula.

5. The fault diagnosis method for workshop equipment based on multimodal data fusion according to claim 4, characterized in that, After the channel dimension is reduced, it enters a multi-scale depthwise separable convolution to extract features from the receptive field of view at different scales. The extracted multi-scale features are then concatenated on the reduced channel dimension, including: After reducing the channel dimension, data of the same modality are stored in the same data set. For each set, N convolutional kernels with a size of [missing value] are performed. Separable convolutions extract features from receptive fields of view at different scales, yielding... The features extracted by these N convolutional kernels are concatenated along the reduced channel dimension to obtain the output of a multi-scale depth-separable convolutional layer. The formula is as follows: ； ； in, Indicates the size is The weights of separable convolutions, Indicates the first Each mode has a weight of The output of separable convolution, This represents the output of a multi-scale depth-separable convolutional layer. This represents the size of the time dimension; n is an integer variable. ; This represents the output feature of the b-th channel after the g-th mode group undergoes group convolution dimensionality reduction.

6. The fault diagnosis method for workshop equipment based on multimodal data fusion according to claim 5, characterized in that, The second, third, and fourth multimodal feature fusion submodules each include: a first input layer, an LCAT layer, a first input layer, a convolutional pooling layer, a normalized convolutional layer, a WFF layer, and an IRMLP layer; the processing includes: The input to the first input layer is processed by the LCAT layer to perform multimodal attention on the extracted results, extracting complementary information between multimodal signals and outputting the first branch result. The input to the second input layer is first processed by a convolutional pooling layer to obtain the convolutional pooling result, and then by a normalized convolutional layer to obtain the second branch result. The first branch result and the second branch result are simultaneously processed by WFF weighted feature fusion and IRMLP inverse residual processing, and the residuals are concatenated with the convolutional pooling results to output the fusion result of this multimodal feature fusion submodule. The extraction results of the second, third, and fourth multimodal feature fusion submodules, which are correspondingly connected to the multi-scale deep residual shrinkage network submodule, are used as the input to the first input layer. The output result of the previous multimodal feature fusion submodule is used as the input to the second input layer.

7. The fault diagnosis method for workshop equipment based on multimodal data fusion according to claim 6, characterized in that, The LCAT layer is a lightweight cross-modal attention module, which is composed of cross-modal self-attention modules connected in series along the channel direction; The LCAT layer is configured as follows: Suppose that the multimodal data input to the LCAT layer has a total of G modes. and All are one of the G modes, that is This represents the current subject mode vector. Represented as Provide a reference mode vector; After being convolved with a scale of 1, the matrix is ​​converted into K and V matrices, which are then processed using... and Indicate; will After being convolved with a scale of 1, it is converted into G Q matrices, and then... The formulas are expressed as follows: ； ； ； ； in , and These represent the weights of the 1×1 convolution kernels used to generate the query, key, and value matrices, respectively; that is, the weights of the 1×1 convolution kernels used to generate the Q, K, and V matrices. This indicates the Softmax normalization operation. This represents the current subject mode vector. Represented as Provide a reference mode vector. and Indicates the current main feature matrix The matrix obtained by convolution, Indicates from the reference matrix The matrix obtained by convolution, This refers to the size of the time dimension; C represents the LCAT layer channel dimension. G modal vectors After cross-modal scaled dot product attention processing, G are generated. Vectors, and G of them The vectors are concatenated along the channel dimension to obtain ; In cross-modal scaled dot product attention processing, generate The formula is as follows: ； ； ； in, It is a ReLU activation function. is the weight of the convolution kernel with size 1; t represents the time parameter; m and n' represent the loop parameters in the channel dimension C of the LCAT layer, respectively; Will The convolution reduces the vector to the current main modality vector. The same dimensions, yield and will Perform residual processing to obtain the current main mode vector. After cross-modal attention processing The formula is as follows: ； ； ； ； ； in, and Refers to learnable weights; Based on the cross-modal attention processing described above, G values ​​are obtained. and these G By concatenating along the channel dimension, we obtain the output after passing through the LCAT layer. The formula is as follows: 。

Images