A method and device for fault diagnosis of electromechanical equipment using multimodal data fusion

Abstract

This invention discloses a method and apparatus for fault diagnosis of electromechanical equipment using multimodal data fusion. The method includes: acquiring a sound signal and an infrared image of the electromechanical equipment to be diagnosed, both with timestamps; generating aligned single-modal representations of the sound signal and the infrared image; inputting the aligned single-modal representations of the sound signal and the infrared image into a trained fusion network model to generate fusion features; and inputting the fusion features into a multilayer perceptron to generate a fault diagnosis result for the electromechanical equipment to be diagnosed. The method of this invention utilizes the correlation information between different modal data, treating the single-modal representation corresponding to the actual fusion result as a positive sample and other representations as negative samples, thereby maximizing the mutual information between the fusion result and the single-modal representation, which helps improve the accuracy and reliability of fault diagnosis.

Description

Technical Field

[0001] This invention relates to the field of intelligent fault diagnosis technology, specifically to a method and apparatus for multimodal data fusion fault diagnosis of electromechanical equipment. Background Technology

[0002] Electromechanical equipment plays a crucial role in various industries, and the occurrence and timely diagnosis of faults are essential for the reliability and safety of such equipment. Traditional fault diagnosis methods are mainly based on single-modal data, such as vibration, temperature, and sound, but these methods have some limitations. For example, single-modal data may not provide enough information to accurately determine the type and severity of the fault, and in complex electromechanical systems, data from different modes may be interrelated, lacking a comprehensive analysis method.

[0003] Compared to traditional fault diagnosis methods based on single-modal data, multimodal data fusion offers several advantages: First, it provides richer information, enabling more accurate determination of fault type and severity. Second, it leverages the interrelationships between different modalities to improve fault diagnosis accuracy. Furthermore, it enhances system robustness, providing better resistance to noise and interference. Therefore, employing multimodal data fusion for fault diagnosis can provide more accurate and reliable results, contributing to improved reliability, safety, and operational efficiency of electromechanical equipment. Summary of the Invention

[0004] In view of this, the present invention provides a multimodal data fusion fault diagnosis method and device for electromechanical equipment, which can use the collected sound and infrared image signals of electromechanical equipment to realize the fusion of different modal data information and ultimately evaluate the status of electromechanical equipment, realize the fault diagnosis function, thereby ensuring the reliable operation of electromechanical equipment, enhancing the safety of electromechanical equipment, and solving the technical problem that single modal data in the prior art cannot provide enough information to accurately determine the fault type and severity.

[0005] To solve the above-mentioned technical problems, the present invention is implemented as follows.

[0006] A multimodal data fusion fault diagnosis method for electromechanical equipment includes:

[0007] Step S1: Acquire the sound signal and infrared image of the electromechanical equipment to be diagnosed with timestamps, align the sound signal and the infrared image based on the timestamps, and slice the aligned sound signal and infrared image according to the time dimension.

[0008] Step S2: For each audio signal slice data: Extract the frequency domain features of the audio signal slice data using the Fourier transform method, and reduce the dimensionality of the frequency domain features using the principal component analysis method to obtain the numerical sequence vector X of the audio signal slice data. a Using an LSTM neural network to convert X a Processed into unit length representation H a For each infrared image slice data: extract the infrared image features of the infrared image slice data, and process the infrared image features into a numerical sequence vector X. f Using an LSTM neural network to convert X f Processed into unit length representation H f ; by all H a The aligned sound signal is represented in a single-mode format by all H... f Forming a single-mode representation of the aligned infrared image;

[0009] Step S3: Input the single-modal representation of the aligned sound signal and the single-modal representation of the aligned infrared image into the trained fusion network model to generate fusion features;

[0010] Step S4: Input the fused features into the multilayer perceptron to generate the fault diagnosis result of the electromechanical equipment to be diagnosed.

[0011] Preferably, in step S2, the frequency domain features are projected into a low-dimensional space using principal component analysis to obtain a numerical sequence vector X of the sound signal. a Infrared image features are extracted using multiple sequentially connected depthwise separable convolutional modules. Pooling layers then process the extracted infrared image features into one-dimensional infrared image features, which are then converted into a numerical sequence vector X. f ;

[0012] The depthwise separable convolutional module includes a depthwise convolutional layer and a pointwise convolutional layer. The infrared image slice data is input into the depthwise convolutional layer; the infrared image slice data is an RGB image with three input channels. The depthwise convolutional layer has three independent convolutional kernels, each kernel processing one channel of the infrared image slice data. That is, each input channel is independently convolved using the three independent kernels to obtain three feature maps as the output of the depthwise convolutional layer. The pointwise convolutional layer receives the three feature maps and uses a 1×1 kernel to perform convolution operations on each feature map. The convolution results of the feature maps are then concatenated as the output of the depthwise separable convolutional module, i.e., the infrared image features extracted by the depthwise separable convolutional module.

[0013] The LSTM neural network processes the numerical sequence vector X of the sound signal. a sum of numerical sequence vectors X f The numerical sequence vector X of the sound signal a sum of numerical sequence vectors X f Using the input sequences as inputs, the internal state of the LSTM neural network is updated by traversing each time step of the input sequences, and a hidden state is generated as the encoding result at the last time step, that is, a unit-length representation H is produced. a and H f .

[0014] Preferably, in step S3, the fusion network model includes multiple sequentially connected linear sub-modules, with the last linear sub-module connected to a linear layer; the linear sub-module includes sequentially connected linear layers and nonlinear activation layers;

[0015] The fusion network model consists of multiple stacked linear layers and activation functions; the fusion network model converts the single-modal representation of the aligned sound signal and the single-modal representation of the aligned infrared image into fusion features;

[0016] The training method for the fusion network model is as follows:

[0017] The fusion network model generates fusion features based on training data and inputs these features into a contrastive prediction coding module. This module includes an inverse prediction network submodule and a noise contrast estimation submodule. The inverse prediction network submodule receives the fusion features and is a linear layer used to invert the fusion features into two single-modal features: an audio single-modal feature and an image single-modal feature. The audio single-modal feature includes a predicted value reflecting the correlation between the audio single-modal feature and the fusion feature, and the image single-modal feature also includes a predicted value reflecting the correlation between the image single-modal feature and the fusion feature. The noise contrast estimation submodule receives the single-modal representation of the audio signal from which the fusion features are generated and the single-modal representation of the infrared image. The system calculates the predicted values ​​of each single-modal feature and the loss function value corresponding to each single-modal feature. The loss function value is used to represent the correlation between the single-modal feature and the fused feature. Based on the correlation represented by the loss function value, it determines whether the single-modal representation corresponding to the single-modal feature matches the fused feature. Single-modal representations that match the fused feature are used as positive samples to generate positive sample labels, and single-modal representations that do not match the fused feature are used as negative samples to generate negative sample labels. Based on all generated labels, the loss generated during the fusion of the single-modal representation of the sound signal and the single-modal representation of the infrared image is determined. The fusion network model is optimized to maximize the correlation between the fused feature and the positive sample and minimize the correlation between the fused feature and the negative sample, and the parameters of the fusion network model are adjusted accordingly.

[0018] Preferably, the loss function value is calculated using a loss function, which is:

[0019]

[0020] Where Z represents the fusion feature, H m For single-mode representation, E H This represents the expectation of the same batch of training samples, s(Z,h) m ) = exp(cos < G Φ (Z),h m >) is the scoring function, where G Φ G represents a neural network with parameter Φ. Φ (Z) represents a neural network that has been fed with fused features Z. and These are positive and negative samples, respectively.

[0021] Preferably, the fusion network model and the fault diagnosis task are trained together based on training data;

[0022] The total loss of the overall training is a weighted combination of cross-entropy loss and the loss generated during the fusion process:

[0023]

[0024] Where α is a hyperparameter, This refers to the loss that occurs during the fusion of the single-modal representation of the sound signal and the single-modal representation of the infrared image;

[0025] The formula for calculating the cross-entropy loss function is as follows:

[0026]

[0027] Where k represents the dimension of the category label, y i It is the i-th element of the real label vector, p i It is the i-th element of the prediction probability vector.

[0028] The present invention provides a multimodal data fusion fault diagnosis device for electromechanical equipment, comprising:

[0029] Preprocessing module: configured to acquire sound signals and infrared images of the electromechanical equipment to be diagnosed with timestamps, align the sound signals and infrared images based on the timestamps, and slice the aligned sound signals and infrared images according to the time dimension;

[0030] Feature extraction module: Configured to extract frequency domain features of each audio signal slice data using Fourier transform, and then reduce the dimensionality of the frequency domain features using principal component analysis to obtain the numerical sequence vector X of the audio signal slice data. a Using an LSTM neural network to convert X a Processed into unit length representation H a For each infrared image slice data: extract the infrared image features of the infrared image slice data, and process the infrared image features into a numerical sequence vector X. f Using an LSTM neural network to convert X f Processed into unit length representation H f ; by all H a The aligned sound signal is represented in a single-mode format by all H... f Forming a single-mode representation of the aligned infrared image;

[0031] Fusion module: configured to input the single-modal representation of the aligned audio signal and the single-modal representation of the aligned infrared image into the trained fusion network model to generate fusion features;

[0032] Diagnostic module: configured to input the fused features into a multilayer perceptron to generate fault diagnosis results for the electromechanical equipment to be diagnosed.

[0033] The present invention provides a computer-readable storage medium storing a plurality of instructions; the plurality of instructions are used by a processor to load and execute the method as described above.

[0034] The present invention provides an electronic device, characterized in that the electronic device comprises:

[0035] A processor is used to execute multiple instructions;

[0036] Memory, used to store multiple instructions;

[0037] The plurality of instructions are to be stored in the memory and loaded and executed by the processor as described above.

[0038] Beneficial effects:

[0039] (1) This invention acquires sound and infrared image signals from electromechanical equipment, extracts the frequency domain features of the sound signals using Fourier transform and principal component analysis, and extracts the spatial features of the infrared images using depthwise separable convolutional layers. Then, the encoded results of the two types of data are fused through a fusion network with stacked linear activation layers to obtain a fused result. This fusion can provide richer information, and by utilizing the correlation information between different modal data, it helps to improve the accuracy and reliability of fault diagnosis.

[0040] (2) During model training, this invention maps the fusion result to the predicted value of the single-modal representation by comparing the prediction encoding module. By treating the actual single-modal representation corresponding to the fusion result as a positive sample and other representations in the same batch as negative samples, a scoring function is used to maximize the ratio between the scores of positive sample pairs and the scores of negative sample pairs, thereby maximizing the mutual information between the fusion result and the single-modal representation. This makes the correlation between the fusion result and the single-modal representation stronger and improves the accuracy of fault diagnosis.

[0041] (3) The present invention can provide more accurate and reliable fault diagnosis results, thereby improving the reliability, safety and operating efficiency of electromechanical equipment. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the multimodal data fusion fault diagnosis method for electromechanical equipment of the present invention;

[0043] Figure 2 This is a schematic diagram of the architecture of the multimodal data fusion fault diagnosis method for electromechanical equipment of the present invention;

[0044] Figure 3This is a schematic diagram of infrared image feature extraction according to the present invention;

[0045] Figure 4 This is a schematic diagram illustrating the multimodal data information fusion of the present invention;

[0046] Figure 5 This is a schematic diagram of the training phase of the present invention;

[0047] Figure 6 This is a structural block diagram of the multimodal data fusion fault diagnosis device for electromechanical equipment according to the present invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] This invention proposes a multimodal data fusion fault diagnosis method for electromechanical equipment.

[0050] like Figure 1 As shown, the multimodal data fusion fault diagnosis method for electromechanical equipment provided by the present invention includes:

[0051] Step S1: Acquire the sound signal and infrared image of the electromechanical equipment to be diagnosed with timestamps, align the sound signal and the infrared image based on the timestamps, and slice the aligned sound signal and infrared image according to the time dimension.

[0052] Step S2: For each audio signal slice data: Extract the frequency domain features of the audio signal slice data using the Fourier transform method, and reduce the dimensionality of the frequency domain features using the principal component analysis method to obtain the numerical sequence vector X of the audio signal slice data. a Using an LSTM neural network to convert X a Processed into unit length representation H a For each infrared image slice data: extract the infrared image features of the infrared image slice data, and process the infrared image features into a numerical sequence vector X. f Using an LSTM neural network to convert X f Processed into unit length representation H f ; by all H a The aligned sound signal is represented in a single-mode format by all H... f Forming a single-mode representation of the aligned infrared image;

[0053] Step S3: Input the single-modal representation of the aligned sound signal and the single-modal representation of the aligned infrared image into the trained fusion network model to generate fusion features;

[0054] Step S4: Input the fused features into the multilayer perceptron to generate the fault diagnosis result of the electromechanical equipment to be diagnosed.

[0055] Step S1 includes: configuring a microphone for acquiring sound signals and an infrared camera for acquiring infrared images for the electromechanical equipment; assigning a timestamp to each frame of the sound signal acquired by the microphone at each sampling point and the infrared image acquired by the infrared camera, the timestamp being used to indicate the time when the corresponding signal was acquired; cropping the infrared image into an RGB image with a resolution of 224×224 pixels; setting a time threshold; for each frame of the cropped infrared image: obtaining its corresponding timestamp, and subtracting the timestamp from all the timestamps of the sound signal; using sound signals whose absolute value of the timestamp difference is less than or equal to the time threshold as sound signals aligned with the infrared image frame; taking an aligned infrared image frame and its aligned sound signal as a matching data pair; forming an aligned sound signal and infrared image from all matching data pairs; and slicing the aligned sound signal and infrared image according to the time dimension.

[0056] In this invention, timestamped audio signals x from electromechanical equipment are collected using a microphone and an infrared sensor, respectively. t,a and infrared images with timestamps x t,f The audio signal is a one-dimensional time-series data sequence, and the infrared image is cropped into a 224×224 pixel resolution RGB image. The timestamp can be absolute time, such as a timestamp in milliseconds. By determining the optimal match, microphone sampling points and infrared camera frames can be aligned to establish a temporal correspondence between them. This allows the audio data and infrared image data to be correlated for subsequent data processing and analysis. The aligned audio signal and infrared image are then sliced ​​along the time dimension.

[0057] In step S2, the frequency domain features are projected into a low-dimensional space using principal component analysis to obtain the numerical sequence vector X of the sound signal. a Infrared image features are extracted using multiple sequentially connected depthwise separable convolutional modules. Pooling layers then process the extracted infrared image features into one-dimensional infrared image features, which are then converted into a numerical sequence vector X. f .

[0058] The depthwise separable convolutional module includes a depthwise convolutional layer and a pointwise convolutional layer. The infrared image slice data, which is an RGB image with three input channels, is input to the depthwise convolutional layer. The depthwise convolutional layer has three independent convolutional kernels, each processing one channel of the infrared image slice data. That is, each input channel is independently convolved using the three independent kernels to obtain three feature maps as the output of the depthwise convolutional layer. The pointwise convolutional layer receives the three feature maps and performs convolution operations on each feature map using a 1×1 kernel. The convolution results of the feature maps are then concatenated as the output of the depthwise separable convolutional module, i.e., the infrared image features extracted by the depthwise separable convolutional module.

[0059] Furthermore, the LSTM neural network processes the numerical sequence vector X of the sound signal respectively. a sum of numerical sequence vectors X f The numerical sequence vector X of the sound signal a sum of numerical sequence vectors X f Using the input sequences as inputs, the internal state of the LSTM neural network is updated by traversing each time step of the input sequences, and a hidden state is generated as the encoding result at the last time step, that is, a unit-length representation H is produced. a and H f .

[0060] In this invention, depthwise separable convolutional modules can be stacked multiple times depending on the specific task objective, typically using 3 to 13 depthwise separable convolutional modules; global pooling layers are used to convert the extracted infrared image features into one dimension.

[0061] like Figure 2 As shown, for audio signals, Fast Fourier Transform (FFT) and Principal Component Analysis (PCA) are used to extract features. Fast Fourier Transform is an algorithm used to convert time-domain signals into frequency-domain representations. Its main advantage lies in its high computational speed, enabling rapid analysis and extraction of frequency-domain features. Principal Component Analysis is a data dimensionality reduction technique used to extract the main features from the data. PCA helps reduce computational complexity and extract more representative audio features.

[0062] like Figure 3As shown, for infrared images, the images are cropped to a fixed size, such as 224×224 pixels. This size is typically used as input for deep learning models. In infrared image processing, a deep convolutional module is used to extract spatial features from the infrared image. The cropped infrared image or the output of the previous deep convolutional module is used as input to the deep convolutional module. In the deep convolutional module, each channel of the input is subjected to an independent convolution operation. For RGB images, there are 3 input channels. Each channel is convolved using multiple convolutional kernels to obtain multiple feature maps. The output of the deep convolutional module is a stack of multiple feature maps, with each feature map corresponding to one convolutional kernel. In the deep convolutional module, the feature map of each channel is convolved independently to obtain a new set of feature maps. In the pointwise convolutional layer, a 1×1 convolutional kernel is used to convolve the feature map of each channel to obtain the final feature representation. Stacking multiple deep separable convolutional layers gradually extracts higher-level feature representations. Finally, pooling layers are used to spatially downsample the stacked feature maps to obtain a one-dimensional feature representation.

[0063] Once the audio features and temperature distribution features (infrared image features) are obtained, an LSTM neural network can be used to unify the dimensionality and convert them into unit-length representations. LSTM is a variant of a recurrent neural network (RNN) that can effectively process sequential data and capture long-term dependencies within the sequence. First, the audio features and temperature distribution features are input into their respective LSTM neural networks. The LSTM neural network converts the feature sequence at each time step into a hidden layer representation, capturing the contextual information in the sequence. Then, the hidden layer representations obtained through the LSTM neural network are further converted into unit-length representations, compressing the information of the entire sequence into a fixed-length vector.

[0064] In step S3, the fusion network model includes multiple sequentially connected linear sub-modules, with the last linear sub-module connected to a linear layer; the linear sub-module includes sequentially connected linear layers and nonlinear activation layers.

[0065] The fusion network model consists of multiple stacked linear layers and activation functions. The fusion network model converts the single-modal representations of the aligned audio signal and the aligned infrared image into fused features.

[0066] The training method for the fusion network model is as follows:

[0067] The fusion network model generates fusion features based on training data and inputs these features into a contrastive prediction coding module. This module includes an inverse prediction network submodule and a noise contrast estimation submodule. The inverse prediction network submodule receives the fusion features and is a linear layer used to invert the fusion features into two single-modal features: an audio single-modal feature and an image single-modal feature. The audio single-modal feature includes a predicted value reflecting the correlation between the audio single-modal feature and the fusion feature, and the image single-modal feature also includes a predicted value reflecting the correlation between the image single-modal feature and the fusion feature. The noise contrast estimation submodule receives the single-modal representation of the audio signal from which the fusion features are generated and the single-modal representation of the infrared image. The system calculates the predicted values ​​of each single-modal feature and the loss function value corresponding to each single-modal feature. The loss function value is used to represent the correlation between the single-modal feature and the fused feature. Based on the correlation represented by the loss function value, it determines whether the single-modal representation corresponding to the single-modal feature matches the fused feature. Single-modal representations that match the fused feature are used as positive samples to generate positive sample labels, and single-modal representations that do not match the fused feature are used as negative samples to generate negative sample labels. Based on all generated labels, the loss generated during the fusion of the single-modal representation of the sound signal and the single-modal representation of the infrared image is determined. The fusion network model is optimized to maximize the correlation between the fused feature and the positive sample and minimize the correlation between the fused feature and the negative sample, and the parameters of the fusion network model are adjusted accordingly.

[0068] In this invention, during feature fusion, there are several single-modal representations in the same batch, denoted as h1, h2, ..., h n The fusion result Z1 is obtained based on h1, so (Z1, h1) is a positive sample pair, and (Z2, h2), ..., (Z n ,h n All are negative sample pairs. The optimization objective can also be to maximize the ratio between the scores of positive sample pairs and the scores of negative sample pairs, thereby maximizing the mutual information between the fusion result and the single-modal representation.

[0069] The loss function value is obtained by calculating a loss function, which is:

[0070]

[0071] Where Z represents the fusion feature, H m For single-mode representation, E H This represents the expectation of the same batch of training samples, s(Z,h) m ) = exp(cos < G Φ (Z),hm >) is the scoring function, where G Φ G represents a neural network with parameter Φ. Φ (Z) represents a neural network that has been fed with fused features Z. and These are positive and negative samples, respectively.

[0072] In this invention, the scoring function is used to measure the correlation between the predicted value and the single-modal representation. The scoring function is defined as an exponential function with the natural constant e as the base and the cosine similarity between the predicted value and the true value of the single-modal representation as the exponent.

[0073] After obtaining unit-length representations of the sound signal and infrared image features respectively, the feature representations of different modalities are fused to fully utilize useful information from different modalities. To ensure that the fused result retains information from each modality and has higher information content, it is desirable to maximize the mutual information between the fused representation and the individual modal representations. Mutual information (MI) can be used to measure the correlation and dependence between two random variables. In this invention, mutual information is used to measure the correlation and information content between the fused representation and the individual modal representation. This invention employs a contrastive predictive coding method to learn feature representations in multimodal data. By requiring the fusion result to predict cross-modal representations in reverse, more modality-invariant information can be passed to the fusion result.

[0074] Further, in step S3, the fusion network model fuses the single-modal representation of the aligned audio signal and the single-modal representation of the aligned infrared image into a fusion feature. The linear layers in each linear submodule of the fusion network model perform a layer-by-layer linear transformation on the input single-modal representation. The linear layers in each linear submodule multiply the input by a weight matrix and add a bias vector. Each nonlinear activation layer is a nonlinear activation function to introduce a nonlinear transformation. The linear layer connected to the last linear submodule maps the output of the last linear submodule to the space of the fusion feature.

[0075] Optional activation functions include ReLU, Sigmoid, Tanh, etc.

[0076] In this invention, a back prediction network submodule is constructed. This submodule is needed to construct single-modal representations from the fusion results. The back prediction network submodule receives the fusion results as input and generates predictions for each single-modal representation.

[0077] Using a backpropagation network submodule and unimodal representations, prediction scores are calculated, with the scores being the values ​​of a scoring function. First, a prediction for each unimodal representation is generated through the backpropagation network submodule. Then, a score is calculated between the fused feature and the prediction for each unimodal representation, using a scoring function to measure their correlation. This invention introduces the scoring function into a Noise-Contrastive Estimation (NCE) framework, treating the unimodal representations that actually correspond to the fused feature as positive samples, indicating a match; while other unimodal representations in the same batch are treated as negative samples. For each fused feature, it is compared with other unimodal representations in the same batch. Using a noise-contrastive estimation loss function, the score between the fused feature and positive samples is calculated and compared with the score of negative samples. The optimization objective is to maximize the score between the fused result and positive samples, while minimizing the score between the fused feature and negative samples.

[0078] In step S4, the multilayer perceptron is a network model consisting of multiple fully connected layers.

[0079] The multilayer perceptron consists of multiple fully connected layers, each composed of multiple neurons. The fusion result of the input is propagated forward through these fully connected layers, with the output of each layer serving as the input to the next layer. An activation function is applied to the output of each fully connected layer to introduce non-linear features. Optional activation functions include ReLU, Sigmoid, and Tanh. During training, the model parameters are optimized by minimizing cross-entropy.

[0080] In this invention, a multilayer perceptron model is used, consisting of multiple fully connected layers. Each fully connected layer comprises a set of neurons, with each neuron connected to all neurons in the previous layer. By stacking multiple fully connected layers, a deep neural network can be constructed. The fused result is passed as input features to the input layer of the MLP model. The dimensionality of the fused result should match the number of neurons in the input layer. After the last fully connected layer, an output layer is added to predict the fault category. The number of neurons in the output layer should match the number of fault categories, and the Softmax function is used as the activation function. The cross-entropy loss function is chosen as the objective function of the multilayer perceptron. The cross-entropy loss function measures the difference between the model's output and the true label.

[0081] In this invention, the fusion network model and the fault diagnosis task are trained together based on training data.

[0082] The total loss of the overall training is a weighted combination of cross-entropy loss and loss generated during the fusion process. Cross-entropy loss measures the accuracy of the diagnostic results, while loss generated during the fusion process measures the accuracy of the fusion of features.

[0083] The formula for calculating the cross-entropy loss function is as follows:

[0084]

[0085] Where k represents the dimension of the category label, y i It is the i-th element of the real label vector, p i It is the i-th element of the prediction probability vector.

[0086]

[0087] Where α is a hyperparameter, This represents the loss incurred during the fusion of the single-modal representation of the sound signal and the single-modal representation of the infrared image. α controls the degree of influence on maximizing mutual information.

[0088] In this invention, during the overall training phase, a total loss function is calculated, including the classification task loss obtained based on the model's output and the true labels, and the loss generated during the information fusion phase. Gradient information is propagated back to each parameter of the model, and the gradient of each parameter with respect to the loss function is calculated. Then, the optimizer updates the model parameters based on the gradients and returns the training results for that epoch.

[0089] This invention also provides a multimodal data fusion fault diagnosis device for electromechanical equipment, such as... Figure 6 As shown, the device includes:

[0090] Preprocessing module: configured to acquire sound signals and infrared images of the electromechanical equipment to be diagnosed with timestamps, align the sound signals and infrared images based on the timestamps, and slice the aligned sound signals and infrared images according to the time dimension;

[0091] Feature extraction module: Configured to extract frequency domain features of each audio signal slice data using Fourier transform, and then reduce the dimensionality of the frequency domain features using principal component analysis to obtain the numerical sequence vector X of the audio signal slice data. a Using an LSTM neural network to convert X a Processed into unit length representation H a For each infrared image slice data: extract the infrared image features of the infrared image slice data, and process the infrared image features into a numerical sequence vector X. f Using an LSTM neural network to convert X f Processed into unit length representation H f; by all H a The aligned sound signal is represented in a single-mode format by all H... f Forming a single-mode representation of the aligned infrared image;

[0092] Fusion module: configured to input the single-modal representation of the aligned audio signal and the single-modal representation of the aligned infrared image into the trained fusion network model to generate fusion features;

[0093] Diagnostic module: configured to input the fused features into a multilayer perceptron to generate fault diagnosis results for the electromechanical equipment to be diagnosed.

[0094] The parts of this invention not described in detail are well-known to those skilled in the art.

[0095] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A method for fault diagnosis of electromechanical equipment using multimodal data fusion, characterized in that, include: Step S1: Acquire the sound signal and infrared image of the electromechanical equipment to be diagnosed with timestamps, align the sound signal and the infrared image based on the timestamps, and slice the aligned sound signal and infrared image according to the time dimension. Step S2: For each audio signal slice data: use the Fourier transform method to extract the frequency domain features of the audio signal slice data, and use principal component analysis to reduce the dimensionality of the frequency domain features to obtain the numerical sequence vector of the audio signal slice data. Using LSTM neural networks to Process into unit length representation ; For each infrared image slice: extract the infrared image features from the infrared image slice data, and process the infrared image features into a numerical sequence vector. Using LSTM neural networks to Process into unit length representation ; by all The aligned audio signal is represented in a single-mode format by all Forming a single-mode representation of the aligned infrared image; Infrared image features are extracted using multiple sequentially connected depthwise separable convolutional modules. Pooling layers then process the extracted infrared image features into one-dimensional infrared image features, which are subsequently converted into a numerical sequence vector. ; The depthwise separable convolutional module includes a depthwise convolutional layer and a pointwise convolutional layer. The infrared image slice data is input into the depthwise convolutional layer; the infrared image slice data is an RGB image with three input channels. The depthwise convolutional layer has three independent convolutional kernels, each kernel processing one channel of the infrared image slice data. That is, each input channel is independently convolved using the three independent kernels to obtain three feature maps as the output of the depthwise convolutional layer. The pointwise convolutional layer receives the three feature maps and uses a 1×1 kernel to perform convolution operations on each feature map. The convolution results of the feature maps are then concatenated as the output of the depthwise separable convolutional module, i.e., the infrared image features extracted by the depthwise separable convolutional module. Step S3: Input the single-modal representation of the aligned sound signal and the single-modal representation of the aligned infrared image into the trained fusion network model to generate fusion features; Step S4: Input the fused features into the multilayer perceptron to generate the fault diagnosis result of the electromechanical equipment to be diagnosed; In step S3, the fusion network model includes multiple sequentially connected linear sub-modules, with the last linear sub-module connected to a linear layer; the linear sub-module includes sequentially connected linear layers and nonlinear activation layers. The fusion network model consists of multiple stacked linear layers and activation functions; the fusion network model converts the single-modal representation of the aligned sound signal and the single-modal representation of the aligned infrared image into fusion features; The training method for the fusion network model is as follows: The fusion network model generates fusion features based on training data and inputs these fusion features into a contrastive prediction coding module. This module includes an inverse prediction network submodule and a noise contrast estimation submodule. The inverse prediction network submodule receives the fusion features and is a linear layer used to invert the fusion features into two single-modal features: an audio single-modal feature and an image single-modal feature. The audio single-modal feature includes a predicted value reflecting the correlation between the audio single-modal feature and the fusion feature. The image single-modal feature also includes a predicted value reflecting the correlation between the image single-modal feature and the fusion feature. The noise contrast estimation submodule receives the single-modal representations of the audio signal and the infrared image, as well as the predicted values ​​of each single-modal feature, to generate fusion features. It calculates the loss function value corresponding to each single-modal feature, which represents the correlation between the single-modal feature and the fusion feature. Based on the correlation represented by the loss function value, it determines whether the single-modal representation corresponding to the single-modal feature matches the fusion feature. Single-modal representations matching the fusion feature are used as positive samples, generating positive sample labels; single-modal representations not matching the fusion feature are used as negative samples, generating negative sample labels. Based on all generated labels, it determines the loss generated during the fusion of the single-modal representations of the audio signal and the infrared image. The fusion network model optimizes by maximizing the correlation between the fusion feature and positive samples and minimizing the correlation between the fusion feature and negative samples, adjusting the parameters of the fusion network model accordingly.

2. The method as described in claim 1, characterized in that, In step S2, the frequency domain features are projected into a low-dimensional space using principal component analysis to obtain a numerical sequence vector of the sound signal. The LSTM neural network processes numerical sequence vectors of sound signals. and numerical sequence vector That is, the numerical sequence vector of the sound signal and numerical sequence vector Using the input sequences as inputs, the internal state of the LSTM neural network is updated by traversing each time step of the input sequences, and a hidden state is generated as the encoding result at the last time step, i.e., a unit-length representation is produced. and .

3. The method as described in claim 1, characterized in that, The loss function value is obtained by calculating a loss function, which is: in, As a feature of fusion, For single-modal representation, This indicates the expectation of the same batch of training samples. Let be the scoring function, where The parameter is neural networks, This indicates that fused features were input. neural networks, and These are positive and negative samples, respectively.

4. The method as described in claim 3, characterized in that, The fusion network model and the fault diagnosis task are trained together based on the training data; The total loss of the overall training is a weighted combination of cross-entropy loss and the loss generated during the fusion process: in, For hyperparameters, This refers to the loss that occurs during the fusion of the single-modal representation of the sound signal and the single-modal representation of the infrared image; The formula for calculating the cross-entropy loss function is as follows: in, The dimension representing the category label. It is the first of the real label vectors One element, It is the first of the predicted probability vectors Each element.

5. A multimodal data fusion fault diagnosis device for electromechanical equipment, characterized in that, include: Preprocessing module: configured to acquire sound signals and infrared images of the electromechanical equipment to be diagnosed with timestamps, align the sound signals and infrared images based on the timestamps, and slice the aligned sound signals and infrared images according to the time dimension; Feature extraction module: Configured to extract frequency domain features of each audio signal slice data using Fourier transform, and then reduce the dimensionality of the frequency domain features using principal component analysis to obtain the numerical sequence vector of the audio signal slice data. Using LSTM neural networks to Process into unit length representation ; For each infrared image slice: extract the infrared image features from the infrared image slice data, and process the infrared image features into a numerical sequence vector. Using LSTM neural networks to Process into unit length representation ; by all The aligned audio signal is represented in a single-mode format by all Forming a single-mode representation of the aligned infrared image; Infrared image features are extracted using multiple sequentially connected depthwise separable convolutional modules. Pooling layers then process the extracted infrared image features into one-dimensional infrared image features, which are subsequently converted into a numerical sequence vector. ; The depthwise separable convolutional module includes a depthwise convolutional layer and a pointwise convolutional layer. The infrared image slice data is input into the depthwise convolutional layer; the infrared image slice data is an RGB image with three input channels. The depthwise convolutional layer has three independent convolutional kernels, each kernel processing one channel of the infrared image slice data. That is, each input channel is independently convolved using the three independent kernels to obtain three feature maps as the output of the depthwise convolutional layer. The pointwise convolutional layer receives the three feature maps and uses a 1×1 kernel to perform convolution operations on each feature map. The convolution results of the feature maps are then concatenated as the output of the depthwise separable convolutional module, i.e., the infrared image features extracted by the depthwise separable convolutional module. Fusion module: configured to input the single-modal representation of the aligned audio signal and the single-modal representation of the aligned infrared image into the trained fusion network model to generate fusion features; The fusion network model includes multiple sequentially connected linear sub-modules, with the last linear sub-module connected to a linear layer; the linear sub-module includes sequentially connected linear layers and nonlinear activation layers; The fusion network model consists of multiple stacked linear layers and activation functions; the fusion network model converts the single-modal representation of the aligned sound signal and the single-modal representation of the aligned infrared image into fusion features; The training method for the fusion network model is as follows: The fusion network model generates fusion features based on training data and inputs these fusion features into a contrastive prediction coding module. This module includes an inverse prediction network submodule and a noise contrast estimation submodule. The inverse prediction network submodule receives the fusion features and is a linear layer used to invert the fusion features into two single-modal features: an audio single-modal feature and an image single-modal feature. The audio single-modal feature includes a predicted value reflecting the correlation between the audio single-modal feature and the fusion feature. The image single-modal feature also includes a predicted value reflecting the correlation between the image single-modal feature and the fusion feature. The noise contrast estimation submodule receives the single-modal representations of the audio signal and the infrared image, as well as the predicted values ​​of each single-modal feature, to generate fusion features. It calculates the loss function value corresponding to each single-modal feature, which represents the correlation between the single-modal feature and the fusion feature. Based on the correlation represented by the loss function value, it determines whether the single-modal representation corresponding to the single-modal feature matches the fusion feature. Single-modal representations matching the fusion feature are used as positive samples, generating positive sample labels; single-modal representations not matching the fusion feature are used as negative samples, generating negative sample labels. Based on all generated labels, it determines the loss generated during the fusion of the single-modal representations of the audio signal and the infrared image. The fusion network model optimizes by maximizing the correlation between the fusion feature and positive samples and minimizing the correlation between the fusion feature and negative samples, adjusting the parameters of the fusion network model accordingly. Diagnostic module: configured to input the fused features into a multilayer perceptron to generate fault diagnosis results for the electromechanical equipment to be diagnosed.

6. A computer-readable storage medium storing a plurality of instructions; the plurality of instructions being loaded by a processor and executing the method as claimed in any one of claims 1-4.

7. An electronic device, characterized in that, The electronic device includes: A processor is used to execute multiple instructions; Memory, used to store multiple instructions; The plurality of instructions are to be stored in the memory and loaded by the processor and executed as described in any one of claims 1-4.

Images