Marine power device fault diagnosis method based on multi-modal information fusion

Abstract

The invention provides a marine power device fault diagnosis method based on multi-modal information fusion, and relates to the technical field of intelligent fault diagnosis. Comprising the steps that operation state data of equipment to be tested are collected, and the operation state data comprise vibration signals and infrared images; preprocessing the running state data to obtain preprocessed running state data; the preprocessed operation state data are input into a fault diagnosis model, a diagnosis result is obtained, and the fault diagnosis model comprises an input module, a feature extractor, a feature enhancement module and a classifier which are connected in sequence; the problems that in the prior art, multi-modal data fusion is difficult, and the fault diagnosis accuracy of the marine power device is low are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent fault diagnosis, and in particular to a marine power plant fault diagnosis method based on multimodal information fusion. Background Art

[0002] Marine power plants are widely used in some civilian and military vessels, such as passenger ships, cargo ships, oil tankers, surface ships, destroyers, and frigates. As the core equipment of a ship, the operating status of the marine power plant not only determines the performance and fuel efficiency of the ship, but is also directly related to the safety of the ship. However, due to the long-term exposure of marine power plants to complex marine environments, equipment failures are common. In particular, failures of key components can lead to reduced equipment operating efficiency and even serious safety accidents. Therefore, effective fault diagnosis of marine power plants is particularly important. Accurately and efficiently extracting the operating characteristics of marine power plants is the core link of equipment health monitoring and the basis for ensuring fault diagnosis and safe operation.

[0003] Traditional fault diagnosis methods mainly rely on single-modal data, such as vibration signals or infrared thermal imaging. Vibration signals can capture the mechanical vibration characteristics of equipment, and are particularly good at detecting structural faults (such as gear breakage, bearing wear, etc.). However, the disadvantage of using vibration signals alone is that it cannot effectively identify certain non-structural faults, such as insufficient lubrication or abnormal temperature rise. In contrast, infrared images can reflect the temperature distribution on the surface of the equipment and are good at capturing non-structural fault characteristics caused by thermal effects. However, image data is easily interfered by environmental noise, and the diagnostic robustness is poor when used alone.

[0004] Unmanned watercraft (UAVs) are waterborne platforms that operate without direct human control and rely on remote control or autonomous navigation systems. With the advancement of artificial intelligence, navigation, positioning, communications, and sensing technologies, UAVs are widely used in ocean patrol, hydrographic surveying, environmental monitoring, search and rescue, salvage, military reconnaissance, and other fields. Compared to traditional manned vessels, UAVs offer advantages such as low cost, high efficiency, and the ability to operate in hazardous waters, making them particularly suitable for deep seas, complex waters, and high-risk missions. In modern ship systems, UAVs have become a crucial component of intelligent ship systems, enhancing overall operational flexibility and safety. Their powertrain is a key mechanical system, primarily consisting of the main engine, gearbox or clutch, drive shaft, and propeller. The coordinated operation of these components, combined with the intelligent control of the power control module, forms the UAV's efficient and reliable mechanical powertrain, providing a solid foundation for autonomous navigation and mission execution.

[0005] To comprehensively characterize the operational characteristics of marine propulsion systems, multimodal information fusion has become a key technology. Multimodal data samples have the following notable characteristics: they come from a rich and diverse range of sources, including images, text, audio, and video. They can comprehensively characterize objects from different dimensions, providing richer and more complementary information. However, multimodal feature extraction faces many complex and critical challenges.

[0006] First, data from different modalities differ significantly in feature dimensions and representation. For example, image data is presented as high-dimensional pixel values, while text data is expressed in the form of discrete word vectors. This difference makes feature fusion and unified representation extremely difficult. Secondly, multimodal data is often unbalanced in quantity, scale, and quality. Data from some modalities may be missing or contain high noise, which further increases the complexity of processing. In addition, the alignment and association of multimodal data are extremely complex. Data from different modalities may be out of sync in time, may not correspond in space, may have significant semantic differences, and lack effective alignment annotation. At the same time, there are complex nonlinear relationships between modalities. Furthermore, the processing and annotation costs of multimodal data are high, the data scale is huge, and the demand for computing resources is extremely high, which to a certain extent limits the efficiency and effectiveness of multimodal feature extraction. These difficulties pose severe challenges to multimodal feature extraction and urgently require further research and breakthroughs. Summary of the Invention

[0007] In order to overcome the shortcomings of the prior art, the present invention aims to provide a method for diagnosing marine power plant faults based on multimodal information fusion. The present invention solves the problems of the prior art in which multimodal data fusion is difficult and the marine power plant fault diagnosis accuracy is low.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A method for fault diagnosis of a marine power plant based on multimodal information fusion, comprising:

[0010] Collecting operating status data of the device under test, wherein the operating status data includes: vibration signals and infrared images;

[0011] Preprocessing the operating status data to obtain preprocessed operating status data;

[0012] The pre-processed operating status data is input into a fault diagnosis model to obtain a diagnosis result, wherein the fault diagnosis model includes:

[0013] The input module, feature extractor, feature enhancement module and classifier are connected in sequence;

[0014] The input module is used to input the preprocessed operating status data, the feature extractor is used to extract features for different modalities respectively to obtain a feature set, the feature enhancement module is used to apply the attention mechanism to enhance the feature set to obtain an enhanced feature set, and the classifier is used to obtain a diagnostic result based on the enhanced feature set.

[0015] Preferably, the preprocessing of the operating status data to obtain the preprocessed operating status data includes:

[0016] performing a normalization operation on the vibration signal to obtain preprocessed vibration signal data;

[0017] Performing dimension adjustment on the infrared image to obtain image data of uniform dimension;

[0018] Normalizing the image data of unified dimension to obtain normalized data;

[0019] Random angle rotation processing is performed on the normalized data to obtain pre-processed infrared image data.

[0020] Preferably, the feature extractor comprises:

[0021] a first sub-extractor and a second sub-extractor;

[0022] The first sub-extractor is used to perform feature extraction on the preprocessed vibration signal data using a long short-term memory network to obtain a first feature subset;

[0023] The second sub-extractor is used to perform feature extraction on the preprocessed infrared image data using a convolutional neural network to obtain a second feature subset.

[0024] Preferably, the expression of the first feature subset is:

[0025] H vib =[h1,h2,...,h T ];

[0026] Among them, h t (t=1,2...T) is the hidden state vector of LSTM at time step t, and T is the total number of time steps of the vibration signal sequence.

[0027] Preferably, the expression of the second feature subset is:

[0028] H img =CNN(X img ), where X img is the preprocessed infrared image data.

[0029] Preferably, the feature enhancement module includes:

[0030] Temporal attention submodule and spatial attention submodule;

[0031] The temporal attention submodule is used to enhance the first feature subset to obtain a first enhanced subset;

[0032] The spatial attention submodule is used to enhance the second feature subset to obtain a second enhanced subset.

[0033] Preferably, the expression of the first enhancer subset is:

[0034] X T =EncoderBlock2(EncoderBlock1(H'));

[0035] Among them, H' is the sequence feature obtained after the first feature subset undergoes linear transformation and causal convolution kernel group, EncoderBlock1 and EncoderBlock2 are two identical temporal encoding blocks, which include multi-head attention mechanism, feedforward network, residual connection and normalization.

[0036] Preferably, the expression of the second enhancer subset is:

[0037]

[0038] M s To generate the attention map using the Sigmoid function, It is the image feature extracted by CNN in the second feature subset.

[0039] The present invention discloses the following technical effects:

[0040] The present invention provides a method for diagnosing faults of a marine power plant based on multimodal information fusion, comprising: collecting operating status data of a device to be tested, wherein the operating status data comprises: a vibration signal and an infrared image; preprocessing the operating status data to obtain preprocessed operating status data; inputting the preprocessed operating status data into a fault diagnosis model to obtain a diagnosis result, wherein the fault diagnosis model comprises: an input module, a feature extractor, a feature enhancement module and a classifier connected in sequence; the input module is used to input the preprocessed operating status data, the feature extractor is used to extract features for different modes respectively to obtain a feature set, the feature enhancement module is used to apply an attention mechanism to perform feature enhancement on the feature set to obtain an enhanced feature set, and the classifier is used to obtain a diagnosis result based on the enhanced feature set. This method combines long-short-term memory (LSTM) networks with convolutional neural networks (CNN) to deeply mine the temporal and spatial features of multimodal data, such as vibration signals and thermal imaging. It then uses feature fusion technology to fully extract data features, enabling efficient and accurate fault diagnosis of marine power plants. This method is suitable for fault monitoring and maintenance of complex mechanical systems, providing technical support for improving ship operational reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A flow chart of a method for fault diagnosis of a marine power plant based on multimodal information fusion provided by an embodiment of the present invention;

[0043] Figure 2 A detailed diagram of a strategy for a marine power plant fault diagnosis method based on multimodal information fusion provided by an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of a feature extraction network provided by an embodiment of the present invention;

[0045] Figure 4 Schematic diagram of the spatial attention submodule provided by an embodiment of the present invention;

[0046] Figure 5 Schematic diagram of the temporal attention submodule provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] like Figure 1 As shown, the present invention provides a method for fault diagnosis of a marine power plant based on multimodal information fusion, comprising:

[0050] Step 100: Collecting operating status data of the device under test, wherein the operating status data includes: vibration signals and infrared images;

[0051] Step 200: pre-processing the operating status data to obtain pre-processed operating status data;

[0052] Step 300: Input the pre-processed operating status data into a fault diagnosis model to obtain a diagnosis result, wherein the fault diagnosis model includes:

[0053] The input module, feature extractor, feature enhancement module and classifier are connected in sequence;

[0054] The input module is used to input the preprocessed operating status data, the feature extractor is used to extract features for different modalities respectively to obtain a feature set, the feature enhancement module is used to apply the attention mechanism to enhance the feature set to obtain an enhanced feature set, and the classifier is used to obtain a diagnostic result based on the enhanced feature set.

[0055] Furthermore, the preprocessing of the operating status data to obtain the preprocessed operating status data includes:

[0056] performing a normalization operation on the vibration signal to obtain preprocessed vibration signal data;

[0057] Performing dimension adjustment on the infrared image to obtain image data of uniform dimension;

[0058] Normalizing the image data of unified dimension to obtain normalized data;

[0059] Random angle rotation processing is performed on the normalized data to obtain pre-processed infrared image data.

[0060] Further, such as Figure 2 As shown in the figure, this method uses accelerometers and infrared thermal imagers as the primary data acquisition devices for data acquisition and preprocessing. Accelerometers are installed on key components of marine power plants (such as bearings or gearboxes) to collect vibration signals during operation, reflecting the health of the equipment's internal mechanical components. Infrared thermal imagers are used to obtain temperature distribution maps during operation to monitor for abnormal heat sources or localized overheating. These two data types comprehensively reflect the equipment's operating status from the perspectives of dynamic and temperature characteristics, respectively. The collected vibration signals and infrared images undergo necessary preprocessing to improve data quality and consistency. The vibration signals are normalized to maintain a uniform dimension. Furthermore, to facilitate subsequent time series analysis in the model, the vibration signals are segmented into fixed-length time windows. Infrared image preprocessing includes uniform resizing, pixel value normalization, and data augmentation (such as rotation and scaling) to ensure image quality during feature extraction and the generalization capability of the model.

[0061] In the model architecture design, a long short-term memory (LSTM) network was used as the feature extraction module to target the time series characteristics of vibration signals. The key feature of LSTM lies in its memory units' ability to capture dynamic patterns and long-term dependencies in time series, making it suitable for analyzing and processing data such as vibration signals. The input layer receives segmented and normalized time series vibration signals, extracts time series features through multiple layers of hidden units, and finally, the output layer generates a high-dimensional feature vector that represents the dynamic characteristics of the signal. A two-dimensional convolutional neural network (CNN) was used to extract infrared image features. CNNs extract spatial features from infrared images by stacking convolutional and pooling layers. The convolutional layers extract local texture and edge features of the image, while the pooling layers reduce dimensionality to retain key features while reducing computational complexity. Subsequently, a fully connected layer maps the extracted features to a high-dimensional space, generating a feature representation of the infrared image. The features of the two modalities are further processed in the classification module through fully connected layers, and a softmax classifier is used to predict the fault type.

[0062] During model training and optimization, supervised learning is employed, optimizing classification performance through a cross-entropy loss function. Differences between classification results for data from different modalities are adjusted by reducing cross-modal consistency loss. Specifically, during model training, the preprocessed vibration signal and infrared image inputs are passed through LSTM and CNN modules, respectively, to extract features for classification. Each modal feature is then applied to a classifier from another modality to generate the output. Finally, the output from the same classifier is measured using cross-modal consistency loss. To prevent overfitting, L2 regularization is applied during training, and the Adam optimizer is used to update network parameters. The initial learning rate is set to 0.001. Furthermore, to improve training efficiency, batch normalization is applied to the output of each network layer to accelerate convergence and improve model stability.

[0063] Specifically, this method uses acceleration sensors and infrared thermal imagers as the main data acquisition devices to obtain the operating status data of the equipment from two dimensions: dynamic vibration characteristics and temperature distribution characteristics. The acceleration sensor is installed on key components such as bearings or gearboxes of the marine power unit to collect vibration signals of mechanical components during operation. When the equipment has faults such as bearing wear or gear breakage, abnormal components of specific frequencies will appear in the vibration signal. Therefore, the vibration signal can directly reflect the health status of the internal mechanical structure (such as gears, bearings, etc.). The infrared thermal imager is used to collect the temperature distribution map of the equipment during operation. It can show whether the equipment has abnormal heat sources or local overheating problems. When lubrication failure or increased friction occurs in the marine power unit, the surface temperature of the component will increase. This fault can be detected and diagnosed by an infrared thermal imager.

[0064] After collecting the vibration signal and temperature distribution map, they need to be preprocessed to varying degrees. The vibration signal is time series data, and when used directly for analysis, there may be large amplitude differences, which will affect the training effect of the model. Therefore, the vibration signal is normalized:

[0065]

[0066] Where x(t) represents the original value of the vibration signal, min(x) and max(x) represent the minimum and maximum values of the vibration signal respectively, and x n (t) represents the normalized signal.

[0067] Infrared images can directly reflect the surface temperature distribution of the equipment, but the original images may have problems such as inconsistent size or insufficient resolution. Therefore, the infrared images are processed as follows:

[0068] All infrared images are uniformly resized to the same size H×W (e.g. 64×64) to ensure consistent input dimensions when inputting the model. The resized image is represented as:

[0069] I′(x,y)=Resize(I(x,y),H,W) (2)

[0070] Where: I(x,y) is the original image, I′(x,y) is the adjusted image, H and W are the height and width of the target image respectively.

[0071] To eliminate the difference in pixel value range, the pixel values of the infrared image are normalized to the range of [0,1]:

[0072]

[0073] In order to improve the robustness and generalization ability of the model, the obtained image is rotated by a random angle θ:

[0074] I′(x′,y′)=I(xcosθ-ysinθ,xsinθ+ycosθ) (4)

[0075] Because vibration signals and infrared images reflect different characteristics of the same equipment, they need to be synchronized during the acquisition and preprocessing stages. This involves temporal alignment and label unification. Specifically, this involves ensuring that the sampling window of the vibration signal aligns with the acquisition time of the infrared image, and that the vibration signal and infrared image share the same label for data under the same equipment operating state (e.g., normal operation, gear fracture, insufficient lubrication, etc.).

[0076] After preprocessing, the normalized fixed-length time series samples are input into the LSTM network in the subsequent network, and the normalized, uniform-sized image samples are input into the CNN network in the subsequent network.

[0077] Furthermore, the feature extractor includes:

[0078] a first sub-extractor and a second sub-extractor;

[0079] The first sub-extractor is used to perform feature extraction on the preprocessed vibration signal data using a long short-term memory network to obtain a first feature subset;

[0080] The second sub-extractor is used to perform feature extraction on the preprocessed infrared image data using a convolutional neural network to obtain a second feature subset.

[0081] Specifically, the feature extraction network as a whole is as follows Figure 3As shown, the network is primarily divided into the following modules: 1) Input module: This module receives data from two modalities—vibration signals and infrared images; 2) Feature extractor: This module extracts features for each modality. Vibration signals are extracted using a long short-term memory (LSTM) network, while infrared images are extracted using a convolutional neural network (CNN); 3) Feature enhancement module: This module applies an attention mechanism to enhance the extracted features; and 4) Classifier: This module processes the features of each modality and converts them into classification results. The following is a detailed description of each module.

[0082] 1) The input module is to place the data collected and preprocessed in step 1 into the feature extractor, input the signal collected by the acceleration sensor in the form of a fixed-length time series, and input the thermal image of the device obtained by the infrared camera in a uniform size.

[0083] 2) The feature extractor is divided into two parts. Since the vibration signal is time series data, a long short-term memory network (LSTM) is used for feature extraction. LSTM can capture long-term temporal dependencies and effectively extract fault mode information. The input of the vibration signal can be expressed as:

[0084]

[0085] Among them, T is the time step and d is the feature dimension.

[0086] The calculation process of LSTM is as follows:

[0087] f t =σ(W f x t +U f h t-1 +b f ) (6)

[0088] i t =σ(W i x t +U i h t-1 +b i ) (7)

[0089] o t =σ(W o x t +U o h t-1 +b o ) (8)

[0090]

[0091] h t =o t ⊙tanh(ct ) (11)

[0092] f t 、i t 、o t They are forget gate, input gate and output gate respectively;

[0093] c t is the unit state;

[0094] h t is a hidden state;

[0095] W and U are the learned weight parameters;

[0096] σ(·) is the Sigmoid activation function, and tanh(·) is the hyperbolic tangent activation function;

[0097] ⊙ represents element-wise multiplication.

[0098] Finally, the features extracted by LSTM are represented as:

[0099] H vib =[h1,h2,...,h T ] (12)

[0100] This feature will be input into the feature enhancement module for fault identification.

[0101] The data structure of infrared images is two-dimensional and spatial features are extracted using convolutional neural networks (CNN). CNN consists of multiple convolutional layers and pooling layers:

[0102] H img =CNN(X img ) (13)

[0103] Among them, X img For the input infrared image, CNN is responsible for extracting high-order features.

[0104] Furthermore, the feature enhancement module includes:

[0105] Temporal attention submodule and spatial attention submodule;

[0106] The temporal attention submodule is used to enhance the first feature subset to obtain a first enhanced subset;

[0107] The spatial attention submodule is used to enhance the second feature subset to obtain a second enhanced subset.

[0108] 3) Feature enhancement module: In order to further improve the expression capability of multimodal features, this method introduces the temporal attention submodule (TAM) and the spatial attention submodule (SAM) in the feature extraction stages of vibration signals and infrared images, respectively, to enhance the expression capability of key time points and key spatial areas in the features, thereby improving the accuracy and robustness of fault diagnosis.

[0109] Spatial Attention Submodule (SAM) Figure 4 , refers to mining the most discriminative spatial regions (such as abnormal fever areas) from the image features extracted by CNN and enhancing them through attention weights.

[0110] The specific calculation steps are:

[0111] Perform global average pooling and global maximum pooling on each channel:

[0112]

[0113] in is the image feature extracted by CNN, GAP is global average pooling, GMP is global maximum pooling, and after splicing, a multi-layer convolution residual structure is used to enhance the perception of local spatial patterns. The specific process is as follows:

[0114] F concat =Concat(F avg ,F max ) (15)

[0115]

[0116] in, and Represents two layers of convolution operations, and the residual connection inputs F concat Added directly to the output to maintain characteristic stability.

[0117] Use the Sigmoid function to generate the attention map:

[0118] M s =σ(Conv 1×1 (F res )) (17)

[0119] Among them, Conv 1×1 Represents the channel compression convolution layer, and the Sigmoid function σ outputs the attention map The original image features are weighted element by element and the output spatial enhancement features are:

[0120]

[0121] TAM is used to enhance the time series features extracted by LSTM, with the goal of modeling the dynamic weight distribution of key time points and the long-term dependencies within the sequence. Figure 5 As shown, it consists of an embedding layer, a position-adaptive coding layer, and a dual coding block.

[0122] The overall structure is as follows:

[0123] The time series features of LSTM output are The original D-dimensional features are mapped to d-dimensional representations through linear transformation:

[0124]

[0125] A set of one-dimensional causal convolution kernels is used for multi-layer temporal modeling. Let the output of the lth layer be X (l) , the basic convolution and gating mechanism is as follows:

[0126] Z (l) =Conv1D(X (l-1) ;W (l) ),G (l) =Conv1D(X (l-1) ; V (l) ) (20)

[0127]

[0128] Where: W (l) ,V (l) is the convolution weight of the lth layer; Represents element-by-element multiplication (gating operation); σ(·) represents the Sigmoid function; all convolutions use dilated convolution to expand the receptive field. The final output is:

[0129] H′=X (L) (twenty two)

[0130] Where L represents the number of convolutional layers, That is, the enhanced sequence feature will be used as the input of the subsequent encoding block.

[0131] Each encoding block in the dual encoding block includes a multi-head self-attention mechanism, a feedforward network, and a residual normalization structure: The multi-head self-attention mechanism:

[0132] Q=HW Q ,K=HW K ,V=HW V

[0133]

[0134] Where: H is the input sequence feature matrix, with a dimension of T×d (T is the time step or sequence length, d is the feature dimension); W Q ,W K ,W V is the projection weight matrix of query, key, and value, all of dimension d×d k ; Q, K, V represent the query, key, and value matrices after the input is linearly transformed, respectively, with dimensions of T×d k ;d k is the feature dimension of each attention head; softmax(·) normalizes each row to represent the attention weight; is the attention score matrix, the scaled dot product, used to stabilize the gradient; Attention(Q,K,V) is the output weighted feature, the dimension is T×d k After splicing multiple heads:

[0135] Z=Concat(Head1,...,Head L )W O (twenty four)

[0136] Head i is the output of the i-th attention head, W O is the final linear transformation weight matrix with a dimension of d×d. Next, the feedforward network + SwiGLU activation is performed:

[0137]

[0138] Where: x is the input feature vector; W1 and V are weight matrices used for feedforward transformation; B and C are bias terms; the dimensions are the same as the output; SwiGLU(x) is a gated activation function that linearly transforms the input features, then performs Sigmoid gating, retains linear output, and finally performs element-by-element multiplication. SwiGLU has stronger nonlinear expression capabilities and gradient stability, and is often used in feedforward networks to improve model performance. The output after gated activation is used as the feedforward network result. Next, residual connection + layer normalization are performed:

[0139] Output=LayerNorm(x+Sublayer(x)) (26)

[0140] Where Sublayer(x) is the transformed output of the feedforward pair x, LayerNorm(x) is the layer normalization, and the output is the enhanced temporal features after processing by two layers of coding blocks:

[0141] X T = EncoderBlock2(EncoderBlock1(H')) (27)

[0142] The final output X T The feature is the enhanced result of the vibration signal channel and is compared with the image enhancement feature X S The input fusion module and classifier are used to implement the multimodal fault recognition task. Here, H' is the sequence feature obtained after the first feature subset undergoes linear transformation and causal convolution kernel group. EncoderBlock1 and EncoderBlock2 are two identical temporal encoding blocks, including a multi-head attention mechanism, a feedforward network, a residual connection, and normalization.

[0143] Furthermore, the obtained features are reduced in dimension through a fully connected layer and then passed through two fully connected layers of exactly the same size to obtain the classification results. Since the training data comes from different modalities, the prediction results of different modal data will be different, so two cross-modal consistency losses are designed. Taking the vibration signal as an example, our goal is to make the vibration signal prediction features of the infrared image classifier and the vibration signal classifier as consistent as possible. Therefore, we input the penultimate layer features of the vibration signal into the infrared image classifier to obtain the prediction result. The prediction result is learned to be consistent with the prediction result of the vibration signal itself. In this process, the Kullback-Leibler divergence (KL divergence) is used. Therefore, the two consistency losses are defined as:

[0144]

[0145] In addition, the classification loss of each modality itself is calculated using cross entropy:

[0146]

[0147] Furthermore, in this method, the model training and optimization process mainly includes loss calculation, parameter optimization and training termination conditions, which are specifically expanded as follows:

[0148] 1) Calculate the loss

[0149] This method designs two types of loss functions: classification loss (Cross-EntropyLoss): used to ensure the classification correctness of training set samples, by calculating the cross entropy between the model prediction value and the true label, training LSTM and CNN to extract effective features; cross-modal consistency loss (KL divergence): used to ensure the consistency of feature extraction results of infrared images and vibration signals, and prevent the features of a certain modality from dominating the classification decision.

[0150] 2) Optimize model parameters

[0151] The goal of model training is to minimize the total loss:

[0152] L total =LC +αL KL (34)

[0153] Among them, α is a hyperparameter used to balance the weights of different losses.

[0154] During the training process, the model updates the feature extraction (θ G1 ,θ G2 ) and the classifier (θ C1 ,θ C2 ) parameters to make the loss function converge.

[0155] 3) Training termination conditions

[0156] The training process will continue until one of the following conditions is met: the number of training epochs reaches a set value (such as 100 epochs); the loss function converges, that is, the loss changes very little after several consecutive rounds of training and the model is stable.

[0157] Through the above optimization process, this method can effectively improve the accuracy of cross-domain fault diagnosis and is suitable for multi-modal equipment monitoring tasks.

[0158] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0159] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for fault diagnosis of marine power plant based on multimodal information fusion, characterized in that: include: Collecting operating status data of the device under test, wherein the operating status data includes: vibration signals and infrared images; Preprocessing the operating status data to obtain preprocessed operating status data; The pre-processed operating status data is input into a fault diagnosis model to obtain a diagnosis result, wherein the fault diagnosis model includes: The input module, feature extractor, feature enhancement module and classifier are connected in sequence; The input module is used to input the preprocessed operating status data, the feature extractor is used to extract features for different modalities respectively to obtain a feature set, the feature enhancement module is used to apply the attention mechanism to enhance the feature set to obtain an enhanced feature set, and the classifier is used to obtain a diagnostic result based on the enhanced feature set.

2. A method for fault diagnosis of a marine power plant based on multimodal information fusion according to claim 1, characterized in that: The preprocessing of the operating status data to obtain the preprocessed operating status data includes: performing a normalization operation on the vibration signal to obtain preprocessed vibration signal data; Performing dimension adjustment on the infrared image to obtain image data of uniform dimension; Normalizing the image data of unified dimension to obtain normalized data; Random angle rotation processing is performed on the normalized data to obtain pre-processed infrared image data.

3. A method for fault diagnosis of a marine power plant based on multimodal information fusion according to claim 2, characterized in that: The feature extractor comprises: a first sub-extractor and a second sub-extractor; The first sub-extractor is used to perform feature extraction on the preprocessed vibration signal data using a long short-term memory network to obtain a first feature subset; The second sub-extractor is used to perform feature extraction on the preprocessed infrared image data using a convolutional neural network to obtain a second feature subset.

4. A method for fault diagnosis of a marine power plant based on multimodal information fusion according to claim 3, characterized in that: The expression of the first feature subset is: H vib =[h1,h2,...,h T ]; Among them, h t (t=1,2...T) is the hidden state vector of LSTM at time step t, and T is the total number of time steps of the vibration signal sequence.

5. A method for fault diagnosis of a marine power plant based on multimodal information fusion according to claim 3, characterized in that: The expression of the second feature subset is: H img =CNN(X img ), where X img is the preprocessed infrared image data.

6. A method for fault diagnosis of a marine power plant based on multimodal information fusion according to claim 6, characterized in that: The feature enhancement module includes: Temporal attention submodule and spatial attention submodule; The temporal attention submodule is used to enhance the first feature subset to obtain a first enhanced subset; The spatial attention submodule is used to enhance the second feature subset to obtain a second enhanced subset.

7. A method for fault diagnosis of a marine power plant based on multimodal information fusion according to claim 6, characterized in that: The expression of the first enhancer subset is: X T =EncoderBlock2(EncoderBlock1(H')); Among them, H' is the sequence feature obtained after the first feature subset undergoes linear transformation and causal convolution kernel group, EncoderBlock1 and EncoderBlock2 are two identical temporal encoding blocks, which include multi-head attention mechanism, feedforward network, residual connection and normalization.

8. A method for fault diagnosis of a marine power plant based on multimodal information fusion according to claim 7, characterized in that: The expression of the second enhancer subset is: M s To generate the attention map using the Sigmoid function, It is the image feature extracted by CNN in the second feature subset.

Images