Davit sample imbalance fault diagnosis method and system based on improved GAN network and multimodal feature fusion

Abstract

This invention discloses a davit sample imbalance fault diagnosis method and system based on an improved GAN network and multimodal feature fusion. By fusing the data features obtained from a one-dimensional time series data diagnostic model and a two-dimensional image data diagnostic model, the complementary information of the two data forms is fully utilized to enhance the performance of the fault diagnosis classifier. The one-dimensional time series data diagnostic model focuses on capturing temporal features, while the two-dimensional image data diagnostic model focuses on extracting frequency features. This combination enables the classifier to fully understand the manifestation and development trends of the fault. This improves the model's ability to recognize complex fault patterns, enhances the robustness and accuracy of the classifier, and thus improves the overall reliability and efficiency of the system.

Description

Technical Field

[0001] The present invention relates to the field of fault diagnosis, and in particular to a method and system for diagnosing faults in a davit electro-hydraulic system based on an improved GAN network and multimodal feature fusion under sample imbalance conditions. Background Art

[0002] Ship davits are specialized equipment installed on ships for lifting and unloading lifeboats or workboats, playing a critical role in ensuring life safety and rapid evacuation in emergency situations. However, due to the intertwined electrical, mechanical, and fluid characteristics of the davit's electro-hydraulic system, its operation exhibits significant nonlinearity, time-varying characteristics, and uncertainty. Davit electro-hydraulic system pipelines are typically closed structures, and failures often occur in difficult-to-monitor internal areas, making it extremely difficult to obtain fault samples. This, in turn, leads to insufficient training data and sample imbalance, increasing the risk of missed and false detections, and significantly affecting the accuracy and reliability of fault diagnosis. Summary of the Invention

[0003] In response to the problems existing in the prior art, the present invention proposes a davit electro-hydraulic system fault diagnosis method and system based on an improved GAN network and multimodal feature fusion to address the problem of imbalanced fault category sample data.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present invention provides a davit sample imbalance fault diagnosis method based on an improved GAN network and multimodal feature fusion, comprising the following steps:

[0006] Step 1: Obtain the status data of the ship's davit electro-hydraulic system and perform noise reduction processing on it;

[0007] Step 2: Use the improved GAN network to balance and expand the unbalanced data after noise reduction;

[0008] Step 3: Divide the expanded data into a training set, a validation set, and a test set (all the data in the set are one-dimensional time series data);

[0009] Step 4 uses the continuous wavelet transform method to convert the one-dimensional time series data in the training set, validation set, and test set into two-dimensional image data;

[0010] Step 5: respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features;

[0011] In step 6, the fused data features are passed to the classifier for fault classification, thereby improving the accuracy and reliability of fault diagnosis.

[0012] As a further technical solution, the improved GAN network is a generator network structure based on a multi-layer Transformer encoder and a self-attention mechanism module.

[0013] As a further technical solution, the specific process of step 2 is as follows:

[0014] Step 2-1: The fully connected layer maps the random noise obtained in step 1 to a high-dimensional space to form the initial features;

[0015] Step 2-2: Multiple Transformer encoders use the Transformer's multi-head self-attention mechanism and feedforward neural network to transform and learn the initial features, capturing the long-range dependencies between the initial features.

[0016] Step 2-3: The self-attention mechanism module further enhances the global information interaction in the initial features, improving the continuity and trend capture capabilities of the generated sequence data;

[0017] Steps 2-4: The convolution layer upsamples and refines the initial features and adjusts the local structure of the generated sequence data;

[0018] Steps 2-5: The fully connected layer maps the features output by the convolutional layer to the length and shape of the target time series, thus generating the final one-dimensional time series data.

[0019] As a further technical solution, the multimodal feature fusion diagnosis model includes a two-dimensional image data diagnosis model and a one-dimensional time series data diagnosis model that are set in parallel.

[0020] As a further technical solution, the specific data processing process of the two-dimensional image data diagnostic model is as follows:

[0021] Step 5-1: The convolution layer extracts local features of the two-dimensional image data and effectively reduces parameters and improves training efficiency by sharing weights.

[0022] Step 5-2: The Swin Transformer Blocks module processes local features and calculates self-attention within the local window to reduce computational complexity. By moving the window, it calculates global features across windows and efficiently captures global and local information.

[0023] Step 5-3: The downsampling layer processes global and local information, reduces the size of input features, retains important features and reduces the amount of computation;

[0024] Step 5-4: The SimAM attention mechanism module enhances the importance of important features, suppresses irrelevant information, and enhances feature representation capabilities;

[0025] Step 5-5: The batch normalization layer performs batch normalization on important features to accelerate convergence and stabilize training;

[0026] Step 5-6: The fully connected layer integrates the features obtained in step 5-5 and maps them to the final output space to output the two-dimensional image data features.

[0027] As a further technical solution, the process of data processing by the one-dimensional time series data diagnostic model is as follows:

[0028] Step 5-1: The convolution layer extracts local data features from the one-dimensional time series data and identifies local trends in the data;

[0029] Step 5-2, the batch normalization layer performs batch normalization on local data features;

[0030] Step 5-3: The activation layer introduces nonlinearity to process the data features obtained in step 5-2;

[0031] Step 5-4: The maximum pooling layer downsamples the data features, extracts important features, and enhances the robustness of the model;

[0032] Step 5-5: The fully connected layer flattens the important features and maps them to a higher-dimensional feature space, integrating the extracted data features;

[0033] Steps 5-6: The Dropout layer trains the data features and randomly discards some neurons during the training process to prevent overfitting and improve the generalization ability of the model;

[0034] Steps 5-7: The LSTM layer captures long-term dependencies in one-dimensional time series data, processes contextual information in the one-dimensional time series data, and effectively retains important information through memory units and gating mechanisms;

[0035] Steps 5-8: The fully connected layer further integrates the data features output by the LSTM layer and maps them to the final output space, outputting the one-dimensional time series data features.

[0036] As a further technical solution, in step 5, the data fusion method is as follows:

[0037] The obtained one-dimensional time series data features and two-dimensional image data features are cascaded in the row dimension. Through the cascade operation, the model can comprehensively utilize the information of the two data features to improve the overall performance.

[0038] As a further technical solution, in step 6, a width learning system is used as a fault diagnosis classifier to perform fault classification. The specific classification method is as follows:

[0039] Input layer: Input the one-dimensional time series data and two-dimensional image data fused data features obtained through cascade operations; Feature mapping layer: Randomly generate weights and map the input features to the new feature space; Augmentation node layer: Add nonlinear nodes to enhance features; Output layer: Use the generalized inverse matrix to solve the output layer weights, output the fault type, model evaluation and parameter optimization.

[0040] In a second aspect, the present invention also provides a davit sample imbalance fault diagnosis system based on an improved GAN network and multimodal feature fusion, as follows:

[0041] The first module is configured to obtain the status data of the ship's davit electro-hydraulic system and perform noise reduction processing on it;

[0042] The second module is configured to use the improved GAN network to balance the imbalanced data after noise reduction;

[0043] The third module is configured to divide the expanded data into a training set, a validation set, and a test set;

[0044] The fourth module is configured to convert the one-dimensional time series data in the training set, the validation set and the test set into two-dimensional image data using a continuous wavelet transform method;

[0045] The fifth module is configured to input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features;

[0046] The sixth module is configured to pass the fused data features into the classifier for fault classification.

[0047] In a third aspect, the present invention further provides a computer device comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0048] The beneficial effects of the present invention are as follows:

[0049] The present invention first obtains the multi-sensor status information of the davit electro-hydraulic system through the data acquisition system, and fuses the data features obtained by the one-dimensional time series data diagnosis model and the two-dimensional image data diagnosis model, making full use of the complementary information of the two data forms and enhancing the performance of the fault diagnosis classifier; the one-dimensional time series data diagnosis model focuses on capturing time series features, while the two-dimensional image data diagnosis model focuses on extracting frequency features. This combination enables the classifier to fully understand the manifestation and development trend of the fault; it improves the model's ability to recognize complex fault modes, enhances the robustness and accuracy of the classifier, and thus improves the overall reliability and efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0051] Figure 1 Fault diagnosis flow chart of the present invention;

[0052] Figure 2 This is a schematic diagram of the GAN network structure;

[0053] Figure 3 This is a schematic diagram of the improved GAN generator network;

[0054] Figure 4 It is a schematic diagram of continuous wavelet (CWT) transform;

[0055] Figure 5 This is a schematic diagram of the multimodal feature fusion diagnosis model;

[0056] Figure 6 It is a schematic diagram of the Swin Transformer Block structure;

[0057] Figure 7 This is a schematic diagram of the LSTM model;

[0058] Figure 8 is a schematic diagram of the width learning system; DETAILED DESCRIPTION

[0059] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0060] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless otherwise clearly indicated in the present invention, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "comprising" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations;

[0061] This embodiment discloses a davit sample imbalance fault diagnosis method based on an improved GAN network and multimodal feature fusion. The method relies on the state data of the ship davit electromechanical and hydraulic system to achieve accurate system fault diagnosis when the electromechanical and hydraulic system training samples are imbalanced. The specific steps are as follows:

[0062] Step 1: Obtain the status data of the ship's davit electro-hydraulic system and perform noise reduction processing on it;

[0063] Step 2: Use the improved GAN network to balance and expand the unbalanced data after noise reduction;

[0064] Step 3: Divide the expanded data into a training set, a validation set, and a test set (all the data in the set are one-dimensional time series data);

[0065] Step 4 uses the continuous wavelet transform method to convert the one-dimensional time series data in the training set, validation set, and test set into two-dimensional image data;

[0066] Step 5: respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features;

[0067] Step 6: The fused data features are passed into the classifier for fault classification.

[0068] Furthermore, the generative adversarial network (GAN) in the prior art includes two adversarial neural networks, a generator and a discriminator. The generator generates realistic data samples from random noise, while the discriminator attempts to distinguish these generated samples from real data. The two networks are gradually improved through adversarial training, the generator continuously improves the quality of the generated samples, and the discriminator continuously improves its discrimination ability, and eventually the generator is able to generate high-quality samples that are difficult to distinguish from real data. The improved GAN network of the present invention is a generator network structure based on Transformer and self-attention mechanism, which aims to make full use of the advantages of Transformer model and self-attention mechanism module in processing long sequence dependencies and capturing global information.

[0069] The multi-layer Transformer encoder design enables the model to extract and integrate feature information from the input data layer by layer. The multi-head self-attention mechanism is a key component of the entire generator network. By processing the relationships between different positions in the input sequence in parallel, it can effectively capture global contextual information. The self-attention mechanism calculates the similarity between each element in the input sequence and all other elements, and then assigns different attention weights based on these similarities. This allows the model to fully consider the global information of all input elements when generating samples, thereby enhancing the model's expressiveness and improving the quality and consistency of generated samples.

[0070] Specifically, the initial input of the generator is random noise generated by Gaussian distribution; the random noise enters the fully connected layer, maps the random noise to a high-dimensional space, and forms an initial feature representation; the random noise data features obtained by the above steps enter multiple Transformer encoders, and the Transformer's multi-head self-attention mechanism and feedforward neural network are used to transform and learn the input features to capture the long-distance dependencies between the input features; the random noise data features obtained by the above steps enter the self-attention mechanism module to further enhance the global information interaction in the input features and improve the continuity and trend capture ability of the generated sequence data; the random noise data features obtained by the above steps enter the convolution layer for upsampling and refinement, and adjust the local structure of the generated sequence data; the random noise data features obtained by the above steps enter the fully connected layer, and map the features output by the convolution layer to the length and shape of the target time series to achieve the final generation of one-dimensional time series data.

[0071] The multi-head attention mechanism described above is based on the self-attention mechanism, which allows each position in the input sequence to pay attention to other positions in the sequence and dynamically calculates the weighted average as the output, thereby capturing the dependencies in the sequence. The core formula of the self-attention mechanism is:

[0072]

[0073] Among them, Q, K, and V represent query, key, and value matrices respectively. is the dimension of the key vector, used to scale the dot product to stabilize the softmax function.

[0074] Furthermore, the multimodal feature fusion diagnostic model in this embodiment includes a two-dimensional image data diagnostic model and a one-dimensional time series data diagnostic model set up in parallel; the one-dimensional time series data is converted into two-dimensional image data through continuous wavelet transform (CWT), and the multimodal data (including the one-dimensional time series data and the two-dimensional image data obtained after conversion) is input into a parallel network structure to extract data features, and then feature fusion and model testing and evaluation are performed.

[0075] Because one-dimensional time series data typically lacks explicit frequency information, making it difficult to directly capture its complex internal features and change patterns, the continuous wavelet transform (CWT) was chosen to extract the data's frequency information. The CWT can convert one-dimensional time series data into two-dimensional image data, thereby extracting the frequency information of the time series data as well as the local characteristics and changes of the signal over time. Through this conversion, deep learning models can more intuitively observe the different frequency components of the signal and its dynamic characteristics over time in the time-frequency domain. The method of converting one-dimensional time series data into two-dimensional image data using the CWT is particularly suitable for processing non-stationary signals in ship davit electro-hydraulic systems. The multi-resolution characteristics of the CWT enable it to capture the details and overall structure of the signal at different scales.

[0076] The mathematical formula of continuous wavelet transform is as follows:

[0077]

[0078] Among them, CWT(a, b) represents the continuous wavelet coefficient, which is the wavelet transform result of the original signal f(t) under the scale a and translation parameter b, f(t) is the original time domain signal, ψ * is the complex conjugate mother wavelet, a is the scale factor used to control the expansion and contraction, and b is the time shift factor used to control the translation.

[0079] The parallel network structure fully leverages the strengths of different modal data. One-dimensional data directly provides detailed information about the original signal, while two-dimensional image data provides a higher-level representation of features. Through multimodal fusion and parallel processing, the network can integrate multiple data features, significantly improving the accuracy and robustness of fault diagnosis.

[0080] The specific processing process of the two-dimensional image data diagnosis model is as follows:

[0081] The two-dimensional image data generated by the CWT method enters the convolution layer. Local features of the image are extracted, and by sharing weights, parameters are effectively reduced, thereby improving training efficiency. The data features obtained by the above steps enter the SwinTransformerBlocks module, which calculates self-attention within the local window to reduce computational complexity. By moving the window, global features across windows are calculated to efficiently capture global and local information. The data features obtained by the above steps enter the downsampling layer, which reduces the size of the input features, retains important features, and reduces the amount of computation. The data features obtained by the above steps enter the SimAM attention mechanism module, which enhances the importance of features, suppresses irrelevant information, and enhances feature representation capabilities. The data features obtained by the above steps enter the batch normalization layer, which performs batch normalization on the input features to accelerate convergence and stabilize training. The data features obtained by the above steps enter the fully connected layer, which integrates the extracted features and maps them to the final output space to output the two-dimensional image data features.

[0082] That is, in this embodiment, in the two-dimensional image data diagnosis model, the Swin Transformer Blocks module is combined with the SimAM attention mechanism module and integrated into the neural network model to significantly improve the model's feature expression ability and diagnostic accuracy. The role of the Swin Transformer Blocks module in fault diagnosis image data is mainly reflected in its powerful local and global feature extraction capabilities. By dividing the image into non-overlapping windows and applying the self-attention mechanism within each window, the Swin Transformer can capture fine-grained local fault features, while capturing global contextual information through hierarchical connections across windows.

[0083] The SimAM attention mechanism module contains an energy function for quantifying the unique properties of each neuron and its role in the neural network, as shown in Equation (3), which reflects the intrinsic properties of neurons and the interactions and spatial inhibition between neurons.

[0084]

[0085] In formula (3), Among them, M is the number of neurons on the channel, w t and b t are weight and bias transformations.

[0086] Minimizing the above formula (3) is essentially equivalent to improving the linear separability between neuron t and other neurons in the same channel. To simplify the processing and introduce stability, the SimAM module adopts a binary label strategy and adds a regularization term to the energy function. The final energy function is as follows:

[0087]

[0088] In theory, each channel has an energy function. The above formula has the following analytical solution:

[0089]

[0090] In formula (5) and formula (6),

[0091] Therefore, the minimum energy can be obtained by the following formula (7):

[0092]

[0093] In formula (7),

[0094] The meaning of the above formula is that when the energy value is lower, the distinction between neuron t and surrounding neurons is greater, which indicates that its importance is higher.

[0095] The key to the attention mechanism is to enhance the features to highlight the information carried by neurons with high importance. It allows the model to dynamically adjust the degree of attention paid to the input features, thereby more accurately capturing the key information. According to the definition of the attention mechanism, the features need to be enhanced, as shown in formula (8).

[0096]

[0097] By dynamically weighting the feature maps, the SimAM attention mechanism module can effectively emphasize important features in the davit electro-hydraulic system image data and suppress irrelevant noise, thereby enhancing the model's robustness and generalization ability for complex images.

[0098] In a one-dimensional time series data diagnostic model, the combination of a Dropout layer and an LSTM layer effectively captures the long- and short-term dependencies in the one-dimensional time series data of the davit electro-hydraulic system and reduces the model's over-reliance on training data. The Dropout layer prevents overfitting during training by randomly discarding some neurons, improving the model's generalization and stability.

[0099] Specifically, the one-dimensional time series data enters the convolution layer, which extracts local features of the one-dimensional time series data and identifies local trends of the data; the data features obtained by the above steps enter the batch normalization layer, which performs batch normalization on the input data features to accelerate convergence and improve model stability; the data features obtained by the above steps enter the activation layer, which introduces nonlinearity and improves the expression ability of the model; the data features obtained by the above steps enter the maximum pooling layer, which downsamples the convolution features, extracts important features, and enhances the robustness of the model; the data features obtained by the above steps enter the fully connected layer, which flattens the input data features and maps them to higher dimensional feature space, integrating the extracted data features; the data features processed by the above steps enter the Dropout layer, randomly discarding a part of neurons during the training process to prevent overfitting and improve the generalization ability of the model; the data features processed by the above steps enter the LSTM layer, capturing the long-term dependencies in the time series data, processing the context information in the time series data, and effectively retaining important information through memory units and gating mechanisms; the data features processed by the above steps enter the fully connected layer, further integrating the data features output by the LSTM layer, and mapping them to the final output space, outputting the one-dimensional time series data features.

[0100] Among them, the feedforward operation of the standard neural network can be described as:

[0101]

[0102] The forward feedback operation of the Dropout neural network can be described as:

[0103]

[0104] In the above formula, l represents the layer of the neural network, z represents the input vector, y represents the output vector, w represents the weight, b represents the bias, and f represents the activation function.

[0105] The LSTM layer effectively captures long-term dependencies and complex dynamic patterns in time series data through its long short-term memory mechanism, enhancing the model's ability to understand and extract sequence features.

[0106] The combination of the Dropout layer and the LSTM layer helps the model capture key patterns more accurately and better learn and understand data when processing time series data.

[0107] Furthermore, in terms of fault diagnosis classification, this embodiment uses a breadth learning system (BLS) as the fault diagnosis classifier. The main advantages of BLS classification over Softmax classification are faster training, simpler model structure, lower computational resource consumption, and strong generalization and adaptability. It can quickly update the model when new data or features are added, meeting the high real-time requirements of fault diagnosis. Specifically, the input layer: inputs the data features of one-dimensional time series data and two-dimensional image data obtained through cascade operations; the feature mapping layer: randomly generates weights and maps the input features to a new feature space; the augmentation node layer: adds nonlinear nodes to enhance features; the output layer: uses the generalized inverse matrix to solve the output layer weights, output the fault type, and conduct model evaluation and parameter optimization.

[0108] This invention enhances the performance of the fault diagnosis classifier by fusing the data features obtained from the one-dimensional time series data diagnostic model and the two-dimensional image data diagnostic model, fully leveraging the complementary information between the two data forms. The one-dimensional time series data diagnostic model focuses on capturing temporal features, while the two-dimensional image data diagnostic model focuses on extracting frequency features. This combination enables the classifier to fully understand the manifestation and development trends of faults. This improves the model's ability to recognize complex fault patterns, enhances the classifier's robustness and accuracy, and thus improves the overall reliability and efficiency of the system.

[0109] Furthermore, this embodiment also provides a davit sample imbalance fault diagnosis system based on an improved GAN network and multimodal feature fusion, including:

[0110] The first module is configured to obtain the status data of the ship's davit electro-hydraulic system and perform noise reduction processing on it;

[0111] The second module is configured to use the improved GAN network to balance the imbalanced data after noise reduction;

[0112] The third module is configured to divide the expanded data into a training set, a validation set, and a test set;

[0113] The fourth module is configured to convert the one-dimensional time series data in the training set, the validation set and the test set into two-dimensional image data using a continuous wavelet transform method;

[0114] The fifth module is configured to input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features;

[0115] The sixth module is configured to pass the fused data features into the classifier for fault classification.

[0116] The specific composition of each module and the method of processing data can be found in the description of the method section and will not be repeated here.

[0117] Furthermore, this embodiment also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method described above.

[0118] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A davit sample imbalance fault diagnosis method based on an improved GAN network and multimodal feature fusion, characterized by: The following steps are involved: Step 1: Obtain the status data of the ship's davit electro-hydraulic system and perform noise reduction processing on it; Step 2: Using an improved GAN network to balance and expand the unbalanced data after noise reduction; the improved GAN network is a generator network structure based on a multi-layer Transformer encoder and a self-attention mechanism module; Step 3: Divide the expanded data into training set, validation set, and test set; Step 4 uses the continuous wavelet transform method to convert the one-dimensional time series data in the training set, validation set, and test set into two-dimensional image data; Step 5: respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features; the multimodal feature fusion diagnosis model includes a two-dimensional image data diagnosis model and a one-dimensional time series data diagnosis model set in parallel; the specific data processing process of the two-dimensional image data diagnosis model is as follows: Step 5-1, the convolution layer extracts the local features of the two-dimensional image data, and by sharing weights, effectively reduces parameters and improves training efficiency; Step 5-2, the Swin Transformer The Blocks module processes local features, calculates self-attention within the local window, reduces computational complexity, and calculates global features across windows by moving the window, efficiently capturing global and local information. In step 5-3, the downsampling layer processes global and local information, reduces the size of the input features, retains important features, and reduces the amount of computation. In step 5-4, the SimAM attention mechanism module enhances the importance of important features, suppresses irrelevant information, and enhances feature representation capabilities. In step 5-5, the batch normalization layer performs batch normalization on important features to accelerate convergence and stabilize training. In step 5-6, the fully connected layer integrates the features obtained in step 5-5 and maps them to the final output space, outputting the two-dimensional image data features. Step 6: The fused data features are passed into the classifier for fault classification. The width learning system is used as the fault diagnosis classifier to perform fault classification. The specific classification method is as follows: Input layer: Input the fused data features of the one-dimensional time series data and the two-dimensional image data obtained through the cascade operation; Feature mapping layer: Randomly generate weights and map the input features to the new feature space; Augmentation node layer: Add nonlinear nodes to enhance the features; Output layer: Use the generalized inverse matrix to solve the output layer weights, output fault types, model evaluation and parameter optimization.

2. The davit sample imbalance fault diagnosis method based on the improved GAN network and multimodal feature fusion according to claim 1 is characterized in that: The specific process of step 2 is as follows: Step 2-1: The fully connected layer maps the random noise obtained in step 1 to a high-dimensional space to form the initial features; Step 2-2: Multiple Transformer encoders use the Transformer's multi-head self-attention mechanism and feedforward neural network to transform and learn the initial features, capturing the long-range dependencies between the initial features. Step 2-3: The self-attention mechanism module further enhances the global information interaction in the initial features, improving the continuity and trend capture capabilities of the generated sequence data; Steps 2-4: The convolution layer upsamples and refines the initial features and adjusts the local structure of the generated sequence data; Steps 2-5: The fully connected layer maps the features output by the convolutional layer to the length and shape of the target time series, thus generating the final one-dimensional time series data.

3. The davit sample imbalance fault diagnosis method based on the improved GAN network and multimodal feature fusion according to claim 1 is characterized in that: The process of data processing by the one-dimensional time series data diagnostic model is as follows: Step 5-1: The convolution layer extracts local data features from the one-dimensional time series data and identifies local trends in the data; Step 5-2, the batch normalization layer performs batch normalization on local data features; Step 5-3: The activation layer introduces nonlinearity to process the data features obtained in step 5-2; Step 5-4: The maximum pooling layer downsamples the data features, extracts important features, and enhances the robustness of the model; Step 5-5: The fully connected layer flattens the important features and maps them to a higher-dimensional feature space, integrating the extracted data features; Steps 5-6: The Dropout layer trains the data features and randomly discards some neurons during the training process to prevent overfitting and improve the generalization ability of the model; Steps 5-7: The LSTM layer captures long-term dependencies in one-dimensional time series data, processes contextual information in the one-dimensional time series data, and effectively retains important information through memory units and gating mechanisms; Steps 5-8: The fully connected layer further integrates the data features output by the LSTM layer and maps them to the final output space, outputting the one-dimensional time series data features.

4. The davit sample imbalance fault diagnosis method based on the improved GAN network and multimodal feature fusion according to claim 1 is characterized in that: In step 5, the data fusion method is as follows: The obtained one-dimensional time series data features and two-dimensional image data features are cascaded in the row dimension. Through the cascade operation, the model can comprehensively utilize the information of the two data features to improve the overall performance.

5. A davit sample imbalance fault diagnosis system based on an improved GAN network and multimodal feature fusion, characterized by: The first module is configured to obtain the status data of the ship's davit electro-hydraulic system and perform noise reduction processing on it; The second module is configured to balance and expand the unbalanced data after noise reduction using an improved GAN network; the improved GAN network is a generator network structure based on a multi-layer Transformer encoder and a self-attention mechanism module; The third module is configured to divide the expanded data into a training set, a validation set, and a test set; The fourth module is configured to convert the one-dimensional time series data in the training set, the validation set and the test set into two-dimensional image data using a continuous wavelet transform method; The fifth module is configured to respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features; the multimodal feature fusion diagnosis model includes a two-dimensional image data diagnosis model and a one-dimensional time series data diagnosis model set in parallel; the specific data processing process of the two-dimensional image data diagnosis model is as follows: Step 5-1, the convolution layer extracts the local features of the two-dimensional image data, and by sharing weights, effectively reduces parameters and improves training efficiency; Step 5-2, Swin Transformer The Blocks module processes local features, calculates self-attention within the local window, reduces computational complexity, and calculates global features across windows by moving the window, efficiently capturing global and local information. In step 5-3, the downsampling layer processes global and local information, reduces the size of the input features, retains important features, and reduces the amount of computation. In step 5-4, the SimAM attention mechanism module enhances the importance of important features, suppresses irrelevant information, and enhances feature representation capabilities. In step 5-5, the batch normalization layer performs batch normalization on important features to accelerate convergence and stabilize training. In step 5-6, the fully connected layer integrates the features obtained in step 5-5 and maps them to the final output space, outputting the two-dimensional image data features. The sixth module is configured to pass the fused data features into the classifier for fault classification. It uses a wide learning system as a fault diagnosis classifier to perform fault classification. The specific classification method is as follows: Input layer: Inputs the fused data features of one-dimensional time series data and two-dimensional image data obtained through cascade operations; Feature mapping layer: Randomly generates weights and maps the input features to a new feature space; Augmentation node layer: Adds nonlinear nodes to enhance features; Output layer: Use the generalized inverse matrix to solve the output layer weights, output fault types, model evaluation and parameter optimization.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to claim 1.

Images