Improved GAN network and multi-modal feature fusion-based boat hanging frame sample imbalance fault diagnosis method and system

Abstract

The invention discloses an improved GAN network and multi-modal feature fusion-based boat hanging frame sample imbalance fault diagnosis method and system, and the method comprises the steps: carrying out the feature fusion of data features obtained by a one-dimensional time sequence data diagnosis model and a two-dimensional image data diagnosis model, and fully utilizing the complementary information of two data forms; and the performance of the fault diagnosis classifier is enhanced. The one-dimensional time sequence data diagnosis model pays attention to capturing time sequence features, the two-dimensional image data diagnosis model pays attention to extracting frequency features, and the classifier can comprehensively understand the performance and development trend of faults through the combination. The method improves the recognition capability of a model for a complex fault mode, enhances the robustness and accuracy of a classifier, and improves the overall reliability and efficiency of a system.

Description

Technical Field

[0001] The present invention relates to the field of fault diagnosis, and in particular to a method and system for diagnosing a davit electro-hydraulic system fault based on an improved GAN network and multi-modal feature fusion under sample imbalance conditions. Background Art

[0002] Ship davits are special equipment installed on ships for lifting and unloading lifeboats or workboats, which play a key role in life safety and rapid evacuation in emergency situations. However, due to the interweaving of electrical, mechanical and fluid characteristics of the davit electro-hydraulic system, its operation process exhibits significant nonlinear, time-varying and uncertain characteristics. The pipelines of the davit electro-hydraulic system are usually closed structures, and faults often occur in internal areas that are difficult to monitor, making it extremely difficult to obtain fault samples, which in turn leads to problems such as insufficient training data and unbalanced samples, increasing the risk of missed detection and false detection, and significantly affecting the accuracy and reliability of fault diagnosis. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention proposes a davit electro-hydraulic system fault diagnosis method and system based on an improved GAN network and multi-modal feature fusion to cope with the problem of imbalanced fault category sample data.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present invention provides a davit sample imbalance fault diagnosis method based on an improved GAN network and multimodal feature fusion, comprising the following steps:

[0006] Step 1: obtaining the status data of the ship davit electro-hydraulic system and performing noise reduction processing on it;

[0007] Step 2: Use the improved GAN network to balance and expand the unbalanced data after noise reduction;

[0008] Step 3: Divide the expanded data into a training set, a validation set, and a test set (all the data in the set are one-dimensional time series data);

[0009] Step 4 uses a continuous wavelet transform method to convert the one-dimensional time series data in the training set, the validation set, and the test set into two-dimensional image data;

[0010] Step 5: respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features;

[0011] Step 6 passes the fused data features into the classifier for fault classification, thereby improving the accuracy and reliability of fault diagnosis.

[0012] As a further technical solution, the improved GAN network is a generator network structure based on a multi-layer Transformer encoder and a self-attention mechanism module.

[0013] As a further technical solution, the specific process of step 2 is as follows:

[0014] Step 2-1, the fully connected layer maps the random noise obtained in step 1 to a high-dimensional space to form the initial features;

[0015] Step 2-2: Multiple Transformer encoders use the Transformer's multi-head self-attention mechanism and feedforward neural network to transform and learn the initial features and capture the long-distance dependencies between the initial features.

[0016] Step 2-3: The self-attention mechanism module further enhances the global information interaction in the initial features, improving the continuity and trend capture capabilities of the generated sequence data;

[0017] Step 2-4: The convolution layer upsamples and refines the initial features and adjusts the local structure of the generated sequence data;

[0018] Step 2-5: The fully connected layer maps the features output by the convolutional layer to the length and shape of the target time series to generate the final one-dimensional time series data.

[0019] As a further technical solution, the multimodal feature fusion diagnosis model includes a two-dimensional image data diagnosis model and a one-dimensional time series data diagnosis model that are set in parallel.

[0020] As a further technical solution, the specific data processing process of the two-dimensional image data diagnosis model is as follows:

[0021] Step 5-1, the convolution layer extracts local features of the two-dimensional image data, and by sharing weights, effectively reduces parameters and improves training efficiency;

[0022] Step 5-2, the Swin Transformer Blocks module processes local features, calculates self-attention within the local window, reduces computational complexity, and calculates global features across windows by moving the window to efficiently capture global and local information;

[0023] Step 5-3, the downsampling layer processes global and local information, reduces the size of input features, retains important features and reduces the amount of calculation;

[0024] Step 5-4, the SimAM attention mechanism module enhances the importance of important features, suppresses irrelevant information, and enhances feature representation capabilities;

[0025] Step 5-5, the batch normalization layer performs batch normalization on important features to accelerate convergence and stabilize training;

[0026] Step 5-6: The fully connected layer integrates the features obtained in step 5-5 and maps them to the final output space to output the two-dimensional image data features.

[0027] As a further technical solution, the process of data processing by the one-dimensional time series data diagnostic model is as follows:

[0028] Step 5-1, the convolution layer extracts local data features from the one-dimensional time series data and identifies local trends of the data;

[0029] Step 5-2, the batch normalization layer performs batch normalization on local data features;

[0030] Step 5-3, the activation layer introduces nonlinearity to process the data features obtained in step 5-2;

[0031] Step 5-4: The maximum pooling layer downsamples the data features, extracts important features, and enhances the robustness of the model;

[0032] Step 5-5, the fully connected layer flattens and maps important features to a higher-dimensional feature space and integrates the extracted data features;

[0033] Step 5-6: The Dropout layer trains the data features and randomly discards some neurons during the training process to prevent overfitting and improve the generalization ability of the model;

[0034] Steps 5-7, the LSTM layer captures the long-term dependencies in the one-dimensional time series data, processes the contextual information in the one-dimensional time series data, and effectively retains important information through memory units and gating mechanisms;

[0035] Step 5-8: The fully connected layer further integrates the data features output by the LSTM layer and maps them to the final output space, outputting the one-dimensional time series data features.

[0036] As a further technical solution, in step 5, the method of data fusion is as follows:

[0037] The obtained one-dimensional time series data features and two-dimensional image data features are cascaded in the row dimension. Through the cascade operation, the model can comprehensively utilize the information of the two data features to improve the overall performance.

[0038] As a further technical solution, in step 6, a width learning system is used as a fault diagnosis classifier to perform fault classification. The specific classification method is as follows:

[0039] Input layer: input the one-dimensional time series data and two-dimensional image data fused data features obtained through cascade operation; feature mapping layer: randomly generate weights and map the input features to the new feature space; augmented node layer: add nonlinear nodes to enhance features; output layer: use the generalized inverse matrix to solve the output layer weights, output fault types, model evaluation and parameter optimization.

[0040] In a second aspect, the present invention also provides a davit sample imbalance fault diagnosis system based on an improved GAN network and multimodal feature fusion, as follows:

[0041] The first module is configured to obtain the status data of the ship's davit electro-hydraulic system and perform noise reduction processing on it;

[0042] The second module is configured to use the improved GAN network to perform balanced expansion on the unbalanced data after noise reduction;

[0043] The third module is configured to divide the expanded data into a training set, a validation set, and a test set;

[0044] The fourth module is configured to convert the one-dimensional time series data in the training set, the validation set and the test set into two-dimensional image data using a continuous wavelet transform method;

[0045] The fifth module is configured to respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features;

[0046] The sixth module is configured to pass the fused data features into the classifier for fault classification.

[0047] In a third aspect, the present invention further provides a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0048] The beneficial effects of the present invention are as follows:

[0049] The present invention first obtains the multi-sensor status information of the davit electro-hydraulic system through the data acquisition system, and fuses the data features obtained by the one-dimensional time series data diagnosis model and the two-dimensional image data diagnosis model, so as to make full use of the complementary information of the two data forms and enhance the performance of the fault diagnosis classifier; the one-dimensional time series data diagnosis model focuses on capturing the time series features, while the two-dimensional image data diagnosis model focuses on extracting the frequency features. This combination enables the classifier to fully understand the manifestation and development trend of the fault; improves the model's recognition ability for complex fault modes, enhances the robustness and accuracy of the classifier, and thus improves the overall reliability and efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0051] Figure 1 Fault diagnosis flow chart of the present invention;

[0052] Figure 2 It is a schematic diagram of the GAN network structure;

[0053] Figure 3 It is a schematic diagram of the improved GAN generator network;

[0054] Figure 4 It is a schematic diagram of continuous wavelet (CWT) transform;

[0055] Figure 5 It is a schematic diagram of the multimodal feature fusion diagnosis model;

[0056] Figure 6 It is a schematic diagram of the Swin Transformer Block structure;

[0057] Figure 7 It is a schematic diagram of the LSTM model;

[0058] Figure 8 is a schematic diagram of a width learning system; DETAILED DESCRIPTION

[0059] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0060] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless otherwise explicitly stated in the present invention, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof;

[0061] This embodiment discloses a davit sample imbalance fault diagnosis method based on an improved GAN network and multimodal feature fusion. The method relies on the state data of the electromechanical and hydraulic system of the ship davit to realize accurate fault diagnosis of the system when the electromechanical and hydraulic system training samples are imbalanced. The specific steps are as follows:

[0062] Step 1: obtaining the status data of the ship davit electro-hydraulic system and performing noise reduction processing on it;

[0063] Step 2: Use the improved GAN network to balance and expand the unbalanced data after noise reduction;

[0064] Step 3: Divide the expanded data into a training set, a validation set, and a test set (all the data in the set are one-dimensional time series data);

[0065] Step 4 uses a continuous wavelet transform method to convert the one-dimensional time series data in the training set, the validation set, and the test set into two-dimensional image data;

[0066] Step 5: respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features;

[0067] Step 6: Pass the fused data features into the classifier for fault classification.

[0068] Furthermore, the generative adversarial network (GAN) in the prior art includes two adversarial neural networks, a generator and a discriminator. The generator generates realistic data samples from random noise, while the discriminator attempts to distinguish these generated samples from real data. The two networks are gradually improved through adversarial training, the generator continuously improves the quality of the generated samples, and the discriminator continuously improves the discrimination ability, and finally the generator is able to generate high-quality samples that are difficult to distinguish from real data. The improved GAN network of the present invention is a generator network structure based on Transformer and self-attention mechanism, which is designed to make full use of the advantages of Transformer model and self-attention mechanism module in processing long sequence dependencies and capturing global information.

[0069] Among them, the design of the multi-layer Transformer encoder enables the model to extract and integrate the feature information of the input data layer by layer. The multi-head self-attention mechanism is a key part of the entire generator network. By processing the relationship between different positions in the input sequence in parallel, it can effectively capture global context information. The self-attention mechanism calculates the similarity between each element in the input sequence and all other elements, and then assigns different attention weights based on these similarities, so that the model can fully consider the global information of all input elements when generating samples, thereby enhancing the model's expressiveness and improving the quality and consistency of generated samples.

[0070] Specifically, the initial input of the generator is random noise generated by Gaussian distribution; the random noise enters the fully connected layer, maps the random noise to a high-dimensional space, and forms an initial feature representation; the random noise data features obtained by the above steps enter multiple Transformer encoders, and the input features are transformed and learned by using the Transformer's multi-head self-attention mechanism and feedforward neural network to capture the long-distance dependencies between the input features; the random noise data features obtained by the above steps enter the self-attention mechanism module to further enhance the global information interaction in the input features and improve the continuity and trend capture ability of the generated sequence data; the random noise data features obtained by the above steps enter the convolution layer for upsampling and refinement, and adjust the local structure of the generated sequence data; the random noise data features obtained by the above steps enter the fully connected layer, and the features output by the convolution layer are mapped to the length and shape of the target time series, so as to achieve the generation of the final one-dimensional time series data.

[0071] The above multi-head attention mechanism is based on the self-attention mechanism, which allows each position in the input sequence to pay attention to other positions in the sequence and dynamically calculates the weighted average as the output to capture the dependencies in the sequence. The core formula of the self-attention mechanism is:

[0072]

[0073] Among them, Q, K, and V represent query, key, and value matrices respectively. is the dimension of the key vector, used to scale the dot product to stabilize the softmax function.

[0074] Furthermore, the multimodal feature fusion diagnostic model in this embodiment includes a two-dimensional image data diagnostic model and a one-dimensional time series data diagnostic model arranged in parallel; the one-dimensional time series data is converted into two-dimensional image data through continuous wavelet transform (CWT), and the multimodal data (including the one-dimensional time series data and the two-dimensional image data obtained after the conversion) is input into a parallel network structure to extract data features, and then feature fusion and model testing and evaluation are performed.

[0075] Since one-dimensional time series data usually lacks explicit frequency information and it is difficult to directly capture its internal complex features and change patterns, continuous wavelet transform is chosen to extract data frequency information. Continuous wavelet transform can convert one-dimensional time series data into two-dimensional image data, thereby extracting the frequency information of the time series data and the local characteristics and changes of the signal in time. Through this conversion, the deep learning model can more intuitively observe the different frequency components of the signal and its dynamic characteristics over time in the time-frequency domain. The method of converting one-dimensional time series data into two-dimensional image data using continuous wavelet transform is particularly suitable for processing non-stationary signals of ship davit electro-hydraulic systems. The multi-resolution characteristics of continuous wavelet transform enable it to capture the details and overall structure of the signal at different scales.

[0076] The mathematical formula of continuous wavelet transform is as follows:

[0077]

[0078] Among them, CWT(a, b) represents the continuous wavelet coefficient, which is the wavelet transform result of the original signal f(t) under scale a and translation parameter b, f(t) is the original time domain signal, ψ * is the complex conjugate mother wavelet, a is the scale factor used to control the expansion and contraction, and b is the time shift factor used to control the translation.

[0079] The parallel network structure can make full use of the advantages of different modal data. On the one hand, one-dimensional data directly provides detailed information of the original signal, and on the other hand, two-dimensional image data provides a higher-level feature representation. Through multimodal fusion and parallel processing, the network can integrate multiple data features and significantly improve the accuracy and robustness of fault diagnosis.

[0080] The specific processing process of the two-dimensional image data diagnosis model is as follows:

[0081] The two-dimensional image data generated by the CWT method enters the convolution layer. Extract local features of the image, and effectively reduce parameters and improve training efficiency by sharing weights; the data features processed by the above steps enter the SwinTransformerBlocks module, calculate self-attention in the local window, reduce computational complexity, and calculate global features across windows by moving windows to efficiently capture global and local information; the data features processed by the above steps enter the downsampling layer to reduce the size of the input features, retain important features and reduce the amount of calculation; the data features processed by the above steps enter the SimAM attention mechanism module, and the importance of features is enhanced through the attention mechanism module, irrelevant information is suppressed, and the feature representation ability is enhanced; the data features processed by the above steps enter the batch normalization layer, batch normalization is performed on the input features, and convergence is accelerated and training is stabilized; the data features processed by the above steps enter the fully connected layer, the extracted features are integrated, and mapped to the final output space, and the two-dimensional image data features are output.

[0082] That is, in the two-dimensional image data diagnosis model in this embodiment, the Swin Transformer Blocks module combined with the SimAM attention mechanism module is integrated into the neural network model to significantly improve the model's feature expression ability and diagnostic accuracy. The role of the Swin Transformer Blocks module in fault diagnosis image data is mainly reflected in its powerful local and global feature extraction capabilities. By dividing the image into non-overlapping windows and applying the self-attention mechanism in each window, the Swin Transformer can capture fine-grained local fault features, while capturing global contextual information through hierarchical connections across windows.

[0083] The SimAM attention mechanism module contains an energy function for quantifying the unique properties of each neuron and its role in the neural network, as shown in formula (3), which reflects the intrinsic characteristics of neurons and the interactions and spatial inhibition between neurons.

[0084]

[0085] In formula (3), Among them, M is the number of neurons on the channel, w t and b t are the weight and bias transformations.

[0086] Minimizing the above formula (3) is essentially equivalent to improving the linear separability between neuron t and other neurons in the same channel. In order to simplify the processing and introduce stability, the SimAM module adopts a binary label strategy and adds a regularization term to the energy function. The final energy function is as follows:

[0087]

[0088] In theory, each channel has an energy function. The above formula has the following analytical solution:

[0089]

[0090] In formula (5) and formula (6),

[0091] Therefore, the minimum energy can be obtained by the following formula (7):

[0092]

[0093] In formula (7),

[0094] The meaning of the above formula is that when the energy value is lower, the distinction between neuron t and surrounding neurons is greater, which indicates that its importance is higher.

[0095] The key to the attention mechanism is to enhance the features to highlight the information carried by neurons with high importance. It allows the model to dynamically adjust the degree of attention to the input features, thereby capturing key information more accurately. According to the definition of the attention mechanism, the features need to be enhanced, as shown in formula (8).

[0096]

[0097] By dynamically weighting the feature maps, the SimAM attention mechanism module can effectively emphasize the important features in the image data of the davit electro-hydraulic system and suppress irrelevant noise, thereby enhancing the model's robustness and generalization ability for complex images.

[0098] In the one-dimensional time series data diagnosis model, the combination of Dropout layer and LSTM layer can effectively capture the long-term and short-term dependencies in the one-dimensional time series data of the davit electro-hydraulic system and reduce the model's excessive dependence on training data. Among them, the Dropout layer randomly discards some neurons to prevent the model from overfitting during the training process, thereby improving the generalization ability and stability of the model.

[0099] Specifically, the one-dimensional time series data enters the convolution layer, and local features are extracted from the one-dimensional time series data to identify local trends of the data; the data features processed by the above steps enter the batch normalization layer, and batch normalization is performed on the input data features to accelerate convergence and improve model stability; the data features processed by the above steps enter the activation layer, and nonlinearity is introduced to improve the expression ability of the model; the data features processed by the above steps enter the maximum pooling layer, and the convolutional features are downsampled to extract important features and enhance the robustness of the model; the data features processed by the above steps enter the fully connected layer, and the input data features are flattened and mapped to higher dimensional feature space, integrating the extracted data features; the data features processed by the above steps enter the Dropout layer, randomly discarding a part of neurons during the training process to prevent overfitting and improve the generalization ability of the model; the data features processed by the above steps enter the LSTM layer, capturing the long-term dependencies in the time series data, processing the contextual information in the time series data, and effectively retaining important information through memory units and gating mechanisms; the data features processed by the above steps enter the fully connected layer, further integrating the data features output by the LSTM layer, and mapping them to the final output space, outputting the one-dimensional time series data features.

[0100] Among them, the forward feedback operation of the standard neural network can be described as:

[0101]

[0102] The forward feedback operation of the Dropout neural network can be described as:

[0103]

[0104] In the above formula, l represents the layer of the neural network, z represents the input vector, y represents the output vector, w represents the weight, b represents the bias, and f represents the activation function.

[0105] The LSTM layer effectively captures long-term dependencies and complex dynamic patterns in time series data through its long short-term memory mechanism, enhancing the model's ability to understand and extract sequence features.

[0106] The combination of the Dropout layer and the LSTM layer helps the model to more accurately capture key patterns and better learn and understand data when processing time series data.

[0107] Furthermore, in terms of fault diagnosis classification, the present embodiment selects the use of the breadth learning system (BLS) as the fault diagnosis classifier. The main advantages of BLS classification over Softmax classification are faster training speed, simpler model structure, lower computing resource consumption, and strong generalization ability and adaptability. It can quickly update the model when new data or features are added, meeting the fault diagnosis needs with high real-time requirements. Specifically, the input layer: input the one-dimensional time series data and two-dimensional image data fusion data features obtained by cascade operation; feature mapping layer: randomly generate weights and map the input features to the new feature space; augmented node layer: add nonlinear nodes to enhance features; output layer: use the generalized inverse matrix to solve the output layer weights, output the fault type, model evaluation and parameter optimization.

[0108] The present invention fuses the data features obtained by the one-dimensional time series data diagnosis model and the two-dimensional image data diagnosis model, making full use of the complementary information of the two data forms, thereby enhancing the performance of the fault diagnosis classifier. The one-dimensional time series data diagnosis model focuses on capturing the time series features, while the two-dimensional image data diagnosis model focuses on extracting the frequency features. This combination enables the classifier to fully understand the manifestation and development trend of the fault. The model's ability to recognize complex fault modes is improved, and the robustness and accuracy of the classifier are enhanced, thereby improving the overall reliability and efficiency of the system.

[0109] Furthermore, this embodiment also provides a davit sample imbalance fault diagnosis system based on an improved GAN network and multimodal feature fusion, including:

[0110] The first module is configured to obtain the status data of the ship's davit electro-hydraulic system and perform noise reduction processing on it;

[0111] The second module is configured to use the improved GAN network to perform balanced expansion on the unbalanced data after noise reduction;

[0112] The third module is configured to divide the expanded data into a training set, a validation set, and a test set;

[0113] The fourth module is configured to convert the one-dimensional time series data in the training set, the validation set and the test set into two-dimensional image data using a continuous wavelet transform method;

[0114] The fifth module is configured to respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features;

[0115] The sixth module is configured to pass the fused data features into the classifier for fault classification.

[0116] The specific composition of each module and the method of processing data can be found in the description of the method section, which will not be elaborated here.

[0117] Furthermore, this embodiment also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method described above.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A davit sample imbalance fault diagnosis method based on improved GAN network and multimodal feature fusion, characterized in that: The following steps are involved: Step 1: obtaining the status data of the ship davit electro-hydraulic system and performing noise reduction processing on it; Step 2: Use the improved GAN network to balance and expand the unbalanced data after noise reduction; Step 3: Divide the expanded data into a training set, a validation set, and a test set; Step 4 uses a continuous wavelet transform method to convert the one-dimensional time series data in the training set, the validation set, and the test set into two-dimensional image data; Step 5: respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features; Step 6: Pass the fused data features into the classifier for fault classification.

2. The davit sample imbalance fault diagnosis method based on improved GAN network and multimodal feature fusion as claimed in claim 1 is characterized in that: The improved GAN network is a generator network structure based on a multi-layer Transformer encoder and a self-attention mechanism module.

3. The davit sample imbalance fault diagnosis method based on improved GAN network and multimodal feature fusion as claimed in claim 1, characterized in that: The specific process of step 2 is as follows: Step 2-1, the fully connected layer maps the random noise obtained in step 1 to a high-dimensional space to form the initial features; Step 2-2: Multiple Transformer encoders use the Transformer's multi-head self-attention mechanism and feedforward neural network to transform and learn the initial features and capture the long-distance dependencies between the initial features. Step 2-3: The self-attention mechanism module further enhances the global information interaction in the initial features, improving the continuity and trend capture capabilities of the generated sequence data; Step 2-4: The convolution layer upsamples and refines the initial features and adjusts the local structure of the generated sequence data; Step 2-5: The fully connected layer maps the features output by the convolutional layer to the length and shape of the target time series to generate the final one-dimensional time series data.

4. The davit sample imbalance fault diagnosis method based on improved GAN network and multimodal feature fusion as claimed in claim 1, characterized in that: The multimodal feature fusion diagnosis model includes a two-dimensional image data diagnosis model and a one-dimensional time series data diagnosis model that are set in parallel.

5. The davit sample imbalance fault diagnosis method based on improved GAN network and multimodal feature fusion as claimed in claim 4 is characterized in that: The specific data processing process of the two-dimensional image data diagnosis model is as follows: Step 5-1, the convolution layer extracts local features of the two-dimensional image data, and by sharing weights, effectively reduces parameters and improves training efficiency; Step 5-2, the Swin Transformer Blocks module processes local features, calculates self-attention within the local window, reduces computational complexity, and calculates global features across windows by moving the window to efficiently capture global and local information; Step 5-3, the downsampling layer processes global and local information, reduces the size of input features, retains important features and reduces the amount of calculation; Step 5-4, the SimAM attention mechanism module enhances the importance of important features, suppresses irrelevant information, and enhances feature representation capabilities; Step 5-5, the batch normalization layer performs batch normalization on important features to accelerate convergence and stabilize training; Step 5-6: The fully connected layer integrates the features obtained in step 5-5 and maps them to the final output space to output the two-dimensional image data features.

6. The davit sample imbalance fault diagnosis method based on improved GAN network and multimodal feature fusion as claimed in claim 4, characterized in that: The process of data processing by the one-dimensional time series data diagnostic model is as follows: Step 5-1, the convolution layer extracts local data features from the one-dimensional time series data and identifies local trends of the data; Step 5-2, the batch normalization layer performs batch normalization on local data features; Step 5-3, the activation layer introduces nonlinearity to process the data features obtained in step 5-2; Step 5-4: The maximum pooling layer downsamples the data features, extracts important features, and enhances the robustness of the model; Step 5-5, the fully connected layer flattens and maps important features to a higher-dimensional feature space and integrates the extracted data features; Step 5-6: The Dropout layer trains the data features and randomly discards some neurons during the training process to prevent overfitting and improve the generalization ability of the model; Steps 5-7, the LSTM layer captures the long-term dependencies in the one-dimensional time series data, processes the contextual information in the one-dimensional time series data, and effectively retains important information through memory units and gating mechanisms; Step 5-8: The fully connected layer further integrates the data features output by the LSTM layer and maps them to the final output space, outputting the one-dimensional time series data features.

7. The davit sample imbalance fault diagnosis method based on improved GAN network and multimodal feature fusion as claimed in claim 1, characterized in that: In step 5, the method of data fusion is as follows: The obtained one-dimensional time series data features and two-dimensional image data features are cascaded in the row dimension. Through the cascade operation, the model can comprehensively utilize the information of the two data features to improve the overall performance.

8. The davit sample imbalance fault diagnosis method based on improved GAN network and multimodal feature fusion as claimed in claim 1, characterized in that: In step 6, a width learning system is used as a fault diagnosis classifier to perform fault classification. The specific classification method is as follows: Input layer: input the one-dimensional time series data and two-dimensional image data fused data features obtained through cascade operation; Feature mapping layer: randomly generate weights and map the input features to the new feature space; Augmentation node layer: add nonlinear nodes to enhance features; Output layer: Use the generalized inverse matrix to solve the output layer weights, output fault types, model evaluation and parameter optimization.

9. A davit sample imbalance fault diagnosis system based on improved GAN network and multimodal feature fusion, characterized by: The first module is configured to obtain the status data of the ship's davit electro-hydraulic system and perform noise reduction processing on it; The second module is configured to use the improved GAN network to perform balanced expansion on the unbalanced data after noise reduction; The third module is configured to divide the expanded data into a training set, a validation set, and a test set; The fourth module is configured to convert the one-dimensional time series data in the training set, the validation set and the test set into two-dimensional image data using a continuous wavelet transform method; The fifth module is configured to respectively input the one-dimensional time series data and the two-dimensional image data into the multimodal feature fusion diagnosis model, extract the data features of the one-dimensional time series data and the two-dimensional image data, and fuse the two extracted data features; The sixth module is configured to pass the fused data features into the classifier for fault classification.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to claim 1.

Images