Multi-modal marine bearing fault detection method based on acoustic denoising

Abstract

The invention discloses a multi-mode marine bearing fault detection method based on acoustic denoising. The method comprises the steps of obtaining marine bearing fault signal data; the marine bearing fault signal data comprises acoustic time sequence data and vibration time sequence data of corresponding bearing faults under different fault working condition types; the method comprises the following steps: constructing a multi-modal marine bearing fault detection model based on acoustic denoising, wherein the model comprises an input layer, a graph construction module, a feature extraction network, an acoustic noise reduction network based on a bidirectional long-short-term memory network, a nearest neighbor interpolation module, a feature fusion network and a classification prediction module; model training is carried out through marine bearing fault signal data to obtain an optimal detection model, and then multi-mode marine bearing fault detection based on acoustic denoising is achieved. According to the method, the problems of poor accuracy, robustness and early warning capability of fault detection under a complex ship working condition due to the fact that an existing method cannot effectively overcome strong noise environment interference of the ship and is insufficient in multi-source information fusion capability are solved.

Description

Technical Field

[0001] This invention relates to the field of marine bearing fault detection technology, and in particular to a multimodal marine bearing fault detection method based on acoustic denoising. Background Technology

[0002] Marine bearings, as core supporting components of ship propulsion and auxiliary systems, directly affect the safety, reliability, and economy of the vessel. Early bearing failures, if not detected in time, can easily escalate into serious damage, leading to equipment downtime or even catastrophic accidents, causing enormous economic losses and safety risks. Therefore, developing efficient and accurate marine bearing fault detection technology is of paramount importance.

[0003] Currently, vibration signal analysis is the mainstream method for monitoring the condition of marine bearings. However, the operating environment of ships is extremely complex and harsh: on the one hand, the strong broadband vibrations generated by the main engine, gearbox, propeller, etc., create significant background vibration interference, which can easily drown out early and subtle fault characteristics of bearings; on the other hand, the enclosed and complex structure of ship cabins results in exceptionally prominent acoustic noise (mechanical noise, fluid noise, and environmental noise), severely polluting the sound signals used for fault diagnosis. This makes detection methods relying solely on vibration signals less sensitive and have a high false alarm rate in the marine environment, making it difficult to reliably capture early faults. Although acoustic signals have unique sensitivity to certain types of bearing faults, the aforementioned severe environmental acoustic noise greatly reduces their signal-to-noise ratio, and traditional sound analysis methods are ineffective in strong noise backgrounds. Furthermore, vibration and acoustic signals reflect the equipment condition from different physical perspectives, and the information provided by a single mode is limited and easily affected by specific interference sources.

[0004] Therefore, there is an urgent need to develop a bearing fault detection method that can effectively overcome the interference of strong noise environment on ships and integrate multi-source information, so as to improve the accuracy, robustness and early warning capability of fault detection under complex ship operating conditions. Summary of the Invention

[0005] This invention provides a multimodal marine bearing fault detection method based on acoustic noise reduction to overcome the above-mentioned technical problems.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A multimodal marine bearing fault detection method based on acoustic denoising, specifically including the following steps: S1: Collect and acquire marine bearing fault signal data; Furthermore, the marine bearing fault signal data includes acoustic time series data and vibration time series data of the corresponding bearing faults under different fault conditions. S2: Construct a multimodal marine bearing fault detection model based on acoustic denoising, which includes an input layer, a graph construction module, a feature extraction network, an acoustic denoising network based on a bidirectional long short-term memory network, a nearest neighbor interpolation module, a feature fusion network, and a classification prediction module; The input layer is used to transmit marine bearing fault signal data to the graph construction module; The graph construction module is used to construct a two-dimensional Mel spectrum graph based on acoustic time series data, and to construct a recursive graph based on vibration time series data. The feature extraction network is used to extract two-dimensional frame-level features from the two-dimensional Mel spectrogram and the recursive graph, respectively; and the two-dimensional frame-level features are acoustic features and vibration features that include time-domain features and frequency-domain features. An acoustic noise reduction network based on a bidirectional long short-term memory network is used to perform noise filtering on acoustic and vibration features to obtain filtered acoustic and vibration features. The nearest neighbor interpolation module is used to perform interpolation operations on the filtered acoustic features to obtain acoustic modal features with an audio time step that matches the vibration frame length of the vibration modal features; and the vibration modal features are the features obtained by performing an up-dimensional operation on the filtered vibration features. The feature fusion network is used to perform feature fusion operations on acoustic modal features and filtered vibration features to obtain the channel attention fusion feature tensor; The classification and prediction module is used to fuse feature tensors based on channel attention to achieve fault classification and detection of marine bearings; S3: Using marine bearing fault signal data, train the constructed multimodal marine bearing fault detection model to obtain the optimal detection model, and implement multimodal marine bearing fault detection based on acoustic denoising according to the optimal detection model.

[0007] Furthermore, the feature extraction network described in S2 includes an input module, a two-dimensional convolutional block, a first batch normalization layer, a first nonlinear activation function layer, a regularization layer, a second batch normalization layer, and a two-dimensional pooling layer connected in sequence. The input module is used to input the two-dimensional Mel spectrogram and the recursive graph into the two-dimensional convolutional block, respectively; Two-dimensional convolutional blocks are used to perform two-dimensional convolution operations on two-dimensional Mel spectrograms and recursive graphs, respectively; The first batch normalization layer is used to perform normalization operations on the output of the two-dimensional convolutional block; The first nonlinear activation function layer is used to perform nonlinear activation operations on the first residual feature; Furthermore, the first residual feature is the feature obtained by performing residual connection on the output of the first batch normalization layer and the input of the input module; The regularization layer is used to perform regularization operations on the output of the first nonlinear activation function layer; The second batch normalization layer is used to perform normalization operations on the output of the regularization layer; A two-dimensional pooling layer is used to perform pooling operations on the second residual feature to obtain two-dimensional frame-level features; and the two-dimensional frame-level features are acoustic features that include temporal and frequency domain features. Vibration characteristics ;in, The temporal frame dimension and feature dimension representing acoustic features; The time frame dimension and feature dimension represent vibration characteristics; The second residual feature is the feature obtained by performing residual connection on the output of the second batch normalization layer and the input of the input module.

[0008] Furthermore, the acoustic noise reduction network based on the bidirectional long short-term memory network in S2 includes a multi-head attention module, a bidirectional long short-term memory network layer, a second nonlinear activation function layer, a noise reduction filter layer, an adaptive pooling layer, and a linear layer connected in sequence. The multi-head attention module is used to separately address acoustic features. Vibration characteristics Perform multi-head attention operations to obtain local attention features; Bidirectional long short-term memory network layers are used to perform temporal context modeling operations on local attention features to obtain global temporal key features; The second nonlinear activation function layer is used to perform nonlinear activation operations on key global temporal features; The noise reduction filtering layer is used to perform global time-domain average pooling operation on the output of the second nonlinear activation function layer to extract the spectral features of acoustic noise and vibration noise respectively to obtain noise vectors, and to filter the noise of acoustic features and vibration noise based on the noise vectors; The adaptive pooling layer is used to perform adaptive pooling operations on the output of the noise reduction filtering layer. Linear layers are used to reconstruct the dimensionality of the output of adaptive pooling layers to obtain filtered acoustic features. Acoustic characteristics of filters .

[0009] Furthermore, the expression for noise filtering of acoustic features and vibration noise based on the noise vector is as follows: , , , , In the formula: Indicates the time dimension; Represents a noise vector; Represents the filter matrix; Represents the basic weight matrix; This represents an enhanced feature representation; This represents the transpose of the filter matrix; This represents the learnable inverse transform matrix that reconstructs the filter activation space back to the spectral space; This indicates the characteristics after noise filtering.

[0010] Furthermore, the expression for the feature fusion network described in S2 is: , , , In the formula: express Activation function; Indicates average pool operation; Max pooling operation; This represents the features after normalizing the output of the nearest neighbor interpolation module, and ; Normalized features representing acoustic modal characteristics; Normalized features representing vibration modal characteristics; The channel attention weights represent modality-specific values ​​and ; express Channel attention weights; express Channel attention weights; Represents the weight matrix and and ; Represents cross-feature vectors; This represents the channel attention fusion feature tensor.

[0011] Furthermore, the classification prediction module in S2 fuses feature tensors based on channel attention to achieve marine bearing fault classification and detection, specifically including: The channel attention fusion feature tensor is applied along the time dimension using a pooling layer. With frequency dimension Perform global average pooling to aggregate and obtain the output vector. ; And the output vector The formula for obtaining it is , Based on the output vector through the fully connected layer Obtain the nonlinear combinations between channels and map them to a preset category space to obtain fully connected vectors; based on The activation function performs a linear transformation on the fully connected vector to obtain the intermediate feature vector. ; Based on the intermediate feature vector Obtain the category probability distribution to achieve marine bearing fault classification and detection; And the expression for obtaining the category probability distribution is: , In the formula: This indicates that the marine bearing fault signal data belongs to the first... The probability of a class; Indicates the total number of categories; Represents the intermediate feature vector Corresponding to the The probability of a class; intermediate feature vector Corresponding to the The probability of a class.

[0012] Furthermore, the method for obtaining the optimal detection model in S3 specifically includes: S31: Divide the marine bearing fault signal data into a training set, a validation set, and a test set according to a preset ratio, with the ratio of training set: validation set: test set = 8:1:1; The acoustic time series data and vibration time series data in the marine bearing fault signal data are used as feature data, and the fault category corresponding to the marine bearing fault is used as label data to obtain a sample dataset; the sample dataset is then randomly divided into a training set and a test set. S32: Train the multimodal marine bearing fault detection model based on acoustic denoising using the training and validation sets to obtain the trained fault detection model: The training set is used to determine the model weights and biases during model training; The validation set is used to select model hyperparameters during model training; and the model hyperparameters include at least the learning rate and the number of iterations. S33: Based on the constructed model loss function, confirm whether the output of the trained fault detection model has converged; If not, then based on the backpropagation method, adaptively adjust the parameter weights and biases of the trained fault detection model, and repeat step S32. If so, then based on the evaluation metrics, the test set is used to confirm whether the fault detection model trained at this time meets the preset evaluation threshold; and the evaluation metrics include at least the model accuracy. If so, the fault detection model trained at this time is the optimal detection model; if not, the parameter weights and biases of the trained fault detection model are adaptively adjusted based on the backpropagation method, and step S32 is repeated.

[0013] Furthermore, the model loss function described in S33 includes a discriminative persistent loss function. Loss function of co-distillation with entropy The construction formula is , , , , In the formula: Indicates single-mode loss; Representation of features and Each is flattened into a one-dimensional array; This represents the output after fusing sound modal features and vibration modal features. Indicates acoustic features or vibration characteristics The output after noise reduction; This represents the entropy-cooperative distillation loss; Indicates the modal loss ratio; Indicates the number of fault categories; Indicates adaptive weights and ; This represents the adaptive weights for the corresponding sound mode and vibration mode; This represents the temperature parameter used for the softening probability distribution; express Activation function; This represents the divergence used to measure the difference between probability distributions; These represent the loss ratios of the sound modes and the vibration modes, respectively. Indicates modal category and ; The output fault category after the acoustic and vibration characteristics are fused is: The probability distribution; Indicates acoustic features The output fault category after noise reduction is: The probability distribution; Vibration characteristics The output fault category after noise reduction is: The probability distribution.

[0014] This invention provides a multimodal marine bearing fault detection method based on acoustic denoising. By constructing a multimodal marine bearing fault detection model based on acoustic denoising, the following beneficial effects are achieved in bearing fault detection: 1. Effectively suppresses environmental noise interference: To address the issue that bearing operating sound signals are easily contaminated by strong background noise, an acoustic noise reduction network based on a bidirectional long short-term memory network was constructed. This network can significantly improve the signal-to-noise ratio of the sound signal and extract purer acoustic features that better reflect the changes in the internal state of the bearing. 2. Integrating complementary information to improve diagnostic robustness: This overcomes the limitations of single-modal (vibration or sound) information, such as its inherent bias and susceptibility to specific operating conditions (e.g., vibration signals are susceptible to the influence of mechanical transmission paths, and sound signals are susceptible to airborne noise). Through a feature fusion network, it can deeply integrate acoustically denoised sound and vibration signal features, fully leveraging their complementary advantages in characterizing different bearing fault types (e.g., inner ring, outer ring, and rolling element damage) and different development stages. 3. Improve the accuracy and reliability of fault diagnosis, enhance adaptability under complex operating conditions, and achieve earlier fault warnings: A. Improve the accuracy and reliability of fault diagnosis: The multimodal marine bearing fault detection model based on acoustic denoising can achieve more accurate identification and classification of early weak and complex faults in bearings, significantly reducing false alarm rate and false negative rate. B. Enhance adaptability under complex working conditions: Improve the diagnostic system's adaptability and diagnostic stability in complex industrial environments such as variable loads, variable speed operation, and strong background noise; C. Achieve earlier fault warning: By integrating the more sensitive acoustic features after noise reduction and reliable vibration features, subtle signs of bearing performance degradation can be detected earlier, providing a more timely and reliable basis for predictive maintenance. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the multimodal marine bearing fault detection method based on acoustic denoising according to the present invention; Figure 2 This is a schematic diagram of the structure of the multimodal marine bearing fault detection model based on acoustic denoising in this embodiment; Figure 3 This is a schematic diagram of the feature extraction network structure in this embodiment; Figure 4 This is a schematic diagram of the acoustic noise reduction network in this embodiment; Figure 5 This is a schematic diagram of the technical route of the feature fusion network in this embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This embodiment provides a multimodal marine bearing fault detection method based on acoustic denoising, such as... Figure 1 As shown, the specific steps include: S1: Collect and acquire marine bearing fault signal data; and the marine bearing fault signal data shall include at least the acoustic time series data and vibration time series data of the bearing faults under different fault conditions. Specifically, by using pre-set sound and vibration sensors and a pre-set fault simulation platform, fault simulation is performed on marine bearings, and bearing fault time series data under different fault conditions are simulated and collected; and different fault conditions include at least the normal bearing operation condition, the bearing inner ring fault operation condition, the bearing outer ring fault operation condition, the bearing ball fault operation condition, and a variety of similar fault conditions. S2: Construct a multimodal marine bearing fault detection model based on acoustic denoising, which includes an input layer, a graph construction module, a feature extraction network, an acoustic denoising network based on a bidirectional long short-term memory network, a nearest neighbor interpolation module, a feature fusion network, and a classification prediction module; The input layer is used to transmit marine bearing fault signal data to the graph construction module; The graph construction module is used to construct a two-dimensional Mel spectrum graph based on acoustic time series data, and to construct a recursive graph based on vibration time series data. Specifically, when traditionally using one-dimensional signals (such as acoustics, vibration, and temperature) as the raw data input for fault diagnosis, redundant feature components are generated due to their susceptibility to external environmental interference. In order to solve this problem, this embodiment converts acoustic and vibration signals, i.e., acoustic time series data and vibration time series data, into two-dimensional spectral images for feature extraction, thereby improving their data representation capabilities. S11: Given by Acoustic time series composed of samples By performing frequency transformation on the acoustic time series using the Mel scale in equation (1), a two-dimensional Mel spectrum is obtained. ,in, This represents the dimensional space of this variable. These represent the number of time frames and the number of frequency bins, respectively. (1) In the formula: Represents the linear frequency of a given acoustic time series data; Represents the Mel scale; Similarly, given by Vibration time series composed of samples The vibration signal is reconstructed in relative space using the graph reconstruction method of equations (2) and (3), which facilitates the analysis of periodicity, chaos and non-stationarity in the time series and obtains the recursive graph. And recorded as : (2) (3) In the formula: This represents a sample extracted from vibration time series data; This represents the trajectory dimension for reconstructing the relative spatial trajectory from vibration time series data; Indicates a time delay; Indicates the parameters for constructing the recursive graph; Represents the Heaviside function; Indicates the threshold parameter; They represent the corresponding sample points respectively. The value of the time series data; Each represents a sample point in a given time series data. The reconstruction space involves calculating the distance between each pair of points and generating a recursive graph based on the distance between the points. The feature extraction network is used to extract two-dimensional frame-level features from the two-dimensional Mel spectrogram and the recursive graph, respectively; and the two-dimensional frame-level features are acoustic features and vibration features that include time-domain features and frequency-domain features. Specifically, the feature extraction network includes an input module, a two-dimensional convolutional block, a first batch normalization layer, a first nonlinear activation function layer, a regularization layer, a second batch normalization layer, and a two-dimensional pooling layer connected in sequence. The input module is used to input the two-dimensional Mel spectrogram and the recursive graph into the two-dimensional convolutional block, respectively; Two-dimensional convolutional blocks are used to perform two-dimensional convolution operations on two-dimensional Mel spectrograms and recursive graphs, respectively; The first batch normalization layer is used to perform normalization operations on the output of the two-dimensional convolutional block; The first nonlinear activation function layer is used to perform nonlinear activation operations on the first residual feature; Furthermore, the first residual feature is the feature obtained by performing residual connection on the output of the first batch normalization layer and the input of the input module; The regularization layer is used to perform regularization operations on the output of the first nonlinear activation function layer; The second batch normalization layer is used to perform normalization operations on the output of the regularization layer; A two-dimensional pooling layer is used to perform pooling operations on the second residual feature to obtain two-dimensional frame-level features; and the two-dimensional frame-level features are acoustic features that include temporal and frequency domain features. Vibration characteristics That is, fine-grained features; among which, The temporal frame dimension and feature dimension representing acoustic features; The time frame dimension and feature dimension represent vibration characteristics; The second residual feature is the feature obtained by performing residual connection on the output of the second batch normalization layer and the input of the input module.

[0019] The feature extraction network in this embodiment has inherent advantages in processing two-dimensional image representations. Convolution operations not only efficiently capture fine-grained features in the image but also maintain the stability of the feature extraction process through their inherent translation invariance, thus continuously preserving key information in the sequence. This is achieved by analyzing the two-dimensional Mel-spectrum. With recursion graph Apply convolutional feature extraction operation Then, frame-level features are obtained. and ;in, and These represent the temporal frame dimension and the feature dimension, respectively. Specifically, the feature extraction network consists of seven sequentially connected modules, where a single module structure is as follows: Figure 3The diagram shows a series of interconnected 2D convolutional blocks (kernel size = 3, stride = 1, padding = 1), a first batch normalization layer, a first nonlinear activation function layer, a regularization layer (dropout), a second batch normalization layer, and a 2D pooling layer. The layers in the feature extraction network are connected via residual connections. In this embodiment, to address the limited vertical resolution of the spectrogram (input), module one uses a reduced vertical stride, while module two uses a rectangular kernel to enhance the capture of time-frequency features. Modules three through seven progressively expand the feature dimension to 47 through 2D convolutions, a process stabilized by batch normalization and nonlinear transformation. Finally, for subsequent acoustic denoising, the Bi-LSTM, a bidirectional long short-term memory-based acoustic denoising network, performs sequence modeling at the original temporal resolution, upsampling the features extracted by the CNN back to the original time-frequency grid dimension to obtain the features. and ; Acoustic noise reduction networks based on bidirectional long short-term memory networks, such as Figure 4 The diagram shows a multi-head attention module, a bidirectional long short-term memory network layer, a second non-linear activation function layer, a noise reduction filter layer, an adaptive pooling layer, and a linear layer connected in sequence. The multi-head attention module is used to separately address acoustic features. Vibration characteristics Perform multi-head attention operations to obtain local attention features; Bidirectional long short-term memory network layers are used to perform temporal context modeling operations on local attention features to obtain global temporal key features; The second nonlinear activation function layer is used to perform nonlinear activation operations on key global temporal features; This embodiment utilizes a CNN (Hypertext Extraction Network) to extract acoustic features. and Sub-features carrying multi-head attention are obtained through a multi-head attention layer, and key features are enhanced by attention weights while maintaining the tensor dimensionality structure. Then, the feature dimensions are transformed into a bidirectional LSTM layer. ,in This represents the size of the hidden layers in the bidirectional LSTM layer during processing. This indicates the size of the hidden layer for sound features during denoising. The bidirectional LSTM layer is essentially feature enhancement. Because attention weights are used, they primarily highlight locally relevant features, i.e., which frequency points are noise. Bidirectional LSTM, however, can highlight key global temporal features through temporal context modeling, i.e., which time points are prone to noise. It then gradually attenuates suspected noise units while strengthening normal sound features, allowing for better learning of noise units using the weight matrix for subsequent deep denoising. The dimensionality transformation performed in the bidirectional LSTM layer reduces the number of parameters in each processing step. Defined as more representative time-series features, while also avoiding overfitting due to excessive parameterization, dimensionality transformation is for ease of processing and setting. The dimensions of the bidirectional LSTM remain unchanged during input and output; the dimension transformation is merely an intermediate variable in the LSTM processing. Subsequently, a nonlinear activation layer is introduced. Enhance feature representation capability; finally, feed the processed features into the noise filtering layer for denoising. The noise reduction filtering layer is used to perform global time-domain average pooling operation on the output of the second nonlinear activation function layer to extract the spectral features of acoustic noise, retain the frequency domain distribution information of the noise, discard the instantaneous speech features to obtain the noise vector, and filter the noise of acoustic features based on the noise vector. The specific steps include: S100: Obtain the noise vector, its expression is: (4) In the formula: Indicates the time dimension; Represents a noise vector; S200: Use an MLP layer to map the noise vector to... dimensional filtering parameters, where The number of filters is represented, and a frequency-dependent filter matrix is ​​generated. This adaptive filter matrix is ​​then compared with the pre-trained base weight matrix. The final filter matrix is ​​obtained by adding them together. And the final filter matrix The expression is: (5) In the formula: Represents the filter matrix; Represents the basic weight matrix; This represents an enhanced feature representation; This represents the transpose of the filter matrix; S300: Obtaining input features in After distributing the weights on each filter, filtering is then performed in two stages: the first stage uses the dynamic filter... To achieve nonlinear interaction between frequency bands and enhance key frequency bands, the expression is as follows: (6) In the formula: This represents an enhanced feature representation; This represents the transpose of the filter matrix; The second stage is feature reconstruction, which involves using the inverse transform matrix to reconstruct the filtered features, learning to preserve speech components and suppress noise. Its expression is: (7) In the formula: This represents a learnable inverse transform matrix used to reconstruct the filter activation space back to the spectral space; This indicates the purity characteristics after noise filtering. The adaptive pooling layer is used to perform adaptive pooling operations on the output of the noise reduction filtering layer. In this embodiment, adaptive pooling is a pooling layer in PyTorch that is used to reduce the dimension of the feature vector output by the convolutional layer. It reduces overfitting, which occurs because the result is too sensitive to certain errors in the input. Max-pooling or mean-pooling can reduce noise. Linear layers are used to reconstruct the dimensionality of the output of adaptive pooling layers to obtain filtered acoustic features. Acoustic characteristics of filters In this embodiment, dimensional space reconstruction refers to transforming data from one dimensional space to another, that is, changing the dimension of the input data through a linear layer using a preset linear transformation matrix. In this embodiment, noise-contaminated single-modal data severely hinders cross-modal dynamic learning, especially significantly impacting acoustic modes susceptible to environmental interference. Such artifacts propagate useless or misleading features to the fusion network, and amplifying these features during fusion may mask key fault characteristics. Although existing technologies focus on prioritizing useful information through complex fusion strategies, residual noise remains difficult to eliminate after optimization. To address this limitation, this embodiment adaptively adjusts the denoising process by reconstructing features and frequency band interactions to filter temporal noise elements, while preserving discriminative embeddings to achieve environmentally adaptive denoising. Furthermore, by combining attention mechanisms and contextual temporal modeling, noise components are effectively separated while maintaining frequency resolution, resulting in the final output... The feature tensor serves as a robust acoustic representation for subsequent processing.

[0020] The nearest neighbor interpolation module is used to perform interpolation operations on the filtered acoustic features to obtain acoustic modal features with an audio time step that matches the vibration frame length of the vibration modal features; and the vibration modal features are the features obtained by performing dimensionality-up operations on the vibration features. Specifically, since the spectrograms of different modalities have different dimensions, in this embodiment, the representations of vibration features and acoustic features share the same channel dimension but have different time steps. Therefore, in order to finely fuse the feature vectors in the time dimension and subsequently match the time steps and number to restore the time dimension of the feature vectors, this embodiment uses cross-time step nearest neighbor interpolation to match the audio time step with the vibration frame length before feature fusion, thereby obtaining the final purified features. First, the vibration features extracted by the convolutional neural network are subjected to dimensionality upscaling. Purification Secondly, use the function. The pure features are obtained by recovering the time dimension of the acoustic feature vector. Finally, before feature fusion, nearest neighbor interpolation across time steps is used. Audio time step With vibration frame length Matching is performed to obtain the final purified features with consistent channel and time step dimensions. and The acoustic or vibration mode is expressed as follows: (8) (9) (10) The feature fusion network is used to perform feature fusion operations on acoustic modal features and vibration features to obtain the channel attention fusion feature tensor; While convolutional neural networks (CNNs), or feature extraction networks, excel at local feature extraction, their pixel-level focus limits the effectiveness of multimodal fusion, often leading to insufficient fusion and impacting classification performance. Simple weighted fusion strategies further exacerbate this problem by neglecting feature importance, resulting in poor fusion outcomes. To address these limitations, this embodiment designs a novel feature fusion network (such as...). Figure 5 As shown in the figure, this feature fusion network achieves channel-dimensional fusion through attention-based fusion operations, thereby improving the discriminative feature representation ability and overall classification robustness. Specifically, the method for feature fusion networks to perform feature fusion operations on acoustic modal features and vibration features is as follows: First, a global layer normalization operation is performed on the acoustic modal features and vibration modal features to match the frequency dimensions of the two modal features, thereby obtaining the normalized features. This is to facilitate subsequent channel attention cross-fusion of the two modal features; for two independent normalized features and Two sets of shapes are generated within the channel attention module using global max pooling and average pooling. The channel vectors; these vectors are processed through a parameter-shared MLP module, and then summed and combined sequentially. function Activation operation to generate channel attention weights Its expression is: (11) In the formula: Indicates average pool operation; Max pooling operation; By applying this mechanism, two independent normalized features are obtained respectively. and Obtain modality-specific channel weights and Subsequently, these weights are multiplied by themselves to generate two shapes of the same value. weight matrix and Next, these weight matrices are respectively compared with... and Multiply and apply Function processing to obtain cross feature vectors Its expression is: (12) Finally, the two cross-modal feature vectors are added together to obtain the channel attention fusion feature. That is, the channel attention fusion feature tensor, its expression is: (13) The classification and prediction module is used to fuse feature tensors based on channel attention to achieve fault classification and detection of marine bearings; Specifically, the method for classifying and detecting marine bearing faults in this embodiment is as follows: The channel attention fusion feature tensor is applied along the time dimension using a pooling layer. With frequency dimension Perform global average pooling to aggregate and obtain the output vector. ; And the output vector The formula for obtaining it is (14) Based on the output vector through the fully connected layer Obtain the nonlinear combinations between channels and map them to a preset category space to obtain fully connected vectors; based on The activation function performs a linear transformation on the fully connected vector to obtain the intermediate feature vector. ; Based on the intermediate feature vector Obtain the category probability distribution to achieve marine bearing fault classification and detection; And the expression for obtaining the category probability distribution is: (15) In the formula: This indicates that the marine bearing fault signal data belongs to the first... The probability of a class; Indicates the total number of categories; Represents the intermediate feature vector Corresponding to the The probability of a class; intermediate feature vector Corresponding to the The probability of a class; S3: Train the constructed multimodal marine bearing fault detection model using marine bearing fault signal data, obtain the optimal detection model, and implement multimodal marine bearing fault detection based on acoustic denoising according to the optimal detection model. Specifically, the method for obtaining the optimal detection model is as follows: S31: Divide the marine bearing fault signal data into a training set, a validation set, and a test set according to a preset ratio, with the ratio of training set: validation set: test set = 8:1:1; The acoustic time series data and vibration time series data in the marine bearing fault signal data are used as feature data, and the fault category corresponding to the marine bearing fault is used as label data to obtain a sample dataset; the sample dataset is then randomly divided into a training set and a test set. S32: Train the multimodal marine bearing fault detection model based on acoustic denoising using the training and validation sets to obtain the trained fault detection model: The training set is used to determine the model weights and biases during model training; The validation set is used to select model hyperparameters during model training; and the model hyperparameters include at least the learning rate and the number of iterations. S33: Based on the constructed model loss function, confirm whether the output of the trained fault detection model has converged; To improve model efficiency and enhance regularization and generalization capabilities, this embodiment introduces a discriminative persistent loss function and an entropy co-distillation loss function to effectively suppress overfitting, smooth decision boundaries, and reduce training variance, thereby improving the model's generalization performance on the test set. At the same time, it achieves efficient model compression and training processes, saves computing resources, and optimizes the parameter configuration of the model network. The discriminative persistence loss function in this embodiment The core lies in flexibly processing the enhanced output, i.e., filtering acoustic features. Acoustic characteristics of filters The loss for backpropagation is determined based on the loss values ​​before and after denoising of the single-modal information. This allows the model to focus on more important branches during training, thereby improving classification or regression performance and enhancing adaptability to specific tasks. Furthermore, the discriminative persistence loss function is the same as the single-mode loss function. The expression is (16) In the formula: Indicates pre-enhanced features Acoustic characteristics or vibration characteristics and post-enhancement features That is, the acoustic characteristics of the filter Or filter acoustic features Each is flattened into a one-dimensional array; finally, in the two cross-entropy losses... The one that is farther away from the true label is used for backpropagation, while the one that is smaller is separated to avoid updating its gradient. Entropy Cooperative Distillation Loss Function Based on single-mode loss function Calculating the loss of different modalities and adjusting the model training process through adaptive weights can improve the learning efficiency of multimodal data. This training process dynamically adjusts the information gain and loss between different modalities, makes full use of the complementarity between modalities, avoids the model from becoming overly dependent on a single modality, and thus enhances the robustness and generalization ability of the model. Adaptive weights The expression is: (17) (18) in Indicates the modal loss ratio; These represent the loss ratios of the sound modes and the vibration modes, respectively. Entropy Co-distillation Loss The calculation is shown in the following formula: (19) In the formula: This represents the divergence used to measure the difference between two probability distributions; This indicates that the output is converted into a probability distribution. function; This represents the temperature parameter used to soften the probability distribution, making the modal interaction loss more effective; Indicates modal category and ; The output fault category after the acoustic and vibration characteristics are fused is: The probability distribution; Indicates acoustic features The output fault category after noise reduction is: The probability distribution; Vibration characteristics The output fault category after noise reduction is: The probability distribution; Indicates the number of fault categories; If not, then based on the backpropagation method, adaptively adjust the parameter weights and biases of the trained fault detection model, and repeat step S32. If so, then based on the evaluation metrics, the test set is used to confirm whether the fault detection model trained at this time meets the preset evaluation threshold; and the evaluation metrics include at least the model accuracy. If so, the fault detection model trained at this time is the optimal detection model; if not, the parameter weights and biases of the trained fault detection model are adaptively adjusted based on the backpropagation method, and step S32 is repeated.

[0021] The overall technical roadmap architecture of the technical framework proposed in this embodiment is as follows: Figure 2 The process includes: first, converting vibration and acoustic signals into recursive graphs and two-dimensional Mel-spectrum graphs, respectively, to amplify fault-related structural patterns; second, through... Figure 3 The feature extraction network constructed as shown extracts modal discriminative features between vibration and acoustic signals; then, through... Figure 4 The acoustic denoising network shown is based on a bidirectional long short-term memory network. It refines acoustic and vibration features using a multi-head attention mechanism combined with bidirectional LSTM layers, suppressing noise-contaminated time segments while preserving key fault features. Finally, the acquired modal features are processed... Figure 5 The attention mechanism shown integrates complementary spatiotemporal features to ensure accurate identification of fault characteristics.

[0022] Compared with the prior art, the beneficial effects of the method described in this embodiment are as follows: 1. Significantly Improved Diagnostic Accuracy and Reliability: By constructing an acoustic noise reduction network based on a bidirectional long short-term memory network, the interference of environmental noise on bearing sound signals is effectively filtered out, significantly improving the signal-to-noise ratio (SNR) of the sound signals and extracting purer acoustic features that better reflect the true internal state of the bearing. Innovatively, high-quality denoised acoustic features are integrated with high SNR vibration features, fully utilizing the complementary characteristics of sound signals being sensitive to surface defects and vibration signals being sensitive to internal impact forces, thus constructing a richer and more comprehensive bearing condition characterization. 2. Construct a fault diagnosis model based on fusion feature training, namely, a multimodal marine bearing fault detection model based on acoustic denoising, which can more accurately distinguish different fault types (such as inner ring, outer ring, etc.) and different fault severity, significantly reducing the misdiagnosis rate and missed diagnosis rate. 3. Enhanced system robustness and environmental adaptability: Overcomes the vulnerability of single-modal diagnostics. When the quality of a certain modal signal degrades due to specific operating conditions (such as strong background noise interference), another modality and its fused features can provide redundant and supplementary information, ensuring the stability of the diagnostic results. Particularly suitable for complex and harsh industrial environments (such as high-noise workshops, equipment locations where vibration sensors are difficult to install). The acoustic denoising module effectively addresses noise challenges, and the combination of non-contact sound acquisition and vibration sensing provides a more flexible deployment solution. 4. Enhanced detection capability for subtle and early-stage faults: The denoised acoustic signal retains characteristic information of early, subtle bearing damage that might be obscured in the original noisy signal. Vibration signals are sensitive to impact-related faults, while acoustic signals are more sensitive to continuous faults such as friction and squealing. The fusion of these two technologies allows the system to detect early signs of bearing performance degradation more accurately and sensitively, facilitating predictive maintenance. This provides a valuable time window to avoid sudden equipment failure; 5. Reduced deployment and maintenance costs, improved economic efficiency: Leveraging the advantages of non-contact sound acquisition (microphone), it provides an effective supplementary or alternative diagnostic tool in areas where vibration sensors are difficult to install (such as high-speed rotating shafts or enclosed structures) or in scenarios requiring reduced wiring complexity. Improved diagnostic accuracy and early warning capabilities enable more precise maintenance planning, avoiding unnecessary downtime and catastrophic equipment damage, thereby significantly reducing the total lifecycle maintenance cost (LCC); reduced unplanned downtime due to misdiagnosis, and improved production efficiency and overall equipment effectiveness (OEE). 6. Provides a more comprehensive and in-depth condition assessment: Multimodal fusion is not only used for fault classification, but the fusion features themselves contain richer information on bearing health status, which helps to more comprehensively assess the overall health status of the bearing and predict the remaining service life (RUL).

[0023] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal bearing fault detection method for marine shafts based on acoustic denoising, characterized by, Specifically comprising the following steps: S1: collect and acquire marine bearing fault signal data; And the marine bearing fault signal data includes acoustic time series data and vibration time series data corresponding to bearing faults under different fault working condition categories; S2: construct an acoustic denoising-based multi-modal marine bearing fault detection model, which includes an input layer, a graph construction module, a feature extraction network, an acoustic denoising network based on a bidirectional long short-term memory network, a nearest neighbor interpolation module, a feature fusion network, and a classification prediction module; The input layer is used to transmit the marine bearing fault signal data to the graph construction module; The graph construction module is used to construct a two-dimensional Mel spectrum graph according to the acoustic time series data and a recurrent graph according to the vibration time series data; The feature extraction network is used to extract two-dimensional frame-level features in the two-dimensional Mel spectrum graph and the recurrent graph, respectively; and the two-dimensional frame-level features are acoustic features and vibration features containing time domain features and frequency domain features; The acoustic denoising network based on the bidirectional long short-term memory network is used to filter noise from the acoustic features and the vibration features to obtain filtered acoustic features and filtered vibration features; The nearest neighbor interpolation module is used to interpolate the filtered acoustic features to obtain acoustic modal features with an audio time step length matching the vibration frame length of the vibration modal features; and the vibration modal features are obtained by dimensionality increasing operation on the filtered vibration features; The feature fusion network is used to perform feature fusion operation on the acoustic modal features and the filtered vibration features to obtain a channel attention fusion feature tensor; The classification prediction module is used to realize marine bearing fault classification detection according to the channel attention fusion feature tensor; S3: train the constructed multi-modal marine bearing fault detection model using marine bearing fault signal data to obtain an optimal detection model, and realize acoustic denoising-based multi-modal marine bearing fault detection based on the optimal detection model.

2. The method of claim 1, wherein, The feature extraction network in S2 includes an input module, a two-dimensional convolution block, a first batch normalization layer, a first nonlinear activation function layer, a regularization layer, a second batch normalization layer, and a two-dimensional pooling layer connected in sequence; The input module is used to input the two-dimensional Mel spectrum graph and the recurrent graph into the two-dimensional convolution block; The two-dimensional convolution block is used to perform two-dimensional convolution operation on the two-dimensional Mel spectrum graph and the recurrent graph, respectively; The first batch normalization layer is used to perform normalization operation on the output of the two-dimensional convolution block; The first nonlinear activation function layer is used to perform nonlinear activation operation on the first residual feature; And the first residual feature is the feature obtained by performing residual connection on the output of the first batch normalization layer and the input of the input module; The regularization layer is used to perform regularization operation on the output of the first nonlinear activation function layer; The second batch normalization layer is used to perform normalization operation on the output of the regularization layer; The two-dimensional pooling layer is configured to perform a pooling operation on the second residual feature to obtain a two-dimensional frame-level feature; and the two-dimensional frame-level feature is an acoustic feature comprising a time domain feature and a frequency domain feature with the vibration feature ; wherein, denotes a time frame dimension and a feature dimension of the acoustic feature; denotes a time frame dimension and a feature dimension of the vibration feature The second residual feature is the feature obtained by performing residual connection on the output of the second batch normalization layer and the input of the input module.

3. The method of claim 1, wherein, The acoustic denoising network in S2 based on a bidirectional long short-term memory network comprises, in sequence, a multi-head attention module, a bidirectional long short-term memory network layer, a second nonlinear activation function layer, a denoising filter layer, an adaptive pooling layer, and a linear layer; The multi-head attention module is used to respectively perform a multi-head attention operation on the acoustic features and the vibration features to obtain local attention features; The bidirectional long short-term memory network layer is configured to perform a timing context modeling operation on the local attention features to obtain global timing key features; The second nonlinear activation function layer is configured to perform a nonlinear activation operation on the global timing key features; The denoising filter layer is configured to perform a global time domain average pooling operation on the output of the second nonlinear activation function layer to extract spectral features of acoustic noise and vibration noise to obtain a noise vector, and to perform noise filtering on the acoustic features and the vibration noise according to the noise vector; The adaptive pooling layer is configured to perform an adaptive pooling operation on the output of the denoising filter layer; The linear layer is used to reconstruct the dimension space of the output of the adaptive pooling layer to obtain the filtered acoustic feature The filtered acoustic feature .

4. The multi-modal bearing fault detection method based on acoustic denoising for marine shafts according to claim 3, characterized in that, The expression for performing noise filtering on the acoustic features and the vibration noise according to the noise vector is wherein: denotes the time dimension; denotes the noise vector; denotes the filter matrix; denotes the base weight matrix; denotes the enhanced feature representation; denotes the transpose of the filter matrix; denotes the learnable inverse transform matrix that reconstructs the filter activation space back to the spectral space; denotes the noise filtered features.

5. The multi-modal bearing fault detection method based on acoustic denoising for marine shafts according to claim 4, characterized in that, The expression of the feature fusion network in S2 is wherein: represents an activation function; represents an average pooling operation; represents a max pooling operation; represents features after performing a normalization operation on the output of the nearest neighbor interpolation module and ; represents normalized features of acoustic modality features; represents normalized features of vibration modality features; represents channel attention weights with respect to modality specificity and ; represents channel attention weights of represents channel attention weights of represents a weight matrix and with ; represents across feature vectors; represents channel attention fusion feature tensors.

6. The multi-modal bearing fault detection method based on acoustic denoising for marine shafts according to claim 5, characterized in that, The classification prediction module in S2 obtains the channel attention fusion feature tensor, The method for ship bearing fault classification detection specifically comprises: pooling layer to fuse the channel attention along the time dimension with the frequency dimension global average pooling operation to aggregate and obtain an output vector ; and the output vector The acquisition formula is by a fully connected layer according to the output vector obtaining a fully connected vector by acquiring a nonlinear combination between channels and mapping to a preset category space, based on obtaining an intermediate feature vector by performing a linear transformation operation on the fully connected vector by an activation function ; According to the intermediate feature vector Obtaining a class probability distribution to implement marine bearing fault classification detection; The expression for obtaining the class probability distribution is wherein: denotes the probability that the ship bearing failure signal data belongs to the class; denotes the total number of classes; denotes the intermediate feature vector corresponding to the probability that the ship bearing failure signal data belongs to the class; intermediate feature vector corresponding to the probability that the ship bearing failure signal data belongs to the class.

7. The multi-modal bearing fault detection method based on acoustic denoising for marine shafts according to claim 6, characterized in that, The method for obtaining the optimal detection model in S3 specifically comprises: S31: The ship bearing fault signal data is divided into a training set, a validation set, and a test set according to a preset ratio, and the training set: validation set: test set = 8:1:1; The acoustic time series data and the vibration time series data in the ship bearing fault signal data are taken as feature data, and the fault class corresponding to the ship bearing fault is taken as label data to obtain a sample data set; and the sample data set is randomly divided into a training set and a test set; S32: The acoustic denoising-based multi-modal ship bearing fault detection model is trained according to the training set and the validation set to obtain a trained fault detection model: The training set is used to determine the model weight and bias in the model training process; The validation set is used to select model hyperparameters in the model training process; and the model hyperparameters at least include a learning rate and an iteration number; S33: Based on the constructed model loss function, it is determined whether the output of the trained fault detection model converges; If not, the parameter weight and bias of the trained fault detection model are adaptively adjusted based on the back propagation method, and step S32 is repeatedly executed; If yes, it is determined whether the trained fault detection model at this time meets a preset evaluation threshold value through the test set based on an evaluation index; and the evaluation index at least includes a model accuracy rate; If yes, the trained fault detection model at this time is the optimal detection model; if not, the parameter weight and bias of the trained fault detection model are adaptively adjusted based on the back propagation method, and step S32 is repeatedly executed.

8. The multi-modal bearing fault detection method based on acoustic denoising for marine shafts according to claim 7, characterized in that, The model loss function in S33 includes a discriminative persistent loss function co-distillation loss function with entropy The construction formula is In the formula: Indicates single-mode loss; Representation of features and Each is flattened into a one-dimensional array; This represents the output after fusing sound modal features and vibration modal features. Indicates acoustic features Vibration characteristics The output after noise reduction; This represents the entropy-cooperative distillation loss; Indicates the modal loss ratio; Indicates the number of fault categories; Indicates adaptive weights and ; This represents the adaptive weights for the corresponding sound mode and vibration mode; This represents the temperature parameter used for the softening probability distribution; express Activation function; This represents the divergence used to measure the difference between probability distributions; These represent the loss ratios of the sound modes and the vibration modes, respectively. Indicates modal category and ; The output fault category after the acoustic and vibration characteristics are fused is: The probability distribution; Indicates acoustic features The output fault category after noise reduction is: The probability distribution; Vibration characteristics The output fault category after noise reduction is: The probability distribution.

Images