Automobile motor abnormal sound detection method and device based on multi-modal deep learning

Abstract

The invention relates to the technical field of industrial intelligent diagnosis and quality control, in particular to an automobile motor abnormal sound detection method and device based on multi-mode deep learning, and the method comprises the steps: extracting the characteristics of a vibration signal through a one-dimensional convolution layer, and introducing a deformable convolution to capture transient impact and a non-stationary mode, thereby achieving the detection of abnormal sound of an automobile motor. Secondly, modeling long-time dependence and dynamic structure characteristics of the vibration signals through a Transform encoder containing causal convolution; extracting features of the acoustic signals through a two-dimensional convolution layer, and modeling cross-time-frequency global semantics and multi-scale features of the acoustic signals in combination with a Swinin-Transform encoder and a cavity space pyramid pooling module; and finally, sound-vibration structured association is established through the graph neural network of the adaptive cross-modal fusion network, so that the detection accuracy and robustness of the abnormal sound of the automobile motor are improved.

Description

Technical Field

[0001] This invention relates to the field of industrial intelligent diagnostics and quality control technology, and in particular to a method and device for detecting abnormal noise in automotive motors based on multimodal deep learning. Background Technology

[0002] With the increasing demand for monitoring the operating status of machinery and equipment and identifying abnormal sounds in the industrial sector, the applicability of single-method abnormal sound detection based on acoustic microphones in complex environments is somewhat limited. Microphones collect sound pressure signals in the air, directly reflecting the sound characteristics ultimately transmitted to the human ear, offering a unique advantage in evaluating customer subjectivity. However, this method is susceptible to interference from environmental noise, spatial reflections, and the location of the sound source. In noisy production line or workshop environments, even with soundproof enclosures or anechoic chambers, it is difficult to completely eliminate background noise and reverberation effects, leading to unstable detection results.

[0003] In contrast, vibration acceleration sensors can be directly attached to the surface of equipment structures to collect vibration acceleration signals from the mechanical body. These signals do not travel through air and are virtually unaffected by external noise, thus more accurately reflecting the true operating state of the equipment. For example, during long-term use, motors may experience bearing wear, rotor imbalance, or abnormal gear meshing. These faults directly manifest as changes in vibration modes on the structure, and acceleration signals can effectively capture these characteristics. However, while detection schemes based on a single vibration sensor offer high signal-to-noise ratios and stability, they cannot directly reflect the radiation of airborne sound, thus lacking sufficient correlation with subjective perception.

[0004] This problem is particularly prominent in the motors and transmission systems of new energy vehicles. These systems are prone to issues such as bearing wear, rotor imbalance, air gap eccentricity, and abnormal gear meshing during long-term operation, often manifesting as perceptible abnormal noises or complex structural vibration mode changes. Traditional microphone-based detection methods can effectively correspond to customers' subjective perception of the vehicle's NVH (Noise, Vibration, and Harshness), but their reliability is insufficient in production line environments. While traditional vibration acceleration sensor-based detection methods are stable, they cannot fully reflect the final auditory experience. Single-modal methods struggle to balance "audible performance" and "source diagnosis," exhibiting inherent limitations.

[0005] Furthermore, the handcrafted features extracted by commonly used time-frequency methods such as Fourier transform, wavelet analysis, and empirical mode decomposition in existing technologies lack generalization ability when facing non-stationary signals, multiple operating conditions, and complex background noise, making it difficult to meet the needs of large-scale industrial quality inspection. Although deep learning methods can automatically learn more representative features, most existing works are still limited to single-modal modeling or simply splicing acoustic and vibration features, lacking cross-modal adaptive fusion and interactive modeling mechanisms, thus failing to fully leverage the complementary advantages of acoustic and vibration signals.

[0006] Therefore, how to effectively integrate the two modalities of microphone and vibration acceleration sensor to ensure sensitive capture of customers' NVH experience while achieving accurate detection of potential equipment anomalies has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0007] Therefore, the technical problem to be solved by the present invention is to overcome the lack of an adaptive fusion and interactive modeling mechanism for acoustic and vibration signals across modes in the prior art, which makes it difficult to accurately detect abnormalities in automobile motors.

[0008] To address the aforementioned technical problems, this invention provides a method for detecting abnormal noise in automotive motors based on multimodal deep learning, comprising: Collect vibration and acoustic signals during the operation of the car motor; The vibration signal is input into a one-dimensional convolutional layer feature extraction module to obtain the initial features of the vibration signal; the initial features of the vibration signal are segmented and embedded into a token sequence; the token sequence is input into a Transformer encoder to obtain the global feature vector of the vibration signal. The acoustic signal is input into the two-dimensional convolutional feature extraction module to obtain the initial features of the acoustic signal; the initial features of the acoustic signal are then passed through the Swin-Transformer encoder and the hollow spatial pyramid pooling module to obtain the global feature vector of the acoustic signal. The global feature vectors of the vibration signal and the acoustic signal are input into an adaptive cross-modal fusion network to obtain the detection and classification results.

[0009] Preferably, before inputting the vibration signal into the one-dimensional convolutional layer feature extraction module, the vibration signal is preprocessed, including: The original vibration signal is approximated as a low-order polynomial trend using least squares, and then subtracted from the original vibration signal to obtain the detrended vibration signal. The de-stressed vibration signal is passed through a bandpass filter to obtain the denoised vibration signal; The denoised vibration signal is normalized and then slicing is performed according to a fixed window length and a fixed step size to obtain the preprocessed vibration signal.

[0010] Preferably, before inputting the acoustic signal into the two-dimensional convolutional feature extraction module, the acoustic signal is preprocessed, including: The original acoustic signal is bandpass filtered to obtain the denoised acoustic signal; The denoised acoustic signal is normalized and segmented to obtain the normalized segmented acoustic signal; The normalized segmented acoustic signal is subjected to Stockwell transform to obtain a two-dimensional time-frequency spectrum, which serves as the preprocessed acoustic signal.

[0011] Preferably, the one-dimensional convolutional layer feature extraction module includes: The first one-dimensional convolutional block includes a first one-dimensional convolutional layer, a batch normalization layer, a ReLU activation function, and a max pooling layer; The second one-dimensional convolutional block includes the first one-dimensional deformable convolutional layer; The third one-dimensional convolutional block includes the second one-dimensional convolutional layer and the second one-dimensional deformable convolutional layer.

[0012] Preferably, the Transformer encoder includes multiple Transformer encoder sub-modules, each Transformer encoder sub-module including a causal convolutional embedding layer, a multi-head self-attention layer and a feedforward network layer connected in sequence, and the layers are connected by residual connections and layer normalization mechanisms.

[0013] Preferably, the multi-head self-attention layer is improved by: A frequency-sensitive weighting matrix based on the spectral energy of the vibration signal is introduced into the multi-head self-attention layer, expressed by the formula: ; in, The attention mechanism of a multi-head self-attention layer. , and These are the query matrix, key matrix, and value matrix obtained from the input feature mapping, respectively. For the softmax function, It is a frequency-sensitive weighted matrix. is the feature dimension of the attention head in the multi-head self-attention layer.

[0014] Preferably, the Swin-Transformer encoder includes multiple Swin-Transformer encoder submodules, each of which includes a window multi-head self-attention layer and a feedforward network layer.

[0015] Preferably, the multi-head self-attention mechanism of the window is improved, including: The multi-head self-attention mechanism for windows introduces a relative position bias matrix and a sliding window mechanism, including: The input features of the multi-head self-attention mechanism are divided into multiple partially overlapping windows using a sliding window mechanism, and multi-head self-attention calculation is performed independently within each window to obtain the attention features of that window. In multi-head self-attention computation, each single-head attention computation introduces a relative position bias matrix, expressed by the formula: ; in, This refers to the attention mechanism of a multi-head self-attention mechanism for windows. , and These are the query matrix, key matrix, and value matrix obtained from the input feature mapping, respectively. For the softmax function, This is the relative position offset matrix. For the feature dimensions of the attention head in the multi-head self-attention mechanism of the window; The output features of the multi-head self-attention mechanism of windows are obtained by concatenating the attention features of all windows.

[0016] Preferably, the adaptive cross-modal fusion network includes a feature adaptive fusion layer, a graph neural fusion layer, and a decision layer connected in sequence; The feature adaptive fusion layer takes the global feature vectors of the vibration signal and the global feature vectors of the acoustic signal as input, and includes a linear mapping layer, a batch normalization layer and a ReLU activation function layer. It outputs the normalized global feature vectors of the vibration signal and the global feature vectors of the acoustic signal. The graph neural network fusion layer takes the normalized global feature vector of the vibration signal and the global feature vector of the acoustic signal as input, and includes two layers of graph neural network and a linear mapping layer, and outputs fused features; The decision layer takes fused features as input and includes a batch normalization layer, a fully connected layer, and a Softmax classification layer, and outputs detection and classification results.

[0017] The present invention also provides a vehicle motor noise detection device based on multimodal deep learning, comprising: The acquisition module is used to acquire vibration and acoustic signals during the operation of the car's motor. The vibration signal feature extraction module is used to input the vibration signal into the one-dimensional convolutional layer feature extraction module to obtain the initial features of the vibration signal; to segment and embed the initial features of the vibration signal into a token sequence; and to input the token sequence into the Transformer encoder to obtain the global feature vector of the vibration signal. The acoustic signal feature extraction module is used to input the acoustic signal into the two-dimensional convolutional feature extraction module to obtain the initial features of the acoustic signal; the initial features of the acoustic signal are then passed through the Swin-Transformer encoder and the hollow spatial pyramid pooling module to obtain the global feature vector of the acoustic signal. The fusion classification module is used to input the global feature vectors of vibration signals and acoustic signals into the adaptive cross-modal fusion network to obtain the detection and classification results.

[0018] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This invention discloses a method for detecting abnormal noises in automotive motors based on multimodal deep learning. It fully integrates the advantages of vibration and acoustic signals. In the vibration signal processing branch, features are extracted using a one-dimensional convolutional layer, incorporating deformable convolutions to capture transient impacts and non-stationary modes. A Transformer encoder with causal convolutions is then used to model the long-term dependencies and dynamic structural features of the vibration signal. In the acoustic signal processing branch, features are extracted using a two-dimensional convolutional layer, combined with a Swin-Transformer encoder and a hollow spatial pyramid pooling module to model the global semantics and multi-scale features of the acoustic signal across time and frequency. Finally, an adaptive cross-modal fusion network graph neural network is used to establish a structured correlation between sound and vibration, improving the accuracy of detecting abnormal noises in automotive motors. This invention utilizes multi-level, cross-modal information integration, preserving both the local energy texture and complex frequency features of multimodal signals, while also modeling long-term dependencies and dynamic relationships between modes, significantly improving the accuracy and robustness of detecting abnormal noises in automotive motors. Attached Figure Description

[0019] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart of a method for detecting abnormal noise in automotive motors based on multimodal deep learning, according to the present invention. Figure 2 This is a schematic diagram of the vibration signal; Figure 3 This is a schematic diagram of converting an acoustic signal into a two-dimensional time-frequency spectrum using the Stockwell transform; Figure 4 This is a structural diagram of the motor anomaly detection model. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0021] Reference Figure 1 As shown, this invention provides a method for detecting abnormal noise in automotive motors based on multimodal deep learning, including: S1: Collect vibration and acoustic signals during the operation of the car motor.

[0022] The input signals of this invention include structural vibration signals acquired by a vibration acceleration sensor and acoustic signals acquired by a microphone. Both types of signals are acquired at a sampling rate of 51200 Hz to ensure the consistency and synchronization of cross-modal characteristics over time.

[0023] In this embodiment, vibration acceleration sensors are placed at key locations on the motor housing to collect structural vibration signals. The collected vibration signals... like Figure 2 As shown, OK samples represent the normal state, and NG samples represent the abnormal state.

[0024] Preferably, the preprocessing of the acquired vibration signal in this embodiment includes: S11-1: The original vibration signal is approximated as a low-order polynomial trend using least squares, and then subtracted from the original vibration signal to obtain the detrended vibration signal.

[0025] To eliminate baseline offset caused by sensor temperature drift or installation errors, it is necessary to first process the original vibration signal. Trend processing is performed. This embodiment uses least squares fitting to approximate the original vibration signal as a low-order polynomial trend. The formula is expressed as: ; Where n is the discrete-time index, representing the nth sampling point of the vibration signal. The polynomial coefficients are estimated using the least squares method. Let be the order of the polynomial.

[0026] Trend of low-order polynomials From the original vibration signal Subtracting from the middle, we obtain the detrended vibration signal. The detrending term of the original vibration signal can effectively eliminate linear or nonlinear drift, so that subsequent frequency domain analysis and feature extraction are not affected by baseline offset.

[0027] S11-2: Pass the de-stressed vibration signal through a bandpass filter to obtain the denoised vibration signal.

[0028] This embodiment employs a bandpass filter based on detrending. To suppress low-frequency interference and high-frequency noise unrelated to the task, the frequency range of 500Hz to 18kHz is selected based on the characteristics of the motor, covering the fundamental frequency, harmonics, gear meshing frequency and high-frequency impact.

[0029] S11-3: Normalize the denoised vibration signal and perform sliding slices according to a fixed window length and a fixed step size to obtain the preprocessed vibration signal.

[0030] To eliminate the impact of amplitude differences acquired under different test conditions on modeling, this embodiment performs RMS normalization on the denoised vibration signal. The basic formula is: ; in, The normalized vibration signal The vibration signal after noise reduction. The number of samples participating in the normalization.

[0031] This normalization process ensures that the signal amplitude is on a uniform scale under different operating conditions, avoiding model deviations caused by differences in signal strength, thereby improving the stability of subsequent CNN-Transformer and DCN feature modeling.

[0032] To capture short-term dynamic characteristics during motor operation, this embodiment performs sliding slices on the normalized vibration signal according to a fixed window length L and a fixed step size S. The resulting preprocessed vibration signal of segment k can be expressed as: ; in, The normalized vibration signal This represents the preprocessed vibration signal of segment k.

[0033] In this embodiment, an airborne sound signal is collected in the motor's working environment using a high-sensitivity microphone, with a sampling rate of 51200 Hz, to ensure that it is aligned with the timing of the vibration signal.

[0034] Preferably, the preprocessing of the acquired acoustic signals in this embodiment includes: S12-1: Bandpass filtering is performed on the original acoustic signal to obtain a denoised acoustic signal, in order to suppress background noise and irrelevant frequency band interference.

[0035] S12-2: Normalize and segment the denoised acoustic signal to obtain the normalized segmented acoustic signal.

[0036] This embodiment normalizes the amplitude of the denoised acoustic signal and eliminates amplitude differences under recording conditions by combining RMS features.

[0037] S12-3: Reference Figure 3As shown, the normalized segmented acoustic signal is subjected to Stockwell Transform (S-Transform) to obtain a two-dimensional time-frequency spectrum, which serves as the preprocessed acoustic signal.

[0038] The Stockwell transform is a time-frequency analysis method that combines the advantages of the short-time Fourier transform and the wavelet transform, enabling it to simultaneously characterize the amplitude, phase, and time-frequency localization features of a signal. Its mathematical expression is: ; in, The original time-domain signal is used; in this embodiment, it is a normalized segmented acoustic signal. For frequency; This is a time variable, corresponding to the position of the signal sampling point on the time axis; This is the center moment of the signal sliding, used to control the analysis position; It is a frequency-dependent Gaussian window function, and the window width is adaptively adjusted as the frequency changes; It is a two-dimensional time-frequency spectrum.

[0039] S-Transform preserves the phase information of the signal and achieves a dynamic balance between frequency resolution and time resolution, thus generating an intuitive two-dimensional time spectrum, providing rich information support for subsequent feature extraction and deep learning modeling.

[0040] This embodiment preprocesses the acquired vibration and acoustic signals. The preprocessing of the vibration signals includes operations such as detrending, normalization, and segmentation to ensure the stability and comparability of the input data. The acoustic signals are further converted into a two-dimensional time-frequency image through Stockwell transform, thereby revealing the energy distribution characteristics of sound in the time and frequency dimensions and providing richer time-frequency information for subsequent deep feature extraction.

[0041] S2: In the vibration signal processing branch, the vibration signal is input into the one-dimensional convolutional layer feature extraction module to obtain the initial features of the vibration signal; the initial features of the vibration signal are segmented and embedded into a token sequence; the token sequence is input into the Transformer encoder to obtain the global feature vector of the vibration signal.

[0042] The original vibration signal was acquired at a sampling rate of 51200 Hz. After preprocessing including detrending, filtering and denoising, normalization, and sliding window segmentation, it was input into the vibration signal processing branch. In order to fully extract local impulse features, non-stationary transient modes, and cross-time dependence information, this branch uses a one-dimensional convolutional layer feature extraction module (1D-CNN) to gradually realize hierarchical extraction and multi-scale representation of features. Subsequently, an improved Transformer encoder was introduced to capture global temporal features.

[0043] To systematically extract hierarchical temporal features from vibration signals, the vibration signal processing branch is divided into two parts: a one-dimensional convolutional layer feature extraction module and a Transformer encoder. The one-dimensional convolutional layer feature extraction module focuses on capturing local pulse and energy patterns, while the Transformer encoder is used for long-term dependent global feature modeling. These will be explained in detail below.

[0044] The one-dimensional convolutional layer feature extraction module includes: The first one-dimensional convolutional block, including the first one-dimensional convolutional layer, batch normalization layer, ReLU activation function and max pooling layer, is used for initial downsampling and extraction of short-time energy patterns; The second one-dimensional convolutional block, including the first one-dimensional deformable convolutional layer, is used to adaptively capture transient shocks and non-stationary modes by combining deformable convolutional layers with residual structures. The third one-dimensional convolutional block, including the second one-dimensional convolutional layer and the second one-dimensional deformable convolutional layer, is used to enable the model to have a larger receptive field and the ability to represent periodic patterns.

[0045] Preferably, the first one-dimensional convolutional layer has a kernel size of 7, a stride of 2, and 32 channels. The first one-dimensional deformable convolutional layer has a kernel size of 3, a stride of 1, a dilation rate of 2, and 64 channels. The second one-dimensional convolutional layer has a kernel size of 7, a stride of 2, and 128 channels. The second one-dimensional deformable convolutional layer also has a kernel size of 7, a stride of 2, a dilation rate of 4, and 128 channels.

[0046] The initial feature representation of the vibration signal output by the one-dimensional convolutional layer feature extraction module is as follows: T is the time step length.

[0047] Specifically, the initial features of the vibration signal are segmented and embedded into a token sequence, including:

[0048] The initial features of the vibration signal are sliced ​​into segments of a fixed length L, with each L = 128 sampling points forming a token, thus creating a local time block. The formula is as follows: ; in, Let L be the i-th token, where i is the token index and L is the sampling length corresponding to each token. Then, each token is embedded into a 256-dimensional feature space through a linear mapping, as shown in the formula: ; in, For the i-th token after mapping, This is the weight matrix. For bias terms; This yields the token sequence. , This represents the number of tokens after splitting.

[0049] To further capture long-term dependencies and achieve global information interaction, this embodiment improves the Transformer encoder. The Transformer encoder in this embodiment includes multiple Transformer encoder sub-modules. Each Transformer encoder sub-module includes a causal convolutional embedding layer, a multi-head self-attention layer, and a feedforward network layer connected in sequence. Residual connections and layer normalization mechanisms are used between the layers to improve model stability and gradient propagation efficiency. Details are as follows.

[0050] To maintain chronological order and prevent future information leakage, a causal convolutional embedding layer is introduced before encoding. Each token embedding is as follows: ; in, Where is the kernel length, i.e., the history window length, and t is the current time step. For the first Each convolution weight.

[0051] The output features of the causal convolutional embedding layer are represented as , as input features of the multi-head self-attention layer, to achieve global dependency modeling.

[0052] Unlike the equal-weighted attention mechanism in the standard Transformer encoder, the Multi-Head Self-Attention (MHSA) layer designed in this embodiment introduces a frequency-sensitive weighting strategy in the time-domain attention calculation. This strategy fully utilizes the spectral energy distribution characteristics of the vibration signal, enabling the attention mechanism to adaptively adjust the attention ratio between high-frequency transient impacts and low-frequency periodic modes in the time-frequency joint space. This allows for multi-scale feature modeling of non-stationary vibration signals and significantly improves the ability to represent complex dynamic modes.

[0053] Specifically, this embodiment improves upon the traditional multi-head self-attention layer by introducing a frequency-sensitive weighting matrix constructed based on the spectral energy of the vibration signal. The formula is as follows: ; in, The attention mechanism of a multi-head self-attention layer. , and These are the query matrix, key matrix, and value matrix obtained from the input feature mapping, respectively. For the softmax function, This is a frequency-sensitive weighting matrix used to adaptively adjust the attention weight distribution based on the spectral energy characteristics of the signal. is the feature dimension of the attention head in the multi-head self-attention layer.

[0054] After the multi-head self-attention layer completes global dependency modeling, its output features are passed as input to the feedforward network layer for further non-linear feature enhancement. The mathematical expression for the feedforward network layer is: ; in, The input features are those of the feedforward network layer. For GELU activation function, , , and All of these are learnable parameters.

[0055] The feedforward network performs deep feature transformation on the output of the multi-head self-attention layer by combining bilinear mapping and nonlinear activation function (GELU) layer by layer, which extracts higher-order nonlinear features of the signal while preserving global dependency information.

[0056] In this embodiment, the Transformer encoder uses four stacked Transformer encoder submodules. The output of the Transformer encoder is a 256-dimensional global feature vector of the vibration signal. The global feature vector of this vibration signal has both local transient modes and cross-cycle global dependencies, providing a high-precision representation for cross-modal fusion.

[0057] This embodiment inputs a one-dimensional vibration signal into the vibration signal processing branch. Deformable convolution is introduced into the one-dimensional convolutional layer feature extraction module to achieve adaptive capture of transient impacts and non-stationary events. A self-attention mechanism improved by causal convolution is combined in the Transformer encoder to model long-term temporal dependencies. In this way, the dynamic features and anomalous patterns of the vibration signal can be comprehensively extracted and characterized, laying the foundation for subsequent cross-modal feature fusion.

[0058] S3: In the acoustic signal processing branch, the acoustic signal is input into the two-dimensional convolutional feature extraction module to obtain the initial features of the acoustic signal; the initial features of the acoustic signal are then passed through the Swin-Transformer encoder and the hollow spatial pyramid pooling module to obtain the global feature vector of the acoustic signal.

[0059] The processing branches for acoustic signals and vibration signals are in parallel.

[0060] In the acoustic signal processing branch, the input acoustic signal is a two-dimensional time-frequency spectrum with a uniform size of 224×224. This branch consists of three parts: a two-dimensional convolutional feature extraction module (2D-CNN), a Swin-Transformer encoder, and a dilated spatial pyramid pooling module.

[0061] To fully extract the multi-layer spatial features of the time-spectrum, the two-dimensional convolutional feature extraction module at the front end of the acoustic signal processing branch adopts a five-layer two-dimensional convolutional structure to achieve step-by-step encoding from low-frequency energy distribution to high-level semantic features. The structure of the two-dimensional convolutional feature extraction module is shown in Table 1 below.

[0062] Table 1. Structure of the 2D Convolutional Feature Extraction Module

[0063] After five layers of convolution and pooling operations, the time-frequency spectrum with an input size of 224×224×1 is encoded into 7×7×512 acoustic signal initial features, providing rich multi-scale spectral representations for the subsequent Swin-Transformer encoder.

[0064] To fully explore the hierarchical relationships of acoustic features and achieve global semantic modeling, this invention introduces an improved Swin-Transformer encoder after the two-dimensional convolutional feature extraction module. The initial acoustic signal features output by the two-dimensional convolutional feature extraction module are first processed through patching and linear mapping to generate a token sequence, which is then input into the Swin-Transformer encoder for hierarchical modeling and global semantic fusion.

[0065] Preferably, this embodiment improves upon the existing Swin-Transformer encoder. The improved Swin-Transformer encoder adopts a three-stage hierarchical structure, comprising multiple Swin-Transformer encoder sub-modules, as shown in Table 2. Each Swin-Transformer encoder sub-module includes a windowed multi-head self-attention layer and a feedforward network layer. A relative position bias matrix and a sliding window mechanism are introduced in the attention calculation to enhance the model's cross-window modeling capability.

[0066] Table 2. Structure of the improved Swing-Transformer encoder

[0067] Compared to the standard Swin-Transformer encoder, this embodiment makes two improvements to the window-based multi-head self-attention (W-MSA) mechanism in the Swin-Transformer encoder submodule: First, it introduces a relative position bias matrix, incorporating relative position information into the attention calculation to describe the spatial relative relationship between tokens within the window, enhancing the model's spatial perception of the time-frequency structure of the spectrogram; Second, it adopts a sliding window mechanism, which translates adjacent windows to allow for partial overlap between windows, thereby achieving cross-window feature sharing and global semantic fusion, significantly improving the model's global dependency expression ability while maintaining local modeling efficiency.

[0068] Specifically, the multi-head self-attention mechanism for windows introduces a relative position bias matrix and a sliding window mechanism, including: The input features of the multi-head self-attention mechanism are divided into multiple partially overlapping windows using a sliding window mechanism, and multi-head self-attention calculation is performed independently within each window to obtain the attention features of that window. In multi-head self-attention computation, each single-head attention computation introduces a relative position bias matrix, expressed by the formula: ; in, This refers to the attention mechanism of a multi-head self-attention mechanism for windows. , and These are the query matrix, key matrix, and value matrix obtained from the input feature mapping, respectively. It is the softmax function; is the relative position offset matrix, used to model the spatial relative positional relationships between tokens within the window; This is the feature dimension of the attention head in the multi-head self-attention mechanism of the window, used for normalization scaling; The output features of the multi-head self-attention mechanism of windows are obtained by concatenating the attention features of all windows.

[0069] The sliding window mechanism enables feature overlap and information interaction between windows through window translation, allowing attention from adjacent regions to cross local window boundaries, thereby achieving global feature integration and multi-scale semantic fusion based on window attention calculation.

[0070] To further enhance the model's ability to fuse multi-scale receptive fields and contextual features, an Atrous Spatial Pyramid Pooling (ASPP) module is introduced after the improved Swin-Transformer encoder. ASPP achieves multi-scale feature extraction through parallel multi-branch convolutions. Its core idea is to extract features at multiple scales using convolutional kernels with different dilation rates, fusing the detailed features obtained from low-dilation rate convolutions with the global semantic information captured by high-dilation rate convolutions, and finally forming a unified contextual semantic representation through multi-scale aggregation. The calculation formula for ASPP is: ; in, This refers to the output characteristics of the ASPP module. These are the input characteristics for the ASPP module. For the set of void ratios, This represents a 3×3 convolution operation with a hole rate of r.

[0071] Specifically, the ASPP module in this embodiment contains four parallel convolutional branches: (1) The standard convolution branch with a void ratio r=1 is used to preserve local spectral details; (2) The mesoscale branch with a void ratio of r=3 is used to capture mid-frequency energy variation patterns; (3) A large receptive field branch with a void ratio r=6 is used to aggregate cross-band energy evolution characteristics; (4) The ultra-large receptive field branch with a void ratio r=9 is used to learn the overall spectral distribution and long-term semantic dependencies.

[0072] The output features of the four branches of the ASPP module are concatenated and fused along the channel dimension, and then dimensionality-reduced using 1×1 convolution to obtain the fused multi-scale feature map. Subsequently, the output features of the ASPP module undergo global average pooling to obtain the final global feature vector of the acoustic signal. The dimension is 256. For the real number field, This indicates that the feature vector consists of 256 real-valued elements.

[0073] The global feature vector of the acoustic signal integrates multi-scale receptive field information under different void ratios, preserving both local time-frequency texture and resonance details, and including cross-band energy distribution trends.

[0074] The acoustic signal processing branch inputs the two-dimensional acoustic time-frequency image obtained through time-frequency transformation into a two-dimensional convolutional neural network to extract local energy distribution and detailed texture features, thereby enhancing the ability to represent acoustic details. Subsequently, a Swin-Transformer encoder is introduced to model global semantic relationships across time-frequency scales, comprehensively understanding the overall structure and potential patterns of acoustic modes. Further, a hollow spatial pyramid pooling module is combined to achieve multi-scale acoustic information fusion through multi-receptive field feature aggregation, effectively compensating for the shortcomings of the Swin-Transformer encoder in cross-window feature integration. Through joint representation of local modeling, global modeling, and multi-scale feature aggregation, both detail recognition and overall pattern understanding are strengthened, significantly enhancing the robustness and discriminative power of acoustic features in various operating conditions and high-noise environments.

[0075] global eigenvectors of acoustic signals The output features of the acoustic signal processing branch will be compared with the global feature vector of the vibration signal extracted from the vibration signal processing branch. Alignment and joint modeling are performed in an adaptive cross-modal fusion network to achieve complementary fusion and joint discrimination of vibration and acoustic information.

[0076] S4: Input the global feature vectors of the vibration signal and the global feature vectors of the acoustic signal into the adaptive cross-modal fusion network to obtain the detection and classification results.

[0077] After the vibration and acoustic signal processing branches output feature vectors respectively, the key issue becomes how to effectively fuse the two types of modal information to achieve high-precision and robust detection. To this end, this invention designs an Adaptive Cross-Modal Fusion Network, whose core objective is to achieve modal alignment and complementary enhancement. By strengthening feature interaction through a dynamic weighting mechanism, vibration and acoustic features can fully interact and compensate for each other, thereby improving the model's ability to discriminate complex working conditions.

[0078] The adaptive cross-modal fusion network comprises a feature adaptive fusion layer, a graph neural fusion layer, and a decision layer connected sequentially. The feature adaptive fusion layer aligns and normalizes features from two modalities, eliminating distribution differences and enhancing feature stability. The graph neural fusion layer establishes a dynamic correlation between vibration and acoustic features through a graph attention mechanism, enabling information interaction and complementary enhancement between modalities. The decision layer performs nonlinear mapping and discriminative learning based on the fused features, outputting the final detection and classification results.

[0079] The feature adaptive fusion layer takes the global feature vectors of the vibration signal and the global feature vectors of the acoustic signal as input, and includes a linear mapping layer, a batch normalization layer and a ReLU activation function layer, and outputs the normalized global feature vectors of the vibration signal and the global feature vectors of the acoustic signal.

[0080] Specifically, to achieve consistency and distribution normalization of the modal space, the feature adaptive fusion layer first performs linear mapping on the global feature vectors of the vibration signal and the global feature vectors of the acoustic signal, respectively, as shown in the formula: ; ; in, and These are the mapped global feature vectors of the vibration signal and the acoustic signal, respectively. This is the weight matrix. This is a bias term.

[0081] To eliminate statistical differences between different modes and improve training stability, batch normalization and ReLU nonlinear activation function processing are applied to the mapped global feature vectors of the vibration signal and the acoustic signal. The formula is as follows: ; in, This is the mapped global feature vector. and These are the batch mean and variance, respectively. and For learnable parameters, To prevent small constants from being divided by zero, This is the normalized global feature vector.

[0082] Finally, the normalized eigenvectors of the vibration signal are obtained. and the normalized global feature vector of the acoustic signal , which serve as the input node features for subsequent graph neural fusion layers.

[0083] The graph neural network fusion layer takes the normalized global feature vector of the vibration signal and the global feature vector of the acoustic signal as input, and includes two layers of graph neural network and a linear mapping layer, outputting fused features.

[0084] In the graph neural network fusion layer, a graph neural network structure is introduced to establish feature relationships between modalities. The features of the two modalities are then combined. and As graph nodes, construct a two-node complete graph. ,in The node features are initialized as follows: ; The feature interactions between nodes employ an attention-based update mechanism: ; in, , Let p be the set of neighbors of the modal node. Let l be the linear weight matrix of the l-th layer. For the modal q node of the l-th layer, For the modal p node of the (l+1)th layer, The inter-modal attention weights for the l-th layer are calculated using feature similarity: ; in, It is a learnable bilinear scoring function. Let be the softmax function of mode q. Let p be the modal node of the l-th layer.

[0085] In this study, the graph neural network for the fusion layer employs a two-layer structure. The first layer captures low-order correlations between vibration and acoustic modes; the second layer further fuses high-order semantic dependencies based on the feature interactions of the first layer, achieving deep alignment and complementary enhancement of multimodal information. After information propagation and fusion through the two GNN layers, the updated node representation is obtained: and .

[0086] After concatenating the representations of the two modalities, a fused feature is generated through a linear mapping layer: ; in, As a feature of fusion, This is the weight matrix. This is a bias term.

[0087] The fusion features include both the dynamic structural patterns of vibration signals and the dynamic spectral energy features of acoustic features, achieving adaptive semantic alignment and feature complementarity among multiple modes.

[0088] In this embodiment, the global feature vectors extracted from the vibration and acoustic branches are input into the feature adaptive fusion layer to achieve feature alignment and dynamic weighting. Then, the normalized global feature vectors are mapped as graph nodes and input into the graph neural network fusion layer. The graph convolution and feature propagation mechanisms of the graph neural network are used to model the deep interaction between acoustic and vibration features, thereby fully exploring the complementary advantages of the two modes and obtaining more discriminative fused features.

[0089] The decision layer takes fused features as input and includes a batch normalization layer, a fully connected layer, and a softmax classification layer, outputting detection and classification results. The decision layer is used to complete feature mapping and classification output, providing a judgment result for the motor's operating status, i.e., OK (normal) / NG (abnormal).

[0090] After aligning and fusing cross-modal features, the fused features need to be mapped to specific classification results. The decision layer designed in this embodiment mainly consists of a normalization layer, a fully connected layer, a regularization mechanism, and a classification output layer, ensuring that the model has both discriminative power and avoids overfitting.

[0091] First, regarding fusion features Batch normalization is performed using the following formula: ; in, and These are the batch mean and variance, respectively. and For learnable parameters, This represents the fusion characteristics after normalization.

[0092] Normalized fusion features The input is fed into two fully connected layers. The first fully connected layer performs linear mapping and nonlinear transformation on the features, while the second fully connected layer performs further feature compression and semantic mapping. The computation process is as follows: ; in, and These are the output features of the first and second fully connected layers, respectively. and This is the weight matrix. and For bias terms, This represents the output features of the first fully connected layer after regularization.

[0093] Specifically, this embodiment introduces a Dropout regularization mechanism between two fully connected layers, used only once between the two fully connected layers, to suppress overfitting and enhance generalization performance by randomly discarding some neurons. Its mathematical form is as follows: ; in, For element-wise multiplication, This represents the probability of discarding the item.

[0094] Output features of the second fully connected layer The classification confidence score is obtained after linear transformation, using the following formula: ; in, The confidence score for category c, which includes OK or NG, where OK represents the normal state and NG represents the abnormal state; This is the weight matrix. This is a bias term.

[0095] Finally, the Softmax classification layer outputs a binary classification probability distribution based on the confidence score, and performs probability normalization calculation for the category (OK or NG) to which the sample belongs, as shown in the following formula: ;

[0096] The adaptive cross-modal fusion network provided in this embodiment not only maintains stable performance under strong noise and multi-condition environments, but also effectively utilizes the complementarity of vibration and acoustic modes to mine high-dimensional discriminative power of cross-modal features. Ultimately, it achieves intelligent OK and NG detection for new energy vehicle motors and other rotating machinery, possessing high precision and good engineering adaptability.

[0097] In the signal acquisition and preprocessing stage, this invention synchronously acquires structural vibration signals and airborne acoustic signals of the motor during operation using a vibration accelerometer and microphone, with a uniform sampling rate of 51200 Hz. During preprocessing, the vibration signal undergoes detrending, normalization, and sliding window segmentation to eliminate trend terms and amplitude differences. The acoustic signal, after filtering and normalization, is further subjected to a Stockwell transform to obtain a time-frequency plot, which serves as input for subsequent modeling. In the vibration signal processing branch, the one-dimensional vibration signal first passes through a one-dimensional convolutional layer feature extraction module to extract local impulse features, and deformable convolution is introduced into the convolution to improve the ability to capture transient impacts and non-stationary modes. Subsequently, the signal enters a Transformer encoder with causal convolution improvement, utilizing a self-attention mechanism to model long-term dependencies, thereby obtaining a comprehensive dynamic structural feature representation. In the acoustic signal processing branch, the two-dimensional time-frequency image obtained by Stockwell transform is first input into a two-dimensional convolutional feature extraction module to extract local energy texture and transient detail features. Then, it is fed into a Swin-Transformer encoder to model global semantic information across time and frequency scales, enhancing the representation ability of the overall acoustic mode. Based on this, a hollow spatial pyramid pooling module is further introduced to achieve multi-scale information fusion through multi-receptive field feature aggregation, thereby compensating for the shortcomings of the Swin-Transformer encoder in cross-window feature integration. This branch can preserve the local details of the acoustic signal while capturing global modes and multi-scale features, resulting in a more accurate and comprehensive characterization of the motor's acoustic performance. Subsequently, the features of the two modalities are fed into an adaptive cross-modal fusion layer. Modal alignment is first completed, and then deep interaction and fusion are achieved through dynamic weighting and combined with a graph neural network, significantly improving the complementarity and discriminative power of cross-modal features. Finally, the fused high-dimensional features are input into the decision layer, processed by batch normalization and multi-layer fully connected processing, and output as OK / NG results through a Softmax classifier, thereby achieving stable and high-precision anomaly detection in noisy and multi-condition environments.

[0098] A motor anomaly detection model is constructed using vibration signal processing branches, acoustic signal processing branches, and an adaptive cross-modal fusion network. Its structure diagram is shown in [reference needed]. Figure 4 As shown.

[0099] During the model training phase, vibration acceleration sensors are used to collect structural vibration signals of the electric motor and its transmission system in new energy vehicles under different operating conditions. These signals directly reflect the dynamic characteristics and potential anomalies of the mechanical body. Subsequently, acoustic signals generated during motor operation are collected via microphones to characterize the motor's airborne acoustic performance. After data collection, the data is labeled to distinguish between normal (OK) and abnormal (NG) states. All data is proportionally divided into training, validation, and test sets to ensure effective training, validation, and testing of the model on different datasets. The constructed training set is input into the proposed deep learning model, and a five-fold cross-validation strategy is used to complete the training process. During training, the model dynamically adjusts parameter settings based on the performance on the validation set, gradually converging and obtaining the optimal model version.

[0100] After model optimization, it was applied to the test set to classify acoustic and vibration samples that were not used in training, outputting OK / NG judgments. The model's performance was evaluated by comparing the predicted results with the actual labels, validating its detection accuracy and robustness. Finally, the model was deployed in an online automotive motor inspection system to achieve real-time quality monitoring on the production line.

[0101] The proposed motor anomaly detection model was validated on a multi-condition dataset of new energy vehicle motors. Results show that the classification accuracies of the vibration signal processing branch using only a one-dimensional convolutional layer feature extraction module (without deformable convolution) and the acoustic signal processing branch using only a two-dimensional convolutional feature extraction module are 92.2% and 91.8%, respectively, which have certain limitations under complex operating conditions. With the introduction of improved modules—specifically, an improved Transformer encoder in the vibration signal processing branch incorporating one-dimensional convolutional layers, deformable convolution, and causal convolution—the accuracy improved by 94.3%. Similarly, the acoustic signal processing branch, combining a two-dimensional convolutional feature extraction module, a Swin-Transformer encoder, and a hollow spatial pyramid pooling module, achieved an accuracy of 93.7%. These results demonstrate that the improved structure effectively enhances the representation capabilities for non-stationary impacts, cross-frequency textures, and complex acoustic patterns.

[0102] Furthermore, when vibration and acoustic features are fused using a simple splicing method, the accuracy reaches 96.4%; while the adaptive cross-modal fusion network proposed in this invention, after combining with a graph neural network, further improves the classification accuracy to 98.6%. Compared with the best single-modal method (94.3%), the error rate is significantly reduced from 5.7% to 1.4%, with a relative error reduction rate of over 75%, fully demonstrating the complementary advantages and synergistic gains of vibration and acoustic signals.

[0103] The results of the ablation experiments are shown in Table 3. Further evidence indicates that the core driving factors for performance improvement are the sensitivity of deformable convolution to transient anomalies, the temporal capture capability of causal convolution + Transformer encoder for operating condition switching, the multi-scale semantic enhancement effect of Swin-Transformer encoder, and the modeling capability of adaptive fusion + graph neural network for cross-modal dependencies. Meanwhile, experimental results under different operating conditions and noise layers show that this method maintains an overall accuracy of ≥97% even under low signal-to-noise ratio, multi-operating condition, and cross-batch testing conditions, highlighting its high accuracy, strong robustness, and good engineering application value in industrial environments.

[0104] Table 3. Ablation Experiment

[0105] Based on the above-mentioned method for detecting abnormal noise from automotive motors using multimodal deep learning, this invention also provides a device for detecting abnormal noise from automotive motors using multimodal deep learning, comprising: The acquisition module is used to acquire vibration and acoustic signals during the operation of the car's motor. The vibration signal feature extraction module is used to input the vibration signal into the one-dimensional convolutional layer feature extraction module to obtain the initial features of the vibration signal; to segment and embed the initial features of the vibration signal into a token sequence; and to input the token sequence into the Transformer encoder to obtain the global feature vector of the vibration signal. The acoustic signal feature extraction module is used to input the acoustic signal into the two-dimensional convolutional feature extraction module to obtain the initial features of the acoustic signal; the initial features of the acoustic signal are then passed through the Swin-Transformer encoder and the hollow spatial pyramid pooling module to obtain the global feature vector of the acoustic signal. The fusion classification module is used to input the global feature vectors of vibration signals and acoustic signals into the adaptive cross-modal fusion network to obtain the detection and classification results.

[0106] In summary, the automotive motor noise detection method based on multimodal deep learning described in this invention fully integrates the advantages of vibration and acoustic signals. In the vibration signal processing branch, features are extracted through one-dimensional convolutional layers, and deformable convolutions are introduced to capture transient impacts and non-stationary modes. Then, a Transformer encoder incorporating causal convolutions is used to model the long-term dependencies and dynamic structural features of the vibration signal. In the acoustic signal processing branch, features are extracted through two-dimensional convolutional layers, and a Swin-Transformer encoder and a hollow spatial pyramid pooling module are combined to model the global semantics and multi-scale features of the acoustic signal across time and frequency. Finally, a graph neural network with an adaptive cross-modal fusion network is used to establish a structured correlation between sound and vibration, improving the detection accuracy of automotive motor noise. This invention utilizes multi-level, cross-modal information integration, which can preserve the local energy texture and complex frequency features of multimodal signals while modeling long-term dependencies and dynamic relationships between modes, significantly improving the accuracy and robustness of automotive motor noise detection. Its performance is significantly better than single-modal or key-module-removed comparative methods.

[0107] The method of this invention has significant application value in the online testing of motors and their transmission systems in new energy vehicles. Through multimodal signal fusion, the system can promptly identify early signs of faults such as bearing wear, rotor imbalance, and abnormal gear meshing during motor operation. This helps factories take preventative maintenance measures in advance, avoiding more serious damage and downtime risks, thereby effectively reducing maintenance costs. The testing system can operate automatically around the clock, reducing manual intervention and lowering the probability of sudden downtime through early warnings, significantly improving production efficiency. Simultaneously, the system continuously accumulates acoustic and vibration data during long-term operation, which can be used for iterative model optimization, further improving detection accuracy and robustness, and assisting in the intelligent quality control and operation and maintenance management of new energy vehicle production lines.

[0108] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for detecting abnormal sound of an automobile motor based on multi-modal deep learning, characterized in that, The method comprises the following steps: Collecting vibration signals and acoustic signals during the operation of an automobile motor; Inputting the vibration signals into a one-dimensional convolution layer feature extraction module to obtain initial vibration signal features; Splitting and embedding the initial vibration signal features into a token sequence; inputting the token sequence into a Transformer encoder to obtain a vibration signal global feature vector; Inputting the acoustic signals into a two-dimensional convolution feature extraction module to obtain initial acoustic signal features; sequentially inputting the initial acoustic signal features into a Swin-Transformer encoder and a hollow spatial pyramid pooling module to obtain an acoustic signal global feature vector; Inputting the vibration signal global feature vector and the acoustic signal global feature vector into an adaptive cross-modal fusion network to obtain a detection classification result.

2. The method of claim 1, wherein the method is based on a multi-modal deep learning. Before inputting the vibration signals into the one-dimensional convolution layer feature extraction module, the vibration signals are preprocessed, including: Approximating the original vibration signals to a low-order polynomial trend by using least squares, and subtracting the low-order polynomial trend from the original vibration signals to obtain a detrended vibration signal; Passing the detrended vibration signal through a band-pass filter to obtain a denoised vibration signal; Normalizing the denoised vibration signal, and performing sliding slicing according to a fixed window length and a fixed step length to obtain a preprocessed vibration signal.

3. The method of claim 1, wherein the method is based on a multi-modal deep learning. Before inputting the acoustic signals into the two-dimensional convolution feature extraction module, the acoustic signals are preprocessed, including: Performing band-pass filtering on the original acoustic signals to obtain a denoised acoustic signal; Normalizing and segmenting the denoised acoustic signal to obtain a normalized segmented acoustic signal; Performing a Stockwell transform on the normalized segmented acoustic signal to obtain a two-dimensional time-frequency spectrum as a preprocessed acoustic signal.

4. The method of claim 1, wherein the method is based on a multi-modal deep learning. The one-dimensional convolution layer feature extraction module comprises: A first one-dimensional convolution block comprising a first one-dimensional convolution layer, a batch normalization layer, a ReLU activation function and a maximum pooling layer; A second one-dimensional convolution block comprising a first one-dimensional deformable convolution layer; A third one-dimensional convolution block comprising a second one-dimensional convolution layer and a second one-dimensional deformable convolution layer.

5. The method of claim 1, wherein the method is based on a multi-modal deep learning. The Transformer encoder comprises a plurality of Transformer encoder sub-modules, each Transformer encoder sub-module comprising a causal convolution embedding layer, a multi-head self-attention layer and a feedforward network layer connected in sequence, and the layers are connected by a residual connection and a layer normalization mechanism.

6. The method of claim 5, wherein the method is based on a multi-modal deep learning. Improvements are made to the multi-head self-attention layer, including: A frequency-sensitive weighting matrix based on the frequency energy of the vibration signals is introduced into the multi-head self-attention layer, and the formula is as follows: ； wherein, is an attention mechanism of the multi-head self-attention layer, , and are a query matrix, a key matrix and a value matrix respectively mapped from the input feature, is a softmax function, is a frequency-sensitive weighting matrix, is a feature dimension of an attention head in the multi-head self-attention layer.

7. The method of claim 1, wherein the method is based on a multi-modal deep learning. The Swin-Transformer encoder comprises a plurality of Swin-Transformer encoder sub-modules, each Swin-Transformer encoder sub-module comprising a window multi-head self-attention layer and a feedforward network layer.

8. The method of claim 7, wherein the method is based on a multi-modal deep learning. Improvements are made to the window multi-head self-attention mechanism, including: A relative position bias matrix and a sliding window mechanism are introduced into the window multi-head self-attention mechanism, including: The input features of the window multi-head self-attention mechanism are divided into multiple partially overlapped windows by using a sliding window mechanism, and multi-head self-attention calculation is independently performed in each window to obtain the attention features of the window. In the multi-head self-attention calculation, a relative position bias matrix is introduced for each single-head attention calculation, and the formula is represented as: ； wherein, is an attention mechanism of the window multi-head self-attention mechanism, , and are a query matrix, a key matrix and a value matrix obtained by mapping the input features respectively, is a softmax function, is a relative position bias matrix, is a feature dimension of an attention head in the window multi-head self-attention mechanism. The attention features of all windows are spliced to obtain the output features of the window multi-head self-attention mechanism.

9. The method of claim 1, wherein the method is based on a multi-modal deep learning. The adaptive cross-modal fusion network comprises a feature adaptive fusion layer, a graph neural fusion layer and a decision layer connected in sequence; The feature adaptive fusion layer takes the vibration signal global feature vector and the acoustic signal global feature vector as input, and comprises a linear mapping layer, a batch normalization layer and a ReLU activation function layer, and outputs the normalized vibration signal global feature vector and the acoustic signal global feature vector; The graph neural fusion layer takes the normalized vibration signal global feature vector and the acoustic signal global feature vector as input, and comprises two layers of graph neural networks and a linear mapping layer, and outputs the fusion features; The decision layer takes the fusion features as input, and comprises a batch normalization layer, a fully connected layer and a Softmax classification layer, and outputs the detection classification result.

10. A multi-modal deep learning based automobile motor abnormal sound detection device, characterized in that, It comprises: The acquisition module is configured to acquire vibration signals and acoustic signals during the operation of an automobile motor; The vibration signal feature extraction module is configured to input the vibration signals into a one-dimensional convolution layer feature extraction module to obtain vibration signal initial features, split and embed the vibration signal initial features into a token sequence, and input the token sequence into a Transformer encoder to obtain a vibration signal global feature vector; The acoustic signal feature extraction module is configured to input the acoustic signals into a two-dimensional convolution feature extraction module to obtain acoustic signal initial features, and sequentially input the acoustic signal initial features into a Swin-Transformer encoder and a hollow spatial pyramid pooling module to obtain an acoustic signal global feature vector; The fusion classification module is configured to input the vibration signal global feature vector and the acoustic signal global feature vector into an adaptive cross-modal fusion network to obtain a detection classification result.

Images