Multi-mode abnormal sound detection system based on deep learning and detection method thereof

Abstract

The invention relates to the technical field of industrial product quality detection and equipment fault diagnosis, in particular to a multi-mode abnormal sound detection system based on deep learning and a detection method thereof. The system comprises an audio signal acquisition module, an audio preprocessing module, a time domain feature extraction module, a frequency domain feature extraction module, a multi-granularity pooling module, a feature fusion module, a context sensing weighting module and an abnormal sound judgment module. The method has the beneficial effects that the defect that the prior art only depends on a single frequency domain feature and ignores time domain transient abnormal information is overcome by constructing a bimodal input mechanism of the time domain waveform and the frequency domain spectrogram, and complete representation of abnormal sound signals in a time-frequency joint space is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial product quality detection and equipment fault diagnosis technology, and particularly relates to a multi-modal abnormal sound detection system based on deep learning and a detection method thereof. BACKGROUND

[0002] In the product quality detection link of modern manufacturing, sound signal detection as a non-contact and real-time detection method has been widely used in various mechanical and electrical products, household appliances, automobile parts and consumer electronics. When the product has defects such as poor assembly, part wear or structural looseness, specific "abnormal sound" will be generated. Therefore, through the analysis of the running sound, the automatic judgment of the product state is realized, which has important application value in the scenes of factory inspection, automatic quality control and predictive maintenance. Traditionally, abnormal sound detection mainly relies on manual listening, that is, experienced quality inspectors judge whether there is an abnormality through subjective hearing. However, this method is significantly affected by individual experience differences, auditory fatigue and environmental noise interference, resulting in inconsistent judgment standards, high misjudgment rate, and difficulty in meeting the requirements of large-scale, high-pace production lines for detection efficiency and consistency.

[0003] Prior art, publication number CN113744756B, "equipment quality inspection and audio data expansion method and related device, equipment, medium", published on August 16, 2024. The patent proposes to obtain the running audio of the equipment, extract acoustic features, and use a quality inspection model trained based on a sub-spectrum graph to make abnormality judgment, thereby improving the efficiency and accuracy of equipment quality inspection. However, this scheme only relies on frequency domain spectrogram as input feature, does not fully exploit the transient change information of sound signal in time domain, and lacks the ability to jointly model the time-frequency dual modal features. In addition, the model structure does not introduce a context perception mechanism, making it difficult to dynamically identify key abnormal frames. When facing background noise interference or weak abnormal sound caused by equipment aging, the detection robustness is limited, and the generalization ability is insufficient.

[0004] In addition, the patent with publication number CN114051076B entitled "A customer service intelligent quality inspection method, device and storage medium" was disclosed on January 14, 2025. This patent constructs a keyword detection model and an emotion recognition model, respectively processes the customer service voice through two-dimensional convolution and multi-head attention, and realizes multi-task parallel quality inspection. Although this scheme introduces a multi-task learning framework, its application scenario is limited to voice content understanding (such as keywords and emotions), rather than unsupervised or weakly supervised anomaly detection for industrial equipment abnormal sound. More importantly, although this method uses time domain acoustic features and three-channel feature maps, it does not systematically integrate time domain waveforms and frequency domain spectrograms, two complementary modalities, nor design a mask weighting or context-aware mechanism for abnormal sound characteristics, making it difficult to effectively capture non-semantic and non-structural abnormal acoustic patterns during product operation.

[0005] Existing deep learning-based audio quality inspection technologies mostly focus on a single modality or a specific semantic scenario, and when facing industrial abnormal sound detection, there are generally defects such as incomplete modality expression, weak context modeling capability, and low sensitivity to weak abnormalities. Therefore, the present application proposes a deep learning-based multi-modal abnormal sound detection method, which cooperatively models time domain and frequency domain features, introduces a context-aware mask mechanism, and uses multi-granularity pooling and residual spectral coding structure to significantly improve the accuracy, robustness and industrial applicability of abnormal sound detection, to meet the urgent need for high-precision and high-efficiency automated quality inspection in intelligent manufacturing. SUMMARY

[0006] The purpose of the present application is to provide a deep learning-based multi-modal abnormal sound detection method to overcome the above-mentioned deficiencies in the prior art.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: A deep learning-based multi-modal abnormal sound detection method, the method comprising the following steps: S1, collecting the original audio signal generated by the product during operation at the quality inspection station, and performing normalization preprocessing on the original audio signal; S2, parallelly inputting the preprocessed audio signal into a time domain feature extraction branch and a frequency domain feature extraction branch, respectively generating an audio embedding vector Faudio and a spectrum embedding vector Fspec; S3, inputting Faudio and Fspec into a multi-modal feature fusion module, and weighting and fusing through a learnable weight a to obtain a fusion feature vector Ffusion; S4, inputting Ffusion into a fully connected classification layer and outputting a binary classification discrimination result through a Softmax function to complete the judgment of the product state OK or NG; wherein the entire model is trained end-to-end using a Softmax cross-entropy loss function.

[0008] Preferably, in the time-domain feature extraction branch of S2, the normalized original audio signal is first input into a front-end convolution module composed of several one-dimensional convolution layers for extracting frame-level acoustic features; then the frame-level feature sequence is input into a densely connected time-delay neural network, each layer of which receives all the outputs of the previous layer within a time window [t-Δ, t+Δ] as input, and the calculation process is as follows: , wherein, denotes the time-delay window, is the number of network layers.

[0009] Preferably, after the densely connected time-delay neural network outputs the frame-level feature sequence, a multi-granularity pooling module is introduced to aggregate context information of different time scales: a global average pooling operation is performed on the entire frame sequence to generate a global context vector G; at the same time, the frame sequence is divided into several continuous segments, a maximum pooling operation is performed on each segment, and the maximum pooling results of each segment are spliced to generate a segment-level context vector S; finally, the feature vector is formed by splicing the global feature G and the local feature S: .

[0010] Preferably, the aggregated context vector Fagg is input into a small multi-layer perceptron, which contains 1-2 hidden layers, the number of neurons in each layer does not exceed 128, and the output dimension is consistent with the length of the frame-level feature sequence; the small multi-layer perceptron outputs a frame-by-frame mask, which is used for point-by-point multiplication operation on the original frame-level features output by the D-TDNN to obtain calibrated features, and then the audio embedding vector Faudio is generated by performing global pooling on the calibrated features.

[0011] Preferably, in the frequency-domain feature extraction branch of S2, the normalized original audio signal is subjected to short-time Fourier transform to obtain a time-frequency spectrum in complex form; the time-frequency spectrum is taken modulo and logarithmically compressed to obtain a logarithmic amplitude spectrum; then the logarithmic amplitude spectrum is input into a residual convolutional neural network; finally, the feature map of the last layer of the residual convolutional neural network is subjected to global average pooling to generate a spectral embedding vector Fspec.

[0012] Preferably, in S3, the multi-modal feature fusion module adopts a learnable linear weighting strategy, and its fusion formula is: , wherein is a weight parameter to achieve optimal proportional fusion of time-domain and frequency-domain features.

[0013] Preferably, linear projection is performed on Faudio to make its dimensionality consistent with Fspec before weighted fusion to calculate Ffusion.

[0014] Preferably, in S4, the fused global feature Ffusion is input into a fully connected classification layer, and a Softmax function is used for probability output. , wherein corresponds to whether the product has abnormal sound; The classification result is output to the quality inspection control system in the form of probability, and when , the threshold is configurable, the system automatically triggers the rejection or manual re-inspection signal to realize real-time automatic screening.

[0015] A multi-modal abnormal sound detection system based on deep learning, comprising: an audio signal acquisition module for acquiring original audio signals generated by a device under test running under standard conditions; an audio preprocessing module for preprocessing the original audio signals to generate time-domain waveform sequences and frequency-domain spectrogram sequences; a time-domain feature extraction module for inputting the time-domain waveform sequences into a one-dimensional convolutional neural network to extract time-domain local transient features and obtain a time-domain feature tensor; a frequency-domain feature extraction module for inputting the frequency-domain spectrogram sequences into a residual spectral encoder to extract frequency-domain deep structure features and obtain a frequency-domain feature tensor; a multi-granularity pooling module for performing multi-granularity time pooling processing on the time-domain feature tensor and the frequency-domain feature tensor respectively to form time-domain multi-granularity representation and frequency-domain multi-granularity representation; a feature fusion module for channel splicing the time-domain multi-granularity representation and the frequency-domain multi-granularity representation to generate a fusion feature vector; a context-aware weighting module for calculating a context-aware attention weight based on the fusion feature vector and performing weighted summation on the fusion feature vector using the weight to generate a context-weighted fusion representation; and an abnormal sound judgment module for inputting the context-weighted fusion representation into a classification decoder to generate an abnormal sound probability value and determining whether there is an abnormal sound defect according to a comparison result of the probability value and a preset threshold.

[0016] The beneficial effects of the present application are: the technical scheme of the present application overcomes the defects of the prior art that only relies on single frequency domain features and ignores time domain transient abnormal information by constructing a dual-mode input mechanism of time domain waveform and frequency domain spectrogram, realizing complete characterization of abnormal sound signals in a time-frequency joint space. The designed residual spectrum encoder effectively enhances the hierarchical expression ability of frequency domain features, avoiding the gradient disappearance problem in deep networks. The multi-granularity time pooling module retains multi-scale abnormal clues from global trends to local mutations, significantly improving the capture ability of weak and short-time abnormal sounds. The context-aware mask weighting mechanism dynamically focuses on the key time frames that are most discriminative for abnormality determination, suppressing the interference of background noise and irrelevant steady-state components, and improving the robustness of the model in complex industrial environments. The overall system adopts an end-to-end trainable architecture, does not require manual design of acoustic features, and is suitable for general abnormal sound detection scenarios of various mechanical and electrical products. The experimental results show that the detection accuracy is improved by about 12 percentage points, the false positive rate is reduced to less than 3%, and the real-time and consistency requirements of automatic quality inspection of high-rhythm production lines are met. BRIEF DESCRIPTION OF DRAWINGS

[0017] Fig. 1 It is a product abnormal sound detection model calculation flowchart of the present application of a multi-modal abnormal sound detection method based on deep learning; Fig. 2 It is a feature fusion structure diagram of the present application of a multi-modal abnormal sound detection method based on deep learning; Fig. 3 It is a system module diagram of the present application of a multi-modal abnormal sound detection method based on deep learning. DETAILED DESCRIPTION

[0018] REFERENCE Figs. 1-2 A multi-modal abnormal sound detection method based on deep learning, the method comprising the following steps: S1, collecting original audio signals generated by the product during operation in the quality inspection station, and performing normalization preprocessing on the original audio signals; S2, parallelly inputting the preprocessed audio signals into a time domain feature extraction branch and a frequency domain feature extraction branch, respectively generating audio embedding vectors Faudio and spectrum embedding vectors Fspec; S3, inputting Faudio and Fspec into a multi-modal feature fusion module, and weighting and fusing through a learnable weight a to obtain a fusion feature vector Ffusion; S4, inputting Ffusion into a fully connected classification layer and outputting a binary classification result through a Softmax function, completing the determination of the product state OK or NG; wherein the entire model is trained end-to-end using a Softmax cross-entropy loss function.

[0019] In step S1, the acquisition of the original audio signal is completed by a high-fidelity audio acquisition device deployed at the production line quality inspection station. At the test station where the product is running, a plurality of high-sensitivity microphone arrays are arranged to synchronously acquire the audio signal s(t). In one of the embodiments, the sensitivity is not less than negative forty-six decibels, the signal-to-noise ratio is not less than seventy percent decibels, the sampling precision is twenty-four bits, and the sampling rate is fixed at forty-eight thousand hertz.

[0020] The device under test needs to run under standard working conditions, that is, the parameters such as power supply voltage, load torque, ambient temperature and humidity are within the normal working range defined in the product specification. The audio acquisition duration is 5-10 seconds, which ensures to cover the typical acoustic behavior in the whole process of device startup, steady-state operation and shutdown.

[0021] In one of the embodiments, the specific acquired original audio signal is stored in a pulse code modulation format, and the total number of sampling points is two hundred and forty thousand points.

[0022] A standardized preprocessing procedure is performed on the original audio signal. First, the original audio signal is divided into frames. In one of the embodiments, a Hanning window function is applied to each frame of signal to suppress the spectral leakage effect. Then, the fast Fourier transform is performed on each windowed frame to obtain the amplitude spectrum as the modulus of the complex result. After this processing, a sequence of frequency domain spectrograms is generated. At the same time, the original time domain waveform data after framing is directly retained as a time domain waveform sequence as the input for subsequent time domain feature extraction. In order to eliminate the gain difference of different devices and environmental noise interference, the original signal is subjected to zero-mean unit-variance normalization processing, that is, s'(t) = (s(t)-μ) / σ, where μ and σ are the mean and standard deviation of the batch of samples, respectively. Then, the normalized audio signal is sent to two parallel processing branches: one is directly used for time domain feature extraction, and the other is used for frequency domain feature generation.

[0023] The time domain waveform sequence is sent to a one-dimensional convolutional neural network for feature extraction. The network is composed of at least four cascaded one-dimensional convolutional layers. Each convolutional layer is followed by batch normalization operation and rectified linear unit activation function. The vector completely retains the high-frequency mutation characteristics of local transient anomalies such as click and burst in the time domain. The time domain feature extraction branch first inputs s'(t) into the front-end convolution module (FCM) with ReLU activation function, followed by batch normalization and max-pooling operation, and finally outputs a sequence of frame-level acoustic features. The feature sequence is then sent to a densely connected time delay neural network (D-TDNN). The D-TDNN contains 5 stacked layers, each layer takes the features from the 5 time steps [t-2, t+2] of the previous layer as input, and the output of the l-th layer at time t is represented as, , where σ is the ReLU activation function, denotes the delay window, is the number of network layers. With this dense connection and delay modeling mechanism, D-TDNN explicitly captures the dynamic dependencies between different time instants of the heterophonic signal, effectively improving the model’s ability to detect intermittent heterophony, such as the intermittent “click” sound produced by a loose bearing or the periodic “squeak” sound caused by brush wear. D-TDNN is only a preferred time series modeling network, and networks with time series modeling capabilities such as TCN, Transformer, etc. can also be used.

[0024] The frame-level feature sequence output by D-TDNN then enters the multi-granularity pooling module. This module performs two types of pooling operations: on the one hand, it performs global average pooling on the entire time series to generate a global context vector G that represents the overall acoustic trend; on the other hand, it divides the time series into 4 consecutive segments, performs max pooling on each segment, and then concatenates the results of each segment to form a segment-level context vector S The final feature vector is formed by concatenating the global feature G and S: Fagg= [G; S], which contains both overall pattern trends and transient mutation information, providing rich input for final classification. This calibration operation effectively suppresses the feature responses of irrelevant frames such as background mechanical noise and steady-state running sound, while enhancing the key frame feature strength related to abnormal events. Finally, the calibrated feature sequence is globally averaged to generate an audio embedding vector Faudio.

[0025] Processing the frequency domain spectrogram sequence. In the frequency domain feature extraction branch, the normalized audio signal s'(t) is sent to the Short-Time Fourier Transform (STFT) module, using a Hamming window function to obtain a complex time-frequency spectrogram: , where is the Hamming window function, is the frequency, is the time frame.

[0026] Take the modulus value of the spectrogram and perform logarithmic compression to obtain a log amplitude spectrogram, which is input into a two-dimensional image residual convolutional neural network (ResNet), This embodiment employs a lightweight variant of ResNet-50, comprising four residual stages. Each stage consists of 3, 4, 6, and 3 residual blocks, respectively. Each residual block is composed of 1×1, 3×3, and 1×1 bottleneck convolutional structures with channel numbers of 256, 512, 1024, and 2048, respectively, and is equipped with skip connections. Through this structure, lower-level convolutions preserve detailed information such as harmonic structures and instantaneous energy changes in the spectrum, while higher-level convolutions extract abstract semantic features such as frequency band energy imbalance and spurious high-frequency noise. The feature map output by the last layer of ResNet is 7×7×2048, which, after global average pooling, generates a 2048-dimensional spectral embedding vector Fspec. ResNet is only one preferred image feature extraction network; structures such as EfficientNet and ConvNeXt can also be used. The audio embedding vector Faudio and the spectral embedding vector Fspec are fed into the multimodal feature fusion module.

[0027] The temporal multi-granularity representation and the frequency multi-granularity representation are concatenated along the channel dimension. Due to the inconsistency in their temporal dimensions, to achieve dimensional alignment, Faudio is first linearly projected to 512 dimensions, and then weighted and fused with Fspec: Ffusion = α· (Faudio) + (1-α)·Fspec, where α is a learnable scalar parameter, initially set to 0.5 and automatically optimized through backpropagation. This fusion mechanism allows the model to adaptively adjust the weight ratio of temporal dynamic characteristics and frequency structural characteristics according to the actual product type (e.g., motors with predominantly high-frequency noise vs. pumps with predominantly low-frequency vibration), thereby fully utilizing the complementarity of the two modes. This vector simultaneously contains temporal transient details and frequency structural information, while preserving multi-scale temporal cues.

[0028] The fused feature vectors (Ffusion) are input into Softmax, a two-layer fully connected network corresponding to the "OK" and "NG" categories, respectively. The final classification stage utilizes the Softmax function. , The system performs binary classification (OK / NG) judgment, outputting probability values ​​with clear physical meaning, facilitating integration into factory quality inspection automation systems for real-time detection and feedback control. The Softmax function normalizes logits and outputs the probability distribution P(y|x) of whether a product is normal or abnormal. When P(NG|x) > θ (θ is a preset threshold, usually set to 0.5, but can be dynamically adjusted according to the production line's false alarm rate requirements), the system determines it as NG, triggering a rejection mechanism or manual re-inspection instruction.

[0029] The processing of Ffusion, including the calculation of context-aware attention weights, effectively enhances feature selection capabilities. Considering that product noise often manifests as short-term abrupt changes, this invention introduces a context-aware masking module onto the time-series features. The mask is generated by a lightweight gating network. , in As a time series feature, The sigmoid function is used. The weighted features are obtained by pointwise multiplying the mask by the original features: , This step can suppress noise interference signals and highlight time segments related to abnormal sounds.

[0030] To address the issues of multiple noise interferences and complex sound source overlap in real-world production environments, this invention utilizes context-aware masking M... t Dynamically weight the frame-level features: , This mechanism can automatically adjust the importance of features in each frame based on temporal context information, thereby suppressing irrelevant information (such as background mechanical noise or worker speech) and strengthening key features related to abnormal sounds, thus greatly improving the discriminative power of feature representation.

[0031] refer to Fig. 3 A deep learning-based multimodal abnormal sound detection system, comprising: The audio signal acquisition module is used to acquire the raw audio signal generated by the device under test under standard operating conditions. An audio preprocessing module is used to preprocess the original audio signal to generate a time-domain waveform sequence and a frequency-domain spectrogram sequence. The temporal feature extraction module is used to input the temporal waveform sequence into a one-dimensional convolutional neural network to extract local transient features in the temporal domain and obtain a temporal feature vector. The frequency domain feature extraction module is used to input the frequency domain spectrogram sequence into the residual spectrum encoder to extract deep structural features in the frequency domain and obtain frequency domain feature vectors. The multi-granularity pooling module is used to perform multi-granularity time pooling processing on the time-domain feature vector and the frequency-domain feature vector respectively to form a time-domain multi-granularity representation and a frequency-domain multi-granularity representation. The feature fusion module is used to concatenate the time-domain multi-granularity representation and the frequency-domain multi-granularity representation through channels to generate a fused feature vector; The context-aware weighted module is used to calculate the context-aware attention weights based on the fused feature vectors, and use these weights to perform a weighted summation of the fused feature vectors to generate a context-weighted fusion representation. The abnormal sound determination module is used to input the context-weighted fusion representation into the classification decoder, generate an abnormal sound presence probability value, and determine whether there is an abnormal sound defect based on the comparison result of the probability value and a preset threshold.

[0032] In one implementation example, the audio signal acquisition module includes a high-fidelity microphone array and a preamplifier circuit, with a microphone sensitivity of not less than -46 dB, a signal-to-noise ratio of not less than 70 dB, and a sampling accuracy of 24 bits.

[0033] In one embodiment of the present invention, the residual spectrum encoder, one-dimensional convolutional neural network, multi-granularity pooling module, context-aware weighted module and classification decoder are all deployed in the same deep learning inference engine, which is built based on vector computation graph and supports batch inference and real-time streaming processing.

[0034] As one embodiment of the present invention, the system further includes a model training module, which adopts an unsupervised contrastive learning strategy to train a feature extraction and fusion network on a normal sample set. By minimizing the distribution entropy of normal samples in the embedding space and maximizing the distance between them and the synthetic abnormal samples, implicit modeling of abnormal patterns is achieved.

[0035] The model training employs a supervised learning strategy, with cross-entropy as the loss function. , in For sample labels, For predicting probabilities; During the training phase, a large-scale labeled audio sample set is used, and data augmentation (noise addition, speed adjustment, reverb) is applied to improve the model's generalization ability. In the deployment phase, after model quantization, it can run on embedded industrial terminals to achieve real-time edge detection.

[0036] As one embodiment of the present invention, the synthetic anomalous sample is generated by injecting pulse noise, modulating harmonic distortion or random phase perturbation into a normal audio signal, and the perturbation intensity is controlled within the range of 20 to 40 decibels of signal-to-noise ratio.

[0037] All modules are integrated on the same computing platform, sharing memory and computing resources to ensure efficient data flow and low-latency response. The system supports online updates and hot model replacement to adapt to the noise detection needs of different product models.

[0038] The advantages of this invention are that, by constructing a dual-modal input mechanism of time-domain waveform and frequency-domain spectrogram, it overcomes the shortcomings of existing technologies that rely solely on single frequency-domain features and ignore transient anomaly information in the time domain, thus achieving a complete representation of abnormal sound signals in the time-frequency joint space. The designed residual spectrum encoder effectively enhances the hierarchical expression capability of frequency-domain features and avoids the gradient vanishing problem in deep networks. The multi-granularity time pooling module retains multi-scale anomaly cues from global trends to local mutations, significantly improving the ability to capture weak, short-term abnormal sounds. The context-aware mask weighting mechanism dynamically focuses on the key time frames most discriminative for anomaly detection, suppressing interference from background noise and irrelevant steady-state components, and improving the robustness of the model in complex industrial environments. The overall system adopts an end-to-end trainable architecture, eliminating the need for manual design of acoustic features, and is suitable for general abnormal sound detection scenarios of various electromechanical products. The detection accuracy is improved by more than 12% compared to existing methods, and the false alarm rate is reduced to below 3%, meeting the real-time and consistency requirements of automated quality inspection for high-speed production lines.

[0039] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal abnormal sound detection method based on deep learning, characterized in that: The method includes the following steps: S1. Collect the raw audio signal generated during the operation of the product at the quality inspection station, and perform normalization preprocessing on the raw audio signal. S2. Input the preprocessed audio signal in parallel to the time-domain feature extraction branch and the frequency-domain feature extraction branch to generate the audio embedding vector Faudio and the spectrum embedding vector Fspec, respectively. S3. Input Faudio and Fspec into the multimodal feature fusion module, and perform weighted fusion through learnable weights α to obtain the fused feature vector Ffusion; S4. Input the Flux data into the fully connected classification layer and output the binary classification result through the Softmax function to determine whether the product status is OK or NG. The entire model is trained end-to-end using the Softmax cross-entropy loss function.

2. The multimodal abnormal sound detection method based on deep learning according to claim 1, characterized in that: In the temporal feature extraction branch of S2, the normalized original audio signal is first input into the front-end convolutional module, which consists of several one-dimensional convolutional layers and is used to extract frame-level acoustic features. The frame-level feature sequence is then input into a densely connected time-delay neural network. Each layer of this densely connected time-delay neural network receives all the outputs of the previous layer within the time window [t-Δ, t+Δ] as input. The calculation process is as follows: ， in, Indicates the delay window, This represents the number of network layers.

3. The multimodal abnormal sound detection method based on deep learning according to claim 2, characterized in that: After the densely connected time-delay neural network outputs a frame-level feature sequence, a multi-granularity pooling module is introduced to aggregate context information at different time scales: a global average pooling operation is performed on the entire frame sequence to generate a global context vector G; simultaneously, the frame sequence is divided into several continuous segments, a max pooling operation is performed on each segment, and the max pooling results of each segment are concatenated to generate a segment-level context vector S; finally, the feature vector is generated from the global feature vector G. With local features Formed by splicing: F agg =[G;S].

4. The multimodal abnormal sound detection method based on deep learning according to claim 3, characterized in that: The aggregation context vector F agg The input is a small multilayer perceptron containing one or two hidden layers, with no more than 128 neurons in each layer, and the output dimension is consistent with the length of the frame-level feature sequence. The output of the small multilayer perceptron is a frame-by-frame mask, which is used to perform point-by-point multiplication on the original frame-level features output by the D-TDNN to obtain calibrated features. Then, the audio embedding vector Faudio is generated by global pooling of the calibrated features.

5. The multimodal abnormal sound detection method based on deep learning according to claim 1, characterized in that: In the frequency domain feature extraction branch of S2, the normalized original audio signal is subjected to short-time Fourier transform to obtain a complex time-frequency spectrum. The time-frequency spectrum is then moduloed and logarithmically compressed to obtain a logarithmic amplitude spectrum. This logarithmic amplitude spectrum is then input into a residual convolutional neural network. Finally, global average pooling is performed on the feature map of the last layer of the residual convolutional neural network to generate a spectral embedding vector Fspec.

6. The multimodal abnormal sound detection method based on deep learning according to claim 1, characterized in that: In S3, the multimodal feature fusion module employs a learnable linear weighting strategy, and its fusion formula is as follows: , in The weighting parameter is used to achieve the optimal ratio of time-domain and frequency-domain features.

7. The multimodal abnormal sound detection method based on deep learning according to claim 6, characterized in that: Before weighted fusion, Faudio is linearly projected so that its dimensions match Fspec. Then, Ffusion is calculated by substituting Faudio into the fusion formula.

8. The multimodal abnormal sound detection method based on deep learning according to claim 1, characterized in that: In S4, the fused global features (Ffusion) are input into the fully connected classification layer, and the Softmax function is used to output the probability. , in Check if the corresponding product has any abnormal noises; The classification results are output to the quality control system in probability form when threshold When configurable, the system automatically triggers rejection or manual re-inspection signals to achieve real-time automated screening.

9. A multimodal abnormal sound detection system based on deep learning according to claims 1-8, characterized in that: It includes: The audio signal acquisition module is used to acquire the raw audio signal generated by the device under test under standard operating conditions. An audio preprocessing module is used to preprocess the original audio signal to generate a time-domain waveform sequence and a frequency-domain spectrogram sequence; a time-domain feature extraction module is used to input the time-domain waveform sequence into a one-dimensional convolutional neural network to extract local transient features in the time domain and obtain a time-domain feature tensor. The frequency domain feature extraction module is used to input the frequency domain spectrogram sequence into the residual spectrum encoder to extract deep structural features in the frequency domain and obtain a frequency domain feature tensor. The multi-granularity pooling module is used to perform multi-granularity time pooling processing on the time-domain feature tensor and the frequency-domain feature tensor respectively to form a time-domain multi-granularity representation and a frequency-domain multi-granularity representation. The feature fusion module is used to concatenate the time-domain multi-granularity representation and the frequency-domain multi-granularity representation through channels to generate a fused feature vector; The context-aware weighted module is used to calculate the context-aware attention weights based on the fused feature vectors, and use these weights to perform a weighted summation of the fused feature vectors to generate a context-weighted fusion representation. The abnormal sound determination module is used to input the context-weighted fusion representation into the classification decoder, generate an abnormal sound presence probability value, and determine whether there is an abnormal sound defect based on the comparison result of the probability value and a preset threshold.

Images