Abnormal sound detection method based on multi-scale time-frequency feature perception

Abstract

The invention provides an abnormal sound detection method based on multi-scale time-frequency feature perception, and relates to the technical field of acoustic detection for industrial machine state monitoring, and the method comprises the steps: inputting an original sound signal, and carrying out the dual-branch feature extraction to generate a time domain coding spectrogram and a logarithmic Mel spectrogram; executing hybrid data enhancement; splicing the enhanced spectrogram and inputting the enhanced spectrogram into a dense encoder for compression modeling; extracting features through a two-stage multi-scale time-frequency sensing network, processing along a time dimension in the first stage, and processing along a frequency dimension in the second stage; inputting the high-order features into a lightweight classifier to output an abnormal score; and comparing the abnormal score with the gamma distribution threshold to judge abnormity. According to the method, the weak anomaly detection rate and the cross-equipment stability are remarkably improved, and the perception and discrimination capability of the model on the multi-scale time-frequency characteristics is effectively enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of acoustic detection for industrial machine status monitoring, and in particular to a method for detecting abnormal sounds based on multi-scale time-frequency feature perception. Background Art

[0002] In recent years, with the rapid development of intelligent manufacturing and industrial automation, real-time monitoring of equipment operating status has become a key technical link in ensuring production efficiency and equipment safety. As a non-contact, non-invasive sensing method, the industrial machine abnormal sound detection (Anomalous Sound Detection, ASD) system has advantages that are different from other modal detection methods. For example, when a small amount of gas leaks from a pump, it cannot be accurately detected through conventional visual information, but acoustic detection can be performed by observing the audio patterns of the pump under different operating conditions. By analyzing the changes in acoustic characteristics during machine operation, the ASD system can promptly identify potential equipment failures or performance anomalies, which is of great significance for predictive maintenance and reducing failure losses.

[0003] Currently, most industrial ASD systems implement automated detection based on machine learning methods. The ASD task is one of the subtasks of sound event detection. Due to the difficulty in obtaining real abnormal sound samples, existing ASD models mostly adopt unsupervised learning methods and only use normal sound data for training. According to the different modeling methods, existing methods are mainly divided into two categories: generative methods and feature classification methods. The generative method maps the input sound into a high-dimensional latent feature representation through the encoder, and the decoder then attempts to restore the feature to the original signal. When the reconstruction error exceeds the preset threshold, the sample is judged to be abnormal. The feature classification method uses the metadata of the audio (such as device type, device number, etc.) to build a classifier, directly model the input sound sample, and calculate the anomaly score based on this. Existing studies have shown that by introducing available additional information, the feature classification method has better detection performance than the generative method in multiple practical scenarios.

[0004] Early classifier-based ASD methods primarily relied on traditional models such as Gaussian mixture models (GMMs) or support vector machines (SVMs). In recent years, as deep neural networks (DNNs) have demonstrated superior performance in feature extraction and nonlinear modeling, DNN-based ASD systems have garnered widespread attention and achieved leading detection results in multiple tasks. To improve classification capabilities, Paul et al. modeled the ASD task as a binary classification problem and introduced proxy anomaly samples similar to the target sound to enhance the diversity and sparsity of anomaly samples. For example, a two-stage classifier model was developed that separately classified device type and data augmentation method, calculating an anomaly score by aggregating the two outputs. Another approach, considering the complementarity of time-domain and time-frequency domain features, was to construct a network that integrated these two features to enhance the model's ability to consistently model sound samples from devices with the same ID. For example, to address the model's bias in data augmentation samples, contrastive learning was introduced to optimize ID-level feature representations to enhance the similarity between samples from the same device. For example, a visual Transformer was introduced to model mel-spectrograms, using a cross-region attention mechanism to learn global information to adapt to sound similarity characteristics under different working conditions. Although existing methods have made some progress in improving model performance, they still generally ignore the natural multi-scale distribution characteristics of abnormal sounds in the time domain or time-frequency domain, resulting in unstable detection performance of the model on some devices and insufficient generalization ability. Summary of the Invention

[0005] In order to solve the technical problems in the prior art, the present invention provides an abnormal sound detection method based on multi-scale time-frequency feature perception, aiming to improve the overall detection performance of the system. Referring to the structural design idea of ​​the STgram model, the proposed method adopts a three-level architecture of encoder-deep feature modeling module-classifier. Among them, the encoder part is based on a densely connected convolutional network (DenseNet), which jointly encodes and compresses the original time domain spectrogram and the logarithmic Mel spectrum to reduce the computational complexity while maintaining key information. The deep feature modeling module introduces a two-stage multi-scale perception network, which consists of multiple depth-separable convolutions (Depth-wise Convolution) and point-wise convolutions (Point-wise Convolution) with different receptive fields, enhancing the model's modeling ability for time-frequency features at different scales and capturing the cross-scale dependencies of abnormal sounds. In addition, in order to improve the generalization ability and detection stability of the model, this paper designs a hybrid data enhancement strategy that integrates mix-up and random masking to enhance the model's robustness to morphological changes of abnormal samples.

[0006] The technical solutions provided by the present invention are as follows:

[0007] The present invention provides an abnormal sound detection method based on multi-scale time-frequency feature perception, comprising:

[0008] S1, input the original sound signal of the target machine;

[0009] S2. Perform dual-branch feature extraction on the original sound signal:

[0010] The first branch directly models the signal in the time domain, using a layer of one-dimensional convolution with a large convolution kernel for compression and channel expansion, followed by three layers of two-dimensional convolution to generate a time-domain coded spectrogram;

[0011] The second branch extracts the logarithmic Mel-spectrogram of the signal;

[0012] S3. Perform hybrid data enhancement on the time-domain coded spectrogram and the log-Mel spectrogram:

[0013] A hybrid strategy of linear interpolation at the sample level is adopted;

[0014] A random masking strategy is used to randomly mask local areas of the feature map;

[0015] S4, concatenating the enhanced time-domain coded spectrogram and the log-mel spectrogram along the channel dimension;

[0016] S5. Input the concatenated features into the dense encoder for compression modeling:

[0017] The dense encoder includes an initial convolution module, a densely connected convolutional network module with an expansion structure, and a subsequent convolution module;

[0018] The densely connected convolutional network module is composed of multiple convolutional layers with different expansion rates in a densely connected manner;

[0019] S6. Input the compressed features output by the dense encoder into the two-stage multi-scale time-frequency perception network for feature extraction:

[0020] The first stage processes along the time dimension and includes depthwise separable convolution and channel attention modules;

[0021] The second stage processes along the frequency dimension and includes multiple sets of mixed convolution layers using different sized prime kernels;

[0022] S7, inputting the high-order features output by the two-stage multi-scale time-frequency perception network into a lightweight classifier, and outputting an anomaly score;

[0023] S8. Compare the abnormality score with a preset threshold. If the abnormality score is greater than the threshold, it is determined to be an abnormal state.

[0024] Furthermore, the densely connected convolutional network module of S5 specifically includes:

[0025] There are four convolutional layers with a dilation rate sequence of d = {1, 2, 4, 8}. The output of the first layer satisfies:

[0026] X l =F l (Cat[X0,...,X l-1 ])

[0027] in d l Represents the expansion rate of the lth layer, and the input feature is the concatenation of the outputs of all previous layers.

[0028] Furthermore, the hybrid data enhancement in step S3 specifically includes:

[0029] S301, mixed strategy:

[0030]

[0031] Where x is a sound sample, y is its corresponding machine identifier, λ∈[0,1] is the mixing coefficient sampled from the Beta distribution Beta(α,α), and α is a hyperparameter that controls the degree of mixing;

[0032] S302, random masking strategy:

[0033] X`=X⊙M

[0034] Where the mask matrix M∈{0,1} C×T×S Generated dynamically, zero-value areas represent shaded areas.

[0035] Furthermore, the calculation of the channel attention module of S6 specifically includes:

[0036] X CA =X R1 ⊙δ(PWC(AP(X R1 )))

[0037] Where AP(·) is the average pooling operation, which compresses each channel sequence, δ(·) is the Sigmoid activation function, ⊙ represents element-wise multiplication, and PWC is point-by-point convolution.

[0038] Furthermore, the hybrid convolutional layer of S6 performs the following operations:

[0039] S601. Divide the input features into four groups along the channel dimension:

[0040] x1,x2,x3,x4=chunk(X R2 )

[0041] S602: Perform mixed convolution calculation on each group:

[0042]

[0043] The prime convolution kernel size k=3, 11, 23, 31, Indicates that the convolution kernel size is k i Channel-wise convolution.

[0044] Furthermore, the lightweight classifier training of S7 adopts the additive angular margin loss function, which specifically includes:

[0045]

[0046] Where m is the additive angular margin penalty term, which enhances intra-class compactness by introducing a fixed margin between the feature vector and the class weight vector; s is the scaling factor used to enhance training stability.

[0047] Furthermore, the lightweight classifier of S7 simultaneously minimizes the cross entropy loss function, specifically including:

[0048]

[0049] where y and They are the real machine identification labels and predicted labels of the training sound samples, and the model is optimized by distinguishing the sound feature differences of different subtypes of devices under the same device type.

[0050] Furthermore, a shape transformation operation is performed between the two-stage processing of S6:

[0051] S801, the time dimension features output from the first stage Adjust the shape;

[0052] S802: Use the adjusted features as input for the second-stage frequency dimension processing.

[0053] Furthermore, the preset threshold setting of S8 specifically includes:

[0054]

[0055] Where Γ represents the gamma distribution. Based on the assumption that the anomaly scores of all samples conform to the gamma distribution, the threshold is determined as the 90th percentile of the distribution.

[0056] Furthermore, the channel-by-channel convolution in S602 satisfies:

[0057] Convolution operations are performed only in the time dimension:

[0058] The convolution kernel size in the time dimension is k i, the size in the frequency dimension is 1;

[0059] The convolution kernel with prime scale is used to avoid periodic response and reduce feature redundancy.

[0060] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0061] (1) In this invention, by introducing a multi-scale time-frequency feature perception network and a dense encoder, the overall accuracy of abnormal sound detection is significantly improved. Experimental results show that the average AUC of this method on the DCASE 2020 dataset is 94.52%, which is better than all the mainstream models compared (such as SW-WAVE NET's 93.25%), especially on pumps (93.42%), sliders (99.59%), and toy conveyor belts (83.43%). The performance is the best. This is due to the two-stage multi-scale network (time and frequency dimensions) that effectively captures cross-scale dependencies and enhances the model's ability to perceive subtle abnormal patterns.

[0062] (2) In the present invention, the generalization ability and detection stability of the model are greatly improved by combining a hybrid data enhancement strategy (mix-up and random masking) with multi-scale modeling. Hybrid data enhancement improves the robustness of the model to noise interference by synthesizing rich sample changes (such as linear interpolation and local masking). In terms of mAUC indicators, the average performance reached 87.38%, an increase of 2.52% over STgram-MFN, and the mAUC on the toy conveyor belt device reached 74.64% (an increase of 10.48%), indicating that this method has higher detection consistency between devices of the same type and reduces performance fluctuations.

[0063] (3) In this paper, the additive angular margin loss function (ArcFace) is used to optimize feature discriminability, combined with a lightweight classifier, to effectively enhance intra-class compactness and inter-class separability. This enables the model to perform well in low false positive rate scenarios, with an average pAUC of 88.09%, which is better than all comparison methods (such as Ge Co's 86.43%). At the same time, ablation experiments confirm that removing the multi-scale module or data augmentation will lead to a significant performance degradation (such as a reduction of about 10% in AUC on a toy conveyor belt device), highlighting the necessity of core components. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1The overall structure diagram of the ASD model of an abnormal sound detection method based on multi-scale time-frequency feature perception provided by an embodiment of the present invention;

[0066] Figure 2 A dense encoder structure diagram of an abnormal sound detection method based on multi-scale time-frequency feature perception provided by an embodiment of the present invention;

[0067] Figure 3 A two-stage multi-scale time-frequency perception network structure diagram of an abnormal sound detection method based on multi-scale time-frequency feature perception provided by an embodiment of the present invention;

[0068] Figure 4 t-SNE feature visualization of the slider and pump in the test set of an abnormal sound detection method based on multi-scale time-frequency feature perception provided by an embodiment of the present invention;

[0069] Figure 5 The model structure ablation experiment results of an abnormal sound detection method based on multi-scale time-frequency feature perception provided by an embodiment of the present invention;

[0070] Figure 6 The present invention provides an embodiment of an abnormal sound detection method based on multi-scale time-frequency feature perception and data enhancement ablation experimental results. DETAILED DESCRIPTION

[0071] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0072] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0073] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0074] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0075] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0076] The embodiment of the present invention provides a method for detecting abnormal sounds based on multi-scale time-frequency feature perception. The processing flow may include the following steps:

[0077] S1, input the original sound signal of the target machine;

[0078] S2. Perform dual-branch feature extraction on the original sound signal:

[0079] The first branch directly models the signal in the time domain, using a layer of one-dimensional convolution with a large convolution kernel for compression and channel expansion, followed by three layers of two-dimensional convolution to generate a time-domain coded spectrogram;

[0080] The second branch extracts the logarithmic Mel-spectrogram of the signal;

[0081] S3. Perform hybrid data enhancement on the time-domain coded spectrogram and the log-Mel spectrogram:

[0082] A hybrid strategy of linear interpolation at the sample level is adopted;

[0083] A random masking strategy is used to randomly mask local areas of the feature map;

[0084] S4, concatenating the enhanced time-domain coded spectrogram and the log-mel spectrogram along the channel dimension;

[0085] S5. Input the concatenated features into the dense encoder for compression modeling:

[0086] The dense encoder includes an initial convolution module, a densely connected convolutional network module with an expansion structure, and a subsequent convolution module;

[0087] The densely connected convolutional network module is composed of multiple convolutional layers with different expansion rates in a densely connected manner;

[0088] S6. Input the compressed features output by the dense encoder into the two-stage multi-scale time-frequency perception network for feature extraction:

[0089] The first stage processes along the time dimension and includes depthwise separable convolution and channel attention modules;

[0090] The second stage processes along the frequency dimension and includes multiple sets of mixed convolution layers using different sized prime kernels;

[0091] S7, inputting the high-order features output by the two-stage multi-scale time-frequency perception network into a lightweight classifier, and outputting an anomaly score;

[0092] S8. Compare the abnormality score with a preset threshold. If the abnormality score is greater than the threshold, it is determined to be an abnormal state.

[0093] In an industrial machine condition monitoring task based on sound signal analysis, the system first uses a microphone to collect the sound signal x of the target machine during operation, and then passes the audio signal x as input to the trained abnormal sound detection model A. θ (·), where θ represents the model parameters. The model outputs the anomaly score s of the current sample and sets a threshold φ for comparison. If s>φ, the sample is considered to be abnormal; otherwise, the machine is considered to be in normal operation.

[0094] In actual industrial production environments, due to the randomness and unpredictability of abnormal conditions, abnormal sound samples are often difficult to collect comprehensively, while sound samples under normal working conditions are easy to obtain in large quantities.

[0095] Therefore, the core challenge of the ASD task is that, based on training only on normal sound samples, the model must be able to identify unseen abnormal events, such as leakage, voltage anomalies, blockages, or structural imbalances. The ASD system must learn the "normal mode" of device operation from a training set containing only normal sounds, and identify abnormal samples that do not appear in the training set during the testing phase. The task can be formally described as follows: Given a training sample set X, t ={x i |x i ∈N}, where N represents the distribution of normal sounds. The goal of the model is to learn a function A:x→s that maps any input sample x to an anomaly score s and determines whether it is abnormal based on a set threshold. In the inference phase, for any input sample x, the system calculates its anomaly score:

[0096] s=A θ (x) (1)

[0097] Finally, by setting the judgment threshold φ, abnormal state recognition is achieved:

[0098]

[0099] The core goal of a classifier-based ASD model is to enable the model to identify the distribution characteristics of different working states (sound samples) of the same device type. During training, the model needs to minimize the classification loss function for each type of sound. This process can be expressed as:

[0100]

[0101] Among them, y and are the true labels and predicted labels of the training sound samples, respectively, and CE(·) is the cross entropy loss function (Cross Entropy Loss). The classifier model aims to learn and distinguish the differences in sound features of different sub-type devices under the same device type. For example, for water pump equipment, the category label of the classifier is provided by the metadata information (i.e., machine ID) of each sound sample, such as "ID-00", "ID-01" and "ID-02", which correspond to three water pumps produced by different manufacturers. Due to the differences in the mechanical structure and operating status of each device, its sound performance also has a certain degree of distinctiveness. By learning the feature distribution of different IDs, the classifier can eventually identify the subtle differences between the current sub-type device and other sub-type devices, thereby improving the ability to distinguish abnormal conditions.

[0102] It is understandable that if Figure 1 As shown, the method provided in this embodiment first performs dual-branch feature preprocessing on the input sound signal. Specifically, the dual branches include a TgramNet branch and a Log-Mel Convertor branch. The TgramNet branch directly models the signal in the time domain, uses a one-dimensional convolution with a large convolution kernel for compression and channel expansion, and is followed by three layers of two-dimensional convolution to calculate the time domain coded spectrogram. The Log-Mel Convertor branch is used to extract the time-frequency domain representation of the signal, calculate its log-Mel spectrogram, and send these two spectrograms as joint input to the subsequent network. The model then goes through a hybrid data enhancement module, a dense encoder module, and a two-stage multi-scale time-frequency perception module to deeply model the input features. Among them, the hybrid data enhancement module combines mix-up and random masking strategies to improve the generalization ability of the model. The dense encoder compresses the feature dimension and enhances the representation capability through a dense connection structure. The multi-scale perception module uses a deep convolutional network with different receptive fields to capture the cross-scale correlation of features in stages. Finally, a lightweight classifier based on MobileFaceNet is used to identify the high-order features after modeling and output the abnormality score of the sample.

[0103] Regarding dense encoders, it should be noted that Figure 2 As shown in , the dense encoder consists of an initial convolution module, a DenseNet module with an expansion structure, and a subsequent convolution module cascade. Its overall structure is as follows Figure 3 The encoder can effectively map the input feature representation to a feature space with lower sampling rate but higher dimension, thereby compressing redundant data while retaining key information.

[0104] In this embodiment, it is assumed that a sound sample of length L is input. The sample is processed by TgramNet and Log-Mel Convertor respectively to obtain its time domain coding representation and log-mel spectrogram representation On this basis, the hybrid data enhancement strategy is applied to process the above two features separately to obtain the enhanced feature representation and The Cat[·] operation is used to concatenate the two enhanced features along the 0th dimension (channel dimension):

[0105]

[0106] The spliced ​​representation As the input feature tensor of the dense encoder. In the designed encoder structure, the first and second convolution blocks are composed of a 2D convolution layer (2D Convolution), an instance normalization layer (InstanceNormalization), and a parametric rectified linear unit (PReLU) activation function. The main function of the first convolution block is to increase the feature dimension by adjusting the number of channels in the convolution layer, thereby enhancing the subsequent network's ability to express complex features. The convolution transformation process can be expressed as:

[0107] X C1 =F C1 (X in ) (5)

[0108] Among them, F C1 (·)→PReLU(IN(Conv2D(·))) represents the cascade calculation process in the convolution block, Conv2D(·), IN(·) and PReLU(·) represent the convolution layer, normalization layer and activation function respectively. The convolution block outputs the feature Where C>2.

[0109] The expanded DenseNet module consists of four densely connected convolutional layers, each of which uses a two-dimensional convolution with a dilation rate to expand the receptive field and improve the ability to model multi-scale features. Assuming the input feature The output of the l-th convolutional block is represented as X l , the forward calculation process of this module can be expressed as:

[0110] X l =F l X0,X1,...,X l-1 dim＝0 ,l= (6)

[0111] Among them, F ll(·) represents the nonlinear transformation function of the lth layer, including the convolution layer with dilation rate (dilatedconvolution), instance normalization layer and PReLU activation function. l The specific form of l(·) can be expressed as F l (·)→PReLU(IN(Conv2D dl (·))), where Conv2D dl (·) represents a two-dimensional convolution operation with a dilation rate of dl. The dilation sequence used in this embodiment is d = {1, 2, 4, 8} to increase the receptive field range layer by layer, and dense connections are applied to all convolutional layers. Finally, the module output is:

[0112] X Dense =F l＝4 (Cat[X0,...,X3] dim＝0 ) (7)

[0113] Then, the post-convolution block performs a convolution operation on X along the frequency dimension. Dense Compression is performed to reduce the complexity of subsequent calculations. This process can be expressed as:

[0114] X C2 =F C2 (X Dense ) (8)

[0115] Among them, the compressed features are S=F / 2. F C2 (·) Calculation process and F C1 (·) are the same, but the convolution kernel size and step size parameters of the convolution layer are set differently.

[0116] Regarding the multi-scale video perception network, it should be noted that, inspired by the excellent modeling capabilities of DPRNN, DPTNet, and Prime K-Net in voice front-end tasks, this embodiment proposes a two-stage multi-scale time-frequency perception network. In the first stage, the network models the input features along the time dimension, and in the second stage, it models along the frequency dimension to fully capture the cross-scale dependencies of time-frequency features. Its structure is as follows Figure 3 This module draws on the design concept of the "attention-feedforward" structure in the Transformer and simplifies it: it uses a channel attention mechanism to replace the original multi-head attention module, and introduces multi-scale convolution in the feedforward network to improve modeling efficiency and expressiveness.

[0117] In this embodiment, it is assumed that the input feature is Where B is the batch size (Batchsize). When feeding into the time-frequency perception network, the shape of X needs to be adjusted to Then, a set of depthwise separable convolutions are used to train X T This process can be described as:

[0118] X R1 =F R1 (X T ) (9)

[0119] Among them, F R1 (·)→σ(DWC(PWC(LN(·)))) is the cascade calculation process of layer normalization (Layer Norm, LN(·)), point-wise convolution (Point-wise Convolution, PW Conv, PWC(·)), depth-wise convolution (Depth-wise Convolution, DWConv, DWC(·)) and gate activation function (Sample Gate, σ(·)). Next, the channel attention module is used to perform global information aggregation and inter-channel information interaction:

[0120] X CA =X R1 ⊙δ(PWC(AP(X R1 ))) (10)

[0121] Where δ(·) is the Sigmoid activation function, ⊙ represents element-wise multiplication, and AP(·) is the average pooling calculation. AP(·) compresses each channel sequence and obtains an attention map containing certain global information through PWC(·) and δ(·). Subsequently, the attention map is reapplied to the input to obtain the attention output X CA .

[0122] In the multi-scale convolution module, a hybrid convolution layer consisting of multiple sets of prime convolution kernels of different scales is introduced. This design helps capture transient changes and their frequency characteristics at different time scales, thereby enhancing the model's ability to perceive multi-scale distribution features. By using prime-scale convolution kernels, periodic responses can be effectively avoided, feature redundancy can be reduced, and the accuracy of feature extraction can be improved. By taking the output X of the previous point-by-point convolution of the multi-scale convolution module as the input, the output X of the previous point-by-point convolution of the multi-scale convolution module is effectively R2 Divided into 4 groups along the channel dimension;

[0123] x1,x2,x3,x4=chunk(X R2 ) (11)

[0124] Referring to the structure of the deep gated fusion network, the i-th hybrid convolutional layer is defined as:

[0125]

[0126] in, Indicates that the convolution kernel size is k i The channel-by-channel convolution of the corresponding prime kernel size corresponding to the group features x1, x2, x3, x4 is [a=3, b=11, c=23, d=31]. After the multi-scale convolution module is calculated, the output of the four hybrid convolution layers is spliced ​​along the channel dimension:

[0127] Y=Cat[MC i (a i ,k i )|a i =chunk(X)[i],i={1,2,3,4}] (13)

[0128] The features extracted by the four MC(·) convolutional layers are fused through a point-by-point convolution, and the channel dimensions are rescaled. Subsequently, the output of the time-multiscale perception network is reshaped into an input format suitable for frequency modeling through a shape transformation operation and input into the frequency-multiscale perception network to further extract global time-frequency perception features.

[0129] Regarding mixed data augmentation, it should be noted that in order to enhance the generalization ability of the model, a data augmentation method combining mix-up and random masking strategies is introduced during the training phase. Mix-up is a data augmentation strategy that performs linear interpolation at the sample level. Its basic idea is to perform a weighted mix of two samples and their labels to generate new training samples, thereby reducing the overfitting of the model to a single feature. Assume that the two input samples are represented as (x i ,y i ) and (x j ,y j ), then the sample after mix-up It can be expressed as:

[0130]

[0131] Where x is a sound sample, y is its corresponding machine ID, λ∈[0,1] is a mixing coefficient sampled from the Beta distribution Beta(α,α), and the hyperparameter α is used to control the degree of mixing. The random masking strategy is a data augmentation strategy that randomly masks local areas of the feature map, which can improve the model's robustness to local perturbations. During the training process, it randomly sets a part of the input feature map to zero to prevent the model from over-relying on local salient areas. Assume that the input feature is Randomly generate a mask matrix M∈{0,1} C×T×S , the enhanced features are:

[0132]

[0133] The zero-value area in the mask M represents the masked part, and the masking position is random for each sample. T and X M The hybrid data enhancement process can be expressed as:

[0134]

[0135] Because ASD models typically only use normal samples during training, they are prone to overfitting to a limited distribution of normal samples. Therefore, a hybrid data augmentation strategy can be used to synthesize richer sample variation patterns, thereby enhancing the model's adaptability to different sound perturbations and its ability to detect short-term or weak abnormal signals. In ASD tasks, audio samples of the same machine type exhibit significant intra-class variation (e.g., recordings at different times or under different operating conditions). Hybrid data augmentation helps improve the model's ability to consistently discriminate between different normal samples under the same ID.

[0136] Regarding the loss function, it should be noted that during the model training phase, this embodiment uses the ArcFace loss function (Additive Angular Margin Loss) to enhance the discriminative ability of the model. This loss function significantly improves the compactness of intra-class features and the separability of inter-class features by introducing an additive angular margin between the feature vector and the class weight vector. The ArcFace loss of the input vector x can be expressed in vector form as:

[0137]

[0138] Where y=[y1,...,y K ] T is the one-hot vector of the true label of sample x, θ=[θ1,...,θ K ] T Represents the angle between the sample and other categories. It is worth noting that the ArcFace loss function only works when the sample’s true label y k =1, an additive angular margin penalty m is introduced between the corresponding feature vector and the category weight vector to promote a compact distribution of features within the category. This margin mechanism further widens the decision boundary between categories by replacing the angle θ with (θ+m·y). At the same time, the hyperparameter s is used to scale the modified log-probability term cos(θ+m·y) to enhance the numerical stability of model training. The core goal of introducing the angular margin is to obtain a more compact distribution of features within the category and reserve a larger discriminant area in the feature space for abnormal samples that have not been seen during training, effectively improving the model's ability to recognize abnormal states.

[0139] In this embodiment, the experiment uses the abnormal sound detection dataset provided by DCASE 2020 Challenge Task 2, and all audio samples are collected in a real industrial environment. The dataset covers six types of typical industrial equipment, namely pumps, fans, sliders, valves, toy cars, and toy conveyors. Except for the toy conveyor, whose device identification number (ID) is 1 to 6, the ID range of the other five types of equipment is 0 to 6. During the model training phase, the dataset only contains sound samples under normal conditions; while in the verification and testing phase, the data is composed of a mixture of normal and abnormal samples to evaluate the anomaly detection performance of the model under unsupervised conditions. The dataset information is shown in Table 1.

[0140] Table 1 Dataset composition

[0141]

[0142] Model performance is evaluated using the area under the receiver operating characteristic curve (AUC) and its partial area (pAUC). pAUC represents the AUC value within a low false positive rate (FPR) range, specifically calculated in the interval [0, p). p＝0.1 In addition, to measure the worst detection effect of each sub-device instance in the same device type, the minimum AUC (mAUC) is introduced as a supplementary evaluation indicator to reflect the consistency and stability of the system's detection performance between different sub-devices. The mathematical definitions of AUC, pAUC and mAUC are:

[0143]

[0144] Among them, C(x) returns 1 when x>0, and returns 0 in other cases. + and A - is the score of normal and abnormal sounds, N + and N - Represents the number of samples of normal sound and abnormal sound respectively. Assuming that the abnormal scores of all samples conform to the gamma distribution, the detection threshold is determined as the 90th percentile of the gamma distribution. In pAUC, is the lower limit function.

[0145] During model training, all audio samples were uniformly resampled to 16kHz, and the audio duration was fixed at 10 seconds. Log-mel spectrograms were extracted using a frame length of 1024 points and a frame shift of 512 points (i.e., 50% frame overlap). The number of mel filter banks was set to 128. Therefore, the log-mel spectrogram size of each 10-second sample was 128 × 313. Model training used the Adam optimizer for parameter updates, with an initial learning rate of 0.0001, a total of 200 training epochs, and a batch size of 64.

[0146] To verify the effectiveness of the proposed method, this embodiment compares its performance with several mainstream abnormal sound detection methods in recent years, covering the following models: Baseline: taking the log-Mel spectrogram as the input feature, using the traditional classifier for anomaly detection; Log-Mel (U-Net reconstruction): using the fully connected U-Net network to reconstruct the non-stationary audio signal, and extracting the reconstruction error as the anomaly indicator; IDNN: reducing the reconstruction error by interpolating the prediction of the center frame in the spectrogram; MobileNetV2: integrating the density estimation-based anomaly detector into the ASD system; Glow-Aff: introducing the normalization flow to achieve accurate likelihood estimation and assigning a higher probability to the target machine sound; STgram-MFN: using one-dimensional convolution to generate the time domain feature spectrogram of the signal from the time domain modeling; GeCo: a generative model with self-attention mechanism for frame-level prediction; SW-WAVENET: using the WaveNet model to extract the waveform domain features of the audio; IDC-TransAE: designing a Transformer autoencoder network combined with ID constraints for anomaly detection. Different from the above methods, the method of this embodiment combines the natural multi-scale distribution characteristics of speech spectrum features in the time and frequency dimensions to construct a two-stage multi-scale modeling framework, thereby achieving more accurate anomaly identification.

[0147] The experimental results on the DCASE 2020Challenge Task 2 dataset are shown in Table 2. The average AUC performance of the method proposed in this embodiment on all machine types reached 94.52%, which is better than all existing methods compared. Among the specific machine types, the method of this embodiment achieved the best performance on pumps, sliders, and toy conveyors, especially on toy conveyors, achieving an AUC score of 83.43%, which is 2.23% higher than the SW-WAVENET method. This result shows that the method of this embodiment still has good anomaly detection capabilities in the face of complex audio backgrounds with weak and discontinuous abnormal features. This performance advantage is mainly attributed to the effective synergy of the proposed two-stage multi-scale time-frequency perception network in local detail features and global structure modeling.

[0148] Table 2 AUC (%) comparison on the test set, bold font is the best value under the corresponding machine

[0149]

[0150] Table 3 shows the experimental results for the pAUC metric. The method in this embodiment achieved an average pAUC of 88.09% across all machine types, outperforming the other comparison methods and demonstrating strong overall detection performance. The proposed method achieved the best pAUC values ​​on fan and slider devices, and its performance on other devices was also near-optimal. These results demonstrate that, under the set low false positive rate threshold, the method in this embodiment exhibits greater stability and robustness in anomaly detection across different machine types.

[0151] Table 3. Comparison of pAUC (%) on the test set. The bold font represents the best value under the corresponding machine.

[0152]

[0153] Table 4 shows the mAUC indicators of different models on the test set. mAUC reflects the worst detection performance achieved by a single machine of the same machine type. In Tables 2 and 3 above, MobileNetV2 outperforms IDNN in AUC and pAUC indicators, but its average mAUC value is 59.73%, which is lower than IDNN's 64.46%, indicating that MobileNetV2 has large fluctuations in performance across devices and poor stability. In contrast, the method of this embodiment shows a significant improvement in mAUC indicators compared to STgram-MFN, with an average mAUC of 87.38%, 2.52% higher. Specifically, the mAUC on the toy conveyor (ToyConveyor) device is 74.64%, significantly better than STgram-MFN's 64.16%, an increase of 10.48%. This result shows that the method of this embodiment can achieve more consistent detection performance between machines of the same type while maintaining high overall detection performance, reflecting higher stability.

[0154] Table 4 Comparison of mAUC (%) on the test set. The bold font is the best value under the corresponding machine.

[0155]

[0156]

[0157] Furthermore, this example performs a visualization analysis of the potential features of the slider and pump data in the test set, and clusters the features learned by the model using the t-distributed stochastic neighbor embedding (t-SNE) method. The results are as follows: Figure 4 As shown. Figure 4 (a) It can be seen that on the slider device, the distribution of normal and abnormal samples of different machine IDs is relatively scattered, and the overlap area between normal and abnormal samples is small, indicating that the model has a higher detection accuracy on this type of device. Figure 4 (b) shows that for pump device ID-9, there is a certain degree of overlap between normal and abnormal samples, which may cause the model to misjudge anomalies on this device. However, overall, the model's feature distribution for other machine IDs is clearly distinct, indicating that it maintains high detection performance on most devices.

[0158] To validate the effectiveness of the proposed model, this section conducts ablation experiments analyzing the main model structures and data augmentation strategies. In the model structure ablation experiments, the dense encoder and multi-scale module are removed to evaluate the contribution of each module to the overall performance. In the data augmentation ablation experiments, the effects of different data augmentation methods on the model's generalization ability are analyzed using only mix-up, only random masking, and a combination of both.

[0159] The results of the model structure ablation experiment are as follows: Figure 5 As shown in the figure, building on the baseline model, the introduction of either a dense encoder or a multi-scale module improves detection performance across different machine types, with a particularly significant improvement on toy conveyor belt machines. Because abnormal sounds from toy conveyor belt machines are brief and subject to significant background noise during operation, the baseline method struggles to effectively detect abnormal signals. With the complete model structure, the model's average detection performance across the six machine types is further improved, and performance differences between different machine types are significantly reduced, demonstrating that the proposed method offers improved detection accuracy and stability.

[0160] The ablation experiment results of the data enhancement method are as follows Figure 6 As shown in the figure, introducing either a mix-up or random masking strategy into the baseline model improved overall detection performance, particularly on fans and toy conveyor belts, demonstrating that data augmentation methods can effectively improve the model's feature representation capabilities. Furthermore, the hybrid data augmentation strategy further improved the model's performance on various machine types, demonstrating that hybrid data augmentation can help the model learn richer acoustic feature representations, thereby effectively enhancing the model's generalization and detection stability.

[0161] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0162] (1) In this invention, by introducing a multi-scale time-frequency feature perception network and a dense encoder, the overall accuracy of abnormal sound detection is significantly improved. Experimental results show that the average AUC of this method on the DCASE 2020 dataset is 94.52%, which is better than all the mainstream models compared (such as SW-WAVE NET's 93.25%), especially on pumps (93.42%), sliders (99.59%), and toy conveyor belts (83.43%). The performance is the best. This is due to the two-stage multi-scale network (time and frequency dimensions) that effectively captures cross-scale dependencies and enhances the model's ability to perceive subtle abnormal patterns.

[0163] (2) In the present invention, the generalization ability and detection stability of the model are greatly improved by combining a hybrid data enhancement strategy (mix-up and random masking) with multi-scale modeling. Hybrid data enhancement improves the robustness of the model to noise interference by synthesizing rich sample changes (such as linear interpolation and local masking). In terms of mAUC indicators, the average performance reached 87.38%, an increase of 2.52% over STgram-MFN, and the mAUC on the toy conveyor belt device reached 74.64% (an increase of 10.48%), indicating that this method has higher detection consistency between devices of the same type and reduces performance fluctuations.

[0164] (3) In this paper, the additive angular margin loss function (ArcFace) is used to optimize feature discriminability, combined with a lightweight classifier, to effectively enhance intra-class compactness and inter-class separability. This enables the model to perform well in low false positive rate scenarios, with an average pAUC of 88.09%, which is better than all comparison methods (such as Ge Co's 86.43%). At the same time, ablation experiments confirm that removing the multi-scale module or data augmentation will lead to a significant performance degradation (such as a reduction of about 10% in AUC on a toy conveyor belt device), highlighting the necessity of core components.

[0165] The above content is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

[0166] There are a few points to note:

[0167] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0168] (2) For the sake of clarity, the thickness of layers or regions in the drawings used to describe the embodiments of the present invention are exaggerated or reduced, that is, these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or intervening elements may be present.

[0169] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to form new embodiments.

[0170] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting abnormal sound based on multi-scale time-frequency feature perception, characterized in that: include: S1, input the original sound signal of the target machine; S2. Perform dual-branch feature extraction on the original sound signal: The first branch directly models the signal in the time domain, using a layer of one-dimensional convolution with a large convolution kernel for compression and channel expansion, followed by three layers of two-dimensional convolution to generate a time-domain coded spectrogram; The second branch extracts the logarithmic Mel-spectrogram of the signal; S3. Perform hybrid data enhancement on the time-domain coded spectrogram and the log-Mel spectrogram: A hybrid strategy of linear interpolation at the sample level is adopted; A random masking strategy is used to randomly mask local areas of the feature map; S4, concatenating the enhanced time-domain coded spectrogram and the log-mel spectrogram along the channel dimension; S5. Input the concatenated features into the dense encoder for compression modeling: The dense encoder includes an initial convolution module, a densely connected convolutional network module with an expansion structure, and a subsequent convolution module; The densely connected convolutional network module is composed of multiple convolutional layers with different expansion rates in a densely connected manner; S6. Input the compressed features output by the dense encoder into the two-stage multi-scale time-frequency perception network for feature extraction: The first stage processes along the time dimension and includes depthwise separable convolution and channel attention modules; The second stage processes along the frequency dimension and includes multiple sets of mixed convolution layers using different sized prime kernels; S7, inputting the high-order features output by the two-stage multi-scale time-frequency perception network into a lightweight classifier, and outputting an anomaly score; S8. Compare the abnormality score with a preset threshold. If the abnormality score is greater than the threshold, it is determined to be an abnormal state.

2. The abnormal sound detection method based on multi-scale time-frequency feature perception according to claim 1 is characterized in that: The densely connected convolutional network module of S5 specifically includes: There are four convolutional layers with a dilation rate sequence of d = {1, 2, 4, 8}. The output of the first layer satisfies: X l =F l (Cat[X0,...,X l-1 ]) Among them F l (·)=PReLU(IN(Conv2D dl (·))), d l Represents the expansion rate of the lth layer, and the input feature is the concatenation of the outputs of all previous layers.

3. The abnormal sound detection method based on multi-scale time-frequency feature perception according to claim 1 is characterized in that: The hybrid data enhancement in step S3 specifically includes: S301, mixed strategy: Where x is a sound sample, y is its corresponding machine identifier, λ∈[0,1] is the mixing coefficient sampled from the Beta distribution Beta(α,α), and α is a hyperparameter that controls the degree of mixing; S302, random masking strategy: Where the mask matrix M∈{0,1} C×T×S Generated dynamically, zero-value areas represent shaded areas.

4. The abnormal sound detection method based on multi-scale time-frequency feature perception according to claim 1 is characterized in that: The calculation of the channel attention module of S6 specifically includes: X CA =X R1 ⊙δ(PWC(AP(X R1 ))) Where AP(·) is the average pooling operation, which compresses each channel sequence, δ(·) is the Sigmoid activation function, ⊙ represents element-wise multiplication, and PWC is point-by-point convolution.

5. The abnormal sound detection method based on multi-scale time-frequency feature perception according to claim 1 is characterized in that: The mixed convolution layer of S6 performs the following operations: S601. Divide the input features into four groups along the channel dimension: x1,x2,x3,x4=because(X R2 ) S602: Perform mixed convolution calculation on each group: MC i (x i ,k i )=PWC(DWC ki (x i ))⊙DWC ki (x i ) The prime convolution kernel size k=3, 11, 23, 31, Indicates that the convolution kernel size is k i Channel-wise convolution.

6. The abnormal sound detection method based on multi-scale time-frequency feature perception according to claim 1 is characterized in that: The S7 lightweight classifier training adopts the additive angular margin loss function, which specifically includes: Where m is the additive angular margin penalty term, which enhances intra-class compactness by introducing a fixed margin between the feature vector and the class weight vector; s is the scaling factor used to enhance training stability.

7. The abnormal sound detection method based on multi-scale time-frequency feature perception according to claim 1 is characterized in that: The S7 lightweight classifier simultaneously minimizes the cross entropy loss function, specifically including: where y and They are the real machine identification labels and predicted labels of the training sound samples, and the model is optimized by distinguishing the sound feature differences of different subtypes of devices under the same device type.

8. The abnormal sound detection method based on multi-scale time-frequency feature perception according to claim 1 is characterized in that: The shape transformation operation is performed between the two-stage processing of S6: S801, the time dimension features output from the first stage Adjust the shape; S802: Use the adjusted features as input for the second-stage frequency dimension processing.

9. The abnormal sound detection method based on multi-scale time-frequency feature perception according to claim 1, characterized in that: The preset threshold setting of S8 specifically includes: Where Γ represents the gamma distribution. Based on the assumption that the anomaly scores of all samples conform to the gamma distribution, the threshold is determined as the 90th percentile of the distribution.

10. The abnormal sound detection method based on multi-scale time-frequency feature perception according to claim 5, characterized in that: The channel-by-channel convolution of S602 satisfies: Convolution operations are performed only in the time dimension: The convolution kernel size in the time dimension is k i , the size in the frequency dimension is 1; The convolution kernel with prime scale is used to avoid periodic response and reduce feature redundancy.

Images