Self-supervised machine abnormal sound detection method based on dual-path WaveNet

Abstract

The invention provides a self-supervised machine abnormal sound detection method based on a dual-path WaveNet, and the method comprises the steps: 1, carrying out the modeling of frequency information of each time frame of an input sound signal in a frequency domain path through employing the WaveNet, and extracting a frequency domain feature; 2, the time domain path takes the output # imgabs0 # of the frequency domain path as the input, and modeling is carried out on the time dimension after transposition is carried out on the time domain path; 3, performing further feature fusion on the output # imgabs1 # of the frequency domain path and the time domain path through a two-dimensional convolutional layer to generate a one-dimensional embedded vector; 4, adopting a self-supervision strategy, and utilizing metadata to construct a classification task so as to train a model; and 5, in a detection stage, the trained model judges whether the model is normal or not according to the abnormal score # imgabs2 of the model test sample. According to the dual-path design, comprehensive modeling of the time-frequency spectrogram is achieved, local details of time and frequency dimensions are reserved, frequency domain harmonic information and time domain dynamic characteristics can be effectively captured, and the problem that complex characteristic modeling in the time-frequency spectrogram is insufficient is solved.

Description

Technical Field

[0001] The present application relates to the technical field of sound signal processing, and in particular to a self-supervised machine abnormal sound detection method based on dual-path WaveNet. Background Art

[0002] As a carrier of information transmission, sound contains rich information. Detection based on acoustic signals has become an indispensable technical means in many fields. In industrial production, by monitoring and analyzing the sound generated by the machine during operation, not only can potential equipment failures or problems be discovered in time, but also it provides a valuable opportunity for taking timely maintenance measures. This operation method helps to reduce equipment downtime and slow down equipment loss, thereby significantly improving overall production efficiency. At the same time, by preventing potential problems, it can also extend the service life of the equipment, reduce the need for unplanned maintenance, and effectively reduce production costs. The abnormal sound detection technology of the machine is not only of great significance in terms of economic benefits, but also has a direct positive impact on the safety of the workplace. Timely detection of abnormal conditions in the operation of the equipment allows early warning of potential safety risks. By preventing accidents and reducing equipment failures, this technology helps to create a safer working environment and effectively protect employees from potential injuries.

[0003] Traditional machine abnormal sound detection methods mainly rely on professional experience and rules, and judge whether the sound is abnormal by setting thresholds or using rules. With the rise of machine learning and deep learning technologies, data-driven methods have gradually become mainstream. By collecting a large amount of sound data and training it with machine learning algorithms, abnormal sounds can be more accurately identified and classified. The new generation of abnormal sound detection systems, especially deep learning models such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), can extract abstract features from large-scale and complex sound data, achieve efficient abnormal sound recognition, and improve the accuracy and adaptability of detection. Researchers collect normal and abnormal samples, extract signal features, and then use machine learning methods to extract the features of these samples to achieve abnormal sound detection.

[0004] Although the abnormal sound detection technology has made significant progress, it is relatively difficult to obtain abnormal sounds in actual environments. Even if abnormal sound samples are obtained by deliberately creating anomalies, the data obtained is often not enough to cover all possible abnormal situations; especially for some situations where machines need to be deliberately damaged to collect abnormal sounds, this method is not only costly but also impractical. Faced with the extreme imbalance of data, the use of mainstream supervised learning methods for abnormal sound detection will result in high accuracy for normal sounds but a large number of abnormal sounds will be missed. Therefore, unsupervised anomaly detection is more realistic. It does not need to add normal and abnormal labels to the data. It only uses the collected daily operation sounds of the machine to judge unknown abnormal sounds based on the distribution characteristics of this information, thereby achieving effective anomaly detection. This method avoids the problems of scarcity of abnormal samples and data imbalance, and provides a more practical solution for the anomaly detection system, making it more flexible to adapt to different actual industrial scenarios.

[0005] To this end, the researchers proposed an unsupervised ASD method that uses only normal sound data during the training phase, allowing the model to learn the patterns and characteristics of normal sounds. Figure 1 As shown in Figure 1, during the detection phase, the model determines whether the input sound is abnormal by calculating the degree of deviation from the normal pattern. Specifically, the model calculates an anomaly score for each input sound. and compare it with the preset threshold When the anomaly score exceeds the threshold, the input sound is judged to be abnormal.

[0006] The main strategies for achieving unsupervised abnormal sound detection include two categories: reconstruction-based methods and self-supervision-based methods. Reconstruction-based methods usually use autoencoders to reconstruct the input sound feature map and use the reconstruction error as the anomaly score. During testing, the test sample is judged to be normal or abnormal by comparing the anomaly score with the threshold. However, this type of method may encounter problems when dealing with non-stationary environments, and the reconstruction error of normal sounds is sometimes higher than that of abnormal sounds. To solve this problem, researchers have proposed a self-supervision-based method. This method uses certain attributes in the data to set up a proxy task, and learns the representation of normal signals from the data by solving supervised proxy tasks. For example, the relevant information of the audio file, such as the machine type or machine ID, is used as a label to train the model to classify it, so that the model learns a more compact feature representation for each category. During testing, the true category probability value predicted by the model is used as the anomaly score, and whether the anomaly score exceeds the threshold is judged to measure whether the sample is abnormal.

[0007] like Figure 2As shown in the figure, the self-supervised abnormal sound detection method designs a predefined pseudo-supervised task so that the model can learn the characteristic distribution of normal sounds without real labels. In the training phase, metadata (such as device category or environmental information) is usually used as pseudo-labels to construct classification tasks so that the model can learn the discrimination boundary between normal sounds. In the detection phase, the input sound is detected for anomalies based on the classification probability of the model. Normal sounds are usually classified with high confidence, while abnormal sounds have low classification confidence due to their deviation from the normal distribution. When the anomaly score is higher than the set threshold, the sample is judged as abnormal.

[0008] The time-spectrogram can intuitively represent the frequency components and time changes of sound signals in the form of a two-dimensional image, thereby capturing the complex time-frequency structural characteristics of the sound. Sound signals are usually non-stationary, and their frequency intensity changes dynamically over time. The time-spectrogram can well describe these characteristics, so that the difference between normal and abnormal sounds can be fully reflected. Compared with directly processing the one-dimensional original waveform, the time-spectrogram can more clearly reveal the frequency distribution, energy changes and harmonic structure unique to abnormal sounds. In addition, the graphical features of the time-spectrogram make it possible to directly combine the model in deep learning to extract deep features from it, greatly improving the detection performance. The time-spectrogram not only enriches the feature information, but also improves the model's recognition ability and robustness for abnormal sounds.

[0009] However, most current methods usually treat the time-frequency spectrogram as ordinary image processing when extracting features, and use two-dimensional convolutional neural networks for modeling. Due to the limited receptive field of two-dimensional convolutional networks, this limitation makes it difficult to capture specific information that is critical for anomaly detection when processing time-frequency spectrograms. On the one hand, the harmonic structure in the frequency domain often contains important feature information. For example, the harmonics of normal sounds usually present a stable and regular frequency distribution, while abnormal sounds may break this regularity. However, due to the fixed receptive field, two-dimensional convolutional networks are difficult to effectively capture these complex harmonic relationships across frequency bands. On the other hand, global changes in the time domain, such as the rhythm, timing patterns, and dynamic characteristics of sound signals, are also important for anomaly detection. However, two-dimensional convolutional networks tend to focus on the characteristics of local areas and cannot effectively model long-distance dependencies or dynamic change trends in the time dimension. Therefore, relying solely on two-dimensional convolutional networks to process time-frequency spectrograms may lead to the neglect of important global information, thereby affecting the accuracy and robustness of anomaly detection.

[0010] To this end, some studies have tried to use Transformer and WaveNet to model information of specific dimensions. With its powerful global modeling capabilities and parallel processing efficiency, Transformer can effectively capture long-distance dependencies in the frequency domain or time domain through the self-attention mechanism. However, Transformer often requires a lot of computing resources when processing time-frequency spectrograms, and its ability to capture local features is relatively limited, making it difficult to fully retain frequency details or subtle changes between time frames.

[0011] like Figure 3 As shown in Figure 1, WaveNet is a one-dimensional convolutional neural network designed for generating raw audio, and it has demonstrated remarkable capabilities in signal feature extraction. Its structure is based on residual blocks, combining dilated convolutions, gating units, residual connections, and skip connections, enabling the model to efficiently capture long-distance dependencies and retain local information. Dilated convolutions gradually expand the receptive field, enabling the model to extract multi-scale features without increasing computational complexity. Residual connections enhance gradient flow and feature learning, while gating units further optimize model performance by combining sigmoid and tanh functions to control the flow of information.

[0012] In contrast, WaveNet gradually expands the receptive field through dilated convolution, which can efficiently capture long-distance dependencies while retaining local information. However, traditional WaveNet is only applicable to one-dimensional signal processing. When faced with two-dimensional time-frequency spectrograms, it is difficult to directly model the complex correlation between time and frequency. Although existing methods have attempted to expand WaveNet to model frequency domain information, this single-dimensional processing method cannot fully reflect the global characteristics of the time-frequency spectrogram, especially in anomaly detection tasks, where harmonic features in the frequency domain and dynamic changes in the time domain are often inseparable. Summary of the invention

[0013] In view of this, the present application provides a self-supervised machine abnormal sound detection method based on dual-path WaveNet, which realizes the gradual extraction of time domain and frequency domain features while effectively maintaining the correlation between time and frequency domain information.

[0014] To achieve the above objectives, the technical solutions adopted in this application are as follows: A self-supervised machine abnormal sound detection method based on dual-path WaveNet, comprising: Step 1: Use WaveNet in the frequency domain path to model the frequency information of each time frame of the input sound signal and extract frequency domain features: ; in The WaveNet mapping function representing the frequency domain path, Represents input data The dimension is , is the output frequency domain feature; Step 2: Output of the time domain path to the frequency domain path As input, after transposing it, it is considered as The length is Then, WaveNet is used to model the time dimension and the output is the time domain feature. : ; in WaveNet mapping function representing the time domain path; Step 3: Output of frequency domain path and time domain path The features are further fused through a two-dimensional convolutional layer to generate a one-dimensional embedding vector; Step 4: Adopt a self-supervised strategy and use metadata to build a classification task to train the model. The classification loss function is ArcFace Loss. Step 5: After the training model, in the detection phase, the model calculates the anomaly score A through the classification probability. If the anomaly score A of the test sample exceeds the set threshold, it is judged as abnormal; otherwise, it is judged as normal.

[0015] Furthermore, the calculation method of the classification loss function ArcFace Loss in step 4 is: Step 4.1: Normalize the one-dimensional embedding vector and the category weight vector; Step 4.2: Obtain the angle value for each category by calculating the cosine similarity between the normalized one-dimensional embedding vector and the category weight vector ; ; in, represents a one-dimensional embedding vector, represents the class weight vector, Represents the input sample With category The angle between Step 4.3: Introduce the angle interval m and calculate the true category The cosine of the angle ; Step 4.4: Introducing a scaling factor , calculate the final logits value , the calculation formula is: ; Step 4.5: Use the Softmax function to convert the logits value Converted into probability distribution, the conversion formula is: ; in, Represents the model prediction sample Belongs to the real category Prediction probability, Yes Category The logits value of Step 4.6: Use the cross entropy loss function to calculate the ArcFace loss. The specific calculation formula is: ; in, represents the batch size, is the number of categories, Indicates The true category labels of samples, Indicates The angle between the one-dimensional embedding vector of a sample and its true label vector, and Represents the input sample With category The angle between is the additional angular margin parameter, is the scaling factor.

[0016] Furthermore, the calculation method of the anomaly score A in step 5 is: ; in, Represents the model prediction sample Belongs to the real category Predicted probability.

[0017] Furthermore, before step 1, the step further includes: The input one-dimensional audio signal is transformed into a time-frequency spectrum. The specific transformation formula is: ; in, is the Log-Mel spectrum, represents the Mel filter bank, The length of the input is One-dimensional audio signal, STFT represents short-time Fourier transform, which converts the audio signal into a one-dimensional Spectrum of represents the number of time frames, represents the number of frequency bands, Indicates that the STFT spectrum is filtered by the Mel filter bank M, where is the number of Mel filters.

[0018] Compared with the prior art, the beneficial effects of this application are: 1. The dual-path design of this application realizes comprehensive modeling of the time-frequency spectrum. In the frequency dimension, the frequency domain path can efficiently extract the frequency features within each time frame; in the time dimension, the time domain path further integrates the frequency domain information to capture the long-distance dependencies and global dynamic changes of the signal; compared with the problem of limited receptive field of traditional two-dimensional convolutional networks, the dual-path WaveNet gradually expands the receptive field through dilated convolution, which not only retains the local details of the time and frequency dimensions, but also effectively captures the frequency domain harmonic information and time domain dynamic characteristics, thereby solving the problem of insufficient modeling of complex features in the time-frequency spectrum; 2. In the abnormal sound detection task, this application uses dual-path WaveNet combined with a self-supervised learning mechanism to enable the model to deeply explore the characteristic distribution of normal sounds. This method does not rely on a large amount of labeled data, and can learn the discrimination boundary of normal sounds through pseudo-label classification tasks; in the detection stage, abnormal sounds deviate from the normal distribution and show lower classification confidence, so they are accurately identified; compared with traditional two-dimensional convolutional networks or single-path methods, the detection accuracy and robustness of this application in abnormal sound detection are significantly improved, especially in unsupervised or pseudo-supervised scenarios; 3. This application introduces ArcFace Loss to optimize the classification target, making the features of normal samples more concentrated and the distribution of abnormal samples in the feature space more discrete, thereby improving the model's ability to distinguish normal and abnormal samples; this feature enables anomaly detection to maintain a high generalization performance between normal sounds of different categories, further enhancing the applicability of the technical solution; 4. This application overcomes the key bottleneck problems of existing methods in time-frequency spectrum feature extraction and anomaly detection through the technical solution of dual-path WaveNet, and provides an efficient and accurate solution for abnormal sound detection. Its innovation and practicality are not only reflected in the deep mining of complex features, but also provide important technical support for the field of unsupervised anomaly detection, which has broad application value and promotion significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a schematic diagram of the unsupervised ASD method in the background technology; Figure 2 It is a schematic diagram of the principle of the self-supervised abnormal sound detection method in the background technology; Figure 3 It is the structural diagram of the WaveNet model in the background technology; Figure 4 A flow chart of a self-supervised machine abnormal sound detection method based on dual-path WaveNet is provided for this application; Figure 5 This application provides a schematic diagram of a self-supervised machine abnormal sound detection method based on dual-path WaveNet. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0022] like Figure 4 and 5 As shown, a self-supervised machine abnormal sound detection method based on dual-path WaveNet includes: Step 1: Use WaveNet in the frequency domain path to model the frequency information of each time frame of the input sound signal and extract frequency domain features; Specifically, when the input data is a time spectrum diagram, and , which can be regarded as The length is A one-dimensional subsequence of and Represent the number of time frames and frequency points respectively; then, the WaveNet of the frequency domain path models the frequency information in each frame and outputs it as frequency domain features : ; in The WaveNet mapping function representing the frequency domain path, Represents input data The dimension is ; The frequency domain path captures information in the frequency dimension and extracts key features related to anomaly detection.

[0023] Step 2: Output of the time domain path to the frequency domain path As input, after transposing it, it is considered as The length is Then, WaveNet is used to model the time dimension and the output is the time domain feature. : ; in WaveNet mapping function representing the time domain path, which integrates the extracted frequency information in time.

[0024] Step 3: Output of frequency domain path and time domain path The features are further fused through a two-dimensional convolutional layer to generate a one-dimensional embedding vector; Step 4: Adopt a self-supervised strategy and use metadata to build a classification task to train the model. The classification loss function is ArcFace Loss. The difference between the predicted probability distribution of the model and the true category distribution is measured by calculating the ArcFace loss, where ArcFace (Additive Angular Margin) represents the additive angular margin.

[0025] Step 5: After the training model, in the detection phase, the model calculates the anomaly score A through the classification probability. If the anomaly score A of the test sample exceeds the set threshold, it is judged as abnormal; otherwise, it is judged as normal.

[0026] Furthermore, the calculation method of the classification loss function ArcFace Loss in step 4 is: Step 4.1: Normalize the one-dimensional embedding vector and the category weight vector; Step 4.2: Obtain the angle value for each category by calculating the cosine similarity between the normalized one-dimensional embedding vector and the category weight vector ; ; in, represents a one-dimensional embedding vector, represents the class weight vector, Represents the input sample With category The angle between Step 4.3: Introduce the angle interval m and calculate the true category The cosine of the angle ; This step can force the model to increase the interval between categories in the feature space and improve the distinction between categories.

[0027] Step 4.4: Introducing a scaling factor , calculate the final logits value , the calculation formula is: ; Step 4.5: Use the Softmax function to convert the logits value Converted into probability distribution, the conversion formula is: ; in, Represents the model prediction sample Belongs to the real category Prediction probability, Yes Category The logits value of Step 4.6: Use the cross entropy loss function to calculate the ArcFace loss. The specific calculation formula is: ; in, represents the batch size, is the number of categories, Indicates The true category labels of samples, Indicates The angle between the one-dimensional embedding vector of a sample and its true label vector, and Represents the input sample With category The angle between is the additional angular margin parameter, is the scaling factor.

[0028] By introducing an angular margin between categories, the embedding features of normal samples are more effectively clustered near the average embedding vector of each category, forming a tighter distribution. After training, during the detection phase, abnormal samples can be more clearly distinguished from normal samples due to their larger angular deviation from the average embedding vector of each category in normal samples. This angle-based feature distribution method strengthens the model's discriminative ability, making the detection of abnormal sounds more accurate and efficient.

[0029] Furthermore, the calculation method of the anomaly score A in step 5 is: ; in, Represents the model prediction sample Belongs to the real category Predicted probability.

[0030] The trained model can usually accurately classify normal sounds with high confidence, but shows lower confidence for abnormal sounds that have never been seen. In order to quantify the degree of abnormality, the classification probability is converted into an abnormality score through negative logarithm, so that normal samples correspond to lower abnormality scores, while abnormal samples correspond to higher abnormality scores. When the abnormality score exceeds the set threshold, the sound is judged to be abnormal. Regarding the setting of the threshold, specifically, a gamma distribution can be fitted to the abnormality scores of normal training samples, and the point in the gamma distribution where 90% of the normal samples have abnormality scores below this value is used as the set threshold (i.e., the 90th percentile).

[0031] Furthermore, before step 1, the step further includes: The input one-dimensional audio signal is transformed into a time-frequency spectrum. The specific transformation formula is: ; in, is the Log-Mel spectrum, represents the Mel filter bank, The length of the input is One-dimensional audio signal, STFT represents short-time Fourier transform, which converts the audio signal into a one-dimensional Spectrum of represents the number of time frames, represents the number of frequency bands, Represents the Mel filter bank The STFT spectrum is filtered, where is the number of Mel filters.

[0032] The Log-Mel spectrogram can effectively capture the time and frequency information of the audio signal; the Log-Mel spectrogram performs time-frequency conversion on the audio signal through short-time Fourier transform (STFT), and then compresses the frequency axis in combination with the Mel filter group, and finally performs logarithmic transformation to make the feature have a better dynamic range.

[0033] This application first models the frequency dimension of the time-spectrogram and extracts the frequency information in each time frame; then integrates the frequency domain features in the time dimension, gradually extracts the time and frequency domain features, and maintains the correlation between the time and frequency domain information. In the abnormal sound detection application, this application uses the dual-path channel attention WaveNet model to deeply mine the feature distribution of normal sounds, thereby accurately distinguishing normal from abnormal sounds, significantly improving the accuracy and robustness of unsupervised abnormal sound detection.

[0034] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A self-supervised machine abnormal sound detection method based on dual-path WaveNet, characterized in that: include: Step 1: Use WaveNet in the frequency domain path to model the frequency information of each time frame of the input sound signal and extract frequency domain features: ; in The WaveNet mapping function representing the frequency domain path, Represents input data The dimension is , is the output frequency domain feature; Step 2: Output of the time domain path to the frequency domain path As input, after transposing it, it is considered as The length is Then, WaveNet is used to model the time dimension and the output is the time domain feature. : ; in WaveNet mapping function representing the time domain path; Step 3: Output of frequency domain path and time domain path The features are further fused through a two-dimensional convolutional layer to generate a one-dimensional embedding vector; Step 4: Adopt a self-supervised strategy and use metadata to build a classification task to train the model. The classification loss function is ArcFace Loss. Step 5: After the training model, in the detection phase, the model calculates the anomaly score A through the classification probability. If the anomaly score A of the test sample exceeds the set threshold, it is judged as abnormal; otherwise, it is judged as normal.

2. A self-supervised machine abnormal sound detection method based on dual-path WaveNet as claimed in claim 1, characterized in that: The calculation method of the classification loss function ArcFace Loss in step 4 is: Step 4.1: Normalize the one-dimensional embedding vector and the category weight vector; Step 4.2: Obtain the angle value for each category by calculating the cosine similarity between the normalized one-dimensional embedding vector and the category weight vector ; ； in, represents a one-dimensional embedding vector, represents the class weight vector, Represents the input sample With category The angle between Step 4.3: Introduce the angle interval m and calculate the true category The cosine of the angle ; Step 4.4: Introducing a scaling factor , calculate the final logits value , the calculation formula is: ; Step 4.5: Use the Softmax function to convert the logits value Converted into probability distribution, the conversion formula is: ; in, Representation sample Belongs to category The predicted probability of Yes Category The logits value of Step 4.6: Use the cross entropy loss function to calculate the ArcFace loss. The specific calculation formula is: ; in, represents the batch size, is the number of categories, Indicates The true category labels of samples, Indicates The angle between the one-dimensional embedding vector of a sample and its true label vector, and Represents the input sample With category The angle between is the additional angular margin parameter, is the scaling factor.

3. A self-supervised machine abnormal sound detection method based on dual-path WaveNet as described in claim 1 or 2, characterized in that: The anomaly score in step 5 The calculation method is: ; in, Represents the model prediction sample Belongs to the real category Predicted probability.

4. A self-supervised machine abnormal sound detection method based on dual-path WaveNet as described in claim 3, characterized in that: The step 1 also includes: The input one-dimensional audio signal is transformed into a time-frequency spectrum. The specific transformation formula is: ; in, is the Log-Mel spectrum, represents the Mel filter bank, The length of the input is One-dimensional audio signal, STFT represents short-time Fourier transform, which converts the audio signal into a one-dimensional Spectrum of represents the number of time frames, represents the number of frequency bands, Represents the Mel filter bank The STFT spectrum is filtered, where is the number of Mel filters.

Images