Self-supervised abnormal sound detection method based on combination of spectral and temporal information

Abstract

The application provides a self-supervised abnormal sound detection method based on spectrum and time information combination, comprising the following steps: collecting original audio clips of each target device in advance, and storing the original audio clips in association with machine ID as an identifier; using a preset mel filter to extract a mel spectrum graph of the original audio clips; using a preset time feature extraction network to extract a time spectrum feature graph of the original audio clips; performing feature fusion on the mel spectrum graph and the time spectrum feature graph of each target device respectively; inputting the mel and time fusion features of each target device into a classifier, taking the machine ID of each target device as a category label, and training to obtain a target classification model; in the test, inputting the to-be-tested sound information into the obtained target classification model to obtain real-time feature sound indicators of each target device, and judging whether the target device is in an abnormal state according to the comparison result of the real-time feature sound indicators and a critical value.

Description

Technical Field

[0001] This invention relates to the field of non-destructive testing technology for industrial equipment, and more specifically, to a self-supervised abnormal sound detection method based on the combination of spectrum and time information. Background Technology

[0002] Methods for detecting defects in industrial equipment are divided into destructive testing and non-destructive testing. Destructive testing requires cutting open the parts and inspecting them visually or by immersing them in chemical liquids to check for defects. Commonly used non-destructive testing methods in industry mainly include radiographic testing, penetrant testing, magnetic particle testing, eddy current testing, and abnormal sound detection.

[0003] The purpose of abnormal sound detection is to automatically identify whether a machine or device is functioning normally or abnormally based on the sounds it emits. With the development of technology, the methods for detecting or monitoring abnormal sounds from machines or devices have gradually become more diverse, from listening with the ears to using portable listening devices, and then to artificial intelligence for "sound diagnosis".

[0004] Currently, the commonly used methods for judging / diagnosing abnormal noises in equipment mainly include auditory testing, ultrasonic testing, and AI-powered intelligent acoustic fault detection.

[0005] (1) Auditory method

[0006] Suitable for experienced workers, who can use the sound and vibration of a machine or equipment to manually determine whether there are defects in the machine or equipment.

[0007] (2) Ultrasonic testing method

[0008] The inability to hear abnormal noises from a machine or equipment does not necessarily mean that the machine or equipment is functioning properly. Therefore, a portable ultrasonic detector can be used to perform ultrasonic testing on the machine or equipment to reflect the amplitude and frequency band information of the noise.

[0009] (3) AI-powered intelligent acoustic fault detection method, mainly including:

[0010] 1) Predict when machine parts will begin to wear out based on basic physical modeling; 2) Label and classify specific sounds, and then use deep learning to associate specific abnormal sounds with specific faults. The software, after learning and training on the sound dataset, can warn of impending problems before a fault occurs.

[0011] It should be noted that the second method of AI intelligent acoustic fault detection relies on pre-collected specific abnormal sounds and pre-established correlations between specific abnormal sounds and specific faults to distinguish whether a machine or equipment is normal or abnormal.

[0012] However, the abnormal sound data from the same batch of devices may differ, as each device has its own unique sound characteristics. Therefore, abnormal sound data in the real world is characterized by diversity and scarcity. Furthermore, since abnormal sound data is a non-stationary signal, collecting and detecting it is not a trivial task. If the actual abnormal sound differs from the specific abnormal sound collected beforehand, it is highly likely that it will not be detected, or even result in a false positive.

[0013] To address this issue, AI-powered intelligent acoustic fault detection methods employ the following approach: pre-collecting normal sounds from the device, learning the characteristics of normal sounds, and then using these learned characteristics to distinguish between normal and abnormal machines or devices. However, existing technologies typically use autoencoders (AEs) to learn the characteristics of normal sounds by minimizing reconstruction errors, and then using these reconstruction errors as a score to detect anomalies. However, since abnormal sounds are not involved in the training process, the effectiveness of such models is limited.

[0014] In order to solve the above problems, people have been seeking an ideal technological solution. Summary of the Invention

[0015] The purpose of this invention is to address the shortcomings of existing technologies by providing a self-supervised abnormal sound detection method based on the combination of spectrum and time information.

[0016] To achieve the above objectives, the technical solution adopted by this invention is: a self-supervised abnormal sound detection method based on the combination of spectrum and time information, characterized by comprising the following steps:

[0017] Step 1: Using an omnidirectional microphone, collect raw audio clips of each target device in a real factory environment in advance, and store them in association using the machine ID as an identifier;

[0018] The omnidirectional microphones are respectively installed at the key monitoring points of each target device. The target devices are of the same type but with different detailed structures. The original audio clips include the normal operation sound of the target device and the operation sound of the machines and equipment around the target device.

[0019] Step 2: Process the original audio segments of each target device using a preset Mel filter to extract the Mel spectrogram corresponding to each target device; process the original audio segments of each target device using a preset time feature extraction network to extract the time spectrogram corresponding to each target device.

[0020] The Mel spectrum and time spectrum feature map of each target device are fused to obtain the Mel and time fused features of each target device.

[0021] Step 3: Input the Mel and temporal fusion features of each target device into the preset classifier model, using the machine ID of each target device as the category label, and use the Arcface loss function to guide the convergence of the classifier model;

[0022] Step 4: During the training process, every N rounds, calculate the accuracy of the classification model on the test set;

[0023] The model is trained iteratively for m×N rounds, and the model with the best accuracy is selected as the target classification model.

[0024] The maximum likelihood estimation algorithm is used to fit the output value of the classification model during training to obtain the Gaussian distribution probability density function of the output value, and then the critical value for abnormal sound detection is determined based on the Gaussian distribution probability density function.

[0025] Step 5: When monitoring the health status of the target equipment on site, collect real-time operating sound clips of each target equipment, and obtain the sound information to be tested after mono sampling;

[0026] The sound information to be tested is input into the target classification model obtained in step 4 to obtain the real-time characteristic sound index of each target device.

[0027] If the real-time characteristic sound index is less than or equal to the threshold value, the target device is determined to be in a normal state.

[0028] If the real-time characteristic sound index is greater than the threshold value, the target device is determined to be in an abnormal state.

[0029] This invention has outstanding substantive features and significant progress compared to the prior art, specifically:

[0030] 1) This invention performs feature fusion on the Mel spectrogram and time spectrogram of the original audio segment of each target device to obtain Mel and time fused features, and uses a classifier model with the machine ID of each target device as the category label to achieve rapid and accurate detection of abnormal sounds, reduce the difficulty of industrial equipment status monitoring, and improve the automation and digitalization level of the factory.

[0031] 2) This invention utilizes convolutional networks to extract the temporal spectrum information of the device's operating sound as supplementary information to the Mel spectrogram of the device's operating sound; the temporal spectrum information includes low-frequency information and high-frequency information, therefore, the fusion of Mel and time features can effectively improve the lack of high-frequency information in audio signals when using only the Mel spectrogram;

[0032] 3) Modeling using only normal audio signals is inherently unlabeled. This invention uses the idea of ​​self-supervision, utilizing machine ID data as labels to transform the unsupervised problem into a supervised classification problem, thereby improving the model's ability to model abnormal data. Attached Figure Description

[0033] Figure 1 This is a flowchart of the self-supervised abnormal sound detection method based on the combination of spectrum and time information of the present invention;

[0034] Figures 2(a) to 2(c) are schematic diagrams of audio acquisition and audio cutting according to the present invention;

[0035] Figure 3 This is a diagram of the network structure for extracting time spectrum information according to the present invention;

[0036] Figure 4 This is a schematic diagram of the model flow of the present invention;

[0037] Figure 5 This is a network structure diagram of the classifier model of the present invention.

[0038] Figure 6 This is a schematic diagram of the bottleneck module in the classifier model of this invention. Detailed Implementation

[0039] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0040] Example 1

[0041] As attached Figure 1 As shown, a self-supervised abnormal sound detection method based on the combination of spectrum and time information includes the following steps:

[0042] Step 1: Using an omnidirectional microphone, collect raw audio clips of each target device in a real factory environment in advance, and store them in association using the machine ID as an identifier;

[0043] The omnidirectional microphones are respectively installed at the key monitoring points of each target device. The target devices are of the same type but with different detailed structures. The original audio clips include the normal operation sound of the target device and the operation sound of the machines and equipment around the target device.

[0044] Step 2: Using a preset Mel filter, extract the Mel spectrogram of the original audio segment from each target device;

[0045] The time-spectral feature map of the original audio segment of each target device is extracted using a pre-set time feature extraction network.

[0046] The Mel spectrum and time spectrum feature map of each target device are fused to obtain the Mel and time fused features of each target device.

[0047] Step 3: Input the Mel-time fusion features of each target device into the preset classifier model, using the machine ID of each target device as the category label, and use the Arcface loss function to guide the convergence of the classifier model;

[0048] Step 4, preset N=10, m=30;

[0049] During training, the accuracy of the classification model on the test set is calculated every 10 rounds.

[0050] The model is trained iteratively for 300 rounds, and the model with the best accuracy is selected as the target classification model.

[0051] The output value of the classification model during training is fitted using the maximum likelihood estimation method to obtain the Gaussian distribution probability density function of the output value. Then, the critical value for abnormal sound detection is determined based on the Gaussian distribution probability density function, and the output value corresponding to the threshold of 0.95 is taken as the critical value.

[0052] Step 5: When monitoring the health status of the target equipment on site, collect the real-time operating sound of each target equipment for 10 seconds every minute. After mono sampling, obtain the sound information to be tested.

[0053] Input the sound information to be tested into the target classification model obtained in step 4 to obtain the real-time characteristic sound index of each target device.

[0054] If the real-time characteristic sound index is less than or equal to the threshold value, the target device is determined to be in a normal state.

[0055] If the real-time characteristic sound index is greater than the threshold value, the target device is determined to be in an abnormal state.

[0056] It should be noted that the fused features are fed into a classifier with the number of classes equal to the number of machines (each target device corresponds to one class). All collected samples can be trained simultaneously, solving the problem of overfitting caused by generative methods that can only use one machine ID for training. The Arcface loss function is used to guide model convergence, which helps to increase the distance between classes and reduce the distance within classes, thereby improving classification accuracy.

[0057] It is understandable that in a factory environment, there are often a large number of similar machines but with different detailed structures. By fixing an omnidirectional microphone (SHURE SM11-CN) at a position of 20 centimeters from the key monitoring part of these machines, the sound of the machines operating normally can be collected and the different machines can be distinguished by machine ID, as shown in Figure 2(a).

[0058] There are two methods for data collection. One method involves collecting ten seconds of sound throughout the entire operation, from machine startup to shutdown, as shown in Figure 2(b). The other method involves running the machine continuously for ten minutes, then cutting the sound into ten-second audio WAV files, all of which are processed into mono, as shown in Figure 2(c). Because the first method is much more expensive than the second, both methods are used alternately for collecting normal samples. When collecting abnormal sounds, only the first method is used.

[0059] When the real-time characteristic sound index is less than the threshold value, it means that the machine is in normal condition. The smaller the real-time characteristic sound index, the better the machine condition. When the real-time characteristic sound index is greater than the threshold value, it means that the machine is in abnormal condition. The larger the real-time characteristic sound index, the worse the machine condition. Thus, a machine health status monitoring table is established to output the health monitoring information of each machine around the clock.

[0060] When the real-time characteristic sound index exceeds the threshold and an anomaly occurs, a second judgment is required, namely, sampling another audio segment for judgment. If an anomaly occurs twice in a row, it is determined that the machine has malfunctioned and needs to be maintained in time, thereby reducing the false judgment rate.

[0061] It should be noted that the Mel spectrum is designed based on human auditory perception and uses Mel filters to capture information of different frequencies. However, the Mel spectrum will filter out the high-frequency part of the sound, which has obvious characteristics. Therefore, this embodiment designs a convolutional network to extract the time-spectral feature map of the sound to supplement the Mel spectrum. After fusion, a complete feature that includes both low and high frequencies is obtained.

[0062] The specific structure of the Mel filter in step 2 will not be described in detail here.

[0063] Furthermore, the time information feature extraction network in step 2 includes a first convolutional layer, a first feature extraction module, a second feature extraction module, and a third feature extraction module. Each of the first, second, and third feature extraction modules includes a LayerNorm layer, a LeakyReLU layer, and a second convolutional layer arranged sequentially, as shown in the attached diagram. Figure 3 As shown.

[0064] Furthermore, the specific parameters of the time information feature extraction network are shown in the table below:

[0065]

[0066] To eliminate more variables and improve the fusion effect of time-spectral features and Mel-spectral features, the parameters n_mel, win-length, and hop-length in the table above are kept consistent with the parameter settings when extracting the audio Mel spectrogram, ensuring that the extracted features are all 128×313 in size, which facilitates subsequent feature fusion.

[0067] In one specific implementation, Mel spectrum extraction employs the following steps:

[0068] First, the audio signal was read using the librosa library in Python, with a sampling rate of 16000, and mono reading was performed.

[0069] Perform a short-time Fourier transform on the read audio signal. The parameters include hop_length: frame shift value is 512, win_length: window length value is 1024, and n_fft: FFT window size is 1024.

[0070] Mel features are extracted from the Fourier transform signal, with parameters including frame shift, window length, FFT window size, n_mel: the number of Mel filters with a value of 128, and power: the exponent of the amplitude spectrum with a value of 2; then converted to a logarithmic scale to obtain the Log-mel spectral features.

[0071] Temporal Spectral Feature Map Extraction: Construct a temporal information feature extraction network. The audio signal first passes through a one-dimensional convolution with a large kernel. Set the number of channels for the convolution, and set the kernel size and stride to be the same as n_mel, win_length and hop_length of the Log-mel spectrogram.

[0072] Then, feature extraction is performed using three stacked LayerNorm, LeakyReLU, and a 1D convolution with a kernel size of 3. Simultaneously, the convolutional blocks do not affect the dimensionality of the output temporal features, maintaining consistency with the dimensionality of the Log-mel spectrum.

[0073] Furthermore, in step 2, when performing feature fusion on the Mel spectrum and time spectrum feature map of each target device to obtain the Mel and time fused features of each target device, the following steps are executed:

[0074] Obtain the Mel spectrum and time spectrum feature map of a target device. The Mel spectrum has a dimension of 128×313, and the time spectrum feature map has a dimension of 128×313.

[0075] The obtained Mel spectrogram and temporal spectrogram feature map are subjected to a cat operation to obtain a Mel and temporal fusion feature with dimensions of 128×313×2.

[0076] It is understandable that the Mel spectrum and the temporal spectrum maintain the same dimension, and are merged at the channel dimension to obtain a fused feature map.

[0077] Specifically, the pre-built classifier model includes two convolutional modules and 18 bottleneck modules positioned between them, as shown in the attached diagram. Figure 5 As shown;

[0078] The structure of the 18 bottleneck modules is shown in the attached figure. Figure 6 As shown, the input first goes into a 2D convolution, where the dimension is expanded by an expansion coefficient e. Then it goes through BN and ReLU, and enters a second Conv2D. This convolution has the same input and output dimensions, but it uses group convolution, meaning that the group parameters are set to have the same number of input channels. The optional SE model can be used later, and it varies at different stages. Then it goes through another Conv2D to change the number of channels to the number of parameters out. There is an add operation between the output and input.

[0079] In one specific implementation, the pre-set classifier model adopts a structure similar to MobileNetV3. The resulting 1×1×512 dimensional vector is passed through a linear layer with K output nodes (K refers to the number of classes, which is equal to the number of machines, with each machine representing one class) to obtain the output, as shown in the attached figure. Figure 5 and 6 As shown. The classifier model uses stacked residuals, which can separate convolutions to further extract audio features, but does not use a linear classifier in the last layer. Instead, it chooses Arcface loss and obtains the predicted class output value by implementing a large margin arc distance.

[0080] This embodiment uses Arcface loss to guide model convergence instead of traditional cross-entropy loss. Arcface loss starts with the cosine expression of cross-entropy loss, normalizes the feature vectors and weights separately, and adds a margin m to the angle between the weights and input features, penalizing the angle between features and weights, thereby reducing intra-class differences and increasing inter-class differences. The formula for the Arcface loss function is as follows:

[0081]

[0082] Here, s and m are two hyperparameters of Arcface Loss, with s taking a value of 32 and m taking a value of 0.5.

[0083] Furthermore, step 4 includes the following steps:

[0084] Step 4a: Iterate the model training for 300 rounds. During the training process, use Mixup data augmentation to enhance the model's fitting ability. Mixup randomly selects two data points in a batch of data and uses the lambd ratio to mix the Log-mel spectral features and temporal spectral features respectively, while also mixing the labels. When calculating the loss, calculate the mixed loss.

[0085] Step 4b: Every ten iterations, use the metrics function of Python's sklearn library to calculate the accuracy of the model, and select the best-performing iterative model as the final model parameters.

[0086] Step 4c: After obtaining the optimal model, load all normal samples. For the output of the normal samples, use the Python scipy library to perform maximum likelihood estimation to fit a Gaussian distribution and obtain the probability density function of the normal samples.

[0087] It should be noted that other artificial intelligence methods for detecting abnormal sounds in industrial equipment only use the Mel-spectrum features of the sound without adding additional supervisory signals, resulting in poor performance. This invention extracts the time signal features of the audio and combines them with Mel-spectrum features to supplement the missing effective features of the Mel-spectrum. Simultaneously, it introduces the machine ID as an additional supervisory signal, transforming the unsupervised abnormal sound detection problem into a supervised classification problem. This achieves non-destructive and rapid detection of abnormal sounds from equipment, enabling accurate and effective monitoring of equipment health status.

[0088] To better model the features of normal sounds, this invention proposes a self-supervised classification method. This method uses machine type and machine ID metadata, along with conditions (normal / abnormal), and an accompanying audio file to detect abnormal sounds, achieving better results than methods using autoencoders.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A self-supervised abnormal sound detection method based on the combination of spectrum and time information, characterized in that, Includes the following steps: Step 1: Using an omnidirectional microphone, collect raw audio clips of each target device in a real factory environment in advance, and store them in association using the machine ID as an identifier; The omnidirectional microphones are respectively installed at the key monitoring points of each target device. The target devices are of the same type but with different detailed structures. The original audio clips include the normal operation sound of the target device and the operation sound of the machines and equipment around the target device. Step 2: Process the original audio segments of each target device using a preset Mel filter to extract the Mel spectrogram corresponding to each target device; process the original audio segments of each target device using a preset time feature extraction network to extract the time spectrogram corresponding to each target device. The Mel spectrum and time spectrum feature map of each target device are fused to obtain the Mel and time fused features of each target device. Step 3: Input the Mel and temporal fusion features of each target device into the preset classifier model, use the machine ID of each target device as the category label, and use the Arcface loss function to guide the convergence of the classifier model; Step 4: During the training process, every N rounds, calculate the accuracy of the classification model on the test set; The model is trained iteratively for m×N rounds, and the model with the best accuracy is selected as the target classification model. The maximum likelihood estimation algorithm is used to fit the output value of the classification model during training to obtain the Gaussian distribution probability density function of the output value, and then the critical value for abnormal sound detection is determined based on the Gaussian distribution probability density function. Step 5: When monitoring the health status of the target equipment on site, collect real-time operating sound clips of each target equipment, and obtain the sound information to be tested after mono sampling; The sound information to be tested is input into the target classification model obtained in step 4 to obtain the real-time characteristic sound index of each target device. If the real-time characteristic sound index is less than or equal to the threshold value, the target device is determined to be in a normal state. If the real-time characteristic sound index is greater than the threshold value, the target device is determined to be in an abnormal state.

2. The self-supervised abnormal sound detection method based on the combination of spectrum and time information according to claim 1, characterized in that: The time information feature extraction network in step 2 includes a first convolutional layer, a first feature extraction module, a second feature extraction module, and a third feature extraction module. The first feature extraction module, the second feature extraction module, and the third feature extraction module all include a LayerNorm layer, a LeakyReLU layer, and a second convolutional layer arranged sequentially.

3. The self-supervised abnormal sound detection method based on the combination of spectrum and time information according to claim 1, characterized in that: In step 2, when performing feature fusion on the Mel spectrum and time spectrum feature map of each target device to obtain the Mel and time fused features of each target device, the following steps are executed: Obtain the Mel spectrum and time spectrum feature map of a target device. The Mel spectrum has a dimension of 128×313, and the time spectrum feature map has a dimension of 128×313. The obtained Mel spectrogram and temporal spectrogram feature map are subjected to a cat operation to obtain a Mel and temporal fusion feature with dimensions of 128×313×2.

Images