Non-supervision industrial abnormal sound detection method and system based on feature fusion and NICE model

Abstract

The invention discloses an unsupervised industrial abnormal sound detection method and system based on feature fusion and an NICE model, and the method comprises the steps: (1) model training: carrying out the feature extraction of three spectrograms of an original sound signal, forming two groups of fusion features through splicing, and inputting the two groups of fusion features into a dual-path NICE, i.e., a dpNICE model for training; and (2) anomaly detection: extracting features of three spectrograms for the acquired unknown sound signal, splicing to form two groups of fusion features, inputting the two groups of fusion features into the model trained in the step (1), and judging whether the model is abnormal or not. According to the technical scheme, training of the dpNICE model does not need metadata assistance, and compared with a self-supervised learning mode, the method is wider in application scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of abnormal sound detection, and in particular relates to an unsupervised industrial abnormal sound detection method and system based on feature fusion and NICE model. Background Art

[0002] In modern industrial production, abnormal sound detection is crucial to maintaining the normal operation of equipment and preventing failures. In industrial abnormal sound detection, traditional supervised deep learning methods are limited in application due to the scarcity of labeled data and the diversity of abnormal situations. Although existing self-supervised deep learning methods have made certain progress, they require metadata-assisted training, and these data may not be available in all cases, so their application scenarios are also limited. In contrast, unsupervised deep learning methods do not have the problem of using metadata-assisted training. However, most of the existing unsupervised learning deep methods learn from a single sound feature and often find it difficult to fully utilize the effective information of the signal. Summary of the invention

[0003] In view of the above problems existing in the prior art, the present invention proposes an unsupervised industrial abnormal sound detection method and system based on feature fusion and nonlinear independent component estimation (Non-linear Independent Components Estimation, NICE) model. The present invention first extracts the log-Mel spectrogram, TgramNet spectrogram and SincNet spectrogram features of the sound; two sets of fusion features are constructed by fusing the TgramNet spectrogram and the log-Mel spectrogram, and the SincNet spectrogram and the log-Mel spectrogram; the fusion features are used as input, and the dual-path NICE model is used for training and detection, and the negative log likelihood is used as the loss in the training stage and the anomaly score in the test stage; the abnormality of the input sound is judged by the anomaly score.

[0004] First, the professional terms in the present invention are explained:

[0005] (I) log-Mel spectrum characteristics

[0006] The log-Mel spectrogram feature is a commonly used sound feature. In the log-Mel spectrogram, the features in the low-frequency area are more prominent and detailed. This is because the Mel scale is more in line with the auditory characteristics of the human ear and is more sensitive to low-frequency sounds. This feature makes the log-Mel spectrogram more effective in processing sound-related tasks.

[0007] (II) TgramNet spectral features

[0008] The TgramNet spectrogram is extracted from the sound signal by the TgramNet network, and its structure is as followsFigure 2 As shown in the figure, the TgramNet network mainly consists of a large kernel one-dimensional convolution layer and three CNN blocks. The three CNN blocks have the same structure, and each block contains layer normalization, LeakyReLU activation function, and one-dimensional convolution with a small kernel. Based on the above structural design, TgramNet can capture temporal dependencies while also refining temporal features to identify more subtle changes through one-dimensional convolutions with different kernel sizes.

[0009] (III) SincNet spectral features

[0010] The SincNet spectrogram is extracted from the sound signal by the SincNet network, and its structure is as follows Figure 3 As shown. SincNet implements bandpass filtering through a parameterized, learnable sinc function. The first layer of its architecture is the time domain convolution between the input data and the finite impulse response (FIR) filter. Its essence is the point-by-point product of the FIR filter and each segment of the input signal and then the summation, that is:

[0011]

[0012] Where x[n] is a sound signal, h[n] is a filter of length L, and y[n] is the filtered output signal. SincNet implements filtering through the following specific form of convolution:

[0013] y[n]=x[n]×g[n,θ]

[0014] Among them, g is a predefined function based on the parameterized sinc function, which is designed to simulate the behavior of a rectangular bandpass filter. Compared with the traditional CNN, which needs to learn all the elements of each filter from the data, the g function only has a small number of parameters θ to learn. Specifically, SincNet only needs to learn the low-frequency and high-frequency cutoff frequencies of the filter. The performance of a general bandpass filter in the frequency domain can be described by the difference between two low-pass filters, that is:

[0015]

[0016] where f1 and f2 are the low and high frequency cutoff frequencies of the filter, respectively, and rect(·) is the response of a rectangular low-pass filter, defined as:

[0017]

[0018] By performing an inverse Fourier transform on the function G, the impulse response of the filter function g can be obtained:

[0019] g[n,f1,f2]=2f2sinc(2πf2n)-2f1sinc(2πf1n)

[0020] The sinc function is defined as sinc(x)=sin(x) / x. To ensure f1≥0 and f2≥f1, the following constraints are imposed:

[0021]

[0022] The above cutoff frequency can be randomly configured in the interval [0, fs / 2], where fs is the sampling rate of the input signal. The cutoff frequency of the Mel-scale filter bank can also be initialized, so that more filters can be configured in the lower part of the spectrum. This area often contains more key features in the audio signal, which helps to enhance the model's ability to capture these key features.

[0023] In addition, since the expression of function g in the time domain has the characteristic of infinite extension, if it is calculated directly, it will inevitably cause the leakage of spectrum energy. In order to reduce this effect, the Hamming window is used to truncate the function g to smooth the discontinuity of the function g at its endpoints. The corresponding calculation process can be expressed as:

[0024] g ω (n,f1,f2)=g(n,f1,f2)·ω(n)

[0025] (IV) Dual-Path NICE (dpNICE) Model

[0026] Based on the NICE model, the dpNICE model constructs a dual-path mechanism to achieve effective fusion of multiple input features, thereby more comprehensively capturing the rich and valuable information in the original sound signal. The dpNICE model is constructed as follows: The dpNICE model consists of several alternating coupling layers and a scale transformation layer. The output node of each alternating coupling layer realizes the information exchange between the two NICE models through two channels, which is specifically implemented as the invariant component of the current coupling layer output in a NICE model is transmitted to the variable component of the corresponding coupling layer output of another NICE model. Specifically, suppose and are the two input components of one of the NICE models, and is the output component of the first alternating coupling layer; and are the two input components of another NICE model, and is the output component of the first alternating coupling layer, then the formula is expressed as:

[0027]

[0028] in, and is the constant component, and is the variable component, g(·) is the coupling function, and m(·) is the reversible transformation function, such as Figure 4 In addition, the input of the scale transformation layer is a vector formed by connecting the four output components of the alternating coupling layer end to end, as shown in Figure 5 shown.

[0029] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0030] An unsupervised industrial abnormal sound detection method based on feature fusion and NICE model uses the features fused from log-Mel spectrogram, TgramNet spectrogram and SincNet spectrogram as input and combines it with a dual-path NICE neural network model. The specific steps are as follows:

[0031] Step (i), model training: extract the features of three spectrograms of the original sound signal, form two sets of fusion features by splicing, and input them into the dual-path NICE, i.e., dpNICE model for training;

[0032] Step (ii), anomaly detection: for the collected unknown sound signal, extract the features of the three spectrograms, form two sets of fusion features by splicing, input them into the model trained in step (i), and determine whether it is abnormal.

[0033] Preferably, (i) model training is specifically as follows:

[0034] Step 1-1: Perform log-Mel spectrogram F on the original sound signal M 、TgramNet Spectrum F T and SincNet spectrogram F S Feature extraction.

[0035] Step 1-2: Concatenate the log-Mel spectrogram with the SincNet spectrogram and the TgramNet spectrogram to form two sets of fusion features F MT (Fusion features of log-Mel spectrogram and TgramNet spectrogram) and F MS (Fusion features of log-Mel spectrogram and SincNet spectrogram).

[0036] In this step, the fusion feature F MT The implementation is specifically F MT =Concat(F M ,F T ), where F M ∈RM×N , F T ∈R M×N , F MT ∈R 2M×N , where Concat(·) represents the concatenation operation, M is the number of filters corresponding to the log-Mel spectrogram, and N is the number of frames of the log-Mel spectrogram.

[0037] In this step, the fusion feature F MS The implementation is specifically F MS =Concat(F M ,F S ), F S ∈R M × N , F MS ∈R 2M × N .

[0038] Step 1-3: Process the sounds of the training set and the test set according to step 1-1 and step 1-2, and obtain the two sets of fusion features F described in step 1-2 for each sample in the data set. MT and F MS .

[0039] Step 1-4: Fusion feature F of each sample in the training set MT and F MS The two NICE models of the dpNICE model are inputted respectively and trained to optimize the model parameters. Specifically, the model training is achieved by minimizing the negative log-likelihood of the latent space data of the dpNICE model and the standard multivariate Gaussian prior distribution, and its implementation details are the same as those of the NICE model.

[0040] Preferably, (ii) abnormality detection: for a collected unknown sound signal data, the following processing is performed to determine whether it is abnormal or not:

[0041] Step 2-1: Extract the log-Mel spectrum feature F according to step 1-1 and step 1-2 in step (a). M 、TgramNet spectral features F T and SincNet spectral features F S ; Then, F M Respectively with F T 、F S Splice to get two sets of fusion features F MT and F MS .

[0042] Step 2-2: The fusion feature F obtained in step 2-1 MT and F MSInput the dpNICE model and calculate the negative log-likelihood value in the latent space as the anomaly score S.

[0043] Step 2-3: Compare the anomaly score S obtained in step 2-2 with the threshold A θ Compared with, greater than threshold A θ Indicates that the input sound signal is abnormal, otherwise it is normal, expressed as:

[0044]

[0045] The present invention also discloses an unsupervised industrial abnormal sound detection system based on feature fusion and NICE model, which specifically includes the following modules based on the above method:

[0046] Model training module: extract the features of three spectrograms from the original sound signal, form two sets of fusion features by splicing, and input them into the dual-path NICE, i.e., dpNICE model, for training;

[0047] Anomaly detection module: For the collected unknown sound signals, the features of the three spectrograms are extracted, and two sets of fusion features are formed by splicing. The features are input into the trained model to determine whether they are abnormal.

[0048] The beneficial effects of the present invention are as follows:

[0049] In the technical solution of the present invention, the training of the dpNICE model does not require labeled data, which saves a lot of labor costs. Compared with the supervised learning method, the present invention has a wider range of application scenarios. In addition, compared with the single feature method, the present invention uses three different sound features to improve the representation of sound signals, capture more comprehensive and richer information of sound signals, and have better detection effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of an unsupervised industrial abnormal sound detection method based on feature fusion and NICE model in a preferred embodiment of the present invention;

[0051] Figure 2 This is the structural diagram of TgramNet;

[0052] Figure 3 This is the structural diagram of SincNet;

[0053] Figure 4 This is the structural diagram of the dpNICE alternating coupling layer;

[0054] Figure 5 This is the structural diagram of the dpNICE scale transformation layer;

[0055] Figure 6The schematic diagram of the structure and parameters of the reversible transformation function in the dpNICE two-way NICE model;

[0056] Figure 7 A detailed flow chart of an unsupervised industrial abnormal sound detection method based on feature fusion and NICE model according to a preferred embodiment of the present invention;

[0057] Figure 8 This is a block diagram of an unsupervised industrial abnormal sound detection system based on feature fusion and NICE model according to a preferred embodiment of the present invention.

[0058] Figure 1 Middle: log-Mel (logarithmic Mel);

[0059] Figure 2 Medium: Large Kernel Conv1D (large kernel one-dimensional convolution), LayerNorm (layer normalization), Conv1D (one-dimensional convolution), Batch norm (batch normalization);

[0060] Figure 6 Middle: Layer Norm (layer normalization), Batchnorm1d (one-dimensional batch normalization), Linear (linear layer);

[0061] Figure 7 Middle: log-Mel (logarithmic Mel). DETAILED DESCRIPTION

[0062] The following uses industrial equipment sound as an example to further illustrate the present invention using an abnormal sound signal detection method based on feature fusion and stacked autoencoders. The following description is only illustrated by preferred embodiments and does not limit the present invention in any form.

[0063] In this embodiment, the public dataset of machine abnormal sound detection of DCASE2020 TASK2 is used, which includes six machine types: Fan, Pump, Slider, Valve, ToyCar and Toyconveyor. Each machine type has 3 or 4 machine IDs. The dataset of each machine ID consists of approximately 1,000 normal sound samples for training and several normal and abnormal sound signal samples for testing. The sampling rate is 16 kHz and the duration is 10 s.

[0064] like Figure 1-7 As shown, the unsupervised industrial abnormal sound detection method based on feature fusion and NICE model in this embodiment is as follows:

[0065] (I) Model training, as follows:

[0066] Step 1: Perform log-Mel spectrogram F on the original sound signal M 、TgramNet Spectrum F T and SincNet spectrogram F S Feature extraction; The log-Mel spectrogram features are extracted using a fast Fourier transform with a window size of 1024 and a step size of 512, and a Mel filter group of 128 is used; the number of channels of TgramNet is 128, the size of the large-core one-dimensional convolution kernel is 1024, and the step size and padding are both 512; the number of SincNet filters is 128, the filter length is 1024, and the step size and padding are 512.

[0067] Step 2: Concatenate the log-Mel spectrogram with the SincNet spectrogram and the TgramNet spectrogram to form the group fusion feature F MT (Fusion features of log-Mel spectrogram and TgramNet spectrogram) and F MS (Fusion features of log-Mel spectrogram and SincNet spectrogram).

[0068] Fusion feature F MT The specific implementation is:

[0069] F MT =Concat(F M ,F T )

[0070] Among them, F M ∈R M×N , F T ∈R M×N , F MT ∈R 2M×N , M is the number of filters corresponding to the log-Mel spectrogram, and N is the number of frames of the log-Mel spectrogram.

[0071] Fusion feature F MS The specific implementation is:

[0072] F MS =Concat(F M ,F S )

[0073] Among them, F S ∈R M×N , F MS ∈R 2M×N .

[0074] Step 3: Process the sounds of the training set and the test set according to steps 1 and 2, and obtain the two sets of fusion features F described in step 2 for each sample in the data set. MT and F MS .

[0075] Step 4: Fusion feature F of each sample in the training set MT and F MS The two NICE models of the dual-path NICE (dpNICE) model are respectively input and trained for 2000 epochs to optimize the model parameters. The two NICE models have the same structure and parameters. The alternating coupling layer is shown in Figure 6 The model training is achieved by minimizing the negative log-likelihood of the latent space data of the dpNICE model and the standard multivariate Gaussian prior distribution. The training uses the Adam optimizer, the batch size is 128, the learning rate is 0.001, and the L1 weight decay is added to the loss function to avoid overfitting. The L1 regularization coefficient is set to 0.1.

[0076] (II) Anomaly detection, as follows:

[0077] Step 1: Perform log-Mel spectrogram F on the original sound signal in the same way as in model training (step 1). M 、TgramNet Spectrum F T and SincNet spectrogram F S Feature extraction.

[0078] Step 2: The fusion feature F obtained in step 1 MT and F MS Input the dpNICE model and calculate the negative log-likelihood value in the latent space as the anomaly score S.

[0079] Step 3: Compare the anomaly score S obtained in step 2 with the threshold A θ Compared with, greater than threshold A θ Indicates that the input sound signal is abnormal, otherwise it is normal, expressed as:

[0080]

[0081] like Figure 8 As shown, this embodiment discloses an unsupervised industrial abnormal sound detection system based on feature fusion and NICE model, which specifically includes the following modules based on the above method embodiment:

[0082] Model training module: extract the features of three spectrograms from the original sound signal, form two sets of fusion features by splicing, and input them into the dual-path NICE, i.e., dpNICE model, for training;

[0083] Anomaly detection module: For the collected unknown sound signals, the features of the three spectrograms are extracted, and two sets of fusion features are formed by splicing. The features are input into the trained model to determine whether they are abnormal.

[0084] For other contents of this embodiment, please refer to the above method embodiment.

[0085] Any matters not described in the present invention are applicable to the prior art.

[0086] The specific implementation modes of the present invention are described above to facilitate the understanding of the present invention by those skilled in the art. However, the present invention is not limited to the scope of the specific implementation modes. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

Claims

1. An unsupervised industrial abnormal sound detection method based on feature fusion and NICE model, characterized in that: The specific steps include: Step (i), model training: extract the features of three spectrograms of the original sound signal, form two sets of fusion features by splicing, and input them into the dual-path NICE, i.e., dpNICE model for training; Step (ii), anomaly detection: for the collected unknown sound signal, extract the features of the three spectrograms, form two sets of fusion features by splicing, input them into the model trained in step (i), and determine whether it is abnormal.

2. The unsupervised industrial abnormal sound detection method based on feature fusion and NICE model as claimed in claim 1, characterized in that: Step (I) is as follows: Step 1-1: Perform log-Mel spectrogram F on the original sound signal M 、TgramNet Spectrum F T and SincNet spectrogram F S Feature extraction; Step 1-2: concatenate the log-Mel spectrogram with the SincNet spectrogram and the TgramNet spectrogram to form two sets of fusion features, namely F MT and F MS , where F MT is the fusion feature of log-Mel spectrum and TgramNet spectrum, F MS It is the fusion feature of log-Mel spectrum and SincNet spectrum; Step 1-3: Process the sound signals of the training set and the test set according to step 1-1 and step 1-2, and each sample in the data set obtains the two sets of fusion features F described in step 1-2. MT and F MS ; Steps 1-4: The fusion feature F of each sample in the training set MT and F MS The two paths are respectively input into the two-path NICE model of the dual-path NICE, i.e., the dpNICE model, and trained to optimize the model parameters.

3. The unsupervised industrial abnormal sound detection method based on feature fusion and NICE model as claimed in claim 2, characterized in that: In step 1-2: Fusion feature F MT The implementation is specifically F MT =Concat(F M ,F T ), where F M ∈R M×N , F T ∈R M×N , F MT ∈R 2M×N , where Concat(·) represents the concatenation operation, M is the number of filters corresponding to the log-Mel spectrogram, and N is the number of frames of the log-Mel spectrogram; Fusion feature F MS The implementation is specifically F MS =Concat(F M ,F S ), F S ∈R M×N , F MS ∈R 2M×N .

4. The unsupervised industrial abnormal sound detection method based on feature fusion and NICE model as claimed in claim 3, characterized in that: Step (ii) is as follows: Step 2-1: Extract the log-Mel spectrogram feature F of the unknown sound signal according to steps 1-1 and 1-2. M 、TgramNet spectral features F T and SincNet spectral features F S ; F M Respectively with F T 、F S Splicing to obtain two sets of fusion features, namely F MT and F MS ; Step 2-2: The fusion feature F obtained in step 2-1 MT and F MS Input the dpNICE model and calculate the negative log-likelihood value in the latent space as the anomaly score S; Step 2-3: Compare the anomaly score S obtained in step 2-2 with the threshold A θ Compare, if it is greater than the threshold A θ , it means the input sound signal is abnormal, otherwise it is normal, specifically expressed as:

5. The unsupervised industrial abnormal sound detection method based on feature fusion and NICE model according to claim 1 is characterized in that: The construction of the dpNICE model is as follows: the dpNICE model consists of several alternating coupling layers and a scale transformation layer; the output node of each alternating coupling layer realizes the information exchange between the two NICE models through two channels, which is specifically realized by transmitting the invariant component of the current coupling layer output in one NICE model to the variable component of the corresponding coupling layer output in the other NICE model. and are the two input components of one of the NICE models, and is the output component of the first alternating coupling layer; and are the two input components of another NICE model, and is the output component of the first alternating coupling layer, then the formula is expressed as: in, and is the constant component, and is the variable component, g(·) is the coupling function, and m(·) is the reversible transformation function. The input of the scale transformation layer is the vector formed by connecting the four output components of the alternating coupling layer end to end.

6. An unsupervised industrial abnormal sound detection system based on feature fusion and NICE model, based on the method described in any one of claims 1 to 5, characterized in that The modules include: Model training module: extract the features of three spectrograms from the original sound signal, form two sets of fusion features by splicing, and input them into the dual-path NICE, i.e., dpNICE model, for training; Anomaly detection module: For the collected unknown sound signals, the features of the three spectrograms are extracted, and two sets of fusion features are formed by splicing. The features are input into the trained model to determine whether they are abnormal.

Images