Abnormal sound detection method and system, electronic equipment and storage medium

Abstract

The invention provides an abnormal sound detection method and system, electronic equipment and a storage medium. The abnormal sound detection method comprises the following steps: acquiring sound information; extracting a first spectrogram feature and a first spectrum feature according to the sound information; determining a first sound feature according to the first spectrogram feature and the first spectrum feature; diffusing and denoising the first sound feature to obtain a second sound feature; determining a deviation between the second sound feature and the first sound feature; and determining whether the sound information is abnormal according to the deviation. According to the invention, after the sound information is obtained, the spectrogram feature and the spectrum feature in the sound information are extracted, and then the sound feature is determined according to the spectrogram feature and the spectrum feature, so that the meta-information of the sound data can be effectively utilized, and the stability of abnormal sound detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sound detection, and in particular to an abnormal sound detection method and system, electronic equipment and storage medium. Background Art

[0002] In the related art, the anomaly detectors of existing generative models are designed to learn the distribution of normal data, so the optimization targets of these methods are mainly focused on the audio data itself, which usually leads to insufficient utilization of metadata and causes unstable performance of ASD (Anomalous sound detection). Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0004] To this end, a first aspect of the present invention provides an abnormal sound detection method.

[0005] A second aspect of the present invention provides an abnormal sound detection system.

[0006] A third aspect of the present invention provides an electronic device.

[0007] A fourth aspect of the present invention provides a storage medium.

[0008] A fifth aspect of the present invention provides a computer program product.

[0009] In view of this, according to a first aspect of the present invention, an abnormal sound detection method is proposed, comprising: obtaining sound information; extracting a first spectrogram feature and a first spectrum feature based on the sound information; determining a first sound feature based on the first spectrogram feature and the first spectrum feature; diffusing and denoising the first sound feature to obtain a second sound feature; determining a deviation between the second sound feature and the first sound feature; and determining whether the sound information is abnormal based on the deviation.

[0010] The abnormal sound detection method provided by the present invention mainly includes: first obtaining sound information, wherein the way of obtaining the sound information can be input by the user or actively obtained by the system. After obtaining the sound information, extract the first spectrum graph feature and the first spectrum feature in the sound information according to the sound information, the first spectrum graph feature can represent the amplitude of the sound information at different frequencies, and the first spectrum feature can represent the characteristics and laws displayed by the sound information in the frequency domain. After obtaining the first spectrum graph feature and the first spectrum feature, process the first spectrum graph feature and the first spectrum feature to obtain the first sound feature. By determining the expression feature of the sound information, i.e., the first sound feature, according to the first spectrum graph feature and the first spectrum feature of the sound information, the metadata of the sound information is effectively utilized, thereby improving the stability of abnormal sound detection. After obtaining the first sound feature of the sound information, the first sound feature is subjected to diffusion processing and denoising processing to obtain the second sound feature, wherein the diffusion processing refers to forward diffusion of the first sound feature, i.e., adding noise to the first sound feature according to the first sound feature and the preset noise level, thereby obtaining noise data. De-noising refers to the reverse denoising of the noise data obtained after diffusion, that is, removing the noise in the noise data and restoring the real sound feature, that is, the second sound feature. The first sound feature and the second sound feature are compared to determine the deviation between the first sound feature and the second sound feature, and then determine whether the sound information is abnormal based on the deviation. If the deviation is too large, it means that the sound information is abnormal sound, and if the deviation is too small, it means that the sound information is not abnormal information. After obtaining the sound information, the present invention extracts the spectrum graph features and spectrum features in the sound information, and then determines the first sound feature based on the spectrum graph features and the spectrum features, and then diffuses and denoises the first sound feature to obtain the second sound feature, and finally determines whether the current sound information is abnormal sound based on the deviation between the second sound feature and the first sound feature, thereby achieving effective use of metadata, and can improve the quality of generated samples and prevent mode collapse.

[0011] The abnormal sound detection method according to the present invention may also have the following technical features:

[0012] In some technical schemes, optionally, the step of determining the first sound feature based on the first spectrum graph feature and the first spectrum feature includes: determining the second spectrum feature based on the first spectrum feature and a one-dimensional convolutional network; determining the second spectrum graph feature based on the first spectrum graph feature and a residual network; determining the first sound feature based on the second spectrum feature and the second spectrum graph feature.

[0013] In this technical solution, the step of determining the first sound feature according to the first spectrum graph feature and the first spectrum feature includes: firstly, determining the second spectrum feature according to the first spectrum feature and the one-dimensional convolutional network, that is, inputting the first spectrum feature into the one-dimensional convolutional network. Since the first spectrum feature is a one-dimensional feature, the first spectrum feature can be processed by the one-dimensional convolutional network, so that a higher-level feature, namely the second spectrum feature, can be gradually extracted. At the same time, the second spectrum graph feature is determined according to the first spectrum graph feature and the residual network. The residual network has a strong feature extraction capability. Therefore, the first spectrum graph feature is input into the residual network, and the high-level feature in the first spectrum graph feature, namely the second spectrum graph feature, can be extracted. Finally, the second spectrum feature and the second spectrum graph feature are combined to obtain the first sound feature. By processing the first spectrum feature using the one-dimensional convolutional network and processing the first spectrum graph feature using the residual network, the potential feature in the sound information, namely the first sound feature, can be obtained.

[0014] In some technical schemes, optionally, the step of determining the second spectrogram feature based on the first spectrogram feature and the residual network includes: using an attention mechanism along the time dimension to determine the first time feature of the first spectrogram feature; using an attention mechanism along the frequency dimension to determine the first frequency feature of the first spectrogram feature; and determining the second spectrogram feature based on the first time feature, the first frequency feature and the residual network.

[0015] In this technical solution, the step of determining the second spectrogram feature based on the first spectrogram feature and the residual network includes: first, using the attention mechanism to determine the first time feature of the first spectrogram feature in the time dimension, and determining the first frequency feature of the first spectrogram feature in the frequency dimension. Since the first spectrogram feature is a multidimensional feature, the first spectrogram feature can be processed in the time dimension and the frequency dimension. Then, the first frequency feature and the first time feature are combined and input into the residual network to obtain the final second spectrogram feature. By determining the features of the first spectrogram feature in the time dimension and the frequency dimension, and then combining the two features and inputting them into the residual network, the accuracy and comprehensiveness of the second spectrogram feature are guaranteed.

[0016] In some technical schemes, optionally, the step of determining the second spectrum feature based on the first spectrum feature and the one-dimensional convolutional network includes: determining the second frequency feature based on the first spectrum feature and the one-dimensional convolutional network; and processing the second frequency feature according to the attention mechanism to obtain the second spectrum feature.

[0017] In this technical solution, the step of determining the second spectrum feature according to the first spectrum feature and the one-dimensional convolutional network includes: firstly determining the second frequency feature according to the first spectrum feature and the one-dimensional convolutional network, that is, inputting the first spectrum feature into the one-dimensional convolutional network. Since the first spectrum feature has only one dimension of frequency, after inputting the first spectrum feature into the one-dimensional convolutional network, the second frequency feature of the first spectrum feature in the frequency dimension will be obtained. Finally, after obtaining the first spectrum feature, the second frequency feature is processed using the attention mechanism to obtain the second spectrum feature. By processing the first spectrum feature using the one-dimensional convolutional network and the attention mechanism, the second spectrum feature is obtained, thereby improving the accuracy of the second spectrum feature.

[0018] In some technical solutions, optionally, the step of diffusing and denoising the first sound feature to obtain the second sound feature includes: forward diffusing the first sound feature according to Gaussian noise to obtain first data information; and reverse denoising the first data information to obtain the second sound feature.

[0019] In this technical solution, the step of diffusing and denoising the first sound feature to obtain the second sound feature includes: firstly, forward diffusing the first sound feature according to Gaussian noise to obtain the first data information, that is, gradually adding random Gaussian noise to the first sound feature, thereby destroying the original data distribution, and then obtaining the first data information. Then, reverse denoising is performed on the first data information to obtain the second sound feature, that is, removing the noise in the noise data to restore the real sound feature, that is, the second sound feature. The second sound feature is obtained by diffusing and denoising the first sound feature, thereby improving the accuracy of the second sound feature.

[0020] In some technical solutions, optionally, the step of performing reverse denoising on the first data information to obtain the second sound feature includes: obtaining at least one trained encoder block and at least one decoder block; processing the first data information according to the at least one encoder block to obtain the second data information; and processing the second data information according to the at least one decoder block to obtain the second sound feature.

[0021] In the technical solution, the step of reverse denoising the first data information to obtain the second sound feature includes: first obtaining at least one trained encoder block and at least one decoder block, then inputting the first data information into the at least one trained encoder block to obtain the second data information, and then inputting the second data information into the at least one decoder block to obtain the second sound feature. The first data information is processed in sequence by the encoder block and the decoder block, thereby achieving reverse denoising of the first data information.

[0022] In some technical schemes, optionally, the step of determining whether the sound information is abnormal based on the deviation includes: determining an abnormality score corresponding to the deviation based on the deviation; matching the abnormality score with a preset score; determining that the sound information is an abnormal sound when the abnormality score is greater than or equal to the preset score; and determining that the sound information is a normal sound when the abnormality score is less than the preset score.

[0023] In the technical solution, the step of determining whether the sound information is abnormal according to the deviation includes: first determining an abnormality score according to the deviation, wherein the abnormality score corresponds to the deviation. Then matching the abnormality score with a preset score. Wherein, the preset score refers to the score corresponding to normal sound information. When the abnormality score is greater than or equal to the preset score, it means that the deviation between the first sound feature and the second sound feature is too large, so it can be determined that the current sound information is abnormal sound information. When the abnormality score is less than the preset score, it means that the deviation between the first sound feature and the second sound feature is small, so it can be determined that the current sound information is a normal sound. By judging whether the current sound information is an abnormal sound through the abnormality score, it can be judged more intuitively.

[0024] According to a second aspect of the present invention, an abnormal sound detection system is proposed, wherein the abnormal sound detection system comprises: a first acquisition module, the first acquisition module is used to acquire sound information; an extraction module, the extraction module is used to extract a first spectrum graph feature and a first spectrum feature according to the sound information; a first processing module, the first processing module is used to determine a first sound feature according to the first spectrum graph feature and the first spectrum feature; a second processing module, the second processing module is used to diffuse and denoise the first sound feature to obtain a second sound feature; a calculation module, the calculation module is used to determine the deviation between the second sound feature and the first sound feature; and a judgment module, the judgment module is used to determine whether the sound information is abnormal according to the deviation.

[0025] The abnormal sound detection system provided by the present invention mainly includes: a first acquisition module, an extraction module, a first processing module, a second processing module, a calculation module and a judgment module. Among them, the first acquisition module can obtain sound information, wherein the method of obtaining the sound information can be input by the user, or the system can actively obtain it. After obtaining the sound information, the extraction module extracts the first spectrum graph feature and the first spectrum feature in the sound information according to the sound information, wherein the first spectrum graph feature can represent the amplitude of the sound information at different frequencies, and the first spectrum feature can represent the characteristics and laws displayed by the sound information in the frequency domain. After obtaining the first spectrum graph feature and the first spectrum feature, the first processing module processes the first spectrum graph feature and the first spectrum feature to obtain the first sound feature. By determining the expression feature of the sound information, i.e., the first sound feature, according to the first spectrum graph feature and the first spectrum feature of the sound information, the metadata of the sound information is effectively utilized, thereby improving the stability of abnormal sound detection. After obtaining the first sound feature of the sound information, the second processing module performs diffusion processing and denoising processing on the first sound feature to obtain the second sound feature, wherein the diffusion processing refers to forward diffusion of the first sound feature, that is, adding noise to the first sound feature according to the first sound feature and the preset noise level, thereby obtaining noise data. Denoising processing refers to reverse denoising processing of the noise data obtained after diffusion, that is, removing the noise in the noise data to restore the true sound feature, that is, the second sound feature. The calculation module compares the first sound feature with the second sound feature to determine the deviation between the first sound feature and the second sound feature, and then the judgment module determines whether the sound information is abnormal based on the deviation. If the deviation is too large, it means that the sound information is abnormal sound, and if the deviation is too small, it means that the sound information is not abnormal information. The present invention extracts spectrogram features and spectrum features from the sound information after obtaining the sound information, then determines a first sound feature based on the spectrogram features and the spectrum features, then performs diffusion and denoising processing on the first sound feature to obtain a second sound feature, and finally determines whether the current sound information is an abnormal sound based on the deviation between the second sound feature and the first sound feature, thereby achieving the technical effect of effectively utilizing metadata, improving the quality of generated samples and preventing mode collapse.

[0026] According to a third aspect of the present invention, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned abnormal sound detection methods when executing the computer program.

[0027] The electronic device provided by the present invention implements the steps of the abnormal sound detection method when the processor executes the computer program, which can achieve the technical effects of any of the above technical solutions and will not be repeated here.

[0028] According to a fourth aspect of the present invention, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above-mentioned abnormal sound detection methods are implemented.

[0029] The storage medium provided by the present invention implements the steps of the above-mentioned abnormal sound detection method when the computer program is executed by the processor, and can achieve the technical effects of any of the above-mentioned technical solutions, which will not be repeated here.

[0030] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the abnormal sound detection method in any of the above technical solutions.

[0031] The computer program product provided by this technical solution implements the steps of the abnormal sound detection method of any technical solution of the present invention, and thus has all the beneficial effects of the abnormal sound detection method of any technical solution of the present invention, which will not be repeated here.

[0032] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0034] Figure 1 A schematic diagram showing a flow chart of an abnormal sound detection method according to an embodiment of the present invention;

[0035] Figure 2 A schematic flow chart showing a step of determining a first sound feature according to a first spectrum graph feature and a first spectrum feature in an abnormal sound detection method according to an embodiment of the present invention;

[0036] Figure 3 A schematic flow chart showing a step of determining a second spectrum feature according to a first spectrum graph feature and a residual network in an abnormal sound detection method according to an embodiment of the present invention;

[0037] Figure 4 A schematic flow chart showing a step of determining a second spectrum feature according to a first spectrum feature and a one-dimensional convolutional network in an abnormal sound detection method according to an embodiment of the present invention;

[0038] Figure 5 A schematic flow chart showing the steps of diffusing and denoising a first sound feature to obtain a second sound feature in an abnormal sound detection method according to an embodiment of the present invention;

[0039] Figure 6A schematic flow chart showing a step of reversely denoising first data information to obtain a second sound feature in an abnormal sound detection method according to an embodiment of the present invention;

[0040] Figure 7 A schematic flow chart showing the step of determining whether sound information is abnormal according to a deviation in an abnormal sound detection method according to an embodiment of the present invention;

[0041] Figure 8 A schematic diagram showing the principle of an abnormal sound detection method according to an embodiment of the present invention;

[0042] Figure 9-a A schematic diagram showing the results without the time-frequency self-supervision module;

[0043] Figure 9-b A schematic diagram of the results of the time-frequency self-supervision module is shown;

[0044] Fig.10 A schematic block diagram of an abnormal sound detection system according to an embodiment of the present invention is shown;

[0045] Fig.11 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0046] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0048] Figure 1 A schematic flow chart of an abnormal sound detection method according to an embodiment of the present invention is shown.

[0049] Among them, the method includes:

[0050] Step 102: Acquire sound information;

[0051] Step 104: extracting a first spectrogram feature and a first spectrum feature according to the sound information;

[0052] Step 106: determining a first sound feature according to the first spectrogram feature and the first spectrum feature;

[0053] Step 108: diffusing and denoising the first sound feature to obtain a second sound feature;

[0054] Step 110: Determine a deviation between the second sound feature and the first sound feature;

[0055] Step 112: Determine whether the sound information is abnormal based on the deviation.

[0056] The abnormal sound detection method provided by the present invention mainly includes: first obtaining sound information, wherein the method of obtaining the sound information can be input by the user or actively obtained by the system. After obtaining the sound information, extracting the first spectrum graph feature and the first spectrum feature in the sound information according to the sound information, wherein the method of extracting the first spectrum graph feature of the sound information can be to perform short-time Fourier transform processing on the sound information, thereby obtaining the first spectrum graph feature of the sound information. The first spectrum graph feature can represent the amplitude of the sound information at different frequencies. The method of extracting the first spectrum feature of the sound information can be to perform fast Fourier transform processing on the sound information, thereby obtaining the first spectrum feature of the sound information, and the first spectrum feature can represent the characteristics and laws displayed by the sound information in the frequency domain. After obtaining the first spectrum graph feature and the first spectrum feature, the first spectrum graph feature and the first spectrum feature are processed to obtain the first sound feature, wherein the first spectrum graph feature and the first spectrum feature can be processed separately by a dual-path ASD (Anomalous sound detection) framework, and then the processed first spectrum graph feature and the first spectrum feature are combined to obtain the final first sound feature. By determining the expression feature of the sound information, i.e., the first sound feature, according to the first spectrum feature and the first spectrum feature of the sound information, the metadata of the sound information is effectively utilized, thereby improving the stability of abnormal sound detection. After obtaining the first sound feature of the sound information, the first sound feature is subjected to diffusion processing and denoising processing to obtain the second sound feature, wherein the diffusion processing refers to forward diffusion of the first sound feature, i.e., adding noise to the first sound feature according to the first sound feature and the preset noise level, thereby obtaining noise data. The denoising processing refers to reverse denoising processing of the noise data obtained after diffusion, i.e., removing the noise in the noise data and restoring the true sound feature, i.e., the second sound feature. The first sound feature and the second sound feature are compared to determine the deviation between the first sound feature and the second sound feature, and then determine whether the sound information is abnormal according to the deviation. If the deviation is too large, it means that the sound information is abnormal sound, and if the deviation is too small, it means that the sound information is not abnormal information. The present invention extracts spectrogram features and spectrum features from the sound information after obtaining the sound information, then determines a first sound feature based on the spectrogram features and the spectrum features, then performs diffusion and denoising processing on the first sound feature to obtain a second sound feature, and finally determines whether the current sound information is an abnormal sound based on the deviation between the second sound feature and the first sound feature, thereby achieving the technical effect of effectively utilizing metadata, improving the quality of generated samples and preventing mode collapse.

[0057] Figure 2A schematic flow chart of a step of determining a first sound feature according to a first spectrogram feature and a first spectrum feature in an abnormal sound detection method according to an embodiment of the present invention is shown; wherein the step of determining the first sound feature according to the first spectrogram feature and the first spectrum feature comprises:

[0058] Step 202: determining a second spectrum feature according to the first spectrum feature and a one-dimensional convolutional network;

[0059] Step 204: determining a second spectrum graph feature according to the first spectrum graph feature and the residual network;

[0060] Step 206: Determine the first sound feature according to the second frequency spectrum feature and the second spectrum graph feature.

[0061] In this embodiment, the step of determining the first sound feature according to the first spectrum graph feature and the first spectrum feature includes: firstly determining the second spectrum feature according to the first spectrum feature and the one-dimensional convolutional network, that is, inputting the first spectrum feature into the one-dimensional convolutional network. Since the first spectrum feature is a one-dimensional feature, the first spectrum feature can be processed by the one-dimensional convolutional network, so that a higher-level feature, namely the second spectrum feature, can be gradually extracted. At the same time, the second spectrum graph feature is determined according to the first spectrum graph feature and the residual network. The residual network has a strong feature extraction capability. Therefore, the first spectrum graph feature is input into the residual network, and the high-level feature in the first spectrum graph feature, namely the second spectrum graph feature, can be extracted. Finally, the second spectrum feature and the second spectrum graph feature are combined to obtain the first sound feature. By processing the first spectrum feature using the one-dimensional convolutional network and processing the first spectrum graph feature using the residual network, the potential feature in the sound information, namely the first sound feature, can be obtained.

[0062] Figure 3 A schematic flow chart of a step of determining a second spectrum feature according to a first spectrum map feature and a residual network in an abnormal sound detection method according to an embodiment of the present invention is shown; wherein the step of determining the second spectrum feature according to the first spectrum map feature and the residual network comprises:

[0063] Step 302: Determine a first time feature of a first spectrogram feature using an attention mechanism along a time dimension;

[0064] Step 304: Determine a first frequency feature of a first spectrogram feature using an attention mechanism along the frequency dimension;

[0065] Step 306: Determine a second spectrum graph feature according to the first time feature, the first frequency feature, and the residual network.

[0066] In this embodiment, the step of determining the second spectrum feature based on the first spectrum feature and the residual network includes: first, using the attention mechanism to determine the first time feature of the first spectrum feature in the time dimension, and determining the first frequency feature of the first spectrum feature in the frequency dimension. Since the first spectrum feature is a multidimensional feature, the first spectrum feature can be processed in the time dimension and the frequency dimension. Then, the first frequency feature and the first time feature are combined and input into the residual network to obtain the final second spectrum feature. By determining the features of the first spectrum feature in the time dimension and the frequency dimension, and then combining the two features and inputting them into the residual network, the accuracy and comprehensiveness of the second spectrum feature are guaranteed.

[0067] Figure 4 A schematic diagram of the steps of determining the second spectrum graph feature according to the first spectrum feature and the one-dimensional convolutional network in an abnormal sound detection method according to an embodiment of the present invention is shown; wherein the step of determining the second spectrum graph feature according to the first spectrum feature and the one-dimensional convolutional network includes:

[0068] Step 402: determining a second frequency feature according to the first frequency spectrum feature and a one-dimensional convolutional network;

[0069] Step 404: Process the second frequency feature according to the attention mechanism to obtain a second spectrum feature.

[0070] In this embodiment, the step of determining the second spectrum feature according to the first spectrum feature and the one-dimensional convolutional network includes: firstly determining the second frequency feature according to the first spectrum feature and the one-dimensional convolutional network, that is, inputting the first spectrum feature into the one-dimensional convolutional network. Since the first spectrum feature has only one dimension of frequency, after inputting the first spectrum feature into the one-dimensional convolutional network, the second frequency feature of the first spectrum feature in the frequency dimension will be obtained. Finally, after obtaining the first spectrum feature, the second frequency feature is processed using the attention mechanism to obtain the second spectrum feature. By processing the first spectrum feature using the one-dimensional convolutional network and the attention mechanism, the second spectrum feature is obtained, thereby improving the accuracy of the second spectrum feature.

[0071] Figure 5 A flow chart showing the steps of diffusing and denoising a first sound feature to obtain a second sound feature in an abnormal sound detection method according to an embodiment of the present invention is shown; wherein the steps of diffusing and denoising the first sound feature to obtain the second sound feature include:

[0072] Step 502: forward diffusion of the first sound feature according to Gaussian noise to obtain first data information;

[0073] Step 504: Perform reverse denoising on the first data information to obtain a second sound feature.

[0074] In this embodiment, the step of diffusing and denoising the first sound feature to obtain the second sound feature includes: first, forward diffusing the first sound feature according to Gaussian noise to obtain the first data information, that is, gradually adding random Gaussian noise to the first sound feature, thereby destroying the original data distribution, and then obtaining the first data information. Then, reverse denoising is performed on the first data information to obtain the second sound feature, that is, removing the noise in the noise data to restore the real sound feature, that is, the second sound feature. In the process of diffusing and denoising the first sound feature to obtain the second sound feature, the first sound feature can be input into a denoising diffusion probability model (Denoising Diffusion Probabilistic Models, DDPM), and the DDPM model is used to diffuse and denoise the first sound feature to obtain the original sound feature, that is, the second sound feature. The second sound feature is obtained by diffusing and denoising the first sound feature, thereby improving the accuracy of the second sound feature.

[0075] Figure 6 A flow chart showing a step of reverse denoising the first data information to obtain a second sound feature in an abnormal sound detection method according to an embodiment of the present invention is shown; wherein the step of reverse denoising the first data information to obtain the second sound feature includes:

[0076] Step 602: Obtain at least one trained encoder block and at least one decoder block;

[0077] Step 604: Process the first data information according to at least one encoder block to obtain second data information;

[0078] Step 606: Process the second data information according to at least one decoder block to obtain a second sound feature.

[0079] In this embodiment, the step of reverse denoising the first data information to obtain the second sound feature includes: first obtaining at least one trained encoder block and at least one decoder block, then inputting the first data information into the at least one trained encoder block to obtain the second data information, and then inputting the second data information into the at least one decoder block to obtain the second sound feature. The number of encoder blocks can be 2, and the number of decoder blocks can also be 2, that is, the number of encoder blocks corresponds to the number of decoder blocks. The first data information is processed in sequence by the encoder block and the decoder block, thereby achieving reverse denoising of the first data information.

[0080] Figure 7A flow chart showing a step of determining whether sound information is abnormal according to a deviation in an abnormal sound detection method according to an embodiment of the present invention; wherein the step of determining whether sound information is abnormal according to the deviation comprises:

[0081] Step 702: determining an anomaly score corresponding to the deviation according to the deviation;

[0082] Step 704: Match the abnormality score with a preset score;

[0083] Step 706: when the abnormality score is greater than or equal to a preset score, determining that the sound information is an abnormal sound;

[0084] Step 708: When the abnormality score is less than a preset score, determine that the sound information is a normal sound.

[0085] In this embodiment, the step of determining whether the sound information is abnormal according to the deviation includes: first determining an abnormality score according to the deviation, wherein the abnormality score corresponds to the deviation, that is, firstly presetting some weight values, and then calculating the abnormality score according to the weight values ​​and the deviation. Then match the abnormality score with the preset score. Among them, the preset score refers to the score corresponding to the normal sound information. When the abnormality score is greater than or equal to the preset score, it means that the deviation between the first sound feature and the second sound feature is too large, so it can be determined that the current sound information is abnormal sound information. When the abnormality score is less than the preset score, it means that the deviation between the first sound feature and the second sound feature is small, so it can be determined that the current sound information is a normal sound. Through the abnormality score, it can be judged more intuitively whether the current sound information is an abnormal sound.

[0086] Figure 8 FIG. 1 is a schematic diagram showing the principle of an abnormal sound detection method according to an embodiment of the present invention; Figure 8As shown, the present invention mainly includes: a self-supervised learning module and a denoising diffusion module. In the self-supervised learning module, a dual-path ASD framework is used to complete the auxiliary classification task, and a self-attention module is introduced to enhance the attention to the time and frequency of the input. Exemplarily, a dual-path ASD framework is used as the backbone of the self-supervised learning module to extract spectrogram features and spectral features respectively. In the spectrogram features can be extracted from the sound information by short-time Fourier transform, and the spectral features can be extracted from the sound information by fast Fourier transform. Then, the spectrogram features are processed by a residual network layer, and the spectral features are sampled by a one-dimensional convolution layer. Compared with using a single path alone, the dual-path framework has been proven to be able to effectively extract expression features from the sound. The self-attention module is then integrated into two branches along a specific dimension. In this work, an improved version of the Squeeze and Excitation Network (SENet) module is used to enhance the information of a specific dimension, and its definition is shown in formula (1):

[0087] y=x+σ(avgpool i (x)×W T +b)×x; (1)

[0088] where avgpool i is the average pooling operation performed along a specific i dimension, and σ is the sigmoid (activation) function operation. W and b are learning parameters, while x represents the input features, y represents the output features, and T represents the steps. In the spectrogram path, self-attention modules, i.e., attention mechanisms, can be used along the time and frequency dimensions. In the spectral path, self-attention modules, i.e., attention mechanisms, can be used along the frequency dimension. The representations encoded from these two paths are combined to form a latent feature z, i.e., a second sound feature, which is used to assist the classification task through the classification head training.

[0089] The denoising diffusion module is designed based on the denoising diffusion probability model (Denoising Diffusion Probabilistic Models, DDPM). DDPM performs well in sample data generation quality and training stability. The denoising diffusion module consists of a forward diffusion process q(z t |z t-1 ) and the reverse denoising process p θ (z t-1 |z t ). The original data distribution is gradually destroyed by latent features and random Gaussian noise, where the Markov process is followed in this process, as shown in formula (2):

[0090]

[0091] where t = 1,…,T. The amount of noise added at each step is determined by the variance schedule β t This makes it possible that at any given moment z t The distribution of can be based only on the original input z ％ and noise adding step t. Its definition is shown in formula (3):

[0092]

[0093] where α t =1-β t , In the denoising process, the model learns to optimize the parameter θ. The reverse process is defined as formula (4) as follows:

[0094]

[0095] where μ θ (z t ,t) is the noise mean predicted by the network, is the operation equation and I is the parameter.

[0096] The training of DDPM can be viewed as an autoencoder. In our work, two encoder blocks (EBs) and two decoder blocks (DBs), each consisting of a dense layer, are used to train DDPM. The size of the input and output are the same, and the output is the predicted noise ε t The goal is to make the predicted noise consistent with the true noise e, so the mean square error (MSE) loss Loss can be used for training, as shown in formula (5):

[0097] Loss = arg min [|| ε t -e || 5 ]; (5)

[0098] Based on the prediction noise ε t , the denoising process is shown in formula (6):

[0099]

[0100] In anomaly detection (ASD), the anomaly score is derived from the reconstruction error. The larger the deviation of a data point, the higher the possibility of being classified as an anomaly. The anomaly score, denoted as error, is shown in formula (7):

[0101]

[0102] Among them, error represents the anomaly score, z ％ is the potential feature, z′ ％ Represents the features after denoising.

[0103] Exemplarily, the following data set can be used to verify the abnormal sound detection method proposed in the present invention, wherein the data set includes seven types of machines: bearings, fans, gearboxes, sliders, toy cars, toy trains, and valves. The data set is divided into a training set and a test set. The training set contains 1000 normal sound samples of each machine type, and the test set contains 100 normal sound samples and 100 abnormal sound samples. In addition, the training set provides the attributes of each sample, which can be used in the training process. These attributes can be used as classification labels in the self-supervised learning module. The performance is evaluated using the area under the curve (Area Under the Curve, AUC) and partial AUC (pAUC) of the receiver operating characteristic (ROC). In the method provided by the present invention, the present invention reports the harmonic mean of all AUCs and pAUCs of each machine type, as well as the harmonic mean of the entire data set.

[0104] The proposed method is compared with the baseline system and other advanced methods, as shown in Table 1 below. The proposed method achieves the highest total harmonic mean, which is 10.41% and 6.09% higher than the baseline model and 1.21% higher than the advanced self-supervised method FeatEx. The self-implemented baseline in the table is a combination of the dual-path ASD framework and KNN as an anomaly detector. The results show that the proposed SSDM improves the overall performance of ASD.

[0105] Table 1

[0106]

[0107]

[0108] In order to verify the role of the self-supervised learning module and the denoising diffusion module in SSDM, the present invention conducted an ablation experiment. Table 2 below shows the harmonic mean of different modules of SSDM in all categories. The results show that SSDM is very effective when both modules are included. The self-supervised learning module significantly improves the overall ASD performance, while the denoising diffusion module complements it to achieve better ASD performance.

[0109] Table 2

[0110]

[0111] To demonstrate the effectiveness of incorporating time-frequency self-attention in the self-supervised learning module, t-SNE can be used to visualize the embedding of the slider. Figure 9-a and Figure 9-bAs shown, after adding time-frequency self-supervision, the overlap between abnormal and normal embeddings is significantly reduced, which will be helpful for the subsequent anomaly detection module.

[0112] Fig.10 A schematic block diagram of an abnormal sound detection system according to an embodiment of the present invention is shown; wherein the abnormal sound detection system 100 comprises:

[0113] The first acquisition module 1002 is used to acquire sound information;

[0114] An extraction module 1004, configured to extract a first spectrogram feature and a first spectrum feature according to the sound information;

[0115] A first processing module 1006, configured to determine a first sound feature according to the first spectrum graph feature and the first spectrum feature;

[0116] A second processing module 1008, configured to diffuse and denoise the first sound feature to obtain a second sound feature;

[0117] A calculation module 1010, configured to determine a deviation between the second sound feature and the first sound feature;

[0118] The judgment module 1012 is used to determine whether the sound information is abnormal according to the deviation.

[0119] The abnormal sound detection system 100 provided by the present invention mainly includes: a first acquisition module 1002, an extraction module 1004, a first processing module 1006, a second processing module 1008, a calculation module 1010 and a judgment module 1012. Among them, the first acquisition module 1002 can obtain sound information, wherein the method of obtaining the sound information can be input by the user, or the system can actively obtain it. After obtaining the sound information, the extraction module 1004 extracts the first spectrum graph feature and the first spectrum feature in the sound information according to the sound information, wherein the method for extracting the first spectrum graph feature of the sound information can be to perform short-time Fourier transform processing on the sound information, thereby obtaining the first spectrum graph feature of the sound information. The first spectrum graph feature can represent the amplitude of the sound information at different frequencies. The method for extracting the first spectrum feature of the sound information can be to perform fast Fourier transform processing on the sound information, thereby obtaining the first spectrum feature of the sound information, and the first spectrum feature can represent the characteristics and laws displayed by the sound information in the frequency domain. After obtaining the first spectrogram feature and the first spectrum feature, the first processing module 1006 processes the first spectrogram feature and the first spectrum feature to obtain the first sound feature, wherein the first spectrogram feature and the first spectrum feature can be processed separately by a dual-path ASD (Anomalous sound detection) framework, and then the processed first spectrogram feature and the first spectrum feature are combined to obtain the final first sound feature. By determining the expression feature of the sound information, i.e., the first sound feature, according to the first spectrogram feature and the first spectrum feature of the sound information, the metadata of the sound information is effectively utilized, thereby improving the stability of abnormal sound detection. After obtaining the first sound feature of the sound information, the second processing module 1008 performs diffusion processing and denoising processing on the first sound feature to obtain the second sound feature, wherein the diffusion processing refers to forward diffusion of the first sound feature, i.e., adding noise to the first sound feature according to the first sound feature and the preset noise level, thereby obtaining noise data. Denoising processing refers to reverse denoising processing of the noise data obtained after diffusion, i.e., removing the noise in the noise data, and restoring the real sound feature, i.e., the second sound feature. The calculation module 1010 compares the first sound feature with the second sound feature to determine the deviation between the first sound feature and the second sound feature, and then the judgment module 1012 determines whether the sound information is abnormal based on the deviation. If the deviation is too large, it means that the sound information is abnormal sound. If the deviation is too small, it means that the sound information is not abnormal information.The present invention extracts spectrogram features and spectrum features from the sound information after obtaining the sound information, then determines a first sound feature based on the spectrogram features and the spectrum features, then performs diffusion and denoising processing on the first sound feature to obtain a second sound feature, and finally determines whether the current sound information is an abnormal sound based on the deviation between the second sound feature and the first sound feature, thereby achieving the technical effect of effectively utilizing metadata, improving the quality of generated samples and preventing mode collapse.

[0120] In some embodiments, optionally, the first processing module 1006 is used to determine the second spectrum feature based on the first spectrum feature and the one-dimensional convolutional network; determine the second spectrum feature based on the first spectrum feature and the residual network; determine the first sound feature based on the second spectrum feature and the second spectrum feature.

[0121] In this embodiment, the first processing module 1006 is used to determine the second spectrum feature according to the first spectrum feature and the one-dimensional convolutional network, that is, the first spectrum feature is input into the one-dimensional convolutional network. Since the first spectrum feature is a one-dimensional feature, the first spectrum feature can be processed by the one-dimensional convolutional network, so that a higher-level feature, namely the second spectrum feature, can be gradually extracted. At the same time, the second spectrum feature is determined according to the first spectrum map feature and the residual network. The residual network has a strong feature extraction capability. Therefore, the first spectrum map feature is input into the residual network, and the high-level feature in the first spectrum map feature, namely the second spectrum map feature, can be extracted. Finally, the second spectrum feature and the second spectrum map feature are combined to obtain the first sound feature. By processing the first spectrum feature using the one-dimensional convolutional network and processing the first spectrum map feature using the residual network, the potential feature in the sound information, namely the first sound feature, can be obtained.

[0122] In some embodiments, optionally, the first processing module 1006 is specifically used to determine a first time feature of a first spectrum graph feature using an attention mechanism along a time dimension; determine a first frequency feature of a first spectrum graph feature using an attention mechanism along a frequency dimension; and determine a second spectrum graph feature based on the first time feature, the first frequency feature, and a residual network.

[0123] In this embodiment, the first processing module 1006 is specifically used to determine the first time feature of the first spectrum graph feature in the time dimension and the first frequency feature of the first spectrum graph feature in the frequency dimension by using the attention mechanism. Since the first spectrum graph feature is a multidimensional feature, the first spectrum graph feature can be processed in the time dimension and the frequency dimension. Then, the first frequency feature and the first time feature are combined and input into the residual network to obtain the final second spectrum graph feature. By determining the features of the first spectrum graph feature in the time dimension and the frequency dimension, and then combining the two features and inputting them into the residual network, the accuracy and comprehensiveness of the second spectrum graph feature are guaranteed.

[0124] In some embodiments, optionally, the first processing module 1006 is further specifically used to determine a second frequency feature according to the first frequency spectrum feature and a one-dimensional convolutional network; and process the second frequency feature according to the attention mechanism to obtain a second frequency spectrum feature.

[0125] In this embodiment, the first processing module 1006 is also specifically used to determine the second frequency feature based on the first spectrum feature and the one-dimensional convolutional network, that is, the first spectrum feature is input into the one-dimensional convolutional network. Since the first spectrum feature has only one dimension of frequency, after the first spectrum feature is input into the one-dimensional convolutional network, the second frequency feature of the first spectrum feature in the frequency dimension will be obtained. Finally, after obtaining the first spectrum feature, the second frequency feature is processed using the attention mechanism to obtain the second spectrum feature. By processing the first spectrum feature using the one-dimensional convolutional network and the attention mechanism, the second spectrum feature is obtained, thereby improving the accuracy of the second spectrum feature.

[0126] In some embodiments, optionally, the second processing module 1008 is used to perform forward diffusion on the first sound feature according to Gaussian noise to obtain first data information; and perform reverse denoising on the first data information to obtain second sound features.

[0127] In this embodiment, the second processing module 1008 is used to forward diffuse the first sound feature according to Gaussian noise to obtain the first data information, that is, to gradually add random Gaussian noise to the first sound feature, thereby destroying the original data distribution, and then obtaining the first data information. Then, the first data information is reversely denoised to obtain the second sound feature, that is, to remove the noise in the noise data and restore the real sound feature, that is, the second sound feature. Among them, in the process of diffusing and denoising the first sound feature to obtain the second sound feature, the first sound feature can be input into a denoising diffusion probability model (Denoising Diffusion Probabilistic Models, DDPM), and the first sound feature is diffused and denoised using the DDPM model to obtain the original sound feature, that is, the second sound feature. The second sound feature is obtained by diffusing and denoising the first sound feature, thereby improving the accuracy of the second sound feature.

[0128] In some embodiments, optionally, the second processing module 1008 is specifically used to obtain at least one trained encoder block and at least one decoder block; process the first data information according to the at least one encoder block to obtain the second data information; and process the second data information according to the at least one decoder block to obtain the second sound feature.

[0129] In this embodiment, the second processing module 1008 is specifically used to obtain at least one trained encoder block and at least one decoder block, and then input the first data information into the at least one trained encoder block to obtain the second data information, and then input the second data information into at least one decoder block to obtain the second sound feature. Among them, the number of encoder blocks can be 2, and the number of decoder blocks can also be 2, that is, the number of encoder blocks corresponds to the number of decoder blocks. The first data information is processed in sequence by the encoder block and the decoder block, thereby realizing reverse denoising of the first data information.

[0130] In some embodiments, optionally, the judgment module 1012 is used to determine an abnormality score corresponding to the deviation based on the deviation; match the abnormality score with a preset score; determine that the sound information is an abnormal sound when the abnormality score is greater than or equal to the preset score; and determine that the sound information is a normal sound when the abnormality score is less than the preset score.

[0131] In this embodiment, the judgment module 1012 is used to determine the abnormal score according to the deviation, wherein the abnormal score corresponds to the deviation, that is, some weight values ​​are first preset, and then the abnormal score is calculated according to the weight value and the deviation. Then the abnormal score is matched with the preset score. Among them, the preset score refers to the score corresponding to the normal sound information. When the abnormal score is greater than or equal to the preset score, it means that the deviation between the first sound feature and the second sound feature is too large, so it can be determined that the current sound information is abnormal sound information. When the abnormal score is less than the preset score, it means that the deviation between the first sound feature and the second sound feature is small, so it can be determined that the current sound information is normal sound. Through the abnormal score, it can be judged more intuitively whether the current sound information is an abnormal sound.

[0132] Fig.11 A schematic block diagram of an electronic device of an embodiment of the present invention is shown; wherein the electronic device 110 includes a memory 1102, a processor 1104, and a computer program stored in the memory 1102 and executable on the processor 1104, and when the processor 1104 executes the computer program, the steps of any of the above-mentioned abnormal sound detection methods are implemented.

[0133] The electronic device 110 provided by the present invention implements the steps of the abnormal sound detection method when the processor 1104 executes the computer program, which can achieve the technical effects of any of the above embodiments and will not be repeated here.

[0134] An embodiment of the present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned abnormal sound detection methods are implemented.

[0135] The storage medium provided by the present invention implements the steps of the abnormal sound detection method when the computer program is executed by the processor, and can achieve the technical effects of any of the above embodiments, which will not be repeated here.

[0136] An embodiment of the present invention provides a computer program product, including a computer program, which implements the steps of the abnormal sound detection method in any of the above embodiments when executed by a processor.

[0137] The computer program product provided in this embodiment implements the steps of the abnormal sound detection method in any embodiment of the present invention, and thus has all the beneficial effects of the abnormal sound detection method in any embodiment of the present invention, which will not be described in detail here.

[0138] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance unless otherwise clearly specified and limited; the terms "connection", "installation", "fixation" and the like should be understood in a broad sense, for example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0139] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0140] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormal sound, characterized in that: include: Get sound information; Extracting a first spectrogram feature and a first spectrum feature according to the sound information; determining a first sound feature according to the first spectrogram feature and the first frequency spectrum feature; Diffusion and denoising of the first sound feature are performed to obtain a second sound feature; determining a deviation between the second sound characteristic and the first sound characteristic; It is determined whether the sound information is abnormal according to the deviation.

2. The abnormal sound detection method according to claim 1, characterized in that: The step of determining the first sound feature according to the first spectrum graph feature and the first spectrum feature comprises: Determining a second spectrum feature according to the first spectrum feature and a one-dimensional convolutional network; Determine a second spectrogram feature based on the first spectrogram feature and the residual network; The first sound feature is determined according to the second frequency spectrum feature and the second spectrogram feature.

3. The abnormal sound detection method according to claim 2, characterized in that: The step of determining the second spectrum map feature according to the first spectrum map feature and the residual network comprises: Determining a first temporal feature of the first spectrogram feature using an attention mechanism along a temporal dimension; Determining a first frequency feature of the first spectrogram feature using an attention mechanism along the frequency dimension; The second spectrum graph feature is determined according to the first time feature, the first frequency feature and the residual network.

4. The abnormal sound detection method according to claim 2, characterized in that: The step of determining the second spectrum feature according to the first spectrum feature and a one-dimensional convolutional network comprises: Determining a second frequency feature according to the first frequency spectrum feature and the one-dimensional convolutional network; The second frequency feature is processed according to the attention mechanism to obtain the second spectrum feature.

5. The abnormal sound detection method according to any one of claims 1 to 4, characterized in that: The step of diffusing and denoising the first sound feature to obtain the second sound feature comprises: Performing forward diffusion on the first sound feature according to Gaussian noise to obtain first data information; The first data information is subjected to reverse denoising to obtain the second sound feature.

6. The abnormal sound detection method according to claim 5, characterized in that: The step of performing reverse denoising on the first data information to obtain the second sound feature includes: Obtaining at least one trained encoder block and at least one decoder block; Processing the first data information according to at least one of the encoder blocks to obtain second data information; The second sound feature is obtained by processing the second data information according to at least one of the decoder blocks.

7. The abnormal sound detection method according to any one of claims 1 to 4, characterized in that: The step of determining whether the sound information is abnormal according to the deviation comprises: determining an anomaly score corresponding to the deviation according to the deviation; Matching the anomaly score with a preset score; When the abnormality score is greater than or equal to the preset score, determining that the sound information is an abnormal sound; When the abnormality score is less than the preset score, the sound information is determined to be a normal sound.

8. An abnormal sound detection system, characterized in that: include: A first acquisition module, wherein the first acquisition module is used to acquire sound information; An extraction module, the extraction module is used to extract a first spectrogram feature and a first spectrum feature according to the sound information; a first processing module, the first processing module being used to determine a first sound feature according to the first spectrum graph feature and the first spectrum feature; A second processing module, the second processing module is used to diffuse and denoise the first sound feature to obtain a second sound feature; a calculation module, the calculation module being used to determine a deviation between the second sound feature and the first sound feature; A judgment module is used to determine whether the sound information is abnormal according to the deviation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the abnormal sound detection method according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the abnormal sound detection method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the abnormal sound detection method according to any one of claims 1 to 7 are implemented.

Images