Unsupervised sound anomaly recognition method and device based on mask strategy

Abstract

The invention provides an unsupervised sound anomaly recognition method and device based on a mask strategy, and the method comprises the steps: obtaining normal audio data of machine equipment, and carrying out the preprocessing of the normal audio data, so as to obtain a preprocessed logarithmic Mel spectrogram; an anomaly recognition model of the deep convolutional generative adversarial network is constructed, and the anomaly recognition model comprises a generator, a global discriminator and a random local discriminator; the preprocessed logarithm Mel spectrogram is masked, an anomaly recognition model is trained according to the masked logarithm Mel spectrogram and the preprocessed logarithm Mel spectrogram so as to obtain a trained anomaly recognition model, and in the training process, the logarithm Mel spectrogram and the preprocessed logarithm Mel spectrogram are matched with each other; optimizing the anomaly recognition model by adopting mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss; adopting a trained anomaly recognition model to obtain an anomaly score so as to determine whether the corresponding to-be-recognized sample is abnormal or not; therefore, the sound anomaly detection effect is improved.

Description

Technical Field

[0001] This invention relates to the field of anomaly recognition technology, and in particular to an unsupervised audio anomaly recognition method based on a masking strategy, an unsupervised audio anomaly recognition device based on a masking strategy, a computer-readable storage medium, and a computer device. Background Technology

[0002] In related technologies, audio data suffers from domain shift, which refers to the difference in acoustic characteristics between training and test data, such as differences in operating speed, machine load, and ambient noise. Because these shifts are caused by factors other than anomalies, models trained using source domain data are prone to false positives. Reconstruction models possess strong fitting capabilities, and reconstruction-based methods are a solution. Reconstruction models mainly include autoencoders, generative adversarial networks, and their variants. Ideally, the reconstruction model is trained on normal audio data, learning its feature distribution. During the detection phase, the reconstruction error is used for anomaly detection. Since the model has not seen anomalous samples, the reconstruction error of anomalous samples is large, thus they are judged as anomalies.

[0003] Reconstruction methods have attracted widespread attention from researchers due to their strong generalization ability. Deep convolutional generative adversarial networks (GANs) are widely used, combining convolutional neural networks and GANs to improve training stability. However, several problems remain unresolved. On one hand, reconstruction models lack generalization ability for anomaly samples. Some researchers use reconstruction error as an indicator of anomaly detection; however, this method ignores the fact that when anomaly data is similar to normal data, the model will also reconstruct some anomaly samples, leading to a decrease in detection performance. On the other hand, most reconstruction models are trained using mean squared error, which is somewhat similar to principal component analysis. When the distributions of normal and anomaly data highly overlap, the reconstruction model focuses on denoising. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose an unsupervised sound anomaly recognition method based on a masking strategy. By introducing a random local discriminator at the feature level, the model can extract normal features from audio data at different granularities. Simultaneously, a random masking strategy is introduced into the generator, enabling it to deeply learn the intrinsic structure and salient features of the data. To avoid the problem of reconstructing abnormal data by the reconstructing model, a feature constraint loss function is designed to obtain the discriminator's judgment at the spectral feature level, guiding the generator to reconstruct a more realistic spectral image, thereby improving the sound anomaly detection effect.

[0005] A second objective of this invention is to provide a computer-readable storage medium.

[0006] The third objective of this invention is to provide a computer device.

[0007] The fourth objective of this invention is to propose an unsupervised sound anomaly recognition device based on a masking strategy.

[0008] To achieve the above objectives, a first aspect of the present invention proposes an unsupervised audio anomaly recognition method based on a masking strategy. The method includes the following steps: acquiring normal audio data from a machine; preprocessing the normal audio data to obtain a preprocessed log-Mel spectrum; constructing an anomaly recognition model using a deep convolutional generative adversarial network, wherein the anomaly recognition model includes a generator, a global discriminator, and a random local discriminator; masking the preprocessed log-Mel spectrum to obtain a masked log-Mel spectrum; and training the anomaly recognition model based on the masked log-Mel spectrum and the preprocessed log-Mel spectrum to obtain a trained anomaly recognition model. During training, a masking loss mechanism is used. The anomaly detection model is optimized using reconstruction loss, local feature constraint loss, and global feature constraint loss. The sample to be identified is input into the trained anomaly detection model to obtain a corresponding anomaly score. This score is then compared with a preset threshold to determine whether the sample to be identified is abnormal. Thus, by introducing a random local discriminator at the feature level, the model can extract normal features from audio data at different granularities. Simultaneously, a random masking strategy is introduced for the generator, enabling it to deeply learn the intrinsic structure and salient features of the data. To avoid the problem of the reconstruction model reconstructing abnormal data, a feature constraint loss function is designed to obtain the discriminator's judgment at the spectral feature level, guiding the generator to reconstruct a more realistic spectral image, thereby improving the sound anomaly detection effect.

[0009] In addition, the unsupervised sound anomaly recognition method based on a masking strategy proposed in the above embodiments of the present invention may also have the following additional technical features:

[0010] Optionally, training the anomaly detection model based on the masked log-Mel spectrum and the preprocessed log-Mel spectrum includes: inputting the masked log-Mel spectrum into a generator to obtain a reconstructed log-Mel spectrum; obtaining a corresponding reconstruction loss and a corresponding masking loss based on the preprocessed log-Mel spectrum and the reconstructed log-Mel spectrum; inputting the reconstructed log-Mel spectrum and the preprocessed log-Mel spectrum into a random local discriminator to extract local feature vectors; inputting the reconstructed log-Mel spectrum and the preprocessed log-Mel spectrum into a global feature vector to extract global feature vectors; obtaining corresponding local feature constraint losses and global feature constraint losses based on the local feature vectors and the global feature vectors; and iteratively optimizing the anomaly detection model using a total loss function to obtain a trained anomaly detection model, wherein the total loss function includes the masking loss, reconstruction loss, local feature constraint loss, and global feature constraint loss.

[0011] Optionally, the local feature constraint loss or the global feature constraint loss can be calculated according to the following formula:

[0012]

[0013] Where, λ f , λ k and λ g Let z be a hyperparameter, μ and σ represent the mean and variance of the hidden layer feature vectors, respectively. i and Corresponding to X respectively i and G(X) i The latent eigenvectors of X) i Let G(X) represent the log-Mel spectrum of the i-th unmasked element. i ) represents the reconstructed output corresponding to the generator, z i It is obtained from the deep convolutional layer of the discriminator, and the reconstructed log-Mel spectrum is input into the encoder in the generator to obtain... a i and Corresponding to X respectively i and G(X) i The two-dimensional feature vector of L is extracted from the first deep convolutional layer of the global discriminator. MSE L represents the mean squared error loss. c This represents the covariance loss.

[0014] Optionally, the sample to be identified is input into the trained anomaly identification model, and the corresponding mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are calculated. The mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are added together to obtain the corresponding anomaly score.

[0015] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing an unsupervised audio anomaly recognition program based on a masking strategy. When executed by a processor, the unsupervised audio anomaly recognition program based on a masking strategy implements the unsupervised audio anomaly recognition method based on a masking strategy as described above.

[0016] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the unsupervised sound anomaly recognition method based on the masking strategy described above.

[0017] To achieve the above objectives, a fourth aspect of the present invention proposes an unsupervised audio anomaly recognition device based on a masking strategy, comprising: an acquisition module for acquiring normal audio data from a machine or device; a preprocessing module for preprocessing the normal audio data to obtain a preprocessed log-Mel spectrogram; a model building module for constructing an anomaly recognition model of a deep convolutional generative adversarial network, wherein the anomaly recognition model includes a generator, a global discriminator, and a random local discriminator; a masking module for masking the preprocessed log-Mel spectrogram to obtain a masked log-Mel spectrogram; a training module for training the anomaly recognition model based on the masked log-Mel spectrogram and the preprocessed log-Mel spectrogram to obtain a trained anomaly recognition model, wherein during training, masking loss, reconstruction loss, local feature constraint loss, and global feature constraint loss are used to optimize the anomaly recognition model; and an anomaly recognition module for inputting a sample to be recognized into the trained anomaly recognition model to obtain a corresponding anomaly score, and comparing the anomaly score with a preset threshold to determine whether the sample to be recognized is abnormal.

[0018] In addition, the unsupervised sound anomaly recognition device based on the masking strategy proposed in the above embodiments of the present invention may also have the following additional technical features:

[0019] Optionally, the training module is further configured to: input the masked log-Mel spectrum into a generator to obtain a reconstructed log-Mel spectrum; obtain a corresponding reconstruction loss based on the preprocessed log-Mel spectrum and the reconstructed log-Mel spectrum, and obtain a corresponding masking loss based on the masked log-Mel spectrum and the reconstructed log-Mel spectrum; input the reconstructed log-Mel spectrum and the preprocessed log-Mel spectrum into the random local discriminator to extract local feature vectors; input the reconstructed log-Mel spectrum and the preprocessed log-Mel spectrum into the global feature vector to extract global feature vectors; obtain corresponding local feature constraint losses and global feature constraint losses based on the local feature vectors and the global feature vector; and iteratively optimize the anomaly recognition model using a total loss function to obtain a trained anomaly recognition model, wherein the total loss function includes the masking loss, reconstruction loss, local feature constraint loss, and global feature constraint loss.

[0020] Optionally, the local feature constraint loss or the global feature constraint loss can be calculated according to the following formula:

[0021]

[0022] Where, λ f , λ k and λ g Let z be a hyperparameter, μ and σ represent the mean and variance of the hidden layer feature vectors, respectively. i and Corresponding to X respectively i and G(X) i The latent eigenvectors of X) i Let G(X) represent the log-Mel spectrum of the i-th unmasked element. i ) represents the reconstructed output corresponding to the generator, z i It is obtained from the deep convolutional layer of the discriminator, and the reconstructed log-Mel spectrum is input into the encoder in the generator to obtain... a i and Corresponding to X respectively i and G(X) i The two-dimensional feature vector of L is extracted from the first deep convolutional layer of the global discriminator. MSE L represents the mean squared error loss. c This represents the covariance loss.

[0023] Optionally, the sample to be identified is input into the trained anomaly identification model, and the corresponding mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are calculated. The mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are added together to obtain the corresponding anomaly score. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating an unsupervised sound anomaly recognition method based on a masking strategy according to an embodiment of the present invention.

[0025] Figure 2 This is a diagram illustrating the overall network framework of the ASD-MS according to an embodiment of the present invention.

[0026] Figure 3 This is a block diagram of an unsupervised sound anomaly recognition device based on a masking strategy according to an embodiment of the present invention. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0028] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0029] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0030] refer to Figure 1 As shown, the unsupervised audio anomaly recognition method based on a masking strategy according to an embodiment of the present invention includes the following steps:

[0031] S101, Obtain normal audio data from the machine / equipment.

[0032] As an example, the audio samples were obtained from the MIMIIDG industrial dataset and the ToyADMOS2 dataset. These datasets contain seven machine types, each with 990 source domain data points and 10 target domain normal data points. The source and target domains differ in operating speed, machine load, viscosity, heating temperature, ambient noise type, and signal-to-noise ratio. Each data point includes both machine sound and ambient noise, is 10 seconds long, and has a sampling rate of 16kHz.

[0033] S102, preprocess the normal audio data to obtain the preprocessed log-Mel spectrum.

[0034] S103, construct an anomaly detection model of a deep convolutional generative adversarial network, wherein the anomaly detection model includes a generator, a global discriminator, and a random local discriminator.

[0035] As an example, such as Figure 2 As shown, the SAD-MS (Sound Anomaly Detection Algorithm Based on Mask Strategy) anomaly detection model based on deep convolutional generative adversarial networks includes a generator, a global discriminator, and a stochastic local discriminator. The stochastic local discriminator network structure consists of five convolutional layers, five normalization layers, five activation function layers, and two deep convolutional layers. The stochastic local discriminator network extracts deeper spectral features through several convolutional and deep convolutional layers, ultimately generating 128 spectral features. The anomaly detection model is trained using the PyTorch framework, optimized with the Adam optimizer at a learning rate of 0.001, a batch size of 512, and 100 training epochs.

[0036] S104, perform masking on the preprocessed log-Mel spectrum to obtain the masked log-Mel spectrum.

[0037] It should be noted that, in order to alleviate the problem of reconstructing some abnormal data in the reconstruction model, a random masking strategy is introduced into the generator to improve the generator's reconstruction capability and avoid the generator focusing on non-target machine sounds or random noise, thereby improving the model's generalization ability and robustness.

[0038] As a specific example, the dimension of the input data is (186×128×128), representing that each sample is divided into 186 (128×128) two-dimensional log-Mel spectrum features. Let X i(t_f, mel_bin) represents the i-th logarithmic Mel spectrum, where (t_f, mel_bin) is the masked region, t_f is the number of time frames, and mel_bin is the length of the Mel frequency interval. When using a time-domain masking strategy, t_f is a variable, and mel_bin is 128. The processing methods for the frequency-domain masking strategy are similar to those for the time-domain masking strategy. In the SAD-MS algorithm, the mask length in the time domain is set to 10 time frames, while complete corresponding data is selected in the frequency domain, and the number of masked regions is set to 2.

[0039] S105, the anomaly recognition model is trained based on the masked log-Mel spectrogram and the preprocessed log-Mel spectrogram to obtain a trained anomaly recognition model. During the training process, the anomaly recognition model is optimized using mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss.

[0040] As an example, training an anomaly detection model based on the masked log-Melogram and the preprocessed log-Melogram includes: inputting the masked log-Melogram into a generator to obtain a reconstructed log-Melogram; obtaining the corresponding reconstruction loss and masking loss based on the preprocessed and reconstructed log-Melograms; inputting the reconstructed and preprocessed log-Melograms into a random local discriminator to extract local feature vectors; inputting the reconstructed and preprocessed log-Melograms into a global feature vector generator to extract global feature vectors; obtaining the corresponding local feature constraint loss and global feature constraint loss based on the local and global feature vectors; and iteratively optimizing the anomaly detection model using a total loss function to obtain a trained anomaly detection model, wherein the total loss function includes masking loss, reconstruction loss, local feature constraint loss, and global feature constraint loss.

[0041] In other words, the stochastic local discriminator receives a log-Mel spectrum as input. By learning local temporal features, the stochastic local discriminator enables the model to capture local spectral features in normal audio data with finer granularity. To reduce the impact of noise on the generator and promote the generator's deeper understanding of the input spectrum, a feature constraint loss is designed to guide the generator to generate more realistic spectra from the feature level. The error of anomaly detection is calculated through a set total loss function. After multiple iterations, the learning process is completed, and the network model is optimized.

[0042] As an example, the local feature constraint loss or the global feature constraint loss is calculated according to the following formula:

[0043]

[0044] Where, λ f , λ k and λ g The hyperparameters are all set to 1; μ and σ represent the mean and variance of the hidden layer feature vectors, respectively. i and Corresponding to X respectively i and G(X) i The latent eigenvectors of X) i Let G(X) represent the log-Mel spectrum of the i-th unmasked element. i ) represents the reconstructed output corresponding to the generator, z i It is obtained from the deep convolutional layer of the discriminator, and the reconstructed log-Mel spectrum is input into the encoder in the generator to obtain... a i and Corresponding to X respectively i and G(X) i The two-dimensional feature vector of L is extracted from the first deep convolutional layer of the global discriminator. MSE L represents the mean squared error loss. c This represents the covariance loss.

[0045] As an example, reconstruction loss is used to quantify the similarity between the reconstructed spectrum and the original spectrum. The loss function formula is shown below:

[0046]

[0047] Where N is the number of samples in the batch.

[0048] As an example, to enhance the generator's ability to reconstruct the input data and enable it to reconstruct the complete audio data from partial information, a masking loss is introduced. The formula for calculating the masking loss is as follows:

[0049]

[0050] Among them, X masked，i G(X) represents the log-Mel spectrum of the i-th masked element. masked，i ) represents the log-Mel spectrum of the generator reconstruction.

[0051] The generator's total loss is as follows:

[0052]

[0053] Where, λ g , λ m , λ u and λ zThe hyperparameters were set to 1, 1, 0.7, and 0.3 respectively, and L... con-g and L con-l The discriminator represents the feature constraint loss; its purpose is to impose constraints on the generator at the feature level, enabling the generator to deeply understand the intrinsic spectral characteristics of the data and thus reconstruct the data.

[0054] S106, Input the sample to be identified into the trained anomaly identification model to obtain the corresponding anomaly score, and compare the anomaly score with a preset threshold to determine whether the sample to be identified is abnormal.

[0055] It should be noted that by designing a dual discriminator network architecture, the model can capture local features in normal audio data with finer granularity. During testing, it can detect local temporal features that are significantly different from normal data, thereby improving the accuracy of anomaly detection. Secondly, a feature constraint loss is introduced into the generator to obtain the discriminator's judgment on the spectral feature level. During gradient backpropagation, the generator can update according to the discriminator's guidance. The reconstructed data not only has a small mean square error, but also has more similar features, thereby improving the generator's reconstruction capability.

[0056] As an example, the sample to be identified is input into the trained anomaly identification model, and the corresponding mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are calculated. The mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are added together to obtain the corresponding anomaly score.

[0057] As a specific implementation, anomalies are detected from both the generator and discriminator perspectives during testing. For the generator, reconstruction error is measured in both the sample space and the feature space. In the sample space, the true log-Melogram is compared with the reconstructed log-Melogram. The feature space contains key information for each sample; therefore, the feature vectors of the true samples are compared with the feature vectors of the reconstructed samples, with each measurement corresponding to an anomaly score. For the global and local discriminators, since the information extracted from the deep convolutional layers contains rich semantic features, the features obtained using Mahalanobis distance are used. Finally, the scores obtained from the two discriminators are summed. The specific formula for the anomaly score is as follows:

[0058]

[0059] Among them, S i Let λ represent the anomaly score of the i-th sample. The first two terms of the formula represent the anomaly scores obtained from the generator, and the last two terms represent the anomaly scores calculated from the discriminator. a , λ b , λ c , λ d Represents the hyperparameter, Dm Represents Mahalanobis distance, u g and u l These represent the feature vector and local feature vector of the sample to be tested, respectively. and These represent the extraction of the average vector and local feature vector from the global discriminator and local discriminator, respectively. The performance of different scores varies significantly across different machine types; therefore, different outlier scores will not be aggregated into a single composite score.

[0060] For two d-dimensional vectors d i and d j The formula for calculating Mahalanobis distance is as follows:

[0061]

[0062] Where, ∑ -1 The matrix represents the inverse of the covariance matrix, and T represents the transpose operation.

[0063] In summary, the unsupervised sound anomaly recognition method based on a masking strategy proposed in this application first preprocesses the original data into a log-Mel spectrogram. Then, a random masking strategy is applied to the log-Mel spectrogram in the time domain. The masked data is then fed into a generator to obtain the reconstructed log-Mel spectrogram. The reconstructed data and the original data are then fed into a global discriminator and a random local discriminator, respectively. The feature vectors of the reconstructed samples and the real samples can be obtained from the discriminator. Subsequently, the mean and standard deviation between the feature vectors are compared, thereby enabling the generator to gain a deeper understanding of the intrinsic structure of the data. To ensure the reconstructed log-Mel spectrogram closely approximates the true log-Mel spectrogram, the generator minimizes not only masking and reconstruction errors during training but also feature matching loss. During testing, the test data is input into the SAD-MS network to extract features, detecting anomalies from both the generator and discriminator perspectives. For the generator, reconstruction errors are measured in both the sample space and feature space. In the sample space, the true log-Mel spectrogram is compared with the reconstructed log-Mel spectrogram. The feature space contains key information for each sample; therefore, the feature vectors of the true samples are compared with the feature vectors of the reconstructed samples. Each measurement corresponds to an anomaly score. For the global discriminator and the random local discriminator, since the information extracted from the deep convolutional layers contains rich semantic features, Mahalanobis distance is used to obtain the features. Finally, the scores from the two discriminators are summed to obtain the anomaly score, thus identifying anomalies.

[0064] In addition, the present invention also proposes a computer-readable storage medium storing an unsupervised audio anomaly recognition program based on a masking strategy, which, when executed by a processor, implements the unsupervised audio anomaly recognition method based on a masking strategy as described above.

[0065] In addition, this invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the unsupervised sound anomaly recognition method based on the masking strategy described above.

[0066] Figure 3 This is a block diagram of an unsupervised sound anomaly recognition device based on a masking strategy according to an embodiment of the present invention. Figure 3 As shown, the unsupervised sound anomaly recognition device includes: an acquisition module 10, a preprocessing module 20, a model building module 30, a mask processing module 40, a training module 50, and an anomaly recognition module 60;

[0067] The system comprises the following modules: Acquisition module 10 acquires normal audio data from the equipment; Preprocessing module 20 preprocesses the normal audio data to obtain a preprocessed log-Mel spectrogram; Model building module 30 constructs an anomaly detection model using a deep convolutional generative adversarial network, which includes a generator, a global discriminator, and a random local discriminator; Masking module 40 masks the preprocessed log-Mel spectrogram to obtain a masked log-Mel spectrogram; Training module 50 trains the anomaly detection model based on the masked and preprocessed log-Mel spectrograms to obtain a trained anomaly detection model, wherein masking loss, reconstruction loss, local feature constraint loss, and global feature constraint loss are used to optimize the anomaly detection model during training; and Anomaly detection module 60 inputs the sample to be identified into the trained anomaly detection model to obtain a corresponding anomaly score, and compares the anomaly score with a preset threshold to determine whether the sample to be identified is abnormal.

[0068] As an example, the training module 50 is further configured to: input the masked log-Mel spectrum into the generator to obtain the reconstructed log-Mel spectrum; obtain the corresponding reconstruction loss based on the preprocessed log-Mel spectrum and the reconstructed log-Mel spectrum, and obtain the corresponding masking loss based on the masked log-Mel spectrum and the reconstructed log-Mel spectrum; input the reconstructed log-Mel spectrum and the preprocessed log-Mel spectrum into a random local discriminator to extract local feature vectors; input the reconstructed log-Mel spectrum and the preprocessed log-Mel spectrum into a global feature vector to extract global feature vectors; obtain the corresponding local feature constraint loss and global feature constraint loss based on the local feature vectors and the global feature vectors; and iteratively optimize the anomaly recognition model using a total loss function to obtain a trained anomaly recognition model, wherein the total loss function includes masking loss, reconstruction loss, local feature constraint loss, and global feature constraint loss.

[0069] As an example, the local feature constraint loss or the global feature constraint loss is calculated according to the following formula:

[0070]

[0071] Where, λ f , λ k and λ g Let z be a hyperparameter, μ and σ represent the mean and variance of the hidden layer feature vectors, respectively. i and Corresponding to X respectively i and G(X) i The latent eigenvectors of X) i Let G(X) represent the log-Mel spectrum of the i-th unmasked element. i ) represents the reconstructed output corresponding to the generator, z i It is obtained from the deep convolutional layer of the discriminator, and the reconstructed log-Mel spectrum is input into the encoder in the generator to obtain... a i and Corresponding to X respectively i and G(X) i The two-dimensional feature vector of L is extracted from the first deep convolutional layer of the global discriminator. MSE L represents the mean squared error loss. c This represents the covariance loss.

[0072] As an example, the sample to be identified is input into the trained anomaly identification model, and the corresponding mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are calculated. The mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are added together to obtain the corresponding anomaly score.

[0073] It should be noted that the foregoing explanation of the embodiments of the unsupervised sound anomaly recognition method based on masking strategy also applies to the unsupervised sound anomaly recognition device based on masking strategy in this embodiment, and will not be repeated here.

[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0079] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0080] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0081] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0082] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0083] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0084] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0085] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An unsupervised sound anomaly recognition method based on a masking strategy, characterized in that, Includes the following steps: Acquire normal audio data from the machine equipment; The normal audio data is preprocessed to obtain a preprocessed log-Mel spectrogram; An anomaly detection model based on a deep convolutional generative adversarial network is constructed, wherein the anomaly detection model includes a generator, a global discriminator, and a random local discriminator; The preprocessed log-Mel spectrogram is masked to obtain a masked log-Mel spectrogram. The anomaly detection model is trained based on the masked log-Mel spectrogram and the preprocessed log-Mel spectrogram to obtain a trained anomaly detection model. During the training process, the anomaly detection model is optimized using mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss. The sample to be identified is input into the trained anomaly identification model to obtain the corresponding anomaly score. The anomaly score is compared with a preset threshold to determine whether the sample to be identified is abnormal.

2. The unsupervised audio anomaly recognition method based on a masking strategy as described in claim 1, characterized in that, The anomaly detection model is trained based on the masked log-Melogram and the preprocessed log-Melogram, including: The masked log-Mel spectrum is input into the generator to obtain the reconstructed log-Mel spectrum. The corresponding reconstruction loss is obtained based on the preprocessed log-Mel spectrogram and the reconstructed log-Mel spectrogram, and the corresponding masking loss is obtained based on the masked log-Mel spectrogram and the reconstructed log-Mel spectrogram; The reconstructed log-Mel spectrogram and the preprocessed log-Mel spectrogram are input into the random local discriminator to extract local feature vectors, and the reconstructed log-Mel spectrogram and the preprocessed log-Mel spectrogram are input into the global feature vector to extract global feature vectors. Based on the local feature vector and the global feature vector, the corresponding local feature constraint loss and global feature constraint loss are obtained; The anomaly detection model is iteratively optimized using a total loss function to obtain a trained anomaly detection model. The total loss function includes the mask loss, reconstruction loss, local feature constraint loss, and global feature constraint loss.

3. The unsupervised audio anomaly recognition method based on a masking strategy as described in claim 2, characterized in that, Calculate the local feature constraint loss or the global feature constraint loss using the following formula: Where, λ f , λ k and λ g Let z be a hyperparameter, μ and σ represent the mean and variance of the hidden layer feature vectors, respectively. i and Corresponding to X respectively i and G(X) i The latent eigenvectors of X) i Let G(X) represent the log-Mel spectrum of the i-th unmasked element. i ) represents the reconstructed output corresponding to the generator, z i It is obtained from the deep convolutional layer of the discriminator, and the reconstructed log-Mel spectrum is input into the encoder in the generator to obtain... a i and Corresponding to X respectively i and G(X) i The two-dimensional feature vector of L is extracted from the first deep convolutional layer of the global discriminator. MSE L represents the mean squared error loss. c This represents the covariance loss.

4. The unsupervised audio anomaly recognition method based on a masking strategy as described in claim 3, characterized in that, The sample to be identified is input into the trained anomaly identification model, and the corresponding mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are calculated. The mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are added together to obtain the corresponding anomaly score.

5. A computer-readable storage medium, characterized in that, It stores an unsupervised audio anomaly recognition program based on a masking strategy. When the unsupervised audio anomaly recognition program based on a masking strategy is executed by the processor, it implements the unsupervised audio anomaly recognition method based on a masking strategy as described in any one of claims 1-4.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the unsupervised sound anomaly recognition method based on a masking strategy as described in any one of claims 1-4.

7. An unsupervised sound anomaly recognition device based on a masking strategy, characterized in that, include: The acquisition module is used to acquire normal audio data from the machine equipment. The preprocessing module is used to preprocess the normal audio data to obtain a preprocessed log-Mel spectrogram. The model building module is used to build an anomaly detection model of a deep convolutional generative adversarial network, wherein the anomaly detection model includes a generator, a global discriminator, and a random local discriminator; The masking module is used to mask the preprocessed log-Mel spectrum to obtain the masked log-Mel spectrum. The training module is used to train the anomaly recognition model based on the masked log-Mel spectrogram and the preprocessed log-Mel spectrogram to obtain a trained anomaly recognition model. During the training process, the anomaly recognition model is optimized using mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss. An anomaly detection module is used to input the sample to be identified into the trained anomaly detection model to obtain the corresponding anomaly score, and compare the anomaly score with a preset threshold to determine whether the sample to be identified is abnormal.

8. The unsupervised sound anomaly recognition device based on a masking strategy as described in claim 7, characterized in that, The training module is further configured to: input the masked log-Mel spectrum into the generator to obtain the reconstructed log-Mel spectrum; obtain the corresponding reconstruction loss based on the preprocessed log-Mel spectrum and the reconstructed log-Mel spectrum, and obtain the corresponding masking loss based on the masked log-Mel spectrum and the reconstructed log-Mel spectrum; input the reconstructed log-Mel spectrum and the preprocessed log-Mel spectrum into the random local discriminator to extract local feature vectors; input the reconstructed log-Mel spectrum and the preprocessed log-Mel spectrum into the global feature vector to extract global feature vectors; and obtain the corresponding local feature constraint loss and global feature constraint loss based on the local feature vector and the global feature vector. The anomaly detection model is iteratively optimized using a total loss function to obtain a trained anomaly detection model. The total loss function includes the mask loss, reconstruction loss, local feature constraint loss, and global feature constraint loss.

9. The unsupervised sound anomaly recognition device based on a masking strategy as described in claim 8, characterized in that, Calculate the local feature constraint loss or the global feature constraint loss using the following formula: Where, λ f , λ k and λ g Let z be a hyperparameter, μ and σ represent the mean and variance of the hidden layer feature vectors, respectively. i and Corresponding to X respectively i and G(X) i The latent eigenvectors of X) i Let G(X) represent the log-Mel spectrum of the i-th unmasked element. i ) represents the reconstructed output corresponding to the generator, z i It is obtained from the deep convolutional layer of the discriminator, and the reconstructed log-Mel spectrum is input into the encoder in the generator to obtain... a i and Corresponding to X respectively i and G(X) i The two-dimensional feature vector of L is extracted from the first deep convolutional layer of the global discriminator. MSE L represents the mean squared error loss. c This represents the covariance loss.

10. The unsupervised sound anomaly recognition device based on a masking strategy as described in claim 9, characterized in that, The sample to be identified is input into the trained anomaly identification model, and the corresponding mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are calculated. The mask loss, reconstruction loss, local feature constraint loss and global feature constraint loss are added together to obtain the corresponding anomaly score.

Images