Equipment anomaly detection method based on audio analysis

Abstract

The invention discloses an equipment anomaly detection method based on audio analysis, and the method comprises the steps: collecting a sound signal from equipment, and carrying out the preprocessing of the sound signal, so as to generate a logarithmic Mel spectrogram; processing the logarithmic Mel spectrogram through a time feature enhancement network TANet and a time domain hybrid enhancement module TMixup so as to enhance the time sequence features of the logarithmic Mel spectrogram and generate feature representation; a multi-scale time sequence feature extraction module MSTNet is utilized to extract multi-scale time sequence features covering short-term, medium-term and long-term dependency relationships from the original audio signals; the method comprises the following steps: performing diffusion and antagonism training in a potential space through a potential diffusion model LDM and a generative adversarial network GAN in combination with enhanced logarithmic Mel spectrogram features and multi-scale time sequence features extracted by MSTNet, generating high-quality features, analyzing the features through a discriminator of the generative adversarial network, and judging whether equipment is in an abnormal state or not. According to the method, efficient anomaly detection performance can be kept in a complex and changeable industrial environment, and a reliable and economical anomaly detection solution is provided for industrial equipment.

Description

Technical Field

[0001] The present invention relates to a technology for evaluating the health status of equipment, and in particular to an equipment anomaly detection method based on audio analysis. Background Art

[0002] With the rapid development of Industry 4.0 and smart manufacturing, intelligent, automated and refined operation and maintenance of equipment have become the key to modern industrial production. However, equipment will face problems such as wear, aging and failure in long-term operation, which will not only affect production efficiency but also bring serious safety hazards. Therefore, how to detect abnormal status of equipment in a timely and accurate manner and ensure the reliable operation of equipment has become the focus of the industry.

[0003] Traditional equipment anomaly detection methods mainly rely on physical sensors, such as vibration sensors, temperature sensors, and pressure sensors, which collect physical parameters of the equipment to determine whether the equipment is in an abnormal state. However, these methods have the problems of complex installation and high maintenance costs, and have a high dependence on the structure and environment of the equipment. In recent years, the anomaly detection method based on audio analysis has gradually become a hot technology for industrial equipment anomaly detection due to its advantages of non-contact, low cost and easy deployment. Audio analysis is a technology that evaluates the health status of equipment by collecting and analyzing the sound signals generated when the equipment is running. Audio analysis can provide a non-contact, low-cost fault detection method, and realize real-time monitoring of equipment anomalies through signal processing, feature extraction and feature modeling. Detect abnormal sounds generated by machines or equipment during operation, and analyze the normal sound characteristics of the equipment. When the sound characteristics change abnormally, the system can identify and mark them as abnormal events.

[0004] The goal of anomaly detection based on audio analysis is to identify abnormal situations by learning the sound characteristics of the equipment during normal operation. At present, the research mainly focuses on anomaly detection methods based on self-supervised classification models and generative reconstruction models. For example, the machine abnormal sound detection method and device of improved AE with patent application number 202310933972.5, the metadata abnormal sound detection method with patent application number 202310768780.3, the unsupervised machine abnormal sound detection method with lightweight network with patent application number 202311316592.3, the self-supervised machine abnormal sound detection method with domain transfer with patent application number 202210863510.6, etc.

[0005] However, the most popular abnormal sound detection technology now, such as the improved AE method with patent application number 202310933972.5, detects abnormal situations by learning the distribution of normal data. These models try to reconstruct or generate samples similar to normal data during training. When the model cannot reconstruct or generate similar samples, it is marked as abnormal. However, this type of method is prone to overfitting, that is, the model performs well on the training data, but when faced with new data or unseen abnormal situations, the model may not be able to effectively detect abnormalities. There are also metadata self-supervised classification models such as patent application number 202310768780.3 that rely on the classification information provided by metadata. However, different types of machines may produce very similar sound features during operation, which makes it difficult for the model to distinguish between different categories of sounds. Even between machines of the same category, due to individual differences and changes in operating conditions, the sound features may show large differences, which further increases the difficulty of the model to extract robust features. Moreover, when the metadata is incomplete, the classification labels are unclear or the labels are wrong, it is difficult for the model to extract effective feature representations from them, resulting in a decrease in classification performance.

[0006] In general, existing abnormal sound detection methods have significant deficiencies in robustness, generalization ability, and detection effect, especially in complex and changeable industrial audio scenarios, where it is difficult to stably detect abnormal states of equipment. Traditional methods have limited feature extraction capabilities for raw audio and logarithmic Mel spectrograms, and it is difficult to fully capture the temporal dynamics and abnormal features in sound signals, resulting in limited detection performance.

[0007] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0008] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and provide a device abnormality detection method based on audio analysis.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect of the present invention, a method for detecting device anomalies based on audio analysis comprises the following steps:

[0011] S1, collecting sound signals from the device, and preprocessing the collected signals to generate a logarithmic Mel spectrogram;

[0012] S2, through the temporal feature enhancement network TANet and the time domain hybrid enhancement module TMixup, multiple feature enhancements are performed on the logarithmic Mel spectrogram generated in step S1 to generate an enhanced temporal feature representation;

[0013] S3, extracting multi-scale temporal features from the original audio signal through the multi-scale temporal feature extraction module MSTNet, including short-term, medium-term and long-term temporal dependencies, thereby capturing the temporal dynamics of the audio signal;

[0014] S4. Using the logarithmic Mel-spectrogram features after multiple feature enhancement obtained in step S2 and the multi-scale time series features extracted in step S3, diffusion and adversarial training are performed in the latent space through the latent diffusion model LDM and the generative adversarial network GAN to generate features that can distinguish between normal and abnormal sound patterns, and the normal operation and abnormal state of the device are distinguished through the discriminator of the generative adversarial network, so as to identify whether the device is in an abnormal state.

[0015] In a second aspect of the present invention, a computer program product includes a computer program, and when the computer program is executed by a processor, the device anomaly detection method based on audio analysis is implemented.

[0016] The present invention has the following beneficial effects:

[0017] This paper proposes a device anomaly detection method based on audio analysis. By introducing a time series feature enhancement network, a multi-scale time series feature extraction module and a potential diffusion adversarial generation model, it deeply mines the multi-scale time series information of audio signals, enhances feature extraction capabilities, and builds an efficient and accurate device anomaly detection system. This method not only improves the robustness and generalization capabilities of detection, but also maintains efficient and accurate detection performance in the face of different equipment types, complex working conditions and no anomalies, providing a more reliable and economical anomaly detection solution for industrial equipment.

[0018] Specifically, the present invention introduces the latent diffusion model (LDM) as a generator into the GAN framework, and performs a diffusion process in a low-dimensional latent space to reduce the computational complexity and training difficulty, while improving the authenticity and diversity of the generated features. Compared with the traditional generative model, the latent diffusion generative adversarial model (TLDiffGAN) based on temporal feature enhancement of the present invention overcomes the problems of mode collapse and training instability. The generated features gradually approach the true distribution during the diffusion process, and through the adversarial training of the discriminator, the generator is able to generate more realistic and more discriminative abnormal features, thereby improving the anomaly detection performance of the model. The present invention also introduces a multi-scale feature extraction network module (MSTNet), which effectively makes up for the shortcomings of the spectrogram in expressing temporal information, and provides a more comprehensive input for the model by fusing with the log-mel spectrogram features. This multi-scale feature extraction strategy significantly enhances the model's ability to recognize complex audio patterns, especially when facing abnormal events, showing stronger sensitivity and accuracy.

[0019] The present invention also introduces a data enhancement network (TANet), which can effectively extract and enhance key time series areas in the spectrogram, especially in identifying abnormal changes. Combined with the TMixup operation, the time series features are further enhanced, so that the model can still maintain a high detection accuracy when facing variable audio signals. Compared with the existing self-supervision and generative methods, the present invention shows excellent adaptability when facing audio signals of various types of machines. Especially when dealing with unseen abnormal situations, it shows higher detection accuracy and robustness. The equipment anomaly detection system of the present invention can be widely used in the fields of manufacturing, energy, transportation, and robots, and is suitable for various types of equipment. By analyzing the audio signals of the equipment in real time during operation, the system can quickly identify abnormal events without affecting the normal operation of the equipment, greatly improving the safety and reliability of industrial production. At the same time, due to the audio analysis and feature extraction method of the present invention, the system has high generalization ability and robustness, and can maintain stable detection performance even in complex environments.

[0020] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 FIG. 4 is an algorithm architecture diagram of an abnormal sound detection network according to an embodiment of the present invention.

[0022] Figure 2 Detailed structural diagram of the MSTNet module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0024] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0025] See also Figure 1 The embodiment of the present invention provides a device abnormality detection method based on audio analysis, comprising the following steps:

[0026] Step S1, sound signal collection and preprocessing: collect sound signals from the device, and preprocess the collected signals to generate a logarithmic Mel-spectrogram.

[0027] In some embodiments, a logarithmic Mel spectrogram is generated by denoising, normalizing and short-time Fourier transform (STFT) the collected signal. Specifically, the audio signal is short-time Fourier transform (STFT), mapped to the Mel scale, and finally logarithm is taken to obtain a logarithmic Mel spectrogram. The feature representation of the logarithmic Mel spectrogram can better reflect the sensitivity of the human ear to different frequencies.

[0028] Step S2, multiple feature enhancement: Through the temporal feature enhancement network TANet and the time domain hybrid enhancement module TMixup, the logarithmic Mel spectrogram generated in step S1 is subjected to multiple feature enhancement to generate an enhanced temporal feature representation.

[0029] In some embodiments, the processing of the log-Mel spectrogram by the temporal feature enhancement network TANet specifically includes: performing a maximum pooling operation to extract the maximum value of a local area of ​​the log-Mel spectrogram and enhance the peak characteristics of the sound signal; performing an average pooling operation to calculate the average value of a local area of ​​the log-Mel spectrogram, smooth the feature map and reduce the impact of noise; performing a power average pooling operation to emphasize the specific time domain information of the log-Mel spectrogram through power operations and capture high-frequency changes; and performing weighted fusion on the results of the above pooling operations, wherein the weight parameters are normalized so that the contribution of different pooling operations to the final features is in a predetermined proportion.

[0030] In some embodiments, the processing performed by the time-domain hybrid enhancement module TMixup on the logarithmic Mel-spectrogram specifically includes: applying data enhancement operations, fusing the time series features obtained by TANet processing with the original spectrogram through mixing coefficients to generate a mixed feature representation; using an activation function to activate the results of the TANet pooling operation to limit the output range and control the feature strength; combining the activated time series features with the original spectrogram features through element-level point-by-point multiplication, and adjusting the key areas in the features point by point.

[0031] By weighted mixing the temporal features of different samples, data with richer temporal features is generated, which not only enhances the robustness of the model but also improves its generalization ability.

[0032] Step S3, multi-scale temporal feature extraction: Multi-scale temporal features are extracted from the original audio signal through the multi-scale temporal feature extraction module MSTNet, including short-term, medium-term and long-term temporal dependencies, so as to capture the temporal dynamics of the audio signal.

[0033] In some embodiments, the processing of the original audio signal by the multi-scale temporal feature extraction module MSTNet specifically includes: dividing the audio signal into multiple segments to maintain the continuity of features and cover the complete audio data; using large convolution kernels with different expansion rates to expand the convolution layer LKDConv to extract features at different time scales; further processing the features extracted by the LKDConv layer through a multi-layer Dilated Conv Block to integrate local and global information; splicing the multi-scale features output by the LKDConv layer and the dilated convolution block in the channel dimension to form a comprehensive multi-scale feature representation.

[0034] Among them, a series of LKDConv layers are applied to each segment of the audio signal, and each LKDConv layer has a specific dilation rate to capture short-term, medium-term and long-term temporal dependency features respectively; after the LKDConv layer, a multi-layer dilated convolution block Dilated Conv Block is used to further refine the features, and the receptive field is expanded by gradually increasing the dilation rate to extract more refined temporal features.

[0035] Step S4, potential diffusion and adversarial generation based on time series feature enhancement: using the log-mel spectrogram features after multiple feature enhancement obtained in step S2 and the multi-scale time series features extracted in step S3, diffusion and adversarial training are performed in the latent space through the latent diffusion model LDM and the generative adversarial network GAN to generate high-quality features that can distinguish between normal and abnormal sound patterns, and the discriminator of the generative adversarial network is used to distinguish the normal operation and abnormal state of the device, thereby accurately identifying whether the device is in an abnormal state.

[0036] The latent diffusion model (LDM) generates data by performing a diffusion process in a low-dimensional latent space. Unlike traditional diffusion models, LDM operates in latent space, which can significantly improve the training and generation efficiency of the model. Generative adversarial networks (GANs) are composed of two parts: a generator and a discriminator. The generator attempts to generate realistic data, while the discriminator tries to distinguish The generated data and the real data are divided into two parts. Through the adversarial training process, the generator can gradually generate data that is very close to the real data.

[0037] In some embodiments, the multi-scale features obtained in step S3 are concatenated with the embedding in the discriminator of the generative adversarial network and input into the final embedding predictor to provide comprehensive feature information for the final anomaly detection.

[0038] In some embodiments, the processing of potential diffusion and adversarial generation in step S4 specifically includes: using the potential diffusion model and the generative adversarial network to jointly optimize the enhanced spectral features to improve the authenticity and discrimination ability of the features; through an iterative process, gradually training the generative model to reconstruct high-quality spectral features so that they are close to the real spectral features in statistical characteristics; designing and using a loss function, which is calculated based on the statistical difference in mean and standard deviation between the generated features and the real features; wherein the loss function is used to compare the mean and standard deviation of the generated features with the real features, and the loss value is adjusted by a weighting parameter to optimize the distribution of the generated features.

[0039] In some embodiments, the potential diffusion and adversarial generation in step S4 includes the following processing methods: performing preliminary feature extraction on the input logarithmic Mel-spectrogram features through an encoder composed of multiple convolutional layers, including using convolutional layers and residual blocks to enhance nonlinear feature representation; downsampling the extracted features to reduce computational complexity and improve processing efficiency; mapping the extracted features to a low-dimensional latent space and performing a diffusion process in the space, wherein the noise intensity is controlled by set parameters; training in the latent space using a noise prediction model to optimize the feature representation by minimizing the noise prediction error; and using a loss function to learn to separate noise and latent features, gradually making the generated feature distribution approach the true feature distribution.

[0040] The device anomaly detection method based on audio analysis proposed by the present invention can build an efficient and accurate device anomaly detection system. By introducing the temporal feature enhancement network (TANet), the multi-scale temporal feature extraction module (MSTNet) and the latent diffusion adversarial generative model (LDiffGAN), the present invention can deeply mine the multi-scale temporal information in the audio signal and significantly improve the feature extraction capability. The combination of TANet and TMixup enhances the features of the key temporal regions in the logarithmic Mel spectrogram, making the model more sensitive and accurate in identifying abnormal sound changes. The MSTNet module effectively captures the short-term, medium-term and long-term dependencies of the audio signal through multi-scale design, making up for the shortcomings of the traditional logarithmic Mel spectrogram in expressing temporal information. At the same time, the combination of the latent diffusion model (LDM) and the generative adversarial network (GAN) optimizes the diffusion process in the latent space, reduces the computational complexity and training difficulty, and enhances the authenticity and diversity of the generated features. The latent diffusion generative adversarial model (TLDiffGAN) based on temporal feature enhancement effectively overcomes the problems of mode collapse and training instability in the traditional model, and improves the detection performance of the model for abnormal situations.

[0041] The device anomaly detection system of the present invention not only improves the robustness and generalization ability of detection, but also can adapt to a variety of equipment types and complex working conditions, and can maintain efficient and accurate detection performance even when facing unprecedented abnormal situations. The system can be widely used in the fields of manufacturing, energy, transportation, and robotics, providing a reliable and economical anomaly detection solution for industrial equipment, greatly improving the safety and reliability of industrial production. By analyzing the audio signals of the equipment in real time during operation, the system of the present invention can quickly identify abnormal events without affecting the normal operation of the equipment, and can maintain stable detection performance even in complex environments.

[0042] The following further describes specific embodiments of the present invention and examples of algorithm implementation thereof.

[0043] Sound signal acquisition and preprocessing

[0044] First, the sound signal generated during the operation of the device is collected through a high-sensitivity microphone, and the collected audio is preprocessed. This includes steps such as denoising, normalization, and short-time Fourier transform (STFT) to generate a log-mel spectrogram as the input of the feature extraction module. The log-mel spectrogram can more intuitively reflect the frequency distribution of the audio signal and effectively enhance the frequency details that are difficult for the human ear to perceive.

[0045] Abnormal sound detection network architecture

[0046] The framework based on potential diffusion model and generative adversarial network with good generalization and sampling performance is adopted. Figure 1 shown.

[0047] The framework mainly includes a combination of a multi-scale convolutional neural network (MSTNet), a temporal feature enhancement network (TANet), a time domain hybrid enhancement module (TMixup) and a diffusion generation module.

[0048] The overall architecture starts with feature extraction of audio signals, undergoes multiple feature enhancements and diffusion generation of latent space, and finally completes anomaly detection by embedding and reconstructing the detector.

[0049] The original audio is used as input, and one branch is logarithmically transformed after passing through 128 mel filters to obtain a log-mel spectrogram. In order to enhance the temporal information of key areas in the spectrogram, the temporal feature enhancement network (TANet) module is used to extract efficient temporal features.

[0050] In order to enhance the temporal information of key areas in the spectrogram, the TANet module adopts a weighted fusion strategy of multiple pooling operations. This strategy aims to enhance the model's sensitivity to time domain information, enabling it to better capture the dynamic characteristics of audio signals. The TANet module uses three pooling operations and performs weighted summation:

[0051]

[0052] MaxPool is a maximum pooling operation, which is used to extract the maximum value of the local area in the input feature map. Enhancing the information of highly activated areas can highlight the peak characteristics of sound signals; AvgPool is an average pooling operation, which smoothes the feature map by calculating the average value of the local area and provides stable time domain information. AvgPool helps reduce the impact of noise; PowerAvgPool is a power average pooling, which is an adjustable pooling method that emphasizes specific time domain information by performing power operations on features. This type of pooling can better capture high-frequency changes. They are the weight parameters corresponding to the pooling operation, which are normalized to ensure that their sum is equal to 1. By adjusting these weights, the contribution of different pooling operations to the final features can be flexibly controlled.

[0053] Through the above operations, the time series information in the spectrogram is enhanced, so that the model can better capture the time domain features. Then, through the TMixup data enhancement operation, the time series features are fused with the original spectrogram to obtain the final input spectrogram features:

[0054]

[0055] Where λ is the mixing coefficient of Beta distribution; the time series features obtained by the pooling operation are passed through the sigmoid activation function σ to limit the output between 0 and 1, further controlling the strength of the time series features; It is an element-level point-by-point multiplication (Hadamard multiplication), which multiplies the activated time series features by the original spectrum features element-by-element, emphasizing or suppressing the key areas in the features point by point.

[0056] Afterwards, the generated spectrogram features are input into the LDiffGAN module, and the authenticity and discrimination ability of the generated features are gradually improved through the joint optimization of the Latent Diffusion process and the generative adversarial network. After multiple iterations, the LDiffGAN module is able to reconstruct high-quality spectrograms. D2GLoss is used as the loss function of the LDiffGAN module. D2GLoss optimizes the distribution of generated features by calculating the difference in statistical characteristics (mean and standard deviation) between the generated features and the real features, making them as close to the real distribution as possible. The loss function calculates the L2 norm of the mean and standard deviation of the generated features and the real features, and performs weighted summation of the loss values ​​in combination with the weight parameters, and finally optimizes the generation process. The specific loss function calculation formula is as follows:

[0057] L=μ eff MSE(μ fake ,μ real )+σ eff MSE(σ fake ,σ real )

[0058] μ eff ,σ eff is the weighting parameter of the mean and standard deviation in the loss function, which is adjusted by the settings in the configuration file; μ fake ,μ real are the means of generated features and real features respectively; σ fake ,σ real are the standard deviations of generated features and true features respectively; MSE represents the mean square error (L2 norm).

[0059] Multi-scale temporal feature network module

[0060] Since log-mel spectrogram has certain limitations in capturing the temporal information of raw audio, the inventors got inspiration from TgramNet and designed a method for extracting multi-scale temporal information from raw audio signals. MSTNet module of temporal features, The MSTNet module extracts short-term and long-term dependent temporal features to make up for the lack of spectrogram information. The specific structure of the MSTNet module is as follows Figure 2 shown.

[0061] First, the input is the original audio waveform signal with a sampling rate of 16kHz. The audio signal is divided into multiple segments, each segment contains 128 frames, and the frame shift is 1 frame. This can maintain the continuity of features on the time axis and cover the complete audio data. Multi-scale feature extraction module

[0062] MSTNet consists of three large-kernel dilated convolution layers (LKDConv, Large-Kernel DilatedConvolution) with different dilation rates to extract features at different time scales. The specific settings of each convolution layer are as follows:

[0063]

[0064] LKDConv1:

[0065] C in Number of input channels, C out The number of output channels, K is the convolution kernel size, D is the dilation rate, P is the padding size (Padding), and n is the number of layers.

[0066] LKDConv1 is mainly used to capture short-term dependencies in audio signals and to model local features in detail through a small expansion rate. LKDConv2 uses a larger expansion rate to capture medium-term temporal dependencies and expand the convolution receptive field, thereby obtaining a wider range of time domain features. LKDConv3 further improves the expansion rate to capture global temporal features with long-term dependencies. This layer captures the long-range dependencies and rhythm information of the audio as a whole through large convolution kernels and high expansion rates.

[0067] In order to further enhance the representation ability of temporal features, MSTNet also includes a multi-layer dilated convolution block (DilatedConv Block).

[0068] These extended convolution blocks are used to further process the features extracted by the LKDConv layer, extracting more refined temporal features by continuously expanding the receptive field, integrating local and global information, and providing richer temporal features for the subsequent generative adversarial module. Finally, the multi-scale features output from each convolutional layer are spliced ​​in the channel dimension to form the final multi-scale features. The purpose of this is to integrate short-term, medium-term and long-term temporal features to provide more comprehensive information for the discriminant and generative modules. The final output feature embedding is spliced ​​with the embedding in the LDiffGAN module discriminator and input into the final embedding predictor.

[0069] Latent Diffusion and Generative Adversarial TLDiffGAN Module Based on Temporal Feature Enhancement

[0070] The TLDiffGAN module combines Latent Diffusion Model (LDM) and Generative Adversarial Network (GAN) and is designed for abnormal sound detection tasks. The input of this module is preprocessed log-mel spectrogram features, which are extracted through multiple specific network layers and diffused and adversarially trained in the latent space.

[0071] First, the input spectrogram is passed through an Encoder module consisting of multiple convolutional layers for preliminary feature extraction:

[0072] Conv2D layer: The input channel is 1 (single-channel spectrogram), the output channel is 256, the convolution kernel size is 4×4, the stride is 1, there is no padding, and the ReLU activation function is used.

[0073] Residual Block: During the feature extraction process, three residual blocks are added, each of which contains two convolutional layers and a skip connection to improve the nonlinear representation capability of feature extraction.

[0074] Downsampling layer: In order to reduce the computational complexity, the features after the residual block are downsampled twice, and the final output feature resolution is 1 / 16 of the original input.

[0075] The feature z0 extracted by the Encoder is mapped to the low-dimensional latent space z t In, z t The distribution at time t is defined as follows:

[0076]

[0077] Among them, α t Control the noise intensity during the diffusion process. LDM trains the noise prediction model in the latent space to minimize the noise prediction error at each time step t, and the objective function is:

[0078]

[0079] Through this loss function, the model learns how to separate noise from latent features, thereby gradually approaching the true feature distribution.

[0080] In summary, the present invention combines audio analysis technology with a deep learning model to propose an efficient and accurate method for detecting equipment anomalies. By adopting a temporal feature enhancement network (TANet), a multi-scale temporal feature extraction module (MSTNet), and a combination of a latent diffusion model (LDM) and a generative adversarial network (GAN), the present invention can deeply mine and analyze the audio signals generated during equipment operation, thereby extracting multi-scale temporal features that reflect the health status of the equipment. This method not only enhances the ability to capture key temporal information in audio signals, but also effectively improves the authenticity and discriminability of features by performing diffusion and adversarial training in the latent space, so that the model can maintain efficient anomaly detection performance when facing different equipment types and complex working conditions.

[0081] The present invention further enhances the model's ability to recognize key timing regions in audio signals by introducing a data augmentation network (TANet) and a time-domain hybrid enhancement module (TMixup), especially showing significant advantages in identifying abnormal changes. This multi-scale feature extraction strategy significantly improves the model's recognition sensitivity and accuracy for complex audio patterns. Compared with traditional abnormal sound detection technology, the present invention shows significant advantages in robustness, generalization ability, and detection effect, especially when dealing with unseen abnormal situations, it can maintain a high detection accuracy.

[0082] The device anomaly detection system based on the present invention can be widely used in the fields of manufacturing, energy, transportation, and robotics, and is applicable to various types of equipment. The system can analyze the audio signals of the equipment in real time during operation without affecting the normal operation of the equipment, quickly identify abnormal events, and greatly improve the safety and reliability of industrial production. At the same time, the system has high generalization ability and robustness, and can maintain stable detection performance even in complex environments, providing a reliable and economical anomaly detection solution for industrial equipment.

[0083] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0084] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0085] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0086] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0087] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0088] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0089] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0090] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0091] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0092] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0093] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0094] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0095] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A device anomaly detection method based on audio analysis, characterized in that: The steps include: S1, collecting sound signals from the device, and preprocessing the collected signals to generate a logarithmic Mel spectrogram; S2, through the temporal feature enhancement network TANet and the time domain hybrid enhancement module TMixup, multiple feature enhancements are performed on the logarithmic Mel spectrogram generated in step S1 to generate an enhanced temporal feature representation; S3, extracting multi-scale temporal features from the original audio signal through the multi-scale temporal feature extraction module MSTNet, including short-term, medium-term and long-term temporal dependencies, thereby capturing the temporal dynamics of the audio signal; S4. Using the logarithmic Mel-spectrogram features after multiple feature enhancement obtained in step S2 and the multi-scale time series features extracted in step S3, diffusion and adversarial training are performed in the latent space through the latent diffusion model LDM and the generative adversarial network GAN to generate features that can distinguish between normal and abnormal sound patterns, and the normal operation and abnormal state of the device are distinguished through the discriminator of the generative adversarial network, so as to identify whether the device is in an abnormal state.

2. The device anomaly detection method based on audio analysis according to claim 1, characterized in that: In step S1, a logarithmic Mel spectrum is generated by denoising, normalizing and performing short-time Fourier transform (STFT) on the collected signal.

3. The device anomaly detection method based on audio analysis according to claim 1 or 2, characterized in that: In step S2, the processing of the logarithmic Mel spectrogram by the temporal feature enhancement network TANet specifically includes: Perform a maximum pooling operation to extract the maximum value of the local area of ​​the logarithmic Mel spectrogram and enhance the peak characteristics of the sound signal; Perform an average pooling operation to calculate the average value of the local area of ​​the logarithmic Mel spectrogram, smooth the feature map and reduce the impact of noise; Perform a power average pooling operation to emphasize specific time domain information of the logarithmic Mel spectrogram through power operation and capture high-frequency changes; The results of the above pooling operations are weighted and fused, where the weight parameters are normalized so that the contribution of different pooling operations to the final features is in a predetermined proportion.

4. The device anomaly detection method based on audio analysis according to any one of claims 1 to 3, characterized in that: In step S2, the time domain mixing enhancement module TMixup processes the logarithmic Mel spectrogram in the following manner: Apply data augmentation operations to fuse the temporal features processed by TANet with the original spectrogram through mixing coefficients to generate mixed feature representations; The activation function is used to activate the TANet pooling operation results to limit the output range and control the feature strength; Through element-wise point-by-point multiplication, the activated temporal features are combined with the original spectrogram features to adjust the key areas in the features point by point.

5. The device anomaly detection method based on audio analysis according to any one of claims 1 to 4, characterized in that: In step S3, the processing of the original audio signal by the multi-scale temporal feature extraction module MSTNet specifically includes: Split the audio signal into multiple segments to maintain feature continuity and cover the complete audio data; The convolutional layer LKDConv is extended with large convolution kernels with different dilation rates to extract features at different time scales; The features extracted by the LKDConv layer are further processed through multi-layer dilated convolution blocks to integrate local and global information; The multi-scale features output by the LKDConv layer and the dilated convolutional block are concatenated in the channel dimension to form a comprehensive multi-scale feature representation.

6. The device anomaly detection method based on audio analysis according to claim 5, characterized in that: In step S3, a series of LKDConv layers are applied to each segment of the audio signal, each LKDConv layer has a specific dilation rate to capture short-term, medium-term and long-term temporal dependency features respectively; after the LKDConv layer, a multi-layer dilated convolution block DilatedConv Block is used to further refine the features, expand the receptive field by gradually increasing the dilation rate, and extract more refined temporal features.

7. The device anomaly detection method based on audio analysis according to any one of claims 1 to 6, characterized in that: In step S4, the multi-scale features obtained in step S3 are combined with the generative adversarial network The embeddings in the discriminator are concatenated and input to the The final embedding predictor is used as the final anomaly detection The measurement provides comprehensive feature information.

8. The device anomaly detection method based on audio analysis according to any one of claims 1 to 7, characterized in that: The processing of potential diffusion and adversarial generation in step S4 specifically includes: The enhanced spectral features are jointly optimized using the latent diffusion model and the generative adversarial network to improve the authenticity and discriminative ability of the features; Through an iterative process, the generative model is gradually trained to reconstruct high-quality spectral features, making them close to the real spectral features in terms of statistical characteristics; A loss function is designed and used, which is calculated based on the statistical difference between the generated features and the real features in terms of mean and standard deviation; wherein the loss function compares the mean and standard deviation of the generated features with the real features, and adjusts the loss value through a weighting parameter to optimize the distribution of the generated features.

9. The device anomaly detection method based on audio analysis according to any one of claims 1 to 7, characterized in that: The potential diffusion and adversarial generation in step S4 further includes the following processing methods: The encoder composed of multiple convolutional layers performs preliminary feature extraction on the input logarithmic Mel-spectrogram features, including using convolutional layers and residual blocks to enhance nonlinear feature representation; Down-sample the extracted features to reduce computational complexity and improve processing efficiency; The extracted features are mapped to a low-dimensional latent space and a diffusion process is performed in the space, where the noise intensity is controlled by a set parameter; The noise prediction model is trained in the latent space to optimize the feature representation by minimizing the noise prediction error. The loss function is used to learn to separate noise and potential features, gradually making the generated feature distribution close to the true feature distribution.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the device abnormality detection method based on audio analysis as described in any one of claims 1 to 9 is implemented.

Images