A device anomaly detection method based on audio analysis

Abstract

A device anomaly detection method based on audio analysis comprises: collecting sound signals from a device and preprocessing to generate a log mel spectrogram; processing the log mel spectrogram through a time feature enhancement network (TANet) and a time domain hybrid enhancement module (TMixup) to enhance its time sequence features and generate feature representation; extracting multi-scale time sequence features covering short-term, medium-term and long-term dependencies from the original audio signal using a multi-scale time sequence feature extraction module (MSTNet); combining the enhanced log mel spectrogram features and the multi-scale time sequence features extracted by the MSTNet, diffusing and adversarially training in the latent space through a latent diffusion model (LDM) and a generative adversarial network (GAN) to generate high-quality features, and analyzing the features through the discriminator of the generative adversarial network to determine whether the device is in an abnormal state. The present application can maintain efficient anomaly detection performance in complex and variable industrial environments, providing a reliable and economical anomaly detection solution for industrial devices.

Description

Technical Field

[0001] The present invention relates to a technology for evaluating the health status of equipment, and in particular to a method for detecting equipment anomalies based on audio analysis. Background Art

[0002] With the rapid development of Industry 4.0 and smart manufacturing, intelligent, automated, and refined equipment operation and maintenance have become key to modern industrial production. However, over long-term operation, equipment can face issues such as wear, aging, and failure. This not only impacts production efficiency but can also pose serious safety risks. Therefore, timely and accurate detection of equipment anomalies to ensure reliable operation has become a key focus within the industry.

[0003] Traditional equipment anomaly detection methods primarily rely on physical sensors, such as vibration sensors, temperature sensors, and pressure sensors. These sensors collect physical parameters of the equipment to determine whether it is in an abnormal state. However, these methods are complex to install and have high maintenance costs, and are highly dependent on the equipment's structure and environment. In recent years, anomaly detection methods based on audio analysis have become a hot topic in industrial equipment anomaly detection due to their non-contact, low-cost, and easy-to-deploy advantages. Audio analysis is a technology that assesses the health of equipment by collecting and analyzing the sound signals generated by equipment during operation. Audio analysis provides a non-contact, low-cost fault detection method. Through signal processing, feature extraction, and feature modeling, it enables real-time monitoring of equipment anomalies. It detects abnormal sounds generated by machines or equipment during operation and analyzes the normal sound characteristics of the equipment. When these sound characteristics change, the system can identify and flag them as abnormal events.

[0004] The goal of anomaly detection based on audio analysis is to identify anomalies by learning the sound characteristics of the device during normal operation. Currently, research focuses on anomaly detection methods based on self-supervised classification models and generative reconstruction models. For example, there are patent applications such as the improved AE machine abnormal sound detection method and device with patent number 202310933972.5, the metadata abnormal sound detection method with patent number 202310768780.3, the lightweight network unsupervised machine abnormal sound detection method with patent number 202311316592.3, and the domain transfer self-supervised machine abnormal sound detection method with patent number 202210863510.6.

[0005] However, the most popular abnormal sound detection technologies currently, such as the improved AE method with patent application number 202310933972.5, detect anomalies by learning the distribution of normal data. During training, these models attempt to reconstruct or generate samples similar to normal data. When the model cannot reconstruct or generate similar samples, it marks them as anomalies. However, such methods are prone to overfitting: the model performs well on the training data, but may not effectively detect anomalies when faced with new data or unseen anomalies. Another example is the metadata self-supervised classification model with patent application number 202310768780.3, which relies on classification information provided by metadata. However, different types of machines may produce very similar sound features during operation, making it difficult for the model to distinguish between different categories of sounds. Even between machines of the same category, sound features can vary significantly due to individual differences and varying operating conditions, further increasing the difficulty of extracting robust features. Furthermore, when metadata is incomplete, the classification labels are unclear, or the labels are incorrect, the model struggles to extract effective feature representations, resulting in reduced classification performance.

[0006] In general, existing abnormal sound detection methods have significant deficiencies in robustness, generalization, and effectiveness. This is particularly true in complex and volatile industrial audio scenarios, where it is difficult to consistently detect abnormal device conditions. Traditional methods have limited feature extraction capabilities for raw audio and log-mel spectrograms, making it difficult to fully capture the temporal dynamics and abnormal characteristics of sound signals, limiting detection performance.

[0007] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0008] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and provide a device abnormality detection method based on audio analysis.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect of the present invention, a method for detecting device anomalies based on audio analysis comprises the following steps:

[0011] S1. Collect sound signals from the device, preprocess the collected signals, and generate a logarithmic Mel spectrogram;

[0012] S2, through the temporal feature enhancement network TANet and the time domain hybrid enhancement module TMixup, the logarithmic Mel spectrogram generated in step S1 is enhanced with multiple features to generate an enhanced temporal feature representation;

[0013] S3, extracting multi-scale temporal features from the original audio signal through the multi-scale temporal feature extraction module MSTNet, including short-term, medium-term and long-term temporal dependencies, thereby capturing the temporal dynamics of the audio signal;

[0014] S4. Using the multi-feature enhanced log-mel spectrogram features obtained in step S2 and the multi-scale time series features extracted in step S3, diffusion and adversarial training are performed in the latent space through the latent diffusion model (LDM) and the generative adversarial network (GAN) to generate features that can distinguish between normal and abnormal sound patterns. The discriminator of the generative adversarial network is used to distinguish between normal operation and abnormal state of the device, thereby identifying whether the device is in an abnormal state.

[0015] In a second aspect of the present invention, a computer program product includes a computer program, which implements the device abnormality detection method based on audio analysis when executed by a processor.

[0016] The present invention has the following beneficial effects:

[0017] This paper proposes a device anomaly detection method based on audio analysis. By introducing a temporal feature enhancement network, a multi-scale temporal feature extraction module, and a latent diffusion adversarial generative model, it deeply mines the multi-scale temporal information of audio signals, enhances feature extraction capabilities, and constructs an efficient and accurate device anomaly detection system. This method not only improves the robustness and generalization of detection, but also maintains efficient and accurate detection performance across different device types, complex operating conditions, and even when no anomalies are observed, providing a more reliable and economical anomaly detection solution for industrial equipment.

[0018] Specifically, the present invention introduces the latent diffusion model (LDM) as a generator into the GAN framework, and performs a diffusion process in a low-dimensional latent space to reduce the computational complexity and training difficulty, while improving the authenticity and diversity of the generated features. Compared with traditional generative models, the latent diffusion generative adversarial model (TLDiffGAN) based on temporal feature enhancement of the present invention overcomes the problems of mode collapse and training instability. The generated features gradually approach the true distribution during the diffusion process, and through the adversarial training of the discriminator, the generator is able to produce more realistic and discriminative abnormal features, thereby improving the anomaly detection performance of the model. The present invention also introduces a multi-scale feature extraction network module (MSTNet), which effectively makes up for the shortcomings of the spectrogram in expressing temporal information, and provides a more comprehensive input for the model by fusing it with the log-mel spectrogram features. This multi-scale feature extraction strategy significantly enhances the model's ability to recognize complex audio patterns, especially when facing abnormal events, showing greater sensitivity and accuracy.

[0019] The present invention also introduces a data augmentation network (TANet), which can effectively extract and enhance key time series areas in the spectrogram, and has significant advantages in identifying abnormal changes. Combined with the TMixup operation, the time series features are further enhanced, so that the model can still maintain a high detection accuracy when facing variable audio signals. Compared with existing self-supervision and generative methods, the present invention shows excellent adaptability when facing audio signals of various types of machines. In particular, when dealing with unseen abnormal situations, it shows higher detection accuracy and robustness. The equipment anomaly detection system of the present invention can be widely used in manufacturing, energy, transportation, robotics and other fields, and is suitable for various types of equipment. By analyzing the audio signals of the equipment in real time during operation, the system can quickly identify abnormal events without affecting the normal operation of the equipment, greatly improving the safety and reliability of industrial production. At the same time, due to the audio analysis and feature extraction method of the present invention, the system has high generalization ability and robustness, and can maintain stable detection performance even in complex environments.

[0020] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 FIG. 4 is a diagram of the algorithm architecture of the abnormal sound detection network according to an embodiment of the present invention.

[0022] Figure 2 Detailed structural diagram of the MSTNet module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0024] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0025] See Figure 1 The embodiment of the present invention provides a device abnormality detection method based on audio analysis, comprising the following steps:

[0026] Step S1, sound signal acquisition and preprocessing: Acquire sound signals from the device, and preprocess the acquired signals to generate a logarithmic Mel-spectrogram.

[0027] In some embodiments, a log-mel spectrogram is generated by denoising, normalizing, and performing a short-time Fourier transform (STFT) on the collected signal. Specifically, the audio signal is subjected to a short-time Fourier transform (STFT), mapped to the Mel scale, and then logarithmized to produce a log-mel spectrogram. This feature representation, called a log-mel spectrogram, can better reflect the human ear's sensitivity to different frequencies.

[0028] Step S2, multiple feature enhancement: The logarithmic Mel spectrogram generated in step S1 is enhanced with multiple features through the temporal feature enhancement network TANet and the time domain hybrid enhancement module TMixup to generate an enhanced temporal feature representation.

[0029] In some embodiments, the processing of the log-Mel spectrogram by the temporal feature enhancement network TANet specifically includes: performing a maximum pooling operation to extract the maximum value of the local area of ​​the log-Mel spectrogram and enhance the peak characteristics of the sound signal; performing an average pooling operation to calculate the average value of the local area of ​​the log-Mel spectrogram, smooth the feature map and reduce the impact of noise; performing a power average pooling operation to emphasize the specific time domain information of the log-Mel spectrogram through power operations and capture high-frequency changes; performing weighted fusion on the results of the above pooling operations, wherein the weight parameters are normalized so that the contribution of different pooling operations to the final features is carried out in a predetermined proportion.

[0030] In some embodiments, the processing performed by the time-domain hybrid enhancement module TMixup on the logarithmic Mel-spectrogram specifically includes: applying a data enhancement operation, fusing the time series features obtained by TANet processing with the original spectrogram through a mixing coefficient to generate a hybrid feature representation; using an activation function to activate the results of the TANet pooling operation to limit the output range and control the feature intensity; through element-level point-by-point multiplication, combining the activated time series features with the original spectrogram features, and adjusting the key areas in the features point by point.

[0031] By weighted mixing of the temporal features of different samples, data with richer temporal features is generated, which not only enhances the robustness of the model but also improves its generalization ability.

[0032] Step S3, multi-scale temporal feature extraction: The multi-scale temporal feature extraction module MSTNet is used to extract multi-scale temporal features from the original audio signal, including short-term, medium-term and long-term temporal dependencies, thereby capturing the temporal dynamics of the audio signal.

[0033] In some embodiments, the processing of the original audio signal by the multi-scale temporal feature extraction module MSTNet specifically includes: dividing the audio signal into multiple segments to maintain the continuity of features and cover the complete audio data; using large convolution kernels with different expansion rates to expand the convolution layer LKDConv to extract features at different time scales; further processing the features extracted by the LKDConv layer through multi-layer dilated convolution blocks Dilated Conv Block to integrate local and global information; splicing the multi-scale features output by the LKDConv layer and the dilated convolution block in the channel dimension to form a comprehensive multi-scale feature representation.

[0034] Among them, a series of LKDConv layers are applied to each segment of the audio signal, and each LKDConv layer has a specific dilation rate to capture short-term, medium-term and long-term temporal dependency features respectively; after the LKDConv layer, a multi-layer dilated convolution block is used to further refine the features, and the receptive field is expanded by gradually increasing the dilation rate to extract more refined temporal features.

[0035] Step S4, latent diffusion and adversarial generation based on temporal feature enhancement: Using the log-mel spectrogram features after multi-feature enhancement obtained in step S2 and the multi-scale temporal features extracted in step S3, diffusion and adversarial training are performed in the latent space through the latent diffusion model LDM and the generative adversarial network GAN to generate high-quality features that can distinguish between normal and abnormal sound patterns. The discriminator of the generative adversarial network is used to distinguish between normal operation and abnormal state of the device, thereby accurately identifying whether the device is in an abnormal state.

[0036] The latent diffusion model (LDM) generates data by performing a diffusion process in a low-dimensional latent space. Unlike traditional diffusion models, LDM operates in the latent space, which can significantly improve the training and generation efficiency of the model. Generative adversarial networks (GANs) are composed of two parts: a generator and a discriminator. Deep learning model composed of groups. The generator attempts to generate realistic data, while the discriminator attempts to distinguish The generated data and the real data are divided into two parts. Through the adversarial training process, the generator can gradually generate data that is very close to the real data.

[0037] In some embodiments, the multi-scale features obtained in step S3 are concatenated with the embedding in the discriminator of the generative adversarial network and input into the final embedding predictor to provide comprehensive feature information for the final anomaly detection.

[0038] In some embodiments, the processing of potential diffusion and adversarial generation in step S4 specifically includes: using the potential diffusion model and the generative adversarial network to jointly optimize the enhanced spectral features to improve the authenticity and discriminability of the features; through an iterative process, gradually training the generative model to reconstruct high-quality spectral features so that they are statistically close to the real spectral features; designing and using a loss function, which is calculated based on the statistical difference in mean and standard deviation between the generated features and the real features; wherein, the loss function is used to compare the mean and standard deviation of the generated features with the real features, and the loss value is adjusted by the weighting parameter to optimize the distribution of the generated features.

[0039] In some embodiments, the potential diffusion and adversarial generation in step S4 includes the following processing methods: performing preliminary feature extraction on the input log-mel spectrum features through an encoder composed of multiple convolutional layers, including using convolutional layers and residual blocks to enhance nonlinear feature representation; downsampling the extracted features to reduce computational complexity and improve processing efficiency; mapping the extracted features to a low-dimensional latent space and performing a diffusion process in the space, where the noise intensity is controlled by set parameters; using a noise prediction model to train in the latent space, and optimizing the feature representation by minimizing the noise prediction error; using a loss function to learn to separate noise and latent features, and gradually making the generated feature distribution approach the true feature distribution.

[0040] The device anomaly detection method based on audio analysis proposed in the present invention can build an efficient and accurate device anomaly detection system. By introducing the temporal feature enhancement network (TANet), the multi-scale temporal feature extraction module (MSTNet) and the latent diffusion generative adversarial model (LDiffGAN), the present invention can deeply explore the multi-scale temporal information in the audio signal and significantly improve the feature extraction capability. The combination of TANet and TMixup enhances the features of the key temporal regions in the log-Mel spectrogram, making the model more sensitive and accurate in identifying abnormal sound changes. The MSTNet module effectively captures the short-term, medium-term and long-term dependencies of the audio signal through multi-scale design, making up for the shortcomings of the traditional log-Mel spectrogram in expressing temporal information. At the same time, the combination of the latent diffusion model (LDM) and the generative adversarial network (GAN) optimizes the diffusion process in the latent space, reduces the computational complexity and training difficulty, and enhances the authenticity and diversity of the generated features. The latent diffusion generative adversarial model (TLDiffGAN) based on temporal feature enhancement effectively overcomes the problems of mode collapse and training instability in traditional models and improves the model's detection performance for abnormal situations.

[0041] The device anomaly detection system of the present application not only improves the robustness and generalization ability of detection, but also can adapt to various device types and complex working conditions, and can maintain high efficiency and accurate detection performance even in the face of unseen abnormal situations. The system can be widely used in manufacturing, energy, transportation and robot fields, providing a reliable and economical anomaly detection solution for industrial devices, greatly improving the safety and reliability of industrial production. By analyzing the audio signals in real time during device operation, the system of the present application can quickly identify abnormal events without affecting the normal operation of the device, and can maintain stable detection performance even in complex environments.

[0042] The specific embodiments of the present application and examples of algorithm implementation are further described below.

[0043] Sound signal acquisition and preprocessing

[0044] First, the sound signals generated during the operation of the device are collected by high-sensitivity microphones, and the collected audio is preprocessed. This includes steps such as noise removal, normalization and short-time Fourier transform (STFT) to generate a log-mel spectrogram as input for the feature extraction module. The log-mel spectrogram can more intuitively reflect the frequency distribution of the audio signal, effectively enhancing the frequency details that the human ear cannot perceive.

[0045] Anomaly sound detection network architecture

[0046] A latent diffusion model and a generative adversarial network framework with good generalization and sampling performance are adopted, and the overall framework is as shown in Figure 1 .

[0047] The framework mainly includes a multi-scale convolutional neural network (MSTNet), a time feature enhancement network (TANet), a time domain mixing enhancement module (TMixup), and a diffusion generation module.

[0048] The overall architecture starts with feature extraction from audio signals, goes through multiple feature enhancement and latent space diffusion generation, and finally completes anomaly detection through embedding and reconstruction detectors.

[0049] The original audio is taken as input, and one branch is subjected to logarithmic transformation after passing through 128 mel filters to obtain a log-mel spectrogram. In order to enhance the timing information of key areas in the spectrogram, a time feature enhancement network (TANet) module is used for efficient timing feature extraction.

[0050] To enhance the temporal information of key regions in the spectrogram, the TANet module adopts a weighted fusion strategy of multiple pooling operations. This strategy aims to enhance the model's sensitivity to temporal information, enabling it to better capture the dynamic characteristics of audio signals. The TANet module employs three pooling operations and performs weighted summation:

[0051]

[0052] MaxPool is a max-pooling operation used to extract the maximum value of a local region in the input feature map, Enhancing information in high-activation regions can highlight the peak features of the sound signal; AvgPool is an average pooling operation that smooths the feature map by calculating the average value of the local region, providing stable temporal information, and AvgPool helps to reduce the impact of noise; PowerAvgPool is a power average pooling, which is an adjustable pooling method that emphasizes specific temporal information by performing power operations on features. This pooling can better capture high-frequency changes. are the weight parameters corresponding to the pooling operations, which are normalized to ensure their sum equals 1. By adjusting these weights, the contribution of different pooling operations to the final features can be flexibly controlled.

[0053] Through the above operations, the temporal information in the spectrogram is enhanced, making the model better capture temporal features. Then, through the TMixup data augmentation operation, the temporal features are fused with the original spectrogram to obtain the final input spectrogram features:

[0054]

[0055] where λ is the mixing coefficient of the Beta distribution; the temporal features obtained by the pooling operation are passed through the sigmoid activation function σ to limit the output between 0 and 1, further controlling the strength of the temporal features; is the element-level point-by-point multiplication (Hadamard multiplication), which multiplies the activated temporal features and the original spectrogram features element by element, emphasizing or suppressing key regions in the features point by point.

[0056] Afterwards, the generated spectrogram features are input into the LDiffGAN module, and through the joint optimization of the Latent Diffusion process and the generative adversarial network, the authenticity and discriminative ability of the generated features are gradually improved. After multiple iterations, the LDiffGAN module is able to reconstruct high-quality spectrograms. D2GLoss is used as the loss function of the LDiffGAN module. D2GLoss optimizes the distribution of generated features by calculating the difference in statistical characteristics (mean and standard deviation) between the generated features and the real features, making it as close to the real distribution as possible. The loss function calculates the L2 norm of the mean and standard deviation of the generated features and the real features, and performs a weighted summation of the loss values ​​in combination with the weight parameters, and finally optimizes the generation process. The specific loss function calculation formula is as follows:

[0057] L=μ eff MSE(μ fake ,μ real )+σ eff MSE(σ fake ,σ real )

[0058] μ eff ,σ eff μ is the weighted parameter of the mean and standard deviation in the loss function, which is adjusted by the settings in the configuration file; fake ,μ real are the means of generated features and real features respectively; σ fake ,σ real are the standard deviations of generated features and true features respectively; MSE represents the mean square error (L2 norm).

[0059] Multi-scale temporal feature network module

[0060] Since the log-mel spectrogram has certain limitations in capturing the temporal information of the original audio, the inventors drew inspiration from TgramNet and designed a method for extracting multi-scale temporal information from the original audio signal. MSTNet module of the time sequence feature, The MSTNet module extracts short-term and long-term dependent temporal features to make up for the lack of spectrogram information. The specific structure of the MSTNet module is as follows Figure 2 shown.

[0061] First, the input is the original audio waveform signal with a sampling rate of 16kHz. The audio signal is divided into multiple segments, each segment contains 128 frames, and the frame shift is 1 frame. This can maintain the continuity of features on the time axis and cover the complete audio data. Multi-scale feature extraction module

[0062] MSTNet consists of three large-kernel dilated convolution layers (LKDConv, Large-Kernel DilatedConvolution), which have different dilation rates to extract features at different time scales. The specific settings of each convolution layer are as follows:

[0063]

[0064] LKDConv1:

[0065] C in Number of input channels, C out The number of output channels, K is the convolution kernel size, D is the dilation rate, P is the padding size (Padding), and n is the number of layers.

[0066] LKDConv1 primarily captures short-term dependencies in audio signals, using a small expansion rate to model local features in detail. LKDConv2 uses a larger expansion rate to capture medium-term temporal dependencies, expanding the receptive field of the convolution and thus acquiring a wider range of temporal features. LKDConv3 further improves the expansion rate to capture global temporal features with long-term dependencies. This layer uses large convolution kernels and a high expansion rate to capture long-range dependencies and rhythmic information in the audio as a whole.

[0067] In order to further enhance the representation ability of temporal features, MSTNet also includes a multi-layer dilated convolution block (DilatedConv Block).

[0068] These dilated convolutional blocks further process the features extracted by the LKDConv layer. By continuously expanding the receptive field, they extract more refined temporal features and integrate local and global information, providing richer temporal features for the subsequent generative adversarial module. Finally, the multi-scale features output from each convolutional layer are concatenated along the channel dimension to form the final multi-scale features. This aims to integrate short-term, medium-term, and long-term temporal features, providing more comprehensive information for the discriminator and generative modules. The final feature embedding is concatenated with the embedding from the LDiffGAN module's discriminator and input into the final embedding predictor.

[0069] Latent Diffusion and Generative Adversarial TLDiffGAN Module Based on Temporal Feature Enhancement

[0070] The TLDiffGAN module combines the Latent Diffusion Model (LDM) and the Generative Adversarial Network (GAN) for the task of abnormal sound detection. The module takes preprocessed log-mel spectrogram features as input, extracts features through multiple specialized network layers, and performs diffusion and adversarial training in the latent space.

[0071] First, the input spectrogram is passed through an Encoder module consisting of multiple convolutional layers for preliminary feature extraction:

[0072] Conv2D layer: The input channel is 1 (single-channel spectrogram), the output channel is 256, the convolution kernel size is 4×4, the stride is 1, there is no padding, and the ReLU activation function is used.

[0073] Residual Block: During the feature extraction process, three residual blocks are added. Each block contains two convolutional layers and a skip connection to improve the nonlinear representation capability of feature extraction.

[0074] Downsampling layer: In order to reduce computational complexity, the features after the residual block are downsampled twice, and the final output feature resolution is 1 / 16 of the original input.

[0075] The feature z0 extracted by the Encoder is mapped to the low-dimensional latent space z t In, z t The distribution at time t is defined as follows:

[0076]

[0077] Among them, α t Control the noise intensity during the diffusion process. LDM trains the noise prediction model in the latent space to minimize the noise prediction error at each time step t. The objective function is:

[0078]

[0079] Through this loss function, the model learns how to separate noise from latent features, thereby gradually approaching the true feature distribution.

[0080] In summary, the present invention combines audio analysis technology with deep learning models to propose an efficient and accurate method for detecting equipment anomalies. By adopting a temporal feature enhancement network (TANet), a multi-scale temporal feature extraction module (MSTNet), and a combination of a latent diffusion model (LDM) and a generative adversarial network (GAN), the present invention can deeply mine and analyze the audio signals generated during equipment operation, thereby extracting multi-scale temporal features that reflect the health status of the equipment. This method not only enhances the ability to capture key temporal information in audio signals, but also effectively improves the authenticity and discriminability of features by performing diffusion and adversarial training in the latent space, so that the model can maintain efficient anomaly detection performance when facing different equipment types and complex working conditions.

[0081] By introducing a data augmentation network (TANet) and a time-domain hybrid enhancement module (TMixup), the present invention further enhances the model's ability to identify key time regions in audio signals, showing a significant advantage in identifying abnormal changes. This multi-scale feature extraction strategy significantly improves the model's sensitivity and accuracy in recognizing complex audio patterns. Compared with traditional abnormal sound detection technologies, the present invention demonstrates significant advantages in robustness, generalization, and detection effectiveness, especially when dealing with unseen abnormal situations, maintaining high detection accuracy.

[0082] The device anomaly detection system based on this invention can be widely used in fields such as manufacturing, energy, transportation, and robotics, and is applicable to a wide range of equipment types. The system can analyze audio signals during device operation in real time without disrupting normal operation, quickly identifying abnormal events and significantly improving the safety and reliability of industrial production. Furthermore, the system exhibits high generalization and robustness, maintaining stable detection performance even in complex environments, providing a reliable and economical anomaly detection solution for industrial equipment.

[0083] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0084] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0085] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0086] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0087] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0088] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0090] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0091] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0092] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0093] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0094] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0095] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A device anomaly detection method based on audio analysis, characterized in that: The steps include: S1. Collect sound signals from the device, preprocess the collected signals, and generate a logarithmic Mel spectrogram; S2, through the temporal feature enhancement network TANet and the time domain hybrid enhancement module TMixup, the logarithmic Mel spectrogram generated in step S1 is enhanced with multiple features to generate an enhanced temporal feature representation; S3, extracting multi-scale temporal features from the original audio signal through the multi-scale temporal feature extraction module MSTNet, including short-term, medium-term and long-term temporal dependencies, thereby capturing the temporal dynamics of the audio signal; S4. Using the multi-feature enhanced log-mel spectrogram features obtained in step S2 and the multi-scale time series features extracted in step S3, diffusion and adversarial training are performed in the latent space through the latent diffusion model (LDM) and the generative adversarial network (GAN) to generate features that can distinguish between normal and abnormal sound patterns. The discriminator of the generative adversarial network is used to distinguish between normal operation and abnormal state of the device, thereby identifying whether the device is in an abnormal state.

2. The device anomaly detection method based on audio analysis according to claim 1, characterized in that: In step S1, the collected signal is denoised, normalized, and subjected to short-time Fourier transform (STFT) to generate a logarithmic Mel-spectrogram.

3. The device anomaly detection method based on audio analysis according to claim 1 or 2, characterized in that: In step S2, the temporal feature enhancement network TANet processes the logarithmic Mel spectrogram in the following ways: Perform a maximum pooling operation to extract the maximum value of the local area of ​​the logarithmic Mel spectrogram and enhance the peak characteristics of the sound signal; Perform an average pooling operation to calculate the average value of the local area of ​​the logarithmic Mel spectrogram, smooth the feature map and reduce the impact of noise; Perform a power average pooling operation to emphasize the specific time domain information of the log-mel spectrogram through power operation and capture high-frequency changes; The results of the above pooling operations are weighted and fused, where the weight parameters are normalized so that the contributions of different pooling operations to the final features are in a predetermined proportion.

4. The device anomaly detection method based on audio analysis according to any one of claims 1 to 3, characterized in that: In step S2, the time domain mixing enhancement module TMixup processes the logarithmic Mel spectrogram in the following manner: Apply data augmentation operations to fuse the temporal features obtained by TANet with the original spectrogram through mixing coefficients to generate a hybrid feature representation; The activation function is used to activate the TANet pooling operation results to limit the output range and control the feature strength; Through element-wise multiplication, the activated temporal features are combined with the original spectrogram features to adjust the key areas in the features point by point.

5. The device anomaly detection method based on audio analysis according to any one of claims 1 to 4, characterized in that: In step S3, the multi-scale temporal feature extraction module MSTNet processes the original audio signal in the following ways: Split the audio signal into multiple segments to maintain feature continuity and cover the complete audio data; The convolutional layer LKDConv is extended with large convolution kernels with different dilation rates to extract features at different time scales; The features extracted by the LKDConv layer are further processed through multi-layer dilated convolution blocks to integrate local and global information; The multi-scale features output by the LKDConv layer and the dilated convolution block are concatenated in the channel dimension to form a comprehensive multi-scale feature representation.

6. The device anomaly detection method based on audio analysis according to claim 5, characterized in that: In step S3, a series of LKDConv layers are applied to each segment of the audio signal, each with a specific dilation rate to capture short-term, medium-term, and long-term temporal dependency features, respectively. After the LKDConv layer, a multi-layer dilated convolution block (DilatedConv Block) is used to further refine the features, expanding the receptive field by gradually increasing the dilation rate and extracting more refined temporal features.

7. The device anomaly detection method based on audio analysis according to any one of claims 1 to 6, characterized in that: In step S4, the multi-scale features obtained in step S3 are concatenated with the embedding in the discriminator of the generative adversarial network and input into the final embedding predictor to provide comprehensive feature information for the final anomaly detection.

8. The device anomaly detection method based on audio analysis according to any one of claims 1 to 7, characterized in that: The processing of potential diffusion and adversarial generation in step S4 specifically includes: The enhanced spectral features are jointly optimized using the latent diffusion model and generative adversarial network to improve the authenticity and discriminative ability of the features; Through an iterative process, the generative model is gradually trained to reconstruct high-quality spectral features, making them close to the real spectral features in terms of statistical properties; A loss function is designed and used, which is calculated based on the statistical difference in mean and standard deviation between the generated features and the true features; wherein the loss function compares the mean and standard deviation of the generated features with the true features, and adjusts the loss value through a weighting parameter to optimize the distribution of the generated features.

9. The device anomaly detection method based on audio analysis according to any one of claims 1 to 7, characterized in that: The potential diffusion and adversarial generation in step S4 further includes the following processing methods: An encoder composed of multiple convolutional layers is used to perform preliminary feature extraction on the input log-mel spectrogram features, including using convolutional layers and residual blocks to enhance nonlinear feature representation; Downsampling the extracted features to reduce computational complexity and improve processing efficiency; The extracted features are mapped to a low-dimensional latent space and a diffusion process is performed in this space, where the noise intensity is controlled by a set parameter; Utilize the noise prediction model trained in the latent space to optimize the feature representation by minimizing the noise prediction error; The loss function is used to learn to separate noise and potential features, and gradually make the generated feature distribution close to the true feature distribution.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the device abnormality detection method based on audio analysis according to any one of claims 1 to 9 is implemented.

Images