Equipment abnormity monitoring method based on audio signal analysis
Through the methods of adaptive feature enhancement and fusion, time-frequency decoupling feature extraction and cross-modal feature fusion, the problems of insufficient robustness and generalization ability of equipment anomaly detection in the existing technology are solved, and efficient and accurate detection of equipment anomalies is achieved, which is suitable for status monitoring of industrial equipment.
Patent Information
- Application Number
- CN202511272147.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-08
- Publication Date
- 2025-10-10
- Estimated Expiration
- 2045-09-08
AI Technical Summary
Existing equipment anomaly detection methods based on audio analysis have significant deficiencies in robustness, generalization ability, and detection effect. In particular, it is difficult to stably detect abnormal equipment conditions in complex and changeable industrial audio scenarios. Traditional methods have limited feature extraction capabilities for raw audio and log-Mel spectrograms, making it difficult to fully capture the temporal dynamics and abnormal characteristics in sound signals.
Adaptive feature enhancement and fusion, time-frequency decoupling feature extraction and cross-modal feature fusion methods are adopted. Multi-channel fusion features are generated through an adaptive gating module. The selective state space model is used for deep feature extraction. The self-supervised ID classification task and the angular interval loss function are combined for model training to realize anomaly score calculation.
It improves the robustness and generalization capabilities of equipment anomaly detection, and can efficiently and accurately detect anomalies in the face of different equipment types and complex working conditions, providing a more reliable and economical anomaly detection solution.
Smart Images

Figure CN120766722A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to audio signal analysis and equipment abnormality monitoring technology, and in particular to an equipment abnormality monitoring method based on audio signal analysis. Background Art
[0002] Intelligent, automated, and refined equipment operation and maintenance are key to modern industrial production. However, over time, equipment can face issues like wear, aging, and failure. This not only impacts production efficiency but can also pose serious safety risks. Therefore, timely and accurate detection of equipment anomalies to ensure reliable operation has become a key focus within the industry.
[0003] Traditional methods for detecting equipment anomaly rely primarily on physical sensors, such as vibration sensors, temperature sensors, and pressure sensors. These sensors collect physical parameters of the equipment to determine whether it is in an abnormal state. However, these methods are complex to install, require high maintenance costs, and are highly dependent on the equipment's structure and environment. In recent years, audio analysis-based anomaly detection methods have become a hot topic for industrial equipment anomaly detection due to their non-contact, low-cost, and easy-to-deploy advantages.
[0004] The goal of anomaly detection based on audio analysis is to identify abnormalities by learning the sound characteristics of a device during normal operation. Currently, research focuses on anomaly detection methods based on self-supervised classification models and generative reconstruction models. For example, there are patent applications such as patent number 202310933972.5 for an improved AE method and device for detecting abnormal machine sounds, patent number 202411896447.1 for a self-supervised method for detecting abnormal machine sounds based on a dual-path WaveNet, patent number 202311316592.3 for an unsupervised method for detecting abnormal machine sounds using a lightweight network, patent number 202210863510.6 for a domain-transferred self-supervised method for detecting abnormal machine sounds, and patent number 202510161917.8 for a multi-class method for detecting abnormal machine sounds based on a diffusion model.
[0005] However, the most popular abnormal sound detection technologies currently receiving the most attention, such as the multi-class machine abnormal sound detection method based on diffusion models (patent application numbers 202310933972.5 and 202510161917.8), improve AE methods by learning the distribution of normal data to detect anomalies. During training, these models attempt to reconstruct or generate samples similar to normal data. When the model cannot reconstruct or generate similar samples, it marks them as anomalies. However, these methods are prone to overfitting: the model performs well on the training data, but may not effectively detect anomalies when faced with new data or unseen anomalies. Another example is patent application number 202411896447.1, which proposes a self-supervised machine abnormal sound detection method based on a dual-path WaveNet. This method models the frequency information of each time frame of the input sound signal and models the time dimension. However, the receptive field of this model is limited to a local time window, making it difficult to effectively capture the characteristics of the machine during long-term steady-state operation and unable to model trend-level abnormal patterns.
[0006] In general, existing abnormal sound detection methods have significant deficiencies in robustness, generalization, and effectiveness. This is particularly true in complex and volatile industrial audio scenarios, where it is difficult to consistently detect abnormal device conditions. Traditional methods have limited feature extraction capabilities for raw audio and log-mel spectrograms, making it difficult to fully capture the temporal dynamics and abnormal characteristics of sound signals, limiting detection performance.
[0007] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention
[0008] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a device abnormality monitoring method based on audio signal analysis.
[0009] To achieve the above object, the present invention adopts the following technical solutions: A device abnormality monitoring method based on audio signal analysis includes the following steps: S1. Adaptive feature enhancement and fusion: The original audio signal is transformed into a logarithmic Mel-spectrogram through time-frequency transformation. The median, root mean square, and variance statistics of each time frame are calculated through an adaptive gating module based on third-order statistical indicators. The gating vector is generated and weightedly fused with the original spectrogram. The supplementary audio features are then spliced together to form a multi-channel fusion feature. S2. Time-frequency decoupling feature extraction: The multi-channel fusion features are divided into time domain block sequences and frequency domain block sequences along the time axis and frequency axis, and input into parallel time domain branches and frequency domain branches respectively; the time domain branch performs causal scanning on the time domain block sequence through a selective state space model to capture long-term steady-state features and trend anomalies, and the frequency domain branch performs causal scanning on the frequency domain block sequence through a selective state space model to analyze cross-band energy distribution and harmonic structure; S3. Cross-modal feature fusion and anomaly scoring: The feature vectors output by the time domain branch and the frequency domain branch are linearly aligned and then added element-by-element to obtain a fused feature representation. Based on the fused feature representation, a self-supervised ID classification task combined with an angular margin loss function is used for model training. During the inference phase, the negative logarithmic probability of the target ID category is used as the anomaly score.
[0010] Furthermore, the adaptive gating module in step S1 generates a gating vector through the following process: The median, root mean square, and variance statistics are calculated frame by frame on the transposed log-mel spectrogram. The three statistics are summed and then the preset offset is subtracted, adjusted by the scaling factor and activated by the Sigmoid function; The weighted fusion is to multiply the gate vector and the original logarithmic Mel spectrogram element by element to achieve time frame weight enhancement.
[0011] Furthermore, the multi-channel fusion feature in step S1 is constructed in the following way: The gated enhanced log-mel spectrogram is used as the first channel; Keep the original log-mel spectrogram as the second channel; Supplement the time domain features of the original audio as the third channel; The three-channel features are concatenated along the channel dimension to form the final input features.
[0012] Furthermore, the processing flow of the time domain branch in step S2 specifically includes: Flatten the time domain block sequence and add class token and position embeddings; It is processed sequentially through layer normalization, linear transformation, two-dimensional convolution and activation function; Dynamically generate selective state-space model parameters based on input features; Capturing long-range dependencies through the state-space modeling layer; The output is added to the original input via a residual connection.
[0013] Furthermore, the frequency domain branch processing flow in step S2 specifically includes: Flatten the frequency domain block sequence and add class token and position embeddings; It is processed sequentially through layer normalization, linear transformation, two-dimensional convolution and activation function; Dynamically generate selective state-space model parameters based on input features; Analyze frequency domain structural characteristics through the state space modeling layer; The output is added to the original input via a residual connection.
[0014] Furthermore, in step S2, the selective state space model implements input-dependent state transition through dynamic parameter calculation, specifically including: Generate time step parameters from intermediate features through linear transformation; Calculate the state transfer matrix through the time step parameters and preset parameters; Generate the input projection matrix by linear transformation of the intermediate features; Perform state-space modeling operations using dynamically generated state transition matrices and input projection matrices.
[0015] Furthermore, the cross-modal feature fusion in step S3 specifically includes: Linearly project the time domain branch output features and the frequency domain branch output features respectively to achieve feature space alignment; The two aligned feature vectors are added element by element to generate the final fusion feature.
[0016] Furthermore, the self-supervised training process in step S3 specifically includes: Construct a classification task using the device ID as the category label; The angular margin loss function is used to optimize the model, forcing the features of different machine IDs to maximize the inter-class distance and minimize the intra-class distance in the angular space; The angular interval loss function introduces additive angular interval constraints on feature distribution.
[0017] Furthermore, the anomaly score calculation method in step S3 includes: Input the audio to be tested into the trained model; Extract the predicted probability of the corresponding real machine ID category; The negative natural logarithm of the predicted probability is calculated as the anomaly score.
[0018] A computer program product includes a computer program, wherein when the computer program is executed by a processor, the device abnormality monitoring method based on audio signal analysis is implemented.
[0019] The present invention has the following beneficial effects: In view of the problems in the prior art, the present application proposes a device anomaly detection method based on audio analysis, which introduces a time sequence feature enhancement network, a multi-scale time sequence feature extraction module and a latent diffusion generative model to deeply mine the multi-scale time sequence information of the audio signal to enhance the feature extraction capability. At the same time, through the innovative end-to-end deep learning framework, the audio signal during the operation of the industrial equipment is analyzed in detail, and the overall technical scheme is divided into three core stages: first, adaptive feature enhancement and fusion based on multiple statistical indicators, second, deep feature extraction using a dual-branch Mamba network for time-frequency decoupling, and finally, model training and anomaly score calculation through a self-supervised classification task and a unique loss function. This method not only improves the robustness and generalization ability of detection, but also maintains efficient and accurate detection performance when facing different equipment types, complex working conditions and unseen anomalies, providing a more reliable and economical anomaly detection solution for industrial equipment.
[0020] Compared with the prior art, the present application has achieved significant improvement in the mechanism and effect of device anomaly sound detection. The fundamental beneficial effect is that the modeling idea of time-frequency decoupling is combined with the selective state space model (SSM) for the first time and introduced into the field of anomaly sound detection. Through the designed spectral-time dual-branch Mamba (STMamba) backbone network, this method completely gets rid of the defects of traditional models limited by local receptive fields: the time domain Mamba (TMamba) branch in this architecture can perform long-range scanning along the time axis, effectively capturing long-period acoustic features and trend changes during machine steady operation; while the parallel spectral Mamba (SMamba) branch accumulates and analyzes the energy distribution and harmonic structure across the frequency band along the frequency axis. This strategy of modeling and fusing the two orthogonal dimensions of time and frequency separately allows the model to analyze machine acoustic features more finely and comprehensively than ever before, thereby overcoming the inherent problem of existing methods that ignore time-frequency dynamic coupling due to independent processing of spectral frames.
[0021] Another significant benefit of this invention lies in its innovative feature enhancement mechanism, which fundamentally improves the model's sensitivity to anomaly patterns and robustness. This innovative TriStat-Gating (TSG) gating mechanism doesn't passively receive acoustic features; instead, it proactively and adaptively enhances them. By constructing a gating signal based on three complementary statistical metrics—median, root mean square, and variance—it intelligently amplifies time regions in the input spectrogram that exhibit potential anomalies in stability, intensity, or distribution uniformity. This mechanism ensures that the enhanced features (ESTgrams) ultimately fed into the backbone network carry more discriminative information, enabling the model to more easily capture subtle differences between normal and abnormal conditions. This front-end intelligent enhancement, combined with the powerful time-frequency decoupling modeling capabilities of the back-end, creates a synergistic effect. This not only improves the model's detection accuracy for diverse equipment and complex operating conditions, but more importantly, enhances its generalization and stability when dealing with unknown anomaly types, providing a more reliable and advanced solution for condition monitoring of industrial equipment.
[0022] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 The figure is an overall flow chart of the device anomaly detection method based on audio analysis of the present invention.
[0024] Figure 2 This is a diagram of the algorithm architecture of the device anomaly detection method based on audio analysis according to an embodiment of the present invention. DETAILED DESCRIPTION
[0025] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.
[0026] Definitions of Abbreviations and Key Terms: Audio analysis: A technology that assesses the health of equipment by collecting and analyzing the sound signals emitted by equipment during operation. It provides a non-contact, low-cost fault detection method. By processing signals, extracting features, and modeling them, it enables real-time monitoring of equipment anomalies.
[0027] Abnormal sound detection: A method used to identify unusual sounds produced by machines or equipment during operation. It analyzes the sound signatures of the equipment during normal operation. When these signatures change, the system identifies them and flags them as abnormal events. This method is widely used in various fields, including industrial manufacturing, energy equipment, transportation, and robotic automation, to improve equipment reliability and safety.
[0028] Selective State-Space Models (SSM): An advanced sequence modeling architecture that adaptively focuses on key information in a sequence by introducing an input-dependent state transition mechanism. SSM effectively models long sequences and is well suited for processing both long-term steady-state features and sudden abnormal events in machine sound signals.
[0029] TriStat-Gating (TSG) gating module: This invention proposes a feature enhancement module that enhances the input mel-spectrogram by introducing an adaptive gating mechanism based on third-order statistical metrics (median, root mean square, and variance). This module can improve the model's sensitivity to unknown abnormal patterns.
[0030] Log-Mel Spectrogram: A two-dimensional time-frequency spectrum representation of audio signals. It converts the audio signal to the frequency domain using the Short-Time Fourier Transform (STFT), then maps it to the Mel scale and takes its logarithm. This representation aligns with the human auditory perception and is a common feature in audio processing.
[0031] This paper aims to improve the robustness and generalization ability of equipment anomaly detection, as well as the detection performance under different equipment types, complex working conditions, and no anomalies. It proposes an equipment anomaly detection method based on audio analysis. By introducing a temporal feature enhancement network, a multi-scale temporal feature extraction module, and a latent diffusion adversarial generative model, this method deeply mines the multi-scale temporal information of audio signals, enhances feature extraction capabilities, and provides a more reliable and economical anomaly detection solution for industrial equipment.
[0032] See Figure 1 The embodiment of the present invention provides a device abnormality monitoring method based on audio signal analysis, comprising the following steps: Step S1, adaptive feature enhancement and fusion: Perform time-frequency transformation on the original audio signal to generate a logarithmic Mel-spectrogram. The median, root mean square, and variance statistics of each time frame are calculated through an adaptive gating module (TSG) based on third-order statistical indicators. A gating vector is generated and weightedly fused with the original spectrogram. The supplementary audio features are then spliced to form a multi-channel fusion feature.
[0033] To eliminate dimensional differences between different statistical indicators, the three calculated statistical vectors can be standardized (for example, Z-score standardization) to a mean of 0 and a variance of 1. These three normalized statistics are calculated frame by frame on the transposed spectrogram, and then concatenated to form multi-channel fusion features.
[0034] In some embodiments, the adaptive gating module generates a gating vector by the following process: calculating three statistics of median, root mean square and variance on the transposed log-mel spectrogram frame by frame; summing the three statistics, subtracting a preset offset, adjusting by a scaling factor and activating by a Sigmoid function; the weighted fusion is to multiply the gating vector and the original log-mel spectrogram element by element to achieve time frame weight enhancement.
[0035] In some embodiments, the multi-channel fusion feature is constructed by the following way: taking the gated enhanced log-mel spectrogram as the first channel; retaining the original log-mel spectrogram as the second channel; supplementing the time domain feature of the original audio as the third channel; the three-channel features are concatenated along the channel dimension to form the final input feature.
[0036] Step S2, time-frequency decoupling feature extraction: the multi-channel fusion features are divided into time domain block sequences and frequency domain block sequences along the time axis and the frequency axis, and are input into parallel time domain branch and frequency domain branch (spectrum-time dual branch Mamba (STMamba) backbone network) respectively; wherein the time domain branch scans the time domain block sequence by the selective state space model (SSM) to capture long-term steady-state features and trend anomalies, and the frequency domain branch scans the frequency domain block sequence by the selective state space model SSM to analyze the cross-band energy distribution and harmonic structure.
[0037] In some embodiments, the processing flow of the time domain branch specifically includes: flattening the time domain block sequence and adding class token and position embedding; sequentially processing by layer normalization, linear transformation, two-dimensional convolution and activation function; dynamically generating selective state space model parameters based on input features; capturing long-range dependencies through the state space modeling layer; adding the output and the original input through residual connection.
[0038] In some embodiments, the processing flow of the frequency domain branch specifically includes: flattening the frequency domain block sequence and adding class token and position embedding; sequentially processing by layer normalization, linear transformation, two-dimensional convolution and activation function; dynamically generating selective state space model parameters based on input features; analyzing frequency domain structure features through the state space modeling layer; adding the output and the original input through residual connection.
[0039] In some embodiments, the selective state space model realizes state transition dependent on input by dynamic parameter calculation, specifically including: generating time step parameters by linear transformation of intermediate features; calculating state transition matrix by time step parameters and preset parameters; generating input projection matrix by linear transformation of intermediate features; performing state space modeling operation by using the dynamically generated state transition matrix and input projection matrix.
[0040] Step S3, cross-modal feature fusion and anomaly scoring: The feature vectors output by the time domain branch and the frequency domain branch are linearly aligned and then added element by element to obtain a fused feature representation; based on the fused feature representation, a self-supervised ID classification task combined with an angular margin loss function is used for model training, and the negative logarithmic probability of the target ID category is used as the anomaly score in the inference stage.
[0041] In some embodiments, the cross-modal feature fusion specifically includes: linearly projecting the time domain branch output features and the frequency domain branch output features respectively to achieve feature space alignment; and performing an element-by-element addition operation on the two aligned feature vectors to generate a final fusion feature.
[0042] In some embodiments, the self-supervised training process specifically includes: constructing a classification task using the device machine ID as the category label; using the angle interval loss function to optimize the model, forcing the features of different machine IDs to maximize the inter-class distance and minimize the intra-class distance in the angle space; the angle interval loss function introduces additive angle interval constraints on feature distribution.
[0043] In some embodiments, the method for calculating the anomaly score specifically includes: inputting the audio to be tested into the trained model; extracting the predicted probability of the corresponding real machine ID category; and calculating the negative natural logarithm of the predicted probability as the anomaly score.
[0044] The main technical advantage of this invention lies in significantly improving the robustness, generalization, and accuracy of abnormal sound detection for industrial equipment through innovative time-frequency decoupling modeling and adaptive feature enhancement. Specifically, by combining the Selective State Space Model (SSM) with time-frequency decoupling, a spectral-time dual-branch Mamba network is designed. The time-domain branch accurately captures the long-term acoustic characteristics and trending anomalies of the equipment's steady-state operation through long-range causal scanning, while the frequency-domain branch deeply analyzes cross-band energy distribution and harmonic structure to identify transient anomalies. This fundamentally overcomes the local receptive field limitations of traditional models and achieves full-granularity analysis of machine acoustic characteristics. Furthermore, the TriStat-Gating mechanism actively enhances input features. Based on third-order statistical metrics of median, root mean square, and variance, it dynamically amplifies time-series regions in the spectrogram that exhibit abnormal stability, intensity, or distribution, resulting in enhanced features (ESTgrams) that carry stronger discriminative information. The synergistic effect of front-end intelligent enhancement and back-end time-frequency decoupling modeling not only improves detection accuracy for diverse equipment and complex operating conditions, but also enhances generalization for unknown anomalies, providing a highly reliable and low-cost anomaly monitoring solution for industrial equipment.
[0045] The following further describes specific embodiments of the present invention and examples of algorithm implementation thereof.
[0046] The present application proposes a device anomaly detection method named "Enhanced Spectrogram-Time Dual-branch Mamba (ESTM)", which performs fine-grained analysis on audio signals during industrial equipment operation through an innovative end-to-end deep learning framework to achieve efficient and accurate anomaly detection. The overall technical solution can be divided into three core stages: first, adaptive feature enhancement and fusion based on multiple statistical indicators; second, deep feature extraction using a time-frequency decoupled dual-branch Mamba network; and finally, model training and anomaly score calculation through a self-supervised classification task and a unique loss function.
[0047] As shown in Figure 2 , which shows the complete link of the audio analysis-based device anomaly detection method of the embodiment of the present application: starting from the input of the original audio signal, the original audio signal is extracted into time-domain features Tgram through a time-spectrum network on one hand, and is transformed into spectrogram features Sgram through a short-time Fourier transform on the other hand, then Sgram is enhanced through a TriStat-Gating (TSG) gating module to obtain enhanced spectrogram features ESgram; then, Tgram, Sgram and ESgram are spliced and fused along the channel dimension to form fused features ESTgram, where T represents the time dimension and S represents the spectrum dimension. After the first part of the fused feature ESTgram generation process, the key input is formed, and then the STMamba network is used for feature extraction through the second part, and finally the classification head is used to complete the anomaly detection.
[0048] I. Adaptive feature generation and fusion based on TriStat-Gating The initial step of the present application is not to directly use traditional acoustic features, but to generate an enhanced multi-channel fused feature, i.e. ESTgram, which is more sensitive to abnormal patterns through an original process. The process starts with the standardization preprocessing of the original monaural audio signal x, through the application of a short-time Fourier transform (STFT) with a window size of 1024 and a hop length of 512, and combined with 128 mel filter banks, to generate the basic two-dimensional log mel spectrogram . However, using only the spectrogram cannot fully highlight all potential abnormal information. To this end, the core of the design of the present invention is the TriStat-Gating (TSG) gating module, the purpose of which is to adaptively enhance the time frames containing key information in the spectrogram. The operating mechanism of the TSG module is that it believes that the abnormal sound of the machine often destroys the stability, intensity consistency or distribution uniformity of the sound characteristics during its normal operation. Therefore, the module calculates three complementary high-order statistical indicators in parallel - Median (Median), Root Mean Square (RMS) and Variance) - to quantify the acoustic characteristics of each time frame from different dimensions. These three statistics are in the transposed spectrogram The above calculations are frame by frame and combined into a gated signal, which is then activated by the Sigmoid function. Generates a gating vector between 0 and 1. This vector is then broadcast and combined with the original spectrogram , this process can automatically identify and increase the weight of time frames where statistical characteristics deviate. Its mathematical expression is as follows:
[0049]
[0050] in is a scaling factor set to 2. It should be noted that the specific values of key hyperparameters such as the scaling factor α and offset μ in the TSG module, as well as the number of blocks in the STMamba network, can be systematically selected using standard optimization methods such as grid search or Bayesian optimization on a standard validation set to maximize the model's anomaly detection performance (e.g., the AUC score on the validation set). Finally, in order to construct an information-complete input, the original log-mel spectrogram is , , and a complementary raw audio feature , (Tgram) is spliced along the channel dimension to form the final three-channel fusion feature ESTgram. This fusion design based on orthogonal statistical indicators ensures the richness and non-redundancy of the input features, providing a solid foundation for the subsequent network to capture subtle time-frequency differences introduced by abnormal patterns.
[0051] 2. Time-Frequency Decoupled Dual-Branch Mamba (STMamba) Backbone Network After the first part of adaptive feature generation and fusion based on TriStat-Gating, the three-channel fusion feature ESTgram containing rich time-frequency information is obtained, and then the feature extraction stage begins.
[0052] After obtaining enhanced ESTgram features, the present invention employs a novel Spectral-Time Dual-Branch Mamba (STMamba) backbone network for deep feature extraction. The core of this network architecture addresses the challenge of applying the advanced one-dimensional sequence model Mamba to two-dimensional time-frequency spectrum analysis. The solution is time-frequency decoupled modeling: rather than treating the time-frequency spectrum as a whole, it splits it into two orthogonal dimensions, time and frequency, and designs a dedicated Mamba processing branch for each dimension. Specifically, the input two-dimensional ESTgram is first patched, splitting it into 12 time-domain patches along the time axis and 16 frequency-domain patches along the frequency axis. The number of blocks mentioned above is mainly determined based on the following principles: the number of time domain blocks is 12, which is chosen to balance computational efficiency and the need to capture the complete operating cycle of the device, ensuring that each block contains sufficient timing information and the total length of the sequence is suitable for modeling long-range dependencies; the number of frequency domain blocks is 16, which is determined based on the total number of mel spectral bands, 128, dividing the total frequency band into 16 sub-bands (each sub-band contains 8 mel frequencies). This can not only preserve the critical frequency band resolution, but also enable the model to effectively learn the correlation between different frequency bands (such as the fundamental frequency and harmonics). These blocks are flattened, linearly projected, and a learnable class token is added. and position embedding , to preserve its global and positional context in the original spectrogram. Subsequently, these two sequences are fed into a parallel two-branch Mamba architecture:
[0053] TMamba (time domain branch) performs causal scanning of the time domain block sequence from left to right, and uses the powerful long-term memory capability of the Selective State Space Model (SSM) to accurately capture the steady-state operation mode, periodic vibration, and trend anomalies in the machine sound that evolve over time. SMamba (spectrogram branch) performs a top-down causal scan of a frequency domain block sequence. Its design aims to accumulate and analyze cross-band energy distribution and harmonic structure, effectively identifying spectra caused by transient anomalies such as impact and friction. The two branches operate in parallel, respectively deeply mining audio features from a global perspective across time and frequency. Finally, the feature vectors output by the two branches are linearly aligned and fused to generate a highly concise and comprehensive final feature representation H.
[0054] The specific calculation process of the dual-branch Mamba backbone network is shown in Table 1. The algorithm embeds two input sequences from the time domain and the frequency domain ( and ) for parallel but weight-sharing processing. In each step of the core loop, the input sequence X first undergoes layer normalization (LayerNorm), linear transformation, and a two-dimensional convolutional layer for preliminary feature mapping. The algorithm then enters the most critical dynamic parameter calculation stage of the Mamba model: a series of data-dependent operations are performed to generate selective parameters , A, B and C. In particular, the parameters and B is obtained by It is calculated after a linear transformation, which enables the state space model (SSM) to dynamically adjust its state transition behavior according to the input content, thereby achieving "selective" attention to key information in the sequence. Using these dynamically calculated parameters, the input sequence is transformed through the core state space modeling (SSM) layer to capture the long-range dependencies within the sequence with extremely high efficiency. Finally, through a residual connection, the output of the SSM module is linearly transformed and added to the original input. This is a technology widely used in deep learning to ensure the smooth flow of information and the stability of the training process. After both branches (time domain branch and frequency domain branch) have independently completed the above processing flow, the algorithm enters the cross-podal fusion stage. The output features obtained by each of the two branches and It first passes through a shared linear alignment layer (Linear align ), whose purpose is to project features from different modalities into the same feature subspace to ensure their semantic comparability. After feature alignment, the two feature vectors are finally fused through simple element-by-element addition, resulting in a more powerful and comprehensive final feature representation H that incorporates both long-term temporal information and global frequency domain information.
[0055] Table 1
[0056] 3. Self-supervised training and anomaly score calculation based on ArcFace loss This paper employs a self-supervised ID classification strategy for end-to-end training of the entire framework, cleverly converting the classification output into anomaly scores. During the training phase, device metadata (i.e., machine type and unique machine ID) is used as readily available labels. The fused features H extracted by the STMamba network are fed into a fully connected layer with a softmax activation function to predict the machine ID to which the input audio belongs. This portion of the network responsible for classification prediction and anomaly score generation is referred to as the classification head. Unlike the cross-entropy loss used in traditional methods, this paper employs the ArcFace loss function. The core advantage of ArcFace lies in its ability to impose an additive angular margin on features in angular space, forcing the model to learn features that have greater separation between different classes (i.e., different machine IDs) while simultaneously making features within the same class more compact. This property is crucial for anomaly detection because it makes the "normal" feature space more clearly and strictly defined, making any samples that deviate from this space easier to identify. The entire network is trained using the AdamW optimizer with a learning rate of 0.0001 for 200 epochs and a batch size of 128. During the inference phase, when a test audio sample is input into the model, its anomaly score is directly defined as the negative log-probability of its corresponding ID category. Because the model has only learned the compact distribution of normal sounds, when encountering an abnormal sound with acoustic characteristics that do not match these, the model cannot confidently classify it into any known normal category. This results in a lower prediction probability, resulting in a higher negative log-probability value, and therefore a higher anomaly score, thereby enabling accurate alerts for abnormal events.
[0057] In summary, this paper proposes a device anomaly monitoring method based on audio signal analysis. Compared to existing technologies, this method achieves significant improvements in both the mechanism and effectiveness of abnormal sound detection. Its fundamental benefit lies in its first integration of the time-frequency decoupling modeling concept with the Selective State Space Model (Mamba), introducing it to the field of abnormal sound detection. Through its designed Spectral-Time Dual-Branch Mamba (STMamba) backbone network, this method completely overcomes the limitations of traditional models, which are limited by a local receptive field. The Time Mamba (TMamba) branch in this architecture performs long-range scanning along the time axis, effectively capturing the long-term acoustic characteristics and trends of the machine during steady-state operation. The parallel Spectral Mamba (SMamba) branch accumulates and analyzes cross-band energy distribution and harmonic structure along the frequency axis. This strategy of modeling two orthogonal dimensions, time and frequency, separately and then fusing them, enables a more refined and comprehensive analysis of machine acoustic characteristics at an unprecedented granularity, overcoming the inherent problem of existing methods that ignore the dynamic coupling of time and frequency by independently processing spectral frames.
[0058] Another great benefit of the present application is that it enhances the sensitivity and robustness of the model to abnormal patterns from the source by innovative feature enhancement mechanism. The inventive TriStat-Gating (TSG) gating module does not passively receive acoustic features, but actively adapts and enhances them. The module constructs a gating signal based on three complementary statistical indicators: median, root mean square and variance, intelligently amplifying time regions in the input spectrogram that show potential abnormalities in stability, intensity or distribution uniformity. This mechanism makes the enhanced features (ESTgram) sent to the backbone network itself carry more discriminative information, making it easier for the model to capture subtle differences between normal and abnormal. This intelligent enhancement at the front end and powerful time-frequency decoupling modeling capability at the back end form a synergistic effect, not only improving the detection accuracy of the model in the face of diverse equipment and complex working conditions, but more importantly, enhancing its generalization ability and stability in the face of unknown abnormal types, providing a more reliable and advanced technical solution for industrial equipment condition monitoring.
[0059] The embodiment of the present application also provides a storage medium for storing a computer program, which is executed to perform at least the method described above.
[0060] The embodiment of the present application also provides a control device, which comprises a processor and a storage medium for storing a computer program; wherein the processor is used to execute the computer program to perform at least the method described above.
[0061] The embodiment of the present application also provides a processor, which executes a computer program to perform at least the method described above.
[0062] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc or a read-only optical disc (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.
[0063] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0064] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0065] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0066] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.
[0067] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0068] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.
[0069] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.
[0070] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0071] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.
Claims
1. A device abnormality monitoring method based on audio signal analysis, characterized in that: The following steps are involved: S1. Adaptive feature enhancement and fusion: The original audio signal is transformed into a logarithmic Mel-spectrogram through time-frequency transformation. The median, root mean square, and variance statistics of each time frame are calculated through an adaptive gating module based on third-order statistical indicators. The gating vector is generated and weightedly fused with the original spectrogram. The supplementary audio features are then spliced together to form a multi-channel fusion feature. S2, time-frequency decoupling feature extraction: the multi-channel fusion features are divided into blocks along the time axis and the frequency axis to form a time domain block sequence and a frequency domain block sequence, which are input into the parallel time domain branch and the frequency domain branch respectively; The time domain branch uses a selective state space model to perform causal scanning on the time domain block sequence to capture long-term steady-state characteristics and trend anomalies, and the frequency domain branch uses a selective state space model to perform causal scanning on the frequency domain block sequence to analyze cross-band energy distribution and harmonic structure. S3. Cross-modal feature fusion and anomaly scoring: The feature vectors output by the time domain branch and the frequency domain branch are linearly aligned and then added element by element to obtain a fused feature representation; Based on the fused feature representation, a self-supervised ID classification task combined with an angular margin loss function is used for model training, and the negative logarithmic probability of the target ID category is used as the anomaly score in the inference stage.
2. The device abnormality monitoring method according to claim 1, characterized in that: The adaptive gating module in step S1 generates a gating vector through the following process: The median, root mean square, and variance statistics are calculated frame by frame on the transposed log-mel spectrogram. The three statistics are summed and then the preset offset is subtracted, adjusted by the scaling factor and activated by the Sigmoid function; The weighted fusion is to multiply the gate vector and the original logarithmic Mel spectrogram element by element to achieve time frame weight enhancement.
3. The device abnormality monitoring method according to claim 1, wherein: The multi-channel fusion features in step S1 are constructed in the following way: The gated enhanced log-mel spectrogram is used as the first channel; Keep the original logarithmic Mel spectrogram as the second channel; Supplement the time domain features of the original audio as the third channel; The three-channel features are concatenated along the channel dimension to form the final input features.
4. The device abnormality monitoring method according to claim 1, wherein: The processing flow of the time domain branch in step S2 specifically includes: Flatten the time domain block sequence and add class token and position embeddings; It is processed sequentially through layer normalization, linear transformation, two-dimensional convolution and activation function; Dynamically generate selective state-space model parameters based on input features; Capturing long-range dependencies through the state-space modeling layer; The output is added to the original input via a residual connection.
5. The device abnormality monitoring method according to claim 1, wherein: The processing flow of the frequency domain branch in step S2 specifically includes: Flatten the frequency domain block sequence and add class token and position embeddings; It is processed sequentially through layer normalization, linear transformation, two-dimensional convolution and activation function; Dynamically generate selective state-space model parameters based on input features; Analyze frequency domain structural characteristics through the state space modeling layer; The output is added to the original input via a residual connection.
6. The device abnormality monitoring method according to claim 1, wherein: In step S2, the selective state space model implements input-dependent state transition through dynamic parameter calculation, specifically including: Generate time step parameters from intermediate features through linear transformation; Calculate the state transfer matrix through the time step parameters and preset parameters; Generate the input projection matrix by linear transformation of the intermediate features; Perform state-space modeling operations using dynamically generated state transition matrices and input projection matrices.
7. The device abnormality monitoring method according to claim 1, wherein: The cross-modal feature fusion in step S3 specifically includes: Linearly project the time domain branch output features and the frequency domain branch output features respectively to achieve feature space alignment; The two aligned feature vectors are added element by element to generate the final fusion feature.
8. The device abnormality monitoring method according to claim 1, wherein: The self-supervised training process in step S3 specifically includes: Construct a classification task using the device ID as the category label; The angular margin loss function is used to optimize the model, forcing the features of different machine IDs to maximize the inter-class distance and minimize the intra-class distance in the angular space; The angular interval loss function introduces additive angular interval constraints on feature distribution.
9. The device abnormality monitoring method according to claim 1, wherein: The anomaly score calculation method in step S3 includes: Input the audio to be tested into the trained model; Extract the predicted probability of the corresponding real machine ID category; The negative natural logarithm of the predicted probability is calculated as the anomaly score.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the device abnormality monitoring method based on audio signal analysis according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Method for detecting abnormal sound of self-supervised machine based on domain transfer
CN115376554A
Machine abnormal sound detection method and device for improving AE by combining Transformer
CN117253505A
Unsupervised machine abnormal sound detection method based on lightweight network
CN117292714A
A Self-Supervised Machine Abnormal Sound Detection Method Based on Dual-Path WaveNet
CN119360884B
A multi-class machine abnormal sound detection method based on diffusion model
CN119626259B
