Abnormal audio visual detection method based on dynamic memory bank
Patent Information
- Application Number
- CN202511031481.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-25
- Publication Date
- 2025-11-07
AI Technical Summary
Existing industrial acoustic detection technologies suffer from black-box characteristics, inability to accurately locate anomalies, and scarcity of abnormal audio samples, resulting in a high false alarm rate and difficulty in effectively detecting the sub-healthy operating state of equipment.
An abnormal audio visualization detection method based on a dynamic memory library is adopted. The audio data features are extracted using an AST pre-trained model, and a core feature library is constructed by combining the k-center greedy algorithm. A heat map is generated by standardizing the score map for anomaly localization. The cache channel and EMA technology are introduced to adaptively update the memory library to reduce false alarms.
It enables precise and visual localization of audio anomalies, reduces false alarm rates, improves detection efficiency, provides real-time health status monitoring, and ensures reliable equipment operation.
Smart Images
Figure CN120913591A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of artificial intelligence and signal processing, and particularly relates to an abnormal audio visual detection method based on a dynamic memory library. BACKGROUND
[0002] In modern industrial production scenarios, key machine equipment is usually in a high-intensity continuous running state for a long time. The core transmission components thereof are inevitably subjected to material fatigue degradation and microstructure damage under the coupling action of cyclic mechanical friction and dynamic impact load. Such progressive aging does not directly cause the equipment to fail to operate normally, but can cause the vibration modal migration and acoustic fingerprint feature shift of the equipment, forming a sub-healthy operation state with concealment. Due to the perceptual delay of the human ear to the nonlinear attenuation of sound waves, combined with the noise interference existing in the industrial field, it is difficult for technical personnel to distinguish the fault state of the machine and the sub-healthy operation state of the equipment through traditional auscultation, causing false positives.
[0003] The current mainstream industrial acoustic detection technology is generally based on an end-to-end deep learning architecture, and typical representatives include a deep autoencoder (AutoEncoder) and a convolutional neural network (CNN). Although this kind of method has strong feature extraction capability, it has two fundamental defects: first, the decision logic of the deep learning model has a significant black box characteristic and lacks explainability; and second, it cannot effectively realize precise positioning of abnormalities and cannot further help engineers to repair industrial equipment.
[0004] From the perspective of technical development, the abnormal audio detection method based on deep learning can be mainly divided into two paradigms of supervised learning and unsupervised learning. In an ideal state, when the abnormal audio sample library is complete enough, the supervised learning method exhibits better detection performance. However, the particularity of the industrial scene makes the scarcity of high-quality abnormal audio samples a normal state. To break through the data bottleneck, some researchers attempt to use a generative adversarial network (GAN) and a diffusion model (Diffusion Model) to generate abnormal samples. However, these models have certain application limitations: on the one hand, there is a risk of collapse caused by unstable model training, and on the other hand, the quality and diversity of the generated samples are difficult to meet the actual needs. SUMMARY
[0005] To solve the above technical problems, the application provides an abnormal audio visual detection method based on a dynamic memory library, which migrates the visualization technology in visual anomaly detection to audio anomaly detection, aiming to solve the two major defects of mainstream acoustic detection.
[0006] To achieve the above purpose, the application provides an abnormal audio visual detection method based on a dynamic memory library, which includes:
[0007] Collecting an audio data set, randomly dividing the audio data to obtain a first audio data set and a second audio data set;
[0008] Using an AST pre-training model to extract local features of the first audio data set and the second audio data set respectively to obtain first local features and second local features;
[0009] Based on the first local features, a core feature library is obtained;
[0010] Based on the core feature library and the second local features, a standardization score is obtained;
[0011] Based on the standardization score, an abnormality judgment is made to obtain a judgment result;
[0012] According to the judgment result, abnormal audio detection is completed.
[0013] Optionally, the first audio data set includes normal log-mel spectrograms, and the second audio data set includes normal log-mel spectrograms and log-mel spectrograms of multiple different abnormal types.
[0014] Optionally, before using the AST pre-training model to extract local features of the first audio data set and the second audio data set respectively, it further includes:
[0015] The audio data set is divided into a plurality of local feature blocks, the local feature blocks are mapped into corresponding embedding vectors by linear projection, and position coding is added to the embedding vectors to obtain time domain feature blocks containing position information.
[0016] Optionally, using the AST pre-training model to extract local features of the first audio data set and the second audio data set respectively includes:
[0017] The time domain feature blocks containing position information are reshaped according to the original grid position to obtain a reshaped feature map;
[0018] Splicing the pre-selected remolding feature map, and performing neighborhood aggregation on the spliced remolding feature map to obtain a neighborhood feature set;
[0019] Performing a maximum aggregation operation on the neighborhood feature set to obtain a local feature.
[0020] Optionally, based on the first local feature, obtaining a core feature library includes:
[0021] Based on the first local feature, constructing a feature memory library;
[0022] Compressing the feature memory library using a k-center greedy algorithm to obtain a core feature library.
[0023] Optionally, based on the core feature library and the second local feature, obtaining a standardized score includes:
[0024] Calculating the sum of distances from each second local feature to the updated core feature library, and performing standardized score processing on the sum of distances to obtain the standardized score.
[0025] Optionally, based on the standardized score, performing anomaly judgment to obtain a judgment result includes:
[0026] Based on the standardized score, obtaining a standardized score set;
[0027] Processing the standardized score set to obtain a processed score map;
[0028] Based on the processed score map, determining whether the audio data set is abnormal to obtain a judgment result.
[0029] Optionally, according to the judgment result, completing abnormal audio detection includes:
[0030] When the judgment result is abnormal, performing abnormal visualization;
[0031] When the judgment result is not abnormal, temporarily storing the standardized score and the second local feature in a cache channel, wherein the cache channel includes a storage window and a score window.
[0032] Optionally, when the judgment result is not abnormal, it further includes:
[0033] Calculating the current mean μ of the standardized score in the score window win And the current variance σ win ;
[0034] Based on the current mean μ win And the current variance σ win , combined with the EMA mechanism, the historical mean μ is maintained by exponential moving averagehistory and historical variance σ history , when the current mean satisfies μ win > μ history + tσ history , then the core feature library needs to be updated.
[0035] Optionally, updating the core feature library further comprises:
[0036] using a k-center greedy algorithm to screen the second local features in the storage window, obtaining optimal features, and updating the core feature library.
[0037] Compared with the prior art, the present application has the following advantages and technical effects:
[0038] 1. The present application can break through the detection black box bottleneck and realize abnormal visualization positioning: the heat map positioning technology in the field of visual anomaly detection is innovatively migrated to the field of audio, a superimposed heat map is generated through a standardized score map, and a pixel-level abnormal area where an anomaly occurs is intuitively identified in an audio data set. Compared with traditional CNN and other black box models, this method breaks through the difficulty of accurately positioning faults in industrial scenes.
[0039] 2. The present application can adaptively reduce the false alarm rate of device progressive aging: the traditional method relies on historical data to construct a static feature distribution model. When the device is in the progressive aging stage after long-term operation, the feature distribution of normal audio will produce a nonlinear shift, causing the false alarm rate of the detection system to increase significantly. Under the synergistic action of the buffer channel, the distribution drift detection mechanism and the EMA technology, the present application can capture the nonlinear change of normal audio features, calibrate the distribution benchmark of the core memory library in a closed-loop feedback mode, effectively suppress the misjudgment caused by device aging, and significantly improve the detection robustness of the system in the whole life cycle of the device.
[0040] 3. The present application can greatly improve the efficiency of technicians in detecting anomalies, and also realize real-time detection and safety monitoring of the health status of the device, thereby providing key technical support for the continuous and reliable operation of the core device. BRIEF DESCRIPTION OF DRAWINGS
[0041] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and their description are used to explain the application and are not intended to limit the application. In the drawings:
[0042] Figure 1 is a flow chart of an abnormal audio visualization detection method based on a dynamic memory library according to an embodiment of the present application;
[0043] Figure 2 is a flow chart of audio data preprocessing according to an embodiment of the present application;
[0044] Figure 3 is a flowchart of constructing a feature library by an embodiment of the present application. DETAILED DESCRIPTION
[0045] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0046] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.
[0047] The present embodiment proposes an abnormal audio visualization detection method based on a dynamic memory library, as shown in Figure 1 The method specifically comprises the following steps:
[0048] Collecting an audio data set, randomly dividing the audio data to obtain a first audio data set and a second audio data set;
[0049] Using an AST pre-training model to extract local features of the first audio data set and the second audio data set respectively to obtain first local features and second local features;
[0050] Based on the first local features, a core feature library is obtained;
[0051] Based on the core feature library and the second local features, a standardization score is obtained;
[0052] Based on the standardization score, an abnormality is judged to obtain a judgment result;
[0053] According to the judgment result, an abnormal audio detection is completed.
[0054] Specifically, the present embodiment includes an offline stage and an online stage. In the offline stage, a pre-training model AudioSpectrogram Transformer is used to extract local features of a log mel spectrogram, and then a k-center greedy algorithm is used to construct and dynamically update a core memory library. In the online stage, the standardization score of the features and the core memory library is calculated to visualize the abnormal area and make an abnormality judgment. At the same time, a cache channel, a distribution drift detection mechanism and an EMA technology are introduced to solve the false alarm problem caused by device aging.
[0055] More specifically, in the online phase, the same feature extraction method as in the offline phase is used to extract local features from the test log-Mel spectrogram. Then, standardized scores are calculated with the core feature library. Anomaly detection is performed based on the standardized scores. For log-Mel spectrograms with abnormal results, anomaly visualization is performed using heatmaps. For log-Mel spectrograms with normal results, the features are placed in a cache channel. Feature distribution statistics are calculated for the local features in the cache channel. Distribution drift detection is performed based on the statistics. The core feature library is dynamically updated.
[0056] Furthermore, the first audio dataset includes normal log-Mel spectrograms, and the second audio dataset includes normal log-Mel spectrograms and log-Mel spectrograms with various different types of anomalies.
[0057] Specifically, firstly, a fixed sampling rate f is used. s (Hz), collecting continuous raw audio signals x(t) from industrial machines operating under high-intensity conditions;
[0058] The continuous raw audio signal x(t) is divided into M 10-second audio segments {x1(t), x2(t), ..., x...}. M (t)}, where the frame length of each segment is T frame It's 50 milliseconds, and the frame shift T for each segment is... hop It is 25 milliseconds, meaning the total number of frames is
[0059] Generate a log-Mel spectrogram for each audio segment in, The log-Melbourne spectrogram scale is represented by T, which represents the number of frames in the log-Melbourne spectrogram. The log-Melbourne spectrogram is also divided into a training set D. train and test set D test The training set contains only normal log-Mel spectrograms (since industrial machines do not initially exhibit aging or failure issues), while the test set contains normal log-Mel spectrograms and log-Mel spectrograms with various abnormal types.
[0060] Furthermore, before extracting local features from the first and second audio datasets using the AST pre-trained model, the process includes:
[0061] The audio dataset is segmented into several local feature blocks. Linear projection is used to map the local feature blocks into corresponding embedding vectors. Position encoding is added to the embedding vectors to obtain temporal feature blocks containing position information.
[0062] Specifically, such as Figure 2 As shown, for any log-Mel spectrogram x in the training and test sets i Extracting x using a pre-trained AST modeli Features;
[0063] The input log-Mel spectrum Divide into N local feature blocks P i,j The size of each block is p t ×p f ,but Where F represents the frequency of the log-Melb spectrum, p t It's the height, p f The width is (i,j), and (i,j) represents the grid position in the input log-Mel spectrogram corresponding to the position of each local feature block.
[0064] Using linear projection to transform local feature block P i,j Mapping to embedding vector That is, E i,j =W·Vec(P i,j )+b, where Vec(·) represents the paving operation. Let b be the projection weight matrix, where b ∈ R. D For bias terms;
[0065] Add positional encoding to each embedding vector Forming a time-frequency block feature representation containing location information The final input embedding aims to preserve time-frequency location information, which is represented as follows:
[0066] Furthermore, the local features extracted from the first and second audio datasets using the AST pre-trained model include:
[0067] The temporal feature blocks containing location information are reshaped according to their original grid positions to obtain the reshaped feature map;
[0068] The pre-selected reshaped feature maps are stitched together, and the stitched reshaped feature maps are then aggregated into neighborhoods to obtain a set of neighborhood features.
[0069] Maximize the aggregation operation on the neighborhood feature set to obtain local features.
[0070] Specifically, such as Figure 3 As shown, the pre-trained AST model contains L layers of Transformer encoders, each layer consisting of a multi-head sub-attention MSA and a feedforward network FNN. The embedding representation for the l-th layer is as follows:
[0071] Z (l) =LayerNorm(MSA(Z) l-1 )+Z (l-1) Z (l) =LayerNorm(FFN(Z)l )+Z (l) );
[0072] Since the output features of the high layer are too abstract, the spatial and temporal information is easily lost, and the output features of the bottom layer include a lot of noise, which easily affects the detection performance, therefore, the output features of the middle two encoding blocks of the pre-trained AST model Encoder are selected, and the output feature of the nth encoding block of the input log mel spectrogram x i is expressed as
[0073] The output features are reshaped according to the original grid position to obtain the reorganized feature map Where f reshape represents the transformation function of reshaping the feature shape;
[0074] The reshaped feature maps of the middle two encoding blocks are fused and spliced along the channel dimension to obtain the fused feature map Where f concat represents the aggregation function, and n' represents the nth' encoding block of the middle layer of the pre-trained model AST Transformer Encoder;
[0075] In order to enhance the perception ability of features to context information, reduce the occupation of memory and inference time, the neighborhood aggregation operation needs to be performed on the spliced features;
[0076] Suppose the neighborhood range of each local feature block (i,j) is defined as Where r is the neighborhood radius;
[0077] For the fused feature map F i Extract the features of all blocks in the neighborhood to form a neighborhood feature set
[0078] Perform maximum pooling aggregation operation on the neighborhood feature set to highlight the significant abnormal features, and the specific operation is
[0079] Finally, let the step size s of the sliding window be 1 to traverse all positions to obtain the aggregated feature set, which is specifically
[0080] Further, based on the first local feature, the core feature library includes:
[0081] Based on the first local feature, a feature memory library is constructed;
[0082] The k-center greedy algorithm is used to compress the feature memory library to obtain the core feature library.
[0083] Specifically, for all normal log-mel spectrograms in the training set, local features are extracted, and then a feature memory library is constructed, specifically
[0084] In order to further reduce the memory occupation and improve the subsequent detection efficiency, the k-center greedy algorithm is used to compress the memory library S to form the core memory library S c , and the size is K c .
[0085] More specifically, when the core memory library S c is constructed or dynamically updated, the average value and the standard deviation of the sum of distances between all feature points are calculated. Specifically: Where d i is the sum of distances from a feature point in the core memory library to the core memory library, which is used for subsequent dynamic updating of the memory library and anomaly detection.
[0086] Further, based on the core feature library and the second local feature, the standardized score includes:
[0087] The sum of distances from each second local feature to the updated core feature library is calculated, and the sum of distances is standardized score processed to obtain the standardized score.
[0088] Specifically, after extracting the local feature x test , set each local fusion feature x i of x , and calculate the sum of distances d c from x i to the core memory library S Where d is the Euclidean distance function, in order to eliminate the dimensional difference and unify the scale, the standardized score s i is processed, specifically
[0089] Further, based on the standardized score, the judgment result includes:
[0090] Based on the standardized score, a standardized score set is obtained;
[0091] The standardized score set is processed to obtain a processed score graph;
[0092] Based on the processed score graph, it is judged whether the audio data set is abnormal, and the judgment result is obtained.
[0093] Specifically, all local feature standardized scores in x test are recorded as a set A(x test )={s i};
[0094] The set of normalized scores A(x test ) is constructed into a low-resolution score map P low according to the position of (i, j), and then up-sampled to the size of the original map by a bilinear interpolation technique, followed by a Gaussian filtering process, and finally normalized to map the normalized scores in the range [0, 1] to obtain the normalized score map P nor ;
[0095] Finally, the maximum value of P nor is taken as the global anomaly score of x test , and then the threshold is judged to obtain the abnormal result.
[0096] Further, according to the judgment result, the abnormal audio detection is completed, including:
[0097] When the judgment result is abnormal, abnormal visualization is performed;
[0098] When the judgment result is normal, the normalized scores and the second local feature are temporarily stored in a cache channel, wherein the cache channel includes a storage window and a score window.
[0099] Specifically, for the log mel spectrum graph with an abnormal result, a heat map P nor is generated according to P heat , and then P heat is set to a certain transparency and superimposed with the original graph, so that the abnormal area can be directly displayed; for the log mel spectrum graph with a normal result, all local feature normalized scores and all local features are temporarily stored in the cache channel for dynamic updating of the core memory library S c .
[0100] More specifically, when detecting in the online phase, the spatial distribution between the new local feature and the core memory library is monitored in real time, and the statistics between the two are calculated, and when a certain condition is met, it indicates that the normal audio signal has produced a nonlinear progressive change, and the core memory library S c needs to be adaptively updated, otherwise, no change is needed.
[0101] Further, when the judgment result is normal, it further includes:
[0102] The current mean μ win and the current variance σ win of the normalized scores in the score window are calculated;
[0103] Based on the current mean μ win and the current variance σ win, combined with the EMA mechanism, the historical mean μ is maintained by exponential moving average history and the historical variance σ history , when the current mean satisfies μ win > μ history + tσ history , then the core feature library needs to be updated.
[0104] Further, updating the core feature library further includes:
[0105] The second local feature in the storage window is screened by using the k-center greedy algorithm to obtain the best feature, and the core feature library is updated.
[0106] Specifically, for the cache channel, it is composed of a reserve window and a score window :
[0107] The reserve window is a FIFO-based window, and its size is twice the size of the core memory library S c (that is, 2K c ), and each local feature in the window is denoted as α;
[0108] Similarly, the score window is also a FIFO-based window, and its size is four times the size of the core memory library S c (that is, 4K c ), which is used to maintain the standardized score s i , and the mean μ win and variance σ win of all standardized scores in the window, which is used to detect distribution drift.
[0109] More specifically, the distribution drift detection of the present application includes the following contents:
[0110] In order to further improve the efficiency and reduce the use of memory, the incremental updating method is used to calculate the mean μ win and variance σ win , specifically wherein, denotes the newly added standardized score, denotes the removed standardized score, and S aq is the incremental sum of squares.
[0111] In order to better capture the long-term distribution of features, combined with the EMA mechanism, the historical mean μ history and the historical variance σ history are maintained by exponential moving average, specifically μ history = λ·μ history+ (1 - λ) · μ win , σ history = λ · σ history + (1 - λ) · | μ win - μ history |, where λ ∈ (0, 1);
[0112] When the condition of k consecutive μ win > μ history + tσ history is met, it indicates that the normal audio signal has produced a nonlinear progressive change, and the core memory library S c needs to be updated to avoid false positives caused by the system. Wherein k is maintained by a continuous step counter, and t is a coefficient value.
[0113] More specifically, since the number of local features stored in the storage window is twice that of the core memory library S c , and the spatial distribution of the former is superior to that of the latter, the k-center greedy algorithm is directly performed on the storage window to select K c best features as the new generation of core memory library S c , and the average value μ s and the standard deviation σ s of the sum of distances between all feature points are recalculated.
[0114] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A method for detecting abnormal audio visualizations based on dynamic memory library, characterized in that, The method comprises the following steps: Collecting an audio data set, randomly dividing the audio data to obtain a first audio data set and a second audio data set; Extracting local features of the first audio data set and the second audio data set by using an AST pre-training model to obtain first local features and second local features; Based on the first local features, a core feature library is obtained; Based on the core feature library and the second local features, a standardization score is obtained; Based on the standardization score, an abnormality judgment is made to obtain a judgment result; According to the judgment result, abnormal audio detection is completed.
2. The method of claim 1, wherein the method is based on a dynamic memory library for detecting abnormal audio visualization. The first audio data set includes normal log-mel spectrograms, and the second audio data set includes normal log-mel spectrograms and log-mel spectrograms of multiple different abnormal types.
3. The method of claim 1, wherein the method is based on a dynamic memory library for detecting abnormal audio visualization. Before extracting the local features of the first audio data set and the second audio data set by using the AST pre-training model, the method further comprises the following steps: Divide the audio data set into a plurality of local feature blocks, map the local feature blocks into corresponding embedding vectors by linear projection, add position coding to the embedding vectors, and obtain time domain feature blocks containing position information.
4. The method of claim 3, wherein the method is based on a dynamic memory library for detecting abnormal audio visualization. Extracting local features of the first audio data set and the second audio data set by using the AST pre-training model comprises the following steps: Reshape the time domain feature blocks containing position information according to the original grid position to obtain a reshaped feature map; Splice the preselected reshaped feature map, and perform neighborhood aggregation on the spliced reshaped feature map to obtain a neighborhood feature set; Perform maximum aggregation operation on the neighborhood feature set to obtain local features.
5. The method of claim 1, wherein the method is based on a dynamic library of abnormal audio visualizations. Based on the first local features, a core feature library is obtained, which comprises the following steps: Based on the first local features, a feature memory library is constructed; Compress the feature memory library by using a k-center greedy algorithm to obtain a core feature library.
6. The method of claim 1, wherein the method is based on a dynamic library of abnormal audio visualizations. Based on the core feature library and the second local features, a standardization score is obtained, which comprises the following steps: Calculate the sum of distances from each second local feature to the updated core feature library, and perform standardization score processing on the sum of distances to obtain the standardization score.
7. The method of claim 1, wherein the method is based on a dynamic library of abnormal audio visualizations. Based on the standardization score, an abnormality judgment is made to obtain a judgment result, which comprises the following steps: Based on the standardization score, a standardization score set is obtained; Process the standardization score set to obtain a processed score graph; Based on the processed score graph, it is judged whether the audio data set is abnormal to obtain a judgment result.
8. The method of claim 2, wherein the method is based on a dynamic memory library for detecting abnormal audio visualization. According to the judgment result, abnormal audio detection is completed, which comprises the following steps: When the judgment result is abnormal, abnormal visualization is performed; When the judgment result is not abnormal, the standardization score and the second local features are temporarily stored in a cache channel, wherein the cache channel comprises a storage window and a score window.
9. The method of claim 8, wherein the dynamic library is based on a library of abnormal audio visualizations. When the judgment result is not abnormal, the core feature library is updated, which comprises the following steps: calculating a current mean μ of the normalized scores in the score window win and a current variance σ win ; Based on the current mean μ win and current variance σ win , combined with the EMA mechanism, the historical mean μ history and historical variance σ history are maintained by exponential moving average. win When the current mean satisfies the condition μ history > μ history + tσ , it is necessary to update the core feature library.
10. The method of claim 9, wherein the method further comprises: Filter the second local features in the storage window by using a k-center greedy algorithm to obtain the best features, and update the core feature library.