Sound anomaly detection method based on prototype network

Abstract

The invention relates to a sound anomaly detection method based on a prototype network, and relates to the technical field of sound anomaly detection. The method comprises the following steps: constructing a feature extractor, and obtaining the feature extractor which can capture difference features of two types of audios and is fixed in structure through supervised training based on normal and abnormal audios; a prototype system is constructed, a prototype set and a prototype network classifier are constructed based on the feature extractor, and normal and abnormal audio prototypes are obtained through calculation; performing anomaly detection judgment, and completing normal and abnormal classification of the to-be-predicted audio through feature extraction and similarity comparison; according to the method, optimization and adaptation are achieved, the prototype set is updated and iterated based on the evaluation set, representative samples of newly-added audio types can be directly added into the prototype set, and adaptation can be achieved without retraining the model. According to the invention, through a prototype network architecture, the detection precision can still be ensured under the condition of a small number of newly added abnormal samples, and the method has relatively high practicability and robustness.

Description

Technical Field

[0001] This invention belongs to the field of sound anomaly detection technology, specifically relating to a sound anomaly detection method based on a prototype network. Background Technology

[0002] Sound anomaly detection technology has a wide range of applications in industrial equipment operation and maintenance, home appliance quality inspection, automotive parts fault diagnosis and electronic equipment reliability testing. Its core purpose is to accurately determine whether there are faults or defects in products or equipment by identifying the sound signals generated during the operation of the product or equipment, thereby realizing early detection and warning of faults, ensuring the operational safety and reliability of products and equipment, and reducing operation and maintenance costs.

[0003] Currently, existing sound anomaly detection technologies can be mainly divided into five categories: rule-based detection methods, feature-based detection methods combined with thresholds, feature-based detection methods combined with statistical machine learning, feature-based detection methods combined with deep learning, and original audio-based detection methods combined with deep learning.

[0004] Among the various detection solutions mentioned above, deep learning-based detection methods are more flexible and can automatically learn the feature patterns of both normal and abnormal audio samples. To a certain extent, they can be generalized and adapted to the sound anomaly detection needs of different product models and under different production conditions, making them a widely used technical solution at present.

[0005] However, existing deep learning-based audio anomaly detection methods still have significant technical shortcomings: for abnormal audio types that have never appeared in the training samples in the application scenario, the classification accuracy of these methods will drop significantly, making it difficult to effectively learn the core features of the newly added abnormal audio and thus failing to achieve accurate detection.

[0006] To address the aforementioned shortcomings, a feasible solution in existing technologies is to collect a certain number of newly emerging anomalous audio samples, add them to the training dataset, and then retrain the deep learning model to detect this new anomaly type. However, this solution has significant limitations. It typically requires collecting a large number of newly emerging anomalous audio samples, while in real-world applications, it is often difficult to collect a sufficient number of samples for newly emerging anomalies in a short period of time, failing to meet the data requirements for model retraining.

[0007] How to accurately identify the new anomaly type when only a small number of new abnormal audio samples are available has become a complex technical challenge to be solved in this field. Although adding a small number of new abnormal audio samples to the training set and retraining the model is still an option, the model cannot fully learn the core features of the new anomaly type due to insufficient sample size, and usually cannot achieve ideal detection results, making it difficult to meet the detection accuracy requirements in practical applications.

[0008] Therefore, there is a need for an efficient sound anomaly detection method to solve the technical problems of low detection accuracy and poor adaptability when the number of newly added anomaly samples is insufficient in the existing technology. The sound anomaly detection method based on prototype network proposed in this patent has been verified by actual implementation. Its detection effect is better than the detection method of adding a small number of newly added anomaly samples to the training set to retrain the model, which can make up for the shortcomings of the existing technology. Summary of the Invention

[0009] To address the aforementioned problems in the existing technology, this invention provides a sound anomaly detection method based on a prototype network. The objective of this invention can be achieved through the following technical solutions: A method for detecting sound anomalies based on prototype networks includes: S1: Construct a feature extractor composed of deep neural networks based on normal and abnormal audio. Through supervised training, the feature extractor captures the common features of normal audio and the difference features between normal and abnormal audio. S2: Based on the feature extractor, construct a prototype set in combination with the application scenario; perform feature extraction and prototype calculation on the audio in the prototype set through the feature extractor to generate normal audio prototypes and abnormal audio prototypes; S3: Based on the normal audio prototype and the abnormal audio prototype, perform feature extraction and similarity comparison on the audio sample to be predicted, and generate a classification judgment result of normal and abnormal. S4: Construct evaluation criteria based on the evaluation set consisting of normal and abnormal audio, and update and iterate the prototype set.

[0010] Specifically, the feature extractor is a backbone network, which is formed by removing the classification head from the original network; The original network includes a deep neural network and a classification head. The deep neural network is used to extract deep features of the audio, and the classification head is used to classify and output the extracted features. The original network distinguishes between normal audio and abnormal audio through supervised training, removes the classification head, and retains the deep neural network part as the feature extractor.

[0011] Specifically, the training process of the original network includes: Based on normal audio and various abnormal audio in the application scenario, a training dataset is generated. Each audio segment in the training dataset is labeled with a corresponding category label, generating normal audio labels and various abnormal audio labels. The original audio and corresponding labels are input into the original network. Supervised training is adopted, with classification loss as the optimization objective. The parameters of the deep neural network in the original network are adjusted through iterative training so that the error between the classification result output by the deep neural network and the labeled label reaches the preset requirement.

[0012] Specifically, during supervised training of the original network, data augmentation processing is required on the input raw audio. The specific process includes: The original audio and corresponding tags are simultaneously augmented in the time domain. The augmented audio samples are then converted into Mel spectrograms. The time domain features of the audio samples are converted into frequency domain features. The generated Mel spectrograms are then augmented in the frequency domain.

[0013] Specifically, the time-domain data enhancement used for the original audio includes: time stretching, pitch shifting, and noise superposition. Time stretching is used to adjust the playback speed without changing the pitch, pitch shifting is used to adjust the pitch without changing the playback speed, and noise superposition is used to add low-intensity background noise to the original audio. The frequency domain data enhancement is used to enhance the frequency domain features of the Mel spectrogram, specifically including translation, flipping, cropping, and adversarial enhancement. Translation is used to shift the Mel spectrogram horizontally or vertically, flipping is used to flip the Mel spectrogram horizontally or vertically, cropping is used to extract the effective feature regions in the Mel spectrogram, and adversarial enhancement is used to improve the network's sensitivity to identifying abnormal features by generating adversarial examples.

[0014] Specifically, the prototype set is used to characterize the core features of normal and abnormal audio, providing a judgment benchmark for the prototype network classifier. The construction process is based on the application scenario and is specifically composed of audio corresponding to each prototype category. The prototype category represents the same type of audio. Each prototype category corresponds to a set of audio samples. The audio samples need to be screened to remove abnormal and interfering audio.

[0015] Specifically, when constructing the prototype set, two construction methods are selected based on the detection accuracy requirements of the application scenario, including: Distinguish between fine-grained prototype categories, further subdivide them according to audio type, and clarify whether each fine-grained prototype category corresponds to a normal audio prototype or an abnormal audio prototype; Instead of distinguishing between fine-grained prototype categories, a coarse-grained classification method is adopted, grouping all normal audio into one category as normal audio prototypes and all abnormal audio into another category as abnormal audio prototypes.

[0016] Specifically, when constructing the prototype set, each prototype category can configure the inclusion strategy of the original training data according to the training effect and detection accuracy requirements: for each prototype category, the corresponding category training data used for the original network supervised training is included in the prototype set, and samples with typical features need to be selected. The selected training data and the newly collected representative samples of the corresponding category together constitute the prototype set; if the training data is not included, then the representative samples actually obtained in the application scenario and verified by the features are selected as the prototype set.

[0017] The aforementioned features typically refer to samples whose core features highly match the preset feature thresholds of the corresponding prototype category, have no redundant interference or missing features, and whose feature vectors are in a preset high similarity range with the feature vectors of most samples in that category, thus accurately representing the essential features of the category; the aforementioned representative audio samples refer to samples that can cover the feature distribution of different scenarios of the corresponding prototype category, reflect the common core features of the category and reasonable feature variations, and whose feature signal-to-noise ratio meets preset requirements after noise suppression, and can provide stable feature input.

[0018] Specifically, the prototype calculation process includes: Based on the feature extractor, the parameters of the feature extractor are fixed to be consistent with the parameters of the original trained backbone network. The audio samples in the prototype set are input one by one into the feature extractor. The audio samples are first converted into Mel spectrograms, and then the deep features in the Mel spectrograms are extracted by the feature extractor to output the feature vector corresponding to each audio sample. For the same prototype category, obtain the feature vectors of all audio samples under the prototype category, calculate the mean of all feature vectors, and generate the average feature vector as the prototype of the prototype category.

[0019] Specifically, the specific process for determining the classification result includes: The audio samples to be predicted are acquired and preprocessed to remove noise from the audio. The preprocessed audio samples are converted into Mel spectrograms, which are consistent with the spectrogram format in the training and prototype calculation processes. The generated samples are expanded to cover a wider range of samples through data augmentation. Calculate the similarity between each generated sample and the prototype of the corresponding prototype category. The result of the similarity calculation for all generated samples is used as the final category of the audio sample to be predicted.

[0020] Specifically, the process of constructing evaluation criteria for the evaluation set includes: An evaluation set consisting of normal and abnormal audio is used, which must include all types of normal and abnormal audio that appear in the application scenario and must not overlap with the training dataset. Evaluation criteria are constructed, including classification accuracy, anomaly recognition rate, and false positive rate. By setting prototype categories of different granularities, classification evaluation indicators on the evaluation set at each granularity are calculated, and the detection effects at different granularities are compared.

[0021] Specifically, the process of updating and iterating the prototype set includes: Based on the evaluation results, the proportion of each category in the training samples is adjusted, samples of categories with missing features are added, and the division method of prototype categories is optimized by merging and splitting to complete the update and iteration of the prototype set.

[0022] The beneficial effects of this invention are as follows: The feature extraction network is trained on a foundation of sufficient data containing both normal audio data and known abnormal audio types. Through supervised learning, it can effectively learn the distinguishing features between normal and abnormal audio. Furthermore, by combining temporal and frequency domain data augmentation techniques, the generalization ability of the feature extraction network is improved. In addition, the factory can directly use existing historical data for training. Even if the historical data is different from the normal and abnormal audio types to be classified, usable feature extraction network parameters can still be obtained without the need to collect a large amount of additional adaptation data, thus reducing data collection costs.

[0023] For normal and abnormal audio to be classified, this method only requires a small amount of data to support the classification detection of each category, without the need to retrain the feature extraction network, which shortens the model computation time and improves the detection efficiency. At the same time, a small number of samples can complete the calculation of prototype features, which compresses the sample size required for detection and solves the problem of poor detection results caused by insufficient sample size in the existing technology.

[0024] For newly added normal or abnormal audio types, there is no need to retrain the feature extraction network and prototype network classifier. Only a small amount of data needs to be collected to form a prototype of the type, which is then added to the original prototype set to achieve the detection of the new type. The operation is convenient and allows users to add adaptations themselves, reducing the adaptation cost and operation difficulty of new types.

[0025] During the prototype set construction process, users can independently set whether to add training data to the prototype set. At the same time, the granularity of the prototype can be flexibly adjusted according to the physical meaning of the actual application scenario and the effect of the detection evaluation index. It has sufficient setting freedom and can adapt to sound anomaly detection scenarios with different fields and different accuracy requirements, further improving the practicality and flexibility of the method. Attached Figure Description

[0026] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0027] Figure 1 This is an overall architecture diagram of a sound anomaly detection method based on a prototype network according to the present invention; Figure 2 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0028] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0029] Please see Figures 1-2 A method for detecting sound anomalies based on prototype networks, comprising: S1: Construct a feature extractor composed of deep neural networks based on normal and abnormal audio. Through supervised training, the feature extractor captures the common features of normal audio and the difference features between normal and abnormal audio. S2: Based on the feature extractor, construct a prototype set in combination with the application scenario; perform feature extraction and prototype calculation on the audio in the prototype set through the feature extractor to generate normal audio prototypes and abnormal audio prototypes; S3: Based on the normal audio prototype and the abnormal audio prototype, perform feature extraction and similarity comparison on the audio sample to be predicted, and generate a classification judgment result of normal and abnormal. S4: Construct evaluation criteria based on the evaluation set consisting of normal and abnormal audio, and update and iterate the prototype set.

[0030] Specifically, the feature extractor is a backbone network, which is formed by removing the classification head from the original network. The original network includes two parts: a deep neural network and a classification head. The deep neural network is used to extract deep features of the audio, and the classification head is used to classify and output the extracted features. The original network first obtains the ability to distinguish between normal and abnormal audio through supervised training, and then the classification head is removed, retaining the deep neural network part as the feature extractor. This provides a stable feature extraction foundation for subsequent prototype calculation and anomaly detection. Moreover, the structure of this backbone network is fixed and does not change with the type of abnormal audio.

[0031] Specifically, the training process of the original network includes: first, collecting normal audio and various abnormal audio in the application scenario, organizing them into a training dataset, labeling each audio segment in the dataset with a corresponding category label, and clearly distinguishing between normal audio labels and various abnormal audio labels; then, inputting the original audio and corresponding labels into the original network, using supervised training, with classification loss as the optimization objective, and adjusting the parameters of the deep neural network in the original network through iterative training until the error between the network output classification result and the labeled label reaches the preset requirement, so that the original network has the ability to stably distinguish between normal audio and abnormal audio features, providing a guarantee for the feature extraction performance of the subsequent backbone network.

[0032] Specifically, during supervised training of the original network, data augmentation processing is required on the input raw audio. The specific process includes: First, temporal data augmentation is performed simultaneously on the original audio and its corresponding labels to ensure that the labels of the augmented audio remain consistent with those of the original audio. Then, the temporally augmented audio samples are converted into Mel spectrograms, transforming the temporal features of the audio into frequency features that are more easily captured by deep neural networks. Finally, frequency domain data augmentation is performed on the generated Mel spectrograms to further enrich the diversity of frequency features. Through the above-mentioned augmentation method combining temporal and frequency domains, the training effect and feature extraction robustness of the original network are comprehensively improved.

[0033] Specifically, the temporal data enhancement is used to enrich the temporal features of the original audio and improve the network's adaptability to temporal changes in audio. It includes at least one of time stretching, pitch shifting, and noise superposition. Time stretching adjusts the playback speed of the audio without changing the pitch, pitch shifting adjusts the pitch without changing the playback speed, and noise superposition adds low-intensity background noise to the original audio to simulate complex audio environments in real-world applications. The frequency domain data enhancement is used to enrich the frequency domain features of the Mel spectrogram and improve the network's ability to recognize changes in frequency domain features. It includes at least one of translation, flipping, cropping, and adversarial enhancement. Translation shifts the Mel spectrogram horizontally or vertically, flipping flips the Mel spectrogram horizontally or vertically, cropping extracts effective feature regions from the Mel spectrogram, and adversarial enhancement improves the network's sensitivity to identifying anomalous features by generating adversarial examples.

[0034] Specifically, the prototype set is used to accurately characterize the core features of normal and abnormal audio, providing a benchmark for the prototype network classifier. Its construction process is deeply integrated with the application scenario, and it consists of audio corresponding to each prototype category. The prototype category represents the same type of audio, that is, audio of the same category has similar feature attributes. Each prototype category corresponds to a set of representative audio samples. This set of samples needs to be screened to remove audio with excessive noise and atypical features, ensuring that the samples can accurately reflect the core features of the audio of that type, providing reliable basic data for subsequent prototype calculation, so that the prototype network classifier can accurately distinguish between normal and abnormal audio based on the prototype set.

[0035] Specifically, when constructing the prototype set, two construction methods are selected based on the detection accuracy requirements of the application scenario, including: One approach is to differentiate between fine-grained prototype categories, which involves further subdividing the audio based on its specific type and clarifying whether each fine-grained prototype category corresponds to a normal audio prototype or an abnormal audio prototype. Fine-grained segmentation can improve the accuracy of anomaly detection and facilitate the identification of specific anomaly types. The other approach is to not differentiate between fine-grained prototype categories and adopt a coarse-grained segmentation method, directly classifying all normal audio into one category as normal audio prototypes and all abnormal audio into another category as abnormal audio prototypes. This method is suitable for scenarios with high detection speed requirements and where it is not necessary to identify specific anomaly types. The two methods can be flexibly switched to adapt to different business needs.

[0036] Specifically, when constructing the prototype set, each prototype category can configure the inclusion strategy of the original training data according to the training effect and detection accuracy requirements: for each prototype category, it can be set independently whether to include the corresponding category training data used for the original network supervised training into the prototype set; if included, it is necessary to screen samples with typical features. The typical features refer to the core features of the sample being highly matched with the preset feature threshold of the corresponding prototype category, without redundant interference or feature loss, and its feature vector being in the preset high similarity interval with the feature vector of most samples in the category, which can accurately represent the essential features of the category; the representative audio samples refer to samples that can cover the feature distribution of different scenarios of the corresponding prototype category, reflect the common core features of the category and reasonable feature variations, and whose feature signal-to-noise ratio meets the preset requirements after noise suppression and can provide stable feature input; the screened training data and the newly collected representative samples of the category together constitute the prototype set; if the training data is not included, only the audio samples actually collected in the application scenario and verified by feature to meet the representative requirements are selected as the prototype set, so as to achieve a dynamic balance between the representativeness of the prototype set and the detection efficiency, and ensure the accuracy of prototype calculation and the robustness of anomaly detection.

[0037] Specifically, the prototype calculation process includes: First, the constructed feature extractor is loaded, ensuring its parameters are fixed and consistent with those of the original trained backbone network. Then, all audio samples from the prototype set are input into the feature extractor one by one. Each audio sample is first converted into a Mel spectrogram, and then the feature extractor extracts deep features from the Mel spectrogram, outputting the feature vector corresponding to each audio sample. Finally, for the same prototype category, the feature vectors of all audio samples in that category are collected, and the mean of all feature vectors is calculated to obtain the average feature vector for that prototype category. This average feature vector is the prototype of that prototype category and serves as the benchmark for subsequent classification.

[0038] Specifically, the specific process for determining the classification result includes: First, the audio samples to be predicted are obtained and preprocessed to remove high-intensity noise and ensure sample integrity. Then, the preprocessed audio is converted into a Mel spectrogram, maintaining the same spectrogram format as used in the training and prototype calculation processes. Next, several samples are generated based on this Mel spectrogram, and slight data augmentation is used to expand the sample coverage and improve accuracy. Then, the similarity between each generated sample and the prototype of each prototype category is calculated, using distance calculation to measure the similarity; higher similarity indicates a closer distance. Finally, based on the similarity calculation results of all generated samples, the category corresponding to the majority of samples is taken as the final category of the audio sample to be predicted, thus determining whether the audio is normal or abnormal. If it is abnormal, its corresponding abnormal prototype category can be further located.

[0039] Specifically, the update and iteration process of the prototype set is as follows: First, an evaluation set consisting of normal and abnormal audio is constructed. This evaluation set must cover all types of normal and abnormal audio that may occur in the application scenario and must not overlap with the training dataset to ensure the objectivity of the evaluation results. Then, evaluation criteria are constructed, which include core indicators such as classification accuracy, anomaly recognition rate, and false positive rate to measure the detection performance of the prototype set. Next, by setting prototype categories of different granularities, classification evaluation indicators on the evaluation set at each granularity are calculated to compare the detection effects at different granularities. Finally, based on the evaluation indicator results, the proportion of each category in the training samples is adjusted, samples of categories with insufficient features are supplemented, and the division method of prototype categories is optimized by merging categories with similar features and splitting categories with large feature differences to achieve the update and iteration of the prototype set and continuously improve detection accuracy.

[0040] This embodiment takes sound anomaly detection in an industrial equipment operation and maintenance scenario as an example. It describes the specific implementation process of the sound anomaly detection method based on a prototype network, focusing on the detection of normal operating sounds (OK audio), equipment friction fault sounds (NG audio), and trailing fault sounds (NG audio). The details are as follows: I. Implementation Preparation 1. Data Preparation: Collect 5000 audio recordings of normal operation of industrial equipment (OK audio), 1000 audio recordings of equipment friction failure (NG1 audio), and 1000 audio recordings of tailing failure (NG2 audio). All audio recordings have the same sampling frequency and duration. Randomly select 4000 OK audio, 800 NG1 audio, and 800 NG2 audio as training data, and use the remaining 1000 OK audio, 200 NG1 audio, and 200 NG2 audio as evaluation data for subsequent prototype set updates and iterations. In addition, select 10 OK audio, 10 NG1 audio, and 10 NG2 audio that did not participate in training and evaluation as samples to be predicted to verify the detection effect.

[0041] 2. Network and Parameter Preparation: The original network consisting of a Convolutional Neural Network (CNN), a Recurrent Neural Network (LSTM), and a classification head is selected. The CNN uses the repvgg_b1 architecture to extract audio frequency domain features, the LSTM is used for sequence modeling of the features, and the classification head is used to output the classification results. The training epochs are set to 100 epochs, the loss function is Focalloss, which is used to model the difference between the network output and the label, and the optimizer is an adaptive gradient descent optimizer to ensure training convergence.

[0042] II. Specific Implementation Steps Step 1: Constructing the Feature Extractor 1. Data Augmentation Calculation: Perform time-domain and frequency-domain data augmentation on the original audio data and corresponding labels (OK label is denoted as 0, NG1 label as 1, NG2 label as 2) in the training data. The specific operations are as follows: (1) Temporal data enhancement: Two methods are used: time stretching and pitch shifting. The time stretching ratio ranges from 0.9 to 1.1. The calculation method is as follows: Let the original audio duration be T, and the stretched audio duration T' = T × k (k is the stretching coefficient, and three values ​​of 0.9, 1.0, and 1.1 are randomly selected). Two shifts are set for pitch shifting, which are +0.5 and -0.5 respectively. The calculation method is as follows: Let the original audio pitch be f, and the shifted pitch f' = f + Δf (Δf is the shift amount, which is ±0.5). At the same time, noise is superimposed on part of the audio. The superimposed noise intensity is 10% of the average amplitude of the original audio. The calculation method is as follows: The enhanced audio amplitude A' = A + 0.1 × A × n (A is the original audio amplitude, and n is the random noise amplitude, which ranges from 0 to 1).

[0043] (2) Mel spectrogram conversion: The time-domain enhanced audio is converted into a Mel spectrogram. The calculation process is as follows: First, a short-time Fourier transform (STFT) is performed on the audio. Let the audio sampling frequency be Fs, the frame length be N, and the frame shift be M to obtain the spectrum matrix S (dimension F×T, where F is the number of frequency points and T is the number of frames). Then, the linear frequencies are converted into Mel frequencies through a Mel filter bank. The Mel frequency calculation formula is: , where f is the linear frequency; perform matrix multiplication on the spectrum matrix S and the Mel filter bank matrix W (with dimensions Mel_F×F, where Mel_F is the number of Mel frequency points) to obtain the Mel spectrum matrix M=W×S, and then take the logarithm of M to obtain the final Mel spectrum diagram with dimensions Mel_F×T.

[0044] (3) Frequency domain data enhancement: The generated Mel spectrogram is enhanced by translation and cropping. The translation calculation method is: the Mel spectrogram is randomly translated along the horizontal direction (time dimension) by ±5 frames, and the part that exceeds the boundary after translation is filled with 0. The cropping calculation method is: a sub-map of size Mel_F×(T-10) is randomly cropped from the Mel spectrogram to ensure that the core frequency domain features are preserved.

[0045] 2. Supervised Training of the Original Network: The enhanced Mel spectrogram and corresponding labels are input into the original network for supervised training, as follows: (1) Feature extraction: Feature extraction of the Mel spectrogram is performed using a CNN with the repvgg_b1 architecture. The formula for each convolution operation is: X_out=σ(W×X_in+b), where X_in is the input feature map, W is the convolution kernel weight, b is the bias term, and σ is the activation function (using the ReLU function). After multiple rounds of convolution and pooling operations, a feature vector of dimension C×1 is obtained (C is the number of feature channels). This feature vector is then input into an LSTM network for sequence modeling. The LSTM cell state update formula is: i_t=σ(W_i×[h_{t-1},x_t]+b_i), f_t=σ( W_f×[h_{t-1},x_t]+b_f), c_t=f_t×c_{t-1}+i_t×tanh(W_c×[h_{t-1},x_t]+b_c), o_t=σ(W_o×[h_{t-1},x_t]+b_o), h_t=o_t×tanh(c_t), where i_t, f_t, and o_t are the input gate, forget gate, and output gate, respectively; c_t is the cell state; h_t is the hidden state; x_t is the input feature at the current time step; W_i, W_f, W_c, and W_o are weight matrices; and b_i, b_f, b_c, and b_o are bias terms.

[0046] (2) Classification and Loss Calculation: The hidden state h_t output by the LSTM is input into the classification head, which uses a fully connected layer with an output dimension of 3 (corresponding to the three classes OK, NG1, and NG2). The output result is converted into a probability distribution using the Softmax function. The probability calculation formula is: p_i=exp(z_i) / Σexp(z_j) (i, j=0,1,2, z_i is the output value of the fully connected layer); the Focalloss is used to calculate the loss value. The loss calculation formula is: , where α_t is the class weight (α=0.4 for OK class, α=0.3 for NG1 and NG2 classes), γ is the focusing parameter (taken as 2), and p_t is the probability of predicting the correct class.

[0047] (3) Parameter update: Update the weight parameters of the original network through the backpropagation algorithm, calculate the gradient and adjust it. Calculate the accuracy of the training set after each epoch. When the accuracy no longer improves after 10 consecutive epochs, stop training and save the network weights at this time as the best model.

[0048] 3. Feature Extractor Construction: Remove the classification head from the trained original network and retain the backbone network composed of CNN and LSTM, which is the feature extractor. Its structure is fixed and does not change with the subsequent addition of NG types. This feature extractor can output a feature vector of dimension C×1 for subsequent prototype calculation.

[0049] Step Two: Building a Prototype System 1. Prototype Set Construction: Fine-grained prototype category division is adopted, which is divided into three categories: OK prototype, NG1 prototype, and NG2 prototype. 50 representative audio samples are selected from each category (screened from the training data to ensure typical features) to form the prototype set. At the same time, the corresponding category samples in the training data are included in the prototype set. That is, the OK prototype set contains 50 OK training samples, the NG1 prototype set contains 50 NG1 training samples, and the NG2 prototype set contains 50 NG2 training samples.

[0050] 2. Prototype Calculation: Load the feature extractor constructed in step one, extract features for each audio sample in the prototype set, and calculate the prototype (mean of feature vector) for each prototype category, as follows: (1) Single sample feature extraction: Convert each audio sample in the prototype set into a Mel spectrogram (without data augmentation) according to the method in step one, input it into the feature extractor, and output the feature vector x_i of the sample (dimension is C×1, i is the sample number, i=1~50 for OK prototype set, i=1~50 for NG1 prototype set, i=1~50 for NG2 prototype set).

[0051] (2) Prototype vector calculation: The prototype vector of the same prototype category is obtained by averaging all feature vectors of the same prototype category. The calculation formula is: P=(1 / N)×Σx_i (N is the number of samples of the prototype category, N=50 here, and Σ is the summation operation).

[0052] For example: Assuming the feature vector dimension C=128, and the feature vectors of the 50 samples in the OK prototype set are x_1, x_2, ..., x_50 (each vector is 128×1), then the OK prototype vector P_OK=(x_1+x_2+...+x_50) / 50, where the value of each dimension is the average value of the corresponding dimension values ​​of the 50 samples; similarly, the NG1 prototype vector P_NG1 and the NG2 prototype vector P_NG2 are calculated.

[0053] 3. Construction of Prototype Network Classifier: The feature extractor is combined with the three prototype vectors calculated above to construct a prototype network classifier. The core function of this classifier is to make classification judgments by calculating the similarity between the sample to be predicted and each prototype vector.

[0054] Step 3: Anomaly Detection and Judgment Ten samples to be predicted (3 OK, 4 NG1, and 3 NG2) were selected and classified as normal or abnormal. The process is as follows: 1. Preprocessing of samples to be predicted: Convert each audio to be predicted into a Mel spectrogram. The conversion method is the same as in step one, without data augmentation. At the same time, generate 5 samples based on the Mel spectrogram (using a cropping ratio of 90%~100% to obtain 5 different sub-images to ensure coverage of core features).

[0055] 2. Feature extraction of the sample to be predicted: Input each generated sample into the feature extractor and output the corresponding feature vector y_j (j=1~5, each vector has a dimension of 128×1).

[0056] 3. Similarity Calculation (Distance Calculation): Euclidean distance is used to measure the similarity between the feature vector of the sample to be predicted and each prototype vector. The smaller the Euclidean distance, the higher the similarity. The calculation formula is: d(y_j,P)=√[Σ(y_jk-Pk) 2 (k=1~128, y_jk is the k-th dimension value of the feature vector of the sample to be predicted, and Pk is the k-th dimension value of the prototype vector).

[0057] 4. Final classification judgment: Perform the above distance calculation and category judgment on the 5 generated samples of the sample to be predicted, and count the category distribution of the 5 samples. Select the category with the most occurrences as the final category of the sample to be predicted. If 3 of the 5 generated samples of a sample to be predicted are judged as NG1 and 2 are judged as NG2, then the sample is finally judged as NG1 (friction fault); if all 5 samples are judged as OK, then the sample is finally judged as OK (normal operation).

[0058] Step 4: Optimization and Adaptation 1. Prototype set update and iteration: Evaluation criteria are constructed using an evaluation set (1000 OKs, 200 NG1s, and 200 NG2s). Accuracy, precision, recall, and F1 score are calculated on the evaluation set, as follows: (1) Evaluation set detection: Each audio sample in the evaluation set is classified according to the method in step three, and the matching status between the predicted category and the actual category is recorded to obtain the confusion matrix: true positive (TP), true negative (TN), false positive (FP), and false negative (FN), where: OK category: TP_OK (predicted OK, actually OK) = 980, FP_OK (predicted OK, actually NG) = 15, FN_OK (predicted NG, actually OK) = 20, TN_OK (predicted NG, actually NG) = 385; NG1 class: TP_NG1 (predicted as NG1, actually NG1) = 192, FP_NG1 (predicted as NG1, actually other class) = 8, FN_NG1 (predicted as other class, actually NG1) = 8, TN_NG1 (predicted as other class, actually other class) = 1192; NG2 class: TP_NG2 (predicted as NG2, actually NG2) = 190, FP_NG2 (predicted as NG2, actually other class) = 7, FN_NG2 (predicted as other class, actually NG2) = 10, TN_NG2 (predicted as other class, actually other class) = 1193.

[0059] (2) Calculation of evaluation indicators: ①Accuracy: The proportion of correctly predicted samples out of the total number of samples. The formula is: Accuracy = (TP_OK + TP_NG1 + TP_NG2) / total number of samples; Substitute the data: Accuracy=(980+192+190) / (1000+200+200)=1362 / 1400≈97.29%.

[0060] ② Precision: The proportion of correctly predicted samples for a certain category out of the total number of predictions for that category. The formula is: Precision = TP / (TP + FP); Precision for OK class: 980 / (980+15)≈98.49%; Precision for NG1 class: 192 / (192+8)=96.00%; Precision for NG2 class: 190 / (190+7)≈96.46%.

[0061] ③ Recall: The proportion of correctly predicted samples in a certain category out of the actual total number of samples in that category. The formula is: Recall = TP / (TP + FN); OK class recall rate: 980 / (980+20)=98.00%; NG1 class recall rate: 192 / (192+8)=96.00%; NG2 class recall rate: 190 / (190+10)=95.00%.

[0062] ④ F1 score: The harmonic mean of precision and recall, calculated as: F1 = 2 × (Precision × Recall) / (Precision + Recall); OK class F1 score: 2×(98.49%×98.00%) / (98.49%+98.00%)≈98.24%; NG1 class F1 score: 2×(96.00%×96.00%) / (96.00%+96.00%)=96.00%; NG2 class F1 score: 2×(96.46%×95.00%) / (96.46%+95.00%)≈95.73%.

[0063] (3) Prototype set optimization: Based on the above evaluation index results, it was found that the recall rate of NG2 class was low (95.00%). The granularity of the prototype set was adjusted, and the number of samples in the NG2 prototype set was increased from 50 to 60 (10 NG2 training samples with typical features were added). The NG2 prototype vector was recalculated and evaluated again. The recall rate of NG2 class was improved to 97.00%, and the F1 value was improved to 96.85%. The prototype set update iteration was completed.

[0064] 2. New Type Adaptation: Assuming a new device noise fault (NG3 audio) is added to the business scenario, select 5 NG3 audio samples as representative samples to form an NG3 prototype, and add it to the original prototype set: (1) Prototype calculation: Convert the 5 NG3 audios into Mel spectrograms, input them into the feature extractor, extract the feature vectors x_1~x_5 (dimension 128×1) of each sample, and calculate the NG3 prototype vector P_NG3=(x_1+x_2+x_3+x_4+x_5) / 5. The calculation method is the same as the prototype calculation in step two.

[0065] (2) New type detection: P_NG3 is incorporated into the prototype system of the prototype network classifier. For the NG3 audio samples to be predicted, the Euclidean distance between its feature vector and P_NG3 is calculated according to the similarity calculation method in step three. If the distance is the smallest, it is determined to be the NG3 category. After testing, all 5 NG3 samples to be predicted were accurately identified. There is no need to retrain the feature extractor and the prototype network classifier, thus achieving rapid adaptation of new types.

[0066] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for detecting sound anomalies based on prototype networks, characterized in that, include: A feature extractor composed of deep neural networks is constructed based on normal and abnormal audio. The feature extractor is trained under supervision to capture the common features of normal audio and the difference features between normal and abnormal audio. Based on the feature extractor, a prototype set is constructed in combination with the application scenario; the feature extractor is used to extract features and calculate prototypes from the audio in the prototype set to generate normal audio prototypes and abnormal audio prototypes. Based on the normal audio prototype and the abnormal audio prototype, feature extraction and similarity comparison are performed on the audio sample to be predicted to generate a classification judgment result of normal and abnormal. Evaluation criteria are constructed based on an evaluation set consisting of normal and abnormal audio, and the prototype set is updated and iterated.

2. The method according to claim 1, characterized in that, The feature extractor is a backbone network, which is formed by removing the classification head from the original network; The original network includes a deep neural network and a classification head. The deep neural network is used to extract deep features of the audio, and the classification head is used to classify and output the extracted features. The original network distinguishes between normal audio and abnormal audio through supervised training, removes the classification head, and retains the deep neural network part as the feature extractor.

3. The method according to claim 2, characterized in that, The training process of the original network specifically includes: Based on normal audio and various abnormal audio in the application scenario, a training dataset is generated. Each audio segment in the training dataset is labeled with a corresponding category label, generating normal audio labels and various abnormal audio labels. The original audio and corresponding labels are input into the original network. Supervised training is adopted, with classification loss as the optimization objective. The parameters of the deep neural network in the original network are adjusted through iterative training so that the error between the classification result output by the deep neural network and the labeled label reaches the preset requirement.

4. The method according to claim 3, characterized in that, During supervised training of the original network, data augmentation processing is required on the input raw audio. The specific process includes: The original audio and corresponding tags are simultaneously augmented in the time domain. The augmented audio samples are then converted into Mel spectrograms. The time domain features of the audio samples are converted into frequency domain features. Frequency domain data augmentation is then performed on the generated Mel spectrograms.

5. The method according to claim 4, characterized in that, The time-domain data enhancement used for the original audio specifically includes: time stretching, pitch shifting, and noise superposition. Time stretching is used to adjust the playback speed without changing the pitch, pitch shifting is used to adjust the pitch without changing the playback speed, and noise superposition is used to add low-intensity background noise to the original audio. The frequency domain data enhancement is used to enhance the frequency domain features of the Mel spectrogram, specifically including translation, flipping, cropping, and adversarial enhancement. Translation is used to shift the Mel spectrogram horizontally or vertically, flipping is used to flip the Mel spectrogram horizontally or vertically, cropping is used to extract the effective feature regions in the Mel spectrogram, and adversarial enhancement is used to improve the network's sensitivity to identifying abnormal features by generating adversarial examples.

6. The method according to claim 1, characterized in that, The prototype set is used to characterize the core features of normal and abnormal audio, providing a judgment benchmark for the prototype network classifier. The construction process is based on the application scenario and is specifically composed of audio corresponding to each prototype category. The prototype category represents the same type of audio. Each prototype category corresponds to a set of audio samples. The audio samples need to be screened to remove abnormal and interfering audio.

7. The method according to claim 6, characterized in that, When constructing the prototype set, two construction methods are selected based on the detection accuracy requirements of the application scenario, specifically including: Distinguish between fine-grained prototype categories, further subdivide them according to audio type, and clarify whether each fine-grained prototype category corresponds to a normal audio prototype or an abnormal audio prototype; Instead of distinguishing between fine-grained prototype categories, a coarse-grained classification method is adopted, grouping all normal audio into one category as normal audio prototypes and all abnormal audio into another category as abnormal audio prototypes.

8. The method according to claim 6, characterized in that, When constructing the prototype set, each prototype category can configure the inclusion strategy of the original training data according to the training effect and detection accuracy requirements: for each prototype category, the corresponding category training data used for the original network supervised training is included in the prototype set, and samples with typical features need to be selected. The selected training data and the newly collected representative samples of the corresponding category together constitute the prototype set; if the training data is not included, then the representative samples actually obtained in the application scenario and verified by the features are selected as the prototype set.

9. The method according to claim 1, characterized in that, The prototype calculation process specifically includes: Based on the feature extractor, the parameters of the feature extractor are fixed to be consistent with the parameters of the original trained backbone network. The audio samples in the prototype set are input one by one into the feature extractor. The audio samples are first converted into Mel spectrograms, and then the deep features in the Mel spectrograms are extracted by the feature extractor to output the feature vector corresponding to each audio sample. For the same prototype category, obtain the feature vectors of all audio samples under the prototype category, calculate the mean of all feature vectors, and generate the average feature vector as the prototype of the prototype category.

10. The method according to claim 1, characterized in that, The specific process for determining the classification result includes: Acquire audio samples to be predicted and preprocess them to remove noise from the audio; convert the preprocessed audio to be predicted into a Mel spectrogram, consistent with the spectrogram format in the training and prototype calculation processes; expand the sample coverage of the generated samples through data augmentation. Calculate the similarity between each generated sample and the prototype of the corresponding prototype category. The result of the similarity calculation for all generated samples is used as the final category of the audio sample to be predicted.

11. The method according to claim 1, characterized in that, The process of constructing evaluation criteria for the evaluation set specifically includes: An evaluation set consisting of normal and abnormal audio is used, which must include all types of normal and abnormal audio that appear in the application scenario and must not overlap with the training dataset. Evaluation criteria are constructed, including classification accuracy, anomaly recognition rate, and false positive rate. By setting prototype categories of different granularities, classification evaluation indicators on the evaluation set at each granularity are calculated, and the detection effects at different granularities are compared.

12. The method according to claim 1, characterized in that, The process of updating and iterating the prototype set specifically includes: Based on the evaluation results, the proportion of each category in the training samples is adjusted, samples of categories with missing features are added, and the classification method of prototype categories is optimized by merging and splitting to complete the update and iteration of the prototype set.

Images