A heterophonic sound detection method using metadata hierarchical information to constrain self-supervised classification

Abstract

The present application belongs to the technical field of abnormal sound detection, and particularly relates to an abnormal sound detection method using metadata hierarchical information constraint self-supervised classification. The present application uses a metadata hierarchical information structure to constrain the learning of low-dimensional features and high-dimensional features of training audio by a neural network, fully utilizes metadata accompanying an audio file, mines the influence of metadata attributes on acoustic features, so that the neural network can learn the changes caused by domain shift on audio features, and further improves the performance of an industrial abnormal sound detection system under domain shift conditions. Meanwhile, the present application proposes an attribute group-centered abnormal score calculation method for evaluating the abnormal score of a test sample under domain shift conditions to judge whether the test audio is normal. Compared with the prior art, the method of the present application can learn audio features more finely and reduce the problems caused by domain shift in abnormal sound detection.

Description

Technical Field

[0001] This invention belongs to the field of abnormal sound detection technology, specifically involving an abnormal sound detection method that utilizes metadata hierarchical information to constrain self-supervised classification. Background Technology

[0002] The purpose of Anomalous Sound Detection (ASD) is to automatically identify whether a target (such as a machine or device) is making unusual sounds, thereby determining whether the target is exhibiting abnormal behavior or state. The diversity and sporadic nature of unusual sounds mean that only normal sounds can be used to train neural networks.

[0003] With the application of deep learning in audio processing, existing research provides two methods for sound anomaly detection: unsupervised and self-supervised. Existing unsupervised methods learn the features of normal sound by minimizing reconstruction error and use this error as a score to detect anomalies. Such industrial sound anomaly detection methods can provide a certain level of performance, but their false detection rate is high and heavily influenced by prior threshold settings. Existing self-supervised methods incorporate metadata attached to the audio data (such as machine type, machine operating power / speed, ambient noise, etc.) into the modeling process. They determine the state of the audio data (normal / abnormal) by judging the status of the audio metadata attributes, achieving a more considerable average detection performance compared to unsupervised methods.

[0004] However, the operating conditions of acoustic targets are not static, and audio recording conditions also vary. Therefore, the state distribution of the training data does not completely represent all normal machine behaviors and states, introducing a domain offset problem for abnormal sound detection. For industrial abnormal sound detection, the source domain records most of the training data and some of the test data, while the target domain records a small amount of training data and some of the test data. The source and target domains differ in terms of operating speed, machine load, viscosity, heating temperature, and type of environmental noise.

[0005] The introduction of domain offset problem significantly limits the performance of anomalous sound detection in practice. Domain offset refers to the difference in data distribution between the source and target domains due to variations in observation conditions, target state, and background noise. Models trained on source domain data may exhibit low accuracy and poor stability when tested on target domain data. The acoustic characteristics of the source and target domains change with metadata attributes, leading to models trained using source domain sounds potentially misidentifying anomalies in the target domain. Since domain offset grouping indexes represent a specific type of domain offset, current mainstream solutions use sectionIDs or attributes as labels for self-supervised learning. However, metadata information related to the audio file and the source domain itself has not been fully utilized to address the domain offset problem in anomalous sound detection. Summary of the Invention

[0006] The purpose of this invention is to solve the problem of incorrect anomaly identification in the target domain after the introduction of domain offset in traditional self-supervised learning methods, and to provide an anomaly detection method that uses metadata hierarchical information to constrain self-supervised classification.

[0007] An anomaly detection method that utilizes hierarchical metadata information to constrain self-supervised classification includes the following steps:

[0008] Step 1: For a specific type of acoustic target k, obtain its original audio signal s, and convert it into a spectrogram feature through short-time Fourier transform; filter the spectrogram feature through a Mel filter bank to obtain the Mel spectrum; perform logarithmic scaling on the Mel spectrum to obtain the frequency domain feature Log-Mel spectrum.

[0009] Step 2: Under the constraints of the hierarchical metadata information structure, feature extraction is performed on the frequency domain feature Log-Mel spectrogram to obtain audio features containing high-dimensional information.

[0010] Step 3: Calculate the anomaly score centered on the attribute group.

[0011]

[0012] Where M represents the total number of attribute groups in the hierarchical metadata information structure; the covariance matrix Σ is obtained from the features of all audio data in the m-th attribute group under the same domain offset grouping index, Σ -1 Let represent the inverse of the covariance matrix Σ; for an attribute group m containing N audio segments, This represents the audio feature containing high-dimensional information extracted from the nth audio audio, with feature center c. m for:

[0013]

[0014] Step 4: The anomaly detection mechanism for acoustic target k is expressed as follows:

[0015]

[0016] in, The mathematical expression representing the anomaly detection mechanism is a binary function; θ is the anomaly threshold selected by the anomaly detection mechanism based on the overall distribution of the training data, applicable to different types of acoustic targets; when the anomaly score exceeds θ, the acoustic target is considered to be outside the distribution range of normal audio features and is judged as abnormal; otherwise, the acoustic target is considered to be within the distribution range of normal audio features and is judged as normal.

[0017] Furthermore, the overall process of step 1 can be summarized as follows:

[0018]

[0019] Where X represents the frequency domain feature Log-Mel spectrum, M represents the dimension of the Mel filter, and N represents the number of time frames for the frequency domain features; Denotes the Mel filter matrix; ||STFT(x)|| 2 This indicates that the power spectrum is taken from the characteristics of the spectrogram.

[0020] Furthermore, in step 2, the hierarchical metadata information structure is a three-layer tree structure, with the acoustic target type as the root node, the second-layer domain offset grouping index as the intermediate node, and the third-layer attribute group information as the leaf node.

[0021] Furthermore, in step 3, feature extraction of the frequency domain feature Log-Mel spectrum is performed using self-supervised learning constrained by a hierarchical metadata information structure, specifically as follows:

[0022] Step 3.1: Hierarchical extraction of audio features;

[0023] The Log-Mel spectrum frequency domain features are input into a feature extractor based on a convolutional neural network structure. Obtain audio features f containing low-dimensional information l Then f l Inputting a 2D convolutional layer Conv2D(·) yields audio features f containing high-dimensional information. h The above feature extraction process can be expressed by the following formula:

[0024]

[0025] f h =Conv2D(f l )

[0026] Step 3.2: Predict self-supervised labels using a linear classifier;

[0027] Based on the extracted features f l and f h The two-layer self-supervised labels are predicted using a linear classifier, as expressed in the following formula:

[0028]

[0029]

[0030] in, Indicates the use of feature f l Prediction domain offset grouping index self-supervised label l ID A linear classifier, the prediction result is denoted as Indicates the use of feature f h Predicted attribute group self-supervised label l AG A linear classifier, the prediction result is denoted as The two linear classifiers have the same structure, each consisting of a single layer of linear units and a softmax activation function;

[0031] Step 3.3: Calculate the loss based on the label prediction results;

[0032] Introducing domain offset grouping index cross-entropy loss Cross-entropy loss of attribute groups To constrain the learning process of low-level and high-level features, the calculations are as follows:

[0033]

[0034]

[0035] Where CE(·,·) represents the cross-entropy loss calculated for two parameters. Assuming x and y are the parameters, the detailed calculation formula is as follows:

[0036] CE(x,y)=-xlogy-(1-y)logx

[0037] Take the empirical weights λ obtained during training. and The weighted average of the two cross-entropy losses is used as the total loss. This process can be expressed as follows:

[0038]

[0039] The beneficial effects of this invention are as follows:

[0040] This invention provides an abnormal sound detection method that utilizes metadata hierarchical information constraints for self-supervised classification, addressing the problem of anomaly identification errors in the target domain caused by domain shift in traditional self-supervised learning methods. This invention uses a metadata hierarchical information structure to constrain the learning of low-dimensional and high-dimensional features of training audio by a neural network. It fully utilizes the metadata of the accompanying audio file to mine the influence of metadata attributes on acoustic features, enabling the neural network to learn the changes in audio features caused by domain shift, thereby improving the performance of industrial abnormal sound detection systems under domain shift conditions. Simultaneously, this invention proposes an anomaly score calculation method centered on attribute groups to evaluate the anomaly score of test samples under domain shift conditions, determining whether the test audio is normal. Compared with existing technologies, the method of this invention can learn audio features more precisely, mitigating the problems caused by domain shift in abnormal sound detection. Attached Figure Description

[0041] Figure 1 This is the overall technical roadmap of the present invention.

[0042] Figure 2 This is a structural diagram of the hierarchical metadata information structure in this invention.

[0043] Figure 3 This is the feature extractor used in this invention to extract low-dimensional and high-dimensional audio features. The structural diagram. Detailed Implementation

[0044] The present invention will now be further described with reference to the accompanying drawings.

[0045] To address the issue of anomaly detection errors in the target domain caused by domain shift in traditional self-supervised learning methods, this invention provides an anomaly detection method that utilizes hierarchical metadata information to constrain self-supervised classification. This invention proposes a hierarchical metadata information structure for self-supervised anomaly detection, which can constrain the neural network's learning of low-dimensional and high-dimensional features of the training audio. It fully leverages the metadata accompanying the audio files in the training data to uncover the impact of metadata attributes on acoustic characteristics, enabling the neural network to learn the changes in audio features caused by domain shift. This improves the performance of industrial anomaly detection systems under domain shift conditions. Furthermore, this invention proposes an anomaly score calculation method centered on attribute groups, which can be used to evaluate the anomaly score of test samples under domain shift conditions to determine whether the test audio is normal.

[0046] An abnormal sound detection method utilizing hierarchical metadata information to constrain self-supervised classification includes: converting audio into frequency domain Log-Mel spectral features; establishing a hierarchical metadata information structure (metadata information tree); extracting learnable features from the Log-Mel spectral frequency domain features under the constraints of the metadata information tree; and evaluating whether test samples are abnormal based on an anomaly score calculation method centered on attribute groups.

[0047] During the training phase, the original audio of the acoustic target is sequentially passed through a short-time Fourier transform and a Log-Mel filter to obtain the Log-Mel spectral frequency domain features of the original audio. Simultaneously, the metadata information carried by the original audio file is split and reorganized to establish a three-layer hierarchical metadata information structure (metadata information tree) containing acoustic target type, domain offset grouping index, and attribute group information. The metadata information tree includes the type characteristics of the domain offset and acoustic features, representing the hierarchical relationship of the metadata information.

[0048] Then, using the domain offset grouping index and attribute group information of the metadata information tree as constraints, a learnable two-stage feature extraction of the Log-Mel spectral frequency domain features is performed through a MobileFaceNet-based neural network and a two-dimensional convolutional layer, generating attribute group feature centers. The learning process of the neural network is constrained by the domain offset grouping index and attribute group information, and its output audio features contain, to some extent, low-level information—the domain offset grouping index—and high-level information—the attribute group information. By constraining the learning process of the neural network through the hierarchical relationship of the domain offset information contained in the metadata information tree, the neural network's attention to the changes in audio features caused by the type and cause of domain offset is strengthened, resulting in more refined features that include the characteristics of domain offset.

[0049] During the testing phase, the test audio was processed through a short-time Fourier transform, a Log-Mel filter, a MobileFaceNet-based neural network, and a two-dimensional convolutional layer to obtain test audio features. The Mahalanobis distance between the test audio features and the feature centers of the attribute groups was calculated as an anomaly score. Finally, an anomaly score threshold was used to determine whether the test audio was normal. Since the feature centers of the attribute groups may contain acoustic features of normal sounds that are related to domain shift, the Mahalanobis distance between the test audio and the feature centers of the attribute groups includes the deviation from the normal distribution caused by domain shift.

[0050] 1. Frequency Domain Log-Mel Spectral Feature Extraction

[0051] Assuming the original single-channel audio signal corresponding to the acoustic target is Where 1 represents the channel dimension corresponding to a single channel, L represents the number of sampling points of the audio digital signal, reflecting the duration of the original audio, and the sampling frequency of the original acoustic signal is 16kHz.

[0052] First, the original audio signal s undergoes a Short-Time Fourier Transform (STFT) to convert it into a spectrogram, which encompasses all frequency bands defined by the sampling frequency. The calculation process uses a STFT time window of 1024 sampling points (64 ms), with a 50% overlap between adjacent time windows, meaning the time window offset step is 512 sampling points (32 ms).

[0053] Subsequently, the spectral features are filtered using a Mel filter bank to obtain the Mel spectrum. To amplify the regions of spectral features that are sensitive to human auditory perception, the Mel spectrum is logarithmically scaled to obtain the Log-Mel spectrum of the frequency domain features. The overall calculation process for the frequency domain features can be summarized as follows:

[0054]

[0055] Where X represents the frequency domain feature Log-Mel spectrum, M represents the dimension of the Mel filter, while N represents the number of time frames for the frequency domain features. This represents the Mel filter matrix, which has 128 dimensions. ||STFT(x)|| 2 This means that the power spectrum of the spectrogram is taken from the features of the spectrum, thus ignoring the computational cost of the short-time Fourier transform results in the complex domain.

[0056] 2. Establish a hierarchical metadata information structure

[0057] Metadata information consists of textual information carried by the original audio of the acoustic target. It includes the acoustic target type, the sequence number of each domain offset condition (domain offset grouping index), and the specific content of each domain offset condition. The acoustic target type refers to the main type of sound-producing machine in the audio, such as a toy car or a fan. Domain offset conditions refer to the specific reasons causing the domain offset, such as the toy car's sequence number, speed, microphone variations, or noise floor type. Metadata indicating the same domain offset condition is grouped into the same attribute group. The attribute groups of the source and target domains are different, and the attribute groups within the same source or target domain are also different. Different domain offset grouping indices indicate different reasons for the domain offset.

[0058] Attached Figure Figure 2The diagram illustrates the hierarchical metadata information structure, a three-layer tree structure. The acoustic target type is the root node, the second layer (domain offset grouping index) is the intermediate node, and the third layer (attribute group information) is the leaf node. The domain offset grouping index abstractly represents the reason for the domain offset under that index, while the attribute group information details the specific content of the domain offset condition. These two layers of information contain the changes in acoustic features caused by the domain offset. Introducing this information into neural networks for learning audio features allows the neural network to focus more on learning the changes in audio features caused by the domain offset.

[0059] 3. Utilizing hierarchical metadata information structure to constrain self-supervised learning

[0060] (1) Hierarchical extraction of audio features

[0061] The Log-Mel spectrum frequency domain features are input into a feature extractor based on the MobileFaceNet network architecture. Obtain audio features f containing low-dimensional information l Then f l Inputting a 2D convolutional layer Conv2D(·) yields audio features f containing high-dimensional information. h The above feature extraction process can be expressed by the following formula:

[0062]

[0063] f h =Conv2D(f l )

[0064] Appendix Figure 3 This demonstrates a feature extractor implemented based on the MobileFaceNet network architecture. It is important to note that MobileFaceNet is a convolutional neural network originally used for human face recognition in the image domain. The main purpose of the feature extractor used in this invention is to extract learnable audio features containing low-dimensional information. This can be implemented not only based on MobileFaceNet, but also based on other convolutional neural networks (such as MobileNetV2).

[0065] (2) Predicting self-supervised labels using a linear classifier

[0066] Based on the extracted low-level features f l and advanced features f h The prediction of two layers of self-supervised labels by a linear classifier can be expressed as the following formula:

[0067]

[0068]

[0069] in, This indicates the use of low-level features f l Prediction domain offset grouping index self-supervised label l ID A linear classifier, the prediction result is denoted as Indicates the use of high-level features f h Predicted attribute group self-supervised label l AG A linear classifier, the prediction result is denoted as The two linear classifiers have the same structure, each consisting of a single layer of linear units and a softmax activation function.

[0070] (3) Calculate the loss based on the label prediction results

[0071] Based on the above prediction of self-supervised labels, this invention introduces domain offset grouping index cross-entropy loss. Cross-entropy loss of attribute groups To constrain the learning process of low-level and high-level features, the calculations are as follows:

[0072]

[0073]

[0074] Where CE(·,·) represents the cross-entropy loss calculated for two parameters. Assuming x and y are the parameters, the detailed calculation formula is as follows:

[0075] CE(x,y)=-xlogy-(1-y)logx

[0076] Finally, the empirical weight λ obtained during training is used as the setpoint. and The weighted average of the two cross-entropy losses is used as the total loss. This process can be expressed as follows:

[0077]

[0078] The values ​​of the empirical weight λ are shown in the table below for different acoustic objectives:

[0079] ToyCar 0.5 ToyTrain 0.4 Bearing 0.7 Fan 0.9 Gearbox 0.5 Slider 0.8 Value 0.9

[0080] The above predicted label-loss calculation process is the constrained self-supervised classification method proposed in this invention. In this method, the domain offset group index contains the type characteristics of the domain offset that constrains the learning of low-level features, and the attribute group information contains the acoustic features that constrain the learning of high-level features. The convolutional neural network trained under these two layers of constraints strengthens the attention to the changes in audio features caused by the type and cause of domain offset, making up for the lack of implicit relationship between the domain offset group index and the attribute group information in traditional methods.

[0081] 4. Calculate the anomaly score centered on the attribute group.

[0082] To address the characteristic feature distribution of normal audio under domain offset conditions, this invention proposes an anomaly score calculation method centered on attribute groups. The closer the feature of the test audio is to the feature center of the attribute group, the smaller the corresponding anomaly score.

[0083] Suppose we have an attribute group m containing N training audio segments, using... Let c represent the audio features containing high-dimensional information extracted from the nth training audio, then its feature center c m The calculation method is as follows:

[0084]

[0085] During the testing phase, the original waveform of the test audio was converted into a frequency domain Log-Mel spectrum, and then high-level features of the test audio were extracted using a neural network. Then calculate the features. With the feature center c of the attribute group m The Mahalanobis distance between them is used as the outlier score.

[0086]

[0087] Where M represents the total number of attribute groups in the hierarchical metadata information structure, and the covariance matrix Σ is obtained from the features of all audio data in the m-th attribute group under the same domain offset grouping index. -1 Σ represents the inverse of the covariance matrix Σ.

[0088] In contrast, this method also proposes a domain-centered anomaly score calculation, which differs from the attribute group-centered calculation method only in the calculation method of the feature center.

[0089] Because the feature center of the attribute group is c m It is the average of high-level features of audio with the same attribute. These high-level features are extracted under the constraints of two layers: domain offset grouping index and attribute group information, which contain domain offset type characteristics and acoustic features. Therefore, the attribute group feature center cm This includes the domain-shifted acoustic features of the group of normal sounds. Under the domain-shift condition, the feature center c of the attribute group... m It better matches the distribution of the characteristic center of normal sound, so the abnormal score obtained is more accurate than that centered on the domain.

[0090] 5. Anomaly detection mechanism

[0091] For a specific type of acoustic target k, the anomaly score of its corresponding original audio waveform obtained through the above calculation is denoted as . The anomaly detection mechanism's handling of this acoustic target can be represented as follows:

[0092]

[0093] in, The mathematical expression for the anomaly detection mechanism is a binary function. θ is the anomaly threshold selected by the anomaly detection mechanism based on the overall distribution of the training data, applicable to different types of acoustic targets. When the anomaly score exceeds θ, the acoustic target is considered to be outside the distribution range of normal audio features and is judged as abnormal; conversely, the acoustic target is considered to be within the distribution range of normal audio features and is judged as normal.

[0094] 6. Excellent ASD performance under domain offset conditions

[0095] The proposed noise detection method, which utilizes hierarchical metadata information to constrain self-supervised learning, effectively leverages the implicit information in metadata under domain offset conditions, thereby improving the performance of current noise detection methods under these conditions. The table below compares the performance of the proposed noise detection method, traditional industrial noise detection methods, and the best existing noise detection methods under domain offset conditions, using AUC and pAUC as evaluation metrics.

[0096]

[0097] As can be seen from the table above, the abnormal sound detection method (HMIC-AGC in the table above) proposed in this invention, which uses metadata hierarchical information to constrain self-supervised learning, has an absolute advantage over other methods under domain offset conditions and ensures high accuracy for different acoustic targets.

[0098] Example 1:

[0099] This invention provides a method for detecting abnormal noise by using hierarchical metadata information to constrain self-supervised classification. It establishes a hierarchical metadata information structure (metadata information tree) using metadata information carried from audio files, and uses this structure to constrain the neural network's learning of low-level and high-level features of the audio. This improves the performance of industrial abnormal noise detection systems under domain offset conditions.

[0100] Domain offset refers to the difference in machine audio data distribution between the source and target domains due to variations in observation conditions, target state, and background noise. Models trained on source domain data may exhibit low accuracy and poor stability when tested on target domain data. In this invention, the source domain data records most of the training data and some of the test data, while the target domain data records a small amount of training data and some of the test data. Even with only a small amount of target domain data in the training data and a large amount in the test data, this strategy still ensures high accuracy in abnormal sound detection thanks to the two-layer constraint of the metadata information tree on self-supervised classification.

[0101] The original waveform s of the acoustic target is transformed into a power spectrum feature through a Short-Time Fourier Transform (STFT), and then extracted into Log-Mel spectral frequency domain features X by passing through a Mel filter matrix and logarithmic transformation.

[0102]

[0103] in, M represents the dimension of the Mel filter, and N represents the number of time frames for the frequency domain features. log(·) is the logarithmic operation. Let ||STFT(s)|| be the filter matrix of the Mel filter bank. 2 The power spectrum features of the spectrogram are extracted from the original signal s of the acoustic target.

[0104] Its short-time Fourier transform uses an overlapping time window for sampling, with a time window length of 1024 sampling points, which is 64ms at a sampling frequency of 16kHz, and an overlap step size of 512 sampling points, which is 32ms at a sampling frequency of 16kHz. The Mel filter bank has a dimension of 128.

[0105] The raw audio file of an acoustic target carries metadata including the acoustic target type, domain offset grouping index, and attribute group information. The acoustic target type indicates the main type of machine emitting the sound in the audio; the domain offset grouping index is the sequence number of different domain offset conditions, which can indicate the type of domain offset; changes in the values ​​in the attribute group information directly indicate the cause of the domain offset. Therefore, based on the description of domain offsets in the metadata, the hierarchical relationship between the domain offset grouping index and the attribute group information can be used as a constraint for learning more refined audio features, thus mitigating the problems caused by domain offsets in abnormal sound detection.

[0106] The metadata carried by the original audio file of the acoustic target can be split and reorganized to establish a hierarchical metadata information structure. This structure is a three-level tree structure, namely the first level with the acoustic target type as the root node, the second level with the domain offset group index as the intermediate node, and the third level with the attribute group information as the leaf node.

[0107] The Log-Mel spectral frequency domain feature X is used to calculate the audio feature f containing low-level information through a hierarchical learning neural network. l and audio features containing advanced information f h .

[0108] Because the learning process of a neural network is constrained by the domain offset grouping index and attribute group information, its output audio features f l and f h To a certain extent, it contains low-level information, namely domain offset grouping index information, and high-level information, namely attribute group information. A hierarchical learning neural network consists of a feature extractor and a 2D convolutional layer.

[0109] Feature extractor Audio features f containing low-level information are used to extract audio features f from Log-Mel spectrum frequency domain features X. l :

[0110]

[0111] Among them, feature extractor This implementation is based on the MobileFaceNet network architecture, but can be replaced with other convolutional neural networks. 2D convolutional layers (Conv2D(·)) are used to extract audio features f containing low-level information. l Extracting audio features containing advanced information f h :

[0112] f h =Conv2D(f l )

[0113] A constrained self-supervised classification method uses a domain offset grouping index (defined as l) obtained from a hierarchical metadata information structure. ID ) and attribute group information (defined as l AG These labels are used as self-supervised labels to constrain the hierarchical learnable neural network pairs f. l and f hThe learning process involves several steps. The domain offset grouping index contains the type characteristics of the domain offsets that constrain low-level feature learning, while the attribute group information contains the acoustic features that constrain high-level feature learning. This constrained self-supervised classification method utilizes hierarchical relationships to enable the neural network to learn features related to domain offsets, enhancing the network's focus on the changes in audio features caused by the type and cause of domain offsets. It also includes a linear ID classifier. Used to predict domain offset grouping index labels, and a linear attribute classifier. Used to predict attribute group information labels:

[0114]

[0115]

[0116] in, This represents the predicted value of the obtained domain offset grouping index. This represents the obtained attribute group information. (Classifier) and All are implemented using a single linear unit and a softmax activation function.

[0117] Calculate l separately ID and Cross-entropy loss between With l AG and Cross-entropy loss between Then, the average loss of the two cross-entropy losses is taken as the total loss, using the empirical weights λ obtained during training (see the instruction manual for details on the values).

[0118]

[0119]

[0120]

[0121] Where CE(·,·) represents the calculation of the cross-entropy of two parameters.

[0122] An attribute-group-centered anomaly score calculation method is proposed. This method averages the audio features learned under the constraints of the entire hierarchical structure and uses the average pooled vector as the attribute group feature center to evaluate the test audio in the anomaly detection phase. Since the attribute group feature center may contain acoustic features related to domain shift of normal sound, it can be used to measure the anomaly score of test samples in the presence of domain shift. For an attribute group m containing N training audio segments, its feature center c... m The calculation method is as follows:

[0123]

[0124] in, This represents the audio features containing high-level information extracted from the nth training audio.

[0125] Calculate the features of the test audio With the feature center c of the attribute group m The Mahalanobis distance between them is used as the outlier score.

[0126]

[0127] Where M represents the total number of attribute groups in the hierarchical metadata information structure, and the covariance matrix Σ is obtained from the features of all audio data in the m-th attribute group under the same domain offset grouping index. -1 Σ represents the inverse of the covariance matrix Σ.

[0128] This invention provides an abnormal sound detection method that utilizes metadata hierarchical information constraints for self-supervised classification, addressing the problem of anomaly identification errors in the target domain caused by domain shift in traditional self-supervised learning methods. The innovative method uses a metadata hierarchical information structure to constrain the learning of low-dimensional and high-dimensional features of training audio by a neural network. It fully leverages the metadata of the accompanying audio file to mine the influence of metadata attributes on acoustic features, enabling the neural network to learn the changes in audio features caused by domain shift, thereby improving the performance of industrial abnormal sound detection systems under domain shift conditions. Simultaneously, this invention proposes an anomaly score calculation method centered on attribute groups to evaluate the anomaly score of test samples under domain shift conditions, determining whether the test audio is normal. Compared to existing technologies, the method of this invention can learn audio features more precisely, mitigating the problems caused by domain shift in abnormal sound detection.

[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting abnormal sounds by using hierarchical metadata information to constrain self-supervised classification, characterized in that, Includes the following steps: Step 1: For a specific type of acoustic target To obtain its original audio signal The spectrum is converted into a spectrum feature through short-time Fourier transform; the spectrum feature is then filtered through a Mel filter bank to obtain the Mel spectrum; the Mel spectrum is then logarithmically scaled to obtain the frequency domain feature Log-Mel spectrum. Step 2: Under the constraints of the hierarchical metadata information structure, feature extraction is performed on the frequency domain feature Log-Mel spectrogram to obtain audio features containing high-dimensional information. The layer metadata information structure is a three-layer tree structure, with the acoustic target type as the root node, the second-layer domain offset grouping index as the intermediate node, and the third-layer attribute group information as the leaf node. Step 3: Calculate the anomaly score centered on the attribute group. ; in, Represents the total number of attribute groups in the hierarchical metadata information structure; covariance matrix It is the first under the same domain offset grouping index The features of all audio data in each attribute group are obtained. Represents the covariance matrix The inverse matrix; for a matrix containing Attribute groups of audio segments , Indicates from the first Audio features containing high-dimensional information extracted from audio samples, feature centers for: Feature extraction from the frequency domain feature Log-Mel spectrum utilizes self-supervised learning constrained by hierarchical metadata information structure, specifically: Step 3.1: Hierarchical extraction of audio features; The Log-Mel spectrum frequency domain features are input into a feature extractor based on a convolutional neural network structure. To obtain audio features containing low-dimensional information Then Input a 2D convolutional layer To obtain audio features containing high-dimensional information The above feature extraction process can be expressed by the following formula: Step 3.2: Predict self-supervised labels using a linear classifier; Based on the extracted features and The two-layer self-supervised labels are predicted using a linear classifier, as expressed in the following formula: in, Indicates the use of features Prediction domain offset grouping index self-supervised label A linear classifier, the prediction result is denoted as ; Indicates the use of features Predicted attribute group self-supervised labels A linear classifier, the prediction result is denoted as The two linear classifiers have the same structure, each consisting of a single layer of linear units and a softmax activation function. Step 3.3: Calculate the loss based on the label prediction results; Introducing domain offset grouping index cross-entropy loss Cross-entropy loss of attribute groups To constrain the learning process of low-level and high-level features, the calculations are as follows: in, This represents the calculation of the cross-entropy loss for two parameters, assuming that... and For the parameter, its detailed calculation formula is as follows: Based on the empirical weights obtained during training Pick and The weighted average of the two cross-entropy losses is used as the total loss. The process is expressed as follows: Step 4: Acoustic target The anomaly detection mechanism is represented as follows: in, The mathematical expression representing the anomaly detection mechanism is a binary function; The anomaly detection mechanism selects anomaly thresholds applicable to different acoustic target types based on the overall distribution of the training data; when the anomaly score exceeds... If the acoustic target exceeds the distribution range of normal audio features, it is considered abnormal; otherwise, it is considered to be within the distribution range of normal audio features and is considered normal.

2. The method for detecting abnormal sounds by utilizing hierarchical metadata information to constrain self-supervised classification, as described in claim 1, is characterized in that: The overall process of step 1 can be summarized as follows: in, Represents the frequency domain characteristic Log-Mel spectrum. , This indicates the dimension of the Mel filter. The number of time frames representing the frequency domain characteristics; Represents the Mel filter matrix; This indicates that the power spectrum is taken from the characteristics of the spectrogram.

Images