A method for detecting abnormal sounds by using metadata to generate audio and predict unknown anomalies

Abstract

The present invention is a method for detecting abnormal sounds that uses metadata to generate audio to predict unknown abnormalities, which solves the problem that the abnormal audio samples of the target machine are invisible to the existing abnormal sound detection model, resulting in the inability to select hyperparameters for the abnormal sound detection model and the limited performance of the abnormal sound detection system. The abnormal sample prediction strategy provided by the present invention uses the alignment of audio features and metadata information features to construct an audio generation method based on metadata information, and then predicts the abnormal sound characteristics of unknown target machine types, and generates predicted sound samples of the target machine that can be used for model training in the First-Shot scenario for the existing abnormal sound detection method. With the help of the predicted sound samples, the abnormal sound detection method can determine the optimal model in complex real-life scenarios for abnormal sound detection, thereby improving the performance and versatility of the abnormal sound detection method. At the same time, the unified sample generation model constructed by the present invention alleviates the problem of scarcity of abnormal samples in real-life scenarios, and effectively reduces the difficulty of industrial deployment of the abnormal sound detection method.

Description

Technical Field

[0001] The present invention relates to the technical field of using metadata to generate audio to predict unknown abnormal sounds, and in particular to a method for detecting unknown abnormal sounds by using metadata to generate audio to predict unknown abnormal sounds. Background Art

[0002] The purpose of the Anomalous Sound Detection (ASD) task is to automatically identify whether a target (such as a machine or device) makes an abnormal sound, thereby determining whether the target has abnormal behavior or status. However, since abnormal sounds rarely occur and are diverse, it is difficult to collect sound samples covering all abnormal situations for model training. To alleviate this problem, existing research has proposed abnormal sound detection in an unsupervised scenario, that is, only using normal sounds to train the abnormal sound detection model.

[0003] However, although the unsupervised scenario defines that only normal sounds can be used to train the abnormal sound detection model, the machine's abnormal sound samples can still be used to determine the model's hyperparameters (such as the number of training iterations, multi-classification loss weights, feature processing weights, etc.) during the training of the abnormal sound detection model, which is inconsistent with the actual scenario and makes the training process impossible to implement in actual applications. This is because the training of deep learning models requires training sets, test sets, and evaluation sets. In the training of abnormal sound detection models, both the test set and the evaluation set contain abnormal audio samples, so that many models select hyperparameters that make the model perform better based on the performance on the test set.

[0004] In order to meet the needs of real-world scenarios, the latest research proposes abnormal sound detection in the First-Shot scenario. The First-Shot scenario means that only the normal and abnormal sound samples of the reference machine, the normal sound samples of the target machine, and the metadata information corresponding to the sound samples are allowed to be used to train the abnormal sound detection model. The abnormal sound samples of the target machine are invisible to the model during the training phase, and the abnormal sound detection model is required to accurately identify the abnormal sound of the target machine. In the First-Shot scenario, the reference machine and the target machine are different, and the abnormal samples of the target machine are completely invisible, which avoids the abnormal samples of the target machine from leaking into the abnormal sound detection model training, and is more in line with unsupervised abnormal sound detection in real-world scenarios. However, in the First-Shot scenario, it is difficult for existing abnormal sound detection models to obtain accurate hyperparameters, and the accuracy of abnormal sound detection is low and the stability is poor. Summary of the invention

[0005] In order to solve the problem of limited performance of the abnormal sound detection system in the First-Shot scenario due to the invisible abnormal audio samples of the target machine, the present invention provides a method for detecting abnormal sounds by using metadata to generate audio to predict unknown abnormalities, which predicts the abnormal audio features of the target machine from the abnormal audio of the reference machine and the metadata information of the target machine, and then generates the corresponding abnormal audio, helping the existing abnormal sound detection model to predict hyperparameters during the training process, thereby improving the performance of the abnormal sound detection system in the First-Shot scenario. In addition, the present invention only uses the abnormal samples of the reference machine to fine-tune the generated model, and provides a unified abnormal feature prediction method for different acoustic target types, which alleviates the problem of difficulty in obtaining abnormal samples sufficient to train the abnormal detection model in real scenarios.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for detecting anomalies of unknown sounds by using metadata to generate audio, comprising the following steps:

[0008] S1: Introduce a cross-modal audio contrast learning strategy to build the potential connection between metadata information and audio features, obtain normal and abnormal sound samples of the reference machine, normal sound samples of the target machine, and metadata information corresponding to the sound samples, and train the audio generation model;

[0009] S2: Obtain metadata information of the target machine, generate metadata information of abnormal samples of the target machine based on the metadata information of normal samples of the target machine, and use the audio generation model to predict features of unknown abnormal audio from the metadata information of abnormal samples of the target machine and generate abnormal audio samples;

[0010] S3: Use the normal and abnormal sound samples of the reference machine, the normal sound samples of the target machine, the abnormal sound samples of the target machine generated by prediction, and the metadata information to train the abnormal sound detection system, so as to help the abnormal sound detection system determine its hyperparameter settings during the training process;

[0011] Preferably, the metadata information corresponding to each audio sample of the reference machine and the target machine consists of discrete attribute values; the metadata information is text information carried by the sound sample, describing the recording status of the sound sample and the operating status of the sound source.

[0012] Preferably, metadata information corresponding to each audio sample of the reference machine and the target machine is converted into its corresponding natural language description through a subtitle conversion algorithm, and used for extracting features of its metadata information by a cross-modal audio contrast learning strategy;

[0013] The subtitle conversion algorithm can be expressed as follows:

[0014] c=Fc (l1,l2,…,l n )

[0015] Among them, l1,l2,...,l n Respectively represent the discrete attribute values ​​that make up the metadata information, f c (·) represents the subtitle conversion algorithm, which is a natural language description of the audio sample obtained by the subtitle conversion algorithm from the metadata information.

[0016] Preferably, the subtitle conversion algorithm is: it integrates the metadata information corresponding to the audio sample into a natural language description of the audio sample using a fixed standard template; wherein the fixed standard template is designed for different machine types and different metadata information;

[0017] Since the pre-training data of the cross-modal audio contrast learning strategy consists of natural language descriptions, the ASD problem in the First-Shot scenario only has discrete attribute values, and the use of discrete attribute values ​​to align with audio features will affect the construction of their potential connections; therefore, the present invention designs a fixed standard template for the purpose of overcoming the distribution differences caused by the pre-training process of the cross-modal audio contrast learning strategy and improving the construction effect of the potential connection between metadata information and audio features;

[0018] As a preferred method, the metadata generates audio to predict unknown abnormal abnormal sound detection method is:

[0019] It adopts contrastive language-audio pre-training (CLAP) as a cross-modal audio contrastive learning strategy;

[0020] CLAP consists of a learnable audio feature extraction network, a learnable text feature extraction network, and an alignment loss function.

[0021] Preferably, the comparative language-audio pre-training strategy is:

[0022] Its audio feature extraction network is used to extract the audio features of the normal and abnormal sound samples of the reference machine and the normal sound samples of the target machine, and its text feature extraction network is used to extract the text features of the natural language description corresponding to the audio samples, which can be expressed as follows:

[0023] e a =f audio (a)

[0024] e c =f text (c)

[0025] Among them, a and c represent the sound sample and the natural language description corresponding to the sound sample, respectively, and f audio (·) and ftext (·) represent audio encoder and text encoder respectively, and Respectively represent the features corresponding to the sound sample a and the natural language description c of a;

[0026] The audio encoder and text encoder are implemented based on the HTSAT model and the RoBERTa model respectively.

[0027] Preferably, the comparative language-audio pre-training strategy is:

[0028] Its alignment loss function can be expressed as follows:

[0029]

[0030]

[0031]

[0032] Where τ represents the learnable temperature coefficient, D represents the batch size set during training, and N represents the number of samples under the machine type currently being trained;

[0033] L1 represents the function used to constrain the audio encoder f audio (·) alignment loss,

[0034] L2 represents the constraint used to constrain the text encoder f text (·) alignment loss;

[0035] During the training process, L1 and L2 in the alignment loss function are used to constrain the audio encoder f audio (·) and the text encoder f text (·), making the features of the two encoder outputs converge;

[0036] Since the metadata information carried by the natural language description is closely related to the characteristics of the sound sample, during inference, only the characteristics of the natural language description can be used to represent the overall characteristics of all the corresponding sound samples;

[0037] This constructs the potential connection between metadata information and machine audio features.

[0038] As a preferred method, by referring to the normal sound sample set A of the machine RN , abnormal sound sample set A of the reference machine RA , the target machine's normal sound sample set A TN and its corresponding metadata information set C RN , C RA , C TN To fine-tune the machine sound predictor The process can be expressed as follows:

[0039]

[0040] Among them, A and C represent the machine's sound sample set and the corresponding metadata information set, as shown below:

[0041]

[0042] Through fine-tuning of sound sample sets and metadata information sets, the machine sound predictor It can be constrained by metadata information, from the input natural language description features e c Generate predicted sound samples corresponding to the natural language description c

[0043] As a preferred embodiment, the machine sound predictor It is an audio waveform generator based on a denoising diffusion probability model, but can also be replaced by other text-based generative neural networks;

[0044] From the input natural language description features e c Generate predicted sound samples corresponding to the natural language description c The process can be expressed as follows:

[0045]

[0046] in, It is a set of normal type predicted sound samples obtained by taking the metadata information set of normal sound samples of the target machine as input, including the estimated normal sound samples of the target machine type. It is a set of abnormal type prediction sound samples obtained by taking the abnormal sound sample metadata information set of the target machine as input, and includes the estimated abnormal sound samples of the target machine type;

[0047] Since the abnormal sound samples of the target machine are invisible, their metadata information set is generated through the metadata information set of the normal sound samples of the target machine. This process only requires changing the "normal" in the metadata information set of the normal sound samples of the target machine to "abnormal" to obtain the metadata information of the abnormal sound samples of the target machine.

[0048] Benefited from contrastive speech-audio pre-training strategy and machine voice predictor The fine-tuning process of the machine sound predictor The differentiated features in the normal sound samples of the reference machine and the abnormal sound samples of the reference machine will be migrated to the process of predicting the sound samples of the target machine, helping to estimate the features of the unknown abnormal audio of the target machine and generate abnormal audio samples for subsequent abnormal sound detection model training.

[0049] As a preference, based on the normal predicted sound samples of the target machine The target machine's normal sound sample metadata information set C TN , abnormal prediction sound samples of the target machine and the abnormal sound sample metadata information set C of the target machine TA ,Will The abnormal samples of the target machine are used to predict the hyperparameters of the abnormal sound detection model. During the training process, a higher-performance abnormal sound detection model is selected to solve the problem of limited performance of the abnormal sound detection system in the First-Shot scenario due to the invisible abnormal audio samples of the target machine.

[0050] Audio feature extraction, metadata information feature extraction, audio feature and metadata information feature alignment, audio generation based on metadata information:

[0051] Firstly, all audio samples of the reference machine type and normal audio samples of the target machine type are passed through the Log-Mel spectral space filter group in the frequency data domain to obtain the Log-Mel spectral space features in the frequency domain. Then, the Log-Mel spectral space features are passed through a learnable audio feature extraction network to obtain the audio features.

[0052] Secondly, the metadata information corresponding to all samples of the reference machine type and normal samples of the target machine type is converted into a natural language description of the audio sample through a subtitle conversion algorithm, and then the natural language description is passed through a learnable text feature extraction network to obtain the metadata information features.

[0053] Subsequently, Contrastive Language-Audio Pretraining (CLAP) is used as a cross-modal audio contrast learning strategy to establish a connection between all audio samples of the reference machine type and the normal audio samples of the target machine type and the features of their corresponding metadata information. Since metadata information is carried by sound samples and describes the text information of the recording status of the sound samples and the operating status of the sound source, the connection between audio and metadata information constructed by CLAP can be used to retrieve and obtain the overall features of all audio samples corresponding to the metadata information from the metadata information features.

[0054] Finally, based on the connection between audio and metadata information constructed by CLAP, the metadata information of the target machine test sample is artificially drafted, and the metadata information features of the target machine test sample are obtained through the text feature extraction network. Finally, the predicted sound sample is generated from the metadata information features of the target machine test sample through the fine-tuned machine sound predictor.

[0055] For the existing abnormal sound detection model, because the original test set consists of sound samples of the reference machine, the predicted model hyperparameters cannot be used for the target machine. Therefore, the training process can use the predicted sound samples as the test set to predict the model hyperparameters, which improves the performance of the abnormal sound detection system in the First-Shot scenario and solves the problem of limited performance of the abnormal sound detection system in the First-Shot scenario due to the invisible abnormal audio samples of the target machine.

[0056] Beneficial effects:

[0057] In order to address the problem that in the First-Shot scenario, the abnormal audio samples of the target machine are invisible, which makes it difficult for the existing abnormal sound detection model to obtain accurate hyperparameters, thereby limiting the performance of the abnormal sound detection system, the present invention designs a cross-modal audio comparison learning strategy between the audio of the reference machine and the target machine and the corresponding metadata information. The potential connection between the metadata information and the machine audio features is mined and constructed through the description of the sound samples by the metadata information, and the metadata-related audio features of all audio samples with the same metadata information are accurately extracted.

[0058] At the same time, the present invention generates abnormal audio samples of the target machine through a machine sound predictor, which solves the problem that the abnormal audio samples of the target machine are invisible to the existing abnormal sound detection model, resulting in the inability to select hyperparameters for the abnormal sound detection model and the limited performance of the abnormal sound detection system.

[0059] In addition, the present invention only uses the abnormal samples of the reference machine to fine-tune the generation model, and can form a unified abnormal feature prediction method for different acoustic target types. It can not only predict the abnormal audio samples of the target machine, but also realizes a strategy of predicting the abnormal samples of the target machine through the normal samples of the target machine by using only one training of the abnormal samples of the reference machine. It constructs a unified generalized generation model, alleviates the problem of scarcity of abnormal samples, and effectively reduces the difficulty of industrial deployment of abnormal sound detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the present invention;

[0061] Figure 2 is a logic diagram of the present invention;

[0062] Figure 3It is a strategy logic diagram of cross-modal audio contrast learning of the present invention;

[0063] Figure 4 A schematic diagram of a machine sound predictor in the present invention generating an audio sample based on metadata information; DETAILED DESCRIPTION

[0064] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0065] The present invention comprises the following steps in the implementation process:

[0066] 1. Audio feature extraction and metadata information feature extraction

[0067] In the First-Shot scenario, the available audio and metadata information sets are the normal sound sample set A of the reference machine and RN , abnormal sound sample set A of the reference machine RA , the target machine's normal sound sample set A TN and its corresponding metadata information set C RN , C RA , C TN Assume that the original audio single-channel signal in any sound sample set is a, and the metadata information corresponding to the original audio single-channel signal is l=l1,l2,...,l n .

[0068] (1) Audio feature extraction

[0069] First, the original audio signal a is converted into a spectrogram feature (Spectrogram) through a short-time Fourier transform (STFT). The spectrogram contains all the frequency bands specified by the sampling frequency. Subsequently, the spectrogram feature is filtered through a Mel filter bank to obtain a Mel spectrogram. In order to amplify the interval of the spectral feature that is sensitive to human auditory perception, the Mel spectrogram is logarithmically scaled to obtain a frequency domain feature Log-Mel spectrogram. Finally, the frequency domain feature Log-Mel spectrogram is passed through a learnable feature extraction network to obtain the final audio feature e a The overall calculation process of obtaining audio features from the original audio signal can be summarized as follows:

[0070] e a =f audio (a)

[0071] Among them, f audio(·) represents a learnable audio encoder that integrates all the above audio signal processing processes. In the present invention, the audio encoder is implemented based on HTSAT, and other audio encoders can also be used to implement it.

[0072] (2) Metadata Information Feature Extraction

[0073] The metadata information corresponding to each audio sample of the machine consists of discrete attribute values. Assume that the metadata information corresponding to the original audio single-channel signal a is l = l1, l2, ..., l n For example, for an audio sample a in the training set whose machine type is ToyCar, the corresponding metadata information l is: "normal", "car_B2", "spd_31V" and "mic_1", then l1 is "normal", l2 is "car_B2", l3 is "spd_31V" and l4 is "mic_1".

[0074] First, the present invention introduces a subtitle conversion algorithm, that is, through an artificially constructed fixed standard template, the metadata information l is converted into a natural language description c of the audio sample a. The fixed standard template is designed for different machine types and different metadata information. For example, for ToyCar, its fixed standard template is "This is the [l1] sound of a toy car with model [l2] and speed [l3], recorded by a microphone placed at the position [l4]". The subtitle conversion algorithm can be expressed as follows:

[0075] c=F c (l1,l2,...,l n )

[0076] Among them, F c (·) indicates the subtitle conversion algorithm.

[0077] The natural language description obtained by the subtitle conversion algorithm facilitates the cross-modal audio contrast learning strategy to extract the features of its metadata information. This is because the pre-training data of the cross-modal audio contrast learning strategy consists of natural language descriptions, and the ASD problem in the First-Shot scenario targeted by the present invention only has discrete attribute values. Aligning discrete attribute values ​​with audio features will affect the construction of their potential connections. Therefore, an artificially defined fixed standard template is used in the present invention to overcome the distribution differences caused by the pre-training process of the cross-modal audio contrast learning strategy and to improve the construction effect of the potential connection between metadata information and audio features.

[0078] Then, the natural language description c is passed through a learnable text feature extraction network to extract the text features e of the natural language description c , which can be expressed as follows:

[0079] e c =f text (c)

[0080] Among them, f text (·) represents a learnable text encoder. In the present invention, the text encoder is implemented based on RoBERTa, and other text encoders can also be used.

[0081] 2. Alignment of audio features and metadata information features

[0082] In order to build a potential connection between an audio sample and its corresponding metadata information, and to facilitate the generation of a target machine prediction of abnormal audio samples through metadata information, the present invention introduces a comparative language-audio pre-training strategy, as shown in the accompanying figure Figure 2 The contrasting language-audio pre-training strategy extracts the features e of the sound sample a from the machine and the natural language description c corresponding to the sound sample through the learnable audio encoder and text encoder. a and the features described in natural language c The L1 and L2 in the alignment loss function are used to constrain the audio encoder f audio (·) and the text encoder f text (·) makes the features of the two encoder outputs converge. The alignment loss function can be expressed as follows:

[0083]

[0084]

[0085]

[0086] Where τ represents the learnable temperature coefficient, D represents the batch size set during training, and N represents the number of samples under the machine type currently being trained. L1 represents the number of samples used to constrain the audio encoder f audio (·) alignment loss, L2 represents the loss used to constrain the text encoder f text Alignment loss of (·)

[0087] Since the metadata information carried by the natural language description is closely related to the characteristics of the sound sample, during inference, only the characteristics of the natural language description can be used to represent the overall characteristics of all the corresponding sound samples, thereby constructing a potential connection between the metadata information and the machine audio characteristics.

[0088] 3. Audio generation based on metadata information

[0089] The present invention adopts a machine sound predictor based on metadata text features e c , generate predicted sound samples, as shown in the attached figure Figure 3 In the present invention, the machine sound predictor is an audio waveform generator based on a denoising diffusion probability model, which is composed of a denoising diffusion probability model, a variational autoencoder and a speech encoder, but can also be replaced by other feature-based generative neural networks.

[0090] (1) Fine-tuning the machine sound predictor

[0091] In order to make the audio samples generated by the machine sound predictor match the characteristic distribution of the abnormal sound detection audio data, the present invention refers to the normal sound sample set A of the machine. RN , abnormal sound sample set A of the reference machine RA , the target machine's normal sound sample set A TN and its corresponding metadata information set C RN , C RA , C TN To fine-tune the machine sound predictor Machine Sound Predictor is an audio waveform generator based on a denoising diffusion probability model, but it can also be replaced by other text-based generative neural networks. The fine-tuning process can be expressed as follows:

[0092]

[0093] Among them, A and C represent the machine's sound sample set and the corresponding metadata information set, as shown below:

[0094]

[0095] Through fine-tuning of sound sample sets and metadata information sets, the machine sound predictor It can be constrained by metadata information, from the input natural language description features e c Generate predicted sound samples corresponding to the natural language description c

[0096] (2) Predict abnormal samples of the target machine

[0097] Machine Sound Predictor From the input natural language description features e c Generate predicted sound samples corresponding to the natural language description c The process can be expressed as follows:

[0098]

[0099] in, It is a set of predicted sound samples obtained by taking the metadata information set of the normal sound samples of the target machine as input. It is a set of predicted sound samples obtained by taking the target machine's abnormal sound sample metadata information set as input. Since the target machine's abnormal sound samples are invisible, the metadata information set used to describe the target machine's abnormal sound samples is generated from the target machine's normal sound sample metadata information set. This process only requires changing "normal" in the target machine's normal sound sample metadata information set to "abnormal" to obtain the target machine's abnormal sound sample metadata information, thereby generating unknown target machine type abnormal sound sample data.

[0100] Benefited from contrastive speech-audio pre-training strategy and machine voice predictor The fine-tuning process of the machine sound predictor The differentiated features in the normal sound samples of the reference machine and the abnormal sound samples of the reference machine will be migrated to the process of predicting the sound samples of the target machine, which will help predict the features of the unknown abnormal audio of the target machine and generate abnormal audio samples for subsequent abnormal sound detection model training.

[0101] Embodiment 1: Combination Figure 1 -Attached Figure 4 A method for detecting abnormal sounds that uses metadata to generate audio to predict unknown abnormalities is described. A cross-modal audio contrast learning strategy is introduced to construct the potential connection between metadata information and audio features, and is used to predict the features of unknown abnormal audio from the metadata information of the target machine and generate abnormal audio samples. This solves the problem of limited performance of the abnormal sound detection system in the First-Shot scenario due to the invisible abnormal audio samples of the target machine.

[0102] In the present invention, the First-Shot scenario refers to the scenario in which only the normal and abnormal sound samples of the reference machine, the normal sound samples of the target machine, and the metadata information corresponding to the sound samples are allowed to be used to train the abnormal sound detection system, while the abnormal sound samples of the target machine are invisible to the model during the training phase, and the abnormal sound detection system is required to accurately identify the abnormal sound of the target machine. The First-Shot scenario reflects the difficulty and challenge of the abnormal sound detection task in practical applications, which is the lack of abnormal sound samples of the target machine. During the training of existing abnormal sound detection systems, abnormal sound samples of the target machine are often used to determine the hyperparameters of the model. However, in the First-Shot scenario, it is difficult to obtain accurate hyperparameters, and the accuracy of abnormal sound detection is low and the stability is poor.

[0103] While making full use of the audio of the reference machine and the target machine and the corresponding metadata information, the present invention benefits from the potential connection between the constructed metadata information and the machine audio features, and realizes the prediction of the abnormal audio features of the target machine from the abnormal audio of the reference machine and the metadata information of the target machine, and then generates the corresponding abnormal audio, helping the existing abnormal sound detection system to infer hyperparameters during the training process, thereby improving the performance of the abnormal sound detection system in the First-Shot scenario.

[0104] Embodiment 2: Combination with Figure 1 -Attached Figure 4 For example, the metadata information corresponding to each audio sample of the machine consists of discrete attribute values. Metadata information is carried by the sound sample, which describes the text information of the sound sample recording status and the operation status of the sound source. For example, the metadata information corresponding to a machine sound sample of ToyCar may include: "normal", "car_B2" (car model B2), "spd_31V" (vehicle speed 31) and "mic_1" (microphone at position 1).

[0105] Embodiment 3: Combination Figure 1 -Attached Figure 4 To illustrate, the metadata information l corresponding to each audio sample of the machine is converted into its corresponding natural language description c through the subtitle conversion algorithm, so that the cross-modal audio contrast learning strategy can extract the features of its metadata information. The subtitle conversion algorithm can be expressed as follows:

[0106] c=F c (l1,l2,...,l n )

[0107] Among them, l1,l2,...,l n Respectively represent the discrete attribute values ​​that make up the metadata information, F c(·) represents the subtitle conversion algorithm, and c is the natural language description of the audio sample obtained by the subtitle conversion algorithm from the metadata information.

[0108] Embodiment 4: Combination Figure 1 -Attached Figure 4 To illustrate, the subtitle conversion algorithm is to integrate the metadata information l corresponding to the audio sample into a natural language description c of the audio sample using a fixed standard template. The fixed standard template is designed for different machine types and different metadata information. For example, for ToyCar, the fixed standard template is "This is the [l1] sound of a toy car with model [l2] and speed [l3], recorded by a microphone placed at the position [l4]" (This is the [l1] sound of a toy car with model [l2] and speed [l3], recorded by a microphone placed at the position [l4]).

[0109] Since the pre-training data of the cross-modal audio contrastive learning strategy consists of natural language descriptions, and the ASD problem in the First-Shot scenario targeted by the present invention only has discrete attribute values, aligning discrete attribute values ​​with audio features will affect the construction of their potential connections. Therefore, the present invention designs a fixed standard template as shown in the above example to overcome the distribution differences caused by the pre-training process of the cross-modal audio contrastive learning strategy and improve the construction effect of the potential connection between metadata information and audio features.

[0110] Embodiment 5: Combination Figure 1 -Attached Figure 4 To illustrate, the method for detecting unknown anomalies by using metadata to generate audio to predict unknown anomalies adopts Contrastive Language-Audio Pretraining (CLAP) as a cross-modal audio contrast learning strategy. CLAP consists of a learnable audio feature extraction network, a learnable text feature extraction network, and an alignment loss function.

[0111] Embodiment 6: Combination Figure 1 -Attached Figure 4 To illustrate, the comparative language-audio pre-training strategy is that its audio feature extraction network is used to extract the audio features of normal and abnormal sound samples of the reference machine and the normal sound samples of the target machine, and its text feature extraction network is used to extract the text features of the natural language description corresponding to the audio samples, which can be expressed as follows:

[0112] e a =f audio (a)

[0113] e c =f text (c)

[0114] Among them, a and c represent the sound sample and the natural language description corresponding to the sound sample, respectively, and f audio (·) and f text (·) represent audio encoder and text encoder respectively, and They represent the features corresponding to the sound sample a and the natural language description c of a respectively. In the present invention, the audio encoder and the text encoder are implemented based on the HTSAT model and the RoBERTa model respectively, and other encoder models can also be used for implementation.

[0115] Example 7: Combination Figure 1 -Attached Figure 4 To illustrate, the comparative language-audio pre-training strategy is that its alignment loss function can be expressed as follows:

[0116]

[0117]

[0118]

[0119] Where τ represents the learnable temperature coefficient, D represents the batch size set during training, and N represents the number of samples under the machine type currently being trained. L1 represents the number of samples used to constrain the audio encoder f audio (·) alignment loss, L2 represents the loss used to constrain the text encoder f text (·) alignment loss.

[0120] During the training process, L1 and L2 in the alignment loss function are used to constrain the audio encoder f audio (·) and the text encoder f text (·) makes the features of the two encoders output similar. Since the metadata information carried by the natural language description is closely related to the features of the sound samples, during inference, only the features of the natural language description can be used to represent the overall features of all the corresponding sound samples. This constructs a potential connection between metadata information and machine audio features.

[0121] Embodiment 8: Combination Figure 1 -Attached Figure 4 For explanation, the normal sound sample set A of the reference machine RN , abnormal sound sample set A of the reference machine R x, the target machine's normal sound sample set A TN and its corresponding metadata information set CRN , C RA , C TN To fine-tune the machine sound predictor The process can be expressed as follows:

[0122]

[0123] Among them, A and C represent the machine's sound sample set and the corresponding metadata information set, as shown below:

[0124]

[0125] Through fine-tuning of sound sample sets and metadata information sets, the machine sound predictor It can be constrained by metadata information, from the input natural language description features e c Generate predicted sound samples corresponding to the natural language description c

[0126] Embodiment 9: Combination Figure 1 -Attached Figure 4 The machine sound predictor is described as follows: It is an audio waveform generator based on a denoising diffusion probability model, but can be replaced by other text-based generative neural networks. From the input natural language description features e c Generate predicted sound samples corresponding to the natural language description c The process can be expressed as follows:

[0127]

[0128] in, The normal type predicted sound samples are obtained by taking the normal sound sample metadata information set of the target machine as input, and the normal sound samples of the target machine type predicted by the present invention are included. The set of abnormal type predicted sound samples obtained by taking the abnormal sound sample metadata information set of the target machine as input includes the abnormal sound samples of the target machine type predicted by the present invention. Since the abnormal sound samples of the target machine are invisible, its metadata information set is generated by the metadata information set of the normal sound samples of the target machine. In this process, only "normal" in the metadata information set of the normal sound samples of the target machine needs to be modified to "abnormal" to obtain the metadata information of the abnormal sound samples of the target machine.

[0129] Benefited from contrastive speech-audio pre-training strategy and machine voice predictor The fine-tuning process of the machine sound predictor The differentiated features in the normal sound samples of the reference machine and the abnormal sound samples of the reference machine will be migrated to the process of predicting the sound samples of the target machine, which will help predict the features of the unknown abnormal audio of the target machine and generate abnormal audio samples for subsequent abnormal sound detection model training.

[0130] Embodiment 10: Combination with Figure 1 -Attached Figure 4 To illustrate, the normal prediction sound sample based on the target machine The target machine's normal sound sample metadata information set C TN , abnormal prediction sound samples of the target machine and the abnormal sound sample metadata information set C of the target machine TA ,Will The abnormal samples of the target machine are used to infer the hyperparameters of the abnormal sound detection model, which helps to select a higher-performance abnormal sound detection model during the training process. This solves the problem of limited performance of the abnormal sound detection system in the First-Shot scenario due to the invisible abnormal audio samples of the target machine.

[0131] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0132] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for detecting abnormal sounds by using metadata to generate audio to predict unknown anomalies, characterized in that: The following steps are involved: S1: Introduce a cross-modal audio contrast learning strategy to build the potential connection between metadata information and audio features, obtain normal and abnormal sound samples of the reference machine, normal sound samples of the target machine, and metadata information corresponding to the sound samples, and train the audio generation model; S2: Obtain metadata information of the target machine, generate metadata information of abnormal samples of the target machine based on the metadata information of normal samples of the target machine, and use the audio generation model to predict features of unknown abnormal audio from the metadata information of abnormal samples of the target machine and generate abnormal audio samples; S3: Use the normal and abnormal sound samples of the reference machine, the normal sound samples of the target machine, the abnormal sound samples of the target machine generated by prediction, and the metadata information to train the abnormal sound detection system, so as to help the abnormal sound detection system determine its hyperparameter settings during the training process.

2. According to claim 1, a method for detecting abnormal sound by using metadata to generate audio to predict unknown anomalies is characterized in that: The metadata information corresponding to each audio sample of the reference machine and the target machine consists of discrete attribute values; Metadata information is text information carried by the sound sample, which describes the recording status of the sound sample and the operating status of the sound source.

3. According to claim 2, a method for detecting abnormal sound by using metadata to generate audio to predict unknown abnormalities is characterized in that: The metadata information corresponding to each audio sample of the reference machine and the target machine is converted into its corresponding natural language description through a subtitle conversion algorithm, which is used for extracting the features of its metadata information by a cross-modal audio contrast learning strategy; The subtitle conversion algorithm is expressed as follows: c=F c (l1,l2,…,l n ) Among them, l1,l2,...,l n Respectively represent the discrete attribute values ​​that make up the metadata information, F c (·) represents the subtitle conversion algorithm, and c is the natural language description of the audio sample obtained by the subtitle conversion algorithm from the metadata information.

4. According to claim 3, a method for detecting abnormal sound by using metadata to generate audio to predict unknown anomalies is characterized in that: The subtitle conversion algorithm is: it integrates the metadata information corresponding to the audio sample into a natural language description of the audio sample with a fixed standard template; wherein the fixed standard template is designed for different machine types and different metadata information.

5. According to claim 4, a method for detecting abnormal sound by using metadata to generate audio to predict unknown anomalies is characterized in that: The metadata-generated audio predicts the unknown abnormal sound detection method as follows: It adopts contrastive language-audio pre-training as a cross-modal audio contrastive learning strategy; Contrastive language-audio pre-training consists of a learnable audio feature extraction network, a learnable text feature extraction network, and an alignment loss function.

6. According to claim 5, a method for detecting abnormal sound by using metadata to generate audio to predict unknown anomalies is characterized in that: The contrastive language-audio pre-training strategy is: Its audio feature extraction network is used to extract the audio features of the normal and abnormal sound samples of the reference machine and the normal sound samples of the target machine, and its text feature extraction network is used to extract the text features of the natural language description corresponding to the audio samples, which are respectively expressed as follows: have been a =f audio (a) e c =f text (c) Among them, a and c represent the sound sample and the natural language description corresponding to the sound sample, respectively, and f audio (·) and f text (·) represent audio encoder and text encoder respectively, and Respectively represent the features corresponding to the sound sample a and the natural language description c of a; The audio encoder and text encoder are implemented based on the HTSAT model and the RoBERTa model respectively.

7. The method for detecting abnormal sound by using metadata to generate audio to predict unknown abnormalities according to claim 6, characterized in that: The contrastive language-audio pre-training strategy is: Its alignment loss function is expressed as follows: Where τ represents the learnable temperature coefficient, D represents the batch size set during training, and N represents the number of samples under the machine type currently being trained; L1 represents the function used to constrain the audio encoder f audio (·) alignment loss, L2 represents the constraint used to constrain the text encoder f text (·) alignment loss; During the training process, L1 and L2 in the alignment loss function are used to constrain the audio encoder f audio (·) and the text encoder F text (·), making the features of the two encoder outputs converge; The metadata information carried by the natural language description is closely related to the characteristics of the sound sample. When making judgments, only the characteristics of the natural language description are used to represent the overall characteristics of all the corresponding sound samples; This constructs the potential connection between metadata information and machine audio features.

8. The method for detecting abnormal sound by using metadata to generate audio to predict unknown abnormalities according to claim 7, characterized in that: By referring to the normal sound sample set A of the machine RN , abnormal sound sample set A of the reference machine RA , the target machine's normal sound sample set A TN and its corresponding metadata information set C RN , C RA , C TN To fine-tune the machine sound predictor The process is represented as follows: Among them, A and C represent the machine's sound sample set and the corresponding metadata information set, as shown below: Through fine-tuning of the sound sample set and metadata information set, the machine sound predictor Constrained by metadata information, the input natural language description features e c Generate predicted sound samples corresponding to the natural language description c 9. The method for detecting abnormal sound by using metadata to generate audio to predict unknown abnormalities according to claim 8, characterized in that: Its machine sound predictor It is an audio waveform generator based on a denoising diffusion probability model, but can also be replaced by other text-based generative neural networks; From the input natural language description features e c Generate predicted sound samples corresponding to the natural language description c The process can be expressed as follows: in, It is a set of normal type predicted sound samples obtained by taking the metadata information set of normal sound samples of the target machine as input, including the estimated normal sound samples of the target machine type. It is a set of abnormal type prediction sound samples obtained by taking the abnormal sound sample metadata information set of the target machine as input, and includes the estimated abnormal sound samples of the target machine type; Since the abnormal sound samples of the target machine are invisible, their metadata information set is generated from the metadata information set of the normal sound samples of the target machine. In this process, only "normal" in the metadata information set of the normal sound samples of the target machine needs to be modified to "abnormal" to obtain the metadata information of the abnormal sound samples of the target machine. Comparing Language-Audio Pretraining Strategies and Machine Voice Predictors The fine-tuning process of the machine sound predictor The differentiated features in the normal sound samples of the reference machine and the abnormal sound samples of the reference machine will be migrated to the process of predicting the sound samples of the target machine, helping to estimate the features of the unknown abnormal audio of the target machine and generate abnormal audio samples for subsequent abnormal sound detection model training.

10. The method for detecting abnormal sound by using metadata to generate audio to predict unknown abnormalities according to claim 9, characterized in that: Normal predicted sound samples based on the target machine Normal sound sample metadata information set C of the target machine TN , abnormal prediction sound samples of the target machine And the abnormal sound sample metadata information set C of the target machine TA ,Will The abnormal samples regarded as the standard machine are used to predict the hyperparameters of the abnormal sound detection model, and the abnormal sound detection model with higher performance is selected during the training process.

Images