System and method for unsupervised abnormal sound detection

Abstract

Systems and methods for detecting abnormal sound are disclosed. The method includes receiving an audio signal from a sound source in a recording environment. The sound source and the recording environment are characterized by a set of attributes including a first attribute related to a first attribute type and a second attribute related to a second attribute type. A multi-head neural network is trained to extract a first embedding vector indicating a first attribute type and a second embedding vector indicating a second attribute type from a received audio signal. The first embedded vector is compared to the first set of embedded vectors to classify attributes of the first attribute type, and the second embedded vector is compared to the second set of embedded vectors to classify attributes of the second attribute type to determine a result of anomaly detection.

Description

Technical Field

[0001] The present disclosure relates generally to sound processing and, more particularly, to detecting anomalies in audio signals and providing explanations for the detected anomalies. Background Art

[0002] Diagnosing and monitoring machine operating performance is important for a wide variety of applications. These diagnostic and monitoring operations are often performed manually by technicians. For example, a technician may listen to and analyze the sounds generated by a machine to identify abnormal sounds. This manual process of analyzing the sounds can be automated to process the machine's sound signals and detect abnormal sounds within them.

[0003] In some scenarios, automated sound diagnosis based on deep learning techniques can be used to detect abnormal sounds. Typically, automated sound diagnosis can be used to detect abnormal sounds using training data corresponding to normal operating conditions for sound diagnosis. Abnormal sound detection based on such training data is an unsupervised method. Unsupervised abnormal sound detection can be suitable for detecting specific types of anomalies, such as sudden transient disturbances or impulsive sounds, which can be detected based on sudden temporal changes.

[0004] Unsupervised anomalous sound detection is the problem of learning a model that can detect anomalies when only data under normal operating conditions is available to train the model parameters. Typical applications are condition monitoring and diagnostics of machine sounds in applications such as predictive maintenance and factory automation. Typical approaches for unsupervised anomalous sound detection include those based on autoencoder-like architectures, where a model trained only on normal data to reconstruct its input should exhibit large reconstruction errors when presented with anomalous examples at inference. Another class of approaches, called surrogate task models, uses an alternative supervised training task to learn a normal model and then measures deviations from the normal to predict anomalies. Example surrogate tasks include: (1) outlier exposure, where sounds known to be significantly different from the machine of interest are used as synthetic anomalies, (2) predicting metadata (e.g., machine instance) or attributes (e.g., operating load), or (3) learning to predict which enhancements (e.g., time stretching or pitch shifting) have been applied to an audio clip.

[0005] A major drawback of existing abnormal sound detection systems is their failure in the presence of domain shift, which occurs when the acoustic characteristics of normal data collected for training and normal data collected during inference change due to factors such as different background noise or different operating voltages. This failure is often caused by algorithms that cannot distinguish between unexpected signal changes caused by abnormal sounds and expected signal changes caused by domain shift.

[0006] Therefore, there is a need to overcome the above-mentioned problems associated with abnormal sound detection under domain shift conditions. More specifically, there is a need to develop a method and system for detecting abnormal sounds in audio signals in an efficient and feasible manner. Summary of the Invention

[0007] Various embodiments of the present disclosure disclose systems and methods for detecting abnormal sounds in audio signals. Some embodiments aim to use deep learning techniques for abnormal sound detection.

[0008] Some embodiments aim to provide unsupervised abnormal sound detection by learning a model that can detect anomalies when only data under normal operating conditions is available for training the model parameters. Typical applications are condition monitoring and diagnostics of machine sounds in applications such as predictive maintenance and factory automation.

[0009] A major drawback of existing anomalous sound detection algorithms is that they fail when presented with domain shift, i.e., acoustic features that change between the normal data collected for training and the normal data collected at inference time due to factors such as different background noise, different operating voltages, etc. This failure is often caused by the algorithm being unable to distinguish between unexpected signal changes caused by anomalous sounds and expected signal changes caused by domain shift.

[0010] Typical approaches for unsupervised anomalous sound detection include those based on autoencoder-like architectures, where a model trained only on normal data to reconstruct its input should exhibit large reconstruction errors when presented with anomalous examples at inference. Another class of approaches, referred to herein as surrogate task models, uses surrogate supervised training tasks to learn a model of normality and then measures deviations from normality to predict anomalies. Example surrogate tasks include: (1) outlier exposure, where sounds known to be significantly different from the machine of interest are used as synthetic anomalies, (2) predicting metadata (e.g., machine instance) or attributes (e.g., operational load), or (3) learning to predict which enhancements (e.g., time stretching or pitch shifting) have been applied to an audio clip.

[0011] Some embodiments are based on the recognition that unsupervised abnormal sound detection using surrogate task models can be adapted to specific operating scenarios by selecting specific surrogate tasks and can therefore be more robust in specific applications than approaches based on autoencoder-like architectures. However, in some scenarios, selecting a specific surrogate task for a specific application may be impractical. Furthermore, even when surrogate task models are developed for specific advantageous tasks, the models still suffer from domain shift.

[0012] To this end, some embodiments aim to provide a system and method for unsupervised anomalous sound detection that is robust to domain shift issues. Additionally or alternatively, some embodiments aim to provide an alternative task model approach to learn a normality model that is resilient to domain shift issues. Additionally or alternatively, some embodiments aim to provide an anomalous sound detection system configured to perform anomaly detection and applications that benefit from such detection.

[0013] Some embodiments are based on the understanding that the surrogate task model approach can be extended to consider not just one surrogate task, but multiple surrogate tasks. In other words, the approach can be extended to learn normality models for multiple surrogate tasks. In theory, since two tasks are better than one, the approach is more resilient to domain shift. However, some embodiments are based on the understanding, supported by experiments, that only the surrogate task model approach can be extended to consider

[0014] Some embodiments are based on the recognition that to increase the resilience of normal models to domain shift, two conditions need to be met: (1) embeddings for classifying multiple surrogate tasks need to be generated together, i.e., interdependent, but (2) these embeddings need to be analyzed separately to detect anomalies. The requirement of interdependent generation allows for learning rich embeddings, since information about some surrogate tasks alone is insufficient to learn a strong model. Separate testing decouples the surrogate tasks to gain resilience to domain shift.

[0015] Some embodiments are based on the recognition that these two conditions can be met when a multi-headed neural network with one input and multiple outputs generates embedding vectors for different alternative tasks, while the outputs of the multi-headed neural network are analyzed separately. For example, one embodiment utilizes a multi-headed neural network to process an audio signal, and the multi-headed neural network is trained to extract a first embedding vector indicating a first attribute type and a second embedding vector indicating a second attribute type from the received audio signal. Because the first type of attributes and the second type of attributes are different from each other, they can be adapted by different embodiments to classify different alternative tasks, and therefore the first condition can be met. An embodiment of the multi-headed neural network includes a convolutional neural network module connected to a plurality of thin output layers, the plurality of thin output layers including a first output layer for outputting attributes of the first attribute type and a second output layer for outputting attributes of the second attribute type.

[0016] Next, some embodiments compare the first embedding vector with a first set of normal embedding vectors to generate a first anomaly score, and compare the second embedding vector with a second set of normal embedding vectors to generate a second anomaly score. An anomaly detection result is determined based on one or a combination of the first anomaly score and the second anomaly score, thereby satisfying the second independent evaluation condition.

[0017] Therefore, one embodiment discloses an abnormal sound detection system, comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the abnormal sound detection system to receive an audio signal generated by a sound source in a recording environment, wherein the sound source and the recording environment are characterized by an attribute set, the attribute set comprising a first attribute belonging to a first attribute type and a second attribute belonging to a second attribute type. The received audio signal is processed using a multi-headed neural network trained to extract a first embedding vector indicating the first attribute type and a second embedding vector indicating the second attribute type from the received audio signal. The first embedding vector is compared with a first set of normal embedding vectors previously generated by the multi-headed neural network to classify attributes of the first attribute type, and the second embedding vector is compared with a second set of normal embedding vectors previously generated by the multi-headed neural network to classify attributes of the second attribute type, thereby determining an abnormality detection result. The abnormal sound detection system is then configured to render the abnormality detection result.

[0018] Another embodiment discloses a computer-implemented method for abnormal sound detection, the method comprising: receiving an audio signal generated by a sound source in a recording environment, wherein the sound source and the recording environment are characterized by an attribute set, the attribute set comprising a first attribute belonging to a first attribute type and a second attribute belonging to a second attribute type. The computer-implemented method further comprises processing the received audio signal using a multi-headed neural network, the multi-headed neural network being trained to extract a first embedding vector indicating the first attribute type and a second embedding vector indicating the second attribute type from the received audio signal. The computer-implemented method further comprises: comparing the first embedding vector with a first normal embedding vector set previously generated by the multi-headed neural network to classify attributes of the first attribute type, and comparing the second embedding vector with a second normal embedding vector set previously generated by the multi-headed neural network to classify attributes of the second attribute type, thereby determining an abnormality detection result; and rendering the abnormality detection result.

[0019] Further features and advantages will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] [ Figure 1A ]

[0021] Figure 1A A recording environment for detecting abnormal sounds according to an embodiment of the present disclosure is shown.

[0022] [ Figure 1B ]

[0023] Figure 1B A schematic diagram illustrating a set of properties of a sound source and a recording environment according to an embodiment of the present disclosure is shown.

[0024] [ Figure 2 ]

[0025] Figure 2 A block diagram illustrating the operation of a multi-headed neural network according to an embodiment of the present disclosure is shown.

[0026] [ Figure 3A ]

[0027] Figure 3A A schematic diagram illustrating the operation of a training phase of an abnormal sound detection system according to an embodiment of the present disclosure is shown.

[0028] [ Figure 3B ]

[0029] Figure 3B A schematic diagram illustrating a test phase operation of an abnormal sound detection system according to an embodiment of the present disclosure is shown.

[0030] [ Figure 4 ]

[0031] Figure 4 A schematic diagram illustrating a process of determining an abnormality score according to an exemplary embodiment of the present disclosure is shown.

[0032] [ Figure 5A ]

[0033] Figure 5A A schematic diagram illustrating an exemplary method of anomaly score generation according to an exemplary embodiment of the present disclosure is shown.

[0034] [ Figure 5B ]

[0035] Figure 5B A block diagram illustrating another method of determining a combined anomaly score while facilitating a learning weighting approach according to an embodiment of the present disclosure is shown.

[0036] [ Figure 6 ]

[0037] Figure 6 A schematic diagram illustrating a use case for collecting a training dataset for embedding vectors for training a multi-head neural network according to an embodiment of the present disclosure is shown.

[0038] [ Figure 7 ]

[0039] Figure 7 A flow chart depicting a method for detecting abnormal sound signals according to an embodiment of the present disclosure is shown.

[0040] [ Figure 8 ]

[0041] Figure 8 A block diagram illustrating a hardware framework of an abnormal sound detection system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it is apparent to those skilled in the art that the present disclosure can be practiced without these specific details. In other cases, devices and methods are shown in block diagram form only to avoid obscuring the present disclosure. Various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the disclosed subject matter set forth in the appended claims.

[0043] As used in this specification and claims, the terms "for example," "for example," and "such as," and the verbs "include," "have," "comprise," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, are each to be interpreted as open-ended, meaning that the list is not to be construed as excluding other additional components or items. The term "based on" means based, at least in part, on. Furthermore, it should be understood that the phraseology and terminology employed herein are for descriptive purposes only and should not be considered limiting. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0044] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, the systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary details. In other cases, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments. In addition, the same reference numerals and symbols in the various figures indicate the same elements.

[0045] System Overview

[0046] Figure 1AA recording environment 100 for detecting unusual sounds according to an embodiment of the present disclosure is shown. Recording environment 100 includes an unusual sound detection system 102, which is coupled to a sound source 108 that generates an audio signal 110. The unusual sound detection system 102 includes a processor 104 and a memory 106. Memory 106 is configured to store instructions for generating anomaly detection results 114, which are used to detect unusual sounds. Unusual sounds include unusual sound signals that are not typically expected to occur during operation. For example, in a manufacturing process, the operational sounds associated with a machine may include some normal sounds and some abnormal or unusual sounds. For example, a crashing sound encountered during machine operation may indicate a malfunction or wear on a machine component and may therefore be considered an unusual sound. Some embodiments aim to use the unusual sound detection system 102 to detect such unusual sounds. Furthermore, the detection of unusual sounds can be used to predict a malfunction associated with the machine. For example, the specific source 108 of the unusual sound detected or unusual is a malfunctioning part of the machine. The operations of the abnormal sound detection system 102 may be embodied in the form of instructions that are stored in the memory 106 and executed by the processor 104 .

[0047] In some embodiments, the memory 106 is configured to store instructions for implementing the multi-headed neural network 112 to facilitate abnormal sound detection. The memory 106 corresponds to at least one of RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassette, magnetic tape, magnetic disk storage, or any other storage medium that can be used to store desired information and can be accessed by the abnormal sound detection system 102. The memory 106 includes non-transitory computer storage media in the form of volatile and / or non-volatile memory. The memory 106 can be removable, non-removable, or a combination thereof. Exemplary memory devices include solid-state memory, hard drives, optical drives, and the like.

[0048] The abnormal sound detection system 102 is configured to receive an audio signal 110 generated by a sound source 108 in a recording environment 100. The recording environment 100 can correspond to an application-specific environment, such as a manufacturing plant, a vehicle, a studio, etc. The audio signal 110 can correspond to non-stationary sounds, such as the sounds of operating machinery, the sounds of a working engine, etc. The audio signal 110 can be converted into a representation in the time-frequency domain, such as a spectrogram. Typically, a spectrogram includes elements defined by values, such as pixels in the time-frequency domain. Each value of each element is identified by a coordinate in the time-frequency domain. For example, time frames in the time-frequency domain are represented as columns, and frequency bands in the time-frequency domain are represented as rows.

[0049] The audio signal 110 is associated with a sound generated from a sound source 108. In the case of a manufacturing setting, the sound source 108 corresponds to at least one of a machine, an electrical device, or an engine. For example, the sound source 108 may include a drill, a grinder, a packaging machine, or the like.

[0050] The sound source 108 and the recording environment 100 are characterized by a set of attributes. Figure 1B A schematic diagram illustrating a sound source 108 and a set of properties 116 of a recording environment 100 is shown.

[0051] For the sound source 108, the attribute set 116 includes at least a first attribute 118 belonging to a first attribute type 118a. For the recording environment 100, the attribute set 116 also includes at least a second attribute 120 belonging to a second attribute type 120a.

[0052] To this end, in one embodiment, the first attribute type and the second attribute type are selected such that the classification of the attributes of first attribute type 118a is independent of one or a combination of the type of recording environment 100 or sound source 108, or the operating state of sound source 108. On the other hand, the classification of the attributes of second attribute type 120a is dependent on one or a combination of the type of recording environment 100 or sound source 108, or the operating state of sound source 108. That is, first attribute type 118a includes attributes that make the classification of the attributes of first attribute type 118a independent of one or a combination of the following: recording environment 100; the type of sound source 108; and the operating state of sound source 108, and second attribute type 120a includes attributes that make the classification of the other attributes of second attribute type 120a dependent on one or a combination of the recording environment 100, the type of sound source 108, and the operating state of sound source 108.

[0053] For example, first attribute 118 of first attribute type 118a is associated with sound source 108, e.g., classifying a motor based on the sound signal. In practice, the motor and / or motor type remains the same regardless of the environment in which the motor operating sound is recorded. Therefore, first attribute 118 is the sound of the motor, and first attribute type 118a can be either a normal sound or an abnormal sound.

[0054] Furthermore, in the above embodiment, the second attribute 120 of the second attribute type 120a may indicate the operating state of the motor. To this end, the motor includes different characteristics related to the operating state of the motor. Characteristics include input characteristics provided to the motor, such as the operating voltage required by the motor. Characteristics also include motor output characteristics, such as the torque generated by the motor. For example, when the motor is operating at low speed, the second attribute 120 may be the state S of the motor. i , and the second attribute type may be for the state S iIn another embodiment, the second attribute type 120a is the torque generated by the motor, which can be changed from state S to state S during operation of the motor. i Change to another state.

[0055] The audio signal 110 is thus a combination of the attribute set 116 including the first attribute 118 of the first attribute type 118a and the second attribute 120 of the second attribute type 120a. The abnormal sound detection system 102 is configured to process the audio signal 110, which is characterized by the attribute set 116 including the first attribute 118 of the first attribute type 118a and the second attribute 120 of the second attribute type 120a, using the multi-headed neural network 112.

[0056] Figure 2 A block diagram 200 illustrating the operation of the multi-headed neural network 112 according to an embodiment of the present disclosure is shown. The multi-headed neural network 112 is configured to process the audio signal 110 (which may be a test audio signal or a received audio signal at the inference time) using one or more linear layers (a first output layer 112a and a second output layer 112b) as thin output layers. To this end, in some implementations, the multi-headed neural network 112 includes a convolutional neural network module connected to a plurality of thin output layers (such as the first output layer 112a and the second output layer 112b). It will be understood that for illustrative purposes only, the multi-headed neural network 112 is configured to process the audio signal 110 (which may be a test audio signal or a received audio signal at the inference time). Figure 2 The number of thin output layers is shown as two in . In fact, any equivalent number of output layers can be used as thin output layers without departing from the scope of the present disclosure.

[0057] The multi-headed neural network 112 is configured to process the audio signal 110 to extract a first embedding vector 202a indicating the first attribute type 118a and a second embedding vector 202b indicating the second attribute type 120a. To this end, the first output layer 112a outputs the extracted first embedding vector 202a, and the second output layer 112b outputs the extracted second embedding vector 202b. In one embodiment, the multi-headed neural network 112 provides a global depthwise convolution (GDC) output of dimension 512-D, which is then connected to each linear output layer: the first output layer 112a and the second output layer 112b. In one embodiment, the first output layer 112a and the second output layer 112b are both 1x1 convolutions.

[0058] The first embedding vector 202a and the second embedding vector 202b are then provided to a classification module 204, which includes a first classifier 204a for classifying attributes of the first attribute type 118a and a second classifier for classifying attributes of the second attribute type 120a, embedded in the first embedding vector 202a and the second embedding vector 202b, respectively. As already understood, the classification of the attributes of the first attribute type 118a is independent of the recording environment 100, while the classification of the attributes of the second attribute type 120a is dependent on the recording environment 100. In one embodiment, each of the first classifier 204a and the second classifier 204b uses a Softmax activation function.

[0059] Some embodiments are based on the recognition that when alternative tasks are decoupled from one another in the abnormal sound detection system 102, the types of tasks can be selected to further improve resilience to domain shift issues. For example, some tasks can be domain-dependent, while others can be domain-independent. To this end, attributes of the first attribute type 118a are classified as independent of the recording environment, while attributes of the second attribute type 120a are classified as dependent on the recording environment. This allows a first alternative task to be based on classifying the first attribute type 118a as domain-independent, while a second alternative task to be based on classifying the second attribute type 120a as domain-dependent.

[0060] An example of an alternative task for attributes of the first attribute type 118a includes determining a motor based on a sound signal. In practice, the motor and / or type of motor remains the same regardless of the environment in which the sound of the motor operating is recorded. In other words, a classifier trained to classify embedding vectors indicating attributes of the first type should produce the same results for classifying the output of multiple executions of the multi-headed neural network 112.

[0061] In contrast, in embodiments having a motor, the second type of attribute may indicate an operating state of the motor and may vary between different executions of multi-headed neural network 112. For example, the second type of attribute may indicate a voltage powering the motor or a torque produced by the motor, which may vary during operation of the motor.

[0062] In one embodiment, the abnormal sound detection system 102 includes a machine learning module (not shown) that includes a multi-head neural network 112 and is configured to operate differently during training and execution. In addition, the abnormal sound detection system 102 also includes an output interface (not shown) to output the results of abnormality detection.

[0063] Figure 3A FIG. 1 is a diagram illustrating the operation of the training phase of the abnormal sound detection system 102 according to an embodiment of the present disclosure. Figure 3AAs shown in FIG, the training phase includes receiving a training audio signal 301, the training audio signal 301 including a first attribute set 118 of a first attribute type 118a and a second attribute set 120 of a second attribute type 120a. The first attribute type 118a includes attributes such that the classification of the attributes of the first attribute type 118a is independent of one or a combination of the following: the recording environment 100; the type of sound source that generated the training audio signal 301; and the operating state of the sound source that generated the training audio signal 301. The second attribute type 120a includes other attributes such that the classification of the other attributes of the second attribute type 120a is dependent on one or a combination of the following: the recording environment 100; the type of sound source that generated the training audio signal 301; and the operating state of the sound source that generated the training audio signal 301.

[0064] Training the multi-headed neural network 112 with training data including training audio signals associated with different values of the attributes of the first attribute type 118a and different values of the attributes of the second attribute type 120a improves the accuracy of the normality model of the abnormal sound detection system 102.

[0065] For example, for different training audio signal 301 samples, there may be different values of the first attribute type 118a, such as a first value 118a1, an nth value 118an, where n is an arbitrary number, etc. In addition, different training audio signal 301 samples may also have similar values of the first attribute type 118a.

[0066] Similarly, for different training audio signal 301 samples, there may be different values of the second attribute type 120a, such as a primary value 120a1, an mth value 120am, where m is an arbitrary number, etc. In addition, different training audio signal 301 samples may also have similar values of the second attribute type 120a.

[0067] The different values of the first attribute type 118a and the second attribute type 120a at training time form the real values associated with different tasks. The multi-head neural network is trained in conjunction with the classification module to generate different encodings that are conducive to classification, thereby matching the real values associated with different tasks. It is worth noting that in some implementations, the classification module is only used to train the multi-head neural network during the training phase, and not during the testing phase.

[0068] Thus, each training audio signal 301 is associated with a corresponding true value of an attribute of a first attribute type (e.g., first value 118a1), which is one of the different values of the first attribute type 118a. Similarly, a corresponding true value of an attribute of a second attribute type 120a (such as primary value 120a1) is one of the different values of the second attribute type 120a. These true values are used to train the multi-headed neural network 112 in conjunction with the classification module 204. The multi-headed neural network 112 processes each training audio signal sample 301, wherein the first output layer 112a extracts a first embedding vector, such as embedding vector 302a1 for the true value 118a1. The second output layer 112b extracts a second embedding vector, such as embedding vector 302b1 for the true value 120a. To this end, there may be many other different embedding vectors obtained from different training audio signal samples that will be associated with the same true value 118a1.

[0069] For example, for another training audio signal sample 301n, another set of embedding vectors, such as embedding vector 302an and embedding vector 302bm, may be extracted. Then, the different embedding vectors extracted for different audio signal samples form a first embedding vector set 302a extracted by the first output layer 112a and a second embedding vector set 302b extracted by the second output layer 112b. Figure 3A In the illustration of , only two training audio signal samples are shown for simplicity of explanation. However, any equivalent number of training audio signal samples may be shown without departing from the scope of the present disclosure.

[0070] For example, the first embedding vector 302a1 and the second embedding vector 302b1 are then passed to the classification module 204, where they are assigned a class label. To this end, the first classifier 204a classifies the first embedding vector 302a1, and the second classifier 204b classifies the second embedding vector 302b1. The parameters of the multi-headed neural network 112 and the classification module 204 are jointly optimized by minimizing the classification loss function based on the classification output of the first classifier 204a and the true value 118a1 of the attribute of the first attribute type 118a, and based on the classification output of the second classifier 204b and the true value 120a of the attribute of the second attribute type. During training, a first embedding vector set 302a consisting of all first embedding vectors obtained for each training audio signal 301 and a second embedding vector set 302b consisting of all second embedding vectors obtained for each training audio signal 301 are stored in the abnormal sound detection system 102, such as in the memory 106 of the abnormal sound detection system 102, as metadata in association with corresponding true values, such as the first value 118a1, the nth value 118an, the main value 120a1 and the mth value 120am of the attribute associated with the training audio signal from which these embedding vectors were generated.

[0071] After this optimization is performed during the first training phase, the parameters of the multi-headed neural network 112 are fixed. The first embedding vector set 302a and the second embedding vector set 302b stored in the abnormal sound detection system 102 at the end of the training period can be used in the testing phase, or they can be replaced by reprocessing all training audio signals using the final multi-headed neural network 112 with the fixed parameters obtained at the end of the training in the same manner as these embedding vector sets were obtained during training.

[0072] During the testing phase, the multi-headed neural network 112 is configured to receive a test audio signal and process it to obtain a first embedding vector and a second embedding vector. Anomaly detection is then performed by comparing the first embedding vector with embedding vectors from the first embedding vector set 302a, and comparing the second embedding vector with embedding vectors from the second embedding vector set 302b. In some embodiments, the first embedding vector obtained from the test audio signal is compared with all embedding vectors in the first embedding vector set 302a, and the second embedding vector obtained from the test audio signal is compared with all embedding vectors in the second embedding vector set 302a.

[0073] Figure 3B A schematic diagram illustrating a test phase operation of an abnormal sound detection system according to an embodiment of the present disclosure is shown.

[0074] In one embodiment, the multi-headed neural network 112 is configured to accept an attribute value 306 of a first attribute type 118a for the test audio signal and select a first subset 302aa' of embedding vectors having a matching first attribute from the first set of embedding vectors 302a, whose attribute values match the accepted attribute value 306 of the test audio signal.

[0075] In another embodiment, the multi-headed neural network 112 is configured to accept a set of acceptable values for an attribute of the first attribute type 118a and place the embedding vector of the training signal in the first subset of embedding vectors 302aa' only if the value of the first attribute type of the training signal belongs to the set of acceptable values for the first attribute type 118a. For example, if the first attribute type 118a corresponds to the operating voltage of the motor, only embedding vectors corresponding to acceptable values of the operating voltage of the motor are placed in the first subset of embedding vectors 302aa', where the acceptable values of the operating voltage of the motor are known not to cause any damage or harm to the motor.

[0076] In one embodiment, the multi-headed neural network 112 is configured to accept an attribute value 308 of the second attribute type 120a of the test audio signal and select a second subset of embedding vectors 302bb' having a matching second attribute from the second embedding vector set 302a, whose attribute values match the accepted attribute value 308 of the second attribute type 120a of the test audio signal.

[0077] In another embodiment, the multi-headed neural network 112 is configured to accept a set of acceptable values for the attribute of the second attribute type 120a and place the embedding vector of the training signal in the second embedding vector subset 302bb' only if the second attribute type value of the training signal belongs to the set of acceptable values of the second attribute type 120a.

[0078] The abnormal sound detection system 102 is configured to store in the memory 106: a training dataset of embedding vectors 302, the training dataset including a first embedding vector set 302a and a second embedding vector set 302b; and associated values of the first attribute type and the second attribute type of the training audio signal from which these values were obtained. Thus, anomaly detection can be performed by comparing the first embedding vector set 302a and the second embedding vector set 302b generated and stored at training time, or a first embedding vector subset 302aa' and a second embedding vector subset 302bb' extracted therefrom based on one of the above criteria, respectively, with the first embedding vector 202a and the second embedding vector 202b generated at execution time, and the first embedding vector 202a and the second embedding vector 202b can be different from the first embedding vector set 302a and the second embedding vector set 302b previously generated and previously classified by the multi-headed neural network 112. For this reason, for the purposes of this disclosure, reference to "previously" can be understood as equivalent to "training time."

[0079] Furthermore, during the training phase of the multi-headed neural network 112, the training dataset of embedding vectors 302 is passed through a classification module such as Figure 2 The classification module 204 is shown in FIG. The classification module 204 is an alternative task classifier (typically implemented as a multinomial logistic regression, also known as a softmax layer) that is used to predict the label of the received audio signal embedding generated by the multi-headed neural network 112. The prediction is based on the alternative task, such as, but not limited to: (1) predicting the exact model of the machine that produces the sound corresponding to the embedding (for example, there may be many different types of valve machines, all with different models); or (2) predicting the properties of the machine based on its operating principle, for example, the operating speed of the fan, or the voltage of the power supply; (3) predicting the background characteristics of the place where the normal sound was recorded, for example, normal sound was recorded in factory A, factory B, a test room, etc.; or (4) creating synthetic anomalies by corrupting the sound signal or presenting examples from different machines. The predicted labels of the alternative task can then be compared with the true labels. In addition, during training, the predicted labels of the alternative task can be compared with the true labels using a cross-entropy loss function.

[0080] Some embodiments are based on the recognition that, while this comparison is advantageous for domain-dependent attributes because the value of the attribute is unknown during execution of the abnormal sound detection system 102, the value of the first attribute type 118a may be known during execution, and comparing the first embedding vector 202a with the embedding vectors in the first embedding vector set 302a corresponding to different values of the first attribute type 118a is unnecessary because, during execution, only those normal first embedding vectors that have the same value as the first attribute 118 of the first attribute type 118a are relevant for computing the overall anomaly score. Therefore, in some embodiments, only the first embedding vector 202a is compared with the first embedding vector subset 302aa' of embedding vectors corresponding to the same value of the first attribute type 118a.

[0081] At execution or inference time, the abnormal sound detection system 102 is configured to compare the first embedding vector 202a with the first embedding vector set 302a or the first embedding vector subset 302aa' previously generated by the multi-headed neural network 112 during the training phase and stored in the memory 106 to classify the attributes of the first attribute type 118a of the received audio signal 110 at the inference time. The second embedding vector 202b is compared with the second embedding vector set 302b or the second embedding vector subset 302bb' previously generated by the multi-headed neural network 112 to classify the attributes of the second attribute type 120a of the received audio signal 110 at the inference time. The result of the comparison is used to determine the result of the anomaly detection of the audio signal 110 received at the inference time.

[0082] In one embodiment, the comparison between the first embedding vector 202a and the first set of embedding vectors 302a or the first subset of embedding vectors 302aa' and the comparison between the second embedding vector 202b and the second set of embedding vectors 302b or the second subset of embedding vectors 302bb' is performed using a nearest neighbor distance metric, where the nearest neighbor distance metric is one of Euclidean distance or cosine distance.

[0083] In addition, the abnormal sound detection system 102 is configured to use the received audio signal 110 and the result of the abnormality detection to determine the abnormality score of the received audio signal 110. The abnormal sound detection system 102 determines whether the received audio signal 110 is a normal audio signal or an abnormal audio signal based on the abnormality score. When the abnormality score is higher than a pre-specified threshold, the received audio signal 110 is determined to be abnormal. In one embodiment, both the abnormality score and the pre-specified threshold can be numerical values between 0 and 1. For example, the pre-specified threshold can be 0.5, and the determined abnormality score can be 0.4. Since the abnormality score of 0.4 is less than the pre-specified threshold of 0.5, the received audio signal 110 is determined to be normal or non-abnormal in this embodiment. In some embodiments, the pre-specified threshold is set based on experimental data.

[0084] Figure 4 Schematic diagram 400 showing a process of abnormality score determination according to an exemplary embodiment of the present disclosure. For the purpose of ease of explanation in this disclosure, Figure 4 The training dataset of the embedding vector 302 shown in FIG includes the above Figure 3A and Figure 3B 302a, the first embedding vector set 302aa', the second embedding vector set 302b and the second embedding vector subset 302bb' explained in FIG.

[0085] like Figure 4 As shown in , the first embedding vector 202a generated at execution time is compared with a training dataset of embedding vectors to generate a first anomaly score 402, where the training dataset of embedding vectors includes a first embedding vector set 302a and a first embedding vector subset 302aa' previously generated by the multi-headed neural network 112 (such as at training time).

[0086] The second embedding vector 202 b generated at execution time is compared to a training dataset of embedding vectors to generate a second anomaly score 404. The training dataset of embedding vectors includes the second embedding vector set 302 b and the second embedding vector subset 302 bb ′ previously generated by the multi-headed neural network 112. The first anomaly score 402 and the second anomaly score 404 are then combined 406 to determine an anomaly detection result 114 based on the combination 406 of the first anomaly score 402 and the second anomaly score 404.

[0087] In one embodiment, the combination 406 of the first anomaly score 402 and the second anomaly score 404 is a weighted combination, wherein the weight of the first anomaly score 402 is less than the weight of the second anomaly score 404. For this reason, a smaller weight is used for the first anomaly score 402 because the first attribute 118 is less likely to cause a failure of the machine that generated the audio signal 110 than the second attribute 120. In one embodiment, if certain attributes prove less reliable in predicting anomalies over time, different weights may be used for different anomaly scores.

[0088] Figure 5A A schematic diagram 500a illustrating an example method of anomaly score generation according to an example embodiment of the present disclosure is shown.

[0089] Abnormal sound detection system 102 is configured to determine an anomaly score, such as combined anomaly score 406, by concatenating first embedding vector 202a and second embedding vector 202b to generate concatenated embedding vector 502. In addition, a nearest neighbor distance algorithm 504 is used to compare concatenated embedding vector 502 with the embedding vector of training dataset 302 by calculating the minimum distance between concatenated embedding vector 502 and the embedding vector of training dataset 302.

[0090] To this end, the abnormal sound detection system 102 is configured to compare the generated concatenated embedding vector 502 with each embedding vector of the training dataset 302 using a distance measurement technique to calculate the minimum distance between the concatenated embedding vector 502 and the embedding vectors of the training dataset 302. The distance measurement technique includes at least one of a Euclidean distance technique, a cosine distance technique, or a weighted Euclidean distance. For example, the first embedding vector 202a is compared with the first embedding vector set 302a and the first embedding vector subset 302aa' to determine a first distance measure. Similarly, the second embedding vector 202b is compared with the second embedding vector set 302b and the second embedding vector subset 302bb' to determine a second distance measure. The first distance measure and the second distance measure are then combined by the distance measurement algorithm 504 to generate a combined anomaly score 406. This combination may include, for example, summing, averaging, weighted averaging, etc.

[0091] Figure 5BA block diagram 500b illustrates another method for determining a combined anomaly score 406 while facilitating a learning weighting approach, according to an embodiment of the present disclosure. The abnormal sound detection system 102 is configured to determine a combined anomaly score 406 for each of the first and second embedding vectors 202a, 202b having decoupled dimensions by applying a distance measurement algorithm 504 to each of the first and second embedding vectors 202a, 202b to determine a first anomaly score 504a, 504b, respectively. The decoupled dimensions correspond to the first and second embedding vectors 202a, 202b having different dimensions, where the dimensions refer to a quantitative or qualitative measure of a feature included in the dimension.

[0092] Furthermore, the individual anomaly scores for the decoupled dimensions are combined using a learned weighting method 506 to determine a combined anomaly score 406. To this end, a weight is assigned to each of the first anomaly score 504a and the second anomaly score 504b before they are combined using known combining techniques (such as summing, averaging, etc.). In one embodiment, the weight of the first anomaly score 504a generated for the first attribute set 118 (independent of the recording environment 100) is less than the weight of the second anomaly score 504b generated for the second attribute set 120 (dependent on the recording environment 100). This is because the values of the first attribute set 118 can be known during execution time, and comparing the first embedding vector 202a with the first embedding vector set 302a and the first embedding vector subset 302aa' corresponding to different values of the first type of attribute is unnecessary or less relevant because, during execution, only those first embedding vector subsets 302aa' that have the same value as the first attribute type 118 are relevant for computing the overall anomaly score. Such a weighted combination increases the accuracy and performance of the multi-headed neural network 112 by placing less emphasis on redundant processing operations.

[0093] Furthermore, based on the contribution of a particular decoupled dimension to the combined anomaly score 406, the abnormal sound detection system 102 generates anomaly detection results 114 to predict the cause behind the abnormal sound. For example, if the decoupled dimension corresponding to the speed prediction during execution contributes most to the combined anomaly score 406, the abnormal sound detection system 102 further causes the processor 104 to generate control commands to investigate only those portions of the machine's controlled speed as possible causes of the detected abnormal sound.

[0094] In one embodiment, during inference, a database of prototype normal sound embeddings in the form of a training dataset of normal embedding vectors is used to compare one or more embedding vectors 202 of the received audio signal 110 with the training dataset of normal embedding vectors. If the training dataset of normal embedding vectors is sufficiently small, all training samples are used as prototypes. However, in some embodiments, the training dataset of normal embedding vectors is further reduced by using an algorithm such as K-means clustering.

[0095] Figure 6 A schematic diagram illustrates a use case 600 for collecting a training dataset for training embedding vectors 302 for multi-headed neural network 112, according to an embodiment of the present disclosure. Abnormal sound detection system 102 utilizes the collected training dataset of embedding vectors 302 to detect abnormal sounds. Use case 600 illustrates different models of drill rigs: drill model 602, drill model 604, and drill model 606 in factory 608 (for simplicity of explanation, the different drill rig models are hereinafter referred to as drills 602-606). Drill rigs 602-606 are monitored using microphone 610 to determine when abnormal sounds occur. In one embodiment, drill model 602 is used for wood, drill model 604 is used for iron, and drill model 606 is used for steel. The training dataset for embedding vectors 302 is created using only sound recordings of drill rigs 602-606 collected during normal operation. Drill rigs 602-606 are complex machines and are controlled by many different operating parameters. Exemplary operating parameters may be the speed at which the drill motor rotates, the type of material being drilled, etc. By varying example operating parameters and recording the sounds they produce, the abnormal sound detection system 102 collects a training dataset of embedding vectors 302 corresponding to normal training data (by collecting corresponding audio signals and extracting embedding vectors from them). Furthermore, an alternative task is defined as a classifier whose goal is to predict a specific operating parameter (e.g., rotational speed) based on the sound signals produced by the drill rigs 602-606. Therefore, since the drill rigs 602-606 have multiple operating parameters, multiple different classifiers can be trained to classify the multiple operating parameters.

[0096] However, once the classifiers are trained, the abnormal sound detection system 102 may not be aware of the specific environment of an individual drill rig at inference time. This specific environment may include, for example, the specific plant 608 (and the associated background noise) in which the drill rigs 602-606 operate. These specific environments are referred to as domains. To make the abnormal sound detection system 102 work well across as many domains as possible, different classifiers are trained to predict domain-independent attributes and domain-dependent features, where domain-dependent attributes are domain-shared attributes (e.g., predicting the model of the drill rig 602-606, predicting how fast the drill rig is spinning, etc.), and domain-dependent features are referred to herein as domain-specific attributes (e.g., predicting the environment in which the drill rig operates or the type of material the drill rig is made of).

[0097] By calculating a separate anomaly score for each classifier at inference time, the abnormal sound detection system 102 can determine anomaly scores based on embedding vectors that are trained differently by different classifiers (e.g., the first classifier 204a and the second classifier 204b). For example, if any drill is currently being used on plastic, and the abnormal sound detection system 102 has never obtained any sound recordings of a drill that has previously drilled plastic, the abnormal sound detection system 102 can weight the anomaly scores from the domain-specific classifiers. Alternatively, the abnormal sound detection system 102 can determine the weights between different classifiers based on how accurate the classifiers are trained (i.e., accurate classifiers have higher weights). An additional method for determining classifier weights for different dimensions can be that after observing some known anomalies, the abnormal sound detection system 102 can adjust the weights for different dimensions so that the observed anomalies can be detected with high confidence in the future without any retraining of the classifiers.

[0098] Example Implementation

[0099] During training, a dataset D of N normal training instances of a given machine type is provided. in is an amplitude spectrogram with F frequencies and T time frames, and is a vector of classification surrogate task labels, where y s Indicates the machine part (functionally equivalent to the machine model) and y am represents the classification label of the mth attribute among the M different attributes available for a given machine type. In addition, the domains share (partial) embedding z S (i.e., the first embedding vector 202a) and the domain-specific (attribute) embedding z A (i.e., the second embedding vector 202b) is obtained as:

[0100]

[0101] Where CNN(.) is a shared convolutional neural network, such as the multi-head neural network 112, and L Sec and L Att denote part- and attribute-specific linear embedding layers (implemented as 1×1 convolutions), respectively. Sec +L Att Minimize to train all parameters, where

[0102]

[0103] are the cross entropy losses for the part and attribute respectively, wi,j and bi,j are the learned weight vectors and biases of the associated classifiers, c indexes the C=6 part, and cm indexes the Cm value of the mth attribute. Because not all attributes are present in all audio examples of a given machine type in the dataset, L Att The attribute loss in (3) is learned from the same embedding z in a multi-task manner. A Instead of learning a disentangled feature dimension for each attribute, a combination is performed over all attributes. If an attribute is unknown for the audio embodiment, the corresponding term in the summation of (3) is ignored.

[0104] During inference, the test embedding z q The training embedding set corresponding to all the anomaly scores used to calculate The nearest neighbor (NN) distance between

[0105]

[0106] Among them, D cos (.,.) is the cosine distance between the two embedding vectors. As described below, the decoupled model allows exploring multiple inference methods, depending on the z in (4) q Which embedding dimension to use.

[0107] For disentangled splicing, using splicing embedding, (4)

[0108] For decoupling weighting, for the partial embedding z S and attribute embedding z A The weighted average of the individually calculated NN distances, i.e.,

[0109]

[0110] Among them, ws and wA are scalar weights that are optimized after training is completed.

[0111] For the decoupling part, only the partial embedding z is used in (4) S .

[0112] For decoupled attributes, only attribute embedding z is used in (4) A .

[0113] At test time, partial labels of the test samples are known, so when computing the NN distance, the restriction of the training set samples from D is set to only those samples belonging to the corresponding part.

[0114] use Figure 7 The flowcharts described in further explain exemplary implementations.

[0115] Figure 7 A flowchart 700 is shown depicting a method for detecting abnormal sound signals according to an embodiment of the present disclosure. The method is performed by the abnormal sound detection system 102. The method begins at operation 702.

[0116] At 704, the method includes receiving an audio signal (e.g., Figure 1A and Figure 1B The sound source 108 corresponds to at least one of a machine, an electrical device, or an engine. In one embodiment, the sound source 108 may include a drill, a grinder, a packaging machine, or the like. Furthermore, the abnormal sound detection system 102 is configured to generate one or more embedding vectors using the received audio signal 110, facilitated by a multi-head neural network, such as a multi-head neural network 112. The sound source 108 and the recording environment 100 are characterized by an attribute set 116, which includes a first attribute 118 associated with a first attribute type 118a and a second attribute 120 associated with a second attribute type 120a. The first attribute 118 is independent of the recording environment 100, while the second attribute depends on the recording environment 100. For example, in the case of a motor operating in an industrial automation environment, attributes associated with the type of motor are first attribute type 118a, which are independent of the recording environment 100 of the industrial automation setting, while attributes associated with the operating state of the motor (such as operating voltage, operating torque, etc.) depend on the type of industrial automation setting. Therefore, these attributes associated with the operating state of the motor are second attribute type 120a. Thus, the received audio signal 110 is formed as a combination of one or more of a first attribute 118 of a first attribute type 118a and a second attribute 120 of a second attribute type 120a.

[0117] At 706, the received audio signal 110 is processed using a multi-headed neural network, such as the previously described multi-headed neural network 112, which is trained to extract a first embedding vector indicating a first attribute type 118a and a second embedding vector indicating a second attribute type 120a from the received audio signal 110. Figure 2As shown in FIG, the multi-headed neural network 112 passes the received audio signal 110 through the first output layer 112a to extract the first embedding vector 202a of the first attribute type 118a, and passes it through the second output layer 112b to extract the second embedding vector 202b of the second attribute type 120a.

[0118] At 708, the first embedding vector is compared with a first embedding vector set previously generated by the multi-head neural network to classify the attributes of the first attribute type, and the second embedding vector is compared with a second embedding vector set previously generated by the multi-head neural network to classify the attributes of the second attribute type to determine the result of anomaly detection. Figure 4 As described, the first embedding vector 202a is compared with the first embedding vector set 302a, which also includes a first embedding vector subset 302aa' of the training data set of embedding vectors 302, the training data set of embedding vectors 302 previously generated by the multi-headed neural network 112 at training time and stored in the memory 106 of the abnormal sound detection system 102. Similarly, the second embedding vector 202b is compared with the second embedding vector set 302b, which also includes a second embedding vector subset 302bb' of the training data set of embedding vectors 302, the training data set of embedding vectors 302 previously generated by the multi-headed neural network 112 at training time and stored in the memory 106 of the abnormal sound detection system 102. This comparison results in the generation of a first anomaly score 402 and a second anomaly score 404, which are combined to generate a combined anomaly score 406. Using the previously combined Figure 5A and Figure 5B The combination can be accomplished using any of the techniques described herein, i.e., using a concatenated combination of embedding vectors or using a learned weighting method for the anomaly score, or any other equivalent technique. The resulting combined anomaly score 406 is then used to determine an anomaly detection result 114. For example, the anomaly detection result 114 is that an abnormal sound is detected when the determined combined anomaly score is above a pre-specified threshold. In one embodiment, the pre-specified threshold can be set by a machine learning module of the processor 104, for example, by using the 95th percentile of the anomaly score calculated over the entire non-anomalous training set.

[0119] At 710, the anomaly detection result 114 is rendered. The rendering can be performed on one or more of a display, a user interface, an audio interface, or a combination thereof associated with the abnormal sound detection system 102. For example, the abnormal sound detection system 102 displays the abnormal sound signal on a display interface that shows spectrograms of different audio signals received in the recording environment 100. The spectrogram of the abnormal sound signal can be highlighted in a different color than the spectrogram of the non-abnormal sound signal. For example, the spectrogram of the abnormal sound signal can be highlighted in red, while the spectrogram of the non-abnormal sound signal can be displayed in green. In addition, the display can also include more information about the source of the abnormal sound signal, which can be obtained from the first embedding vector 202a or the second embedding vector 202b generated for the received audio signal 110. The method terminates at 712.

[0120] Figure 7 The illustrated method provides for more accurate and efficient detection of abnormal sound signals, which may be used in various applications, such as for machines, engines, numerically controlled components, and the like.

[0121] Figure 8 A block diagram 800 illustrates a hardware framework of the abnormal sound detection system 102 according to an embodiment of the present disclosure. In some exemplary embodiments, the block diagram 800 includes one or more microphones 802a that collect the audio signal 110 and a training dataset 806 (which is equivalent to the training dataset 302 shown in FIG. 3 ).

[0122] The unusual sound detection system 102 includes a hardware processor 808. The hardware processor 808 communicates with computer memory, such as memory 810. The memory 810 includes stored data, including algorithms, instructions, and other data implemented by the hardware processor 808. It is contemplated that the hardware processor 808 may include two or more hardware processors, depending on the requirements of a particular application. The two or more hardware processors may be internal or external. The unusual sound detection system 102 is integrated with other components, including devices such as output interfaces and transceivers.

[0123] In some alternative embodiments, the hardware processor 808 is connected to a network 804 that communicates with the source of the audio signal 110. As non-limiting examples, the network 804 includes, but is not limited to, one or more local area networks (LANs) and / or wide area networks (WANs). The network 804 also includes enterprise-wide computer networks, intranets, and the Internet. The abnormal sound detection system 102 includes one or more client devices, storage components, and data sources. Each of the one or more client devices, storage components, and data sources includes one or more devices that collaborate in a distributed environment within the network 804.

[0124] In some other alternative embodiments, hardware processor 808 is connected to a network-enabled server 814, which is connected to a client device 816. Network-enabled server 814 corresponds to a dedicated computer connected to a network running software designed to process client requests received from client device 816 and provide appropriate responses on client device 816. Hardware processor 808 is connected to transmitter 820 and external memory device 818, which stores all necessary data used in the detection of abnormal sound signals. Transmitter 820 facilitates the transmission of data between network-enabled server 814 and client device 816. In addition, output 822 associated with the detection of abnormal sound signals is generated.

[0125] The audio signal 110 and the training dataset 806 are further processed by the multi-headed neural network 112. The multi-headed neural network 112 is trained using the training dataset 806 of normal embedding vectors (as described above).

[0126] The abnormal sound detection system 102 is configured to detect a malfunctioning operating portion in a manufacturing setting based on abnormal sound detection disclosed in various embodiments described herein.

[0127] Those skilled in the art will appreciate many modifications and other embodiments of the disclosure described herein after having benefited from the teachings presented in the foregoing description and the associated drawings. It should be understood that the disclosure is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. In addition, although the above description and the associated drawings describe exemplary embodiments in the context of certain example combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, combinations different from the elements and / or functions explicitly described above are also contemplated, which will be listed in some of the appended claims. Although specific terms are used herein, they are general and descriptive only and are not intended to be limiting.

Claims

1. An abnormal sound detection system, comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the abnormal sound detection system to: receiving an audio signal generated by a sound source in a recording environment, wherein the sound source and the recording environment are characterized by a set of attributes, the set of attributes including a first attribute belonging to a first attribute type and a second attribute belonging to a second attribute type; Processing the received audio signal using a multi-headed neural network trained to extract a first embedding vector indicating the first attribute type and a second embedding vector indicating the second attribute type from the received audio signal; Comparing the first embedding vector with a first embedding vector set previously generated by the multi-headed neural network to classify attributes of the first attribute type, and comparing the second embedding vector with a second embedding vector set previously generated by the multi-headed neural network to classify attributes of the second attribute type, thereby determining an anomaly detection result; and Render the anomaly detection result.

2. The abnormal sound detection system according to claim 1, wherein: The multi-headed neural network includes a convolutional neural network module connected to a plurality of thin output layers, the plurality of thin output layers including a first output layer for outputting attributes of the first attribute type and a second output layer for outputting attributes of the second attribute type.

3. The abnormal sound detection system according to claim 1, wherein: The first attribute type includes attributes such that classification of the attributes of the first attribute type is independent of one or a combination of the recording environment, the type of the sound source, and the operating state of the sound source, and The second attribute type includes other attributes, such that classification of the other attributes of the second attribute type depends on one or a combination of the recording environment, the type of the sound source, and the operating state of the sound source.

4. The abnormal sound detection system according to claim 3, wherein: The sound source is a machine that generates the audio signal during its operation, wherein the first embedding vector indicates a type of the machine, and wherein the second embedding vector indicates an operating state of the machine.

5. The abnormal sound detection system according to claim 4, wherein: The machine includes a motor, and the type of machine includes the type of motor, wherein the operating state includes one or a combination of a motor input characteristic and a motor output characteristic.

6. The abnormal sound detection system according to claim 4, wherein: The machine is one or a combination of a drone, a robot, a CNC machine and an engine.

7. The abnormal sound detection system according to claim 1, wherein: Classification of attributes of the first attribute type is managed by a first classifier used to train the multi-headed neural network to produce a first set of normal embedding vectors.

8. The abnormal sound detection system according to claim 1, wherein: Classification of attributes of the second attribute type is managed by a second classifier used to train the multi-headed neural network to produce a second set of normal embedding vectors.

9. The abnormal sound detection system according to claim 1, wherein: The multi-headed neural network is trained with training data comprising audio signals associated with different attribute values of the first attribute type and different attribute values of the second attribute type, wherein, during processing of the received audio signal with the multi-headed neural network, the attribute values of the first attribute type of the received audio signal are known, and wherein the first set of normal embedding vectors comprises a first subset of embedding vectors indicating the known attribute values of the first attribute type.

10. The abnormal sound detection system according to claim 1, wherein: The multi-headed neural network is trained using training data, the training data including audio signals associated with different attribute values of the first attribute type and different attribute values of the second attribute type, wherein during processing of the received audio signal by the multi-headed neural network, the processor is configured to: accepting a property value of the first property type of the received audio signal; and Only a first subset of embedding vectors indicating accepted attribute values of the first attribute type is placed in the first set of embedding vectors.

11. The abnormal sound detection system according to claim 1, wherein: The first embedding vector is compared to the set of first embedding vectors previously generated by the multi-headed neural network, and the second embedding vector is compared to the set of second embedding vectors previously generated by the multi-headed neural network using a nearest neighbor distance metric, wherein the nearest neighbor distance metric is one of Euclidean distance, cosine distance, or weighted Euclidean distance.

12. The abnormal sound detection system according to claim 1, wherein: The at least one processor causes the abnormal sound detection system to: The abnormality detection result is determined based on a combined abnormality score to detect an abnormal sound signal, wherein the abnormal sound signal is detected when the combined abnormality score is higher than a pre-specified threshold.

13. The abnormal sound detection system according to claim 1, wherein: To determine the anomaly detection result, the processor is configured to compare the first embedding vector with the first set of embedding vectors to generate a first anomaly score; comparing the second embedding vector to the second set of embedding vectors to generate a second anomaly score; and The anomaly detection result is determined based on a combination of the first anomaly score and the second anomaly score, the combination of the first anomaly score and the second anomaly score generating a combined anomaly score.

14. The abnormal sound detection system according to claim 13, wherein: The combination of the first anomaly score and the second anomaly score is a weighted combination, wherein a weight of the first anomaly score is less than a weight of the second anomaly score.

15. The abnormal sound detection system according to claim 1, wherein: In order to determine the abnormality detection result, the processor is configured to: combining the first embedding vector and the second embedding vector to obtain a combined embedding vector; comparing the combined embedding vector with a set of normal combined embedding vectors previously obtained by combining a first embedding vector and a second embedding vector generated by the multi-headed neural network with respect to a normal audio signal to generate a combined anomaly score; and An anomaly detection result is determined based on the combined anomaly score.

16. The abnormal sound detection system according to claim 15, wherein: To combine the first embedding vector and the second embedding vector to obtain the combined embedding vector, the processor is configured to concatenate the first embedding vector and the second embedding vector.

17. The abnormal sound detection system according to claim 1, wherein: The rendered anomaly detection results are used to predict a fault associated with at least one of the sound source and the recording environment.

18. A computer-implemented method for detecting abnormal sounds, the computer-implemented method comprising the following steps: receiving an audio signal generated by a sound source in a recording environment, wherein the sound source and the recording environment are characterized by a set of attributes, the set of attributes including a first attribute belonging to a first attribute type and a second attribute belonging to a second attribute type; Processing the received audio signal using a multi-headed neural network trained to extract a first embedding vector indicating the first attribute type and a second embedding vector indicating the second attribute type from the received audio signal; Comparing the first embedding vector with a first embedding vector set previously generated by the multi-headed neural network to classify attributes of the first attribute type, and comparing the second embedding vector with a second embedding vector set previously generated by the multi-headed neural network to classify attributes of the second attribute type, thereby determining an anomaly detection result; and Render the anomaly detection result.

19. The computer-implemented method of claim 18, wherein: The multi-headed neural network includes a convolutional neural network module connected to a plurality of thin output layers, the plurality of thin output layers including a first output layer for outputting attributes of the first attribute type and a second output layer for outputting attributes of the second attribute type.

20. The computer-implemented method of claim 18, wherein: in: The first attribute type includes attributes such that classification of the attributes of the first attribute type is independent of one or a combination of the recording environment, the type of the sound source, and the operating state of the sound source, and The second attribute type includes other attributes, such that classification of the other attributes of the second attribute type depends on one or a combination of the recording environment, the type of the sound source, and the operating state of the sound source.

Images