Sound anomaly detection with mixed augmented dataset

Abstract

Embodiments of the invention relate to sound anomaly detection with mixed augmented datasets. A method and computer program product for training a neural network performs multiple forms of data augmentation on sample waveforms of a training dataset that includes both normal samples and abnormal samples to generate normal data augmentation samples and abnormal data augmentation samples. The normal data augmentation samples are labeled according to the type of data augmentation performed on each respective normal data augmentation sample. The abnormal data augmentation samples are labeled according to a type of data augmentation that is different from the type of data augmentation performed on each respective abnormal data augmentation sample. Using the normal data augmentation samples and the abnormal data augmentation samples, a neural network model is trained to identify the form of data augmentation that has been performed on a waveform.

Description

[0001] Statement of the Prior Art Disclosure in Respect of the Invention or Joint Invention

[0002] The following disclosure is submitted pursuant to 35 U.S.C. § 102(b)(1)(A):

[0003] DISCLOSURE

[0004] DETECTION OF ANOMALOUS SOUNDS FOR MACHINE CONDITION MONITORING USING CLASSIFICATION CONFIDENCE, Tadanobu Inoue, Phongtharin Vinayavekhin, Shu Morikuni, Shiqiang Wang, Tuan Hoang Trong, David Wood, Michiaki Tatosubori, Ryuki Tachibana, available as of July 1, 2020.

[0005] DETECTION OF ANOMALOUS SOUNDS FOR MACHINE CONDITION MONITORING USING CLASSIFICATION CONFIDENCE (paper), Inoue, P. Vinayavekhin, S. Morikuni, S. Wang, T. H. Trong, D. Wood, M. Tatosubiri, R. Tachibana, available as of November 2, 2020.

[0006] DETECTION OF ANOMALOUS SOUNDS FOR MACHINE CONDITION MONITORING USING CLASSIFICATION CONFIDENCE (presentation), Tadanobu Inoue, Phongtharin Vinayavekhin, Shu Morikuni, Shiqiang Wang, Tuan Hoang Trong, David Wood, Michiaki Tatosubori, Ryuki Tachiban, available as of November 2, 2020. TECHNICAL FIELD

[0007] The present invention relates generally to detecting anomalies in sound data, and more specifically to using data augmentation on sound samples to distinguish anomalous sound data from normal sound data. BACKGROUND

[0008] Anomaly detection seeks to find unusual samples in audio data. For example, using a dataset of audio samples representing “normal” data for training a model, the trained model can be utilized to identify anomalous input audio data. Examples of attempts to perform audio anomaly detection include reconstruction (where anomalies can be detected from reconstruction error of a trained autoencoder or generative adversarial network), feature learning (where a feature extraction model maps normal data into a small region of a feature space), classification (where a classifier is used to find samples outside of an expected distribution), and geometric transformation (where a classifier is trained to infer a geometric transformation of image data).

[0009] Each of these examples faces respective challenges. For example, geometric transformation is designed for images, and native application of geometric transformation principles to audio samples performs poorly. SUMMARY

[0010] A computer program product for training a neural network performs multiple forms of data augmentation on sample waveforms of a training dataset including both normal samples and anomalous samples to generate normal data augmented samples and anomalous data augmented samples. The normal data augmented samples are labeled according to a type of data augmentation performed on each respective normal data augmented sample. The anomalous data augmented samples are labeled according to a type of data augmentation different from the type of data augmentation performed on each respective anomalous data augmented sample. A neural network model is trained to identify the form of data augmentation that has been performed on the waveforms using the normal data augmented samples and the anomalous data augmented samples.

[0011] A computer program product for training a neural network performs multiple forms of data augmentation on sample waveforms of a training dataset including both normal samples and anomalous samples to generate normal data augmented samples and anomalous data augmented samples. A neural network model is trained to identify the form of data augmentation that has been performed on the waveforms using the normal data augmented samples and the anomalous data augmented samples. The anomalous data augmented samples contribute differently to a training loss function than the normal data augmented samples.

[0012] These and other features and advantages will be apparent from the following detailed description, which proceeds with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0013] The following description will provide details of preferred embodiments with reference to the following drawings, in which:

[0014] Figure 1 is a diagram showing various different forms of data augmentation performed on input waveform samples according to embodiments of the present application, including distinct types of data augmentation performed to different extents;

[0015] Figure 2 is a diagram showing segments of an input waveform according to an embodiment of the application;

[0016] Figure 3 is a block / flow diagram of a method of detecting anomalies and performing responsive actions according to an embodiment of the application;

[0017] Figure 4 is a block diagram / flow diagram of a method of training a classifier to detect forms of data augmentation performed on waveform samples according to an embodiment of the application;

[0018] Figure 5 is a block diagram / flow diagram of a method of detecting anomalies using data augmentation classification according to an embodiment of the application;

[0019] Figure 6 is a block diagram of an anomaly detection and response system using data augmentation classification according to an embodiment of the application;

[0020] Figure 7 is a high-level diagram of a neural network layer that can be used to classify data augmentations according to an embodiment of the application;

[0021] Figure 8 is a diagram of a neural network architecture that can be used to classify data augmentations according to an embodiment of the application;

[0022] Figure 9 is a diagram of a neural network classifier that can be used to classify data augmentations in waveform samples according to an embodiment of the application;

[0023] Figure 10 is a block diagram illustrating an illustrative cloud computing environment with one or more cloud computing nodes that are in communication with a local computing device used by a cloud consumer according to one embodiment; and

[0024] Figure 11 is a block diagram illustrating a set of functional abstraction layers provided by the cloud computing environment according to one embodiment. DETAILED DESCRIPTION

[0025] To detect anomalous sound data, sound augmentation can be used on the acoustic dataset using a variety of different augmentation types. A machine learning model can then be trained to classify input sound clips according to which type of data augmentation was applied. During anomaly detection, input sound data is augmented in the same way as the training dataset. The machine learning model is then used to classify the input sound data according to the different types of data augmentation. An anomaly score can then be produced based on the confidence that the augmented input data was classified into the trained augmentation type categories. Anomalous sound data can generally have lower confidence values and higher anomaly scores than normal sound data.

[0026] To further improve the efficacy of the model, a training dataset is formed using a combination of samples from normal operation and samples from anomalous operation. These anomalous samples are augmented in a way that provides intentionally incorrect labels for the augmentations performed. The number of anomalous samples in the training dataset can be relatively small compared to the number of normal samples. In many cases, the number of samples from anomalous operation used for the training dataset is small enough that naive binary classification performs poorly.

[0027] It should be appreciated that while the present disclosure includes detailed descriptions regarding cloud computing, implementations of the teachings set forth herein are not limited to a cloud computing environment. Rather, embodiments of the present invention are capable of being implemented in conjunction with any other type of computing environment now known or later developed.

[0028] Referring now to Figure 1 , a diagram is shown illustrating the application of a variety of different data augmentation processes to a sound clip 100. In this example, two data augmentations are performed to produce a plurality of augmented samples. This particular example shows pitch shifting and time stretching being performed, each to three different degrees. Thus, nine samples are shown, including the original input sample 100 and eight augmented samples 102a-102h, each with a different respective degree (e.g., a magnitude selected from -1, 0, +1).

[0029] It should be appreciated that any number and type of data augmentation processes can be performed with any appropriate number of degrees. Exemplary types of sound data augmentation include pitch shifting, time stretching, low / high pass filters, overlaid noise sounds, time shifting, decomposition of sound into harmonic and impulse components, shuffling of the time sequence order of sound clips, averaging sounds, and spectral warping. While two types of data augmentation are shown (each with three degrees) to produce a field of nine outputs, any number of data augmentation types (with any appropriate number of degrees) can be used to produce a set of augmented samples of any appropriate size. For example, as described in more detail below, a subset of four, five, or eight of the nine possible outputs can be implemented and used.

[0030] Referring now to Figure 2 , the augmented samples can be divided into a set of segments 202. It should be understood that the segmentation can be performed before or after the data augmentation. The segments 202 can all have the same length, or can have different lengths. The start time of each subsequent segment 202 can be separated from the start time of the previous segment 202 by a hop size 204. Thus, the segments 202 can overlap each other to varying degrees depending on the hop size 204. The segment length and hop size are hyperparameters that can be tuned to maximize anomaly detection performance. As an example in real time, a ten second sample size can have a segment length of 2-3 seconds, with a 50% hop size ratio relative to the segment length.

[0031] The input samples 100 can have any appropriate length, and one sample can differ in length from the next sample. Additionally, some types of data augmentation (e.g., time stretching) will affect the length of the sample. Regardless of the length of the input, using a consistent segment size helps to provide uniform samples to the classifier. Thus, for a ten second long input sample, the data augmentation can produce a set of samples that vary from about 9 seconds to about 11 seconds. In this example, the segments 202 can have a length of about three seconds, with a hop size of about one second. Thus, each of the augmented samples in the augmented sample set can have a different number of segments 202 associated therewith.

[0032] Referring now to Figure 3 , a method of detecting and responding to anomalous audio is shown. In block 302, training is performed. The training uses a machine learning model, such as a machine learning model implemented through an artificial neural network (ANN). The training takes an input dataset that includes various “normal” sound samples and some “anomalous” sound samples, and performs N types of data augmentation on each sample. For the anomalous sound samples, incorrect labels are applied to improve the performance of the trained classifier. Each sample is divided into multiple sound segments. The segments are used to train the machine learning model.

[0033] While it is specifically contemplated that the dataset can include sound information recorded within the frequency range of human hearing, it should be understood that the present principles can be readily extended to inaudible pressure waves (e.g., ultrasonic sensor data or acoustic emission data), seismic information, biometric information (e.g., heart rate or brain waves or breathing sounds), vibrations, accelerometer data, and any other kind of data that can be converted into a waveform. For example, a recorded time series from a sensor within the system can be expressed as a waveform, even if no sound information is involved at all.

[0034] The machine learning model can include a set of classifiers, each trained to recognize a respective data augmentation or combination of data augmentations, and to output a respective probability. The probability reflects the likelihood that the input segment was augmented according to the respective combination of data augmentations. Thus, for example, a classifier trained to recognize segments that have undergone pitch shift with degree -1 and time stretch with degree +1 will provide a high probability output for normal segments that have undergone those data augmentations, but will provide a lower probability output for segments that have not undergone those data augmentations.

[0035] The trained classifier for a particular set of data augmentations can also provide a lower probability output for input segments that have actually undergone the respective combination of data augmentations but were generated from abnormal sound data. This can occur because abnormal data can behave differently under data augmentations than normal sound data. To increase the likelihood that such mismatches can occur, a variety of different data augmentations and augmentation degrees can be performed. Introducing abnormal training data with incorrect labels enhances this effect.

[0036] The training in block 302 can divide the training data set into a training subset and a validation subset. As will be described in greater detail below, the training data set can be used in a backpropagation-style training process in which the output of the machine learning model is compared to an expected output for a portion of the training data, and error information is propagated back through the model to update the model. Once training is complete, the model can be further evaluated against new training information from the validation subset to assess whether the machine learning model has been trained to be sufficiently general. The training of block 302 can be repeated and refined until the accuracy of classifying the validation subset exceeds a threshold.

[0037] Block 304 receives a new input audio sample. The sample can originate from any source. As with the training data set, the input audio sample can be audible sound information, or can represent any appropriate waveform that matches the type of audio information used for training. Thus, the sample can originate from any source suitable for recording the relevant type of waveform, such as a microphone, seismograph, heart rate monitor, electroencephalogram, etc.

[0038] Block 306 performs anomaly detection on the new sample. The anomaly detection outputs an anomaly score for the sample based on how well the data augmentations performed on the sample can be correctly classified, and will be described in greater detail below. Once an anomaly is detected, block 308 performs a responsive action.

[0039] Anomaly detection can be used in a variety of applications, such as equipment failure detection, product defect detection, network intrusion detection, fraud detection, medical diagnosis, and earthquake detection. Response actions can be used to quickly and automatically respond to any such anomalies, providing a rapid response to new situations. For example, an anomaly can indicate a product defect in a factory, in which case the defective product can be removed from the production line and repaired. An anomaly can also indicate equipment failure, in which case the production line can be stopped to repair the equipment and prevent further damage. In some cases where anomalies can be resolved automatically, response actions can adjust system operating parameters to compensate, such as increasing cooling action when overheating is detected.

[0040] Now for reference Figure 4 Additional details regarding training box 302 are shown. Box 402 selects initial raw samples from the training dataset. Box 404 then generates data-augmented samples from the raw samples, for example, by performing different types of data augmentation to varying degrees. Thus, each data-augmented sample is characterized by the type of data augmentation performed on it, and the corresponding degree of each type of data augmentation. Data-augmented samples may include zero degree for each type of data augmentation and are therefore identical to the raw samples.

[0041] according to Figure 1 In the example, when performing two distinct types of augmentation, each type of augmentation has possible values ​​{-1, 0, +1}, a total of nine different augmented samples are possible for each original sample. In some cases, four samples can be selected (corresponding to...). Figure 1 The corner samples and the original samples are omitted), five samples (corresponding to Figure 1 The original samples and corners), eight samples (corresponding to Figure 1 Eight external samples (and omitting the original samples) or nine samples (corresponding to Figure 1 All samples in the sample).

[0042] Box 405 determines whether the selected sample corresponds to a "normal" or "abnormal" operation. If the sample is "normal," then Box 406 addresses what type of data augmentation was performed to correctly label the data-augmented sample. If not, then Box 407 addresses what type of data augmentation was performed to incorrectly label the data-augmented sample.

[0043] Box 408 then segments the data augmentation samples, for example, by using jump size to iterate through each data augmentation sample and select a fixed-length segment. Box 409 determines whether there are other original samples in the training dataset. If so, Box 412 selects the next sample, and processing returns to Box 404.

[0044] If no other original samples are available, box 410 uses data augmentation fragments to train the machine learning model to identify the type and extent of data augmentation. Training can utilize a loss function to characterize the difference between the model's output and its expected output. Losses can include, for example, a normalized exponential function (softmax) loss and a center loss, where the former characterizes descriptiveness and the latter characterizes compactness. Therefore, the loss function can be expressed as:

[0045]

[0046] in, It is the normalized exponential function loss. It is a central loss, and These are the parameters that determine the weights between the components of the loss. Center loss can be used to map normal input data to a minimized volumetric hyperspace in the latent feature space. In some other examples, the loss function can be defined separately. .

[0047] Training is completed at box 414. The model can be tested on the validation dataset and can be repeated if needed.

[0048] In some embodiments, it can be used Figure 1 The table is used to perform labeling of boxes 406 and 407, where each type of data augmentation of different types can be assigned a corresponding label. Thus, augmentation 102 represents pitch shift of -1 and time stretch of -1. a It can have an enhancement of 102, representing a pitch shift of +1 and a time stretch of +1. h Differentiated labeling. In some instances, normal samples can be labeled in box 406 to reflect the actual enhancements performed; for example, an enhancement that results in a pitch shift of +1 and a time shift of +1 would be labeled 102. h In such an example, anomalous samples can be labeled by box 407 with labels that are diametrically opposite to the actual enhancement performed. Therefore, an anomalous sample increased by pitch shift +1 and time shift +1 can be labeled as 102. a — Figure 1 The opposite square in the middle. Similarly, it has been based on 102 d Enhanced anomalous samples can be marked as if they have undergone 102e enhancement.

[0049] Although about Figure 1The specific type of enhancement shown describes the intentional mislabeling of box 407, but it should be understood that it can be extended to any form of enhancement. Therefore, if a particular anomalous sample is enhanced in the first manner, then it can be labeled as if it had been enhanced in the second manner, which is the opposite of the first manner. Anomalous samples that have not undergone enhancement can be accurately labeled.

[0050] Other labeling patterns for box 407 were also envisioned. For example, a "cyclic" labeling system could be used, where each anomalous sample is labeled as if a "next" type enhancement has already been performed. Following... Figure 1 For example, if an abnormal sample is subjected to an action based on 102 a The enhancement can reach 102 b Labels can be applied. In this instance, each anomalous sample (including those without applied enhancements) can be labeled according to the "next" enhancement, where any appropriate order of enhancements is chosen.

[0051] As an alternative to intentionally mislabeling anomalous samples in box 407, the loss function can be modified to remove anomalous samples from the correct classification during training in box 410. In some instances, normal samples are paired with... The contribution of normal samples can be added to the cross-entropy loss, while the contribution of outlier samples can be subtracted from the cross-entropy loss. In some other examples, for normal samples... For abnormal samples ,in m It is the number of normal samples and n It represents the number of outlier samples. p i It is aimed at the first i The normalized exponential function probability of each sample, and t i It is aimed at the first i A given label for each sample.

[0052] Now for reference Figure 5 Additional details regarding anomaly detection in box 306 are shown. After receiving a new sample in box 304, box 502 generates a data-enhanced sample from the new sample, for example, using the same set of data enhancement types and degrees used in box 404 to generate the training data-enhanced sample. In some cases, the data enhancement in box 502 does not need to replicate all the data enhancement types and degrees used during training. For example, some data enhancement types or degrees may be skipped to make inference faster.

[0053] After the new data augmentation sample has been generated, block 504 then segments the new data augmentation sample using the same hop size and segment length as were used in block 406 to segment the training data augmentation samples. Segmenting the samples can improve anomaly detection, as anomalies can only occur in a small portion of a larger sample. Furthermore, dividing the samples into multiple segments increases the amount of training data available, which can improve the accuracy of the classifier.

[0054] Block 506 uses the trained model to classify the data augmentation segments into the labels for normal samples used during the training phase. Each segment is classified according to the type and extent of data augmentation performed, with an associated probability score being generated for the segment. For example, a normalized exponential function probability can be determined for each segment.

[0055] Block 508 then determines the mean of the probabilities for the segments of each respective new data augmentation sample. Thus, each new data augmentation sample will have an associated score that is the mean of the probabilities for each of its component segments. Block 510 then determines an anomaly score for the new sample. For example, this score can be determined as:

[0056]

[0057] where x is the new sample, is the output of the classifier used to determine what type and extent of data augmentation was performed on the new data augmentation sample, x is the output of the data augmentation type and extent execution, j is the total number of combinations of data augmentation types and extents performed. Specifically, k the value of may be the mean probability for the segments of the data augmentation sample. For example, following the instructions of Figure 1 , k may be 9.

[0058] Once the anomaly score for the new sample has been determined by block 510, block 512 uses the anomaly score to determine whether the new sample represents an anomaly. For example, this can include comparing the anomaly score to a threshold, where an anomaly score above the threshold indicates that an anomaly has occurred, and where an anomaly score at or below the threshold indicates that no anomaly has occurred.

[0059] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include one or more computer readable storage media (storage media) having computer readable program instructions stored therein to implement the various aspects of the present application.

[0060] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0061] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions to storage media within the respective computing / processing device for execution by a processor. Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing / processing device, partly on the user's computing / processing device, as a stand-alone software package, partly on the user's computing / processing device and partly on a remote computing / processing device or entirely on the remote computing / processing device or server. In the latter scenario, the remote computing / processing device can be connected to the user's computing / processing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing / processing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions to implement aspects of the present application.

[0062] Various aspects of the present application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0063] These computer readable program instructions can be provided to a processor of a computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0064] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0065] Reference throughout this specification to "an embodiment" or "the embodiment" or "one embodiment", and their variations, means that a particular feature, structure, characteristic, and so forth being described in connection with these embodiments is included in at least one embodiment of the present application. Thus, appearances of the phrase "in one embodiment" or "in an embodiment", as well as any other variations, anywhere in this specification are not necessarily all referring to the same embodiment.

[0066] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to encompass selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” this terminology is intended to encompass selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be readily apparent to those skilled in the art and related fields, this can be extended to as many of the listed items as possible. The flowcharts and block diagrams in the accompanying drawings illustrate the structure, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in a flowchart or block diagram may represent a module, segment, or instruction portion containing one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions indicated in the boxes may occur in a different order than those shown in the figures. For example, two consecutive boxes shown may actually be implemented as a single step, executed simultaneously, substantially simultaneously, executed with partial or complete temporary overlap, or these boxes may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart illustrations, and combinations of boxes in the block diagram and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified function or action, or by a combination of dedicated hardware and computer instructions.

[0067] eAs used herein, the term "hardware processor subsystem" or "hardware processor" can refer to a processor, memory, software, or combination thereof that coordinates to perform one or more particular tasks. In useful embodiments, a hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, and / or controllers based on individual processor or computing elements (e.g., logic gates, etc.). A hardware processor subsystem can include one or more on-board memories (e.g., cache, dedicated memory arrays, read-only memory, etc.). In some embodiments, a hardware processor subsystem can include one or more memories, which can be on-board memories or off-board memories, or can be exclusively used by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0068] In some embodiments, a hardware processor subsystem can include and execute one or more software elements. The one or more software elements can include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0069] In other some embodiments, a hardware processor subsystem can include specialized, dedicated circuitry to perform one or more electronic processing functions to achieve a specified result. These circuits can include one or more application-specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0070] These and other variations of hardware processor subsystems are contemplated in accordance with embodiments of the present application.

[0071] Referring now to Figure 6 , an anomaly detection and response system 600 is shown. The system 600 includes a hardware processor 602 and a memory 604. A sensor interface 606 provides communication with one or more sensors, which can for example include a microphone that collects audio data, or alternatively can be any sensor or combination of sensors that provides a waveform or time series output.

[0072] A classifier 610 is trained by a model trainer 614, and can be implemented as any appropriate machine learning model, such as an ANN. A data augmenter 608 is used by the model trainer 614 to perform data augmentation on each original sample waveform from a training dataset to generate a set of data augmented samples, for example using a variety of types and degrees of data augmentation. The classifier 610 is trained to identify the type and degree of data augmentation that has been applied to a given segment of a sample.

[0073] The anomaly detector 612 receives new samples from the sensor interface 606 and uses the data augmenter 608 to generate data augmented samples. The classifier 610 is then used to determine what type and degree of data augmentation was performed on each of the data augmented samples, resulting in respective probabilities for each combination of augmentation type and degree. The anomaly detector uses these probabilities to generate an anomaly score for the new sample, and then uses the anomaly score to determine whether the new sample represents an anomaly.

[0074] The response function 616 is triggered by detecting an anomaly. The response function 616 can include any appropriate action to correct, report, or otherwise address the detected anomaly.

[0075] Reference is now made to Figure 7 , which shows a diagrammatic illustration of generalization of an ANN. As noted above, the classifier 610 can be implemented as an ANN. ANNs are information processing systems inspired by biological neural systems such as the brain. The key element of an ANN is the structure of the information processing system, which includes a large number of highly interconnected processing elements (called “neurons”) working in parallel to solve a particular problem. ANNs are also trained in use, with learning involving adjustment of weights present between neurons. ANNs are configured for a particular application such as pattern recognition or data classification by such a learning process.

[0076] ANNs demonstrate the ability to derive meaning from complex or inexact data, and can be used to extract patterns, as well as detect trends that are too complex to be detected by humans or other computer-based systems. Known neural networks generally have an input neuron 702 that provides information to one or more “hidden” neurons 704. Connections 708 between input neurons 702 and hidden neurons 704 are weighted, and these weighted inputs are then processed by the hidden neurons 704 according to some function in the hidden neurons 704 and the weighted connections 708 between layers. There can be any number of layers of hidden neurons 704, as well as neurons performing different functions. There are also different neural network structures, such as convolutional neural networks, maxout networks, and others. Finally, a set of output neurons 706 accept and process weighted inputs from the last set of hidden neurons 704.

[0077] This represents a "feed forward" computation, in which information propagates from the input neurons 702 to the output neurons 706. After the feed forward computation is complete, the output is compared to the desired output available from the training data. Errors associated with the training data are then processed in a "feed back" computation, in which the hidden neurons 704 and the input neurons 702 receive information about the error backpropagated from the output neurons 706. Once the backpropagation of error has been completed, weight updates are performed, and the weighted connections 708 are updated to take the received error into account. The above represents only one ANN.

[0078] Reference is now made to Figure 8 which shows an exemplary ANN architecture for the classifier 610, which corresponds to the generalized structure of Figure 7 It should be understood that the present architecture is merely exemplary, and other architectures or types of neural networks can be used instead. In particular, while hardware embodiments of ANNs are described in the present application, it should be understood that neural network architectures can be implemented or simulated in software. Also, the hardware embodiments described in the present application are intended to illustrate the general principles of neural network computation in a high level of generality, and should not be understood as limiting in any way.

[0079] Furthermore, the neuron layers described below and the weights connecting them are described in a general manner, and can be replaced by any type of neural network layer having any appropriate degree or type of interconnectivity. For example, the layers can include convolutional layers, pooling layers, fully connected layers, normalization exponential function (softmax) layers, or any other appropriate type of neural network layer. Furthermore, layers can be added or removed as desired, and weights can be omitted for more complex forms of interconnectivity.

[0080] During the feed forward operation, a set of input neurons 802 each provide an input voltage in parallel to a respective row of weights (W) 804. In the hardware embodiments described in the present application, the weights 804 each have a settable resistance value, such that a current output flows from the weights 804 to the respective hidden neuron 806 to represent the weighted input. In software embodiments, the weights 804 can simply be represented as coefficient values that are multiplied with the relevant neuron output.

[0081] According to the hardware embodiments, the current output by a given weight 804 is determined as where V is the input voltage from the input neuron 802 and r is the set resistance of the weight 804. The current from each weight is summed column-wise and flows to the hidden neuron 806. A set of reference weights 807 have a fixed resistance and their outputs are combined into a reference current that is provided to each of the hidden neurons 806. Since conductance values can only be positive, some reference conductance is needed to encode both positive and negative values in the matrix. The current produced by the weights 804 is a continuous value and is positive, so the reference weights 807 are used to provide a reference current above which the current is considered positive and below which the current is considered negative. The use of reference weights 807 is not needed in a software embodiment where the values of the outputs and weights can be obtained exactly and directly. As an alternative to using reference weights 807, another embodiment can use a separate array of weights 804 to obtain negative values.

[0082] The hidden neurons 806 use the currents from the array of weights 804 and the reference weights 807 to perform certain calculations. The hidden neurons 806 then output their own voltage to another array of weights 804. This array performs in the same manner, with a column of weights 804 receiving a voltage from their respective hidden neuron 806 to produce a weighted output current that is summed row-wise and provided to the output neuron 808.

[0083] It should be understood that any number of the above steps can be implemented by inserting additional array layers and hidden neurons 806. It should also be noted that some neurons can be constant neurons 809 that provide a constant output to the array. Constant neurons 809 can be present among the input neurons 802 and / or hidden neurons 806 and are used only during the forward pass operation.

[0084] During backpropagation, the output neuron 808 provides a voltage that is backpropagated across the array of weights 804. The output layer compares the generated network response to the training data and computes an error. This error is applied as a voltage pulse to the array, where the height and / or duration of the pulse is modulated in proportion to the error value. In this example, a row of weights 804 receives a voltage from the corresponding output neuron 808 in parallel and converts this voltage to a current that is summed column-wise to provide input to the hidden neurons 806. The hidden neurons 806 combine the weighted feedback signal with the derivative of their forward pass calculation and store the error value before outputting the feedback signal voltage to their respective column of weights 804. This backpropagation is carried through the entire network 800 until all of the hidden neurons 806 and input neurons 802 have stored an error value.

[0085] During weight update, the input neurons 802 and hidden neurons 806 are forward applied with a first weight update voltage, and the output neurons 808 and hidden neurons 806 are backward applied with a second weight update voltage, throughout the network 800. The combination of these voltages produces a state change within each weight 804, such that the weights 804 assume a new resistance value. In this way, the weights 804 can be trained to adapt the neural network 800 to errors in its processing. It should be noted that these three modes of operation, feedforward, backpropagation, and weight update, do not overlap one another.

[0086] As mentioned above, the weights 804 can be implemented in software or hardware, for example using relatively complex weighted circuitry or using resistive cross-point devices. Such resistive devices can have non-linear switching characteristics that can be used to process data. The weights 804 can belong to a class of devices known as resistive processing units (RPUs) in that their non-linear characteristics are used to perform computations in the neural network 800. RPU devices can be implemented with resistive random access memory (RRAM), phase change memory (PCM), programmable metallization cell (PMC) memory, or any other device having non-linear resistive switching characteristics. Such RPU devices can also be considered to be memristive systems.

[0087] Referring now to Figure 9 , an exemplary architecture for a classifier is shown. For an implementation that includes two types, with three possible degrees of data augmentation for each type, the classifier can be a nine-class classifier. The input can be a spectrogram of a sound snippet, and the output is a confidence value for each augmentation type. This input can be provided to one or more convolutional neural network (CNN) layers 902. The output of the CNN layers 902 is provided to a fully connected layer 904. Then, a normalization exponential function layer 906 generates the confidence values.

[0088] In more detail, the layers can be implemented as follows:

[0089] Input: Log Mel spectrogram (ch, freq, time)

[0090] CNN [64, k=(7, 1)] + BN + ReLU

[0091] Max pooling [k=(4, 1)] + Dropout (0.2)

[0092] CNN [128, k=(10, 1)] + BN + ReLU

[0093] CNN [256, k=(1, 7)] + BN + ReLU

[0094] Global max pooling (ch axis) + Dropout (0.5)

[0095] Dense (128)

[0096] Dense (class) + Softmax

[0097] In the above text, "BN" refers to batch normalization, ReLU refers to rectified linear unit, "max pooling" refers to the pooling layer of a CNN, "Dropout" refers to the random deactivation (dropout) layer, "Dense" refers to the densely connected layer, and "Softmax" refers to the normalized exponential function layer.

[0098] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0099] The features are as follows:

[0100] On-demand self-service: Cloud consumers can automatically and unilaterally supply computing power, such as server time and network storage, on demand, without requiring human interaction with the service provider.

[0101] Extensive network access: Capabilities are available on the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0102] Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. Location independence is significant because consumers typically do not have control or knowledge of the exact location of the provided resources, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0103] Rapid flexibility: Capacity can be supplied quickly and flexibly, and in some cases automatically, to rapidly shrink and rapidly expand. For the consumer, the available supply capacity often appears unlimited and can be purchased at any time and in any quantity.

[0104] Measured service: cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported providing transparency for both the provider and consumer of the service.

[0105] Service models are as follows:

[0106] Software as a Service (SaaS): the capability provided to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based e-mail). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0107] Platform as a Service (PaaS): the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.

[0108] Infrastructure as a Service (IaaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0109] Deployment models are as follows:

[0110] Private cloud: the cloud infrastructure is operated solely for the organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0111] Community cloud: the cloud infrastructure is shared by several organizations and supports mission-oriented business

[0112] Public cloud: the cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.

[0113] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursts for load balancing between clouds).

[0114] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. The core of cloud computing is its infrastructure, which includes a network of interconnected nodes.

[0115] Now for reference Figure 10 This describes an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. Nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services, without requiring cloud consumers to maintain resources on their local computing devices for these services. It should be understood that... Figure 10 The types of computing devices 54A-N shown are intended to be illustrative only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0116] Now for reference Figure 11 This demonstrates the 50 (cloud computing environment) Figure 10 This provides a set of functional abstractions. It should be understood beforehand. Figure 2 The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0117] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0118] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage device 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0119] In one example, management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources can include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which future requirements are anticipated in accordance with an SLA.

[0120] Workloads layer 90 provides examples of functionality for which the cloud computing environment can be utilized. Examples of workloads and functions which can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics processing 94; transaction processing 95; and abnormality detection 96.

[0121] Having described preferred embodiments for sound abnormality detection with mixed augmented dataset (which are intended to be illustrative and not limiting), it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes can be made in the particular embodiments of the application disclosed which are within the scope and spirit of the application as outlined by the appended claims. Having thus described the aspects of the present application, the same is now claimed as new and desired to be protected by Letters Patent and certain aspects are set forth in the appended claims.

Claims

1. A computer-implemented method for training a neural network to identify a form of data augmentation performed on a waveform, comprising: performing multiple forms of data augmentation on sample waveforms of a training dataset to generate normal data augmentation samples and abnormal data augmentation samples, the training dataset including both normal samples and abnormal samples; labeling the normal data augmentation samples to reflect a type of data augmentation performed on each respective normal data augmentation sample; labeling the abnormal data augmentation samples according to a type of data augmentation that is different from the type of data augmentation performed on each respective abnormal data augmentation sample; and training a neural network model using the normal data augmentation samples and the abnormal data augmentation samples to identify a form of data augmentation that has been performed on a waveform.

2. The method of claim 1, wherein the multiple forms of data augmentation include one or more types of data augmentation selected from a group comprising: pitch shift, time stretch, low / high pass filter, overlaid noise sound, time shift, decomposing a sound into harmonic and impulse components, shuffling a time sequence order of sound segments, averaging a sound, and spectral warping.

3. The method of claim 1, wherein the multiple forms of data augmentation include different degrees of a single type of data augmentation.

4. The method of claim 3, wherein the multiple forms of data augmentation include at least two distinct types of data augmentation.

5. The method of claim 4, wherein each form of data augmentation is performed to at least three different degrees to provide at least eight different forms of combined data augmentation.

6. The method of claim 4, wherein each form of data augmentation is performed to at least two different degrees to provide at least four different forms of combined data augmentation.

7. The method of claim 4, wherein the multiple forms of data augmentation include samples on which zero augmentation has been performed.

8. The method of claim 1, wherein the sample waveforms are selected from a group comprising: audio waveforms, accelerometer waveforms, vibration waveforms, inaudible pressure waveforms, seismic waveforms, and biometric waveforms. the normal samples are recorded from systems exhibiting normal behavior and the abnormal samples are recorded from systems exhibiting abnormal behavior.

9. The method of claim 1, wherein, 10. A computer program product for training a neural network, the computer program product comprising program instructions executable by a hardware processor to cause the hardware processor to perform the method of any one of claims 1-9.

11. A computer system, comprising: a processor; instructions that, when executed by the processor, perform the method of any one of claims 1-9.

12. A computer-implemented method for training a neural network to identify a form of data augmentation performed on a waveform, comprising: performing multiple forms of data augmentation on sample waveforms of a training dataset to generate normal data augmentation samples and abnormal data augmentation samples, the training dataset including both normal samples and abnormal samples; and ​ The normal data augmented samples and the abnormal data augmented samples are used to train a neural network model to identify the form of data augmentation that has been performed on the waveforms, wherein the abnormal data augmented samples contribute differently to a training loss function than the normal data augmented samples by moving abnormal samples away from correct classification.

13. The method of claim 12, wherein, The normal data augmented samples contribute to the loss function as and the abnormal data augmented samples contribute to the loss function as where, m is the number of normal samples, and n is the number of abnormal samples, p i is the normalized exponential function probability of the i th sample, and t i is the given label for the i th sample.

14. The method of claim 12, wherein, The normal data augmented samples are added to a cross-entropy loss and the abnormal data augmented samples are subtracted from the cross-entropy loss.

15. The method of claim 12, wherein the plurality of forms of data augmentation include one or more types of data augmentation selected from a group comprising: pitch shift, time stretch, low / high pass filter, overlaid noise sound, time shift, decomposing sound into harmonic and impulse components, shuffling the time sequential order of sound segments, averaging sound, and spectral warping.

16. The method of claim 12, wherein, The normal samples are recorded from systems that are behaving normally and the abnormal samples are recorded from systems that are behaving abnormally.

17. A computer program product for training a neural network, the computer program product comprising program instructions executable by a hardware processor to cause the hardware processor to perform the method of any one of claims 12-16.

18. A computer system comprising: a processor; instructions that, when executed by the processor, perform the method of any one of claims 12-16.

Images