Computer programs and computer-implemented methods for training neural networks (speech anomaly detection using mixed, augmented datasets)

Abstract

To solve the following problem: the number of abnormal samples in a training dataset may be relatively small, compared with the number of normal samples.SOLUTION: Methods and computer program products for training a neural network perform multiple forms of data augmentation on sample waveforms of a training dataset that includes both normal and abnormal samples to generate normal data augmentation samples and abnormal data augmentation samples. The normal data augmentation samples are labeled according to a type of data augmentation that was performed on each respective normal data augmentation sample. The abnormal data augmentation samples are labeled according to a type of data augmentation other than that which was performed on each respective abnormal data augmentation sample. A neural network model is trained to identify a form of data augmentation that has been performed on a waveform using the normal data augmentation samples and the abnormal data augmentation samples.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] Statements by the inventor or co-inventor regarding prior disclosures The following disclosure is filed pursuant to Section 102(b)(1)(A) of the United States Patent Act.

[0002] Disclosure

[0003] Detection of abnormal noises in machine conditions using classification confidence, Tadanobu Inoue, Phongtharin Vinayavekhin, Shu Morikuni, Shiqiang Wang, Tuan Hoang Trong, David Wood, Michiaki Tatsubori, Ryuki Tachibana, available from July 1, 2020.

[0004] Detection of abnormal sounds in machine conditions using classification confidence (paper), T. Inoue, P. Vinayavekhin, S. Morikuni, S. Wang, TH Trong, D. Wood, M. Tatsubori, R. Tachibana, available from November 2, 2020.

[0005] Detection of abnormal sounds in machine conditions using classification confidence (announcement), Tadanobu Inoue, Phongtharin Vinayavekhin, Shu Morikuni, Shiqiang Wang, Tuan Hoang Trong, David Wood, Michiaki Tatsubori, Ryuki Tachibana, available from November 2, 2020.

[0006] The present invention relates, in general terms, to the detection of anomalies in audio data, and more specifically, to distinguishing abnormal audio data from normal audio data using data augmentation of audio samples. [Background technology]

[0007] Anomaly detection attempts to find unusual samples in audio data. For example, anomaly input audio data can be recognized using a trained model, where a dataset of audio samples representing "normal" data is used to train the model. Examples of attempts to perform audio anomaly detection include reconstruction, where anomalies can be detected from the reconstruction errors of a trained autoencoder or generative adversarial network; feature learning, where a feature extraction model maps normal data to a small region of the feature space; classification, where a classifier is used to find samples outside the expected distribution; and geometric transformation, where a classifier is trained to infer geometric transformations of image data.

[0008] Each of these examples faces its own challenges. Geometric transformations, for example, are designed for images and perform poorly in applications where there is no prior knowledge of the geometric transformation laws for audio samples. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Detection of abnormal noises in machine conditions using classification confidence, Tadanobu Inoue, Phongtharin Vinayavekhin, Shu Morikuni, Shiqiang Wang, Tuan Hoang Trong, David Wood, Michiaki Tatsubori, Ryuki Tachibana, July 1, 2020 [Non-Patent Document 2] Detection of abnormal noises in machine conditions using classification confidence, T. Inoue, P. Vinayavekhin, S. Morikuni, S. Wang, TH Trong, D. Wood, M. Tatsubori, R. Tachibana, November 2, 2020. [Non-Patent Document 3] Detection of abnormal noises in machine conditions using classification confidence, Tadanobu Inoue, Phongtharin Vinayavekhin, Shu Morikuni, Shiqiang Wang, Tuan Hoang Trong, David Wood, Michiaki Tatsubori, Ryuki Tachibana, November 2, 2020. [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] The number of abnormal samples in a training dataset may be relatively small compared to the number of normal samples.

[0011] Computer programs for training neural networks perform multiple forms of data augmentation on sample waveforms in a training dataset containing both normal and abnormal samples, generating normal and abnormal data augmentation samples. Normal data augmentation samples are labeled according to the type of data augmentation performed on their respective normal samples. Abnormal data augmentation samples are labeled according to the type of data augmentation other than that performed on their respective abnormal samples. The neural network model is trained to identify the forms of data augmentation performed on the waveforms using the normal and abnormal data augmentation samples.

[0012] A computer program product for training a neural network performs multiple forms of data augmentation on the sample waveforms of a training data set that includes both normal samples and abnormal samples to generate normal data-augmented samples and abnormal data-augmented samples. The neural network model is trained to identify the forms of data augmentation performed on the waveforms using the normal data-augmented samples and the abnormal data-augmented samples. The abnormal data-augmented samples contribute to the training loss function so as to be different compared with the normal data-augmented samples.

[0013] These and other features and advantages will become apparent from the following detailed description of these exemplary embodiments when read in conjunction with the accompanying drawings.

[0014] The following description will provide details of the preferred embodiments in relation to the following figures.

Brief Description of the Drawings

[0015] [Figure 1] FIG. is a diagram showing various different forms of data augmentation performed on an input waveform sample, including different types of data augmentation performed at different degrees, according to an embodiment of the present invention.

[0016] [Figure 2] FIG. is a diagram showing the segmentation of an input waveform according to an embodiment of the present invention.

[0017] [Figure 3] FIG. is a block / flow diagram of a method for detecting an abnormality and performing a response operation according to an embodiment of the present invention.

[0018] [Figure 4] FIG. is a block / flow diagram of a method for training a classifier for detecting the form of data augmentation performed on a waveform sample according to an embodiment of the present invention.

[0019] [Figure 5] This is a block / flow diagram of a method for detecting anomalies using data augmentation classification according to one embodiment of the present invention.

[0020] [Figure 6] This is a block diagram of an anomaly detection and response system using data augmentation classification according to one embodiment of the present invention.

[0021] [Figure 7] This is a high-level diagram of a neural network layer that may be used to classify data augmentations, according to one embodiment of the present invention.

[0022] [Figure 8] This diagram shows a neural network architecture that could be used to classify data augmentations, according to one embodiment of the present invention.

[0023] [Figure 9] This is a diagram of a neural network classifier, according to one embodiment of the present invention, which may be used to classify data augmentations in waveform samples.

[0024] [Figure 10] This block diagram shows an exemplary cloud computing environment according to one embodiment, having one or more cloud computing nodes with which a local computing device used by a cloud consumer communicates.

[0025] [Figure 11] This is a block diagram showing a set of functional abstraction layers provided by a cloud computing environment, according to one embodiment. [Modes for carrying out the invention]

[0026] To detect anomalous speech data, speech augmentation using various different augmentation types can be applied to acoustic datasets. Machine learning models can also be trained to classify input speech segments according to the type of data augmentation applied. During anomaly detection, the input speech data is augmented in the same way as the training dataset. Machine learning models are further used to classify the input speech data according to different types of data augmentation. Anomaly scores can also be generated based on the confidence level at which the augmented input data is classified into the trained augmentation type class. Anomaly speech data generally has lower confidence levels and higher anomaly scores than normal speech data.

[0027] To further increase the effectiveness of this model, a training dataset is created using a combination of samples from normal and abnormal behavior. Abnormal samples are augmented in a way that intentionally provides inaccurate labels for the augmentation being performed. The number of abnormal samples in the training dataset may be relatively small compared to the number of normal samples. Often, the number of abnormal samples in the training dataset is small enough that binary classification without prior knowledge is inadequately performed.

[0028] While this disclosure includes a detailed description of cloud computing, it should be understood that the implementations described herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment that is currently known or may be developed in the future.

[0029] Referring to Figure 1, a diagram is shown illustrating the application of various different data augmentation processes to sound clip 100. In this example, two types of data augmentation are performed to create multiple augmented samples. This particular example shows pitch shifting and time stretching, each performed to three different degrees. For example, the original input sample 100 and eight augmented samples 102, each with a different corresponding degree (e.g., magnitude selected from -1, 0, and +1). a ~102 h Nine samples are shown, including [the one shown].

[0030] It should be understood that any number and type of data augmentation processes can be performed using any appropriate number of degrees. Exemplary types of speech data augmentation include pitch shifting, time stretching, low / high-pass filtering, noise overlay, temporal shifting, decomposition of speech into harmonic and percussive components, time-series shuffling of speech segments, speech averaging, and spectral warping. While two types of data augmentation are shown, each using three degrees, any number of data augmentation types can be used at any appropriate degree to produce a set of augmented samples of any appropriate size, resulting in a field of nine outputs. For example, a subset of four, five, or eight of the nine possible outputs may be implemented and used as described in more detail below.

[0031] Referring to Figure 2, the augmented sample can be divided into a set of 202 segments. It should be understood that segmentation can be performed before or after data augmentation. All segments 202 may have the same length or may have varying lengths. The start time of each subsequent segment 202 may be separated from the start time of the preceding segment 202 by a hop size of 204. Thus, segments 202 may overlap each other to varying degrees depending on the hop size 204. Segment length and hop size are hyperparameters that can be adjusted to maximize anomaly detection performance. As an example of one practical implementation, a 10-second sample size may have segment lengths of 2-3 seconds, where the hop size-to-segment length ratio is 50%.

[0032] The input samples 100 can have any suitable length, and the length of one sample may differ from that of the next. Furthermore, several types of data augmentation (e.g., time stretching) will affect the sample length. Using an invariant segment size helps to provide the classifier with a uniform sample regardless of the input length. For example, for an input sample with a length of 10 seconds, data augmentation may result in a set of samples that vary from approximately 9 seconds to approximately 11 seconds. In this example, segment 202 may have a length of approximately 3 seconds, with a hop size of approximately 1 second. Thus, each augmented sample may have a different number of segments 202 associated with it.

[0033] Referring to Figure 3, a method for detecting and responding to abnormal speech segments is shown. Training takes place in block 302. Training uses a machine learning model, such as one implemented by an artificial neural network (ANN). Training employs an input dataset containing various "normal" speech samples and several "abnormal" speech samples, and applies N types of data augmentation to each sample. For abnormal speech samples, inaccurate labels are applied to improve the performance of the classifier being trained. Each sample is divided into multiple speech segments. The segments are used to train the machine learning model.

[0034] While it is conceivable that a dataset may specifically include auditory information recorded within the frequency range of human hearing, it should be understood that this principle can be easily extended to inaudible pressure waves (e.g., ultrasonic sensor data or acoustic emission data), seismic information, biometric information (e.g., heart rate or electroencephalogram or respiratory sounds), vibration, accelerometer data, and any other types of data that can be converted into waveforms. For example, a time series recorded from sensors in a system can be represented as a waveform even if it does not contain any auditory information.

[0035] A machine learning model may include a set of classifiers, each of which is trained to recognize a corresponding data augmentation or combination of data augmentations and to output a corresponding probability. This probability reflects the likelihood that the input segment has been augmented according to the corresponding combination of data augmentations. For example, a classifier trained to recognize segments that have undergone a pitch shift of -1 and a time stretch of +1 will give a high probability output for normal segments that have undergone those data augmentations, but a lower probability output for segments that have not undergone those data augmentations.

[0036] A trained classifier for a particular set of data augmentation may also give a lower probability output to input segments generated from anomalous speech data, even though the corresponding combination of data augmentation was actually performed. This can occur because anomalous data may behave differently under data augmentation compared to normal speech data. Different data augmentations and degrees of augmentation can be used to increase the likelihood of such discrepancies occurring. This effect is amplified by introducing anomalous training data with inaccurate labels.

[0037] In training in block 302, the training dataset may be split into a training subset and a validation subset. As will be described in more detail below, the training dataset may be used in a backpropagation training process, in which case the output of the machine learning mode is compared to the expected output for a portion of the training data, and the error information is backpropagated through the model to update it. Once training is complete, the model may further be evaluated against the new training information from the validation subset to assess whether the machine learning model has been trained with sufficient generality. Training in block 302 may be repeated and refined until the accuracy of classifying the validation subset exceeds a threshold.

[0038] Block 304 receives a new input audio sample. This sample can originate from any source. Like a training dataset, the input audio sample may be audible speech information or represent any suitable waveform that matches the type of audio information used for training. Therefore, the sample may originate from any source suitable for recording such a type of waveform, such as a microphone, seismograph, heart rate monitor, or electroencephalogram.

[0039] Block 306 performs anomaly detection on new samples. Anomaly detection outputs an anomaly score for the sample based on the degree to which the data augmentation applied to the sample can be correctly classified, which is described in more detail below. If an anomaly is detected, block 308 performs a response action.

[0040] Anomaly detection can be used in a variety of applications, including equipment failure detection, product defect detection, network intrusion detection, fraud detection, medical diagnosis, and earthquake detection. Response actions can be used to provide a rapid response to new situations by responding quickly and automatically to any such anomaly. For example, an anomaly may indicate a product defect in a factory, in which case the defective product can be removed from the factory line and repaired. An anomaly may also indicate equipment failure, in which case the factory line may be stopped to repair the equipment and prevent further damage. In some cases, where the anomaly can be handled automatically, the response action may adjust the system's operating parameters to compensate, such as increasing cooling action when an overheating condition is detected.

[0041] Referring to Figure 4, further details about training block 302 are shown. Block 402 selects the first original sample from the training dataset. Block 404 then generates augmented samples from the original sample, for example, by using various types of data augmentation performed to different degrees. Each augmented sample is therefore characterized by the type of data augmentation performed on it and the corresponding degree of each type of data augmentation. An augmented sample may be identical to the original sample, as it may contain a degree of zero for each type of data augmentation.

[0042] If two distinct types of extensions are performed following the example in Figure 1, each of these has possible values ​​of {-1, 0, +1}, resulting in a total of nine different extension samples possible for each original sample. In some cases, four samples (corresponding to the corner samples in Figure 1 and omitting the original samples), five samples (corresponding to the original samples and corners in Figure 1), eight samples (corresponding to the eight outer samples in Figure 1 and omitting the original samples), or nine samples (corresponding to all the samples in Figure 1) may be selected.

[0043] Block 405 determines whether the selected sample corresponds to "normal" or "abnormal" operation. If the sample is "normal," block 406 correctly labels the data-augmented sample regarding the type of data augmentation that occurred. Otherwise, block 407 incorrectly labels the data-augmented sample regarding the type of data augmentation that occurred.

[0044] In block 408, the data augmentation samples are then segmented, for example, using a hop size to select fixed-length segments for stepping through the data augmentation samples. In block 409, it is determined whether there are any further original samples in the training dataset. If so, in block 412, the next sample is selected, and the process returns to block 404.

[0045] If no further original samples are available, the data augmentation segment is used by block 410 to train a machine learning model to recognize the type and degree of data augmentation. Training may utilize a loss function to characterize the difference between the model's output and the model's expected output. The loss may include, for example, a softmax loss and a center loss, in which case the former is characterized by descriptiveness and the latter by compactness. The loss function may be, for example, L=L S +λL C It can be expressed as, in the formula, L S This is the softmax loss, LC is the center loss, and λ is a parameter that determines the weight between the components of the loss. The center loss can be used to map normal input data to a minimized volume hyperspace in the latent feature space. In other examples, the loss function can simply be L = L S can be defined as.

[0046] Training is completed at block 414. The model can be tested against a validation dataset and repeated as necessary.

[0047] [[ID=,10]] In some embodiments, the labeling of blocks 406 and 407 can be performed using the table of FIG. 1, in which case each of the various types of data augmentation can be assigned to a corresponding label. Thus, augmentation 102 representing a pitch shift of -1 and a time stretch of -1 a may have a label separate from that of augmentation 102 representing a pitch shift of +1 and a time stretch of +1. In some examples, normal samples can be labeled in block 406 to reflect the actual augmentation being performed such that, for example, an augmentation with a pitch shift of +1 and a time shift of +1 will be labeled as 102 h . In such an example, abnormal samples can be labeled by block 407 with a label opposite to the actual augmentation being performed. Thus, an abnormal sample augmented with a pitch shift of +1 and a time shift of +1 can be labeled as 102, which is the opposing rectangle in FIG. 1 h . Similarly, an abnormal sample augmented according to 102[[ID=,18]] a can be labeled as if the augmentation of 102 d had been performed. e

[0048] The intended mislabeling in block 407 is explained in relation to a specific type of expansion shown in Figure 1, but it should be understood that it can be extended to any form of expansion. Therefore, if a particular anomalous sample is expanded in the first way, it may be labeled as if it were expanded in the second way, the opposite of the first. For anomalous samples that do not undergo expansion, they can be labeled accurately.

[0049] Other labeling patterns for block 407 are also possible. For example, a "round-robin" labeling method could be used, in which case each abnormal sample is labeled as if it had undergone the "next" type of expansion. Following the example in Figure 1, 102 a If expansion by 102 is performed on an abnormal sample, b The following labels may be applied. In this example, each abnormal sample (including samples to which no extensions have been applied) may be labeled according to the "next" extension, and any appropriate order of extensions can be selected.

[0050] As an alternative to the intended mislabeling of anomalous samples in block 407, the loss function can be modified during training in block 410 to move anomalous samples away from the correct classification. In some examples, L S The contribution of a normal sample may add to the cross-entropy loss, while the contribution of an abnormal sample may subtract from the cross-entropy loss. In other examples, for normal samples,

number

number

[0051] Referring to Figure 5, further details regarding anomaly detection in block 306 are shown. After a new sample is received in block 304, block 502 generates data-augmented samples from the new sample using, for example, the same set of data augmentation types and degrees used to generate the training data augmented samples in block 404. In some cases, the data augmentation in block 502 does not need to replicate all of the data augmentation types and degrees used during training. For example, some data augmentation types or degrees may be skipped to speed up inference.

[0052] After the new data augmentation samples are generated, block 504 then segments the new data augmentation samples using the same hop size and segment length that were used to segment the training data augmentation samples in block 406. Segmenting the samples can improve anomaly detection because anomalies can only occur in small parts of larger samples. Furthermore, dividing the samples into multiple segments increases the amount of training data available, which can improve the accuracy of the classifier.

[0053] In block 506, the trained model is used to classify the data-augmented segments into labels for the normal samples used during the training phase. Each segment is classified according to the type and degree of data augmentation performed, and an associated probability score is generated for the segment. For example, a softmax probability may be determined for each segment.

[0054] In block 508, the mean value for the probability of each corresponding new data augmentation sample segment is determined. Thus, each new data augmentation sample will have an associated score, which is the mean of the probabilities of each of its component segments. In block 510, an anomaly score for the new sample is determined. For example, this score is:

number

[0055] Once block 510 determines the anomaly score of a new sample, block 512 uses the anomaly score to determine whether the new sample represents an anomaly. For example, this may involve comparing the anomaly score to a threshold, where an anomaly score above the threshold indicates that an anomaly occurred, and an anomaly score at or below the threshold indicates that no anomaly occurred.

[0056] The present invention may be a system, method, or computer program product or combination thereof in an integration of any possible level of technical detail. The computer program product may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform an aspect of the present invention.

[0057] A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. A computer-readable storage medium may, for example, be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any preferred combination thereof. A non-exclusive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooves on which instructions are recorded, and any preferred combination thereof. Computer-readable storage media are not to be interpreted, as used herein, as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through optical fiber cables), or transient signals themselves, such as electrical signals transmitted through wires.

[0058] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the corresponding computing / processing device.

[0059] The computer-readable program instructions for performing the operation of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially as a standalone software package on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or a connection to an external computer may be made (for example, through the internet using an internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may, in order to perform an aspect of the present invention, utilize state information of computer-readable program instructions to execute computer-readable program instructions and personalize the electronic circuit.

[0060] Aspects of the present invention will be described herein with reference to flowcharts or block diagrams or combinations thereof of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram or combination thereof, as well as combinations of blocks in a flowchart or block diagram or combination thereof, can be implemented by computer-readable program instructions.

[0061] By providing these computer-readable program instructions to a processor of a computer or other programmable data processing device to manufacture a machine, the instructions executed by the processor of the computer or other programmable data processing device may provide a means for performing a function / operation specified in one or more blocks of a flowchart or block diagram or a combination thereof. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing device, or other device or a combination thereof to function in a particular way, so that the computer-readable storage medium on which the instructions are stored may contain a product containing instructions that perform a mode of function / operation specified in one or more blocks of a flowchart or block diagram or a combination thereof.

[0062] Computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device and perform a series of operational steps on the computer, other programmable device, or other device to produce a computer-executed process, so that instructions executed on a computer, other programmable device, or other device perform a function / operation specified in one or more blocks of a flowchart or block diagram or a combination thereof.

[0063] In this specification, any reference to “one embodiment” or “one embodiment” of the present invention, and to other variations thereof, means that certain features, structures, and properties, etc., described in relation to the embodiments, are included in at least one embodiment of the present invention. Therefore, the phrases “in one embodiment” or “in one embodiment,” and any other variations, appearing in various places throughout this specification, do not necessarily all refer to the same embodiment.

[0064] For example, in the cases of "A / B", "A or B or a combination thereof", and "at least one of A and B", it should be understood that the use of any of the following " / ", "~ or ... or a combination thereof", and "~ at least one of ~" is intended to encompass the selection of only the first enumerated option (A), or only the second enumerated option (B), or the selection of both options (A and B). As a further example, in the cases of "A, B, or C or a combination thereof", and "at least one of A, B, and C", such phrasing is intended to encompass the selection of only the first enumerated option (A), or only the second enumerated option (B), or only the third enumerated option (C), or only the first and second enumerated options (A and B), or only the first and third enumerated options (A and C), or only the second and third enumerated options (B and C), or the selection of all three options (A, B, and C). This can be expanded to include many of the listed items, as will be readily apparent to those skilled in the art in this and related fields.

[0065] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the system, method, and computer program product according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in a block may occur in an order other than that shown in the diagram. For example, two consecutively shown blocks may actually be implemented as a single stage that is executed substantially simultaneously, with some or all time overlap, or the blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, or a combination thereof, and any combination of blocks in a block diagram or flowchart, or a combination thereof, can be implemented by a dedicated hardware-based system that performs a specified function or operation, or a combination of dedicated hardware and computer instructions.

[0066] As used herein, the terms “hardware processor subsystem” or “hardware processor” may refer to a processor, memory, software, or combination thereof that works together to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices). One or more data processing elements may be included in a central processing unit, a graphics processing unit, or a separate processor-based or computing element-based controller or a combination thereof (e.g., logic gates). A hardware processor subsystem may include one or more onboard memories (e.g., caches, dedicated memory arrays, read-only memory). In some embodiments, a hardware processor subsystem may include one or more memories that can be onboard or offboard, or that can be dedicated to use by a hardware processor subsystem (e.g., ROM, RAM, Basic Input / Output System (BIOS)).

[0067] In some embodiments, a hardware processor subsystem may include and execute one or more software elements. These one or more software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0068] In other embodiments, the hardware processor subsystem may include specialized circuits dedicated to performing one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), FPGAs, or PLAs, or a combination thereof.

[0069] Other variations of these and other hardware processor subsystems according to embodiments of the present invention are also conceivable.

[0070] Referring here to Figure 6, the anomaly detection and response system 600 is shown. The system 600 includes a hardware processor 602 and memory 604. The sensor interface 606 provides communication with one or more sensors, which may include, for example, a microphone for collecting audio data, or alternatively, any sensor or combination of sensors that provide waveform or time-series output.

[0071] The classifier 610 is trained by the model trainer 614 and can be implemented as any suitable machine learning model, such as an ANN. The data augmentation unit 608 is used by the model trainer 614 to perform data augmentation on each original sample waveform from the training dataset, for example, using multiple types and degrees of data augmentation, in order to generate a set of data augmentation samples. The classifier 610 is trained to recognize the type and degree of data augmentation applied to a given segment of the sample.

[0072] The anomaly detector 612 receives a new sample from the sensor interface 606 and uses the data augmentation unit 608 to generate a data augmented sample. The classifier 610 further determines what type and degree of data augmentation has been performed on each data augmented sample and uses this to generate a probability for each combination of augmentation type and degree. The anomaly detector uses these probabilities to generate an anomaly score for the new sample and then uses the anomaly score to determine whether the new sample represents an anomaly.

[0073] The response function 616 is triggered by the detection of an anomaly. The response function 616 may include any appropriate action to correct, report, or otherwise address the detected anomaly.

[0074] Referring to Figure 7, a generalized diagram of an ANN is shown. As described above, classifier 610 can be implemented as an ANN. An ANN is an information processing system inspired by the biological nervous system, such as the brain. A key element of an ANN is the structure of an information processing system that includes a large number of highly interconnected processing elements (called "neurons") that work in parallel to solve a specific problem. An ANN is further trained in use through learning, which involves adjustments to the weights present between neurons. Through such a learning process, an ANN is configured for a specific application, such as pattern recognition or data classification.

[0075] ANNs demonstrate the ability to derive meaning from complex or ambiguous data and can be used to extract patterns that are too complex to detect by humans or other computer-based systems, and to detect trends. The structure of a neural network is generally known to have input neurons 702 that provide information to one or more "hidden" neurons 704. The connections 708 between the input neurons 702 and the hidden neurons 704 are weighted, and these weighted inputs are further processed by the hidden neurons 704 according to a certain function in the hidden neurons 704, also using the weighted connections 708 between layers. There can be any number of layers of hidden neurons 704, and neurons that perform different functions. Different neural network structures also exist, such as convolutional neural networks and max-out networks. Finally, a set of output neurons 706 accept and process weighted inputs from the last set of hidden neurons 704.

[0076] This represents a "feedforward" computation, in which case information propagates from input neuron 702 to output neuron 706. Once the feedforward computation is complete, the output is compared to the desired output available from the training data. The error to the training data is further processed in a "feedback" computation, in which hidden neuron 704 and input neuron 702 receive information about the error that is backpropagated from output neuron 706. Once backpropagation is complete, a weight update is performed so that the weighted connection 708 is updated to take the received error into account. This represents a single type of ANN.

[0077] Referring here to Figure 8, an exemplary ANN architecture for classifier 610 is shown, corresponding to the generalized structure in Figure 7. It should be understood that this architecture is illustrative in all respects, and other architectures or types of neural networks may be used instead. In particular, while hardware embodiments of ANNs are described herein, it should be understood that neural network architectures can be implemented or simulated in software. The hardware embodiments described herein are included to illustrate general principles of neural network computation with a high level of generality and should not be construed as limiting in any way.

[0078] Furthermore, the layers of neurons and the weights connecting them described below are generally described and can be replaced with any type of neural network layer with any appropriate degree or type of interconnectivity. For example, the layers may include convolutional layers, pooling layers, fully connected layers, softmax layers, or any other appropriate type of neural network layer. Moreover, layers can be added or removed as needed, and weights can be omitted for more complex forms of interconnectivity.

[0079] During feedforward operation, each set of input neurons 802 provides input voltages in parallel to the corresponding row of weights 804. In the hardware embodiments described herein, each weight 804 has a configurable resistance value such that a current output flows from the weight 804 to the corresponding hidden neuron 806 to represent the weighted input. In software embodiments, the weights 804 may simply be represented as coefficient values ​​multiplied by the relevant neuron outputs.

[0080] Following the hardware embodiment, the current output with a given weight 804 is:

number

[0081] Hidden neurons 806 use current from an array of weights 804 and a reference weight 807 to perform a certain calculation. Hidden neurons 806 also output their own voltages to another array of weights 804. This array does the same by having columns of weights 804 receive voltages from their corresponding hidden neurons 806 to produce a weighted current output that is added in the row direction and provided to output neurons 808.

[0082] It should be understood that any number of these stages can be implemented by interposing additional layers of the sequence and hidden neurons 806. It should also be noted that some neurons can be constant neurons 809 that provide a constant output to the sequence. Constant neurons 809 can reside among input neurons 802 or hidden neurons 806 or a combination thereof, and are used only during feedforward operation.

[0083] During backpropagation, the output neuron 808 provides a voltage that returns across the array of weights 804. The output layer compares the generated network response to the training data and calculates the error. The error is applied to the array as a voltage pulse, in which case the height, duration, or combination of the pulse is modulated proportionally to the error value. In this example, the rows of weights 804 receive voltages in parallel from the corresponding output neuron 808 and convert these voltages into currents that they add column by column to provide input to the hidden neuron 806. The hidden neuron 806 combines the weighted feedback signal with the derivative of its feedforward calculation and stores the error value before outputting the feedback signal voltage to the corresponding column of its weights 804. Backpropagation propagates throughout the network 800 until all hidden neurons 806 and input neurons 802 have stored the error value.

[0084] During weight updates, input neurons 802 and hidden neurons 806 apply a first weight update voltage forward, while output neurons 808 and hidden neurons 806 apply a second weight update voltage backward through the network 800. The combination of these voltages causes a state change within the range of each weight 804, giving the weights 804 new resistance values. In this way, the weights 804 can be trained to fit the neural network 800 to its processing errors. It should be noted that the three modes of operation—feedforward, backpropagation, and weight updates—do not overlap with each other.

[0085] As described above, the weights 804 can be implemented in software or hardware, for example, using a relatively complex weighting circuit or a resistive crosspoint device. Such resistive devices may have switching characteristics with nonlinearity that are usable for processing data. The weights 804 can belong to a class of devices called resistive processing units (RPUs), because their nonlinear characteristics are used to perform calculations in the neural network 800. RPU devices can be implemented with resistive random-access memory (RRAM®), phase-change memory (PCM), programmable metallization cell (PMC) memory, or any other device with nonlinear resistive switching characteristics. Such RPU devices may also be considered memristor systems.

[0086] Referring here to Figure 9, an illustrative structure of the classifier is shown. For an implementation that includes two types of data augmentation, each with three possible degrees, the classifier can be a 9-class classifier. The input can be a spectrogram of an audio segment, and the output is a confidence score for each augmentation type. The input can be supplied to one or more convolutional neural network (CNN) layers 902. The output of the CNN layer 902 is supplied to a fully connected layer 904. A softmax layer 906 further also generates confidence scores.

[0087] More specifically, the layer can be implemented as follows:

[0088] Input: Log-mel spectrogram (channel, frequency, time)

[0089] CNN[64, k=(7,1)]+BN+ReLU

[0090] Maximum pooling [k=(4,1)] + dropout (0.2)

[0091] CNN[128, k=(10,1)]+BN+ReLU

[0092] CNN[256, k=(1,7)]+BN+ReLU

[0093] Global max pooling (channel-axis) + dropout (0.5)

[0094] Density(128)

[0095] Density (Class) + Softmax

[0096] In the above, "BN" refers to batch normalization, ReLU refers to normalized linear units, "maximum pooling" refers to the pooling layer of a CNN, "dropout" refers to the dropout layer, "dense" refers to a densely connected layer, and "softmax" refers to the softmax layer.

[0097] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0098] The characteristics are as follows:

[0099] On-demand self-service: Cloud consumers can unilaterally provision computer functions such as server time and network storage automatically as needed, without requiring human interaction with the service provider.

[0100] Broad network access: Features that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs) are available over the network and accessible through standard mechanisms.

[0101] Resource Pool: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated as needed. It offers a sense of location independence in that consumers generally have no control or knowledge of the exact location of the resources provided, although they may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0102] Rapid scalability: Features can be provisioned quickly and flexibly, sometimes automatically, to scale out rapidly and release quickly to scale in rapidly. To consumers, the features available for provisioning often appear unlimited and can be purchased at any time and in any quantity.

[0103] Pay-as-you-go services: Cloud systems automatically control and optimize resource usage by leveraging instrumentation at an appropriate level of abstraction for the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage is monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0104] The service model is as follows:

[0105] Software as a Service (SaaS): The functionality provided to consumers is the use of a provider's applications running on cloud infrastructure. These applications are accessible from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functionalities, with the exception of limited user-specific application configuration settings.

[0106] Platform as a Service (PaaS): The functionality offered to consumers is the deployment of consumer-created or acquired applications on cloud infrastructure, using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they control the deployed applications and, in some cases, the applications hosting the environment configuration.

[0107] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provisioning of processing, storage, networking, and other underlying computing resources that enable consumers to deploy and run any software they choose, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating system, storage, deployed applications, and, in some cases, limited control over selected networking components (e.g., host firewalls).

[0108] The deployment model is as follows:

[0109] Private Cloud: Cloud infrastructure operates solely for a specific organization. This can be managed by that organization or a third party, and can reside on-premises or off-premises.

[0110] Community Cloud: A cloud infrastructure is shared by several organizations and supports a specific community that shares common concerns (e.g., mission, security requirements, policies, and compliance considerations). This can be managed by this organization or a third party and can reside on-premises or off-premises.

[0111] Public cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0112] Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain separate entities but are bound together by standardized or patented technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0113] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0114] Referring here to Figure 10, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N or a combination thereof. The nodes 10 can communicate with each other. They may be physically or virtually grouped (not shown) in one or more networks, such as private, community, public, or hybrid clouds, or a combination thereof, as described above. This makes it possible for the cloud computing environment 50 to provide infrastructure, platforms, or software or a combination thereof as a service, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N shown in Figure 10 are for illustrative purposes only, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device by any type of network or network-addressable connection or a combination thereof (e.g., using a web browser).

[0115] Referring here to Figure 11, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 10) is shown. It should be understood in advance that the components, layers, and functionalities shown in Figure 2 are for illustrative purposes only and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functionalities are provided:

[0116] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include a mainframe 61, a RISC (Reduced Instruction Set Computer) architecture-based server 62, a server 63, a blade server 64, a storage device 65, and network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0117] The virtualization layer 70 provides an abstraction layer that may offer the following examples of virtual entities: virtual servers 71, virtual storage devices 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0118] In one example, the management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 82 provides cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection of data and other resources. The user portal 83 gives consumers and system administrators access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management to meet required service levels. Service level agreement (SLA) planning and execution 85 provides proactive preparation and procurement of cloud computing resources where future requirements are anticipated in accordance with the SLA.

[0119] The workload layer 90 provides examples of functions that can be utilized in a cloud computing environment. Examples of workloads and functions that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, provision of virtual classroom education 93, data analysis processing 94, transaction processing 95, and anomaly detection 96.

[0120] While preferred embodiments of speech anomaly detection using mixed, augmented datasets (intended to be illustrative and not limiting) have been described, it should be noted that modifications and variations can be made by those skilled in the art in light of the above teachings. Accordingly, it should be understood that changes within the scope of the invention outlined by the appended claims can be made in the particular embodiments disclosed. Having thus described aspects of the invention with the details and specificity required by patent law, what is claimed and desired to be protected by patent certificate is set forth in the appended claims.

Claims

1. A computer program for training a neural network, which is powered by a hardware processor. A procedure for generating normal data augmentation samples and abnormal data augmentation samples by performing multiple types of data augmentation on sample waveforms of a training dataset that includes both normal and abnormal samples, A procedure for labeling each corresponding normal data augmentation sample according to the type of data augmentation performed on it, A procedure for labeling the anomalous data augmentation samples according to the type of data augmentation other than that performed on each corresponding anomalous data augmentation sample, A procedure for training a neural network model to identify the type of data augmentation performed on a waveform using the aforementioned normal data augmentation samples and the aforementioned abnormal data augmentation samples. A computer program designed to execute something.

2. The computer program according to claim 1, wherein the data augmentation of the plurality of forms includes one or more types of data augmentation selected from the group consisting of pitch shifting, time stretching, low / high-pass filtering, noise overlay, temporal shifting, decomposition of speech into harmonic and percussive components, shuffling of speech segments in chronological order, averaging of speech, and spectral warping.

3. The computer program according to claim 1, wherein the aforementioned multiple forms of data augmentation include a single type of data augmentation of different degrees.

4. The computer program according to claim 3, wherein the plurality of data extensions include at least two distinct types of data extensions.

5. The computer program according to claim 4, wherein each data extension of a given form is performed to at least three different degrees in order to provide a combination of at least eight different forms of data extension.

6. The computer program according to claim 4, wherein each type of data augmentation is performed to at least two different degrees to provide a combination of at least four different types of data augmentation.

7. The computer program according to claim 4, wherein the data augmentation of the plurality of forms includes a sample in which the augmentation performed is zero.

8. The computer program according to any one of claims 1 to 7, wherein the sample waveform is selected from the group consisting of audio waveforms, accelerometer waveforms, vibration waveforms, inaudible pressure waveforms, seismic waveforms, and biological waveforms.

9. The computer program according to any one of claims 1 to 7, wherein the normal sample is recorded from a system behaving normally, and the abnormal sample is recorded from a system behaving abnormally.

10. A computer-implemented method for training neural networks, The steps include: generating normal data augmentation samples and abnormal data augmentation samples by performing data augmentation in multiple formats on sample waveforms of a training dataset that includes both normal and abnormal samples; A step of labeling the normal data augmentation samples according to the type of data augmentation performed on each corresponding normal data augmentation sample, The steps include labeling the anomalous data augmentation samples according to the type of data augmentation other than that performed on each corresponding anomalous data augmentation sample, The steps include: training a neural network model to identify the type of data augmentation performed on a waveform using the normal data augmentation samples and the abnormal data augmentation samples; and A method that includes [a certain feature].

11. The method according to claim 10, wherein the plurality of data augmentation forms include one or more types of data augmentation selected from the group consisting of pitch shifting, time stretching, low / high-pass filtering, noise overlay, temporal shifting, decomposition of speech into harmonic and percussive components, shuffling of speech segments in chronological order, averaging of speech, and spectral warping.

12. The method according to claim 10, wherein the aforementioned multiple forms of data augmentation include a single type of data augmentation of different degrees.

13. The method according to claim 12, wherein the plurality of data augmentations include at least two distinct types of data augmentations, each performed to at least three different degrees, in order to provide at least nine different combined data augmentations.

14. The method according to any one of claims 10 to 13, wherein the sample waveform is selected from the group consisting of audio waveforms, accelerometer waveforms, vibration waveforms, inaudible pressure waveforms, seismic waveforms, and biological waveforms.

15. The method according to any one of claims 10 to 13, wherein the normal sample is recorded from a system behaving normally, and the abnormal sample is recorded from a system behaving abnormally.

16. A computer program for training a neural network, which is powered by a hardware processor. A procedure for generating normal data augmentation samples and abnormal data augmentation samples by performing multiple types of data augmentation on sample waveforms of a training dataset that includes both normal and abnormal samples, A procedure for training a neural network model to identify the form of data augmentation performed on a waveform using the normal data augmentation samples and the abnormal data augmentation samples, wherein the normal data augmentation samples and the abnormal data augmentation samples contribute to the training loss function such that the training correctly classifies the data augmentation performed on the normal data augmentation samples and moves away from the correct classification performed on the abnormal data augmentation samples. A computer program designed to execute something.

17. The aforementioned normal data augmentation sample is 【Number 5】 The aforementioned training loss function is used to contribute to the training loss function, and the aforementioned anomalous data augmentation samples are used to contribute to the training loss function. [Math 6] The above training loss function is contributed as follows, where m is the number of normal samples, n is the number of abnormal samples, and p ｉ is the softmax probability of the i-th sample, and t ｉ The computer program according to claim 16, wherein is a given label for the i-th sample.

18. The computer program according to claim 16, wherein the contribution of the normal data augmentation sample is added to the cross-entropy loss, and the contribution of the abnormal data augmentation sample is subtracted from the cross-entropy loss.

19. The computer program according to any one of claims 16 to 18, wherein the data augmentation of the plurality of forms includes one or more types of data augmentation selected from the group consisting of pitch shifting, time stretching, low / high-pass filtering, noise overlay, temporal shifting, decomposition of speech into harmonic and percussive components, shuffling of speech segments in chronological order, averaging of speech, and spectral warping.

20. The computer program according to any one of claims 16 to 18, wherein the normal samples are recorded from a system behaving normally, and the abnormal samples are recorded from a system behaving abnormally.

Images