Acoustic Event Detection System

Abstract

Generally, this disclosure describes a computing system that automatically identifies and classifies audio inputs that include non-speech audio signals, and is also capable of adding new classes based on only a limited number of examples of the new classes in order to identify classes of sounds for which the system has not been trained.

Description

[Technical field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 301,439, filed January 20, 2022, the entire contents of which are incorporated herein by reference.

[0002] This disclosure relates to automatic classification and identification of audio information.

[0003] (Government Interests) This invention was made with Government support under Contract No. N4175620C3057 awarded by the Irregular Warfare Technical Support Directorate (IWTSD). The United States Government has certain rights in this invention. [Background technology]

[0004] Audio data may include human utterances, including speech. Audio data may also include non-speech sounds, including human non-linguistic vocalizations such as humming, crying, yodeling, and similar vocalizations. Other examples of non-linguistic audio data may include non-human vocalizations, such as vocalizations from whales and other marine creatures, and a wide range of other non-linguistic audio, such as running water, weather sounds, and other sounds from nature, and mechanically generated sounds. Summary of the Invention

[0005] Generally, this disclosure describes a computing system that can implement techniques for automatically identifying and classifying audio inputs that include non-speech audio signals, and that can add new classes based on a limited number of examples of the new classes to identify classes of sounds that the system has not been trained on.

[0006] In one embodiment, the present disclosure describes a system comprising: an embedding extractor trained using an audio space including a plurality of non-speech sounds to generate an embedded representation of an input audio waveform; and a classifier configured to determine a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the classes of non-speech sounds by comparing the embedded representation of the input audio waveform to classes of non-speech sounds for which the classifier was trained, and to output the score for the input audio waveform.

[0007] In another embodiment, the disclosure describes a method including: generating, by an embedding extractor of a machine learning system, the embedding extractor being trained with an audio space including a plurality of non-speech sounds, an embedded representation of an input audio waveform, the embedded representation of the input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determining, by a classifier of the machine learning system, a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the classes of non-speech sounds for which the classifier was trained, by comparing the embedded representation of the input audio waveform to classes of non-speech sounds for which the classifier was trained; and outputting, by the classifier, the score for the input audio waveform.

[0008] In another embodiment, the disclosure describes a non-transitory computer-readable storage medium including instructions that, when executed, cause a processing circuit to: generate, by an embedding extraction device of a machine learning system, the embedding extraction device being trained using an audio space including a plurality of non-speech sounds, an embedded representation of an input audio waveform, the embedded representation of the input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determine, by a classifier device of the machine learning system, a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the classes of non-speech sounds for which the classifier device was trained, by comparing the embedded representation of the input audio waveform to classes of non-speech sounds for which the classifier device was trained; and outputting, by the classifier device, the score for the input audio waveform.

[0009] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the detailed description below. Other features, objects, and advantages of the disclosure will become apparent from the detailed description and drawings, and from the claims. [Brief description of the drawings]

[0010] [Figure 1] 1A and 1B are conceptual block diagrams illustrating one possible example arrangement of components for a sound event detection pipeline in accordance with one or more techniques of this disclosure.

[0011] [Diagram 2] FIG. 2 is a block diagram illustrating a computing system configured to implement a sound discrimination system in accordance with one or more techniques of this disclosure.

[0012] [Diagram 3] FIG. 3 is a conceptual diagram illustrating details of an example implementation of an embedding extraction device in accordance with one or more techniques of this disclosure.

[0013] [Figure 4] FIG. 4 is a block diagram of several electronic systems and devices communicating with each other in a network environment in accordance with one or more techniques of this disclosure.

[0014] [Diagram 5] FIG. 5 is a flow diagram illustrating an example of the operation of the acoustic event detection system of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In general, this disclosure describes a computing system that can implement techniques for automatically identifying and classifying audio inputs that include non-speech audio signals. In some examples, the computing system can add new classes by providing only a limited number of examples for which the computing system is not trained to identify sound classes. The computing system can include at least two components. A first component, an embedding extractor, can create a representation of world knowledge of the non-speech audio space, configured to distinguish non-speech phenomena from other non-speech phenomena. A second component, a classifier, can receive these embedding representations, including representations of non-speech sounds, and discriminate non-speech sounds from one another. The classifier can be trained, for example, to focus on specific non-speech sound classes that a user wants to discriminate and identify, rather than being trained on all non-speech sound classes.

[0016] The classifier is further configured to receive input that adds new classes of non-speech sounds that a user may want to identify without having to retrain the world knowledge contained in the embedding extractor. In other words, the classifier can register a new class of sound using a limited number of examples of the new class, e.g., using only a single example, and use the new class to discriminate and identify other input sounds. The classifier can operate based on only a few examples with extensive world knowledge provided by the embedded representations generated by the embedding extractor. A developer may register new classes, the computing system may operate on them, and deploy them as updates to the computing system. End users of the computing system may also register new classes with the system.

[0017] The computing system can enable a user to detect various desired sound classes within an audio recording, and can also process extensive data collections in an embedding extractor to find only those recordings that contain acoustic events of interest, saving a great deal of human time and effort.

[0018] Compared to detecting linguistic phenomena, the linguistic domain is much more limited than the general non-linguistic audio domain. Speech parts have limited frequency ranges, identifiable patterns, and include words, parts of words, intonation, and other features. However, the representations in an automated speech recognition system may not be adequate for a wide range of non-linguistic sounds, such as gunshots, birds chirping, and sounds caused by weather. In some examples, the classifier of the present disclosure can still identify some non-linguistic sounds using language-based representations, as long as the classifier senses non-linguistic events. However, the representations output by the embedding extractor of the present disclosure may be more diverse and larger, and may have a much larger number of dimensions, for example, compared to the embedding representations from an embedding extractor trained to identify speakers in human conversations.

[0019] FIG. 1A is a block diagram illustrating one possible example arrangement of components for a sound detection pipeline according to one or more techniques of the present disclosure. In the example of FIG. 1A, the system 100 includes an input device 102, an embedding extractor 106, a back-end classifier 108, a user interface 124, and a calibration module 110. The sound detection pipeline of the system 100 may receive an input sound, for example an input audio waveform 116 at the input device 102, and output a score 112 for the input sound. The score 112 may indicate whether the received input audio waveform 116 is the same as or different from a particular class of sound. The score 112 may be a number, text, a flag, or other value that indicates a class classification, for example a binary class classification. One or more components of the system 100 may be part of a machine learning system executed by processing circuitry of a computing engine. The computing engine may operate on a single computing device of the computing system 200 or may be distributed across multiple computing devices.

[0020] The input device 102 may be any device configured to receive sound and convert the sound into an electronic representation of the sound. In the example of FIG. 1A, the input device 102 may receive an input audio waveform 116 and output an audio spectrogram 104. In some examples, the input device 102 may include one or more microphones, cameras, and similar devices, in addition to circuitry for directly capturing an audio waveform and outputting an audio spectrogram 104. In other examples, the input device 102 may represent a peripheral device, such as a storage or memory device, an interface device, such as a USB port or a network interface, or other device configured to receive a representation of an audio waveform 116 and output an audio spectrogram 104. In some examples, the recording may include audio, video, data, and other information in analog or digital format that is stored in memory, streamed over a network to the input device 102, or obtained by the system 100 via the input device 102. In some examples, the input audio waveform 116 and the audio spectrogram 104 are substantially similar or identical.

[0021] The embedding extractor 106 may receive an input audio waveform 116, which may have been converted to an audio spectrogram 104 in some examples, and output an embedded representation 114 of the input audio waveform 116. The embedding extractor 106 may be trained with an audio space that includes a plurality of sounds, which in some examples may include non-speech sounds. The non-speech sounds may include naturally generated sounds, such as avalanches, birds chirping, beach waves, as well as mechanically generated sounds, such as sounds generated by motors, clocks, bells, explosions, and the like.

[0022] The system 100 of the present disclosure, and more specifically the embedding extractor 106, may include a deep neural network (DNN) model that is trained to generate an embedding representation and then utilized to create one or more other embedding representations for different sets of data. A speaker embedding representation is a highly discriminative, low-dimensional representation of an audio sample from a speaker. More generally, an embedding representation is a highly discriminative, low-dimensional representation of an audio sample. Embedding extraction refers to the process of extracting a portion of something and may be described as a technique that is learned to represent discrete variables as continuous vectors. The embedding extractor is typically a neural network trained to discriminate among classes of sounds, such as speakers, languages, phones, etc. For the purposes of the system of the present disclosure, embedding extraction may include mapping sounds, such as from the audio spectrogram 104, into vectors on which the processing circuitry of the present disclosure may perform analysis and association mapping. Embedding extraction may include the process of converting high-dimensional data into low-dimensional data in vector form in such a way that two distinct sounds may be categorized as similar or different to each other. The embedded representation 114 of the input audio waveform 116 may be a vector in some examples. In some examples, the embedded representation 114 of the input audio waveform 116 may include data for distinguishing particular non-speech sounds from other non-speech sounds in an audio space of sounds.

[0023] The back-end classifier 108 may receive an embedded representation 114 of the input audio waveform 116. As described above, the embedded representation 114 may be generated based on world knowledge of the audio space trained on the embedding extractor 106. The back-end classifier 108 is configured to discriminate between multiple sound classes. In some examples, the back-end classifier 108 may be implemented using Probabilistic Linear Discriminant Analysis (PLDA). The processing circuitry executing the back-end classifier 108 may use Linear Discriminant Analysis (LDA) to map high-dimensional features in the embedded representation 114 to a lower-dimensional space to perform class classification using PLDA. In some examples, the PLDA implementation may be trained to determine only two distinct classes, such as a binary choice. The two classes may be "same" or "different". In this manner, the backend classifier 108 can receive the embedded representation 114 for an input audio waveform 114 and determine whether the input audio waveform 114 can be classified as either "the same" as or "different" from one or more representations on which the backend classifier 108 was trained.

[0024] As an example to illustrate the operation of the system 100, the system 100 may receive an input audio waveform 116 via the input device 102. The back-end classifier 108 may be trained based on the representations of cat sounds, more specifically, based on the representations of cat sounds under specific conditions, such as cats fighting, being content (purring), being hungry, or based on other specific cat sound representations. By processing the embedding representations 114 generated from the embedding extractor 106, the back-end classifier 108 may calculate the likelihood that the received new audio waveform 116 processed using the pipeline of the system 100 is the same or different from those cat sound classes, generally the cat sound classes that the system 100 was trained on (or other classes in other use cases). In some examples, the back-end classifier 108 may further calculate the likelihood that the received audio is the same or different from the cat sound under the above specific conditions, such as purring. In some examples, the backend classifier 108 may be trained with a small number of examples of cat sounds, which may be as few as one example.

[0025] Other applications in which the system 100 of FIG. 1 may be useful may include a birdwatcher walking through a forest and hearing a new bird song not previously heard by the birdwatcher. The birdwatcher may want to know the species of bird producing the song. This may be accomplished by recording the sound and creating a detector, such as by training the backend classifier 108 with the new class of sounds. In some examples, the detector may then be used to match the novel bird song to a catalog of bird songs labeled by bird species. Additionally, the birdwatcher may upload the new bird song detector software to an audio recording device that is left in the forest to detect future occurrences of the bird song and record the date, time, temperature, and other parameters for future study.

[0026] Another application is a situation where a machine repair technician analyzes a motor that may be failing. The motor may emit some strange noises during operation, for example at random intervals. The technician detects the future occurrence of these noises and matches them to other measurable parameters in other examples (load, temperature, revolutions per minute, etc.) to determine whether these random sounds may be an indication of a possible catastrophic failure. Preventing that failure may prevent costly downtime and potentially expensive repair costs.

[0027] Thus, determining whether a particular sound contains important information about some event may be desirable for real-world applications. Therefore, it may be useful to have a system such as system 100 that can record past occurrences or future occurrences of the same sound in real time at short sample times to create a detector. In contrast to system 100 of the present disclosure, some other examples of audio detectors may only perform detection of a limited number of sound classes. Also, these other examples do not allow users to register new classes of sounds or update existing sound classes. The current systems and methods described in this disclosure may include detection of any existing sound events, such as gunshots, music, moving vehicles, background noise, animals, etc. Additionally, the systems of the present disclosure may include the ability to detect new classes of sounds by providing audio examples in addition to updating existing sound class detectors with new audio samples.

[0028] A calibration module 110 (illustrated as “calibrator 110”) of system 100 is configured to output a score used for comparing sounds. Calibrator 110 may include a score calibrator that provides an interpretable score. In some examples, calibration module 110 may run a logistic regression calibration module trained on the sound detection scores.

[0029] Although aspects of the system 100 are illustrated and primarily described as an end-to-end system for sound detection, they may also be implemented as a plug-in included in a speaker identification, sound detection, or other sound analysis system. For example, the plug-in may include an embedding extractor 106, a back-end classifier 108, and a calibration module 110 that receives audio spectrograms 104 from the sound analysis system and outputs scores 112 for use by the sound analysis system.

[0030] 2 is a block diagram illustrating a computing system in accordance with one or more techniques of this disclosure. In the example of FIG. 2, a computing engine 230 executes a machine learning system 232 on a computing system 200.

[0031] The computing engine 230 includes a processing circuit 220 for executing the machine learning system 232. The processing circuit 220 may be operatively connected to a recording device 226. The processing circuit 220 may include one or more processors in some examples. Examples of the processing circuit 220 may include one or more of a microcontroller (MCU), such as a computer on a single integrated circuit including a processor core, memory, and programmable input / output peripherals, and a microprocessor, such as a central processing unit (CPU) in an integrated circuit (IC) controller, controller, digital signal processor (DSP), application specific integrated circuit (ASIC), field-programmable gate array (FPGA), system on chip (SoC), or equivalent discrete or integrated logic circuit. The processor may be an integrated circuit, i.e., an integrated processing circuit, which may be embodied as a fixed hardware processing circuit, a programmable processing circuit, and / or a combination of processing circuitry that is both fixed and programmable. Thus, the terms "processing circuitry," "processor," or "controller" as used herein may refer to one or more of the above structures or any other structures operable to implement the techniques described herein. Processing circuitry 220 may be distributed across one or more computing devices interconnected by a computer network and may execute components of computing system 200, such as embedding extractor 206, classifier 208, calibrator 210, and user interface 224.

[0032] Examples of the recording device 226 may include a memory device or any type of computer-readable recording medium. The recording device 226 may comprise one or more of a random access memory (RAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), a one-time programmable memory (OTP), an electronically erasable programmable read only memory (EEPROM), a flash memory, or other types of volatile or non-volatile memory devices. In some examples, the computer-readable recording medium may store instructions that cause a processing circuit to perform the functions described herein. In some examples, the computer-readable recording medium may comprise data, such as configuration information, temporary values, and other types of data used to perform the functions of the present disclosure.

[0033] The machine learning system 232 includes an embedding extractor 206, a classifier 208, and a calibration module 210. The embedding extractor 206, the classifier 208, and the calibration module 210 are illustrative examples of the embedding extractor 106, the back-end classifier 108, and the calibration module 110 described above in connection with FIG. 1 and may have the same or similar features and functions. For example, the embedding extractor 206 may be trained using an audio space that includes world knowledge of sounds, including non-speech sounds. The embedding extractor 206 may receive an input audio waveform 216 via the input device 202 and output an embedded representation 214 of the input audio waveform 216 to the classifier 208. In some examples, the embedding extractor 206 includes a deep neural network trained to process the input audio waveform 216 using the audio space to generate high-dimensional features in the embedded representation 214 for the input audio waveform 216.

[0034] The classifier 208 is configured to receive an embedded representation 214 of the input audio waveform 216 from the embedding extractor 206. In some examples, the classifier 208 may compare the embedded representation 214, which may be a vector representation of the input audio waveform 216, to classes of non-speech sounds. In response to the comparison, the classifier 208 may determine a score 212 for the input audio waveform 216 that indicates whether the input audio waveform 216 is the same as or different from a selected class of non-speech sounds, as described above in connection with FIG. 1. In some examples, the calibration module 210 may comprise programming instructions, such as a logistic regression calibration module trained based on the sound detection scores. In some examples, determining the score 212 may include performing one or more classification operations (classification).

[0035] The computing engine 230 may output the scores 212 and other data or information via one or more output devices 228. The output devices 228 may include one or more of the following: an audio device configured to output sound, a display configured to present graphics, text, and similar information to a user, and communications circuitry configured to transmit the output to other devices, such as handheld computing devices, remote displays or audio devices, and other computing devices, e.g., in a network.

[0036] The machine learning unit 232 of the present disclosure may provide advantages over other types of audio classification machine learning systems. For example, the classifier may be further configured to receive the classification data 225, e.g., via the user interface 224, and the input audio waveforms 216 that add new classes of non-speech sounds that a user may want to identify, without having to retrain the world knowledge contained in the embedding extractor 206. The classifier 208 may register a new class of sound using only a limited number of examples of the new class, which in some instances may be a single example, and use the new class to discriminate and identify other input sounds.

[0037] In one implementation of adding a new class, a user may first identify, for example, via the user interface 224, a new input audio waveform 216 as a new class. This new input audio waveform 216 is processed by the DNN of the embedding extractor 206 as described above in connection with FIG. 1. The computing system 200 may extract high-dimensional features from the DNN and project them through the LDA space to reduce the dimensionality of the extracted embedding representation. A pre-trained PLDA classifier, for example the classifier 208, may model and decompose the extracted embedding representations into an acoustic model subspace and a channel space. The final model for the new class is this embedding representation projected into the PLDA acoustic model subspace.

[0038] In some examples, the embedding extractor 206 may receive the new audio waveform via a microphone or similar device, or the new audio waveform may be a recording or data, such as an audio spectrogram, as described above for the input device 102 in connection with FIG. 1. The classifier 208 may also receive user input, which may include a suggestion that the second input audio waveform is an example of a new class of non-speech sounds. In some examples, this user input may also include a label for the new class of non-speech sounds.

[0039] The classifier 208 may be configured to compare the embedded representations 214 of subsequent input audio waveforms received via the input device 202 to the new class of non-speech sounds. The classifier 208 may then output a score 212 for each subsequent waveform to indicate whether the subsequent waveform is the same or different from the new class. In some examples, the calibration module 210 may perform score calibration to provide interpretable scores, as described above in connection with FIG. 1. One example of an interpretable score is that if the system provides a score of 0.8 for an event, the event should occur 80% of the time.

[0040] An advantage of the machine learning system 232 of the present disclosure is that new classes can be added without the need to retrain the world knowledge of the audio space for the embedding extractor 206. In other words, the embedding extractor 206 is not changed as a result of adding a new class of non-speech sounds. Because training the embedding extractor 206 may use significant computational resources over a significant amount of time (e.g., weeks or months), adding a new class and adding the ability to discriminate received sounds against the new class may be useful compared to other sound event detection pipeline examples. In some examples, a developer or administrator of the machine learning system 232 may add a new class. In other examples, an end user of the machine learning system 232 may add a new class, for example, by adding an example audio waveform and an identification of that example via a user interface, as described above.

[0041] FIG. 3 is a conceptual diagram illustrating an example neural network structure for computing embedded representations according to one or more techniques of this disclosure. Embedding extractor 300 may be an example illustrating embedding extractor 106 or 206. The input sound representation may include an embedding representation of the sound estimated from a deep learning network pre-trained on audio samples. The embedding representation 314 is a 128-dimensional vector in the example of FIG. 3. The input features may be a 64 log Mel filter bank generated by Mel filter bank 340. In some examples, the features may be computed using a 1 second window of the audio spectrogram 344.

[0042] The neural network architecture components of the embedding extractor 300 may include one or more NN layers executed by a machine learning system, such as the machine learning system 232. The components in this example include a convolutional (CONV)64 layer block 321, a CONV128 block 322, a CONV256 block 324, a CONV512 block 326, a CONV512 block 328, and a separate pooling layer 346. The CONV128 block 322, the CONV256 block 324, the CONV512 block 326, and the CONV512 block 328 are associated with respective input pooling layers. The numbers 64, 128, 256, and 512 refer to the dimensionality of the input features.

[0043] In some examples, the Mel filter bank 340 may process an input audio spectrogram 344 and pass the input signal through a pre-emphasis filter. The filtered signal may be sliced ​​into (overlapping) frames and a window function may be applied to each frame. The Mel filter bank 340 may perform a Fourier transform, for example a short-time Fourier transform (STFT or SFT), on each frame and calculate a power spectrum for each frame, followed by the filter bank. In some examples, the final step of the Mel filter bank 340 may include mean normalization.

[0044] In some examples, the high-pass filter balances the frequency spectrum to correct high frequencies that are smaller in amplitude than lower frequencies. The high-pass filter may also perform other functions, such as to improve the signal-to-noise ratio (SNR) of the input signal 344. In other examples, this high-pass filter function may be omitted in lieu of a mean normalization step.

[0045] By dividing the signal into short frames, the frequency profile of the input signal can be preserved over time. A Fourier transform can be performed on the short time frames to provide an approximation of the frequency profile of the signal when concatenating adjacent frames. In some examples, the Mel filter bank 340 can also apply a windowing function, such as a Hamming window, to the frames. In some examples, the size of the frames can be on the order of milliseconds (ms), e.g., 5-60 ms, while in other examples the size of the frames can be on the order of about 1 second. The overlap can be about half the frame size in some examples, while in other examples there can be a larger or smaller overlap.

[0046] Computing the filter bank may involve applying a filter, such as a triangular filter, to the power spectrum output from the Fourier transform to extract frequency bands. The Mel scale roughly mimics the human ear's perception of sound, which is nonlinear, by being more discriminatory at lower frequencies and less discriminatory at higher frequencies.

[0047] The network structure in the example of Figure 3 also includes a CONV256 block 324, a CONV512 block 326, and a CONV512 block 328, and the embedded representation 314 is output after processing by a fully connected (FC) layer 330. However, Figure 3 illustrates only one example of an implementation of the embedding extraction of the present disclosure. In other examples, the embedding extractor may include more, fewer, or different components than in the arrangement. For example, the embedded representation 314 output from the FC layer 330 may comprise a 128-bit vector in some examples.

[0048] After a first representation of the input audio spectrogram 344 by the Mel filter bank 340, which adjusts the audio spectrum based on human perceptual hearing, the DNN component of the embedding extractor 300 can determine which of the feature vectors are important for isolating non-language audio events. In other words, the DNN can create features that contain information useful for isolating non-language audio events, for example in the CONV128 block 322. As described above in connection with FIG. 1, the embedding extractor 300 can be trained on a very large set of audio samples to develop world knowledge, for example knowledge of the non-language audio space, to determine which features stored in the representations can distinguish audio events from one another. These embedded representations 314 provided to a back-end classifier, for example the back-end classifier 108 of FIG. 1, enable the back-end classifier 108 to determine when audio information is different from other audio information.

[0049] Some examples of features / phenomena that can be included and used to distinguish sounds from one another may include frequency, frequency patterns over time, loudness, sharpness of the sound, e.g. a gunshot or a door slam compared to a siren or whale cry, duration, timbre, discreteness, structure, decay time, and similar features. A neural network can analyze the sound energy in each frame, e.g. a 10 ms frame, and a pooling layer at each step can combine the frames to determine the changes over time, the decays and patterns mentioned above over a longer period of time, e.g. on the order of 1 second.

[0050] The example of FIG. 3 includes one or more fully connected layers, the FC layer 330. The FC layer 330 may be implemented as a feed to a neural network forming the last few layers of the network. The input to the FC layer 330 may include the output from the final pooling or convolutional layer 346, which is flattened and then fed to the FC layer 330. In some examples, the FC layer 330 may compile the data extracted by the previous layers to form a final output, for example, an embedding representation 314. The embedding representation 314 may also be described as a nonlinear transformation of the input, for example, the world knowledge contained in the different non-speech sounds of the audio spectrogram 104 of FIG. 1, where the DNN of the embedding extractor 300 may be trained to discriminate between the classes of sounds. The output, i.e., the embedding representation 314, may comprise rich information that facilitates discriminating between different sounds in the space of non-speech audio. The embedding representation 314 becomes a feature vector for a simpler back-end classifier.

[0051] In this way, the disclosed system is in contrast to some models, such as support vector machine (SVM) techniques for language recognition, which may start directly from the MFCC (Mel-frequency cepstral coefficient) representation of the received language and arrive at a language class classification. In language recognition, the audio information is of smaller magnitude and fewer features are required for classification, and this world knowledge step as used by the disclosed system may not be used. The models are trained to detect specific features and classify specific phenomena. Instead, the disclosed system further culls the world knowledge from this largely amorphous non-language audio into an embedding representation that can be used by the back-end classifier. The embedding extractor is also not a classifier for the disclosed system.

[0052] An autoencoder that receives an input and performs feature identification can have an internal representation, the input features, and its output is a prediction of the input, e.g., how to best reproduce the input. Autoencoders do not use labels, since they are trained without labels, but the disclosed system, in contrast to autoencoders, can be equipped with labels for classes, which can provide a model that can determine whether an audio event is the same or different from another event with a pre-determined error rate.

[0053] The embedding extractor 300 may be trained on a large amount of acoustic events for predicting acoustic events. Once trained, the embedding extractor 300 maps any acoustic event (part of the training data or unseen data) into a high-dimensional space. However, as described above in connection with Figs. 1 and 2, adding a new class may require a large effort. Also, if most of the trained classes are not relevant to a particular task, retraining the full model is time-consuming and resource-intensive. However, the internal representation of the audio data generalizes the model to discriminate patterns (sounds) for which the model has not yet been trained. Now, when the embedding extractor 300 provides the embedding representations 314 to the back-end classifier of the present disclosure, the back-end classifier can identify new classes that the embedding extractor 300 has not been trained on with a low error rate, even if the back-end classifier has only been trained on a few examples. The back-end classifier can compare the new sound representations with sound representations, such as the embedding representations 314 from the embedding extractor 300. In this manner, the system of the present disclosure can distinguish new classes of sounds without having to retrain the embedding extractor 300.

[0054] In addition, because the embedded representations 314 may comprise reduced-dimensional vectors whose dimensions are focused on distinguishing non-speech sounds from each other, the back-end classifier of the present disclosure may perform comparisons using reduced computational resources compared to performing comparisons with representations of a larger number of dimensions.

[0055] 4 is a block diagram of multiple electronic systems and devices communicating with each other in a network environment in accordance with one or more techniques of this disclosure. Components of an audio classifier system, such as the system 100 of FIG. 1, the embedding extractor of FIG. 3, and the computing system 200 of FIG. 2, may operate in a distributed network environment.

[0056] The network environment includes a communication network 320 connecting server computing systems 304A-304B and at least one client computing system 302A-302G. As shown, there may be multiple server computing systems 304A-304B and multiple client computing systems 302A-302G connected to each other via the network 320. Examples of the network 320 may be or include one or more of an optical network, a cellular network, the Internet, a local area network (LAN), a wide area network (WAN), a satellite link, a fiber network, a cable network, or combinations thereof and / or others. The server computing systems 304A-304B may each include circuitry and software for communicating across the network 320 with the other server computing systems 304A-304B and the client computing systems 302A-302G. The server computing systems 304A-304B may each be associated with one or more databases 306A-306B and may be configured to store information, such as the storage device 226 of FIG. 2. The servers 304A-304B may each have one or more instances of a virtual server running on a physical server, and multiple virtual instances may be implemented depending on the design. A firewall may be established between the client computing systems, such as 302D, and the network 320 to protect data integrity on the client computing system 302D. Processing circuitry on one or more of the servers 304A-304B may execute components of the computing engine 230 and perform the functions described above in connection with FIGS. 1-3.

[0057] Cloud provider services can install and run application software in the cloud, and users can access the software services from client devices. Cloud users with sites in the cloud may not just manage the cloud infrastructure and platform on which their applications run; that is, servers and databases can be shared hardware where users are given dedicated use of a certain amount of these resources. A user's cloud-based site is given a virtual amount of dedicated space and bandwidth in the cloud. Cloud applications can differ from other applications in terms of scalability, which can be achieved by cloning tasks on multiple virtual machines during runtime to accommodate changing work demands. A load balancer distributes the work to the set of virtual machines. This process is transparent to the cloud users who only see a single access point.

[0058] The cloud-based remote access is coded to perform request and response cycles with both mobile device applications residing on the client devices 302A-302G and web browser applications residing on the client devices 302A-302G using protocols such as HTTP. In some circumstances, the cloud-based remote access for the wearable electronic device 302C may be accessed via a mobile device, desktop, or tablet device cooperating with the electronic device 302C. The cloud-based remote access between the client devices 302A, 302G and the cloud-based provider site 304A is coded to perform one or more of the following: (1) request and response cycles from all web browser-based applications, (2) SMS / twitter-based request and response message exchanges, (3) request and response cycles from a dedicated online server, (4) direct request and response cycles between native mobile applications on the client devices and the cloud-based remote access for the wearable electronic device, and (5) combinations thereof.

[0059] In one embodiment, the server computing system 304A may comprise a server engine, a web page management component or online services or online application component, a content management component, and a database management component. The server engine performs basic processing and operational system level tasks. The web page management component, online services, or online application component may handle the creation and display or routing of web pages or screens associated with receiving and serving digital content and digital advertisements. Users may access the server computing system through its associated URL. The content management component handles most of the functionality in the embodiments described herein. The database management component includes database related tasks of recording and searching, queries against the database, and recording data.

[0060] Figure 5 is a flow diagram illustrating an example of the operation of the acoustic event detection system of the present disclosure. As described above in connection with Figures 1-3, an embedding extractor of the present disclosure, for example the embedding extractor 106 of Figure 1, may receive (500) an input audio waveform 116, which may be a recording, a data file, or direct input via a microphone, via an input device 102. The embedding extractor may be trained with a large audio space that may include sounds including speech and non-speech sounds, and may generate (502) an embedded representation 114 (or 314) of the input audio waveform 116 as shown in Figures 1 and 3.

[0061] The embedding extractor may output 504 a representation of the input audio waveform to a classifier, such as the back-end classifier 108. This embedded representation 114 of the input audio waveform may comprise data, for example in the form of vectors, for distinguishing certain non-speech sounds from other non-speech sounds.

[0062] The classifier may receive an embedded representation of the input audio waveform, e.g., from the embedding extractor (506), and may compare the embedded representation of the input audio waveform to the classes of non-speech sounds (508). In response to the comparison, the classifier may determine a score for the input audio waveform (510). The score may indicate whether the input audio waveform is the same as or different from the classes of non-speech sounds, e.g., scores 112 and 212 of FIGS. 1 and 2, respectively. The classifier may output the score for the input audio waveform (512), e.g., via a user interface or other output device, as described above in connection with FIG. 2. In some examples, the machine learning system of the present disclosure implementing the acoustic event detection system may further apply a calibration to the scores of the audio waveform.

[0063] In one or more examples, the functions described above may be implemented in hardware, software, firmware, or a combination thereof. For example, various components of FIGS. 1-4, such as the embedding extractor 106, the machine learning system 232, the filter bank 340, and the convolution block of FIG. 3, may be implemented in hardware, software, firmware, or a combination thereof. If implemented in software, the functions may be stored on or transmitted on a computer-readable recording medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable recording medium that corresponds to a tangible medium, such as a data recording medium, or a communication medium, including a medium that facilitates transfer of a computer program from one place to another, such as according to a communication protocol. In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable recording medium, or (2) a communication medium, such as a signal or carrier wave. A data recording medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. The computer program product may include a computer-readable medium.

[0064] The techniques described in this disclosure may be implemented, at least in part, in hardware or a combination of hardware and software, firmware, or any combination thereof. For example, various aspects of the described techniques may be implemented in one or more processors, including one or more microprocessors, DSPs, ASICs, FPGAs, or any other equivalent logic circuitry, integrated or discrete, or any combination of such components. The term "processor" or "processing circuitry," such as with respect to processing circuitry 220, may generally refer to any of the aforementioned logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit including hardware may also perform one or more of the techniques of this disclosure.

[0065] Such hardware, software, and firmware may be implemented in the same device or in separate devices to support the various techniques described in this disclosure. In addition, any of the described units, modules, or components may be implemented together or separately as discrete but interoperable logic devices. The description of different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be realized by separate hardware, firmware, or software elements. Rather, functionality associated with one or more modules or units may be performed by separate hardware, firmware, or software elements or may be integrated within a common or separate hardware, firmware, or software element.

[0066] The techniques described in this disclosure may also be embodied or encoded in an article of manufacture that includes a computer-readable medium encoded with instructions. Instructions embedded or encoded in an article of manufacture that includes an encoded computer-readable medium may, for example, implement any of the techniques described herein in one or more programmable or other processors when the instructions contained in or encoded on the computer-readable medium are executed by the one or more processors. The computer-readable recording medium may include RAM, ROM, PROM, EPROM, EEPROM, flash memory, hard disk, compact disc ROM, floppy disk, cassette, magnetic medium, optical medium, or other computer-readable medium. In some examples, an article of manufacture may include one or more computer-readable media.

[0067] In some examples, a computer-readable recording medium may include a non-transitory medium. The term "non-transitory" may imply that the recording medium is not embodied in a carrier wave or propagated signal. In certain examples, a non-transitory recording medium may store data that may change over time (e.g., in RAM or cache).

Claims

1. 1. A system comprising: A computing engine including processing circuitry for executing a machine learning system, the machine learning system comprising: an embedding extractor trained with an audio space containing a plurality of non-speech sounds to generate an embedding representation of an input audio waveform; A classification device, comprising: determining a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the input audio waveform to classes of non-speech sounds on which the classifier has been trained; and a classification device configured to output the score for the input audio waveform; The system comprises:

2. 2. The system of claim 1, wherein the embedding extractor comprises a deep neural network (DNN) trained using the audio space to process the input audio waveform and generate high-dimensional features of the input audio waveform.

3. 10. The system of claim 1, wherein the machine learning system further comprises a calibration module executed by the computing engine, the calibration module being a logistic regression calibration module trained based on the sound detection scores.

4. The system of claim 1 , wherein to determine the score, the classifier is configured to perform class classification using Probabilistic Linear Discriminant Analysis (PLDA).

5. the input audio waveform is a first input audio waveform; The machine learning system includes: configured to receive a second input audio waveform; and configured to receive a user input, the user input comprising: an indication that the second input audio waveform is an example of a new class of non-speech sounds; and and labels for the new classes of non-speech sounds, the machine learning system further comprising: The second input audio waveform is processed by the embedding extraction device, and processing the second input audio waveform includes extracting a second extracted embedded representation comprising high-dimensional features from a deep neural network (DNN) and reducing a dimensionality of the second extracted embedded representation, and the classification device is decomposing the second extracted embedded representation into an acoustic model subspace and a channel subspace; and 2. The system of claim 1, wherein the trained classifier is configured to generate a final model of the new class including the high-dimensional features of the second extracted embedded representation projected into the acoustic model subspace without retraining the embedding extractor.

6. the classifier is configured to compare the embedded representation of the subsequent input audio waveform to the new classes of non-speech sounds; and The system of claim 5 , wherein the embedding extractor is not changed as a result of adding the new class of non-speech sounds.

7. 1. A method, comprising: generating an embedding representation of an input audio waveform by an embedding extractor of a machine learning system, the embedding extractor being trained using an audio space that includes a plurality of non-speech sounds, the embedding representation of the input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determining, by a classifier of the machine learning system, a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the input audio waveform with classes of non-speech sounds for which the classifier has been trained; outputting, by the classification device, the score for the input audio waveform; A method comprising:

8. 8. The method of claim 7, wherein the embedding extractor comprises a deep neural network (DNN) trained using the audio space to process the input audio waveform and generate high-dimensional features of the input audio waveform.

9. 8. The method of claim 7, further comprising calibrating the scores output by the classifier by a calibration module of the machine learning system, the calibration module being a logistic regression calibration module trained based on sound detection scores.

10. The method of claim 7 , wherein determining the score includes performing class classification using probabilistic linear discriminant analysis (PLDA).

11. The input audio waveform is a first input audio waveform, and the method comprises: receiving, by the machine learning system, a second input audio waveform; receiving, by the machine learning system, a user input, the user input comprising: an indication that the second input audio waveform is an example of a new class of non-speech sounds; and receiving user input including a label for said new class of non-speech sounds; processing the second input audio waveform by the machine learning system, where processing the second input audio waveform includes extracting a second extracted embedded representation comprising high dimensional features from a deep neural network (DNN) and reducing a dimensionality of the second extracted embedded representation; decomposing the second extracted embedded representation into an acoustic model subspace and a channel subspace by the machine learning system; 8. The method of claim 7, further comprising: generating, by the machine learning system, a final model of the new class that includes the high-dimensional features of the second extracted embedded representation projected into the acoustic model subspace without retraining the embedding extractor.

12. The method further includes comparing, by the classifier, the embedded representation of the subsequent input audio waveform to the new classes of non-speech sounds; The method of claim 11 , wherein the embedding extractor is not changed as a result of adding the new class of non-speech sounds.

13. 1. A non-transitory computer-readable storage medium containing instructions that, when executed, cause a processing circuit to: generating an embedding representation of an input audio waveform by an embedding extractor of a machine learning system, the embedding extractor being trained using an audio space that includes a plurality of non-speech sounds, the embedding representation of the input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determining, by a classifier of the machine learning system, a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the input audio waveform with classes of non-speech sounds for which the classifier has been trained; outputting, by the classification device, the score for the input audio waveform; A non-transitory computer-readable recording medium that causes the

14. 14. The non-transitory computer-readable storage medium of claim 13, wherein the embedding extractor comprises a deep neural network (DNN) trained using the audio space to process the input audio waveform and generate high-dimensional features of the input audio waveform.

15. 14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further cause the processing circuitry to calibrate the scores output by the classifier with a calibration module of the machine learning system, the calibration module being a logistic regression calibration module trained based on sound detection scores.

16. The non-transitory computer-readable storage medium of claim 13 , wherein determining the score includes performing class classification using probabilistic linear discriminant analysis (PLDA).

17. The input audio waveform is a first input audio waveform, and the instructions cause the processing circuit to: receiving a second input audio waveform; receiving a user input, the user input comprising: an indication that the second input audio waveform is an example of a new class of non-speech sounds; and receiving user input including a label for said new class of non-speech sounds; processing the second input audio waveform, the processing comprising: extracting a second extracted embedded representation comprising high dimensional features from a deep neural network (DNN) and reducing a dimensionality of the second extracted embedded representation; decomposing the second extracted embedded representation into an acoustic model subspace and a channel subspace; generating a final model of the new class including the high-dimensional features of the second extracted embedded representation projected into the acoustic model subspace without retraining the embedding extractor; The non-transitory computer-readable storage medium of claim 13 , further comprising:

18. The non-transitory computer-readable storage medium of claim 17 , wherein the classifier is a pre-trained PLDA classifier.

Images