Acoustic Event Detection System

Abstract

Generally, this disclosure describes a computing system that automatically identifies and classifies audio inputs that include non-speech audio signals, and is also capable of adding new classes based on only a limited number of examples of the new classes in order to identify classes of sounds for which the system has not been trained.

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 301,439, filed January 20, 2022, the entire contents of which are incorporated herein by reference.

[0002] TECHNICAL FIELD This disclosure relates to automatic classification and identification of audio information.

[0003] (Government Interests) This invention was made with government support under Contract No. N4175620C3057 awarded by the Irregular Warfare Technical Support Directorate (IWTSD). The United States Government has certain rights in this invention. [Background technology]

[0004] Audio data may include human utterances, including speech. Audio data may also include non-speech sounds, including human non-verbal vocalizations such as humming, crying, yodeling, and similar vocalizations. Other examples of non-verbal audio data may include non-human vocalizations, such as vocalizations from whales and other marine life, a wide variety of other non-verbal audio, such as the sounds of flowing water, weather sounds, and other natural sounds, and mechanically generated sounds. Summary of the Invention

[0005] Generally, this disclosure describes a computing system that can implement techniques for automatically identifying and classifying audio inputs, including non-speech audio signals, that can add new classes based on a limited number of examples of the new classes in order to identify classes of sounds for which the system has not been trained.

[0006] In one embodiment, the present disclosure describes a system comprising: a computing engine including processing circuitry for executing a machine learning system; an embedding extractor trained using an audio space including a plurality of non-speech sounds to generate an embedded representation of an input audio waveform; and a classifier configured to: determine a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the input audio waveform to classes of non-speech sounds on which the classifier was trained; and output the score for the input audio waveform.

[0007] In another embodiment, the present disclosure describes a method including: generating, by an embedding extractor of a machine learning system, the embedding extractor being trained using an audio space that includes a plurality of non-speech sounds, an embedded representation of an input audio waveform, the embedded representation of the input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determining, by a classifier of the machine learning system, a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the input audio waveform to classes of non-speech sounds for which the classifier was trained; and outputting, by the classifier, the score for the input audio waveform.

[0008] In another embodiment, the present disclosure describes a non-transitory computer-readable storage medium including instructions that, when executed, cause a processing circuit to: generate, by an embedding extractor of a machine learning system, the embedding extractor being trained using an audio space that includes a plurality of non-speech sounds, an embedded representation of an input audio waveform, the embedded representation of the input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determine, by a classifier of the machine learning system, a score for the input audio waveform that indicates whether the input audio waveform is the same as or different from the classes of non-speech sounds for which the classifier was trained, the score for the input audio waveform; and output, by the classifier, the score for the input audio waveform.

[0009] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the detailed description below. Other features, objects, and advantages of the disclosure will become apparent from the detailed description and drawings, and from the claims. [Brief explanation of the drawings]

[0010] [Figure 1] 1A and 1B are conceptual block diagrams illustrating one possible example arrangement of components for a sound event detection pipeline in accordance with one or more techniques of this disclosure.

[0011] [Figure 2] FIG. 2 is a block diagram illustrating a computing system configured to implement a sound discrimination system in accordance with one or more techniques of this disclosure.

[0012] [Figure 3] FIG. 3 is a conceptual diagram illustrating details of an example implementation of an embedding extraction device in accordance with one or more techniques of this disclosure.

[0013] [Figure 4] FIG. 4 is a block diagram of multiple electronic systems and devices communicating with each other in a network environment in accordance with one or more techniques of this disclosure.

[0014] [Figure 5] FIG. 5 is a flow diagram illustrating an example of the operation of the acoustic event detection system of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] Generally, this disclosure describes a computing system capable of implementing techniques for automatically identifying and classifying audio inputs, including non-speech audio signals. In some examples, the computing system can add new classes by providing only a limited number of examples for identifying sound classes for which the computing system has not been trained. The computing system can include at least two components. A first component, an embedding extractor, can create a representation of world knowledge of the non-speech audio space, the representation configured to distinguish non-speech phenomena from other non-speech phenomena. A second component, a classifier, can receive these embedding representations, including representations of non-speech sounds, and can discriminate non-speech sounds from one another. For example, the classifier can be trained to focus on specific non-speech sound classes that a user wants to distinguish and identify, rather than being trained on all non-speech sound classes.

[0016] The classifier is further configured to receive input that adds new classes of non-speech sounds that a user may want to identify, without having to retrain the world knowledge contained in the embedding extractor. In other words, the classifier can register a new class of sound using a limited number of examples of the new class, e.g., using only a single example, and then use the new class to discriminate and identify other input sounds. The classifier can function based on only a few examples with extensive world knowledge provided by the embedded representations generated by the embedding extractor. A developer may register new classes, and the computing system may operate on them and deploy them as updates to the computing system. End users of the computing system may also register new classes with the system.

[0017] The computing system can enable a user to detect various desired sound classes within an audio recording, and can also process extensive data collections in an embedding extractor to find only recordings containing acoustic events of interest, saving a great deal of human time and effort.

[0018] Compared to detecting linguistic phenomena, the linguistic domain is much more limited than the general non-linguistic audio domain. Speech parts have limited frequency ranges, identifiable patterns, and include words, parts of words, intonation, and other features. However, representations in automated speech recognition systems may not be adequate for a wide range of non-linguistic sounds, such as gunshots, birdsong, and weather-related sounds. In some examples, the classifier of the present disclosure can still identify some non-linguistic sounds using language-based representations, as long as the classifier is sensitive to non-linguistic events. However, the representations output by the embedding extractor of the present disclosure may be more diverse and larger, and may have a significantly greater number of dimensions than embedding representations from an embedding extractor trained to identify speakers in human conversations, for example.

[0019] FIG. 1A is a block diagram illustrating one possible example arrangement of components for a sound detection pipeline in accordance with one or more techniques of this disclosure. In the example of FIG. 1A, system 100 includes input device 102, embedding extractor 106, back-end classifier 108, user interface 124, and calibration module 110. The sound detection pipeline of system 100 may receive an input sound, e.g., an input audio waveform 116, at input device 102, and output a score 112 for the input sound. The score 112 may indicate whether the received input audio waveform 116 is identical to or different from a particular sound class. The score 112 may be a number, text, a flag, or other value indicating a classification, such as a binary classification. One or more components of system 100 may be part of a machine learning system executed by processing circuitry of a computing engine. The computing engine may operate on a single computing device of computing system 200 or may be distributed across multiple computing devices.

[0020] The input device 102 may be any device configured to receive sound and convert it into an electronic representation of sound. In the example of FIG. 1A , the input device 102 can receive an input audio waveform 116 and output an audio spectrogram 104. In some examples, the input device 102 may include one or more microphones, cameras, and similar devices, in addition to circuitry for directly capturing an audio waveform and outputting the audio spectrogram 104. In other examples, the input device 102 may represent a peripheral device such as a storage or memory device, an interface device such as a USB port or network interface, or other device configured to receive a representation of the audio waveform 116 and output the audio spectrogram 104. In some examples, a recording may include audio, video, data, and other information in analog or digital format that is stored in memory, streamed over a network to the input device 102, or obtained by the system 100 via the input device 102. In some examples, the input audio waveform 116 and the audio spectrogram 104 are substantially similar or identical.

[0021] The embedding extractor 106 can receive an input audio waveform 116, which in some examples has been converted to an audio spectrogram 104, and output an embedded representation 114 of the input audio waveform 116. The embedding extractor 106 can be trained using an audio space containing a plurality of sounds, which in some examples include non-speech sounds. The non-speech sounds can include naturally occurring sounds, such as avalanches, birdsong, and waves on a beach, as well as mechanically occurring sounds, such as sounds produced by motors, clocks, bells, explosions, and the like.

[0022] The system 100 of the present disclosure, and more specifically the embedding extractor 106, may include a deep neural network (DNN) model trained to generate an embedding representation and then utilized to create one or more other embedding representations for different sets of data. A speaker embedding representation is a highly discriminative, low-dimensional representation of an audio sample from a speaker. More generally, an embedding representation is a highly discriminative, low-dimensional representation of an audio sample. Embedding extraction refers to the process of extracting a portion of something and may be described as a technique learned to represent discrete variables as continuous vectors. An embedding extractor is typically a neural network trained to distinguish among classes of sounds, such as speakers, languages, phones, etc. For the system of the present disclosure, embedding extraction may include mapping sounds, for example, from the audio spectrogram 104, into vectors on which processing circuitry of the present disclosure can perform analysis and association mapping. Embedding extraction may include the process of converting high-dimensional data into low-dimensional data in vector form in such a way that two distinct sounds can be categorized as similar or different from each other. The embedded representation 114 of the input audio waveform 116 may be a vector in some examples. In some examples, the embedded representation 114 of the input audio waveform 116 may include data for distinguishing particular non-speech sounds from other non-speech sounds in the audio space of sounds.

[0023] The backend classifier 108 may receive an embedded representation 114 of the input audio waveform 116. As described above, the embedded representation 114 may be generated based on world knowledge of the audio space trained on the embedding extractor 106. The backend classifier 108 is configured to distinguish between multiple sound classes. In some examples, the backend classifier 108 may be implemented using probabilistic linear discriminant analysis (PLDA). The processing circuitry executing the backend classifier 108 may use linear discriminant analysis (LDA) to map high-dimensional features in the embedded representation 114 to a lower-dimensional space and perform class classification using PLDA. In some examples, such PLDA implementations may be trained to distinguish between only two distinct classes, such as a binary choice. The two classes may be "same" or "different." In this manner, the backend classifier 108 can receive the embedded representation 114 for the input audio waveform 114 and determine whether the input audio waveform 114 can be classified as either "the same" as or "different" from one or more representations on which the backend classifier 108 was trained.

[0024] As an example illustrating the operation of the system 100, the system 100 can receive input audio waveforms 116 via the input device 102. The backend classifier 108 can be trained based on representations of cat sounds, more specifically, based on representations of cat sounds under specific conditions, such as cats fighting, being content (purring), being hungry, or other specific cat sound representations. By processing the embedded representations 114 generated from the embedding extractor 106, the backend classifier 108 can calculate the likelihood that a new received audio waveform 116 processed using the pipeline of the system 100 is the same as or different from the class of cat sounds, and generally, the class of cat sounds for which the system 100 was trained (or, in other use cases, other classes). In some examples, the backend classifier 108 can further calculate the likelihood that the received audio is the same as or different from a cat sound under the specific conditions, such as purring. In some examples, the backend classifier 108 can be trained with a small number of examples of cat sounds, which may be as few as one example.

[0025] Another application in which the system 100 of FIG. 1 may be useful may include a birdwatcher walking through a forest and hearing a new bird song not previously heard by the birdwatcher. The birdwatcher may want to know the species of bird producing the song. This can be accomplished by recording the sound and creating a detector, for example, by training the backend classifier 108 with the new class of sound. In some examples, this detector can then be used to match the novel bird song to a catalog of bird songs labeled by bird species. Additionally, the birdwatcher can upload new bird song detector software to an audio recording device left in the forest to detect future occurrences of the bird song and record the date, time, temperature, and other parameters for future study.

[0026] Another application is a situation in which a machine repair technician analyzes a motor that may be failing. The motor may emit some unusual noises during operation, for example, at random intervals. The technician detects the future occurrence of these noises and matches them to other measurable parameters (load, temperature, revolutions per minute, etc.) to determine whether these random sounds may be an indication of a possible catastrophic failure. Preventing that failure may prevent costly downtime and potentially expensive repair costs.

[0027] Thus, determining whether a particular sound contains important information about some event may be desirable for real-world applications. Therefore, it may be useful to have a system such as system 100 that can record past or future occurrences of the same sound in real time with short sample times to create a detector. In contrast to system 100 of the present disclosure, some other examples of audio detectors may only perform detection of a limited number of sound classes. Furthermore, these other examples do not allow users to register new classes of sounds or update existing sound classes. The current systems and methods described in this disclosure may include detection of any existing sound events, such as gunshots, music, moving vehicles, background noise, animals, etc. Additionally, the systems of the present disclosure may include the ability to detect new classes of sounds by providing audio examples, in addition to updating existing sound class detectors with new audio samples.

[0028] Calibration module 110 (illustrated as "calibrator 110") of system 100 is configured to output scores used for sound comparison. Calibration module 110 may include a score calibrator that provides interpretable scores. In some examples, calibration module 110 may run a logistic regression calibration module trained on the sound detection scores.

[0029] Although aspects of system 100 are illustrated and primarily described as an end-to-end system for sound detection, they may also be implemented as a plug-in included in a speaker identification, sound detection, or other sound analysis system. For example, the plug-in may include an embedding extractor 106, a back-end classifier 108, and a calibration module 110 that receives audio spectrograms 104 from the sound analysis system and outputs scores 112 for use by the sound analysis system.

[0030] 2 is a block diagram illustrating a computing system in accordance with one or more techniques of this disclosure. In the example of FIG. 2, a computing engine 230 executes a machine learning system 232 on a computing system 200.

[0031] The computing engine 230 includes processing circuitry 220 for executing the machine learning system 232. The processing circuitry 220 may be operatively connected to a recording device 226. The processing circuitry 220 may, in some examples, include one or more processors. Examples of the processing circuitry 220 may include one or more of: a microcontroller (MCU), such as a computer on a single integrated circuit including a processor core, memory, and programmable input / output peripherals; and a microprocessor, such as a central processing unit (CPU) in an integrated circuit (IC) controller, controller, digital signal processor (DSP), application specific integrated circuit (ASIC), field-programmable gate array (FPGA), system on chip (SoC), or equivalent discrete or integrated logic circuitry. The processor may be an integrated circuit, i.e., an integrated processing circuit. The integrated processing circuit may be embodied as a fixed hardware processing circuit, a programmable processing circuit, and / or a combination of both fixed and programmable processing circuitry. Thus, the terms "processing circuitry," "processor," or "controller," as used herein, may refer to one or more of the above structures or any other structure operable to implement the techniques described herein. Processing circuitry 220 may be distributed across one or more computing devices interconnected by a computer network and may implement components of computing system 200, such as embedding extractor 206, classifier 208, calibrator 210, and user interface 224.

[0032] Examples of the recording device 226 may include a memory device or any type of computer-readable storage medium. The recording device 226 may comprise one or more of random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable memory (OTP) memory, electronically erasable programmable read-only memory (EEPROM), flash memory, or other types of volatile or non-volatile memory devices. In some examples, the computer-readable storage medium may store instructions that cause a processing circuit to perform the functions described herein. In some examples, the computer-readable storage medium may comprise data, such as configuration information, temporary values, and other types of data used to implement the functions of the present disclosure.

[0033] The machine learning system 232 includes an embedding extractor 206, a classifier 208, and a calibration module 210. The embedding extractor 206, the classifier 208, and the calibration module 210 are illustrative examples of the embedding extractor 106, the backend classifier 108, and the calibration module 110 described above in connection with FIG. 1 and may have the same or similar features and functionality. For example, the embedding extractor 206 may be trained using an audio space that includes world knowledge of sounds, including non-speech sounds. The embedding extractor 206 may receive an input audio waveform 216 via the input device 202 and output an embedded representation 214 of the input audio waveform 216 to the classifier 208. In some examples, the embedding extractor 206 includes a deep neural network trained to process the input audio waveform 216 using the audio space to generate high-dimensional features in the embedded representation 214 for the input audio waveform 216.

[0034] The classifier 208 is configured to receive an embedded representation 214 of an input audio waveform 216 from the embedding extractor 206. In some examples, the classifier 208 may compare the embedded representation 214, which may be a vector representation of the input audio waveform 216, to classes of non-speech sounds. In response to this comparison, the classifier 208 may determine a score 212 for the input audio waveform 216 that indicates whether the input audio waveform 216 is the same as or different from a selected class of non-speech sounds, as described above in connection with FIG. 1. In some examples, the calibration module 210 may comprise programming instructions, such as a logistic regression calibration module trained based on the sound detection scores. In some examples, determining the score 212 may include performing one or more classification operations (classification).

[0035] The computing engine 230 may output the scores 212 and other data or information via one or more output devices 228. The output devices 228 may include one or more of the following: an audio device configured to output sound, a display configured to present graphics, text, and similar information to a user, and communications circuitry configured to transmit output to other devices, such as handheld computing devices, remote displays or audio devices, and other computing devices, for example, in a network.

[0036] The machine learning unit 232 of the present disclosure may provide advantages over other types of audio classification machine learning systems. For example, the classifier may be further configured to receive classification data 225, e.g., via user interface 224, and input audio waveforms 216 that add new classes of non-speech sounds that a user may want to identify, without having to retrain the world knowledge contained in the embedding extractor 206. The classifier 208 can register a new class of sound using only a limited number of examples of the new class, which in some instances may be a single example, and use the new class to discriminate and identify other input sounds.

[0037] In one implementation of adding a new class, a user may first identify a new input audio waveform 216 as a new class, for example, via user interface 224. This new input audio waveform 216 is processed by the DNN of embedding extractor 206, as described above in connection with FIG. 1. Computing system 200 can extract high-dimensional features from the DNN and project them through an LDA space to reduce the dimensionality of the extracted embeddings. A pre-trained PLDA classifier, for example, classifier 208, can model and decompose these extracted embeddings into an acoustic model subspace and a channel space. The final model for the new class is this embedding projected into the PLDA acoustic model subspace.

[0038] In some examples, the embedding extractor 206 may receive the new audio waveform via a microphone or similar device, or the new audio waveform may be a recording or data, such as an audio spectrogram, as described above for the input device 102 in connection with FIG. 1. The classifier 208 may also receive user input, which may include a suggestion that the second input audio waveform is an example of a new class of non-speech sounds. In some examples, this user input may also include a label for the new class of non-speech sounds.

[0039] The classifier 208 may be configured to compare the embedded representations 214 of subsequent input audio waveforms received via the input device 202 to the new class of non-speech sounds. The classifier 208 may then output a score 212 for each subsequent waveform to indicate whether the subsequent waveform is the same as or different from the new class. In some examples, the calibration module 210 may perform score calibration to provide interpretable scores, as described above in connection with FIG. 1. One example of an interpretable score is if the system provides a score of 0.8 for an event, then the event should occur 80% of the time.

[0040] An advantage of the machine learning system 232 of the present disclosure is that new classes can be added without having to retrain the world knowledge of the audio space for the embedding extractor 206. In other words, the embedding extractor 206 is not changed as a result of adding a new class of non-speech sounds. Because training the embedding extractor 206 can take a significant amount of time (e.g., weeks or months) and significant computational resources, adding a new class and the ability to discriminate received sounds against that new class can be useful compared to other sound event detection pipeline examples. In some examples, a developer or administrator of the machine learning system 232 may add a new class. In other examples, an end user of the machine learning system 232 may add a new class by, for example, adding an example audio waveform and an identification of that example via a user interface, as described above.

[0041] FIG. 3 is a conceptual diagram illustrating the structure of an example neural network for computing embedded representations, according to one or more techniques of this disclosure. Embedding extractor 300 may represent an example illustrating embedding extractor 106 or 206. The representation of the input sound may include an embedded representation of the sound estimated from a deep learning network pre-trained on audio samples. In the example of FIG. 3, the embedded representation 314 is a 128-dimensional vector. The input features may be a 64 log Mel filter bank generated by Mel filter bank 340. In some examples, the features may be computed using a 1-second window of the audio spectrogram 344.

[0042] The neural network architecture components of the embedding extractor 300 may include one or more NN layers executed by a machine learning system, such as the machine learning system 232. In this example, the components include a convolutional (CONV)64 layer block 321, a CONV128 block 322, a CONV256 block 324, a CONV512 block 326, a CONV512 block 328, and a separate pooling layer 346. The CONV128 block 322, the CONV256 block 324, the CONV512 block 326, and the CONV512 block 328 are associated with respective input pooling layers. The numbers 64, 128, 256, and 512 refer to the dimensionality of the input features.

[0043] In some examples, the Mel filter bank 340 may receive an input audio spectrogram 344 and pass the input signal through a pre-emphasis filter. The filtered signal may be sliced ​​into (overlapping) frames and a window function may be applied to each frame. The Mel filter bank 340 may perform a Fourier transform, such as a short-time Fourier transform (STFT or SFT), on each frame, calculate a power spectrum for each frame, and then calculate the filter bank. In some examples, the final step of the Mel filter bank 340 may include mean normalization.

[0044] In some examples, the high-pass filter balances the frequency spectrum to compensate for high frequencies that have smaller amplitudes than lower frequencies. The high-pass filter may also perform other functions, such as to improve the signal-to-noise ratio (SNR) of the input signal 344. In other examples, this high-pass filter function may be omitted in lieu of a mean normalization step.

[0045] Dividing the signal into short frames allows the frequency contour of the input signal to be preserved over time. Performing a Fourier transform on these short-time frames can provide an approximation of the frequency contour of the signal when concatenating adjacent frames. In some examples, the Mel filter bank 340 can also apply a windowing function, such as a Hamming window, to the frames. In some examples, the frame size can be on the order of milliseconds (ms), e.g., 5-60 ms, while in other examples, the frame size can be approximately 1 second. The overlap can be approximately half the frame size in some examples, and can be larger or smaller in other examples.

[0046] Computing the filter bank may involve applying a filter, such as a triangular filter, to the power spectrum output from the Fourier transform to extract frequency bands. The Mel scale roughly mimics the nonlinear human ear's perception of sound by being more discriminative at lower frequencies and less discriminative at higher frequencies.

[0047] The network structure in the example of Figure 3 also includes a CONV256 block 324, a CONV512 block 326, and a CONV512 block 328, and the embedded representation 314 is output after processing by a fully connected (FC) layer 330. However, Figure 3 illustrates only one example of an implementation of the embedding extraction of this disclosure. In other examples, the embedding extractor may include more, fewer, or different components than shown in the arrangement. For example, the embedded representation 314 output from the FC layer 330 may comprise a 128-bit vector in some examples.

[0048] After first representing the input audio spectrogram 344 with a Mel filter bank 340, which adjusts the audio spectrum based on human perceptual hearing, the DNN component of the embedding extractor 300 can determine which feature vectors are important for isolating non-language audio events. In other words, the DNN can create features containing information useful for isolating non-language audio events, for example, in the CONV128 block 322. As described above in connection with FIG. 1, the embedding extractor 300 can be trained on a very large set of audio samples to develop world knowledge, e.g., knowledge of the non-language audio space, to determine which features stored in the representations can distinguish audio events from one another. These embedded representations 314 provided to a backend classifier, for example, the backend classifier 108 of FIG. 1, enable the backend classifier 108 to determine when audio information is different from other audio information.

[0049] Some examples of features / phenomena that can be included and used to distinguish sounds from one another may include frequency, frequency patterns over time, loudness, sharpness of the sound, e.g., a gunshot or door slam compared to a siren or whale call, sound duration, timbre, discreteness, structure, decay time, and similar features. A neural network can analyze the sound energy in each frame, e.g., 10 ms frames, and a pooling layer at each step can combine those frames to determine changes over time, such as the decay and patterns described above, over a longer time period, e.g., about 1 second.

[0050] The example of FIG. 3 includes one or more fully connected layers, the FC layer 330. The FC layer 330 can be implemented as a feed to a neural network that forms the last few layers of the network. The input to the FC layer 330 can include the output from the final pooling or convolutional layer 346, which is flattened and then fed to the FC layer 330. In some examples, the FC layer 330 can compile data extracted by previous layers to form a final output, such as an embedded representation 314. The embedded representation 314 can also be described as a nonlinear transformation of the input, e.g., the world knowledge contained in different non-speech sounds in the audio spectrogram 104 of FIG. 1, where the DNN of the embedding extractor 300 can be trained to distinguish between sound classes. The output, i.e., the embedded representation 314, can provide rich information that facilitates discrimination between different sounds in the non-speech audio space. The embedded representation 314 becomes a feature vector for a simpler back-end classifier.

[0051] In this way, the disclosed system contrasts with some models, such as support vector machine (SVM) techniques for language recognition, which can arrive at language classification starting directly from MFCC (mel-frequency cepstral coefficient) representations of received language. In language recognition, audio information is smaller in size, requiring fewer features for classification, and this world knowledge step, as used by the disclosed system, may not be used. These models are trained to detect specific features and classify specific phenomena. Instead, the disclosed system further culls world knowledge from this largely amorphous, non-linguistic audio into embedded representations that can be used by a back-end classifier. The embedding extractor is also not a classifier for the disclosed system.

[0052] An autoencoder that receives an input and performs feature identification can have an internal representation, the input features, and its output is a prediction of the input, e.g., how best to reproduce the input. Autoencoders do not use labels because they are trained without labels, but the disclosed system, in contrast to autoencoders, can be equipped with labels for classes, thereby providing a model that can determine whether an audio event is the same as or different from another event with a predetermined error rate.

[0053] The embedding extractor 300 can be trained based on a large volume of acoustic events to predict them. Once trained, the embedding extractor 300 maps any acoustic event (either the training data or a portion of the unseen data) into a high-dimensional space. However, as described above in connection with FIGS. 1 and 2, adding new classes can require significant effort. Furthermore, if most of the trained classes are not relevant to a particular task, retraining the entire model is time-consuming and resource-intensive. However, the internal representation of the audio data generalizes the model to discriminate patterns (sounds) for which the model has not yet been trained. Now, when the embedding extractor 300 provides the embedding representations 314 to the backend classifier of the present disclosure, the backend classifier can identify new classes for which the embedding extractor 300 has not been trained with a low error rate, even if the backend classifier has only been trained on a few examples. The backend classifier can compare the new sound representations with sound representations, such as the embedding representations 314 from the embedding extractor 300. In this way, the system of the present disclosure can distinguish new classes of sounds without having to retrain the embedding extractor 300.

[0054] Additionally, because the embedded representations 314 may comprise reduced-dimensional vectors whose dimensions are focused on distinguishing non-speech sounds from one another, the back-end classifier of the present disclosure may perform comparisons using reduced computational resources compared to performing comparisons with representations of greater dimensions.

[0055] 4 is a block diagram of multiple electronic systems and devices communicating with each other in a network environment in accordance with one or more techniques of this disclosure. Components of an audio classification system, such as system 100 of FIG. 1, the embedding extractor of FIG. 3, and the computing system 200 of FIG. 2, may operate in a distributed network environment.

[0056] The network environment includes a communications network 320 connecting server computing systems 304A-304B and at least one client computing system 302A-302G. As shown, there may be multiple server computing systems 304A-304B and multiple client computing systems 302A-302G connected to each other via the network 320. Examples of the network 320 may be or include one or more of an optical network, a cellular network, the Internet, a local area network (LAN), a wide area network (WAN), a satellite link, a fiber network, a cable network, or combinations thereof and / or others. The server computing systems 304A-304B may each include circuitry and software for communicating across the network 320 with the other server computing systems 304A-304B and the client computing systems 302A-302G. Server computing systems 304A-304B may each be associated with one or more databases 306A-306B and may be configured to store information, such as storage device 226 of FIG. 2. Servers 304A-304B may each have one or more instances of a virtual server running on a physical server, and multiple virtual instances may be implemented depending on the design. Firewalls may be established between client computing systems, such as 302D, and network 320 to protect data integrity on client computing system 302D. Processing circuitry on one or more servers 304A-304B may execute components of computing engine 230 and perform the functions described above in connection with FIGS. 1-3.

[0057] Cloud provider services allow application software to be installed and run in the cloud, and users can access the software services from client devices. Cloud users with sites in the cloud may not simply manage the cloud infrastructure and platform on which their applications run; that is, servers and databases may be shared hardware, with users being given dedicated use of a certain amount of these resources. A user's cloud-based site is given a virtual amount of dedicated space and bandwidth in the cloud. Cloud applications may differ from other applications in terms of scalability, which may be achieved by cloning tasks onto multiple virtual machines during runtime to accommodate changing work demands. A load balancer distributes work across the set of virtual machines. This process is transparent to cloud users, who only see a single access point.

[0058] The cloud-based remote access is coded to perform request-response cycles with both mobile device applications residing on the client devices 302A-302G and web browser applications residing on the client devices 302A-302G using protocols such as HTTP. In some situations, the cloud-based remote access for the wearable electronic device 302C may be accessed via a mobile device, desktop, or tablet device cooperating with the electronic device 302C. The cloud-based remote access between the client devices 302A, 302G and the cloud-based provider site 304A is coded to perform one or more of the following: (1) request-response cycles from all web browser-based applications; (2) SMS / twitter-based request-response message exchanges; (3) request-response cycles from a dedicated online server; (4) direct request-response cycles between native mobile applications on the client devices and the cloud-based remote access for the wearable electronic device; and (5) combinations thereof.

[0059] In one embodiment, the server computing system 304A may include a server engine, a web page management component or online service or application component, a content management component, and a database management component. The server engine performs basic processing and operational system-level tasks. The web page management component, online service, or online application component may handle the creation and display or routing of web pages or screens associated with receiving and serving digital content and digital advertisements. Users can access the server computing system through its associated URL. The content management component handles most of the functionality in the embodiments described herein. The database management component includes database-related tasks such as logging and retrieval, database queries, and data logging.

[0060] Figure 5 is a flow diagram illustrating an example of the operation of the acoustic event detection system of the present disclosure. As described above in connection with Figures 1-3, an embedding extractor of the present disclosure, for example, the embedding extractor 106 of Figure 1, may receive an input audio waveform 116 (500) via input device 102, which may be a recording, a data file, or direct input via a microphone. The embedding extractor may be trained using a large audio space that may include sounds, including speech and non-speech sounds, and may generate an embedded representation 114 (or 314) of the input audio waveform 116 (502), such as those shown in Figures 1 and 3.

[0061] The embedding extractor may output 504 a representation of the input audio waveform to a classifier, such as the back-end classifier 108. This embedded representation 114 of the input audio waveform may comprise data, for example in the form of vectors, for distinguishing particular non-speech sounds from other non-speech sounds.

[0062] The classifier may receive an embedded representation of an input audio waveform, e.g., from an embedding extractor (506), and may compare the embedded representation of the input audio waveform to classes of non-speech sounds (508). In response to this comparison, the classifier may determine a score for the input audio waveform (510). This score, e.g., scores 112 and 212 in FIGS. 1 and 2, respectively, may indicate whether the input audio waveform is the same as or different from the non-speech sound classes. The classifier may output the score for the input audio waveform (512), e.g., via a user interface or other output device, such as those described above in connection with FIG. 2. In some examples, a machine learning system of the present disclosure implementing an acoustic event detection system may further apply calibration to the scores of the audio waveforms.

[0063] In one or more examples, the functionality described above may be implemented in hardware, software, firmware, or a combination thereof. For example, various components in FIGS. 1-4, such as the embedding extractor 106, the machine learning system 232, the filter bank 340, and the convolution block in FIG. 3, may be implemented in hardware, software, firmware, or a combination thereof. If implemented in software, the functionality may be stored on or transmitted over a computer-readable storage medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium that corresponds to a tangible medium, such as a data storage medium, or a communication medium, including a medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. The computer program product may include a computer-readable medium.

[0064] The techniques described in this disclosure may be implemented, at least in part, in hardware, a combination of hardware and software, firmware, or any combination thereof. For example, various aspects of the described techniques may be implemented within one or more processors, including one or more microprocessors, DSPs, ASICs, FPGAs, or any other equivalent logic circuitry, integrated or discrete, or any combination of such components. The terms "processor" or "processing circuitry," such as with respect to processing circuitry 220, may generally refer to any of the foregoing logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit including hardware may also perform one or more of the techniques of this disclosure.

[0065] Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various techniques described in this disclosure. Additionally, any of the described units, modules, or components may be implemented together or separately as discrete but interoperable logic devices. The description of different features as modules or units is intended to emphasize different functional aspects and does not necessarily imply that such modules or units must be realized by separate hardware, firmware, or software elements. Rather, functionality associated with one or more modules or units may be performed by separate hardware, firmware, or software elements, or may be integrated within common or separate hardware, firmware, or software elements.

[0066] The techniques described in this disclosure may also be embodied or encoded in an article of manufacture that includes a computer-readable medium encoded with instructions. Instructions embedded or encoded in an article of manufacture that includes an encoded computer-readable medium can, for example, implement any of the techniques described herein in one or more programmable or other processors when the instructions contained in or encoded on the computer-readable medium are executed by the one or more processors. The computer-readable storage medium may include RAM, ROM, PROM, EPROM, EEPROM, flash memory, hard disk, compact disc ROM (CD-ROM), floppy disk, cassette, magnetic medium, optical medium, or other computer-readable medium. In some examples, an article of manufacture may include one or more computer-readable media.

[0067] In some examples, a computer-readable storage medium may include a non-transitory medium. The term "non-transitory" may imply that the storage medium is not embodied in a carrier wave or propagated signal. In particular examples, a non-transitory storage medium may store data (e.g., in RAM or cache) that may change over time.

Claims

1. 1. A system comprising: a computing engine including processing circuitry for executing a machine learning system, the machine learning system comprising: an embedding extractor having a deep neural network trained with world knowledge of an audio space containing a plurality of non-speech sounds to generate an embedded representation of an input audio waveform; A classification device, comparing the embedded representation of the input audio waveform with classes of non-speech sounds on which the classifier has been trained to determine a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the classes of non-speech sounds; adding new classes of non-speech sounds by extracting high-dimensional features from the deep neural network, projecting the high-dimensional features into a space for the classifier, and updating the classifier with the projected high-dimensional features without retraining the world knowledge of the audio space for the embedding extractor; and a classifier configured to output the score for the input audio waveform; A system comprising:

2. The system of claim 1 , wherein the embedding extractor processes the input audio waveform to generate high-dimensional features of the input audio waveform.

3. 10. The system of claim 1, wherein the machine learning system further comprises a calibration module executed by the computing engine, the calibration module being a logistic regression calibration module trained based on the sound detection scores.

4. The system of claim 1 , wherein the classifier is configured to perform classification using probabilistic linear discriminant analysis (PLDA) to determine the score.

5. A system, comprising: a computing engine including processing circuitry for executing a machine learning system, the machine learning system comprising: an embedding extractor trained with an audio space containing a plurality of non-speech sounds to generate an embedded representation of a first input audio waveform; A classification device, determining a score for the first input audio waveform indicating whether the first input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the first input audio waveform with classes of non-speech sounds on which the classifier has been trained; and a classification device configured to output the score for the first input audio waveform; The machine learning system includes: configured to receive a second input audio waveform; and configured to receive user input, said user input comprising: an indication that the second input audio waveform is an example of a new class of non-speech sounds; and and a label for the new class of non-speech sounds, the machine learning system further comprising: The second input audio waveform is processed by the embedding extraction device, and processing the second input audio waveform includes extracting a second extracted embedded representation comprising high-dimensional features from a deep neural network (DNN) and reducing the dimension of the second extracted embedded representation, and the classification device decomposing the second extracted embedded representation into an acoustic model subspace and a channel subspace; and a trained classifier configured to generate a final model of the new class including the high-dimensional features of the second extracted embedded representation projected into the acoustic model subspace without retraining the embedding extractor.

6. the classifier is configured to compare the embedded representation of the subsequent input audio waveform with the new class of non-speech sounds; and The system of claim 5 , wherein the embedding extractor is not changed as a result of adding the new class of non-speech sounds.

7. 1. A method, comprising: generating an embedded representation of an input audio waveform by an embedding extractor of a machine learning system, the embedding extractor having a deep neural network trained with world knowledge of an audio space containing a plurality of non-speech sounds, the embedded representation of the input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determining, by a classifier of the machine learning system, a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the input audio waveform with classes of non-speech sounds for which the classifier has been trained; adding new classes of non-speech sounds by extracting high-dimensional features from the deep neural network, projecting the high-dimensional features into a space for the classifier, and updating the classifier with the projected high-dimensional features without retraining the world knowledge of the audio space for the embedding extractor; outputting, by the classifier, the score for the input audio waveform; A method comprising:

8. The method of claim 7 , wherein the embedding extractor processes the input audio waveform to generate high-dimensional features of the input audio waveform.

9. 8. The method of claim 7, further comprising calibrating the scores output by the classifier by a calibration module of the machine learning system, the calibration module being a logistic regression calibration module trained based on sound detection scores.

10. The method of claim 7 , wherein determining the score includes performing class classification using probabilistic linear discriminant analysis (PLDA).

11. A method, comprising: generating an embedded representation of a first input audio waveform by an embedding extractor of a machine learning system, the embedding extractor being trained using an audio space containing a plurality of non-speech sounds, the embedded representation of the first input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determining, by a classifier of the machine learning system, a score for the first input audio waveform indicating whether the first input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the first input audio waveform with classes of non-speech sounds for which the classifier has been trained; outputting, by the classifier, the score for the first input audio waveform; receiving, by the machine learning system, a second input audio waveform; receiving, by the machine learning system, a user input, the user input comprising: an indication that the second input audio waveform is an example of a new class of non-speech sounds; and receiving user input including a label for said new class of non-speech sounds; processing the second input audio waveform by the machine learning system, wherein processing the second input audio waveform includes extracting a second extracted embedded representation comprising high-dimensional features from a deep neural network (DNN) and reducing the dimensionality of the second extracted embedded representation; decomposing the second extracted embedded representation into an acoustic model subspace and a channel subspace by the machine learning system; generating, by the machine learning system, a final model of the new class that includes the high-dimensional features of the second extracted embedded representation projected into the acoustic model subspace without retraining the embedding extractor.

12. The method further includes comparing, by the classifier, the embedded representation of the subsequent input audio waveform with the new classes of non-speech sounds; The method of claim 11 , wherein the embedding extractor is not changed as a result of adding the new class of non-speech sounds.

13. A non-transitory computer-readable storage medium containing instructions that, when executed, cause a processing circuit to: generating an embedded representation of an input audio waveform by an embedding extractor of a machine learning system having a deep neural network, the embedding extractor being trained with world knowledge of an audio space containing a plurality of non-speech sounds, the embedded representation of the input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determining, by a classifier of the machine learning system, a score for the input audio waveform indicating whether the input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the input audio waveform with classes of non-speech sounds for which the classifier has been trained; adding new classes of non-speech sounds by extracting high-dimensional features from the deep neural network, projecting the high-dimensional features into a space for the classifier, and updating the classifier with the projected high-dimensional features without retraining the world knowledge of the audio space for the embedding extractor; outputting, by the classifier, the score for the input audio waveform; A non-transitory computer-readable recording medium that causes the

14. The non-transitory computer-readable storage medium of claim 13 , wherein the embedding extractor processes the input audio waveform to generate high-dimensional features of the input audio waveform.

15. 14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further cause the processing circuit to calibrate the scores output by the classifier with a calibration module of the machine learning system, the calibration module being a logistic regression calibration module trained based on sound detection scores.

16. The non-transitory computer-readable storage medium of claim 13 , wherein determining the score includes performing class classification using probabilistic linear discriminant analysis (PLDA).

17. A non-transitory computer-readable storage medium containing instructions that, when executed, cause a processing circuit to: generating an embedded representation of a first input audio waveform by an embedding extractor of a machine learning system, the embedding extractor being trained using an audio space containing a plurality of non-speech sounds, the embedded representation of the first input audio waveform including data for distinguishing a particular non-speech sound from the plurality of non-speech sounds; determining, by a classifier of the machine learning system, a score for the first input audio waveform indicating whether the first input audio waveform is the same as or different from the class of non-speech sounds by comparing the embedded representation of the first input audio waveform with classes of non-speech sounds for which the classifier has been trained; outputting, by the classifier, the score for the first input audio waveform; receiving a second input audio waveform; receiving a user input, the user input comprising: an indication that the second input audio waveform is an example of a new class of non-speech sounds; and receiving user input including a label for said new class of non-speech sounds; processing the second input audio waveform, wherein processing the second input audio waveform includes extracting a second extracted embedded representation comprising high-dimensional features from a deep neural network (DNN) and reducing the dimensionality of the second extracted embedded representation; decomposing the second extracted embedded representation into an acoustic model subspace and a channel subspace; generating a final model of the new class that includes the high-dimensional features of the second extracted embedded representation projected into the acoustic model subspace without retraining the embedding extractor; A non-transitory computer-readable storage medium that further causes the computer to perform the steps of:

18. 20. The non-transitory computer-readable storage medium of claim 17, wherein the classifier is a trained PLDA classifier.

Images