Method and apparatus for normalizing features extracted from audio data for signal recognition or modification

Abstract

A feature vector can be extracted from each frame of input digitized microphone audio data. The feature vector can include a power value for each of a plurality of frequency bands. A feature history data structure including a plurality of feature vectors can be formed. A normalized feature set including a normalized feature data structure can be generated by determining normalized power values for the plurality of frequency bands of each feature vector of the feature history data structure. A signal recognition or modification process can be based at least in part on the normalized feature data structure.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 880,167, filed July 30, 2019, U.S. Provisional Patent Application No. 62 / 950,013, filed December 18, 2019, and European Patent Application No. 19217579.2, filed December 18, 2019, each of which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to systems and methods for audio signal recognition or modification, including but not limited to speech recognition. BACKGROUND

[0004] Devices capable of signal recognition or modification have a wide range of applicability, including but not limited to wake-word detection and other speech recognition processes, speaker recognition processes, language recognition processes, emotion recognition processes, non-speech sound classification processes, and noise suppression processes. Although existing systems and methods for implementing audio signal recognition or modification provide benefits, improved systems and methods would be desirable.

[0005] Symbols and Terminology

[0006] The expression “smart audio device” is used herein to mean a smart device that is a single-purpose audio device or a virtual assistant (e.g., a connected virtual assistant). A single-purpose audio device is a device that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one loudspeaker) and is largely or primarily designed to implement a single purpose (e.g., a television (TV) or a mobile phone). Although a TV can typically play (and is considered capable of playing) audio from program material, in most cases a modern TV runs some sort of operating system on which applications (including the application of watching TV) run locally. Similarly, audio input and output in a mobile phone can do many things, but these are served by applications running on the phone. In this sense, a single-purpose audio device with loudspeaker(s) and microphone(s) is typically configured to run local applications and / or services to directly use the loudspeaker(s) and microphone(s). Some single-purpose audio devices can be configured to combine together to implement playing audio over a zone or user-configured area.

[0007] In this document, a "virtual assistant" (e.g., a connected virtual assistant) is a device (e.g., a smart speaker, a smart display, or a voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one loudspeaker), and that can provide the ability to use multiple devices (different from the virtual assistant) to enable, in some sense, a cloud or an application that is not implemented in or on the virtual assistant itself. Virtual assistants can sometimes work together, e.g., in a very discrete and conditionally defined manner. For example, two or more virtual assistants can work together in the sense that one of them (i.e., the virtual assistant that is most confident that it has heard a wake word) responds to the word. Connected devices can form a constellation that can be managed by one master application, which can be (or include or implement) a virtual assistant.

[0008] In this document, a "wake word" is used in a broad sense to mean any sound (e.g., a word spoken by a human or other sound) for which a smart audio device is configured to wake up in response to detecting ( "hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "waking up" means that the device enters a state of waiting (i.e., is listening) for a sound command.

[0009] In this document, the expression "wake word detector" means a device (or software that includes instructions for configuring a device) that is configured to continuously search for an alignment between real-time sound (e.g., speech) features and a trained model. Typically, a wake word event is triggered whenever the wake word detector determines that the probability of having detected a wake word exceeds a predefined threshold. For example, the threshold can be a predetermined threshold that is adjusted to give a reasonable tradeoff between false acceptance rate and false rejection rate. After a wake word event, the device can enter a state (which can be referred to as a "woken up" state or an "attention" state) in which the device listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0010] Throughout this disclosure, including in the claims, "speaker" and "loudspeaker" are used synonymously to mean any sound-emitting transducer (or set of transducers) that is driven by a single speaker feed. A typical earphone includes two speakers. A speaker can be implemented to include multiple transducers (e.g., a woofer and a tweeter) that are all driven by a single common speaker feed. In some cases, the speaker feed can undergo different processing in different circuit branches that are coupled to different transducers.

[0011] Throughout this disclosure, including in the claims, the expression "operating on" a signal or data (e.g., filtering, scaling, transforming, or applying gain to a signal or data) is used in a broad sense to mean either directly operating on the signal or data or operating on a processed version of the signal or data (e.g., a version of the signal that has been preliminarily filtered or pre-processed prior to operating on it).

[0012] Throughout this disclosure, including in the claims, the term "system" is used in a broad sense to mean a device, system, or subsystem. For example, a subsystem that implements a decoder can be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to a plurality of inputs, where the subsystem generates M of the inputs, while the other X-M inputs are received from an external source) can also be referred to as a decoder system.

[0013] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to mean a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, video, or other image data). Examples of processors include a field-programmable gate array (or other configurable integrated circuit or chipset), a digital signal processor programmed and / or otherwise configured to perform pipelined processing on audio or other sound data, a programmable general-purpose processor or computer, and a programmable microprocessor chip or chipset. SUMMARY

[0014] At least some aspects of the present disclosure can be implemented via one or more methods. Some such methods can optionally involve receiving microphone audio data from a microphone system, and digitizing the microphone audio data to produce digitized microphone audio data. Some methods can optionally involve framing the digitized microphone audio data to produce frames of the digitized microphone audio data, and extracting a feature vector from each frame of the digitized microphone audio data. The feature vector can be or can include a power value for each of a plurality of frequency bands. In some examples, the power value can be a log of the power. Some such methods can involve forming a feature history data structure. The feature history data structure can include a plurality of feature vectors.

[0015] Some such methods can involve producing a set of normalized features including a normalized feature data structure by determining normalized power values for the plurality of frequency bands of each feature vector of the feature history data structure. Some such methods can involve performing a signal recognition or modification process based at least in part on the normalized feature data structure.

[0016] In some such methods, the set of normalized features can optionally include a set of cepstral coefficients that describe an overall average spectral shape of the normalized feature data structure.

[0017] Some implementations can involve determining an average spectral representation based on the current frame and at least one previous frame, including computing, for each of a plurality of frequency bands, an average or a weighted average of a power value of a respective frequency band of the current frame and a power value of the respective frequency band of the at least one previous frame, and determining the set of cepstral coefficients based on the average spectral representation.

[0018] Some implementations according to the first aspect can involve determining an average wideband level representing an average over all frequency bands of the average spectral representation, determining an average level independent spectral representation by subtracting the average wideband level from the average spectral representation, determining a cepstral decomposition of the average level independent spectral representation, and determining the set of cepstral coefficients based on the cepstral decomposition. In some examples, generating the normalized feature data structure includes smoothing the average level independent spectrum and subtracting the smoothed average level independent spectrum and the average wideband level from each feature vector of the feature history data structure. Optionally, smoothing the average level independent spectrum includes performing a spectral re-synthesis on the cepstral decomposition.

[0019] In some implementations according to the second aspect, the method further includes determining a cepstral decomposition of the average spectral representation or a smoothed version thereof, and determining the set of cepstral coefficients based on the cepstral decomposition. In some examples, generating the normalized feature data structure includes subtracting the average level independent spectrum or a smoothed version thereof from each feature vector of the feature history data structure.

[0020] In some implementations according to the first or second aspect, the cepstral decomposition includes cepstra of individual digits, and the set of cepstral coefficients corresponds at least to a subset of the cepstra of the cepstral decomposition, optionally to a subset excluding the first cepstra.

[0021] In some examples, based on the current frame and a most recent previous frame, the average spectral representation is determined as an infinite impulse response (IIR) average. Alternatively, based on all frames of the feature history data structure, the average spectral representation is determined as a finite impulse response (FIR) average by computing, for each of a plurality of frequency bands, an average of power values of the respective frequency band of all frames of the feature history data structure.

[0022] The signal recognition or modification process can be or can include a speech recognition process. In some cases, the speech recognition process can be or can include a wake-word recognition process. According to some examples, performing the signal recognition or modification process can involve providing the normalized feature data structure to an acoustic model and determining a phoneme probability estimate based on an output of the acoustic model. In some cases, the acoustic model can be implemented via a neural network. Alternatively or additionally, the signal recognition or modification process can involve a speaker recognition process, a language recognition process, an emotion recognition process, a non-speech sound classification process, and / or a noise suppression process.

[0023] According to some examples, the frequency bands can have equal or substantially equal widths in a frequency space. For example, the frequency space can be a mel-frequency space, a Bark scale, or an equivalent rectangular bandwidth scale.

[0024] In some examples, generating the normalized feature data structure can involve determining an average wideband level based on the current frame and the most recent previous frame. For example, generating the normalized feature data structure can involve determining an average spectral representation based on the current frame and the most recent previous frame. According to some implementations, generating the normalized feature data structure can involve determining an average wideband level based on all frames of the feature history data structure. In some cases, generating the normalized feature data structure can involve determining an average spectral representation based on all frames of the feature history data structure.

[0025] In some implementations, generating the normalized feature data structure can involve determining a confidence weighting factor for each feature. Determining the confidence weighting factor can be based at least in part on a noise estimate, a signal-to-noise estimate, an echo prediction, a signal-to-echo prediction, a sound direction estimate, and / or a reverberation estimate.

[0026] According to some examples, generating the normalized feature set can involve obtaining an average level independent spectral representation. In some examples, generating the normalized feature set can involve performing a cepstral decomposition of the average level independent spectral representation. In some cases, performing the cepstral decomposition can involve applying a decorrelation transform represented by a basis matrix. For example, the basis matrix can represent a discrete sine transform basis, a discrete cosine transform basis, and / or a Karhunen-Loève transform basis.

[0027] Some methods can involve performing a subsequent or parallel task based at least in part on the signal recognition or modification process. For example, the subsequent or parallel task can involve controlling a loudspeaker to produce a responsive sound, performing an internet search, initiating a telephone call, sending a text message, sending an email message, and / or changing a setting of a device.

[0028] Some or all of the operations, functions, and / or methods described herein can be performed by one or more means according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure can be implemented in non-transitory media on which software is stored.

[0029] For example, the software may include instructions for controlling one or more devices to perform the methods described above.

[0030] At least some aspects of this disclosure can be implemented via apparatus. For example, one or more apparatuses may be able to perform at least partially the methods disclosed herein. In some embodiments, the apparatus may include an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof. According to some examples, the control system may be configured to perform the methods as described above.

[0031] Details of one or more embodiments of the subject matter described herein are set forth in the following figures and description. Other features, aspects, and advantages will become apparent from this specification, the figures, and the claims. Note that the relative dimensions in the following figures may not be drawn to scale. Attached Figure Description

[0032] Figure 1 This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0033] Figure 2 It outlines what can be achieved by, for example Figure 1 The flowchart shows an example of a method performed by the apparatus or other devices shown.

[0034] Figure 3A A framework of a system that can be used to implement some aspects of this disclosure is shown.

[0035] Figure 3B It outlines what can be achieved by, for example Figure 1 The flowchart shows an example of a method performed by the apparatus or other devices shown.

[0036] Figure 4 The following is shown for implementation Figure 3A An example of the architecture of an acoustic model.

[0037] In the various drawings, like reference numbers and designations indicate similar elements. DETAILED DESCRIPTION

[0038] Many embodiments of the present disclosure are technically possible. It will be apparent to those of ordinary skill in the art from the present disclosure how to implement them. Some embodiments of systems and methods are described herein.

[0039] Speech analysis systems include wake-word detectors, speech recognizers, speaker recognizers, sentiment recognizers, and the like. Speech analysis systems are often required to work under a wide variety of acoustic conditions and with a wide variety of microphone hardware, neither of which can often be fully characterized when a speech analysis system is designed or trained. Many feature extraction techniques and feature normalization schemes have been proposed over the years to deal with these issues, such as Mel-Frequency Cepstral Coefficient features (MFCC), fMLLR (Feature Space Maximum Likelihood Regression), CMS (Cepstral Mean Subtraction), and CMVN (Cepstral Mean and Variance Normalization).

[0040] While such approaches have some advantages, none has proven to be completely satisfactory. For example, normalization and regression techniques must collect statistics over a period of time before they can operate effectively. In the case of speech recognition, these techniques often require tens of seconds or tens of minutes of speech before they take effect. In many speech analysis use cases, there is no guarantee that new speech will be spoken under the same acoustic conditions as past speech. For example, in the case of far-field wake-word detection, it can be expected that a previously unheard user can enter a room, stand in a new position not occupied by any previous speaker, and the first word spoken from his or her mouth can be the wake-word. Nonetheless, it is expected that the device will detect the wake-word with a low false trigger rate in this case. Furthermore, if the device that detects the wake-word is a “smart speaker” located in a residential kitchen, it can be expected that the user will move the device frequently during food preparation activities, which can produce a wide range of associated sounds over time as well as different levels and spectra of reception of the user’s speech. This means that it is not possible to speak two wake-word utterances under the same acoustic conditions.

[0041] Furthermore, unweighted normalization techniques are often sensitive to background noise, such as the sounds associated with food preparation discussed above. The MFCC approach often involves reducing the first cepstral coefficient (often denoted as CO), which has the effect of removing an unweighted average of the data in the Mel-frequency space from the feature set. If the estimate of the cepstral mean is heavily influenced by background noise, then subtracting this cepstral mean will distort all of the cepstral coefficients, including those that best represent the desired speech signal.

[0042] For many years, speech technology systems have worked based on the assumption that the features representing speech sounds can be modeled as Gaussians or Gaussian mixtures. This has limited system design to using feature sets with limited covariance between features. Often, the assumption of statistical independence between features (diagonal covariance) is made to make Gaussian mixture modeling computationally tractable. Typically, MFCCs, as well as deltas and delta-deltas from one time step to the next are chosen as the feature set, meaning that the classification of what the user is saying at any time depends only on three time analysis frames (typically a 30 ms total time history). For example, since typical syllable durations in human speech are on the order of 100 ms, this often results in a poor estimate of which phoneme is being spoken in a speech recognition task.

[0043] In contrast, most modern speech technology systems rely on neural network based speech models. Such models typically have a greater ability to handle static, correlated (covarying) features than previous time Gaussian or Gaussian mixture models. These systems typically "stack" several feature analysis frames, and feed them into the network in each frame. This stacking approach can result in a more robust estimate of which phoneme is being spoken (or whether a wake word has been spoken, which speaker is speaking, etc.). A typical case in modern speech recognizers is to compute a number (e.g., 23) of "logmelspec" values (i.e., the power log of a number of spectral frequency bands distributed at equal intervals in mel frequency) per 10 ms, and the most recent 10-20 frames of such features are "stacked" into an extended 230-460 dimensional feature vector and presented to the network.

[0044] Some disclosed implementations provide novel approaches to normalization of spectral features that are "stacked" over multiple analysis frames for input to a neural network speech technology model. Although in some disclosed examples the spectral features are "logmelspec" features, the disclosed approaches are also applicable if the frequency bands in which power is computed are not organized on a mel scale but instead are organized in a different way (e.g., bands at logarithmic intervals in frequency, bands organized according to a Bark scale, etc.). Some disclosed normalization approaches can be able to overcome at least some of the difficulties discussed above.

[0045] Some such normalization approaches can involve normalization only between features in the stack (e.g., over a 100-200 ms time window). Such approaches can help to accurately recognize speech or detect a wake word even when a user starts speaking suddenly under acoustic conditions that have never been heard before.

[0046] Some methods can involve separating macro-scale spectral shape from micro-scale detail. Such methods can help create systems that are robust to, for example, microphones with different equalization curves or room impulse responses (RIRs) with different spectral shapes. Systems implementing such methods can have relatively greater robustness to background noise than previous methods.

[0047] Some methods can accommodate weighting in features, such as deterministic-based weighting. Systems implementing such methods can be configured to obtain additional inputs from pre-processing systems such as echo suppressors, noise suppressors, and / or nonlinear beamforming systems, which in some cases can greatly enhance robustness.

[0048] Figure 1 is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure. According to some examples, the apparatus 110 can be or can include a smart audio device configured for performing at least some of the methods disclosed herein. For example, the smart audio device can be a virtual assistant that includes a wake-word detector. In other implementations, the apparatus 110 can be or can include another device configured for performing at least some of the methods disclosed herein, such as a laptop or desktop computer, a mobile device such as a cellular phone or tablet computer, etc. In some such implementations, the apparatus 110 can be or can include a server.

[0049] In this example, the apparatus 110 includes an interface system 115 and a control system 120. In some implementations, the interface system 115 can be configured for receiving input from a microphone system including one or more microphones.

[0050] The interface system 115 can include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some implementations, the interface system 115 can include one or more wireless interfaces. The interface system 115 can include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 115 can include one or more interfaces between the control system 120 and a memory system, such as the optional memory system 125 shown in Figure 1 However, in some cases, the control system 120 can include a memory system.

[0051] Control system 120 may include, for example, a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components. In some embodiments, control system 120 may reside in more than one device. For example, a portion of control system 120 may reside in device 110, and another portion of control system 120 may reside in another device, such as a server, a smart audio device, a mobile device (e.g., a smartphone or tablet computer), etc. In some such examples, interface system 115 may also reside in more than one device. Therefore, in some embodiments, device 100 may be one of multiple devices for implementing one or more cloud-based services for implementing aspects of this disclosure, including but not limited to cloud-based speech recognition services.

[0052] In some embodiments, the control system 120 may be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 120 may be configured to perform signal recognition or modification processes, such as one or more of those processes disclosed herein. In some such examples, the control system 120 may be configured to perform a speech recognition process. For example, the speech recognition process may involve a wake-word recognition process.

[0053] Some or all of the methods described herein can be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media can include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media can, for example, be located on... Figure 1 The optional memory system 125 and / or control system 120 shown are included. Therefore, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media on which software is stored. For example, the software may include instructions for controlling at least one device to process audio data. For example, the software may be provided by, for example, Figure 1 The control system 120 and other control system components are used to perform the operation.

[0054] Figure 2 It outlines what can be achieved by, for example Figure 1 The flowchart illustrates an example of a method performed by the apparatus shown. As with other methods described herein, the blocks of method 200 need not be performed in the indicated order. Furthermore, this method may include more or fewer blocks than those shown and / or described.

[0055] In this example, block 205 involves receiving microphone audio data from a microphone system. According to some such examples, all of the microphones of the microphone system are part of the apparatus (e.g., apparatus 110) that performs method 200. However, in some implementations, the apparatus that performs method 200 can not include at least a portion of the microphone system. In some such examples, the microphone system can include one or more microphones located at one or more other locations of the environment (e.g., one or more other devices of the environment).

[0056] According to this example, block 205 involves receiving analog microphone audio data, and block 210 involves digitizing the microphone audio data to produce digitized microphone audio data. However, in alternative implementations, method 200 can involve receiving digitized microphone audio data as the initial block of the process.

[0057] Similarly, in this example, block 215 involves framing the digitized microphone audio data to produce frames of digitized microphone audio data. According to some implementations, the step size can be 10 ms. In such examples, a new frame of logmelspec features is computed every 10 ms. Other implementations can use longer or shorter step sizes. For example, some implementations can have a step size of 5 ms, 15 ms, 20 ms, 25 ms, 30 ms, etc. To compute logmelspec features using a step size of 10 ms, in some examples, 20 ms of input microphone audio data can be used, with 50% overlap in the transform. Alternative implementations can use longer or shorter overlap. For example, in some implementations, 30 ms, 40 ms, 50 ms, 60 ms, 70 ms, 80 ms, 90 ms, or 100 ms of input microphone audio data can be used to compute each 10 ms frame. Some systems can choose to use a step size that calculates a convenient power of two count of input microphone audio samples, as this can result in an efficient transform implementation using an FFT. For example, some implementations can use 256 input samples at Fs = 16 kHz (16 ms), or 128 input samples (8 ms). Some alternative implementations can use 256 input samples or 512 input samples at Fs = 32 kHz. However, in alternative implementations, method 200 can involve receiving frames of digitized microphone audio data as the initial block of the process.

[0058] According to this example, block 220 involves extracting a feature vector from each frame of the digitized microphone audio data. In this example, the feature vector includes a power value for each of a plurality of frequency bands. Thus, in some examples, block 220 (or a previous block, such as block 215) can involve transforming the digitized microphone audio data from the time domain to a plurality of frequency bands in the frequency domain. In some examples, the feature vector can include 15, 20, 25, 30, 40, 45, or 50 values.

[0059] In some examples, the frequency bands can have equal or substantially equal widths in the frequency space. For example, the frequency space can be a mel-frequency space, a bark scale, or an equivalent rectangular bandwidth scale. For example, the power values can be logarithms of power. According to some examples, the logarithmic representation is in decibels.

[0060] In this implementation, block 225 involves forming a feature history data structure. Block 225 can involve a process of "stacking" the feature vectors. In this example, the feature history data structure includes a plurality of feature vectors. In some such examples, there can be 5, 10, 15, 20, 25, 30, or more feature vectors. According to some implementations, the feature history data structure can include feature vectors having a total duration of 100 ms to 200 ms. For example, some such implementations with a step size of 10 ms can include between 10 and 20 feature vectors in the feature history data structure. Some such implementations with a step size of 20 ms can include between 5 and 10 feature vectors in the feature history data structure. Creating a feature history data structure that includes feature vectors having a total duration of 100 ms to 200 ms can be advantageous for speech-related tasks, at least in part because the average syllable rate of speech is about one syllable per 100 ms. Thus, a total duration of 100 ms to 200 ms can be a good choice for implementations involving recognizing phonemes or syllables. Longer or shorter total durations can be suitable for implementations involving recognizing non-speech sounds, recognizing entire words, and the like.

[0061] According to this implementation, block 230 involves producing a normalized feature set that includes a normalized feature data structure. In this example, block 230 involves determining normalized power values for the plurality of frequency bands of each feature vector of the feature history data structure.

[0062] According to some examples, box 230 may relate to obtaining an average-level independent spectral representation or an average spectral representation. In some such examples, box 230 may relate to performing cepstral decomposition of the average-level independent spectral representation or the average spectral representation. According to some such examples, the normalized feature set may include a set of cepstral coefficients corresponding to the overall average spectral shape of the normalized feature data structure. For example, the coefficient set may be at least partially based on cepstral decomposition.

[0063] exist Figure 2 In the example shown, box 235 relates to performing a signal recognition or modification process based at least in part on the normalized feature data structure generated in box 230. In some cases, the signal recognition or modification process may involve a speech recognition process. In some such examples, the speech recognition process may be a wake-word recognition process.

[0064] According to some implementations, performing a signal recognition or modification process may involve providing a normalized feature data structure to an acoustic model and determining a phoneme probability estimate based on the output of the acoustic model. In some examples, the acoustic model may be implemented via a neural network.

[0065] Alternatively or additionally, performing a signal recognition or modification process may involve processes other than speech recognition. For example, performing a signal recognition or modification process may involve speaker recognition, speech recognition, emotion recognition, non-verbal sound classification, and / or noise suppression.

[0066] In some implementations, method 200 may involve performing at least one follow-up task, which is at least partially based on a signal identification or modification process. According to some such implementations, the follow-up task may involve controlling a speaker to generate a response sound, performing an Internet search, initiating a telephone call, sending a text message, sending an email message, and / or changing device settings (e.g., television settings, speaker settings, thermostat settings, home security system settings, home appliance settings, heating or air conditioning settings, etc.).

[0067] Figure 3A A framework of a system that can be used to implement some aspects of this disclosure is shown. For example, Figure 3A Boxes 303, 304, 305, 307, 309, and 311 can be controlled via a control system (as referenced above). Figure 1 The control system 120 described herein is used for implementation.

[0068] In this example, speaker 301 is emitting utterance 313, which is picked up by microphone system 302. The speaker may sometimes be referred to herein as a "speaker". Depending on the specific implementation, microphone system 302 may include a single microphone or an array of two or more microphones.

[0069] In this example, the microphone system 302 provides analog microphone audio data to an analog-to-digital converter 303. This is an example of block 205. Figure 2 The analog-to-digital converter 303 is configured to digitize the analog microphone audio data to produce digitized microphone audio data. This is an example of block 210. Figure 2

[0070] According to this example, the digitized microphone audio data is provided to a digital pre-processing unit 304. The analog-to-digital converter 303 or the digital pre-processing unit 304 can be configured to frame the digitized microphone audio data to produce frames of digitized microphone audio data (block 215). The digital pre-processing unit 304 can be configured to provide one or more types of audio pre-processing such as echo suppression, noise suppression, beamforming, etc.

[0071] In this example, a feature extraction block 305 is configured to extract a feature vector from each frame of digitized microphone audio data that is output from the digital pre-processing unit 304. This is an example of block 220. The feature vector can be or can include a power value for each of a plurality of frequency bands. In some examples, the feature extraction block 305 can be configured to convert pulse code modulation (PCM) data to "logmelspec" data. For example, the feature extraction block 305 can be configured to compute a log of power in each of a plurality of frequency bands of equal width in a mel frequency space at a time interval. In some examples, the time interval can be 6 ms, 8 ms, 10 ms, 12 ms, 14 ms, etc. According to some examples, the number of frequency bands can be 8, 10, 12, 16, 18, 20, 22, 24, 25, 26, 28, 30, 40, 45, 50, etc. Figure 2 Block 306 depicts a feature vector produced by the feature extraction block 305 from a frame of digitized microphone audio data that is output from the digital pre-processing unit 304. In one example, the feature vector 306 includes 25 real numbers. In alternative examples, the feature vector 306 can include more or fewer real numbers, e.g., 15, 20, 30, 35, etc.

[0072] According to this implementation, the stacking unit 307 has a history buffer and is configured to store or "stack" a plurality of feature vectors into a 2-dimensional array 308 of features over time and frequency. The array 308 is an example of block 225.

[0073] Figure 2 ​​an example of a feature history data structure formed in block 225. In one non-limiting example, the feature extraction block 305 can compute a new 25-band feature vector 306 every 10 ms. In one such example, the stacking unit 307 holds the most recent 10 feature vectors 306, so that the output of the stacking unit 307 is a 25 (in frequency) by 10 (in time) array 308 of the feature history.

[0074] In this example, the array 308 is provided to a normalization block 309. In some examples, the normalization block 309 can be configured to produce a normalized feature set 310 by determining normalized power values for the multiple frequency bands of each feature vector 306 of the array 308. Some detailed examples of the normalization block 309 are described below. In this example, the normalized feature set 310 includes a two-dimensional array (or "stack") 310A of normalized features. In some implementations, this array 310A will be the same size as the array 308. According to this example, the normalized feature set 310 also includes a cepstral feature 310B that describes the overall average spectral shape of the array 310A. In this example, the cepstral feature 310B includes only 2 real numbers. In various examples, the number of cepstral features 310B is much smaller than the number of values of the array 310A.

[0075] In this example, the normalized feature set 310 is provided to an acoustic model 311, which in some cases can be implemented via a neural network such as a deep neural network (DNN), a convolutional neural network (CNN), or a recurrent neural network (RNN). In some examples, the acoustic model 311 can be configured for speech recognition. In some such examples, the acoustic model 311 will have one input for each real-valued of the normalized feature set 310, and one output for each phoneme or group of phonemes probability that the acoustic model models. Figure 3A An example of phoneme probability estimates 312 produced by the acoustic model 311 is shown.

[0076] Figure 3B is a flowchart outlining one example of a method that can be performed by an apparatus such as the apparatus shown in Figure 1 is a flowchart outlining one example of a method that can be performed by an apparatus such as the apparatus shown in

[0077] In this example, blocks 325 through 355 of the method 320 are examples of blocks that can be performed by the normalization block 309 of Figure 3A According to this example, block 325 involves computing the average spectrum over all frames of the input stack. For example, the input stack can be the array 308. Figure 3Aof the array 308. According to some examples, block 325 can involve computing the average spectrum as follows:

[0078] (Equation 200)

[0079] In Equation 200, x[f,t] represents the un-normalized input features of the array 308 at a particular time t in the past (within the range [1, T], where index 1 corresponds to the most recent data) and a particular frequency band f (within the range [1, F]). In Equation 200, represents the average spectrum averaged over all frames [1, T].

[0080] According to this example, block 330 involves computing the average level over all frames in the input stack. In this example, block 330 involves, for example, obtaining the average of the spectrum over all frequencies as follows:

[0081] (Equation 201)

[0082] In Equation 201, L represents the average wideband level. However, L can be computed in other ways. In some implementations, L can be computed directly from x by substituting Equation 200 into Equation 201 and forming a double summation. For example, some implementations involving the use of MATLAB or Julia can set L equal to the average of x.

[0083] In this example, block 335 involves computing the average level-independent spectrum. According to some examples, block 335 can involve computing the average level-independent spectrum as follows:

[0084] (Equation 202)

[0085] In Equation 202, represents the average level-independent spectrum averaged over all frames [1, T].

[0086] In Figure 3B the example shown in FIG. 3B, block 340 involves computing a smoothed cepstral approximation of the average level-independent spectrum. In some examples, block 340 involves computing a smoothed cepstral approximation for a small number of cepstra, “dnab”. Note that the term cepstral is used herein to mean a discrete cosine transform of logmelspec data. To derive the term cepstral corresponding to the term spectrum, one typically reverses the characters in a portion of the word. For example, filtering implemented in the cepstral domain is often referred to as “liftering”. Thus, we refer to the cepstral equivalent of a spectral bin as “dnab” herein.

[0087] The cepstral components other than those corresponding to the flat basis functions (typically, this means excluding the first cepstral component) can be considered as a set of cepstral output components (e.g., Figure 3A The cepstral feature 310B), the set of cepstral output components, is a format that allows the speech recognizer to summarize the general spectral shape of the audio data in the stack.

[0088] Based on some examples, box 340 may involve calculating a smoothed cepstral approximation as follows:

[0089] (Equation 203)

[0090] In equation 203, c[b] represents Cepstral decomposition. In some examples, this can be achieved by obtaining [the cepstral DNAb]. Obtaining the DCT relative to the truncated discrete cosine transform (DCT) basis matrix C[b,f] The cepstral decomposition, b = [1...B]. However, other implementations can apply different solution correlation transformations, such as... The Karhunen-Loève transform (KLT) or discrete sine transform (DCT) can be used. Therefore, any suitable basis matrix can be involved. For implementations where the statistics of the specific audio data are known, KLT can be an advantageous decorrelation transform, while for implementations where the statistics of the specific audio data are unknown, DCT can be an advantageous decorrelation transform. In some implementations, B is a single-digit integer, such as 2, 3, 4, 5, etc. In this example, c[b] for b>1 (i.e., excluding the first cepstral dnab) is... Figure 3A An example of the cepstral feature 310B. For example, the DCT basis matrix C[b,f] can be calculated according to the following equations 206 and 207.

[0091] exist Figure 3B In the example shown, box 345 involves transforming the smoothed cepstral approximation of box 340 back to a smoothed spectrum, for example, as follows:

[0092] (Equation 204)

[0093] In Equation 204, s[f] denotes the spectral resynthesis of c[b] obtained by acquiring the IDCT of c relative to the truncated inverse discrete cosine transform (IDCT) basis matrix S[f,b]. In Equation 204, S[f,b] denotes the IDCT basis that can be computed by acquiring the Moore-Penrose pseudoinverse of C[b,f].

[0094] Instead of using cepstral analysis and spectral resynthesis to smooth Different smoothing operations can be applied to to obtain a smoothed average spectrum s[f].

[0095] In Figure 3B examples, block 350 involves computing normalized feature data structures, e.g., by removing the smoothed average spectrum from the input:

[0096] (Equation 205)

[0097] In Equation 205, y[f, t] represents normalized output features that are an array 310A of Figure 3A Examples of DCT basis matrix C[b, f] can be computed according to Equations 206 and 207:

[0098] (Equation 206)

[0099] (Equation 207)

[0100] Examples of Equations 206 and 207 involve using DCT-I bases. As noted above, other examples can involve using another type of basis, such as a different type of DCT basis, a DST basis, or a KLT basis. The scale factor introduced in Equation 207 is one convenient way of making the cepstral decorrelation dnabhave approximately the same scale as the input band energies.

[0101] Some alternative implementations involve supplying an efficient DCT function to implement Equation 203 and avoid computing or storing a library of matrices C. Similarly, in some alternative examples, Equation 204 can be replaced by a call to an IDCT function. However, in some such implementations, these types of libraries would compute as many cepstra dnabas there are spectral bands, since these types of libraries can assume B = F. It can be relatively more desirable to retain a relatively small number of cepstra dnab. In other words, Equation 204 produces a smoothed spectrum because .

[0102] Alternatively, normalized feature data structures are computed by removing the average spectrum from the input:

[0103] (Equation 205a)

[0104] In this implementation, the cepstral decomposition of can be determined (rather than the cepstral decomposition of ):

[0105] (Equation 203a),

[0106] to obtain a set of cepstral coefficients for use in a signal recognition or modification process c[b] .

[0107] Optionally, in such embodiments, the may be smoothed by a suitable smoothing operation so that equation 205a includes a smoothed version of x[f,t] subtracted from the normalized feature history data structure and equation 203a includes determining a cepstral decomposition of the smoothed version of .

[0108] In further alternatives, the normalized feature data structure is computed according to equation 205a and the set of cepstral coefficients is computed according to equation 203. In further alternatives, the normalized feature data structure is computed according to equation 205 and the set of cepstral coefficients is computed according to equation 203a.

[0109] According to some alternative embodiments, if a confidence weighting w[f,t] is available for each input feature x[f,t], then both the average over time ) and the average over time and frequency (L) can be considered weighted averages. Such embodiments can potentially provide additional robustness under noisy conditions.

[0110] Depending on the particular embodiment, the confidence weighting can be based on a number of different factors. According to some embodiments, the confidence weighting can be based at least in part on a stationary noise estimate. For example, the stationary noise estimate can be obtained by using a minimum follower in each frequency band. In some such examples, for each input frame, if the input band energy is below the current noise floor estimate for the band, then the noise floor estimate can be immediately updated to the new energy level (or can be aggressively tilted closer to the new energy level value). According to some such embodiments, if the input band is greater than the noise floor estimate, then the current estimated noise floor estimate can be very slowly tilted toward the new energy level value. It can be advantageous to not only track the minimum noise, but also to track the variance of the noise in each band and / or to track the covariance of the noise between bands.

[0111] In some examples, after a stationary noise estimate has been made, it can be assumed that if the stationary noise estimate is close to the noise floor (low estimated SNR), then the particular time-frequency tile is unlikely to be relevant for a given speech analysis and / or speech modification task, so w can be set close to zero for such frequency bands. On the other hand, if the time-frequency tile is greater than a threshold amount above the noise floor (high estimated SNR), then the time-frequency tile is likely to be relevant, so in some examples the value of w can be set close to 1 for that frequency band. Some examples can implement this feature by using a logistic function as the mapping between SNR and confidence w.

[0112] According to some implementations, the confidence weighting can be based at least in part on an acoustic echo estimate. For example, if a device includes one or more microphones in addition to one or more loudspeakers, the device (or another device) can predict how the loudspeaker acoustically couples to the microphone(s) over time by observing both signals. Such a prediction or determination can typically be made in the context of echo cancellation or suppression. However, when used with some disclosed implementations, the acoustic echo prediction or determination can not necessarily be used to cancel or suppress the echo, but rather merely to predict the echo and then provide an indication to the normalization stage disclosed herein to ignore frequency bands that are highly contaminated by echo when normalizing.

[0113] In some examples, an echo canceller can cancel approximately 20 dB of echo. That is, the ERLE (echo return loss enhancement) can be approximately 20 dB. In a communication system, the echo cancellation process is typically followed by an echo suppression process, which operates in the frequency band energy domain and is used to actively suppress residual echo that can remain after the echo cancellation process. Such an echo suppression process typically works by first predicting the SER (signal-to-echo ratio) of each frequency band in each frame and then determining a suppression gain for the frequency band based on the SER. According to some disclosed implementations, the SER can be used to drive w, e.g., by using a logistic function as described for the SNR in the case of noise suppression, rather than applying a suppression gain.

[0114] According to some implementations, the confidence weighting can be based at least in part on sound direction / correlation (when multiple microphones are present). If a time-frequency tile contains mostly coherent plane wave energy arriving from a direction known to correspond to a speaker of interest (e.g., determined by a spatial covariance estimate), then in some disclosed implementations the corresponding frequency band can be highly weighted when computing the normalization. If a time-frequency tile appears to come from a known source of interference or is a mix of diffuse ambient sound, then in some disclosed implementations the corresponding frequency band can be given a lower weight when computing the normalization.

[0115] In some examples, the confidence weighting can be based at least in part on the presence or absence of reverberation (e.g., by ignoring reverberation). It can be possible to predict whether certain time-frequency tiles are caused primarily by reverberation (from a single microphone) rather than direct sound, e.g., by knowing how long it takes for sound to decay in a particular environment. For example, such a prediction can be based on an estimate of the RT60 for a particular environment, which is the time (in seconds) it takes for sound to decay 60 dB in a reverberant environment. If the disclosed implementation estimates that a particular frequency band contains direct speech from a speaker with high confidence, the implementation can set the value of w to be close to 1. If the disclosed implementation estimates that a frequency band is more likely to be composed of reverberation energy, the implementation can set the value of w to be close to 0.

[0116] In some alternative implementations, the above equations 200 and 201 can be replaced with equations 200A and 201A, respectively:

[0117] (Equation 200A)

[0118] (Equation 201A)

[0119] Finite Impulse Response (FIR)

[0120] Equations 200, 200A, 201, and 201A describe a process of averaging over time. These are Finite Impulse Response (FIR) averages. In other words, to implement these equations, it is necessary to know x[f, t-T] at frame t. According to some examples described above, a history buffer (e.g., an array 308 of Figure 3A ) can be maintained in order to retain data from T previous frames.

[0121] In some alternative implementations involving the application of equations 200A and 201A (or similar equations), instead of using a history buffer, a running approximation (an "IIR average") of is maintained and updated over time as follows:

[0122] (Equation 208)

[0123] In equation 208, represents a constant. An example value for is 0.96, but in other examples, may be larger or smaller. Some such implementations can be more computationally efficient than the examples described above with reference to equations 200 and 201. Furthermore, by adjusting the value of , such implementations can optionally be more dependent on recent data than older data (or vice versa).

[0124] In Figure 3B the example shown in FIG. 3, block 355 involves outputting a normalized feature data structure, which can be an example of the normalized feature set 310 shown in FIG. 3. According to this example, block 360 involves performing a signal recognition or modification process based at least in part on the normalized feature data structure. In some such examples, block 360 can involve inputting the normalized feature data structure to an acoustic model, such as one of the acoustic models referenced herein (e.g., the acoustic model 311 of FIG. 3). Figure 3A Figure 3A According to some implementations, block 360 can involve one or more of the processes described above with reference to block 235 of FIG. 2. Figure 2

[0125] The following is an example of implementing the normalization block 309 via Julia 1.0:

[0126]

[0127] Figure 4 An example of an architecture for implementing the acoustic model of FIG. 3 is shown. In this implementation, the acoustic model 311 is implemented according to one example of a DNN. Figure 3A

[0128] In this example, the normalized feature set 310 is provided to the acoustic model 311. According to this implementation, the array 310A of the normalized feature set 310 is provided to a flattening block 401. According to this example, the flattening block 401 is configured to flatten the stack of spectral features 310A into a one-dimensional vector. In some examples, the flattening block 401 can be configured to flatten the stack of spectral features 310A by reading the rows of the stack of spectral features 310A, while in other examples, the flattening block 401 can be configured to flatten the stack of spectral features 310A by reading the columns of the stack of spectral features 310A down. In alternative implementations, the flattening block 401 can be configured to flatten the stack of spectral features 310A by reading the stack of spectral features 310A in another manner (e.g., along the diagonal).

[0129] ​​​According to this example, the flattened spectral feature vectors output by the flattening block 401 are provided to a concatenation block 402. In this example, the cepstral features 310B are provided directly to the concatenation block 402 because the cepstral features 310B do not need to be flattened. In this implementation, the concatenation block 402 is configured to combine the flattened spectral feature vectors together with the cepstral feature vectors to form a single unified feature vector 410. In some implementations, the cepstral features are placed first, followed by the flattened spectral feature vectors, while in other implementations the flattened spectral feature vectors can be placed first. In other implementations, the flattened spectral feature vectors and the cepstral features can be interleaved.

[0130] In this example, the concatenation block 402 is configured to output the single unified feature vector 410 to an input layer 403 of a DNN. For example, the input layer 403 can be a fully connected (i.e., dense) layer with bias and a non-linear activation function (e.g., logistic, hyperbolic tangent, rectified linear unit (ReLU), LeakyReLU). For example, the input layer 403 can produce between 64 and 1024 output values.

[0131] According to this example, the DNN includes a plurality of hidden layers 404. In some implementations, the DNN can include between 2 and 6 hidden layers, while in other implementations the DNN can include more hidden layers. For example, each hidden layer 404 can be a fully connected (i.e., dense) layer with bias and a non-linear activation function (e.g., logistic, hyperbolic tangent, rectified linear unit (ReLU), LeakyReLU). For example, each hidden layer 404 can produce between 64 and 1024 output values.

[0132] In this implementation, the output layer 405 produces a phoneme probability estimate (312) corresponding to each of the plurality of phoneme (e.g., phoneme, diphone, triphone) classes for the input feature set 310 as a probability vector. For example, one output can correspond to the user making the sound “oo” as in “book”.

[0133] Some embodiments include a system or device configured (e.g., programmed) to perform one or more disclosed methods, and a tangible computer readable medium (e.g., a disk) storing code for implementing one or more disclosed methods or steps thereof. For example, a system can be or include a programmable general purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general purpose processor can be or include a computer system including an input device, a memory, and a processing subsystem programmed (and / or otherwise configured) to perform embodiments of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0134] Some embodiments of the disclosed system can be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform the required processing on the audio signal(s), including performance of embodiments of the disclosed methods. Alternatively, embodiments of the disclosed system (or elements thereof) can be implemented as a general purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor that can include an input device and a memory) programmed with software or firmware and / or otherwise configured to perform any of a variety of operations including embodiments of the disclosed methods. Alternatively, elements of some embodiments of the disclosed system can be implemented as a general purpose processor or DSP configured (e.g., programmed) to perform embodiments of the disclosed methods, and the system can further include other elements (e.g., one or more loudspeakers and / or one or more microphones). A general purpose processor configured to perform embodiments of the disclosed methods can generally be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and in some examples a display device.

[0135] Another aspect of the present disclosure can be implemented in one or more non-transitory computer readable media (e.g., one or more RAM, ROM, magnetic disks, or other tangible storage media) storing code (e.g., an encoder executable to perform any embodiment of the disclosed methods or steps thereof) for performing any embodiment of the disclosed methods or steps thereof.

[0136] While specific embodiments and applications of the present disclosure have been described herein, it is to be understood that numerous modifications and variations have been omitted, and it is contemplated to be within the scope of the disclosure.

[0137] Various aspects of the application can be appreciated from the following enumerated example embodiments (EEEs):

[0138] EEE 1. An audio processing method comprising:

[0139] receiving microphone audio data from a microphone system;

[0140] digitizing the microphone audio data to produce digitized microphone audio data;

[0141] framing the digitized microphone audio data to produce frames of digitized microphone audio data;

[0142] extracting a feature vector from each frame of the digitized microphone audio data, the feature vector comprising a power value for each of a plurality of frequency bands;

[0143] forming a feature history data structure comprising a plurality of feature vectors;

[0144] producing a normalized feature set comprising a normalized feature data structure by determining normalized power values for the plurality of frequency bands for each feature vector of the feature history data structure; and

[0145] performing a signal recognition or modification process based at least in part on the normalized feature data structure.

[0146] EEE 2. The audio processing method of EEE 1, wherein the signal recognition or modification process comprises a speech recognition process.

[0147] EEE 3. The method of EEE 2, wherein the speech recognition process comprises a wake-word recognition process.

[0148] EEE 4. The method of EEE 2 or EEE 3, wherein performing the signal recognition or modification process comprises:

[0149] providing the normalized feature data structure to an acoustic model; and

[0150] determining a phoneme probability estimate based on an output of the acoustic model.

[0151] EEE 5. The method of EEE 4, wherein the acoustic model comprises a neural network.

[0152] EEE 6. The method of any of EEEs 1-5, wherein the signal recognition or modification process comprises one or more processes selected from a list of processes consisting of: a speaker recognition process; a language recognition process; an emotion recognition process; a non-verbal sound classification process; and a noise suppression process.

[0153] EEE 7. The method of any of EEEs 1-6, wherein the frequency bands have equal or substantially equal widths in frequency space.

[0154] EEE 8. The method of EEE 7, wherein the frequency space is a mel-frequency space, a bark scale, or an equivalent rectangular bandwidth scale.

[0155] EEE 9. The method of any of EEEs 1-8, wherein the power values comprise a logarithm of power.

[0156] EEE 10. The method of any of EEEs 1-9, wherein generating the normalized feature data structure involves determining an average wideband level based on a current frame and a most recent previous frame.

[0157] EEE 11. The method of EEE 10, wherein generating the normalized feature data structure involves determining an average spectral representation based on the current frame and the most recent previous frame.

[0158] EEE 12. The method of any of EEEs 1-9, wherein generating the normalized feature data structure involves determining an average wideband level based on all frames of the feature history data structure.

[0159] EEE 13. The method of EEE 10, wherein generating the normalized feature data structure involves determining an average spectral representation based on all frames of the feature history data structure.

[0160] EEE 14. The method of any of EEEs 1-13, wherein generating the normalized feature data structure involves determining a confidence weighting factor for each feature.

[0161] EEE 15. The method of EEE 14, wherein determining the confidence weighting factor is based at least in part on one or more of a noise estimate, a signal-to-noise ratio estimate, an echo prediction, a signal-to-echo ratio prediction, a sound direction estimate, or a reverberation estimate.

[0162] EEE 16. The method of EEE 13, wherein generating the normalized feature set involves obtaining an average level independent spectral representation.

[0163] EEE 17. The method of EEE 16, wherein generating the set of normalized features involves performing a cepstrum decomposition of the average level independent spectral representation.

[0164] EEE 18. The method of EEE 17, wherein performing the cepstrum decomposition involves applying a decorrelation transform represented by a basis matrix.

[0165] EEE 19. The method of EEE 18, wherein the basis matrix represents one or more of a discrete sine transform basis, a discrete cosine transform basis, or a Karhunen-Loeve transform basis.

[0166] EEE 20. The method of EEE 13, wherein the set of normalized features further includes a set of coefficients corresponding to an overall average spectral shape of the normalized feature data structure, the set of coefficients based at least in part on the cepstrum decomposition.

[0167] EEE 21. The method of any of EEEs 1-20, further comprising performing a subsequent task based at least in part on the signal recognition or modification process, wherein the subsequent task includes one or more tasks selected from a list of tasks consisting of: controlling a speaker to produce a responsive sound; performing an internet search; initiating a telephone call; sending a text message; sending an email message; and changing a setting of a device.

[0168] EEE 22. An apparatus configured to perform the method of any of EEEs 1-21.

[0169] EEE 23. One or more non-transitory media having software recorded thereon, the software comprising instructions for controlling one or more apparatuses to perform the method of any of EEEs 1-21.

[0170] EEE 24. An apparatus comprising:

[0171] an interface system configured to receive microphone audio data from a microphone system; and

[0172] a control system configured to:

[0173] digitize the microphone audio data to produce digitized microphone audio data;

[0174] frame the digitized microphone audio data to produce frames of digitized microphone audio data;

[0175] extracting a feature vector from each frame of the digitized microphone audio data, the feature vector including a power value for each of a plurality of frequency bands;

[0176] forming a feature history data structure including the plurality of feature vectors;

[0177] generating a normalized feature set including a normalized feature data structure by determining normalized power values for the plurality of frequency bands of each feature vector of the feature history data structure; and

[0178] performing a signal recognition or modification process based at least in part on the normalized feature data structure.

[0179] EEE 25. One or more non-transitory media having software recorded thereon, the software including instructions for controlling one or more devices to perform a method, the method comprising:

[0180] receiving microphone audio data from a microphone system;

[0181] digitizing the microphone audio data to produce digitized microphone audio data;

[0182] framing the digitized microphone audio data to produce frames of digitized microphone audio data;

[0183] extracting a feature vector from each frame of the digitized microphone audio data, the feature vector including a power value for each of a plurality of frequency bands;

[0184] forming a feature history data structure including the plurality of feature vectors;

[0185] generating a normalized feature set including a normalized feature data structure by determining normalized power values for the plurality of frequency bands of each feature vector of the feature history data structure; and

[0186] performing a signal recognition or modification process based at least in part on the normalized feature data structure.

[0187] EEE 26. The one or more non-transitory media of EEE 25, wherein the signal recognition or modification process includes a speech recognition process.

[0188] EEE 27. The one or more non-transitory media of EEE 26, wherein the speech recognition process includes a wake-word recognition process.

Claims

1. A method of audio processing, comprising: receiving frames of digitized microphone audio data; extracting, from each frame of the digitized microphone audio data, a feature vector comprising a power value for each of a plurality of frequency bands; forming a feature history data structure comprising a plurality of feature vectors; producing a normalized feature set comprising a normalized feature data structure by determining normalized power values for the plurality of frequency bands of each feature vector of the feature history data structure; and performing a signal recognition or modification process based at least in part on the normalized feature set, wherein the normalized feature set further comprises a set of cepstral coefficients describing an overall average spectral shape of the normalized feature data structure; determining an average spectral representation based on a current frame and at least one previous frame, comprising computing, for each of the plurality of frequency bands, an average or weighted average of a power value of a respective frequency band of the current frame and a power value of the respective frequency band of the at least one previous frame; and determining the set of cepstral coefficients based on the average spectral representation, wherein producing the normalized feature data structure is based on the average spectral representation; determining an average wideband level representing an average over all frequency bands of the average spectral representation; determining an average level independent spectral representation by subtracting the average wideband level from the average spectral representation; determining a cepstral decomposition of the average level independent spectral representation; and determining the set of cepstral coefficients based on the cepstral decomposition.

2. The method of claim 1, wherein, Producing the normalized feature data structure comprises: smoothing the average level independent spectral; and subtracting the smoothed average level independent spectral and the average wideband level from each feature vector of the feature history data structure.

3. The method of claim 2, wherein, Smoothing the average level independent spectral comprises performing spectral resynthesis on the cepstral decomposition.

4. The method of any one of claims 1 to 3, wherein, The cepstral decomposition comprises individual cepstrum coefficients dnalb, and the set of cepstral coefficients corresponds at least to a subset of the cepstrum coefficients dnalb of the cepstral decomposition, optionally to a subset excluding a first cepstrum coefficient dnalb.

5. The method of any one of claims 1 to 3, wherein, The average spectral representation is determined as an infinite impulse response, IIR, average based on the current frame and a most recent previous frame.

6. The method of any one of claims 1 to 3, wherein, The average spectral representation is determined as a finite impulse response, FIR, average based on all frames of the feature history data structure by computing, for each of the plurality of frequency bands, an average of power values of a respective frequency band of all frames of the feature history data structure.

7. The method of any one of claims 1 to 3, wherein, The signal recognition or modification process comprises a speech recognition process.

8. The method of claim 7, wherein, The speech recognition process comprises a wake-word recognition process.

9. The method of claim 7, wherein, Performing the signal recognition or modification process comprises: providing the normalized feature set to an acoustic model; and determining a phoneme probability estimate based on an output of the acoustic model.

10. The method of claim 9, wherein, The acoustic model comprises a neural network.

11. The method of any one of claims 1 to 3, wherein, The signal recognition or modification process comprises one or more processes selected from a list of processes consisting of: a speaker recognition process; a language recognition process; an emotion recognition process; a non-speech sound classification process; and a noise suppression process.

12. The method of any one of claims 1 to 3, wherein, The plurality of frequency bands have equal or substantially equal widths in a frequency space.

13. The method of claim 12, wherein, The frequency space is a mel-frequency space, a bark scale, or an equivalent rectangular bandwidth scale.

14. The method of any one of claims 1 to 3, wherein, The power values comprise logarithms of power.

15. The method of any one of claims 1 to 3, wherein, Generating the normalized feature data structure involves determining a confidence weighting factor for each feature, wherein determining the confidence weighting factor is based at least in part on one or more of a noise estimate, a signal-to-noise estimate, an echo prediction, a signal-to-echo prediction, a sound direction estimate, or a reverberation estimate.

16. The method of any one of claims 1 to 3, further comprising performing a subsequent task, the subsequent task based at least in part on the signal identification or modification process, wherein, The subsequent task comprises one or more tasks selected from a list of tasks consisting of: controlling a loudspeaker to produce a responsive sound; performing an internet search; initiating a telephone call; sending a text message; sending an email message; and changing a setting of the device.

17. The method of any one of claims 1 to 3, wherein, Receiving frames of digitized microphone audio data comprises: receiving microphone audio data from a microphone system; digitizing the microphone audio data to produce digitized microphone audio data; and framing the digitized microphone audio data to produce frames of digitized microphone audio data.

18. An audio processing method comprising: receiving frames of digitized microphone audio data; extracting a feature vector from each frame of the digitized microphone audio data, the feature vector comprising a power value for each of a plurality of frequency bands; forming a feature history data structure comprising a plurality of feature vectors; generating a normalized feature set comprising a normalized feature data structure by determining normalized power values for the plurality of frequency bands of each feature vector of the feature history data structure; and performing a signal recognition or modification process based at least in part on the normalized feature set, wherein the normalized feature set further comprises a set of cepstral coefficients describing an overall average spectral shape of the normalized feature data structure; determining an average spectral representation based on a current frame and at least one previous frame, comprising computing, for each of the plurality of frequency bands, an average or a weighted average of a power value of a respective frequency band of the current frame and a power value of the respective frequency band of the at least one previous frame; and determining the set of cepstral coefficients based on the average spectral representation, wherein generating the normalized feature data structure is based on the average spectral representation; determining a cepstral decomposition of the average spectral representation or a smoothed version thereof; and determining the set of cepstral coefficients based on the cepstral decomposition.

19. The method of claim 18, wherein, Generating the normalized feature data structure comprises: subtracting an average level independent spectrum or a smoothed version thereof from each feature vector of the feature history data structure.

20. The method of any one of claims 18-19, wherein, The cepstral decomposition comprises cepstrum dnas of an order, and the set of cepstral coefficients corresponds at least to a subset of the cepstrum dnas of the cepstral decomposition, optionally to a subset excluding a first cepstrum dna.

21. The method of any one of claims 18-19, wherein, The average spectral representation is determined as an infinite impulse response, IIR, average based on the current frame and a most recent previous frame.

22. The method of any one of claims 18-19, wherein, The average spectral representation is determined based on all frames of the feature history data structure by computing, for each of the plurality of frequency bands, an average of power values of the respective frequency band for all frames of the feature history data structure.

23. The method of any one of claims 18-19, wherein, The signal recognition or modification process comprises a speech recognition process.

24. The method of claim 23, wherein, The speech recognition process comprises a wake-word recognition process.

25. The method of claim 23, wherein, Performing the signal recognition or modification process comprises: providing the set of normalized features to an acoustic model; and determining a phoneme probability estimate based on an output of the acoustic model.

26. The method of claim 25, wherein, The acoustic model comprises a neural network.

27. The method of any one of claims 18-19, wherein, The signal recognition or modification process comprises one or more processes selected from a list of processes consisting of: a loudspeaker recognition process; a language recognition process; an emotion recognition process; a non-speech sound classification process; and a noise suppression process.

28. The method of any one of claims 18-19, wherein, The plurality of frequency bands have equal or substantially equal widths in frequency space.

29. The method of claim 28, wherein, The frequency space is a mel-frequency space, a Bark scale, or an equivalent rectangular bandwidth scale.

30. The method of any one of claims 18-19, wherein, The power values comprise a logarithm of power.

31. The method of any one of claims 18-19, wherein, Producing the normalized feature data structure involves determining a confidence weighting factor for each feature, wherein determining the confidence weighting factor is based at least in part on one or more of a noise estimate, a signal-to-noise ratio estimate, an echo prediction, a signal-to-echo ratio prediction, a sound direction estimate, or a reverberation estimate.

32. The method of any of claims 18 to 19, further comprising performing a subsequent task, the subsequent task based at least in part on the signal identification or modification process, wherein, The subsequent task comprises one or more tasks selected from a list of tasks consisting of: controlling a loudspeaker to produce a responsive sound; performing an internet search; initiating a telephone call; sending a text message; sending an email message; and changing a setting of a device.

33. The method of any one of claims 18 to 19, wherein, Receiving frames of digitized microphone audio data comprises: receiving microphone audio data from a microphone system; digitizing the microphone audio data to produce digitized microphone audio data; and framing the digitized microphone audio data to produce frames of digitized microphone audio data.

34. An audio processing apparatus, the apparatus configured to perform the method of any one of claims 1 to 33.

35. One or more non-transitory media having software recorded thereon, the software comprising instructions for controlling one or more devices to perform the method of any one of claims 1 to 33.

36. A computer program product, the computer program product comprising a program, the program comprising computer executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 33.

Images