Learned audio front-end machine learning models for audio understanding

Abstract

Methods, systems, and apparatuses including computer programs encoded on a computer storage medium for processing audio input using a learned audio front-end machine learning model to generate a representation of the audio input. This representation can then be processed by an audio understanding model to generate a corresponding output for each of one or more audio understanding tasks.

Description

BACKGROUND

[0001] This specification relates to processing audio using a machine learning model.

[0002] Examples of machine learning models include support vector machines, decision forests, linear models, and neural networks.

[0003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY

[0004] This specification describes a system implemented as computer programs on one or more computers in one or more locations, the computer programs using a machine learning model to process an audio signal - i.e., an audio waveform comprising a sequence of audio samples, e.g., amplitude values - to generate an output for an audio understanding task.

[0005] An audio understanding task can be any task that requires processing an audio waveform to generate a prediction characterizing the audio waveform.

[0006] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0007] Many existing neural network systems that perform audio understanding tasks use a mel filterbank representation of the input audio waveform. Mel filterbanks are fixed, designed audio features that mimic human perception. However, mel filterbanks are limited by the fundamental limitations of the hand-designed representation. This specification describes a single, general, learnable front-end that outperforms mel filterbanks over a wide range of audio domains, including speech, music, audio events, and animal sounds, thereby providing a general learning front-end for audio. In particular, this specification describes a lightweight, fully learnable architecture that can be used as a direct replacement for mel filterbanks. In other words, this learned audio front-end model has very few parameters, and because those parameters are learned, it can achieve improved performance with minimal computational overhead during training and inference.

[0008] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 An example audio processing system is shown.

[0010] Figure 2 is a flowchart of an example process for training a policy neural network and an embedding neural network.

[0011] Figure 3 is a flowchart of an example process for training a policy neural network on a demonstration sequence.

[0012] Figure 4 training a policy neural network using a demonstration sequence is illustrated.

[0013] The same reference numbers and designations in the various drawings indicate the same elements. DETAILED DESCRIPTION

[0014] This specification describes a system implemented as computer programs on one or more computers in one or more locations, which use a machine learning model to process an audio signal - i.e. an audio waveform comprising a sequence of audio samples, e.g. amplitude values - to generate an output for an audio understanding task.

[0015] The audio understanding task can be any task that requires processing of an audio waveform to generate a prediction characterising the audio waveform.

[0016] For example, if the input is an audio sequence representing a spoken utterance, i.e. an audio waveform, the audio understanding task can be speech recognition, e.g. with an output defining a score for each of a set of text segments, each score representing an estimated likelihood that the text segment is a correct transcription of the utterance.

[0017] As another example, the audio understanding task can be a keyword spotting task, in which, if the input is an audio sequence representing a spoken utterance, the output can indicate whether a particular word or phrase (a "hotword") was spoken in the utterance.

[0018] As another example, the audio understanding task can be a language identification task, in which, if the input is an audio sequence representing a spoken utterance, the output generated by the student neural network can identify the natural language in which the spoken utterance was spoken.

[0019] Audio understanding tasks can also be tasks performed on audio that is not speech. Examples of such audio understanding tasks include: a music recognition task that receives an audio sequence as input and generates a prediction of the name or other attribute of a piece of music represented in the input; an animal classification task that receives an audio input that includes one or more animal sounds and generates a prediction of the class of animal that is making the animal sound; an audio event classification task in which the input is an audio sequence, is a prediction of what time instances each of a plurality of different sounds is active within the audio sequence; etc.

[0020] More generally, while this specification describes a front-end that replaces a Mel- filterbank representation in processing an audio signal, the front-end can instead replace a hand- designed or otherwise learned representation of a different kind of input signal - e.g., seismic data or a physiological recording.

[0021] Figure 1 An example audio processing system 100 is shown. The audio processing system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0022] The audio processing system 100 obtains an audio waveform 102 as input. The audio waveform 102 is a sequence of audio samples, e.g., amplitude values, at a first frequency (“sample frequency”).

[0023] The audio processing system 100 processes the audio waveform 102 using a learned audio front-end model 110 to generate a feature representation 112 of the audio waveform.

[0024] In general, the audio front-end model 110 is a machine learning model that is configured to apply a learned filtering operation with a plurality of filtering parameters, a learned pooling operation with a plurality of pooling parameters, and a learned normalization operation with a plurality of normalization parameters to generate the feature representation 112 of the audio waveform 102.

[0025] Reference is made below to Figures 2 to 4 The operations performed by the audio front-end model 110 to generate the feature representation 112 are described in more detail.

[0026] The system 100 processes the feature representation 112 using an audio understanding machine learning model 120. The audio understanding machine learning model 120 is a machine learning model with parameters (“audio understanding parameters”) that is configured to process the feature representation 112 to generate a respective output 122 for each of one or more audio understanding tasks—e.g., for one or more of the tasks described above or for a different task that requires a prediction of the content of the audio waveform 102.

[0027] The audio understanding model 120 can be any appropriate model, e.g., a model that was previously configured to receive as input a mel-filterbank representation of an audio signal. That is, as a particular example, the audio front-end model 110 can replace in an audio processing pipeline a system that maps an audio waveform to a mel-filterbank representation of the audio waveform that is provided as input to the audio understanding model 120.

[0028] Particular examples of audio understanding models that can receive as input the feature representation 112 include: a convolutional neural network, e.g., a convolutional neural network with an EfficientNet architecture; a fully connected neural network, e.g., a multi-task neural network with a respective set of linear layers for each of multiple tasks; a recurrent neural network, e.g., a neural network based on long short-term memory (LSTM) or gated recurrent units (GRU); or a self-attention neural network, e.g., a transformer neural network.

[0029] The audio front-end model 110 is referred to as a “learned” audio front-end model because the values of the parameters of the learned audio front-end model—i.e., the values of the filtering, pooling, and normalization parameters—are learned end-to-end with the audio understanding model 120. In other words, the operations performed by the audio front-end model 110 are fully differentiable, allowing the audio front-end model 110 to be jointly trained with the “back-end” model by gradient descent. This is in contrast to other prominent representations—e.g., mel-filterbank representations—that are hard-coded and thus cannot be fine-tuned to improve the performance of a given model 120 on a given set of one or more audio processing tasks.

[0030] In particular, the system 100 includes a training engine 150 that trains the audio front-end model 110 and the audio understanding model 120 on respective training data for each of one or more audio understanding tasks.

[0031] The training data for a given task includes a set of training audio inputs, and for each audio input, includes a target output for the given task that should be generated by the audio understanding model 120 by processing a feature representation of the training audio input generated by the audio front-end model 110.

[0032] Training engine 150 trains models 110 and 120 on training data for a given task using gradient descent. Specifically, audio front-end model 110 is trained using audio understanding model 120 and by backpropagating the gradient of the loss function for a given task to audio front-end model 110. The loss function can be any suitable loss function for the task, which measures the performance of audio understanding model 120 on a given task given the target output in the training data for that task; for example, a classification loss if the audio understanding task is a classification task, or a regression loss if the audio understanding task is a regression task. Specifically, the system can train models 110 and 120 to determine the training values ​​of parameters θ of audio understanding model 120 and parameters ψ of audio front-end model 110 using any suitable gradient descent-based technique (e.g., stochastic gradient descent, Adam optimization, rmsProp optimization, etc.) to minimize the following:

[0033] argmin θ，ψ E (x，y)∈D L(g θ (F ψ (x))), y),

[0034] Where x is the training audio input sampled from the training dataset D, y is the target output for the training input, and F... ψ (x) is the representation generated by model 110 by processing the training audio input x, g θ (F ψ (x) is the output generated by model 120 by processing this representation, and L is the loss function for a given task.

[0035] In some cases, the training engine 150 can jointly pre-train the audio front-end model 110 with the original back-end model on one or more tasks, and then train the audio front-end model 110 on different tasks using the new back-end model. For example, it can simultaneously keep the pre-trained values ​​of the parameters of model 110 fixed and update the parameters of the new back-end model, or update the parameters of the new back-end model while also fine-tuning the pre-trained values ​​of the parameters of model 110. This can be accomplished, for example, when there is a large amount of training data available for one or more tasks that the original back-end model can perform, while only a limited amount of training data is available for the tasks that the new back-end model can perform.

[0036] Figure 2 This is a flowchart of an example process 200 for processing audio input. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed audio processing system—for example, Figure 1 The audio processing system 100 is capable of executing process 200.

[0037] The system obtains the audio waveform (step 202). The audio waveform is a sequence of audio samples at a first frequency. That is, the audio waveform is a sequence with a corresponding scalar audio sample at each of the T first time steps. In other words, the audio waveform is a one-dimensional waveform with T samples available at the sampling frequency Fs Hz.

[0038] The system uses the learned audio front-end model to process the audio waveform to generate a feature representation of the audio waveform (step 204).

[0039] The feature representation includes a feature sequence at a second frequency. Typically, the second frequency is lower than the first frequency.

[0040] More specifically, the operation of the learned audio front-end can be represented as a parameterized function, which has been learned by training the learned audio front-end, and maps the audio waveform from a one-dimensional space to a two-dimensional feature space. That is, the learned parameters map T scalar values ​​in the audio waveform to a 2D M×N representation, where M represents the number of time frames (“features”) and N represents the number of feature channels in each time frame (“feature”) of the representation. Because the second frequency will typically be lower than the first frequency, the number of time frames M will typically be less than the number of audio samples T in the audio waveform.

[0041] The following is for reference. Figure 3 and Figure 4 A more detailed description is given of using a learned audio front-end model to process audio waveforms to generate feature representations.

[0042] The system uses an audio understanding machine learning model with multiple audio understanding parameters to process the feature representation to generate the corresponding output for each of the one or more audio understanding tasks as described above (step 206).

[0043] Figure 3 This is a flowchart of an example process 300 for generating a feature representation of an audio waveform using a learned audio front-end model. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed audio processing system—e.g., Figure 1 The audio processing system 100 is capable of executing process 300.

[0044] The system applies a learned filtering operation with multiple filtering parameters to the audio waveform to generate a filtered representation including a filter feature sequence of a first frequency (step 302).

[0045] At higher levels, the learned filtering operation passes the audio waveform through a set of bandpass filters, followed by non-linear operation at the original sampling frequency Fs.

[0046] In other words, the filter feature sequence includes the corresponding filter features for each first time step in the audio waveform, and therefore, each filter feature in the sequence has a corresponding audio sample in the audio waveform.

[0047] Typically, each filter feature includes a corresponding value for each of the multiple channels. That is, the learned filtering operation maps T scalar audio samples in the input waveform to T filter features, each filter feature having N feature channels, where N is greater than one.

[0048] To generate values ​​for any given channel of the filter features, the system applies one or more one-dimensional convolutional filters to the audio waveform to generate values ​​for the filter features used for that channel.

[0049] In some implementations, for each channel, the system can apply a single one-dimensional convolutional filter with a stride of 1 to the audio waveform to generate values ​​for the filtering characteristics of that channel.

[0050] As a specific example, each convolutional filter can be a complex-valued filter, and the system can generate T values ​​f for the filter features of the nth channel by performing the following operations. n :

[0051]

[0052] Where x is the audio waveform. It is a complex-valued filter with n channels, and the * operation represents a convolution with a stride of 1. Applying the square modulus operator brings the output of the convolution back to the real domain.

[0053] In some other implementations, for each channel, the system can apply multiple one-dimensional convolutional filters with a stride of 1 to the audio waveform to generate values ​​for the filtering characteristics of that channel.

[0054] As a specific example, for each channel, the system is able to apply multiple one-dimensional convolutional filters to the audio waveform to generate multiple convolutional values, and then combine the convolutional values ​​to generate a value for that channel.

[0055] More specifically, the system can apply a total of 2N real-valued one-dimensional filters to generate T values ​​f for the filter characteristics of the nth channel by performing the following operations. n :

[0056]

[0057] in, and These are two of a total of 2N real-valued one-dimensional filters used to calculate the value of the nth channel. For example, and It can be the real and imaginary parts of a single complex-valued one-dimensional filter. Applying a total of 2N filters in this way can generate an output equivalent to that generated using the complex-valued filters described above, without requiring the system to explicitly manipulate the complex numbers.

[0058] In some implementations, the system does not impose any constraints on the values ​​(“coefficients”) of the one-dimensional filter learned during training; that is, the one-dimensional filter is a standard 1D convolution. For example, the system is able to initialize the filter coefficients to approximate the computation of a Mel filter bank and then adjust the filter via backpropagation during training without imposing any constraints on any coefficients in the filter.

[0059] In some of these implementations, to prevent instability during training, the system applies normalization, such as L2 normalization, to the filter coefficients before calculating the convolution.

[0060] In some implementations, the system does impose constraints on the coefficients of the one-dimensional filters. In particular, the system can require that each of a plurality of one-dimensional convolutional filters is a Gabor filter.

[0061] Typically, Gabor filters are generated by modulating a Gaussian kernel with a sinusoidal signal. Formally, a Gabor filter has a length W. Through its center frequency η n and inverse bandwidth σ n The parameters are as follows:

[0062] For t = -W / 2, ..., W / 2,

[0063] In other words, the equation above defines the W coefficients of a Gabor filter of length W. Therefore, the Gabor filter... The frequency response is expressed in terms of frequency η n Centered on and with a bandwidth of 1 / σ n Both are Gaussians, and both are expressed in normalized frequency units in [-1 / 2, 1 / 2].

[0064] For each of the N Gabor filters, the system only needs to learn two parameter values ​​(center frequency and inverse bandwidth), and learning these two parameter values ​​for each filter allows for the learning of a set of smooth quasi-analytical bandpass filters with controllable center frequency and bandwidth.

[0065] Therefore, Gabor filters have significantly fewer parameters than normalized 1D convolutions. For example, N Gabor filters of length W are entirely specified by 2N parameters, i.e., a center frequency of N and a bandwidth of N, relative to a standard 1D convolution of W×N. As a specific example, when using a window length of 25ms and operating at a sampling rate of 16kHz, W = 401 samples, and Gabor-based filtering considers 200 times fewer parameters than its unconstrained alternative. This enables a significant reduction in the computational intensity of training and a significant reduction in the memory footprint of the learned front-end without compromising its performance.

[0066] To apply the Gabor filter, the system obtains the impulse response of each of the N Gabor filters over the range t = -W / 2,..., to W / 2, and then convolves these impulse responses into the input waveform.

[0067] Alternatively, the system can apply Gabor filters using 2N real-valued filters instead of N complex-valued Gabor filters as described above, i.e., by splitting the real and imaginary parts of the Gabor filters, convolving the input with each filter independently, and then merging the results using squared l2 pooling.

[0068] In some cases, to aid in stable training, the system limits the center frequencies of N Gabor filters to a predefined range (e.g., [0, 1 / 2]), ensuring they fall within the positive portion of the frequency range. The system can also optionally constrain the center frequencies to a predefined range—e.g., the range [0, 1 / 2][i]. — This makes the full width at half maximum (FWHM) of the frequency response within 1 / W and 1 / 2.

[0069] The system applies a learned pooling operation with multiple pooling parameters to the filtered representation to generate a pooling representation that includes a sequence of pooling features with a second frequency (step 304). Each pooling feature has a corresponding value for each of the multiple channels.

[0070] At higher levels, learned pooling operations extract the signal represented by the filtered representation to reduce the time resolution of the filtered representation.

[0071] More specifically, the system implements learned pooling operations through depthwise convolutions with low-pass filters. Therefore, each input channel is associated with a low-pass filter.

[0072] In other words, for each channel, the system performs a learned pooling operation by applying the corresponding learned low-pass filter for that channel to the values ​​in the channel's filter features with a step size greater than one, to generate a set of pooled values ​​for the channel with a second frequency. By making the step size greater than one, the system reduces the number of values ​​in each channel, thereby reducing the temporal resolution of the filtered representation. The value of the step size can be a hyperparameter.

[0073] In some cases, the system parameterizes the corresponding learned low-pass filter for each channel to have a Gaussian impulse response. That is, for channel n, there is a learned low-pass filter of length W. It can satisfy:

[0074] For t = -W / 2, ..., W / 2,

[0075] Therefore, the system is able to learn the low-pass pooling function per channel while adding only N parameters to the front-end model.

[0076] The system applies a learned normalization operation with multiple normalization parameters to the pooling representation to generate a feature representation (step 306).

[0077] At higher levels, the learned normalization operation normalizes the pooling representation, and then nonlinearity is applied to the pooling representation to reduce its dynamic range.

[0078] Specifically, for each corresponding value in each of the multiple channels of each pooling feature, the system normalizes the corresponding value according to one or more normalization parameters, and then applies a nonlinearity to the normalized corresponding value according to one or more of the normalization parameters to generate the corresponding value in the feature representation.

[0079] As a specific example, the system can apply nonlinearity to the corresponding normalized values ​​by adding learned offsets to generate a sum and applying compression to that sum using learned exponents. For instance, the system can learn the corresponding offset and corresponding exponent for each channel.

[0080] As another specific example, for each corresponding value of each channel in multiple channels of each pooling feature, the system is able to normalize the corresponding value using an exponential moving average of the values ​​of the channels in any pooling feature preceding the pooling feature in the sequence. The exponential moving average is controlled by a corresponding smoothing coefficient.

[0081] For example, the exponential moving average M(t,n) of the values ​​F(t,n) of channel n in a feature with index t can be recursively defined as:

[0082] M(t, n) = (1-sn M(t-1,n)+s n F(t, n),

[0083] Where s n It is the smoothing coefficient for channel n.

[0084] Therefore, specific examples of the learned normalization operation can satisfy:

[0085]

[0086] Where ε is a constant value, α n It is the learned exponent used for normalization of channel n, δ n It is the learned offset of channel n, and r n It is the learned exponent used for channel n. Therefore, in this particular example, the system learns s for each of the N channels. n α n δ n and r n The corresponding value.

[0087] Figure 4 The illustration shows an example of how the learned audio front-end model operates.

[0088] like Figure 4 As shown, a learned filtering operation is first used to process an audio waveform 102 with T audio samples to generate a filtered representation, i.e., a sequence of filtered features, each with N channels. The learned filtering operation first applies N learned Gabor filters 410 of length W to the input audio waveform 120, and then applies a squared modulus operator 240 to the output to generate the filtered representation.

[0089] Alternatively, the system can apply Gabor filters using 2N real-valued filters instead of N complex-valued Gabor filters as described above, i.e., by splitting the real and imaginary parts of the Gabor filters, convolving the input with each filter independently, and then merging the results using squared l2 pooling.

[0090] The learned pooling operation is then used to process the filtered representation to generate a pooled representation. The learned pooling operation applies a corresponding Gaussian low-pass filter 430 to each channel of the filtered representation to reduce the time resolution of the filtered representation and generate a pooled representation.

[0091] The learned normalization operation is then used to process the pooling representation to generate feature representations. The learned normalization operation applies the sPCEN operation 440 described above to generate feature representations.

[0092] The feature representation can then be provided to the downstream audio processing neural network for processing, as described above.

[0093] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof are already installed on the system, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a specific operation or action, this means one or more programs comprising instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0094] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware—including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0095] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include a dedicated logic circuit system, such as a FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0096] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data—for example, one or more scripts stored in a markup language document—stored in a single file dedicated to the program in question, or stored in multiple harmonizing files—for example, a file that stores one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a data communication network.

[0097] In this specification, the term "database" is used broadly to refer to any data collection: the data does not need to be structured in any particular way, or not at all, and it can be stored on storage devices in one or more locations. Therefore, for example, an indexed database can include multiple data collections, each of which can be organized and accessed differently.

[0098] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0099] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by a dedicated logic circuit system, such as an FPGA or ASIC, or by a combination of a dedicated logic circuit system and one or more programmed computers.

[0100] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0101] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0102] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device—for example, a smartphone running a messaging application—and subsequently receiving response messages from the user.

[0103] The data processing apparatus for implementing machine learning models may also include, for example, a dedicated hardware accelerator unit for processing the common and computationally intensive portions of machine learning training or production, namely inference and workloads.

[0104] It is possible to use machine learning frameworks, such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet, to implement and deploy machine learning models.

[0105] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server; or middleware components, such as an application server; or front-end components, such as a client computer having a graphical user interface, web browser, or application that a user can interact with through embodiments of the subject matter described in this specification; or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium—e.g., a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0106] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML pages, to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from it. It is possible to receive data generated on the user device, such as the result of user interaction, from the device at the server.

[0107] While this specification contains numerous details of specific implementation, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments in this specification can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination can in some cases be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0108] Similarly, although operations are depicted in a specific order in the accompanying drawings and recited in the claims, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all of the shown operations, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0109] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: Obtain an audio waveform that includes a sequence of audio samples at a first frequency; The audio waveform is processed using a machine learning-based audio front-end model to generate a feature representation of the audio waveform, wherein the feature representation includes a feature sequence at a second frequency, and wherein the machine learning-based audio front-end model is configured to: A machine learning-based filtering operation with multiple filtering parameters is applied to the audio waveform to generate a filtered representation including a sequence of filtered features at the first frequency. A machine learning-based pooling operation with multiple pooling parameters is applied to the filtered representation to generate a pooled representation including a sequence of pooled features at the second frequency; and A machine learning-based normalization operation with multiple normalization parameters is applied to the pooling representation to generate the feature representation; and The feature representation is processed using an audio understanding machine learning model with multiple audio understanding parameters to generate a corresponding output for each of one or more audio understanding tasks. Each filtered feature has a corresponding value for each of the multiple channels, and the application of machine learning-based pooling operations includes: For each channel, the corresponding learned low-pass filter for that channel is applied to the values ​​in the filter features of that channel with a step size greater than one, to generate a set of pooling values ​​for the channel having the second frequency, and The corresponding machine learning-engineered low-pass filter for each channel has a Gaussian impulse response.

2. The method according to claim 1, wherein, The machine learning-based audio front-end model and the audio understanding machine learning model have been trained end-to-end on at least one of the one or more audio understanding tasks to determine the plurality of filtering parameters, the plurality of pooling parameters, and the plurality of normalization parameters.

3. The method according to claim 1, wherein, The second frequency is lower than the first frequency.

4. The method according to any one of claims 1 to 3, wherein, Each filter feature includes a corresponding value for each of the multiple channels, and wherein, for each channel, applying the machine learning-based filtering operation includes: One or more one-dimensional convolutional filters are applied to the audio waveform to generate corresponding values ​​for the channels.

5. The method according to claim 4, wherein, Applying one or more one-dimensional convolutional filters to the audio waveform to generate values ​​for the channel includes: Multiple one-dimensional convolutional filters are applied to the audio waveform to generate multiple convolutional values; and The multiple convolution values ​​are combined to generate corresponding values ​​for the channels.

6. The method according to claim 5, wherein, Each of the plurality of one-dimensional convolutional filters is a Gabor filter.

7. The method according to any one of claims 1 to 3, wherein, Each pooling feature has a corresponding value for each of the multiple channels, and wherein applying the learned normalization operation includes, for each of the multiple channels of each pooling feature: The values ​​of the plurality of channels are normalized using an exponential moving average of the channel values ​​in any pooling feature preceding the pooling feature in the pooling feature sequence, wherein the moving average is controlled by a corresponding smoothing coefficient.

8. The method according to claim 7, wherein, The different corresponding smoothing coefficients are learned for each of the multiple channels.

9. The method according to claim 7, wherein, Applying the learned normalization operation includes, for each of the plurality of channels of each pooling feature, the corresponding value: The learned offset is summed with the normalized value to generate a total; as well as Compression is applied to the sum using the learned exponent.

10. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations according to any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method according to any one of claims 1 to 9.

Images