Audio processing using neural networks

Abstract

The invention relates to audio processing using neural networks. Methods, systems, and apparatus, including computer programs encoded on computer storage media, for audio processing using neural networks. One of the systems includes a plurality of neural network layers, wherein a neural network system is configured to receive time-domain features of an audio sample and process the time-domain features to generate a neural network output for the audio sample, the plurality of neural network layers including: a frequency transform (F-T) layer configured to apply a transform that transforms windows of the time-domain features into frequency-domain features defined by a set of F-T layer parameters; and one or more other neural network layers with respective layer parameters, wherein the one or more neural network layers are configured to process the frequency-domain features to generate the neural network output.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese Invention Patent Application No. 201780029191.6, filed on May 10, 2017. Technical Field

[0003] This specification relates to processing inputs through layers of a neural network to generate outputs. Background Technology

[0004] Neural networks are machine learning models that use one or more non-linear units to predict outputs in response to received inputs. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer serves as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates its output from the received input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0005] Typically, an innovative aspect of the subject matter described in this specification can be embodied in a method for providing a neural network comprising multiple neural network layers, wherein the neural network is configured to receive temporal features of audio samples and process the temporal features to generate a neural network output from the audio samples, wherein the neural network comprises: (i) a frequency transform (FT) layer configured to apply a transform defined by a set of FT layer parameters to convert the temporal features into frequency domain features; and (ii) one or more other neural network layers having corresponding layer parameters and configured to process the frequency domain features to generate a neural network output. The method may also include training the neural network. The method of training the neural network may include: obtaining training data comprising the temporal features of each of a plurality of training audio samples and a known output of the training audio samples; and training the neural network on the training data to adjust the values ​​of the parameters of the other neural network layers and to adjust the values ​​of the FT layer parameters to define an optimal transform from the temporal features to the frequency domain features.

[0006] Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. A system of one or more computers may be configured to perform a specific operation or action by means of software, firmware, hardware, or any combination thereof installed on the system, which, in operation, may cause the system to perform the action. One or more computer programs may be configured to perform a specific operation or action by including instructions that, when executed by a data processing device, cause the device to perform the action.

[0007] According to another aspect, a system is provided, comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the computers to perform operations to implement: a neural network system comprising a plurality of neural network layers, wherein the neural network system is configured to receive temporal features of audio samples and process the temporal features to generate a neural network output for the audio samples, the plurality of neural network layers comprising: a frequency transform (FT) layer configured to apply a transform of the temporal features to frequency domain features defined by a set of FT layer parameters; and one or more other neural network layers having corresponding layer parameters, wherein the one or more neural network layers are configured to process the frequency domain features to generate a neural network output. The FT layer parameters may be trained FT layer parameters that define the optimal transform from the time domain to the frequency domain.

[0008] According to another aspect, a method is provided for training a neural network comprising multiple neural network layers on training data, wherein the neural network is configured to receive temporal features of audio samples and process the temporal features to generate a neural network output from the audio samples, wherein the neural network comprises: (i) a frequency transform (FT) layer configured to apply a transform defined by a set of FT layer parameters to transform the temporal features into frequency domain features; and (ii) one or more other neural network layers having corresponding layer parameters and configured to process the frequency domain features to generate the neural network output, and wherein the method comprises: obtaining training data, the training data comprising temporal features of the training audio samples for each of a plurality of training audio samples and a known output of the training audio samples; and training the neural network on the training data to adjust the values ​​of the parameters of the other neural network layers and to adjust the values ​​of the FT layer parameters to define an optimal transform from the temporal features to the frequency domain features.

[0009] According to another aspect, a system is provided, comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the computers to perform operations for: training a neural network comprising multiple neural network layers on training data, wherein the neural network is configured to receive temporal features of audio samples and process the temporal features to generate a neural network output from the audio samples, wherein the neural network comprises: (i) a frequency transform (FT) layer configured to apply a transform defined by a set of FT layer parameters, the transform transforming a window of the temporal features into frequency domain features; and (ii) one or more other neural network layers having corresponding layer parameters and configured to process the frequency domain features to generate the neural network output, and wherein the operations comprise: obtaining training data, the training data comprising temporal features of the training audio samples for each of a plurality of training audio samples and a known output of the training audio samples; and training the neural network on the training data to adjust the values ​​of the parameters of the other neural network layers and to adjust the values ​​of the FT layer parameters to define an optimal transform from the temporal features to the frequency domain features.

[0010] The foregoing aspects and other embodiments may each optionally include one or more of the following features, individually or in combination. In some embodiments, the window transform of the time-domain features to the frequency-domain features, defined by the FT layer parameter set, is represented by a Discrete Fourier Transform (DFT) matrix.

[0011] In some cases, the set of FT layer parameters includes a transformation matrix, wherein each entry in each row of the transformation matrix includes a corresponding sampling basis function.

[0012] In some implementations, a first parameter for each row determines the frequency of each entry in the row, and a second parameter for each row determines the magnitude of each entry in the row.

[0013] In some cases, the number of rows in the transformation matrix is ​​equal to the number of frequencies extracted.

[0014] In some implementations, the FT layer parameter set includes a transformation matrix, wherein the transformation matrix includes a window function.

[0015] In some cases, the neural network is configured to apply a window function to the input of the FT layer.

[0016] In some respects, the training data includes multiple multi-channel training data.

[0017] In some implementations, the neural network includes multiple Fourier Transform (FT) layers, each FT layer being configured to apply a window transform of time-domain features to frequency-domain features as defined by a corresponding set of FT layer parameters, and training the neural network includes connecting the frequency-domain features generated by each of the multiple FT layers to perform a multi-timescaled Fourier Transform (DFT).

[0018] In some cases, each of the plurality of FT layers is configured to apply a transformation defined by a shared set of FT layer parameters.

[0019] In some implementations, the method further includes merging the plurality of multichannel training data into a single training data input.

[0020] In some cases, training the neural network on the training data includes a window for each temporal feature and for the FT layer: receiving the backpropagation gradient of the window of the temporal feature from a neural network layer above the FT layer; calculating the gradient of the error function of the neural network, wherein the error function depends on the product of the FT layer parameters and the received backpropagation gradient; and updating the FT layer parameters using the calculated gradient.

[0021] In some implementations, the window on which the time-domain features are processed by the frequency transform (FT) layer depends on the number of received time-domain features.

[0022] In some cases, the FT layer is a fully connected neural network layer.

[0023] In some implementations, the neural network is configured to receive temporal features of audio samples and process the temporal features to generate a neural network output that classifies the audio samples, and the training data includes the temporal features of each of a plurality of training audio samples and the known classification of the training audio samples.

[0024] The subject matter described herein may be implemented in certain embodiments to achieve one or more of the following advantages.

[0025] Audio processing tasks such as speech recognition, content recognition, music recognition, siren detection, and other classification problems often utilize frequency domain features as input. Common algorithms for converting time-domain features to frequency-domain features produce fixed-scale frequency domain features, such as linear-scale frequency domain features or other predefined-scale frequency domain features, which can be provided as input to audio classification systems such as neural network systems.

[0026] However, important information may be lost during the fixed-scale transformation process, and providing fixed-scale frequency domain features as input to an audio classification system may degrade the system's accuracy and performance. Furthermore, hard-coded fixed-scale transformations from time-domain to frequency-domain features may not provide the optimal mapping from the time domain to the frequency domain for a particular task. For example, a linear-scale frequency mapping may be optimal for some recognition tasks, while a logarithmic scale or major musical scale may be optimal for others. As another example, some recognition tasks benefit from a more detailed frequency scale in the mid-frequency spectrum and less detail in the low- and high-frequency spectra.

[0027] The adaptive time-to-frequency conversion neural network system described in this specification learns the optimal mapping between time-domain and frequency-domain features for a given recognition task. Specifically, the adaptive time-to-frequency conversion neural network system learns different frequency mappings for different recognition tasks and different input signal characteristics. Therefore, compared to systems that apply fixed-scale frequency mappings to time-domain features and provide fixed-scale frequency-domain features to the neural network for processing, the neural network system implementing the adaptive time-to-frequency conversion described in this specification can achieve a higher level of accuracy when classifying raw input samples.

[0028] Furthermore, the adaptive time-to-frequency conversion neural network system described in this specification can use techniques based on the Discrete Fourier Transform (DFT) to learn the optimal mapping between time-domain and frequency-domain features. This leads to improved performance and efficiency in some cases compared to other neural network systems that do not implement adaptive time-to-frequency conversion, because although the computational complexity of the DFT may actually be higher than other conversion techniques such as the Fast Fourier Transform (FFT), it is often faster in practice due to the efficiency of matrix multiplication and the large constants associated with the FFT. Additionally, the adaptive time-to-frequency conversion neural network system described in this specification maps to k frequencies, where k is significantly smaller than n. Therefore, the system performs n*k operations, which is even more efficient.

[0029] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0030] Figure 1 This is an example of an adaptive time-domain to frequency-domain conversion neural network system.

[0031] Figure 2This is a flowchart illustrating an example process for generating the output of a neural network system that performs an adaptive time-domain to frequency-domain transformation from input.

[0032] Figure 3 This is a flowchart of an example process for training a frequency transformation neural network layer and one or more other neural network layers.

[0033] The same reference numerals and names in the various figures denote the same elements. Detailed Implementation

[0034] This specification describes a neural network system and method for learning the optimal mapping between time-domain input samples and frequency-domain output samples for a given classification or regression task.

[0035] Figure 1 An example of an adaptive time-domain to frequency-domain conversion neural network system 100 is shown. The adaptive time-domain to frequency-domain conversion neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the system, components and techniques described below can be implemented.

[0036] The adaptive time-domain to frequency-domain conversion neural network system 100 is a machine learning system that receives system input and generates system output from the system input. The neural network system 100 can be configured to receive the time-domain features of audio sample 102 and generate a corresponding output 112 based on the input.

[0037] In some implementations, the corresponding output may include a corresponding score or audio sample classification. For example, if the input to the neural network system 100 is a window representing the temporal features of an audio sample of a utterance, the output generated by the neural network system 100 may be a score for each text block in a set of text blocks, each score representing an estimated probability that the text block is a correct transcription of a utterance or utterance sequence. In some examples, the neural network system 100 may be part of a content recognition system, and the output generated by the neural network system 100 may be a score for each content block in a set of content blocks or for each segment in a set of content blocks. As another example, the neural network system 100 may be part of an emergency alarm detection system, and the output generated by the neural network system 100 may be a score classifying the input as an alarm or not as an alarm.

[0038] In other implementations, the corresponding output can be used to predict a single value or multiple values ​​(e.g., regression). For example, the neural network system output 112 can estimate the level of a certain type of noise (e.g., speech) in an audio sample input to the system. As another example, the neural network system output 112 can include a prediction of the year of a song represented by the audio sample. As yet another example, in the case of echo cancellation, the neural network system can be used to generate audio. For example, the audio sample input 102 can include two sub-inputs, such as a mixed signal and a background signal, and the corresponding neural network system output 112 can include an audio sample generated by subtracting the background signal from the mixed signal.

[0039] The adaptive time-to-frequency domain conversion neural network system 100 includes multiple neural network layers, such as a frequency transform (FT) layer 104 and a neural network layer 108. Each neural network layer in the adaptive time-to-frequency domain conversion neural network system 100 is configured to receive and process a corresponding layer input to generate a corresponding layer output. The neural network layers collectively process the neural network inputs received by the neural network system 100 to generate a corresponding neural network output for each received neural network input. (See below for reference.) Figure 2 A more detailed description of the neural network system that generates an adaptive time-domain to frequency-domain transformation from the input.

[0040] Some or all layers of the neural network system 100 are associated with a corresponding set of parameters. A neural network layer generates an output from an input based on the current values ​​of its parameters. For example, as part of generating an output from a received input, some layers may multiply the received input by a corresponding parameter matrix with the current parameter values.

[0041] At least one neural network layer in the adaptive time-to-frequency conversion neural network system 100 is a Fourier Transform (FT) layer, for example, FT layer 104. The FT layer is a neural network layer that transforms a window of received time-domain features 102 into frequency-domain features 106. For example, if the window of received time-domain features includes L time-domain features, the FT layer can transform the window of time-domain features into any number of frequency-domain features. The number of frequency-domain features can depend on the computational cost and size of the output from the layer. In some cases, the number of frequency-domain features may be less than the number of time-domain features. The number of frequency-domain features (e.g., the number of nodes in the FT layer) can be the same as or different from the number of time-domain features in the window of time-domain features. The number of time-domain features in the window of time-domain features can be selected to determine the time window of the time-domain features that the FT layer operates on. For example, a 50ms time window with an 8kHz mono input takes 400 time-domain features as input.

[0042] The Fourier Transform (FT) layer transforms the received time-domain features into frequency-domain features by applying a transform defined by the corresponding FT layer parameter set. In some implementations, a window function, such as a hanning window, can be applied to the input signal for the FT layer to avoid leakage. The transform defined by the corresponding FT layer parameter set can be represented as a transform matrix. The FT layer can use matrix multiplication to apply the transform matrix to the vector representation of the time-domain features to generate frequency-domain features. In some implementations, each entry in each row of the corresponding transform matrix includes a corresponding sampling basis function, such as a sampled sine wave like the Discrete Fourier Transform (DFT) or a sampled cosine wave like the Discrete Cosine Transform (DCT). The entries in the transform matrix can be controlled by multiple learnable parameters. For example, a first parameter of each row of the transform matrix can determine the frequency of each entry in that row, and a second parameter of each row of the transform matrix can determine the amplitude of each entry in that row. In some implementations, the number of rows in the transform matrix is ​​equal to the number of frequencies extracted by the FT layer. In some cases, the window function can be incorporated into the transform matrix defined by the corresponding FT layer parameter set by multiplying the window function by each row of the transform matrix pointwise.

[0043] The coefficients in the frequency transform matrix can include complex numbers, such as those calculated using the standard DFT formula. As a result, the frequency transform layer can output complex numbers. In this case, the system can use a norm or logarithm of the order of magnitude to generate the real-valued layer output, which can be fed into the rest of the network. In some implementations, the FT layer can include two matrices calculated by decomposing the DFT into cosine (real) and sine (imaginary) components. The two matrices can remain separate or can be combined using the order of magnitude of each complex weight. In some cases, the FT layer can apply a discrete cosine transform, which takes real numbers as input and produces real numbers as output.

[0044] In some implementations, the transform defined by the corresponding FT layer parameter set includes a DFT; for example, the transform defined by the corresponding FT layer parameter set is represented by a DFT matrix. Although for convenience... Figure 1 Only one FT layer 104 is depicted in this paper, but the adaptive time-to-frequency transformation neural network system 100 may include multiple FT layers. For example, multiple FT layers can be used on different time windows and concatenated to perform a multi-scale DFT. In some implementations, multiple FT layers may apply corresponding transformations defined by a shared set of FT layer parameters.

[0045] The FT layer (e.g., FT layer 104) can be a fully connected neural network layer. In some implementations, the FT layer (e.g., FT layer 104) may not include biases or activations, such as linear activation functions.

[0046] The adaptive time-domain to frequency-domain transformation neural network system 100 includes one or more other neural network layers 108, such as neural network layer 110. Although for convenience... Figure 1 Only one neural network layer 110 is depicted, but the neural network system 100 may include multiple neural network layers. The number of neural network layers 108 included in the neural network system 100 depends on the task that the neural network system 100 is configured to perform. Furthermore, the type of neural network layers 108 included in the neural network system 100 depends on the task that the neural network system is configured to perform. For example, for a speech recognition task, the neural network system 100 may include multiple neural network layers 108, which may include multiple Long Short-Term Memory (LSTM) neural network layers. The use of LSTM neural network layers in speech recognition tasks is described in more detail below: “Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition” Sak, Andrew Senior Beaufays, http: / / arxiv.org / abs / 1402.1128. As another example, for an instrument recognition task, the neural network system 100 may include multiple neural network layers 108, which include multiple convolutional neural network layers. The application of convolutional neural networks to the instrument recognition task is described in more detail below: “Automatic Instrument Recognition in Polyphonic Music Using Convolutional Neural Networks” Peter Li, Jiyuan Qian, Tian Wang, http: / / arxiv.org / abs / 1511.05520.

[0047] In some implementations, the neural network system 100 may include one or more recurrent neural network (RNN) layers, such as LSTM neural network layers. In these implementations, the neural network system 100 may receive as input a sequence of windows of temporal features of audio samples corresponding to a set of time steps. For example, the neural network system 100 may receive audio samples that have been divided into a sequence of multiple data frames corresponding to a set of time steps, for example, where each frame is associated with a 10-millisecond audio sample and includes the temporal features of that frame. The FT layer may process each input in the received temporal feature sequence and transform each input into a corresponding frequency domain feature output.

[0048] An adaptive time-to-frequency domain conversion neural network system 100 can be trained based on multiple training audio samples to determine the training values ​​of the parameters of the neural network layers, i.e., adjusting the parameter values ​​from their initial values ​​to the training values. For example, during training, the adaptive time-to-frequency domain conversion neural network system 100 can process features of each of the multiple training audio samples and generate a corresponding neural network output for each training audio sample. The neural network output can then be used to adjust the values ​​of the parameters of the adaptive time-to-frequency domain conversion neural network system 100, for example, adjusting the transform defined by the corresponding FT layer parameter set and adjusting the values ​​of the parameters of the neural network layer 110. In some embodiments, the adaptive time-to-frequency domain conversion neural network system 100 uses gradient descent and backpropagation neural network training techniques to adjust the parameter values. (See below for reference.) Figure 3 A more detailed description of the training adaptive time-domain to frequency-domain conversion neural network system 100.

[0049] Once the adaptive time-to-frequency conversion neural network system 100 has been trained, the neural network system can receive new neural network inputs for processing and through neural network layers to generate new neural network outputs for the inputs based on the training values ​​of the parameters of the adaptive time-to-frequency conversion neural network system 100.

[0050] Figure 2 This is a flowchart of an example process 200 for generating an output from an input using a neural network system that performs an adaptive time-domain to frequency-domain transformation. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification that includes a frequency transformation layer (e.g., Figure 1 An adaptive time-domain to frequency-domain conversion neural network system 100 with a frequency conversion layer 104 can execute process 200.

[0051] The system obtains the input time-domain features of the audio sample (step 202). The input time-domain features of the audio sample may include, for example, one or more windows of time-domain features arranged in sequence. In some embodiments, the input time-domain features include time-domain features from a single channel. In other embodiments, the input time-domain features include time-domain features from multiple channels. In this case, the system can combine the input time-domain features from multiple channels into a single channel input.

[0052] The system transforms the temporal features of the audio samples into frequency domain features using a window transformation (step 204). For example, the system can use one or more FT layers (e.g., Figure 1 The FT layer (104) is used to process the time-domain features through a window to generate the corresponding frequency-domain features. (See reference above.) Figure 1The FT layer can transform the time-domain features of an audio sample into frequency-domain features by applying a transformation defined by the corresponding FT layer parameter set to a window of time-domain features. For example, the transformation defined by the corresponding FT layer parameter set can be represented by a transformation matrix, where applying the transformation to the window of time-domain features includes multiplying the transformation matrix with the vector representation of the window of time-domain features using matrix multiplication. (See above reference.) Figure 1 The example transformation matrix is ​​described in more detail.

[0053] The system processes frequency domain features to generate neural network output (step 206). For example, the system can use one or more other neural network layers (e.g., Figure 1 A neural network layer 108 is used to process frequency domain features to generate a neural network output. The generated neural network output can include a classification of the audio sample described above with reference to step 202. For example, if the audio sample represents a utterance or a sequence of utterances, the generated neural network output can include a score for each text block in the set of text blocks, each score representing an estimated probability that the text block is a correct transcription of the utterance or the sequence of utterances. In some cases, the generated neural network output can be used to predict a single value or multiple values ​​(e.g., regression).

[0054] As described above, in some implementations, the window of time-domain features may include time-domain features from multiple channels, and the system may use multiple FT layers to process the multi-channel window of time-domain features to generate multiple corresponding frequency-domain features. In this case, the system may concatenate the generated multiple corresponding frequency-domain features into a single channel input and use one or more neural network layers to process the single channel input to generate a neural network output.

[0055] As part of processing neural network inputs through neural network system layers to generate neural network outputs for those inputs, the system can execute process 200. Process 200 can be executed for neural network inputs for which the desired output (i.e., the neural network output that the system should generate for that input) is unknown. The system can also execute process 200 on inputs in a training sample set (i.e., the set of inputs for which the system knows the outputs should be predicted) to train the system (i.e., to determine training values ​​for the parameters used in the FT neural network layers and other neural network layers). Specifically, as part of a machine learning training technique for training the neural network (e.g., stochastic gradient descent backpropagation training technique), process 200 can be repeatedly executed on inputs selected from the training sample set. See below. Figure 3 A more detailed description of an example training process for an adaptive time-to-frequency conversion neural network system is provided.

[0056] Figure 3This is a flowchart of an example process 300 for training a frequency-transformation neural network layer and one or more other neural network layers. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification (e.g., Figure 1 The adaptive time-domain to frequency-domain conversion neural network system 100 can execute process 300.

[0057] The system obtains training data, which includes the temporal features of each training audio sample in the training audio sample set and the known output of the training audio samples (step 302). In some embodiments, the training data may include the temporal features of the training audio samples and the known classification of the training audio samples. The temporal features of the training audio samples may include one or more windows of temporal features. The length of the windows of the temporal features may depend on the size of the corresponding training audio sample and the number of temporal features of the corresponding training audio sample. For example, an 8kHz single-channel training audio sample with 400 temporal features may include multiple windows of temporal features with a length of 50ms. In some cases, there may be a step size, such as 25ms, between the windows of the temporal features, such that there may be semi-overlap between each window.

[0058] In some implementations, the training data may include single-channel training data input, such as mono sound. In other implementations, the training data input may include multiple multi-channel training data, such as stereo sound.

[0059] The system trains the neural network on the training data to adjust the values ​​of parameters in other neural network layers, and adjusts the values ​​of the FT layer parameters to determine the training parameter set (step 304). For example, as referenced above. Figure 1 In some embodiments, the FT layer parameters may be represented as a transformation matrix, wherein each entry in each row of the transformation matrix includes a corresponding sampling basis function (e.g., a sampled sine or cosine wave) specified by a first parameter and a second parameter, the first parameter determining the frequency of each entry in that row, and the second parameter determining the amplitude of each entry in that row. In such an embodiment, training the neural network on training data may include adjusting the values ​​of the first and second parameters to determine training values ​​for the first and second parameters of each row of the transformation matrix.

[0060] The system trains the neural network by processing training data as input to generate the neural network output, as shown in the reference above. Figure 1 As described above. For example, the system uses one or more FT layers (e.g., Figure 1The FT layer 104 processes the training data input to generate the training data output of the one or more FT layers. Once the training data output of the one or more FT layers is generated, the system can provide training data for the one or more FT layers as training data input for the neural network layers above the one or more FT layers for further processing.

[0061] As described above with reference to step 304, in some embodiments, the system obtains training data comprising multiple multi-channel training data. In some cases, the system may combine the multi-channel training data into a single training data input and train the neural network on that single training data input. Alternatively, in other cases, the neural network may include multiple Fourier Transform (FT) layers, each FT layer being configured to apply a transform defined by a shared or corresponding set of FT layer parameters, which transforms the time-domain features into frequency-domain features via a window transformation, as described above with reference to... Figure 1 As described. The system can then train the neural network by concatenating the frequency domain features generated by each of the multiple FT layers to perform a multi-timescale DFT.

[0062] In some implementations, training a neural network system on training data includes: for each window of a temporal feature and for each Fourier transform (FT) layer, receiving the backpropagation gradient of that window of the temporal feature from a neural network layer above the FT layer. For example, the backpropagation gradient can be generated by: computing the gradient of the top layer in the neural network, and then using backpropagation techniques to backpropagate the computed gradients through the neural network layers.

[0063] Training a neural network system on training data may also include, for each window of the temporal features and for each FT layer, computing the gradient of the neural network's error function with respect to the current values ​​of the FT layer parameters, and updating the FT layer parameters using the computed gradient. For example, the system can update the values ​​of a vector that fully specifies the FT layer parameters using machine learning training techniques, such as by summing the gradient and the vector, or by multiplying the gradient by the learning rate and then adding the product to the vector. In some implementations, during training, the fundamental parameters of the FT matrix associated with the FT layer are modified such that the fundamental frequency and amplitude parameters are adjusted during backpropagation. The matrix weights are then updated based on the new frequency / amplitude parameters.

[0064] The trained set of FT layer parameters defines the optimal transformation from time-domain features to frequency-domain features. The determined optimal transformation depends on the task performed by the adaptive time-to-frequency transformation neural network system, as specified by the training data described above with reference to step 302. For example, the determined optimal transformation for a speaker recognition task may differ from the determined optimal transformation for a speech recognition task. Similarly, the determined optimal transformation depends on the characteristics of the input signal. For example, the determined optimal transformation for an input audio sample from a specific microphone type may differ from the determined optimal transformation for the same audio sample from different microphone types.

[0065] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly contained computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by or control of the operation of a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiving device for execution by the data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination thereof. However, the computer storage medium is not a propagation signal.

[0066] The term "data processing apparatus" includes all types of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0067] A computer program (which may also be referred to as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but does not need to, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple co-files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located in one place or distributed across multiple locations.

[0068] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs distinct from its inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet, laptop, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device that includes one or more processors and computer-readable media. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0069] The processes and logic flows described in this specification can be executed by one or more programmable processors, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)), and the apparatus can also be implemented as special-purpose logic circuitry.

[0070] Computers suitable for executing computer programs include, for example, those based on general-purpose and special-purpose microprocessors or both, and any kind of central processing unit (CPU). Typically, the CPU receives instructions and data from read-only memory or random access memory or both. Essential components of a computer are the CPU for executing or running instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer may be operatively coupled to such one or more mass storage devices to receive data from or transfer data to or both. However, a computer does not necessarily need to have such a device. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device (e.g., a Universal Serial Bus (USB) flash drive), to name just a few.

[0071] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as: semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0072] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and, for example, a mouse or trackball, a pointing device through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a webpage in response to a request received from a webpage on the user's client device.

[0073] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components such as data servers, middleware components such as application servers, or front-end components such as client computers with graphical user interfaces or web browsers through which users can interact with the implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”) (e.g., the Internet).

[0074] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the user device as a client and receiving user input from that user. Data generated at the user device, for example, as a result of user interaction, may be received at the server from the user device.

[0075] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or claim, but rather as descriptions of features specific to particular embodiments of a particular invention. Some features described in this specification in the context of discrete embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented discretely or in any suitable sub-combination in multiple embodiments. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from the combination in some cases.

[0076] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order, or that all shown operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0077] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, executed on data processing hardware, causing the data processing hardware to perform operations, the operations including: The temporal features of the received input audio samples are used as input to a trained frequency transform (FT) layer of the neural network, the input audio samples including speech signals representing utterance and background signals; The trained FT layer applies a transformation, defined by a set of trained FT layer parameters, to transform the temporal features of the input audio sample into frequency domain features. The trained FT layer is trained to learn the mapping between temporal and frequency domain features for a specific audio processing task. The frequency domain features transformed by the transform applied by the trained FT layer are processed using multiple convolutional layers of the neural network. Based on the processed frequency domain features, an output audio sample is generated as the output from the neural network by subtracting the background signal from the speech signal of the input audio sample. as well as The neural network is trained on training data for each of a plurality of training audio samples, the training data including the corresponding temporal features of the training audio samples and the corresponding known outputs of the training audio samples, wherein training the neural network includes updating the adjustable parameters of the trained FT layer.

2. The computer-implemented method according to claim 1, wherein: Applying the transformation includes applying the transformation defined by the set of trained FT layer parameters to window the temporal features of the input audio sample into the frequency domain features; and The window for the time-domain features depends on the number of time-domain features received.

3. The computer-implemented method according to claim 2, wherein, The transformation, defined by the set of trained FT layer parameters, which transforms the time-domain features of the input audio sample into the frequency-domain features, is represented by a Discrete Fourier Transform (DFT) matrix.

4. The computer-implemented method according to claim 1, wherein: The set of trained FT layer parameters includes a transformation matrix; and Each row of the transformation matrix contains entries for the corresponding sampling basis function.

5. The computer-implemented method according to claim 4, wherein, The first parameter of each row of the transformation matrix determines the frequency of each entry in that row, and the second parameter of each row of the transformation matrix determines the magnitude of each entry in that row.

6. The computer-implemented method according to claim 1, wherein: The set of trained FT layer parameters includes a transformation matrix; and The transformation matrix includes a window function.

7. The computer-implemented method according to claim 1, wherein, The trained FT layer includes a fully connected neural network layer.

8. The computer-implemented method according to claim 1, wherein, Training the neural network on the training data includes each window for the temporal features and the trained FT layer: The backpropagation gradient of the window, which receives temporal features, is received from the neural network layer above the trained FT layer; Calculate the gradient of the error function of the neural network, wherein the error function depends on the product of the trained FT layer parameters and the received backpropagation gradient; and The calculated gradients are used to update the parameters of the trained FT layer.

9. The computer-implemented method according to claim 1, wherein: The neural network includes multiple trained FT layers, each of which is configured to apply a window transformation of time-domain features to frequency-domain features, defined by a set of parameters of the corresponding trained FT layer. as well as Training the neural network includes connecting frequency domain features generated from each of the plurality of trained FT layers to perform a multi-timescale DFT.

10. A system comprising: Data processing hardware; as well as Memory hardware that communicates with and stores instructions, the instructions causing the data processing hardware to perform operations when executed by the data processing hardware, the operations including: The temporal features of the received input audio samples are used as input to a trained frequency transform (FT) layer of the neural network, the input audio samples including speech signals representing utterance and background signals; The trained FT layer applies a transformation, defined by a set of trained FT layer parameters, to transform the temporal features of the input audio sample into frequency domain features. The trained FT layer is trained to learn the mapping between temporal and frequency domain features for a specific audio processing task. The frequency domain features transformed by the transform applied by the trained FT layer are processed using multiple convolutional layers of the neural network. Based on the processed frequency domain features, an output audio sample is generated as the output from the neural network by subtracting the background signal from the speech signal of the input audio sample; and The neural network is trained on training data for each of a plurality of training audio samples, the training data including the corresponding temporal features of the training audio samples and the corresponding known outputs of the training audio samples, wherein training the neural network includes updating the adjustable parameters of the trained FT layer.

11. The system according to claim 10, wherein: Applying the transformation includes applying the transformation defined by the set of trained FT layer parameters to window the temporal features of the input audio sample into the frequency domain features; and The window for the time-domain features depends on the number of time-domain features received.

12. The system according to claim 11, wherein, The transformation, defined by the set of trained FT layer parameters, which transforms the time-domain features of the input audio sample into the frequency-domain features, is represented by a Discrete Fourier Transform (DFT) matrix.

13. The system according to claim 10, wherein: The set of trained FT layer parameters includes a transformation matrix; and Each row of the transformation matrix contains entries for the corresponding sampling basis function.

14. The system according to claim 13, wherein, The first parameter of each row determines the frequency of each entry in that row, and the second parameter of each row determines the magnitude of each entry in that row.

15. The system according to claim 10, wherein: The set of trained FT layer parameters includes a transformation matrix; and The transformation matrix includes a window function.

16. The system according to claim 10, wherein, The trained FT layer includes a fully connected neural network layer.

17. The system according to claim 10, wherein, Training the neural network on the training data includes each window for the temporal features and the trained FT layer: The backpropagation gradient of the window, which receives temporal features, is received from the neural network layer above the trained FT layer; Calculate the gradient of the error function of the neural network, wherein the error function depends on the product of the trained FT layer parameters and the received backpropagation gradient; and The calculated gradients are used to update the parameters of the trained FT layer.

18. The system according to claim 10, wherein: The neural network includes multiple trained FT layers, each of which is configured to apply a window transformation of time-domain features to frequency-domain features, defined by a set of parameters of the corresponding trained FT layer. as well as Training the neural network includes connecting frequency domain features generated from each of the plurality of trained FT layers to perform a multi-timescale DFT.

Images