System and method for processing audio data with multi-rate learnable audio front end

Abstract

Systems and methods for processing audio data with a multi-rate learnable audio front end. Methods and systems for processing audio data with a multi-level audio front-end model are provided. A one-dimensional audio waveform is received as input and processed using a multi-stage audio front-end model to convert the one-dimensional waveform into a two-dimensional matrix representing features of the audio waveform. The multi-level learnable audio front-end model is configured to apply a first filter bank to the audio waveform to generate a first time-frequency representation of the audio waveform; applying a first decimation filter to the audio waveform to generate a first decimated audio input; applying a second filter bank to the first decimated audio input to generate a second time-frequency representation of the audio waveform; and stacking the first time-frequency representation and the second time-frequency representation together to generate a two-dimensional matrix.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for processing audio data with a multi-rate learnable audio front end. Background Art

[0002] With the success of Convolutional Neural Networks (CNNs) in the field of computer vision, recent years have seen a lot of increasing adoption of audio-based CNNs on a variety of tasks. Pioneering research in the field of automatic speech recognition paved the way for the adoption of deep neural networks (DNNs) on a variety of audio tasks. Sound event detection and environmental sound classification with and without visual cues have dominated recent scientific publications, also thanks to the widespread success of the DCASE challenge. Recently released datasets on industrial machine fault detection and monitoring via audio recordings are driving a new wave of audio DNN architectures tailored for industrial use cases.

[0003] Audio DNN architectures are typically constructed as a sequence of three components: a front-end, an encoder, and a head. Raw audio samples are fed into the front-end to generate a time-frequency representation, converted into an embedding by the encoder, and finally fed into the head to produce the desired output. Summary of the invention

[0004] According to one embodiment, a method for processing audio data with a multi-stage audio front-end model is provided. The method includes receiving a one-dimensional audio waveform as input. The method includes processing the audio waveform using a multi-stage audio front-end model to convert the one-dimensional waveform into a two-dimensional matrix representing features of the audio waveform, wherein the multi-stage learnable model is configured to perform the following operations: applying a first filter group to the audio waveform to generate a first time-frequency representation of the audio waveform; applying a first extraction filter to the audio waveform to generate a first extracted audio input; applying a second filter group to the first extracted audio input to generate a second time-frequency representation of the audio waveform; and stacking the first time-frequency representation and the second time-frequency representation together to generate a two-dimensional matrix. The method also includes processing the two dimensions using an audio understanding machine learning model having multiple audio understanding parameters to generate a corresponding output for each of one or more audio understanding tasks.

[0005] In another embodiment, an audio processing system includes a processor and a memory having instructions. When executed by the processor, the memory causes the processor to perform the following operations: receiving a one-dimensional audio waveform; processing the one-dimensional waveform via a multi-stage learnable audio front-end model to convert the one-dimensional audio waveform into a two-dimensional matrix representing features of the audio waveform; and processing the two-dimensional matrix using an audio understanding machine learning model with multiple audio understanding parameters to generate a corresponding output for each of one or more audio understanding tasks. The multi-stage learnable audio front-end model is configured to: apply a first filter group to an audio waveform to generate a first time-frequency representation of the audio waveform; apply a first extraction filter to the audio waveform to generate a first extracted audio input; apply a second filter group to the first extracted audio input to generate a second time-frequency representation of the audio waveform; and stack the first time-frequency representation and the second time-frequency representation together to generate a two-dimensional matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A schematic example of an audio processing system according to an embodiment is illustrated.

[0007] Figure 2 A schematic example of an architecture of a learnable audio front end according to an embodiment is illustrated.

[0008] Figure 3A Illustrated are the impulse response (left) and frequency response magnitude (right) of a sinc low pass filter with different low pass frequencies according to an embodiment.

[0009] Figure 3B Illustrated are the real component of the impulse response (left) and the frequency response magnitude (right) of a Gabor filter according to an embodiment.

[0010] Figure 3C Illustrated are the real component of the impulse response (left) and the frequency response magnitude (right) of a Gabor filter according to another embodiment.

[0011] Figure 4 The diagram illustrates an algorithm executed by a processor for applying a multi-stage learnable audio front-end model according to an embodiment, specifically how to allocate desired initialization frequencies and associated bandwidths to different stages of a multi-rate learnable audio front-end, and how to calculate the decimation rate for each stage.

[0012] Figure 5 A flowchart of a method for processing audio data using a multi-stage learnable audio front-end model according to an embodiment is illustrated. DETAILED DESCRIPTION

[0013] Embodiments of the present disclosure are described herein. However, it is to be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms. The drawings are not necessarily to scale; some features may be exaggerated or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein are not to be interpreted as restrictive, but merely as a representative basis for teaching those skilled in the art to use the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any of the accompanying drawings may be combined with the features illustrated in one or more other drawings to produce embodiments that are not explicitly described or described. The combination of the illustrated features provides representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired.

[0014] Unless the context clearly dictates otherwise, as used herein, "a", "an", and "the" refer to both singular and plural references. As an example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor programmed together to perform each of the various functions.

[0015] The present disclosure relates to one-dimensional (1D) and two-dimensional (2D) representations of sound. Although the representation of an audio waveform involves two axes (time and amplitude), it can be referred to as a 1D signal due to the way it is typically processed and analyzed. This is because each point on the waveform corresponds to a specific moment in time, and the value at that point represents the amplitude of the sound at that moment in time. When audio data is stored or manipulated in digital form, it can be arranged as a sequence of single values ​​(amplitude) that vary over time, which is why it can be referred to as one-dimensional.

[0016] With the success of convolutional neural networks (CNNs) in the field of computer vision, recent years have seen a lot of increasing adoption of audio-based CNNs on a variety of tasks. Pioneering research in the field of automatic speech recognition paved the way for the adoption of deep neural networks (DNNs) on a variety of audio tasks. Sound event detection and environmental sound classification with and without visual cues dominate recent scientific publications, also thanks to the widespread success of the DCASE challenge. Recently released datasets on industrial machine fault detection and monitoring via audio recordings are driving a new wave of audio DNN architectures tailored for industrial use cases. Audio-based anomaly detection with convolutional neural networks or auto-regressive models are just examples of how audio CNNs are approaching the needs of the industry.

[0017] While researchers are exploring the benefits of self-supervised, multimodal learning for audio-visual datasets, the input signals for audio and images remain fundamentally different. At a distance of a few hundred pixels, images contain all the information needed for object or face recognition. This does not directly apply to raw audio signals, which are long sequences of thousands of samples, where important clues about the content are often distributed across the entire sequence. To overcome this difference and quickly adopt vision-based architectures for audio use cases, it is common to first compute a time-frequency representation of the audio signal using some filter banks. While most of the layers adopted for audio DNN architectures are therefore vision-based, with small-sized 2D convolutional, pooling, and normalization layers being widely used, recent developments in speaker recognition and sound classification have shown that learnable filter banks (i.e., 1D convolutional layers with kernel sizes on the order of hundreds of samples) can be very beneficial.

[0018] Based on these recent works, the present disclosure turns to deploying audio-based CNNs with learnable front ends on vector processors and hardware accelerators. Since the current literature mainly focuses on the optimization of quantization strategies for vision models, the large 1D convolution kernels required by learnable filter banks cannot be directly deployed on current platforms. While most toolchains for embedded DNN conversion and deployment work well for 2D convolution filters up to 11×11, the use of large 1D kernels with 300 coefficients or more is either not supported or suffers from large losses in accuracy. The present disclosure proposes a design method for multi-rate learnable filter banks for audio-based DNNs, which is subject to kernel size constraints derived from the selected hardware platform, which paves the way for the efficient implementation of modern learnable audio front ends on existing hardware accelerators.

[0019] Audio DNN architectures are typically constructed as a sequence of three components: a front end, an encoder, and a head. Raw audio samples are fed to the front end to generate a time-frequency representation, converted to an embedding by the encoder, and finally fed to the head to produce the desired output. The first wave of deep audio architectures was based on a fixed, non-learnable front end, typically a log-Mel spectrogram. Recent work has shown how learnable front ends can improve accuracy and be more suitable for a wide variety of tasks. The first level of the learnable front end is a 1D convolutional filter bank whose coefficients are learned via back propagation. When it comes to deploying audio DNNs on embedded platforms, the front end can be implemented with custom code to run on the CPU, or it can be integrated as a sequence of one or more network layers to run on a dedicated vector processor (e.g., a neural engine). Although a discussion of the pros and cons of these two options is outside the scope of this disclosure, this disclosure focuses primarily on the latter approach. The basic principle is that in order to simplify the development and integration of updated DNN models with learnable front ends, it is usually desirable to feed raw audio samples from a microphone to a vector processor, and then have the front end, encoder, and head compute on the neural engine.

[0020] When referring to the design of a 1D convolutional filter bank, in an embodiment, the main hyperparameter to be set is the kernel size K. Given a sampling rate f s , the lowest frequency component that can be represented by a filter of length K is f s / K, so K is usually chosen to be large enough to accommodate the lowest frequency of interest to be represented. In practice, with a sampling rate of 16kHz and a lowest frequency to be represented of 40Hz, K will be greater than 400.

[0021] A drawback of running the audio front end on the Neural Engine is that current vector processors do not handle large kernel sizes K very well. Vector processors and associated toolchains designed primarily for image processing are tailored to work at most on kernel sizes of 11×11, resulting in a maximum K = 121. When allowed to work with larger kernels, large model transfer losses may be experienced, resulting in significant differences between the output produced by a conventional CPU and a vector processor for the same input signal.

[0022] Therefore, according to various embodiments herein, an audio processing system is disclosed having a multi-rate learnable audio front end subject to kernel size constraints. The system overcomes the above limitations and allows the audio front end to perform correctly on a variety of vector processing. The system also retains the representation capabilities of the classical filter bank while reducing conversion errors.

[0023] Embodiments of the present disclosure apply CNNs to process audio signals for spectrogram representation. In these embodiments, audio signals are converted to spectrograms, which are two-dimensional matrix representations that show how the frequency content of audio changes over time. Each spectrogram can essentially be a visual (image) representation of sound, where the x-axis represents time, the y-axis represents frequency, and the color / intensity represents the amplitude of the frequency component. CNNs can then be applied to these spectrogram images.

[0024] The heart of a CNN lies in its convolutional layers. In the context of audio, these layers slide small filters (kernels) across a spectrogram to detect different patterns at different scales. Low-level filters might detect simple features, such as edges or corners, while higher-level filters can capture more complex patterns, such as textures or motifs.

[0025] There are many different ways to construct a spectrogram, which can lead to different results. The standard way is fixed, where the way to convert a 1D raw audio input into a 2D matrix is ​​set and fixed. Another way of doing the conversion is a "learned" conversion, where the system learns about the data it processes. Here, the system learns the filters or kernels that should be used to convert the 1D audio input into a 2D matrix or spectrogram. Each row of the 2D representation represents the response to the filter being used.

[0026] Figure 1 An example audio processing system 100 is shown in accordance with an embodiment of the present disclosure. The audio processing system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0027] The audio processing system 100 obtains an audio waveform 102 (also represented by y (0) As input). The audio waveform 102 may be a sequence of audio samples (e.g., amplitude values) at a first frequency (“sampling frequency”). The audio processing system 100 processes the audio waveform 102 using a learnable or learned audio front-end model 110 to generate a feature representation 112 of the audio waveform, such as a 2D matrix or spectrogram.

[0028] In an embodiment, the audio front-end model 110 is a machine learning model configured to apply a learned filtering operation having a plurality of filtering parameters, a learned pooling operation having a plurality of pooling parameters, and a learned normalization operation having a plurality of normalization parameters to generate a feature representation 112 of the audio waveform 102. Figure 2-4The operations performed by the audio front-end model 110 to generate the feature representation 112 are described in more detail. The system 100 uses an audio understanding machine learning model 120 to process the feature representation 112. The audio understanding machine learning model 120 is a machine learning model having parameters ("audio understanding parameters") that are configured to process the feature representation 112 to generate a corresponding output 122 for each of one or more audio understanding tasks, for example, for one or more of the tasks described above or for a different task that requires predictions about the content of the audio waveform 102.

[0029] The audio understanding model 120 can be any suitable model, for example, a model previously configured to receive a Mel filter bank representation of an audio signal as input. That is, as a specific example, the audio front-end model 110 can replace a system in an audio processing pipeline that maps an audio waveform to a Mel filter bank representation of an audio waveform, which is provided as an input to the audio understanding model 120.

[0030] Specific examples of audio understanding models that can receive feature representations 112 as input include convolutional neural networks (such as those having the EfficientNet architecture), fully connected neural networks (such as a multi-task neural network with a corresponding set of linear layers for each of multiple tasks), recurrent neural networks (such as neural networks based on long short-term memory (LSTM) or gated recurrent units (GRU)), or self-attention neural networks (such as Transformer neural networks).

[0031] The audio front-end model 110 is referred to as a "learned" or "learnable" audio front-end model because the values ​​of the parameters of the learned audio front-end model (i.e., the values ​​of the filtering, pooling, and normalization parameters) are (or can be) learned end-to-end with the audio understanding model 120. In other words, the operations performed by the audio front-end model 110 are fully differentiable, allowing the audio front-end model 110 to be jointly trained with the "back-end" model via gradient descent. This is different from other prominent representations (e.g., Mel filter bank representations), which are hard-coded and therefore cannot be fine-tuned to improve the performance of a given model 120 on a given set of one or more audio processing tasks.

[0032] In particular, the system 100 includes a training engine 150 that trains an audio front-end model 110 and an audio understanding model 120 on corresponding training data for each of one or more audio understanding tasks. The training data for a given task includes a set of training audio inputs and, for each audio input, a target output for the given task that should be generated by the audio understanding model 120 by processing the feature representation of the training audio input generated by the audio front-end model 110. In an embodiment, the training engine 150 trains the models 110 and 120 on the training data for the given task by gradient descent, and in particular, trains the audio front-end model 110 by back-propagating the gradient of the loss function for the given task through the audio understanding model 120 into the audio front-end model 110. The loss function can be any appropriate loss function for the task that measures the performance of the audio understanding model 120 on the given task given the target output in the training data for the given task, for example, if the audio understanding task is a classification task, the loss function is a classification loss, or if the audio understanding task is a regression task, the loss function is a regression loss.

[0033] In some cases, the training engine 150 may pre-train the audio front-end model 110 on one or more tasks in conjunction with the original back-end model, and then train the audio front-end model 110 on a different task using the new back-end model, e.g., while keeping the pre-trained values ​​of the parameters of the model 110 fixed and updating the parameters of the new back-end model, or updating the parameters of the new back-end model while also fine-tuning the pre-trained values ​​of the parameters of the model 110. This may be done, for example, where a large amount of training data is available for one or more tasks performed by the original back-end model, but only a limited amount of training data is available for the task(s) performed by the new back-end model.

[0034] Figure 2 FIG. 1 illustrates an example of an architecture of a learnable audio front end 110 according to an embodiment. As shown, the learnable audio front end model 110 is a multi-stage learnable audio front end model. With the illustrated multi-stage approach, the front end model receives the original audio input y (0) (which may originate from a microphone) as input, and a small set of filters are applied, including a first filter bank h (0) and the first decimation layer or filter d (0) The first filter bank h applied to the original audio input (0) Generate a first time-frequency representation f (0) The first decimation filter d (0) Generate the first extracted original audio input y (1)(which can be used in subsequent processing stages), thus being a multi-stage approach. In particular, the multi-stage learnable audio front-end model 110 can apply another small set of filters, including a second filter group h (1) and the second decimation filter d (1) The second filter bank h (1) Generate a second time-frequency representation f (1) , and the second decimation filter d (1) Generate the second extracted original audio input y (2) The second extracts the original audio input y (2) Can be subjected to the third filter bank h (2) is processed to produce a third time-frequency representation f (2) .

[0035] The first and second time-frequency representations f (0) and f (1) Having a third time-frequency representation f (2) higher time resolution, so they pass through the decimation filter d (0) and d (1) Decimation is performed along the time axis to obtain three time-coherent representations f (0)2 、f (1)2 and f (2) These three time-coherent representations are stacked together in the output time-frequency matrix or spectrogram f.

[0036] In short, the original audio input y at stage l (l) is fed to the filter bank h (l) To obtain the time-frequency representation f (l) , and is fed to the decimation filter d (l) To obtain y (l+1) , i.e. the extracted original audio input of stage l+1.

[0037] Filter Bank h (l) and the decimation filter d (l) is learnable and is constrained to a fixed functional form.

[0038] In an embodiment, K represents the maximum filter length as constrained by the selected hardware accelerator, and Represents the sample index of a discrete filter with K coefficients. The decimation filter d (l) is a truncated sinc as defined in the following formula (1), which is a learnable low-pass filter with a desired bandwidth ω lp ∈[0,0.5] has an almost flat frequency response. Given the desired decimation rate r, according to the Nyquist theorem, ω lp <1 / (2r). Figure 3ATwo simple sinc filters are depicted in .

[0039]

[0040] In an embodiment, a set of Gabor filters is used for h (l) , so that each filter is fully described by only two learnable parameters, namely the center frequency and the bandwidth. The impulse response of a Gabor filter centered at the normalized frequency ω∈[0,0.5] with a standard deviation σ is defined in the following formula (2), where j represents an imaginary unit.

[0041]

[0042] The corresponding amplitude in the frequency domain is defined in the following formula (3), where k∈[-0.5,0.5] is the normalized frequency.

[0043]

[0044] Given K, two constraints can be derived that will drive the design of the multirate filter bank: how many proficiencies the system can tolerate, and how much noise can be allowed in the filter. Regarding the former, the range of frequencies that can be represented by the filter length K (excluding ω = 0) is constrained to be between 1 / K and 0.5 (Nyquist). Figure 3B The impulse and frequency responses of two Gabor filters designed with K = 31 are shown for the lower and upper frequencies ω, respectively. lo =1 / K and ω up =0.5.

[0045] Regarding the second constraint, that on σ, this is necessary to control the amount of maximum tolerable ripple (i.e. the amplitude of the sidelobes in the frequency domain). Due to the finite length of the filter, a filter with a small bandwidth results in a truncated impulse response, resulting in stronger sidelobes. The lower limit of σ, expressed in terms of normalized frequency, is defined as σ min The specific value depends on the application, and in the embodiments, Provides a good trade-off between sidelobe attenuation and filter bank depth. Figure 3C It is shown how the impulse and frequency responses change for varying values ​​of σ.

[0046] Prior knowledge about a particular task may indicate that a certain set of center frequencies and associated bandwidths F = {(f i ,b i ),i∈[1,M]} will be a good initialization for the filter bank. The Mel scale can be used to determine F, and in the following it can be assumed that f i and b iExpressed in Hz. Given the sampling rate f of level l s (l) The above two constraints determine which (f i ,b i )right.

[0047] In fact, the decimation rate r between levels l and l+1 (l) There is also an upper limit, as defined in the following formula (4), which is directly derived from the Nyquist theorem. in (l) Defined as the set of frequencies and bandwidths that have not been allocated at the beginning of level l, then r (l) The value of is chosen to be an integer subject to equation (4), which allows the frequency-bandwidth pair (f i ,b i ) is maximized.

[0048]

[0049] Assume the native sample rate is f s (0) , also expressed in Hz, as is necessary to satisfy the Nyquist criterion for the desired frequency and hardware constraints, Figure 4 Algorithm 1 shown in FIG. 1 details how to convert the desired initialization frequency and the associated bandwidth F (j) Assigning to multi-rate allows learning the different stages of the audio front end and how to calculate the decimation rate r for each stage (j) The entry point of the algorithm is GETMULTISTAGEFB, which gives the filter length K, the set of initialization frequencies and bandwidths F, and the native sampling rate f. s (0) and minimum filter bandwidth σ min , constructs L stages of the filter bank and returns the frequency-bandwidth pair (F (j) ) and the decimation factor r (j) At a given level (f s ) and the minimum bandwidth (σ min ), the auxiliary function GETSTAGEIDXS determines the frequency-bandwidth pairs (F in GETDECIMATIONRATE calculates the decimation rate (r) that allows most frequencies to be represented in the following stage.

[0050] In this paper, we disclose a novel multi-rate learnable front-end model for audio-based neural networks. The proposed front-end builds on other learnable front-ends to exploit the benefits of learnable filter banks in audio applications while considering practical limitations of embedded development of audio DNNs. Experimental evaluation shows that the performance of the multi-rate front-end is comparable to that of an unconstrained learnable front-end while still outperforming the classical fixed front-end.

[0051] The present disclosure provides a type of learnable front-end audio processing system. Learnable filter banks are implemented. This can be difficult to implement (and may be the reason why it has not been done before) because they require a large number of coefficients that cannot be used in embedded applications. Therefore, the present disclosure proposes a multi-stage approach. Figure 5 A method 500 for processing audio data using a multi-stage learnable audio front-end model such as disclosed above is illustrated. The method may be performed by one or more of the processors and / or computing systems disclosed herein. For example, the method may be implemented by the audio processing system 100 and its multi-stage learnable audio front-end model 110.

[0052] At 502, the audio processing system 100 receives a one-dimensional audio waveform y (0) As input. This may be raw audio input such as generated by a microphone, for example. The audio input may be event specific and last for a specific period of time, such as in the range of 10-20 seconds. At 504, the audio processing system 100 processes the one-dimensional audio waveform y using a multi-stage learnable audio front-end model (e.g., model 110). (0) , to convert the one-dimensional audio waveform into a two-dimensional matrix or spectrogram representing the features of the audio waveform. Here, the multi-level learnable audio front-end model is configured to perform the following steps 506-516.

[0053] At 506, the model 110 applies the first filter bank h (0) Applied to the audio waveform to generate a first time-frequency representation of the audio waveform f (0) At 508, the model 110 applies the first decimation filter d (0) Applied to the audio waveform to generate the first decimated audio input y (1) At 510, the model 110 applies the second filter bank h (1) Applied to the first extracted audio input y (1) To generate a second time-frequency representation of the audio waveform f (1) At 512, the model 110 applies the second decimation filter d (1) Applied to the first extracted audio input y (1) To generate the second extracted audio input y (2) At 514, the model 110 applies the third filter bank h (2)Applied to the second extracted audio input y (2) To generate a third time-frequency representation of the audio waveform f (2) At 516, the model converts the first time-frequency representation f (0) , the second time-frequency representation f (1) and the third time-frequency representation f (2) are stacked together to produce a two-dimensional matrix f.

[0054] At 518 , the audio processing system 100 processes the two-dimensional matrix using the audio understanding machine learning model 120 having a plurality of audio understanding parameters to generate a corresponding output for each of the one or more audio understanding tasks.

[0055] It should be understood that Figure 5 The steps shown in are merely exemplary, and more or fewer iterations of the filter bank and decimation filter may actually be implemented. For example, in one embodiment, only two stages of the filter bank and decimation filter are implemented. In other words, steps 512 and 514 may be omitted, and instead only two time-frequency representations may be stacked at step 516. Of course, in other embodiments, more than one iteration of the filter bank and decimation filter may be implemented. Figure 5 More filter banks and decimation filters as shown in FIG.

[0056] Depending on the embodiment, the filters or kernels disclosed herein may be implemented (i.e., tuned to or viewed) on specific portions of the audio. (0) Can be applied to high frequency sounds (i.e. sounds above an upper threshold in Hz), h (1) can be applied to mid-frequency sounds (e.g., sounds below the upper threshold but above the lower threshold), and h (2) Can be applied to low frequency sounds (i.e. sounds below the lower threshold). Since we want the same number of samples when stacking, this is where the decimation filter d is used. (0) and d (1) The reason for this is to move the sound so that each part of the final spectrogram has the same number of samples.

[0057] The embodiments of the subject matter and functional operations described in this specification can be implemented in a digital electronic circuit system, in a tangibly implemented computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-temporary storage medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0058] The term "data processing apparatus" refers to data processing hardware and covers all kinds of apparatus, devices and machines for processing data, including by way of example a programmable processor, a computer or a plurality of processors or computers. The apparatus may also be or further include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to the hardware, the apparatus may optionally include code for creating an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0059] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0060] The processes and logic flows described in this specification can be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system (such as an FPGA or ASIC), or by a combination of a dedicated logic circuit and one or more programmed computers.

[0061] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or implementing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented or incorporated into the special logic circuit system by a special logic circuit system. Typically, the computer will also include or be operably coupled to one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk or an optical disk, to receive data from the large-capacity storage device or to transmit data or both to it. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few examples.

[0062] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including by way of example semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0063] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and having a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smart phone running a messaging application) and receiving a response message from the user in return.

[0064] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0065] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0066] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., such as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0067] A computing system may include a client and a server. The client and the server are typically remote from each other and typically interact via a communication network. The relationship between the client and the server is due to the computer programs running on the respective computers, and each has a client-server relationship with the other. In some embodiments, the server transmits data, such as an HTML page, to a user device, such as for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from the user. Data generated at the user device, such as the result of a user interaction, may be received at the server from the device.

[0068] Although exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms covered by the claims. The words used in the specification are descriptive rather than restrictive words, and it is understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of the various embodiments can be combined to form further embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations with respect to one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics may be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. Therefore, in terms of one or more characteristics, to the extent that any embodiment is described as not as desirable as other embodiments or prior art implementations, these embodiments are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method for processing audio data using a multi-level learnable audio front-end model, the method comprising: Receive a one-dimensional audio waveform y (0) As input; Use a multi-stage learnable audio front-end model to process the audio waveform y (0) , to convert the one-dimensional audio waveform into a two-dimensional matrix representing the features of the audio waveform, wherein the multi-level learnable audio front-end model is configured as: The first filter bank h (0) Applied to the audio waveform to generate a first time-frequency representation of the audio waveform f (0) ; The first decimation filter d (0) Applied to the audio waveform to generate the first decimated audio input y (1) ; The second filter bank h (1) Applied to the first extracted audio input y (1) To generate a second time-frequency representation of the audio waveform f (1) ;as well as The first time-frequency representation f (0) and the second time-frequency representation f (1) Stacked together to generate a two-dimensional matrix f; as well as The two-dimensional matrix is ​​processed using an audio understanding machine learning model having a plurality of audio understanding parameters to generate a corresponding output for each of one or more audio understanding tasks.

2. The method according to claim 1, wherein the multi-stage learnable audio front-end model is further configured to: The second decimation filter d (1) Applied to the first extracted audio input y (1) To generate the second extracted audio input y (2) ; The third filter bank h (2) Applied to the second extracted audio input y (2) To generate a third time-frequency representation of the audio waveform f (2) .

3. The method according to claim 2, wherein the multi-stage learnable audio front-end model is further configured to convert the third time-frequency representation f (2) With the first time-frequency representation f (0) and the second time-frequency representation f (1) Stacking is performed to produce a two-dimensional matrix f.

4. The method of claim 3, wherein the third time-frequency representation has a time resolution, the method further comprising: The first time-frequency representation and the second time-frequency representation are decimated to match a time resolution of a third time-frequency representation.

5. The method of claim 1 , wherein the first time-frequency representation has a first time resolution and the second time-frequency representation has a second time resolution, the method further comprising: The first time-frequency representation and the second time-frequency representation are decimated such that the first time resolution matches the second time resolution.

6. The method according to claim 1, further comprising: The first and second filter banks are determined based on an initial set of frequencies of interest, maximum allowed ripples on the frequency responses of the first and second filter banks, and an original sampling rate of the one-dimensional audio waveform. The method of claim 1 , wherein the one-dimensional audio waveform is generated by a microphone.

8. The method of claim 1, wherein a first filter bank is applied to a portion of a one-dimensional audio waveform having frequencies above a threshold, and a second filter bank is applied to a first decimated audio input waveform having frequencies below a threshold. (1) part of.

9. An audio processing system, comprising: processor; as well as A memory having instructions which, when executed by a processor, cause the processor to Receive a one-dimensional audio waveform; The one-dimensional waveform is processed via a multi-stage learnable audio front-end model to convert the one-dimensional audio waveform into a two-dimensional matrix representing features of the audio waveform, wherein the multi-stage learnable audio front-end model is configured to: The first filter bank h (0) Applied to the audio waveform to generate a first time-frequency representation of the audio waveform f (0) ; The first decimation filter d (0) Applied to the audio waveform to generate the first decimated audio input y (1) ; The second filter bank h (1) Applied to the first extracted audio input y (1) To generate a second time-frequency representation of the audio waveform f (1) ;as well as The first time-frequency representation f (0) and the second time-frequency representation f (1) Stacked together to generate a two-dimensional matrix f; as well as The two-dimensional matrix is ​​processed using an audio understanding machine learning model having a plurality of audio understanding parameters to generate a corresponding output for each of one or more audio understanding tasks.

10. The system of claim 9, wherein the multi-stage learnable audio front-end model is further configured to: The second decimation filter d (1) Applied to the first extracted audio input y (1) To generate the second extracted audio input y (2) ; The third filter bank h (2) Applied to the second extracted audio input y (2) To generate a third time-frequency representation of the audio waveform f (2) .

11. The system of claim 10, wherein the multi-stage learnable audio front-end model is further configured to convert the third time-frequency representation f (2) With the first time-frequency representation f (0) and the second time-frequency representation f (1) Stacking is performed to produce a two-dimensional matrix f.

12. The system of claim 11, wherein the third time-frequency representation has a time resolution, and wherein the instructions further cause the processor to: The first time-frequency representation and the second time-frequency representation are decimated to match a time resolution of a third time-frequency representation.

13. The system of claim 9, wherein the first time-frequency representation has a first time resolution and the second time-frequency representation has a second time resolution, and wherein the instructions further cause the processor to: The first time-frequency representation and the second time-frequency representation are decimated such that the first time resolution matches the second time resolution.

14. The system of claim 9, wherein the instructions further cause the processor to: The first and second filter banks are determined based on an initial set of frequencies of interest, maximum allowed ripples on the frequency responses of the first and second filter banks, and an original sampling rate of the one-dimensional audio waveform.

15. The system of claim 9, wherein the one-dimensional audio waveform is generated by a microphone.

16. The system of claim 9, wherein a first filter bank is applied to a portion of a one-dimensional audio waveform having frequencies above a threshold, and a second filter bank is applied to a first decimated audio input waveform having frequencies below a threshold. (1) part of.

17. A method for processing audio data using a multi-stage learnable audio front-end model, the method comprising: Receive a one-dimensional audio waveform y (0) As input; Use a multi-stage learnable audio front-end model to process the audio waveform y (0) , to convert the one-dimensional audio waveform into a two-dimensional matrix representing the features of the audio waveform, wherein the multi-level learnable audio front-end model is configured as: The first filter bank h (0) Applied to the audio waveform to generate a first time-frequency representation of the audio waveform f (0) ; The first decimation filter d (0) Applied to the audio waveform to generate the first decimated audio input y (1) ; The second filter bank h (1) Applied to the first extracted audio input y (1) To generate a second time-frequency representation of the audio waveform f (1) ; The second decimation filter d (1) Applied to the first extracted audio input y (1) To generate the second extracted audio input y (2) ; The third filter bank h (2) Applied to the second extracted audio input y (2) To generate a third time-frequency representation of the audio waveform f (2) ;as well as The first time-frequency representation f (0) , the second time-frequency representation f (1) and the third time-frequency representation f (2) Stacked together to generate a two-dimensional matrix f; as well as The two-dimensional matrix is ​​processed using an audio understanding machine learning model having a plurality of audio understanding parameters to generate a corresponding output for each of one or more audio understanding tasks.

18. The method according to claim 17, further comprising: The first and second filter banks are determined based on an initial set of frequencies of interest, maximum allowed ripples on the frequency responses of the first and second filter banks, and an original sampling rate of the one-dimensional audio waveform.

19. The method of claim 17, wherein a first filter bank is applied to a portion of a one-dimensional audio waveform having frequencies above an upper threshold, and a second filter bank is applied to a first decimated audio input waveform having frequencies below the upper threshold. (1) part of.

20. The method of claim 17, wherein the third filter bank is applied to a portion of the one-dimensional audio waveform having frequencies below a lower threshold.

Images