Method and apparatus for processing audio features using broadcast residual learning

Abstract

Certain aspects of the present disclosure provide techniques for efficient broadcasted residual machine learning. An input tensor comprising a frequency dimension and a time dimension is received, and the input tensor is processed using a first convolution operation to generate a multi-dimensional intermediate feature map comprising the frequency dimension and the time dimension. The multi-dimensional intermediate feature map is converted to a one-dimensional intermediate feature map over the time dimension using a frequency dimension reduction operation, and the one-dimensional intermediate feature map is processed using a second convolution operation to generate a time feature map. The time feature map is expanded to the frequency dimension using a broadcast operation to generate a multi-dimensional output feature map, and the multi-dimensional output feature map is augmented using the multi-dimensional intermediate feature map via a first residual connection.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 656,621, filed March 25, 2022, and U.S. Provisional Patent Application No. 63 / 166,161, filed March 25, 2021, the entire contents of each of which are incorporated herein by reference.

[0003] introduction

[0004] This disclosure relates to various aspects of machine learning, and more specifically to efficient data processing.

[0005] Designing efficient machine learning architectures is a crucial topic in neural speech processing. Specifically, Keyword Localization (KWS), which aims to detect predefined keywords, has become increasingly important. KWS plays a key role in device wake-up and user interaction on smart devices. However, providing models that minimize errors while operating efficiently is challenging. Model efficiency is particularly important in KWS because the process is often performed on edge devices (e.g., in resource-constrained devices such as mobile phones, smart speakers, and Internet of Things (IoT) devices) while requiring low latency.

[0006] Accordingly, there is a need for systems and methods that utilize efficient model design to provide high-accuracy classification.

[0007] Brief Overview

[0008] Some aspects provide a method comprising: receiving an input tensor including a frequency dimension and a time dimension; processing the input tensor using a first convolution operation to generate a multidimensional intermediate feature map including the frequency dimension and the time dimension; using a frequency dimension reduction operation to transform the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension; processing the one-dimensional intermediate feature map using a second convolution operation to generate a time feature map; expanding the time feature map to the frequency dimension using a broadcast operation to generate a multidimensional output feature map; and using the multidimensional intermediate feature map to augment the multidimensional output feature map via a first residual connection.

[0009] Other aspects provide a processing system configured to perform the aforementioned methods and those described herein, a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the aforementioned methods and those described herein, a computer program product embodied on a computer readable storage medium comprising code for performing the aforementioned methods and those further described herein, and a processing system comprising means for performing the aforementioned methods and those further described herein.

[0010] The following description and the related drawings, in which like elements are referred to with like reference numerals, detail certain illustrative features of one or more aspects. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings depict certain aspects of the one or more aspects and are therefore not to be considered limiting in scope, so far as scope is deemed appropriate.

[0013] Figure 1 An example workflow for broadcasted residual learning is depicted.

[0014] Figure 2 An example block diagram for residual learning techniques is depicted.

[0015] Figure 3 is an example broadcasted residual learning block for use in efficiently processing input data.

[0016] Figure 4 is an example broadcasted residual learning block for use in efficiently processing input data in a transition layer.

[0017] Figure 5 is an example flow diagram illustrating a method for processing data using broadcasted residual learning.

[0018] Figure 6 An example processing system configured to perform various aspects of the disclosure is depicted.

[0019] To facilitate understanding, like reference numerals have been used, where possible, to designate identical elements common to the figures. It is contemplated that elements and features of one aspect can be beneficially incorporated into other aspects without further recitation.

[0020] DETAILED DESCRIPTION

[0021] Aspects of the disclosure provide techniques for broadcasted residual learning. The techniques described herein provide high model accuracy and significantly improved computational efficiency (e.g., small model size and light computational burden) compared to existing approaches.

[0022] A variety of efficient convolutional neural networks (CNNs) have recently been developed. Generally, CNNs are composed of repeating blocks of the same structure and tend to be based on residual learning and depthwise separable convolutions. This has led to several CNN-based approaches to KWS. Existing approaches either use one-dimensional temporal convolutions or two-dimensional (e.g., frequency and time) convolutions. Each approach has corresponding benefits and drawbacks.

[0023] For example, for models that use one-dimensional temporal convolutions, generally fewer computational resources are needed compared to models that rely on two-dimensional approaches. However, with one-dimensional convolutions, internal biases of the convolution (such as translational equivariance) cannot be obtained with respect to the frequency dimension.

[0024] On the other hand, two-dimensional convolution-based approaches require significantly more computational resources than one-dimensional approaches, even when using efficient designs and architectures such as depthwise separable convolutions. This can prevent such two-dimensional approaches from being useful for a variety of devices and implementations.

[0025] The broadcasted residual learning techniques described herein can be used to efficiently process data both during training (as training data passes through the model) and during runtime (as new data passes through to generate inferences).

[0026] In some aspects, broadcasted residual learning is used to process and classify audio data and features (e.g., to perform KWS). Generally, audio data and features can be represented using two-dimensional tensors (e.g., with a frequency dimension and a time dimension). Although audio is used in examples herein, aspects of the disclosure can be readily applied to a variety of data.

[0027] In some aspects, broadcasted residual learning generally involves performing a convolution on an input tensor to extract two-dimensional features, reducing the dimensionality of the two-dimensional features to allow efficient convolution of the features (e.g., requiring reduced computation, processing steps, and energy), expanding the resulting tensor to the original dimensionality of the two-dimensional features, and augmenting the expanded tensor using the original two-dimensional features. In some aspects, the expanded tensor is further augmented using the original input tensor.

[0028] In some aspects, the broadcasted residual learning described herein can be performed in a neural network architecture to perform a variety of tasks, such as classifying input audio. For example, the techniques described herein can be implemented as broadcasted residual learning blocks, and several of these blocks can be used in a sequence within a neural network architecture.

[0029] Advantageously, broadcasted residual learning preserves many of the residual functions of one-dimensional temporal convolutions while still allowing two-dimensional convolutions to be used together via broadcasted residual connections that expand the temporal outputs to the frequency dimension. This residual mapping enables the network to efficiently represent useful audio features using much less computation than a regular convolutional neural network, which reduces computational complexity, latency, computational requirements, memory requirements, and so on. In aspects, the broadcasted residual learning techniques described herein can use less computation and parameters to achieve state-of-the-art accuracy on speech command datasets compared to regular systems.

[0030] Example workflow for broadcasted residual learning

[0031] Figure 1 An example workflow 100 for broadcasted residual learning is depicted. Workflow 100 begins with an input tensor 105. In some examples, tensor 105 can be audio data (e.g., represented by a log Mel spectrogram indicating a frequency spectrum over time), or audio features (e.g., features generated by processing audio data). In aspects, input tensor 105 is a two-dimensional tensor having a frequency dimension and a time dimension. The time dimension can be depicted as time bins or steps, while the frequency dimension is depicted based on frequency values or frequency bands. The frequencies present at each bin (e.g., the amplitude of sound at each frequency) can be reflected via values in the tensor.

[0032] Input tensor 105 is processed using a first convolution operation 110, which results in a set of two-dimensional feature maps 115. As illustrated, feature maps 115 have dimensions H x W x c, where H and W are spatial dimensions (e.g., time and frequency dimensions, respectively), and c is the number of channels.

[0033] In one aspect, convolution operation 110 is a depthwise convolution performed using one or more kernels configured to extract features of the frequency dimension. For example, convolution operation 110 can use n x 1 kernels, where n corresponds to the frequency dimension. That is, the depthwise kernels used for convolution operation 110 can have a length greater than 1 in the frequency dimension, with a length of 1 in the time dimension. This allows convolution operation 110 to act as a frequency depthwise convolution that extracts frequency features (e.g., feature maps 115) for tensor 105.

[0034] As illustrated, these feature maps 115 are two-dimensional (having a length greater than one in both the frequency dimension and the time dimension). In the illustrated workflow 100, a dimension reduction operation 120 is performed to reduce the dimensionality of the feature maps 115. Specifically, the dimension reduction operation 120 can reduce the feature maps 115 to eliminate the frequency dimension and to preserve the time dimension. This results in one-dimensional feature maps 125. The feature maps 125 can have the same time dimensionality and number of channels as the feature maps 115, but have a length of one in the frequency dimension.

[0035] The dimension reduction operation 120 is generally performed on a per-frequency (or per-band) basis, and can include various techniques, including max pooling (so that the maximum value, or the feature with the most activation presence, is preserved), average pooling (so that the average value is preserved), min pooling (so that the minimum value is preserved), and so forth. In some aspects, the dimension reduction operation 120 can also be performed by convolving the feature maps 115 using an Hxl kernel without padding, where H corresponds to the size of the frequency dimension.

[0036] Advantageously, the one-dimensional feature maps 125 (which correspond to the time dimension) can be convolved using significantly less computational resources than a traditional two-dimensional convolution. This significantly improves the efficiency of the broadcasted residual learning.

[0037] As illustrated, a second convolution operation 130 is used to process the feature maps 125. In some aspects, the convolution operation 130 is a depthwise separable convolution (e.g., a depthwise convolution followed by a pointwise convolution). The convolution operation 130 can be performed using one or more kernels configured to extract features for the time dimension, as compared to the convolution operation 110 (which corresponds to the frequency dimension). For example, the convolution operation 130 can use a l x m kernel, where m corresponds to the time dimension.

[0038] That is, the depthwise kernel used for the convolution operation 130 can have a length greater than one in the time dimension, with a length of one in the frequency dimension. This allows the convolution operation 130 to function as a time depthwise convolution, which extracts time features for the feature maps 125. In some aspects, the convolution operation 130 can be a depthwise separable convolution. Following the time depthwise convolution, the convolution operation 130 can apply one or more pointwise kernels in such aspects. This results in feature maps 135.

[0039] In the workflow 100, the feature maps 135 are then broadcasted to the frequency dimension, as indicated by arrow 137. This broadcast operation (also referred to as an expansion operation) generally converts the one-dimensional feature maps 135 into a multi-dimensional feature map 140 having the same dimensionality as the feature maps 115. In some aspects, this broadcast involves replicating and stacking the feature maps 135 until they reach a height H (in this example).

[0040] The residual connection 150 reflects the residual nature of broadcast residual learning. In the workflow 100, the input tensor 105 is augmented with the feature map 140 using an operation 145 to generate the output 155. In some aspects, the feature map 140 can also or instead be augmented with the feature map 115. This operation 145 can generally include any number of combining techniques, including element-wise summation, averaging, multiplication, and so forth. Advantageously, the residual connection 150 allows the system to preserve the two-dimensional nature of the input, regardless of the dimensionality reduction operation 120.

[0041] Example residual learning techniques

[0042] Figure 2 Example residual blocks 200A and 200B for residual learning techniques are depicted.

[0043] The residual block 200A reflects a conventional residual block used in some residual models. This residual block 200A can be expressed as y = x + f(x), where x and y are the input and output features, respectively, and the function f(·) computes a convolution output. The identity shortcut and the result of f(x) have the same dimensionality and can be summed by a simple element-wise addition.

[0044] In particular, as illustrated by the residual block 200A, the input 205 is processed using the same convolution operation 210. The resulting tensor can then be summed with the original input 205 (via the identity shortcut 215), as indicated by the operation 220. This results in the output 225 of the residual block 200A.

[0045] In aspects of the disclosure, to leverage both one- and two-dimensional features together, the function f(x) (reflected by the convolution operation 210) can be decomposed into f1 and f2, which correspond to the temporal and two-dimensional operations, respectively. This is reflected in the broadcast residual block 200B.

[0046] The broadcast residual block 200B can be expressed as:

[0047] y = x + BC(f1(reduction(f2(x)))),

[0048] where x and y are the input and output features, respectively, f1 and f2 are convolution operations, BC(·) is a broadcast or expansion operation, and reduction(·) is a dimensionality reduction operation (e.g., average pooling over the frequency dimension). In this expression, the batch and channel dimensions are omitted for conceptual clarity, and the input feature x is in R H×W and H and W are the frequency and temporal step sizes, respectively.

[0049] As illustrated by residual block 200B, input 250 is processed using a convolution operation 255 to extract two-dimensional features. The resulting tensor can then be reduced using a dimension reduction 260, and the reduced tensor is processed using a convolution operation 265 to extract temporal features. These features are then expanded to the frequency dimension, and augmented with the original input 250 via an identity shortcut 270, thereby producing output 280.

[0050] Example broadcast residual learning block

[0051] Figure 3 is an example broadcast residual learning block 300 for use in efficiently processing input data, such as audio input data.

[0052] As illustrated, a first operation 310 (labeled f2 in Figure 3 ) is used to receive and process input tensor 305. Operation 310 corresponds to the two-dimensional feature extraction discussed above (e.g., convolution operation 110), and produces R H×W two-dimensional feature maps in R Figure 1 (e.g., feature maps 115 in R

[0053] As illustrated, operation 310 also includes a sub-spectrum normalization (SSN) operation 325. SSN operation 325 generally operates by splitting the input features (generated by frequency-deep convolution 320) into sub-bands in the frequency dimension, and separately normalizing each sub-band (e.g., using batch normalization). This allows the system to implement frequency-aware temporal features, as compared to ordinary batch normalization over the entire set of features.

[0054] The system can then perform a dimension reduction using operation 330. In the illustrated example, broadcast residual learning block 300 uses frequency average pooling to average the input features by frequency, thereby producing the features discussed above in R 1×W (e.g., feature maps 125 in R Figure 1 ).

[0055] These features are then processed using a second operation 340 (labeled f1 in Figure 3 ). Operation 320 can correspond to the temporal convolution operation discussed above (e.g., convolution operation 130). In one aspect, operation 340 is a depthwise separable convolution (e.g., a composition of temporal depthwise convolution 345 and pointwise convolution 355).

[0056] Temporal depthwise convolution 345 can include one or more 1xm temporal depthwise convolution kernels to generate temporal features (e.g.,Figure 1 features 135) in FIG. 1.

[0057] As illustrated, operation 340 then includes a batch normalization operation 350 followed by a swish activation (also indicated by 350). Although a swish activation is depicted in Figure 3 FIG. 1, in aspects, any suitable activation function can be used.

[0058] Following the point-wise convolution 355, operation 340 can also include a channel-wise dropout (indicated by 360) at a dropout rate p. This dropout can be used as a regularization for the model in order to prevent overfitting and improve generalization. A broadcast operation (which can correspond to the broadcast operation 137 of FIG. 1) represented by operation 365 (which also includes the tensor expansion discussed above with respect to operation 145 of FIG. 1) can then be used to expand the features from operation 340 (in R Figure 1 Figure 1 1×W H×W .

[0059] In some aspects, in order to be frequency convolution aware over sequential blocks (e.g., sequential applications of the broadcasted residual learning block 300), the system not only uses the residual connection 315 (sometimes referred to as an “identity shortcut”) to augment the features using the original input 305 (at operation 365), but also uses a secondary residual connection 335 (at operation 365) from the two-dimensional features output by the frequency depth-wise convolution 320. The secondary residual connection 335 enables the system to maintain the frequency-aware features of the input despite the dimension reduction operation. The output of this broadcast and expansion operation 365 (also referred to as a broadcast sum operation in some aspects) can then be processed using one or more activation functions (e.g., a ReLU function 370), and then provided as an output 375 from the residual learning block 300.

[0060] In this way, the broadcasted residual learning block 300 can be expressed as y = x + f2(x) + BC(f1(reduction(f2(x)))), where x and y are the input and output features, respectively, f1and f2are convolution operations, BC(·) is a broadcast or expansion operation, and reduction(·) is a dimension reduction operation (e.g., average pooling by frequency dimension).

[0061] Using the broadcasted residual learning block 300, machine learning modules can provide, for example, more efficient KWS while maintaining two-dimensional features compared to conventional techniques. By performing time depth-wise and point-wise convolutions on one-dimensional temporal features, the convolutional load is reduced to one over the frequency step size H (often 40 or more) compared to traditional two-dimensional depth-wise separable convolutions.

[0062] ​​​Example transition broadcast residual learning block

[0063] Figure 4 is an example transition broadcast residual learning block 400 for use in efficiently processing input data, such as audio input data.

[0064] The transition broadcast residual learning block 400 is similar to the conventional broadcast residual learning block 300, but has two differences that enable the transition broadcast residual learning block 400 to be used in a transition layer in which the number of channels in the input 305 is different than the number of channels in the output 475.

[0065] In particular, the operation 410 replaces the operation 310 in Figure 3 The operation 410 includes an additional point-wise convolution 412 that is used to change the number of channels in the input 405 to the desired number of channels for the output 475. As illustrated, this point-wise convolution 412 can be followed by batch normalization and an activation function (such as ReLU) indicated by 413.

[0066] The second difference between the transition broadcast residual learning block 400 and the conventional broadcast residual learning block 300 is that the transition broadcast residual learning block 400 does not include the identity shortcut ( Figure 3 residual connection 315 in

[0067] In other respects, the transition broadcast residual learning block 400 largely borrows from the conventional broadcast residual learning block 300 described above with reference to Figure 3

[0068] Example method for broadcast residual learning

[0069] Figure 5 is an example flowchart illustrating a method 500 for processing data using broadcast residual learning.

[0070] The method 500 begins at block 505, where a processing system receives an input tensor that includes a frequency dimension and a time dimension.

[0071] At block 510, the processing system processes the input tensor using a first convolution operation to generate a multi-dimensional intermediate feature map that includes a frequency dimension and a time dimension. In some cases, the multi-dimensional intermediate feature map is a two-dimensional intermediate feature map.

[0072] In some aspects, the first convolution operation uses one or more depth-wise convolution kernels having a size that is greater than 1 in the frequency dimension and equal to 1 in the time dimension.

[0073] ​In some aspects, the input tensor is output from a point-wise convolution operation configured to change a number of channels in the input tensor.

[0074] At block 515, the processing system converts the multi-dimensional intermediate feature map to a one-dimensional intermediate feature map using a frequency dimension reduction operation.

[0075] In some aspects, the frequency dimension reduction operation includes at least one of: a max-pooling operation, an average-pooling operation, or a convolution operation.

[0076] In some aspects, the method 500 further includes performing a sub-spectral normalization (SSN) operation on the multi-dimensional intermediate feature map prior to converting the multi-dimensional intermediate feature map to the one-dimensional intermediate feature map.

[0077] In some aspects, wherein the SSN operation includes: dividing the multi-dimensional intermediate feature map into a plurality of sub-bands in a frequency dimension; and performing a batch normalization on each of the plurality of sub-bands.

[0078] At block 520, the processing system processes the one-dimensional intermediate feature map using a second convolution operation to generate a temporal feature map.

[0079] In some aspects, the second convolution operation includes a depth-wise separable convolution operation, wherein a depth-wise convolution of the depth-wise separable convolution operation is configured to use one or more depth-wise convolution kernels having a size equal to 1 in a frequency dimension and greater than 1 in the temporal dimension.

[0080] In some aspects, a point-wise convolution of the depth-wise separable convolution operation is configured to use one or more point-wise convolution kernels after the depth-wise convolution.

[0081] At block 525, the processing system expands the temporal feature map to a frequency dimension using a broadcast operation to generate a multi-dimensional output feature map.

[0082] At block 530, the processing system augments the multi-dimensional output feature map using the multi-dimensional intermediate feature map via a first residual connection.

[0083] In some aspects, the method 500 further includes outputting the augmented multi-dimensional output (e.g., as an output from a residual block to another residual block or another block or layer of a model, as an output from a model, etc.).

[0084] In some aspects, the method 500 further includes augmenting the multi-dimensional output feature map using the input tensor via a second residual connection.

[0085] In some aspects, the input tensor includes input audio features; and the first convolution operation and the second convolution operation are part of a broadcast residual neural network configured to classify the input audio features.

[0086] Example processing systems for broadcasted residual learning

[0087] In some aspects, regarding Figures 1-5 The described techniques, methods, and workflows can be performed on one or more devices.

[0088] Figure 6 An example processing system 600 is depicted, which can be configured to perform aspects of the various methods described herein, including, for example, regarding Figures 1-5 The described methods.

[0089] The processing system 600 includes a central processing unit (CPU) 602, which in some examples can be a multi-core CPU. Instructions executed at the CPU 602 can be loaded, for example, from a program memory associated with the CPU 602 or can be loaded from the memory partition 624.

[0090] The processing system 600 also includes additional processing components customized for particular functions, such as a graphics processing unit (GPU) 604, a digital signal processor (DSP) 606, a neural processing unit (NPU) 608, a multimedia component 610, and a wireless connectivity component 612.

[0091] An NPU, such as 608, is generally a specialized circuit configured to implement all the necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), and the like. An NPU can sometimes alternatively be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligent processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0092] An NPU, such as 608, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs can be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, multiple NPUs can be part of a specialized neural network accelerator.

[0093] An NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs capable of performing both training and inference, the two tasks can generally still be performed independently.

[0094] An NPU configured to accelerate training is generally configured to accelerate optimization of a new model, which is a highly computation-intensive operation involving inputting an existing dataset (typically labeled or tagged), iterating over the dataset, and then adjusting model parameters (such as weights and biases) in order to improve model performance. Generally, optimization based on error prediction involves backpropagating through layers of the model and determining gradients to reduce prediction error.

[0095] An NPU designed to accelerate inference is generally configured to operate on a complete model. Such an NPU can thus be configured to input a new piece of data and quickly process it through a trained model to generate a model output (e.g., an inference).

[0096] In one implementation, the NPU 608 is part of one or more of the CPU 602, GPU 604, and / or DSP 606.

[0097] In some examples, the wireless connectivity components 612 can include subcomponents, for example, for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity components 612 are further connected to one or more antennas 614.

[0098] The processing system 600 can also include one or more sensors 616 associated with any manner of sensors, one or more image signal processors (ISPs) 618 associated with any manner of image sensors, and / or a navigation component 620, which can include satellite-based positioning system components (e.g., GPS or GLONASS) as well as inertial positioning system components.

[0099] The processing system 600 can also include one or more input and / or output devices 622, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0100] In some examples, one or more processors of the processing system 600 can be based on an ARM or RISC-V instruction set.

[0101] The processing system 600 also includes memory 624, which is representative of one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 624 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 600.

[0102] In particular, in this example, the memory 624 includes a machine learning component 624A, which can be configured in accordance with one or more aspects described herein. For example, the machine learning component 624A can provide data or audio analysis using one or more machine learning models (e.g., neural networks) configured with one or more broadcast residual learning blocks.

[0103] The memory 624 further includes a set of frequency-wise depth kernels 624B and a set of time-wise depth kernels 624C. As discussed above, the frequency-wise depth kernels 624B generally include one-dimensional kernels having a length greater than one in the frequency dimension, while the time-wise depth kernels 624C include one-dimensional kernels having a length greater than one in the time dimension.

[0104] The frequency-wise depth kernels 624B can generally be used to perform frequency-wise depth convolutions (e.g., convolution operation 110 in FIG. 1), while the time-wise depth kernels 624C can generally be used to perform time-wise depth convolutions (e.g., convolution operation 130 in FIG. 1). Figure 1 Figure 1

[0105] The processing system 600 further includes a machine learning circuit 626, such as, for example, described above with respect to Figures 1-5 .

[0106] Although depicted as separate circuits in Figure 6 for the sake of clarity, the machine learning circuit 626 can be implemented in other processing devices of the processing system 600, such as within the CPU 602, GPU 604, DSP 606, NPU 608, etc.

[0107] In general, the processing system 600 and / or its components can be configured to perform the methods described herein.

[0108] Notably, in other aspects, aspects of the processing system 600 can be omitted, such as where the processing system 600 is a server computer, etc. For example, in other aspects, the multimedia component 610, wireless connectivity component 612, sensors 616, ISP 618, and / or navigation component 620 can be omitted. Further, aspects of the processing system 600 can be distributed among multiple devices.

[0109] The depicted components, as well as other un-depicted components, can be configured to perform various aspects of the methods described herein.

[0110] Example Clauses

[0111] ​​Clause 1 : A method comprising: receiving an input tensor comprising a frequency dimension and a time dimension; processing the input tensor using a first convolution operation to generate a multi-dimensional intermediate feature map comprising the frequency dimension and the time dimension; converting the multi-dimensional intermediate feature map into a one-dimensional intermediate feature map over the time dimension using a frequency dimension reduction operation; processing the one-dimensional intermediate feature map using a second convolution operation to generate a time feature map; expanding the time feature map into the frequency dimension using a broadcast operation to generate a multi-dimensional output feature map; and augmenting the multi-dimensional output feature map using the multi-dimensional intermediate feature map via a first residual connection.

[0112] Clause 2: The method of clause 1, wherein the multi-dimensional intermediate feature map is a two-dimensional intermediate feature map.

[0113] Clause 3: The method of any of clauses 1-2, further comprising: augmenting the multi-dimensional output feature map using the input tensor via a second residual connection.

[0114] Clause 4: The method of any of clauses 1-3, wherein the first convolution operation uses one or more depthwise convolution kernels having a size greater than 1 over the frequency dimension and equal to 1 over the time dimension.

[0115] Clause 5: The method of any of clauses 1-4, wherein the input tensor is output from a pointwise convolution operation configured to change a number of channels in the input tensor.

[0116] Clause 6: The method of any of clauses 1-5, further comprising: performing a sub-spectral normalization (SSN) operation on the multi-dimensional intermediate feature map prior to converting the multi-dimensional intermediate feature map into the one-dimensional intermediate feature map.

[0117] Clause 7: The method of any of clauses 1-6, wherein the SSN operation comprises: dividing the multi-dimensional intermediate feature map into a plurality of sub-bands over the frequency dimension; and performing batch normalization on each of the plurality of sub-bands.

[0118] Clause 8: The method of any of clauses 1-7, wherein the frequency dimension reduction operation comprises at least one of: a max pooling operation, an average pooling operation, or a convolution operation.

[0119] Clause 9: The method of any of clauses 1-8, wherein the second convolution operation comprises a depthwise separable convolution operation, wherein a depthwise convolution of the depthwise separable convolution operation is configured to use one or more depthwise convolution kernels having a size equal to 1 over the frequency dimension and greater than 1 over the time dimension.

[0120] Clause 10: The method of any of clauses 1-9, wherein the point-wise convolutions of the depth-wise separable convolution operation are configured to use one or more point-wise convolution kernels after the depth-wise convolution.

[0121] Clause 11 : The method of any of clauses 1-10, wherein: the input tensor comprises input audio features; and the first and second convolution operations are part of a broadcast residual neural network configured to classify the input audio features.

[0122] Clause 12: A system comprising means for performing the method of any of clauses 1-11.

[0123] Clause 13: A system comprising: memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform the method of any of clauses 1-11.

[0124] Clause 14: A non-transitory computer-readable medium including computer- executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any of clauses 1-11.

[0125] Clause 15: A computer program product embodied on a computer-readable storage medium comprising code for performing the method of any of clauses 1-11.

[0126] Additional Considerations

[0127] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not meant to limit the scope of the claims presented herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects. For example, features described in relation to one example can be combined with features described in relation to a different example. Various examples can omit, substitute, or add various procedures or components as appropriate. For instance, the methods described can be performed in an order different from that described, and other steps can be added, omitted, or combined. Also, features described with respect to certain examples can be combined in some other examples. For example, an apparatus can be implemented or a method can be practiced using any number of the aspects set forth herein. In addition, the scope of the disclosure is intended to cover any resulting apparatus or method which falls within the scope of the claims, including all aspects of such resulting apparatus and methods, regardless of whether such aspects are recited herein. It should be appreciated that any aspect of the disclosure disclosed herein can be implemented by one or more elements of a claim.

[0128] As used herein, the term "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0129] As used herein, the phrase "at least one of" followed by a listing of two or more items means any of the listed items individually, or any combination of two or more of the listed items. As an example, "at least one of a, b, or c" means "a" or "b" or "c" or "a-b" or "a-c" or "b-c" or "a-b-c" or any combination thereof.

[0130] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, "determining" can include resolving, selecting, choosing, establishing and the like.

[0131] As used herein, the term "connected to" in the context of sharing electronic signals and data between elements described herein can generally refer to being in data communication between respective elements that are connected to one another. In some cases, elements can be directly connected to one another, such as via one or more conductive traces, lines, or other conductive carriers capable of carrying signals and / or data between respective elements that are directly connected to one another. In other cases, elements can be indirectly connected to one another, such as via one or more data buses or similar shared circuitry and / or integrated circuit elements for communicating signals and data between respective elements that are indirectly connected to one another.

[0132] The methods disclosed herein comprise one or more steps or actions for accomplishing a method. The method steps and / or actions can be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. Further, the various operations of methods described above can be performed by any suitable means capable of performing the corresponding functions. Such means can include various hardware and / or software component(s) and / or module(s), including, but not limited to a circuit, an application specific integrated circuit (ASIC), or processor. Generally, where there are operations illustrated in figures, those operations can have corresponding counterpart means-plus-function components with similar numbering.

[0133] The following claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language of the claims. Within the claims, the singular forms "a," "an," and "the" do not exclude the plural reference unless the context clearly dictates otherwise. Unless specifically stated otherwise, the term "some" refers to one or more. Unless specifically stated otherwise, the term "any" refers to one or more. Any element expressed by the phrase "means for" is intended to functionally equate to the structures implied by such phrase, for example, a "means for viewing" would equate to a "viewer." The use of the term "about" with respect to a given numerical value shall mean that the value is the stated value, plus or minus ten percent, unless specifically stated otherwise. The use of the term "about" with respect to a given numerical value shall mean that the value is the stated value, plus or minus ten percent, unless specifically stated otherwise. The use of the term "about" with respect to a given numerical value shall mean that the value is the stated value, plus or minus ten percent, unless specifically stated otherwise. The elements of various aspects described throughout this disclosure are intended to be all structural and functional equivalents of the elements described in this specification that enable the claimed aspects. Additionally, means-plus-function or step-plus-function elements described herein are intended to perform the functions described in connection with such elements. Furthermore, to the extent that the term "includes" is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term "comprising" as "comprising" is interpreted when employed as a transitional term in the claim.

Claims

1. A computer-implemented method, comprising: Receive an input tensor that includes a frequency dimension and a time dimension, wherein the input tensor includes input audio data or input audio features; The input tensor is processed using a first convolution operation to generate a multidimensional intermediate feature map that includes the frequency dimension and the time dimension. The frequency dimension reduction operation is used to transform the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension. The first and second convolution operations are used to process the one-dimensional intermediate feature map to generate a temporal feature map, wherein the first and second convolution operations are part of a broadcast residual neural network configured to classify the input audio data or the input audio features; The time feature map is extended to the frequency dimension using a broadcast operation to generate a multidimensional output feature map. The multidimensional output feature map is augmented using the multidimensional intermediate feature map via the first residual connection; as well as Output the amplified multidimensional output feature map.

2. The computer-implemented method of claim 1, wherein the multidimensional intermediate feature map is a two-dimensional intermediate feature map, and wherein converting the multidimensional intermediate feature map into the one-dimensional intermediate feature map reduces the number of computations performed by the processor when generating the temporal feature map.

3. The computer-implemented method as described in claim 1, further comprising: The multidimensional output feature map is augmented using the input tensor via a second residual connection.

4. The computer-implemented method of claim 1, wherein the first convolution operation uses one or more depthwise convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension.

5. The computer-implemented method of claim 4, wherein the input tensor is output from a pointwise convolution operation configured to change the number of channels in the input tensor.

6. The computer-implemented method of claim 1, further comprising: Before converting the multidimensional intermediate feature map into a one-dimensional intermediate feature map, a subspectral normalization operation is performed on the multidimensional intermediate feature map.

7. The computer-implemented method as described in claim 6, wherein, The sub-spectrum normalization operation includes: The multidimensional intermediate feature map is divided into multiple sub-bands along the frequency dimension; and Batch normalization is performed on each of the plurality of subbands.

8. The computer-implemented method as described in claim 1, wherein... The frequency dimension reduction operation includes at least one of the following: max pooling, average pooling, or convolution.

9. The computer-implemented method as described in claim 1, The second convolution operation includes a depthwise separable convolution operation, wherein the depthwise convolution of the depthwise separable convolution operation is configured to use one or more depthwise convolution kernels having a size equal to 1 in the frequency dimension and greater than 1 in the time dimension.

10. The computer-implemented method of claim 9, wherein... The pointwise convolution of the depthwise separable convolution operation is configured to use one or more pointwise convolution kernels after the depthwise convolution.

11. A non-transient computer-readable medium comprising computer-executable instructions, which, when executed by one or more processors of a processing system, cause the processing system to perform operations including: Receive an input tensor that includes a frequency dimension and a time dimension, wherein the input tensor includes input audio data or input audio features; The input tensor is processed using a first convolution operation to generate a multidimensional intermediate feature map that includes the frequency dimension and the time dimension. The frequency dimension reduction operation is used to transform the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension. The first and second convolution operations are used to process the one-dimensional intermediate feature map to generate a temporal feature map, wherein the first and second convolution operations are part of a broadcast residual neural network configured to classify the input audio data or the input audio features; The time feature map is extended to the frequency dimension using a broadcast operation to generate a multidimensional output feature map. The multidimensional output feature map is augmented using the multidimensional intermediate feature map via the first residual connection; as well as Output the amplified multidimensional output feature map.

12. The non-transient computer-readable medium of claim 11, wherein the operation further comprises: The multidimensional output feature map is augmented using the input tensor via a second residual connection.

13. The non-transient computer-readable medium of claim 11, wherein the first convolution operation uses one or more depthwise convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension.

14. The non-transient computer-readable medium of claim 13, wherein the input tensor is output from a pointwise convolution operation configured to change the number of channels in the input tensor.

15. The non-transient computer-readable medium of claim 11, further comprising: Before converting the multidimensional intermediate feature map into the one-dimensional intermediate feature map, a subspectral normalization operation is performed on the multidimensional intermediate feature map.

16. The non-transient computer-readable medium of claim 15, wherein, The sub-spectrum normalization operation includes: The multidimensional intermediate feature map is divided into multiple sub-bands along the frequency dimension; and Batch normalization is performed on each of the plurality of subbands.

17. The non-transient computer-readable medium of claim 11, wherein the frequency dimension reduction operation comprises at least one of: (i) max pooling, (ii) average pooling, or (iii) convolution.

18. The non-transient computer-readable medium of claim 11, wherein the second convolution operation includes a depthwise separable convolution operation, wherein the depthwise convolution of the depthwise separable convolution operation is configured to use one or more depthwise convolution kernels having a size equal to 1 in the frequency dimension and greater than 1 in the time dimension.

19. The non-transient computer-readable medium of claim 18, wherein the pointwise convolution of the depthwise separable convolution operation is configured to use one or more pointwise convolution kernels after the depthwise convolution.

20. A processing system, comprising: The memory includes computer-executable instructions; One or more processors, the one or more processors being configured to execute the computer-executable instructions and cause the processing system to perform operations including the following: Receive an input tensor that includes a frequency dimension and a time dimension, wherein the input tensor includes input audio data or input audio features; The input tensor is processed using a first convolution operation to generate a multidimensional intermediate feature map that includes the frequency dimension and the time dimension. The frequency dimension reduction operation is used to transform the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension. The first and second convolution operations are used to process the one-dimensional intermediate feature map to generate a temporal feature map, wherein the first and second convolution operations are part of a broadcast residual neural network configured to classify the input audio data or the input audio features; The time feature map is extended to the frequency dimension using a broadcast operation to generate a multidimensional output feature map. The multidimensional output feature map is augmented using the multidimensional intermediate feature map via the first residual connection; as well as Output the amplified multidimensional output feature map.

21. The processing system of claim 20, wherein the operation further comprises: The multidimensional output feature map is augmented using the input tensor via a second residual connection.

22. The processing system of claim 20, wherein the first convolution operation uses one or more depthwise convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension.

23. The processing system of claim 22, wherein the input tensor is output from a pointwise convolution operation configured to change the number of channels in the input tensor.

24. The processing system of claim 20, further comprising: Before converting the multidimensional intermediate feature map into a one-dimensional intermediate feature map, a subspectral normalization operation is performed on the multidimensional intermediate feature map.

25. The processing system of claim 24, wherein the sub-spectrum normalization operation comprises: The multidimensional intermediate feature map is divided into multiple sub-bands along the frequency dimension; as well as Batch normalization is performed on each of the plurality of subbands.

26. The processing system of claim 20, wherein the frequency dimension reduction operation includes at least one of the following: (i) max pooling, (ii) average pooling, or (iii) convolution.

27. The processing system of claim 20, wherein the second convolution operation includes a depthwise separable convolution operation, wherein the depthwise convolution of the depthwise separable convolution operation is configured to use one or more depthwise convolution kernels having a size equal to 1 in the frequency dimension and greater than 1 in the time dimension.

28. A processing system, comprising: A means for receiving an input tensor including a frequency dimension and a time dimension, wherein the input tensor includes input audio data or input audio features; A means for processing the input tensor using a first convolution operation to generate a multidimensional intermediate feature map including the frequency dimension and the time dimension; Apparatus for converting a multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension using a frequency dimension reduction operation; A means for processing the one-dimensional intermediate feature map using a second convolution operation to generate a temporal feature map, wherein the first convolution operation and the second convolution operation are part of a broadcast residual neural network configured to classify the input audio data or the input audio features; Apparatus for extending the temporal feature map to the frequency dimension using a broadcast operation to generate a multidimensional output feature map; as well as A means for amplifying the multidimensional output feature map using the multidimensional intermediate feature map via a first residual connection.

Images