Broadcast Residual Learning

Abstract

Some aspects of the present disclosure provide a technique for efficient broadcast residual machine learning. An input tensor with a frequency dimension and a time dimension is received, and the input tensor is processed using a first convolution operation to generate a multi-dimensional intermediate feature map with a frequency dimension and a time dimension. The multi-dimensional intermediate feature map is converted to a one-dimensional intermediate feature map in the time dimension using a frequency dimension reduction operation, and the one-dimensional intermediate feature map is processed using a second convolution operation to generate a time feature map. The time feature map is expanded in the frequency dimension using a broadcasting operation to generate a multi-dimensional output feature map, and the multi-dimensional output feature map is expanded with the multi-dimensional intermediate feature map via the first residual connection.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 166,161, filed March 25, 2021, the entire contents of each of which are incorporated herein by reference.

[0002] Aspects of the present disclosure relate to machine learning, and more particularly, to efficient data processing. [Background technology]

[0003]

[0003] Designing efficient machine learning architectures is an important topic in neural speech processing. In particular, keyword spotting (KWS), which aims to detect predefined keywords, is becoming increasingly important. KWS plays an important role in device activation and user interaction on smart devices. However, it is difficult to provide a model that minimizes errors but also operates efficiently. Model efficiency is particularly important in KWS because the process is generally performed on edge devices (e.g., devices with limited resources such as mobile phones, smart speakers, and Internet of Things (IoT) devices) but at the same time requires low latency.

[0004]

[0004] Therefore, there is a need for a system and method for providing highly accurate classification using efficient model design. Summary of the Invention

[0005]

[0005] Some aspects provide a method comprising receiving an input tensor having a frequency dimension and a temporal dimension, processing the input tensor with a first convolution operation to generate a multidimensional intermediate feature map having a frequency dimension and a temporal dimension, converting the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the temporal dimension using a frequency dimension reduction operation, processing the one-dimensional intermediate feature map using a second convolution operation to generate a temporal feature map, expanding the temporal feature map into the frequency dimension using a broadcasting operation to generate a multidimensional output feature map, and augmenting the multidimensional output feature map with the multidimensional intermediate feature map via a first residual connection.

[0006]

[0006] Other aspects provide a processing system configured to perform the above-mentioned methods and methods described herein; a non-transitory computer-readable medium comprising instructions which, when executed by one or more processors of the processing system, cause the processing system to perform the above-mentioned methods and methods described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the above-mentioned methods and methods further described herein; and a processing system comprising means for performing the above-mentioned methods and methods further described herein.

[0007] The following description and the related drawings set forth in detail certain illustrative features of the one or more aspects.

[0008]

[0008] The accompanying drawings illustrate some aspects of the one or more aspects and therefore should not be considered as limiting the scope of the present disclosure. [Brief description of the drawings]

[0009] [Figure 1]

[0009] FIG. 1 illustrates an example workflow for broadcasted residual learning. [Diagram 2]

[0010] 1 is an exemplary block diagram for a residual learning technique. [Diagram 3]

[0011] 1 is a diagram of an example broadcast residual learning block for use in efficiently processing input data. [Figure 4]

[0012] 1 is a diagram of an example broadcast residual learning block for use in efficiently processing input data in a transitional layer. [Diagram 5]

[0013] 1 is an example flow diagram illustrating a method for processing data using broadcast residual learning. [Figure 6]

[0014] FIG. 1 illustrates an exemplary processing system configured to implement various aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010]

[0015] For ease of understanding, wherever possible, identical reference numbers have been used to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation.

[0011]

[0016] Aspects of the present disclosure provide techniques for broadcasted residual learning. The techniques described herein provide high model accuracy and significantly improved computational efficiency (e.g., small model size and light computational load) compared to existing approaches.

[0012]

[0017] A wide variety of efficient convolutional neural networks (CNNs) have been developed recently. Generally, CNNs consist of repeated blocks of the same structure, and are often based on residual learning and depthwise separable convolution. This has given rise to several CNN-based KWS methods. Existing methods use either one-dimensional time convolution or two-dimensional (e.g., frequency and time) convolution. Each method has its own advantages and disadvantages.

[0013]

[0018] For example, models using one-dimensional time convolution generally require fewer computing resources compared to models relying on two-dimensional techniques. However, one-dimensional convolution does not capture internal biases of the convolution (such as translation equivariance) for the frequency dimension.

[0014]

[0019] On the other hand, approaches based on two-dimensional convolutions require significantly more computational resources than one-dimensional methods, even when using efficient designs and architectures such as depthwise separable convolutions, which may prevent such two-dimensional approaches from being useful for a wide variety of devices and implementations.

[0015]

[0020] The broadcast residual learning techniques described herein can be used to efficiently process data both during training (while training data is passed through the model) and during runtime (when new data is passed to generate inferences).

[0016]

[0021] In some aspects, broadcast residual learning is used to process and classify audio data and features (e.g., to implement KWS). In general, audio data and features may be represented using two-dimensional tensors (e.g., with frequency and time dimensions). Although audio is used in the examples herein, aspects of the present disclosure may be readily applied to a wide variety of data.

[0017]

[0022] In some aspects, broadcast residual learning generally involves performing a convolution on an input tensor to extract a two-dimensional feature, reducing the dimensionality of the two-dimensional feature to enable efficient convolution on the feature (e.g., requiring reduced computation, processing steps, and energy), expanding the resulting tensor to the original dimensionality of the two-dimensional feature, and augmenting the augmented tensor with the original two-dimensional feature. In some aspects, the augmented tensor is further augmented with the original input tensor.

[0018]

[0023] In some aspects, the broadcast residual learning described herein may be implemented in a neural network architecture to perform various tasks, such as classifying input audio. For example, the techniques described herein may be implemented as broadcast residual learning blocks, and several of these blocks may be used in sequence within a neural network architecture.

[0019]

[0024] Advantageously, broadcast residual learning preserves many residual features of one-dimensional temporal convolutions while still allowing two-dimensional convolutions to be used together via broadcast residual connections that expand the time output into the frequency dimension. This residual mapping allows the network to effectively represent useful audio features with many fewer computations than traditional convolutional neural networks, which reduces computational complexity, latency, compute requirements, memory requirements, etc. In aspects, the broadcast residual learning techniques described herein can achieve state-of-the-art accuracy on voice command datasets using fewer computations and parameters compared to traditional systems.

[0020] Example Workflow for Broadcasted Residual Learning

[0025] FIG. 1 illustrates an example workflow 100 for broadcast residual learning. The workflow 100 starts with an input tensor 105. In some examples, the tensor 105 can be audio data (e.g., represented by a log Mel spectrogram showing a spectrum of frequencies over time) or audio features (e.g., features generated by processing audio data). In some aspects, the input tensor 105 is a two-dimensional tensor with a frequency dimension and a time dimension. The time dimension can be defined into time intervals or steps, and the frequency dimension is defined based on frequency values ​​or bands. The frequencies present in each interval (e.g., the loudness at each frequency) can be reflected via values ​​in the tensor.

[0021]

[0026] The input tensor 105 is processed using a first convolution operation 110, which results in a set of two-dimensional feature maps 115. As shown, the feature maps 115 have dimensionality H×W×c, where H and W are spatial dimensions (e.g., the time and frequency dimensions, respectively) and c is the number of channels.

[0022]

[0027] In one aspect, the convolution operation 110 is a depthwise convolution performed using one or more kernels configured to extract features in the frequency dimension. For example, the convolution operation 110 may use an n×1 kernel, where n corresponds to the frequency dimension. That is, a depthwise kernel for the convolution operation 110 may have a length greater than 1 in the frequency dimension and a length of 1 in the time dimension. This allows the convolution operation 110 to act as a frequency depthwise convolution that extracts frequency features (e.g., feature map 115) for the tensor 105.

[0023]

[0028] As shown, these feature maps 115 are two-dimensional (having lengths greater than one in both the frequency and time dimensions). In the illustrated workflow 100, a dimension reduction operation 120 is performed to reduce the dimensionality of the feature maps 115. In particular, the dimension reduction operation 120 may reduce the feature maps 115 to eliminate the frequency dimension and preserve the time dimension. This results in a one-dimensional feature map 125. The feature map 125 may have the same time dimensionality and number of channels as the feature map 115, but with a length of one in the frequency dimension.

[0024]

[0029] The dimensionality reduction operation 120 is generally performed per frequency (or per frequency band) and may include a variety of techniques, including max pooling (where the features with the maximum or most activated presence are retained), average pooling (where the average value is retained), min pooling (where the minimum value is retained), etc. In some aspects, the dimensionality reduction operation 120 may also be performed by convolving the feature maps 115 with an H×1 kernel without padding to reduce dimensionality, where H corresponds to the size of the frequency dimension.

[0025]

[0030] Advantageously, the one-dimensional feature map 125 (corresponding to the time dimension) can be convolved using significantly fewer computational resources compared to traditional two-dimensional convolution, which significantly improves the efficiency of broadcast residual learning.

[0026]

[0031] As shown, the feature map 125 is processed using a second convolution operation 130. In some aspects, the convolution operation 130 is a depthwise separable convolution (e.g., a depthwise convolution followed by a pointwise convolution). In contrast to the convolution operation 110 (which corresponds to the frequency dimension), the convolution operation 130 may be performed using one or more kernels configured to extract features for the time dimension. For example, the convolution operation 130 may use a 1×m kernel, where m corresponds to the time dimension.

[0027]

[0032] That is, the depthwise kernel for the convolution operation 130 may have a length greater than one in the time dimension and a length of one in the frequency dimension. This allows the convolution operation 130 to act as a temporal depthwise convolution that extracts temporal features for the feature map 125. In some aspects, the convolution operation 130 may be a depthwise separable convolution. In such aspects, following the temporal depthwise convolution, the convolution operation 130 may apply one or more pointwise kernels. This results in the feature map 135.

[0028]

[0033] In workflow 100, feature map 135 is then broadcast in the frequency dimension, as indicated by arrow 137. This broadcasting operation (also called a dilation operation) generally transforms one-dimensional feature map 135 into a multi-dimensional feature map 140 with the same dimensionality as feature map 115. In some aspects, broadcasting involves copying and stacking feature maps 135 (in this example) until they reach a height of H.

[0029]

[0034] The residual connections 150 reflect the residual nature of broadcast residual learning. In the workflow 100, the input tensor 105 is augmented with the feature map 140 using an operation 145 to generate the output 155. In some aspects, the feature map 140 may also or alternatively be augmented with the feature map 115. This operation 145 may generally include any number of combination techniques, including element-wise summation, averaging, multiplication, etc. Advantageously, the residual connections 150 allow the system to preserve the two-dimensional features of the input despite the dimensionality reduction operation 120.

[0030] Example Residual Learning Techniques

[0035] FIG. 2 shows example block diagrams 200A and 200B for a residual learning technique.

[0031]

[0036] Block 200A reflects a traditional residual block used in some residual models. This block 200A can be expressed as y=x+f(x), where x and y are the input and output features, respectively, and the function f(·) computes the convolution output. The identity shortcut of x and the result of f(x) are of the same dimensionality and can be added by simple element-wise addition.

[0032]

[0037] In particular, as shown by residual block 200A, input 205 is processed using some convolution operation 210. The resulting tensor may then be added (via identity shortcut 215) with the original input 205, as shown by operation 220. This results in output 225 of regular residual block 200A.

[0033]

[0038] In an aspect of the present disclosure, to utilize both one-dimensional and two-dimensional features together, the function f(x) (reflected by the convolution operation 210) may be decomposed into f1 and f2, which correspond to a time operation and a two-dimensional operation, respectively. This is reflected in the broadcast residual block 200B.

[0034]

[0039] The broadcast residual block 200B is

[0035]

number

[0036] It can be expressed as:

[0040] where x and y are the input and output features, respectively, f1 and f2 are the convolution operations, BC(·) is the broadcasting or augmentation operation, and reduction(·) is the dimensionality reduction operation (e.g., average pooling over the frequency dimension). In this formulation, the batch and channel dimensions are ignored for conceptual clarity, and the input feature x is

[0037]

number

[0038] where H and W are the frequency step and time step, respectively.

[0039]

[0041] As indicated by residual block 200B, input 250 is processed using a convolution operation 255 to extract two-dimensional features. The resulting tensor may then be reduced using dimensionality reduction 260, and the reduced tensor(s) are processed using a convolution operation 265 to extract temporal features. These features are then expanded into the frequency dimension and augmented with the original input 250 via identity shortcut 270, which results in output 280.

[0040] Example Broadcasted Residual Learning Block

[0042] FIG. 3 is an exemplary broadcast residual training block 300 for use in efficiently processing input data, such as audio input data.

[0041]

[0043] As shown, an input tensor 305 is received and processed using a first operation 310 (labeled f2 in FIG. 3). Operation 310 corresponds to the two-dimensional feature extraction (e.g., convolution operation 110) described above,

[0042]

number

[0043] (e.g., feature map 115 in FIG. 1 ). As shown, the convolution operation 310 is performed using a frequency-depth-wise convolution 320 with one or more n×1 frequency-depth-wise convolution kernels.

[0044] As shown, the operation 310 also includes a SubSpectral Normalization (SSN) operation 325. The SSN operation 325 generally operates by splitting the input features (produced by the frequency-depth unit convolution 320) into sub-bands in the frequency dimension and normalizing each sub-band separately (e.g., using batch normalization). This allows the system to achieve frequency-aware temporal features compared to regular batch normalization on the entire feature set.

[0045]

[0045] The system can then perform dimensionality reduction using operation 330. In the illustrated example, the broadcast residual learning block 300 uses frequency average pooling to average the input features by frequency, which, as explained above,

[0046]

number

[0047] 1.

[0048] These features are then processed using a second operation 340 (labeled f1 in FIG. 3). The operation 320 may correspond to the time convolution operation described above (e.g., convolution operation 130). In one aspect, the operation 340 is a depthwise separable convolution (e.g., a composite of a time-depthwise convolution 345 and a pointwise convolution 355).

[0049]

[0047] The time-depth unit convolution 345 may comprise one or more 1xm time-depth unit convolution kernels for generating temporal features (eg, feature map 135 in FIG. 1).

[0050] As shown, operation 340 then includes a batch normalization operation 350 followed by swish activation (also indicated by 350). Although swish activation is shown in FIG. 3, in aspects any suitable activation function may be used.

[0051] Following the point-wise convolution 355, the operation 340 may also include a channel-wise dropout (denoted by 360) at a dropout rate p. This dropout may be used as a regularization for the model to prevent overfitting and improve generalization. A broadcasting operation (which may correspond to the broadcasting operation 137 of FIG. 1), represented by operation 365 (which also includes the tensor expansion described above with reference to operation 145 of FIG. 1), is then performed (

[0052]

number

[0053] (in) the feature from the operation 340

[0054]

number

[0055] can be used to expand

[0056] In some aspects, to be frequency convolution aware across successive blocks (e.g., successive applications of the broadcast residual learning block 300), the system not only uses the residual connections 315 (sometimes called “identity shortcuts”) to expand features with the original input 305 (in operation 365), but also uses auxiliary residual connections 335 from the two-dimensional features output by the frequency-depth unit convolution 320 (in operation 365). This auxiliary residual connection 335 allows the system to preserve the frequency-aware features of the input despite the dimensionality reduction operation. The output of this broadcasting and expanding operation 365 (also called a broadcast sum operation in some aspects) may then be processed using one or more activation functions (e.g., a ReLU function 370) and then provided as an output 375 from the residual learning block 300.

[0057]

[0051] In this way, the broadcast residual learning block 300 can be expressed as y = x + f2(x) + BC(f1(reduction(f2(x)))), where x and y are the input and output features, respectively, f1 and f2 are convolution operations, BC(·) is the broadcasting or augmentation operation, and reduction(·) is a dimensionality reduction operation (e.g., average pooling on the frequency dimension).

[0058]

[0052] Using the broadcast residual learning block 300, the machine learning model can provide a more efficient KWS, for example, compared to conventional techniques, while preserving two-dimensional features. By performing time-depthwise convolution and point-wise convolution on one-dimensional time features, the computational burden is reduced by a factor of frequency steps H (often 40 or more) compared to traditional two-dimensional depthwise separable convolution.

[0059] Example Transitional Broadcasted Residual Learning Block FIG. 4 is an exemplary transition broadcast residual training block 400 for use in efficiently processing input data, such as audio input data.

[0060]

[0054] The transition broadcast residual learning block 400 is similar to the normal broadcast residual learning block 300, with two differences that allow the transition broadcast residual learning block 400 to be used in a transition layer where the number of channels at the input 305 is different from the number of channels at the output 475.

[0061]

[0055] In particular, operation 410 replaces operation 310 in Fig. 3. Operation 410 includes an additional point-wise convolution 412, which is used to change the number of channels in the input 405 to a desired number of channels for the output 475. As shown, this point-wise convolution 412 may be followed by batch normalization and an activation function (such as ReLU), indicated by 413.

[0062] The second difference between the transition broadcast residual learning block 400 and the normal broadcast residual learning block 300 is that the transition broadcast residual learning block 400 does not include an identity shortcut (residual connection 315 in FIG. 3 ), i.e., the transition broadcast residual learning block 400 does not use the input 405 to augment the output (because of the different dimensionality).

[0063] In other respects, the transition broadcast residual learning block 400 largely mirrors the regular broadcast residual learning block 300 described above with reference to FIG.

[0064] Example Method for Broadcasted Residual Learning FIG. 5 is an example flow diagram illustrating a method 500 for processing data using broadcast residual learning.

[0065]

[0059] The method 500 begins at block 505, where a processing system receives an input tensor having a frequency dimension and a time dimension.

[0066]

[0060] In block 510, the processing system processes the input tensor using a first convolution operation to generate a multi-dimensional intermediate feature map having a frequency dimension and a time dimension. In some cases, the multi-dimensional intermediate feature map is a two-dimensional intermediate feature map.

[0067]

[0061] In some aspects, the first convolution operation uses one or more depthwise convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension.

[0068]

[0062] In some aspects, the input tensor is output from a pointwise convolution operation configured to change the number of channels in the input tensor.

[0069]

[0063] In block 515, the processing system converts the multi-dimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension using a frequency dimension reduction operation.

[0070] In some aspects, the frequency dimension reduction operation comprises at least one of a maximum pooling operation, an average pooling operation, or a convolution operation.

[0071]

[0065] In some aspects, the method 500 further comprises performing a subspectral normalization (SSN) operation on the multi-dimensional intermediate feature map prior to converting the multi-dimensional intermediate feature map to a one-dimensional intermediate feature map.

[0072]

[0066] In some aspects, the SSN operation comprises splitting the multi-dimensional intermediate feature map into a plurality of sub-bands in a frequency dimension and performing batch normalization on each sub-band of the plurality of sub-bands.

[0073]

[0067] At block 520, the processing system processes the one-dimensional intermediate feature map using a second convolution operation to generate a temporal feature map.

[0074]

[0068] In some aspects, the second convolution operation comprises a depthwise separable convolution operation, wherein the depthwise convolution of the depthwise separable convolution operation is configured to use one or more depthwise convolution kernels having a size equal to one in the frequency dimension and greater than one in the time dimension.

[0075]

[0069] In some aspects, the pointwise convolution of the depthwise separable convolution operation is configured to use one or more pointwise convolution kernels after the depthwise convolution.

[0076]

[0070] At block 525, the processing system expands the time feature map into the frequency dimension using a broadcasting operation to generate a multi-dimensional output feature map.

[0077]

[0071] In block 530, the processing system augments the multi-dimensional output feature map with the multi-dimensional intermediate feature map via a first residual connection.

[0078]

[0072] In some aspects, the method 500 further includes outputting the expanded multidimensional output (e.g., from the residual block, as an output to another residual block or other block or layer of the model, as an output from a mode, etc.).

[0079]

[0073] In some aspects, the method 500 further comprises augmenting the multi-dimensional output feature map with the input tensor via a second residual connection.

[0080]

[0074] In some aspects, the input tensor comprises an input audio feature, and the first convolution operation and the second convolution operation are part of a broadcast residual neural network configured to classify the input audio feature.

[0081] Example Processing System for Broadcasted Residual Learning In some aspects, the techniques, methods, and workflows described with respect to FIGS. 1-5 may be implemented on one or more devices.

[0082] FIG. 6 illustrates an example processing system 600 that may be configured to perform aspects of the various methods described herein, including, for example, the methods described with respect to FIGS. 1-5.

[0083]

[0077] The processing system 600 includes a central processing unit (CPU) 602, which may be a multi-core CPU in some examples. Instructions to be executed in the CPU 602 may be loaded from a program memory associated with the CPU 602 or may be loaded from a memory partition 624, for example.

[0084]

[0078] The processing system 600 also includes additional processing components adapted to specific functions, such as a graphics processing unit (GPU) 604, a digital signal processor (DSP) 606, a neural processing unit (NPU) 608, a multimedia processing unit 610, and wireless connectivity components 612.

[0085]

[0079] An NPU, such as 608, is generally a specialized circuit configured to implement all the necessary control and arithmetic logic to run machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), visual processing units (VPUs), or graph processing units.

[0086]

[0080] NPUs, such as 608, are configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, multiple NPUs may be part of a dedicated neural network accelerator.

[0087]

[0081] NPUs can be optimized for training or inference, or in some cases configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still be performed largely independently.

[0088]

[0082] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimizing based on erroneous predictions involves backpropagating through layers of the model and determining gradients to reduce prediction errors.

[0089]

[0083] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs can be configured to input new data and rapidly process it through already-trained models to generate model outputs (e.g., inferences).

[0090]

[0084] In one implementation, the NPU 608 is part of one or more of the CPU 602, GPU 604, and / or DSP 606.

[0091] In some examples, the wireless connectivity component 612 may include sub-components for, for example, third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 612 is further connected to one or more antennas 614.

[0092]

[0086] The processing system 600 may also include one or more sensor processing units 616 associated with any type of sensor, one or more image signal processors (ISPs) 618 associated with any type of image sensor, and / or a navigation processor 620, which may include satellite-based positioning system components (e.g., GPS or GLONASS) as well as inertial positioning system components.

[0093]

[0087] The processing system 600 may also include one or more input and / or output devices 622, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0094]

[0088] In some examples, one or more of the processors of processing system 600 may be based on the ARM or RISC-V instruction set.

[0095] Processing system 600 also includes memory 624, which represents one or more static and / or dynamic memories, such as dynamic random access memories, flash-based static memories, etc. In this example, memory 624 includes computer-executable components that may be executed by one or more of the above-mentioned processors of processing system 600.

[0096]

[0090] In particular, in this example, the memory 624 includes a machine learning component 624A, which may be configured according to one or more aspects described herein. For example, the machine learning component 624A may provide data or audio analysis using one or more machine learning models (e.g., neural networks) configured with one or more broadcast residual learning blocks.

[0097]

[0091] The memory 624 further includes a set of frequency depth unit kernel(s) 624B and a set of time depth unit kernel(s) 624C. As explained above, the frequency depth unit kernel 624B generally includes a one-dimensional kernel having a length greater than one in the frequency dimension, and the time depth unit kernel 624C generally includes a one-dimensional kernel having a length greater than one in the time dimension.

[0098]

[0092] The frequency depth unit kernel 624B may generally be used to perform a frequency depth unit convolution (e.g., convolution operation 110 in FIG. 1), and the time depth unit kernel 624C is generally used to perform a time depth unit convolution (e.g., convolution operation 130 in FIG. 1).

[0099]

[0093] The processing system 600 further comprises a machine learning circuit 626, such as, for example, that described above with respect to Figures 1-5.

[0100]

[0094] Although shown as a separate circuit in FIG. 6 for clarity, the machine learning circuit 626 may be implemented in other processing devices of the processing system 600, such as the CPU 602, the GPU 604, the DSP 606, the NPU 608, etc.

[0101] Generally, the processing system 600 and / or its components may be configured to perform the methods described herein.

[0102]

[0096] Notably, in other aspects, aspects of the processing system 600 may be omitted, such as when the processing system 600 is a server computer, etc. For example, the multimedia components 610, the wireless connectivity 612, the sensors 616, the ISP 618, and / or the navigation components 620 may be omitted in other aspects. Additionally, aspects of the processing system 600 may be distributed among multiple devices.

[0103]

[0097] The shown components, and other non-shown components, may be configured to perform various aspects of the methods described herein.

[0104] Example Clauses

[0098] Clause 1: A method comprising: receiving an input tensor having a frequency dimension and a time dimension; processing the input tensor using a first convolution operation to generate a multidimensional intermediate feature map having a frequency dimension and a time dimension; converting the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension using a frequency dimension reduction operation; processing the one-dimensional intermediate feature map using a second convolution operation to generate a time feature map; expanding the time feature map into the frequency dimension using a broadcasting operation to generate a multidimensional output feature map; and augmenting the multidimensional output feature map with the multidimensional intermediate feature map via a first residual connection.

[0105]

[0099] Clause 2: The method of clause 1, wherein the multi-dimensional intermediate feature map is a two-dimensional intermediate feature map.

[0106]

[0100] Clause 3: The method of any of clauses 1 to 2, further comprising augmenting the multidimensional output feature map with the input tensor via a second residual connection.

[0107]

[0101] Clause 4: A method according to any one of clauses 1 to 3, wherein the first convolution operation uses one or more unit-depth convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension.

[0108]

[0102] Clause 5: A method according to any one of clauses 1 to 4, in which the input tensor is output from a pointwise convolution operation configured to change the number of channels in the input tensor.

[0109]

[0103] Clause 6: A method described in any one of clauses 1 to 5, further comprising performing a sub-spectral normalization (SSN) operation on the multi-dimensional intermediate feature map prior to converting the multi-dimensional intermediate feature map into a one-dimensional intermediate feature map.

[0110]

[0104] Clause 7: A method according to any one of clauses 1 to 6, wherein the SSN operation comprises dividing the multi-dimensional intermediate feature map into a plurality of subbands in the frequency dimension and performing batch normalization on each subband of the plurality of subbands.

[0111]

[0105] Clause 8: A method according to any one of clauses 1 to 7, wherein the frequency dimension reduction operation comprises at least one of a max pooling operation, an average pooling operation, or a convolution operation.

[0112]

[0106] Clause 9: A method according to any one of clauses 1 to 8, wherein the second convolution operation comprises a depthwise separable convolution operation, wherein the depthwise convolution of the depthwise separable convolution operation is configured to use one or more depthwise convolution kernels having a size equal to one in the frequency dimension and greater than one in the time dimension.

[0113]

[0107] Clause 10: A method according to any one of clauses 1 to 9, wherein the point-wise convolution of the depthwise separable convolution operation is configured to use one or more point-wise convolution kernels after the depthwise convolution.

[0114]

[0108] Clause 11: A method according to any one of clauses 1 to 10, wherein the input tensor comprises input audio features and the first convolution operation and the second convolution operation are part of a broadcast residual neural network configured to classify the input audio features.

[0115]

[0109] Clause 12: A system comprising means for carrying out the method according to any one of clauses 1 to 11.

[0116]

[0110] Clause 13: A system comprising a memory having computer-executable instructions and one or more processors, wherein the one or more processors are configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of clauses 1 to 11.

[0117]

[0111] Clause 14: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method described in any one of clauses 1 to 11.

[0118]

[0112] Clause 15: A computer program product embodied on a computer-readable storage medium comprising code for performing the method according to any one of clauses 1 to 11.

[0119] Additional Considerations

[0113] The above description is provided to enable those skilled in the art to implement various aspects described herein. The examples described herein are not intended to limit the scope, applicability, or aspects described in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made in the function and configuration of the elements described without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components, as appropriate. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be performed using any number of the aspects described herein. Furthermore, the scope of the disclosure is intended to cover such apparatus or methods implemented using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be implemented by one or more elements of a claim.

[0120]

[0114] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0121]

[0115] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0122]

[0116] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Also, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0123]

[0117] As used herein, the term "connected to" in the context of sharing electronic signals and data between elements described herein may generally mean in data communication between the respective elements connected to each other. In some cases, the elements may be directly connected to each other, such as through one or more conductive traces, lines, or other conductive carriers capable of carrying signals and / or data between the respective elements directly connected to each other. In other cases, the elements may be indirectly connected to each other, such as through one or more data buses or similar shared circuitry and / or integrated circuit elements for communicating signals and data between the respective elements indirectly connected to each other.

[0124]

[0118] The methods disclosed herein comprise one or more steps or actions for achieving the method. The steps and / or actions of the method may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. Those means may include various hardware and / or software components and / or modules, including, but not limited to, circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are operations illustrated in the figures, those operations may have corresponding counterpart means-plus-function components with similar numbering.

[0125]

[0119] The following claims are not limited to the embodiments set forth herein, but are to be accorded the full scope consistent with the language of the claims. In the claims, reference to an element in the singular does not mean "one and only," but rather "one or more," unless expressly stated otherwise. The term "some" refers to one or more, unless expressly stated otherwise. No claim element is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase "means for," or, in the case of a method claim, unless the element is recited using the phrase "step for." All structural and functional equivalents of the elements of the various embodiments described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and are encompassed by the claims. Moreover, nothing disclosed herein is made public, regardless of whether such disclosure is expressly recited in the claims.

Claims

1. receiving an input tensor having a frequency dimension and a time dimension, wherein the input tensor comprises input audio data or input audio features; processing the input tensor with a first convolution operation to generate a multi-dimensional intermediate feature map having the frequency dimension and the time dimension; converting the multi-dimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension using a frequency dimension reduction operation; processing the one-dimensional intermediate feature map using a second convolution operation to generate a temporal feature map, wherein the first convolution operation and the second convolution operation are part of a broadcast residual neural network configured to classify the input audio data or the input audio features. expanding the time feature map into the frequency dimension using a broadcasting operation to generate a multi-dimensional output feature map; augmenting the multi-dimensional output feature map with the multi-dimensional intermediate feature map via a first residual connection; outputting the extended multi-dimensional output feature map; A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1 , wherein the multi-dimensional intermediate feature map is a two-dimensional intermediate feature map, and wherein converting the multi-dimensional intermediate feature map to the one-dimensional intermediate feature map reduces a number of calculations performed by a processor when generating the temporal feature map.

3. 2. The computer-implemented method of claim 1, further comprising augmenting the multidimensional output feature map with the input tensor via a second residual connection.

4. 2. The computer-implemented method of claim 1, wherein the first convolution operation uses one or more unit-depth convolution kernels having a size greater than one in the frequency dimension and equal to one in the time dimension.

5. 5. The computer-implemented method of claim 4, wherein the input tensor is output from a pointwise convolution operation configured to change a number of channels in the input tensor.

6. 2. The computer-implemented method of claim 1, further comprising performing a sub-spectral normalization (SSN) operation on the multi-dimensional intermediate feature map prior to converting the multi-dimensional intermediate feature map to a one-dimensional intermediate feature map.

7. The SSN calculation is dividing the multi-dimensional intermediate feature map into a plurality of sub-bands in the frequency dimension; performing batch normalization on each subband of the plurality of subbands; The computer-implemented method of claim 6 , comprising:

8. The computer-implemented method of claim 1 , wherein the frequency dimensionality reduction operation comprises at least one of a max pooling operation, an average pooling operation, or a convolution operation.

9. 2. The computer-implemented method of claim 1 , wherein the second convolution operation comprises a depthwise separable convolution operation, wherein a depthwise convolution of the depthwise separable convolution operation is configured to use one or more depthwise convolution kernels having a size equal to one in the frequency dimension and greater than one in the time dimension.

10. 10. The computer-implemented method of claim 9, wherein a point-wise convolution of the depthwise separable convolution operation is configured to use one or more point-wise convolution kernels after the depthwise convolution.

11. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of claims 1 to 10.

12. a memory having computer-executable instructions; One or more processors; 11. A processing system comprising: the one or more processors configured to execute the computer-executable instructions, causing the processing system to perform the method of any one of claims 1 to 10.

Images