Broadcast residual learning

Abstract

Some aspects of the present disclosure provide a technique for efficient broadcast residual machine learning. An input tensor with a frequency dimension and a time dimension is received, and the input tensor is processed using a first convolution operation to generate a multi-dimensional intermediate feature map with a frequency dimension and a time dimension. The multi-dimensional intermediate feature map is converted to a one-dimensional intermediate feature map in the time dimension using a frequency dimension reduction operation, and the one-dimensional intermediate feature map is processed using a second convolution operation to generate a time feature map. The time feature map is expanded in the frequency dimension using a broadcasting operation to generate a multi-dimensional output feature map, and the multi-dimensional output feature map is expanded with the multi-dimensional intermediate feature map via the first residual connection.

Description

Technical Field

[0001] Cross - reference to Related Applications

[0001] This application claims the benefit and priority of U.S. Patent Application No. 17 / 656,621, filed on March 25, 2022, which claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 166,161, filed on March 25, 2021, the entire contents of each of which are incorporated herein by reference.

[0002]

[0002] Aspects of the present disclosure relate to machine learning, and more particularly, to efficient data processing.

Background Art

[0003]

[0003] Designing an efficient machine learning architecture is an important topic in neural audio processing. In particular, keyword spotting (KWS), which aims to detect predefined keywords, is becoming increasingly important. KWS plays an important role in device activation and user interaction on smart devices. However, it is difficult to provide a model that operates efficiently while minimizing errors. Model efficiency is particularly important in KWS because the process is generally implemented on edge devices (such as devices with limited resources, such as mobile phones, smart speakers, and Internet of Things (IoT) devices) and at the same time requires low latency.

[0004]

[0004] Therefore, there is a need for systems and methods for providing high - precision classification using efficient model design.

Summary of the Invention

[0005]

[0005] Some embodiments provide a method comprising: receiving an input tensor having a frequency dimension and a temporal dimension; processing the input tensor using a first convolution operation to generate a multidimensional intermediate feature map having a frequency dimension and a temporal dimension; converting the multidimensional intermediate feature map to a one-dimensional intermediate feature map in the temporal dimension using a frequency dimension reduction operation; processing the one-dimensional intermediate feature map using a second convolution operation to generate a temporal feature map; expanding the temporal feature map to the frequency dimension using a broadcasting operation to generate a multidimensional output feature map; and augmenting the multidimensional output feature map using the multidimensional intermediate feature map via a first residual connection.

[0006]

[0006] Another embodiment provides a processing system configured to carry out the above-described method and the method described herein; a non-transitory computer-readable medium having instructions that, when executed by one or more processors of the processing system, cause the processing system to carry out the above-described method and the method described herein; a computer program product executed on a computer-readable storage medium having code for carrying out the above-described method and the method described herein; and a processing system having means for carrying out the above-described method and the method described herein.

[0007]

[0007] The following description and related drawings detail some exemplary features of one or more embodiments.

[0008]

[0008] The attached figures show some of one or more embodiments and should therefore not be considered to limit the scope of the present disclosure. [Brief explanation of the drawing]

[0009] [Figure 1]

[0009] A diagram illustrating an exemplary workflow for broadcasted residual learning. [Figure 2]

[0010] An exemplary block diagram for residual learning techniques. [Figure 3]

[0011] A diagram illustrating an exemplary broadcast residual learning block for use in efficient processing of input data. [Figure 4]

[0012] A diagram illustrating an exemplary broadcast residual learning block for use in the efficient processing of input data in the transitional layer. [Figure 5]

[0013] An illustrative flowchart showing how to process data using broadcast residual learning. [Figure 6]

[0014] A diagram illustrating an exemplary processing system configured to implement various aspects of this disclosure. [Modes for carrying out the invention]

[0010]

[0015] For ease of understanding, the same reference numerals are used to designate identical elements common to the drawings where possible. It is intended that elements and features of one embodiment can be usefully incorporated into other embodiments without further description.

[0011]

[0016] Aspects of this disclosure provide techniques for broadcasted residual learning. The techniques described herein offer higher model accuracy and significantly improved computational efficiency (e.g., smaller model size and lighter computational load) compared to existing methods.

[0012]

[0017] A wide variety of efficient convolutional neural networks (CNNs) have been developed recently. Generally, CNNs consist of repeating blocks of the same structure and are often based on residual learning and depthwise separable convolution. This has given rise to several CNN-based KWS methods. Existing methods use either one-dimensional time convolution or two-dimensional (e.g., frequency and time) convolution. Each method has its own advantages and disadvantages.

[0013]

[0018] For example, models using one-dimensional time convolution generally require fewer computing resources compared to models relying on two-dimensional methods. However, one-dimensional convolution cannot capture the internal bias of the convolution (such as translation equivariance) in terms of the frequency dimension.

[0014]

[0019] On the other hand, two-dimensional convolution-based methods require significantly more computational resources than one-dimensional methods, even when using efficient designs and architectures such as depth-separable convolution. This can prevent such two-dimensional methods from being useful for a wide variety of devices and implementations.

[0015]

[0020] The broadcast residual learning techniques described herein can be used to efficiently process data both during training (while training data is passed through the model) and during runtime (when new data is passed through to generate inferences).

[0016]

[0021] In some embodiments, broadcast residual learning is used to process and classify audio data and features (for example, to perform KWS). Generally, audio data and features can be represented using two-dimensional tensors (for example, having frequency and time dimensions). While audio is used in the examples herein, embodiments of this disclosure can be readily applied to a wide variety of data.

[0017]

[0022] In some embodiments, broadcast residual learning generally involves performing convolution on an input tensor to extract 2D features, reducing the dimensionality of the 2D features to enable efficient convolution on the features (e.g., requiring reduced computation, processing steps, and energy), expanding the resulting tensor to the original dimensionality of the 2D features, and extending the expanded tensor using the original 2D features. In some embodiments, the expanded tensor is further extended using the original input tensor.

[0018]

[0023] In some embodiments, the broadcast residual learning described herein may be implemented in a neural network architecture to perform a variety of tasks, such as classifying input audio. For example, the technique described herein may be implemented as a broadcast residual learning block, and several of these blocks may be used sequentially within a neural network architecture.

[0019]

[0024] Advantageously, broadcast residual learning retains many of the residual capabilities of a one-dimensional time convolution, while still allowing two-dimensional convolutions to be used together via broadcast residual connections that extend the time output to the frequency dimension. This residual mapping allows the network to effectively represent useful audio features using far fewer computations than conventional convolutional neural networks, thereby reducing computational complexity, latency, compute requirements, memory requirements, etc. In aspects, the broadcast residual learning techniques described herein can achieve state-of-the-art accuracy on speech command datasets using fewer computations and parameters compared to conventional systems.

[0020] Example Workflow for Broadcasted Residual Learning

[0025] Figure 1 shows an exemplary workflow 100 for broadcasted residual learning. The workflow 100 begins with an input tensor 105. In some examples, the tensor 105 can be audio data (e.g., represented by a log Mel spectrogram that shows the spectrum of frequencies over time) or audio features (e.g., features generated by processing audio data). In some aspects, the input tensor 105 is a two-dimensional tensor having a frequency dimension and a time dimension. The time dimension can be partitioned into time intervals or steps, and the frequency dimension is partitioned based on frequency values or bands. The frequencies present at each interval (e.g., the loudness of the sound at each frequency) can be reflected via the values in the tensor.

[0021]

[0026] The input tensor 105 is processed using a first convolutional operation 110, which results in a set of two-dimensional feature maps 115. As shown, the feature maps 115 have a dimensionality H×W×c, where H and W are spatial dimensions (e.g., the time dimension and the frequency dimension, respectively), and c is the number of channels.

[0022]

[0027] In one aspect, the convolution operation 110 is a depthwise convolution implemented using one or more kernels configured to extract features in the frequency dimension. For example, the convolution operation 110 may use an n×1 kernel, where n corresponds to the frequency dimension. That is, the depthwise kernel for the convolution operation 110 may have a length greater than 1 in the frequency dimension and a length of 1 in the time dimension. This enables the convolution operation 110 to act as a frequency depthwise convolution that extracts frequency features (e.g., feature map 115) for the tensor 105.

[0023]

[0028] As shown, these feature maps 115 are two-dimensional (having a length greater than 1 in both the frequency dimension and the time dimension). In the illustrated workflow 100, a dimension reduction operation 120 is performed to reduce the dimensionality of the feature map 115. Specifically, the dimension reduction operation 120 may reduce the feature map 115 so as to eliminate the frequency dimension and preserve the time dimension. This results in a one-dimensional feature map 125. The feature map 125 may have the same time dimensionality and the same number of channels as the feature map 115, but has a length of 1 in the frequency dimension.

[0024]

[0029] The dimension reduction operation 120 may generally be performed for each frequency (or for each frequency band) and may include various techniques, including max pooling (such that the feature with the maximum value, or the most activated presence, is retained), average pooling (such that the average value is retained), min pooling (such that the minimum value is retained), etc. In some aspects, the dimension reduction operation 120 may also be performed by convolving the feature map 115 using an H×1 kernel without padding to reduce the dimension, where H corresponds to the size of the frequency dimension.

[0025]

[0030] Advantageously, the one-dimensional feature map 125 (corresponding to the time dimension) can be convolved using significantly fewer computational resources compared to conventional two-dimensional convolution. This significantly improves the efficiency of broadcast residual learning.

[0026]

[0031] As shown in the figure, the feature map 125 is processed using a second convolution operation 130. In some embodiments, the convolution operation 130 is a depth-separable convolution (e.g., a depth-based convolution followed by a pointwise convolution). In contrast to the convolution operation 110 (corresponding to the frequency dimension), the convolution operation 130 may be performed using one or more kernels configured to extract features about the time dimension. For example, the convolution operation 130 may use a 1 × m kernel, where m corresponds to the time dimension.

[0027]

[0032] That is, the depth-unit kernel for the convolution operation 130 may have a length greater than 1 in the time dimension and a length of 1 in the frequency dimension. This allows the convolution operation 130 to function as a temporal depthwise convolution that extracts temporal features for the feature map 125. In some embodiments, the convolution operation 130 may be a depth-unit separable convolution. In such embodiments, following the temporal depth convolution, the convolution operation 130 may apply one or more point-unit kernels. This results in the feature map 135.

[0028]

[0033] In workflow 100, the feature map 135 is then broadcast to the frequency dimension, as indicated by arrow 137. This broadcasting operation (also called augmentation) generally transforms the one-dimensional feature map 135 into a multi-dimensional feature map 140 having the same dimensionality as feature map 115. In some embodiments, the broadcasting involves copying and stacking the feature maps 135 (in this example) until they reach a height of H.

[0029]

[0034] The residual connection 150 reflects the residual properties of broadcast residual learning. In workflow 100, the input tensor 105 is augmented with feature map 140 using operation 145 to produce output 155. In some embodiments, feature map 140 may be augmented with feature map 115, either similarly or alternatively. This operation 145 can generally include any number of combination techniques, including element-wise summation, averaging, multiplication, etc. Advantageously, the residual connection 150 allows the system to retain the 2D features of the input despite the dimensionality reduction operation 120.

[0030] Example Residual Learning Techniques

[0035] Figure 2 shows exemplary block diagrams 200A and 200B for residual learning techniques.

[0031]

[0036] Block 200A reflects a conventional residual block used in several residual models. This block 200A can be represented as y = x + f(x), where x and y are the input and output features, respectively, and the function f(·) calculates the convolutional output. The identity shortcut of x and the result of f(x) are of the same dimensionality and can be added by simple element-wise addition.

[0032]

[0037] More specifically, as shown by residual block 200A, the input 205 is processed using a convolution operation 210. The resulting tensor can then be added to the original input 205 (via the identification shortcut 215), as shown by operation 220. This yields the output 225 of the normal residual block 200A.

[0033]

[0038] In aspects of this disclosure, in order to utilize both one-dimensional and two-dimensional features together, the function f(x) (reflected by the convolution operation 210) may be decomposed into f1 and f2, which correspond to a time operation and a two-dimensional operation, respectively. This is reflected in the broadcast residual block 200B.

[0034]

[0039] Broadcast residual block 200B is,

[0035]

number

[0036] It can be expressed as,

[0040] Here, x and y are the input and output features, respectively, f1 and f2 are convolution operations, BC(·) is a broadcasting or augmentation operation, and reduction(·) is a dimensionality reduction operation (e.g., average pooling by frequency dimension). In this equation, batch and channel dimensions are ignored for conceptual clarity, and the input feature x is:

[0037]

number

[0038] This is the case where H and W are the frequency step and time step, respectively.

[0039]

[0041] As shown by residual block 200B, the input 250 is processed using a convolution operation 255 to extract two-dimensional features. The resulting tensor can then be reduced using dimensionality reduction 260, and the (one or more) reduced tensors are processed using a convolution operation 265 to extract temporal features. These features are then expanded to the frequency dimension and extended using the original input 250 via an identification information shortcut 270, which yields output 280.

[0040] Example Broadcasted Residual Learning Block

[0042] Figure 3 shows an exemplary broadcast residual learning block 300 for use in the efficient processing of input data, such as audio input data.

[0041]

[0043] As shown in the figure, the input tensor 305 is received and processed using the first operation 310 (labeled f2 in Figure 3). Operation 310 corresponds to the two-dimensional feature extraction (e.g., convolution operation 110) described above.

[0042]

number

[0043] This yields a two-dimensional feature map (for example, feature map 115 in Figure 1). As shown in the figure, the convolution operation 310 is performed using a frequency-depth unit convolution 320 comprising one or more n × 1 frequency-depth unit convolution kernels.

[0044]

[0044] As shown in the figure, operation 310 also includes a subspectral normalization (SSN) operation 325. The SSN operation 325 generally involves splitting the input features (generated by the frequency-depth unit convolution 320) into subbands in the frequency dimension and normalizing each subband separately (for example, using batch normalization). This allows the system to achieve frequency-aware time features compared to normal batch normalization on the entire feature set.

[0045]

[0045] The system can then perform dimensionality reduction using operation 330. In the illustrated example, the broadcast residual learning block 300 uses frequency-averaged pooling to average the input features by frequency, which is as described above.

[0046]

number

[0047] This produces the characteristics shown in (for example, feature map 125 in Figure 1).

[0048]

[0046] These features are then processed using a second operation 340 (labeled f1 in Figure 3). Operation 320 may correspond to the time convolution operation described above (e.g., convolution operation 130). In one embodiment, operation 340 is a depth-separable convolution (e.g., a combination of a time-depth convolution 345 and a point-level convolution 355).

[0049]

[0047] The time-depth unit convolution 345 may comprise one or more 1×m time-depth unit convolution kernels for generating time features (for example, feature map 135 in Figure 1).

[0050]

[0048] As shown in the figure, operation 340 then includes a batch normalization operation 350 followed by a swish activation (also indicated by 350). The swish activation is shown in Figure 3, but in some embodiments, any suitable activation function may be used.

[0051]

[0049] Following the point-wise convolution 355, operation 340 may also include a channel-wise dropout (indicated by 360) at a dropout rate p. This dropout may be used as a regularization for the model to prevent overfitting and improve generalization. A broadcasting operation (which may correspond to the broadcasting operation 137 in Figure 1), represented by operation 365 (which also includes the tensor extension described above with reference to operation 145 in Figure 1), is then performed,

[0052]

number

[0053] Features from operation 340 in

[0054]

number

[0055] It can be used to expand to [a certain extent].

[0056]

[0050] In some embodiments, in order to be frequency convolution aware over a series of blocks (e.g., a series application of the broadcast residual learning block 300), the system uses not only a residual connection 315 (sometimes called an "identification shortcut") to extend the features using the original input 305 (in operation 365), but also an auxiliary residual connection 335 from the 2D features output by the frequency-depth unit convolution 320 (in operation 365). This auxiliary residual connection 335 allows the system to retain the frequency-aware features of the input despite the dimensionality reduction operation. The output of this broadcasting and extension operation 365 (also called a broadcast sum operation in some embodiments) may then be processed using one or more activation functions (e.g., a ReLU function 370) and then provided as output 375 from the residual learning block 300.

[0057]

[0051] In this way, the broadcast residual learning block 300 can be expressed as y = x + f2(x) + BC(f1(reduction(f2(x)))), where x and y are the input and output features, respectively, f1 and f2 are convolution operations, BC(·) is a broadcasting or augmentation operation, and reduction(·) is a dimensionality reduction operation (e.g., average pooling by frequency dimension).

[0058]

[0052] Using the broadcast residual learning block 300, a machine learning model can provide a more efficient KWS compared to conventional techniques, for example, while preserving 2D features. By performing unit time-depth and unit point convolution on 1D time features, the computational load is reduced to 1 / frequency step H (often 40 or more) compared to conventional 2D unit depth separable convolution.

[0059] Example Transitional Broadcasted Residual Learning Block

[0053] Figure 4 shows an exemplary transition broadcast residual learning block 400 for use in the efficient processing of input data, such as audio input data.

[0060]

[0054] The transition broadcast residual learning block 400 is similar to the regular broadcast residual learning block 300, with two differences: the transition broadcast residual learning block 400 can be used in a transition layer where the number of channels in input 305 is different from the number of channels in output 475.

[0061]

[0055] In detail, operation 410 replaces operation 310 in Figure 3. Operation 410 includes an additional point-by-point convolution 412, which is used to change the number of channels in input 405 to a desired number of channels for output 475. As shown in the figure, this point-by-point convolution 412 may be followed by batch normalization and an activation function (such as ReLU), indicated by 413.

[0062]

[0056] The second difference between the transition broadcast residual learning block 400 and the normal broadcast residual learning block 300 is that the transition broadcast residual learning block 400 does not include an identification information shortcut (residual connection 315 in Figure 3). That is, the transition broadcast residual learning block 400 does not extend the output using input 405 (because of the difference in dimensionality).

[0063]

[0057] In other respects, the transition broadcast residual learning block 400 largely mirrors the normal broadcast residual learning block 300 described above with reference to Figure 3.

[0064] Example Method for Broadcasted Residual Learning

[0058] Figure 5 is an illustrative flowchart illustrating a method 500 for processing data using broadcast residual learning.

[0065]

[0059] Method 500 begins in block 505, where the processing system receives an input tensor having a frequency dimension and a time dimension.

[0066]

[0060] In block 510, the processing system processes the input tensor using a first convolution operation to generate a multidimensional intermediate feature map having a frequency dimension and a time dimension. In some cases, the multidimensional intermediate feature map is a two-dimensional intermediate feature map.

[0067]

[0061] In some embodiments, the first convolution operation uses one or more depthwise convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension.

[0068]

[0062] In some embodiments, the input tensor is output from a pointwise convolution operation configured to change the number of channels in the input tensor.

[0069]

[0063] In block 515, the processing system uses frequency dimension reduction operations to convert the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension.

[0070]

[0064] In some embodiments, the frequency dimensionality reduction operation comprises at least one of the following: a maximum pooling operation, an average pooling operation, or a convolution operation.

[0071]

[0065] In some embodiments, the method 500 further comprises performing a subspectral normalization (SSN) operation on the multidimensional intermediate feature map before converting the multidimensional intermediate feature map to a one-dimensional intermediate feature map.

[0072]

[0066] In some embodiments, the SSN operation comprises dividing the multidimensional intermediate feature map into multiple subbands in the frequency dimension and performing batch normalization on each of the multiple subbands.

[0073]

[0067] In block 520, the processing system processes the one-dimensional intermediate feature map using a second convolution operation to generate a time feature map.

[0074]

[0068] In some embodiments, the second convolution operation comprises a depthwise separable convolution operation, wherein the depthwise convolution of the depthwise separable convolution operation is configured to use one or more depthwise convolution kernels having a size equal to 1 in the frequency dimension and greater than 1 in the time dimension.

[0075]

[0069] In some embodiments, the pointwise convolution of a depth-separable convolution operation is configured to use one or more pointwise convolution kernels after the depth-separable convolution.

[0076]

[0070] In block 525, the processing system uses a broadcasting operation to expand the time feature map to the frequency dimension in order to generate a multidimensional output feature map.

[0077]

[0071] In block 530, the processing system extends the multidimensional output feature map using the multidimensional intermediate feature map via the first residual connection.

[0078]

[0072] In some embodiments, the method 500 further includes outputting an extended multidimensional output (for example, from a residual block, as an output to another residual block or other block or layer of the model, as an output from a mode, etc.).

[0079]

[0073] In some embodiments, the method 500 further comprises extending the multidimensional output feature map using an input tensor via a second residual connection.

[0080]

[0074] In some embodiments, the input tensor comprises an input audio feature, and the first and second convolution operations are part of a broadcast residual neural network configured to classify the input audio feature.

[0081] Example Processing System for Broadcasted Residual Learning

[0075] In some embodiments, the techniques, methods, and workflows described with respect to Figures 1 to 5 may be performed on one or more devices.

[0082]

[0076] Figure 6 shows an exemplary processing system 600 that can be configured to carry out various aspects of the methods described herein, including, for example, the methods described with respect to Figures 1 to 5.

[0083]

[0077] The processing system 600 includes a central processing unit (CPU) 602, which in some examples may be a multi-core CPU. Instructions to be executed in the CPU 602 may be loaded, for example, from program memory associated with the CPU 602 or from a memory partition 624.

[0084]

[0078] The processing system 600 also includes additional processing components adapted to specific functions, such as a graphics processing unit (GPU) 604, a digital signal processor (DSP) 606, a neural processing unit (NPU) 608, a multimedia processing unit 610, and a wireless connectivity component 612.

[0085]

[0079] 608, for example, an NPU is a special circuit configured to implement all the necessary controls and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), visual processing units (VPUs), or graph processing units.

[0086]

[0080] NPUs, such as 608, are configured to accelerate performance on common machine learning tasks, including image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples, multiple NPUs may be part of a dedicated neural network accelerator.

[0087]

[0081] The NPU may be optimized for training or inference, or in some cases, configured to balance performance between both. In the case of an NPU capable of performing both training and inference, the two tasks can still be performed independently in general.

[0088]

[0082] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation involving inputting existing datasets (often labeled or tagged), iterating over those datasets, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimizing based on incorrect predictions involves backpropagating through layers of the model and determining gradients to reduce prediction errors.

[0089]

[0083] NPUs designed to accelerate inference are generally configured to operate on a complete model. Thus, such an NPU may be configured to take new data as input and process it rapidly through an already trained model in order to generate model outputs (e.g., inferences).

[0090]

[0084] In one implementation configuration, the NPU608 is one or more parts of the CPU602, GPU604, and / or DSP606.

[0091]

[0085] In some examples, the wireless connectivity component 612 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE®), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi® connectivity, Bluetooth® connectivity, and other wireless data transmission standards. The wireless connectivity processing component 612 is further connected to one or more antennas 614.

[0092]

[0086] The processing system 600 may also include one or more sensor processing units 616 associated with any form of the sensor, one or more image signal processors (ISPs) 618 associated with any form of the image sensor, and / or a navigation processor 620, the navigation processor 620 may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0093]

[0087] The processing system 600 may also include one or more input and / or output devices 622, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, and a microphone.

[0094]

[0088] In some examples, one or more of the processors of the processing system 600 may be based on an ARM or RISC-V instruction set.

[0095]

[0089] The processing system 600 also includes memory 624 which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this example, memory 624 includes computer executable components which can be executed by one or more of the processors of the processing system 600 described above.

[0096]

[0090] In particular, in this example, the memory 624 includes a machine learning component 624A which may be configured according to one or more embodiments described herein. For example, the machine learning component 624A may provide data or audio analysis using one or more machine learning models (e.g., neural networks) which consist of one or more broadcast residual learning blocks.

[0097]

[0091] The memory 624 further includes a set of (one or more) frequency depth unit kernels 624B and a set of (one or more) time depth unit kernels 624C. As described above, the frequency depth unit kernels 624B generally include one-dimensional kernels with a length greater than 1 in the frequency dimension, and the time depth unit kernels 624C include one-dimensional kernels with a length greater than 1 in the time dimension.

[0098]

[0092] The frequency-depth unit kernel 624B can generally be used to perform frequency-depth unit convolution (e.g., convolution operation 110 in Figure 1), and the time-depth unit kernel 624C can generally be used to perform time-depth unit convolution (e.g., convolution operation 130 in Figure 1).

[0099]

[0093] The processing system 600 further comprises a machine learning circuit 626, such as those described above with respect to Figures 1 to 5.

[0100]

[0094] Although shown as a separate circuit in Figure 6 for clarity, the machine learning circuit 626 can be implemented in other processing devices of the processing system 600, such as the CPU 602, GPU 604, DSP 606, and NPU 608.

[0101]

[0095] Generally, the processing system 600 and / or its components may be configured to carry out the methods described herein.

[0102]

[0096] In particular, in other embodiments, the embodiments of the processing system 600 may be omitted, for example, when the processing system 600 is a server computer. For example, the multimedia components 610, wireless connectivity 612, sensors 616, ISP 618, and / or navigation components 620 may be omitted in other embodiments. Furthermore, the embodiments of the processing system 600 may be distributed among multiple devices.

[0103]

[0097] The shown components, and other unshown components, may be configured to carry out various aspects of the methods described herein.

[0104] Example Clauses

[0098] Clause 1: A method comprising: receiving an input tensor having a frequency dimension and a time dimension; processing the input tensor using a first convolution operation to generate a multidimensional intermediate feature map having a frequency dimension and a time dimension; converting the multidimensional intermediate feature map to a one-dimensional intermediate feature map in the time dimension using a frequency dimension reduction operation; processing the one-dimensional intermediate feature map using a second convolution operation to generate a time feature map; expanding the time feature map to the frequency dimension using a broadcasting operation to generate a multidimensional output feature map; and extending the multidimensional output feature map using the multidimensional intermediate feature map via a first residual connection.

[0105]

[0099] Clause 2: The method according to Clause 1, wherein the multidimensional intermediate feature map is a two-dimensional intermediate feature map.

[0106]

[0100] Clause 3: The method according to any one of Clauses 1 to 2, further comprising extending the multidimensional output feature map using an input tensor via a second residual connection.

[0107]

[0101] Clause 4: The method according to any one of Clauses 1 to 3, wherein the first convolution operation uses one or more depth unit convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension.

[0108]

[0102] Clause 5: The method according to any one of Clauses 1 to 4, wherein the input tensor is output from a point-by-point convolution operation configured to change the number of channels in the input tensor.

[0109]

[0103] Clause 6: The method according to any one of Clauses 1 to 5, further comprising performing a subspectral normalization (SSN) operation on the multidimensional intermediate feature map before converting the multidimensional intermediate feature map to a one-dimensional intermediate feature map.

[0110]

[0104] Clause 7: The method according to any one of Clauses 1 to 6, wherein the SSN operation comprises dividing a multidimensional intermediate feature map into multiple subbands in the frequency dimension and performing batch normalization on each of the multiple subbands.

[0111]

[0105] Clause 8: The method according to any one of Clauses 1 to 7, wherein the frequency dimensionality reduction operation comprises at least one of a maximum pooling operation, an average pooling operation, or a convolution operation.

[0112]

[0106] Clause 9: The method according to any one of Clauses 1 to 8, wherein the second convolution operation comprises a depth-separable convolution operation, wherein the depth-unit convolution of the depth-unit separable convolution operation is configured to use one or more depth-unit convolution kernels having a size equal to 1 in the frequency dimension and greater than 1 in the time dimension.

[0113]

[0107] Clause 10: The method according to any one of Clauses 1 to 9, wherein the point-by-point convolution of a depth-unit separable convolution operation is configured to use one or more point-by-point convolution kernels after the depth-unit convolution.

[0114]

[0108] Clause 11: The method according to any one of Clauses 1 to 10, wherein the input tensor comprises input audio features, and a first convolution and a second convolution are part of a broadcast residual neural network configured to classify the input audio features.

[0115]

[0109] Clause 12: A system comprising means for carrying out the method described in any one of Clauses 1 to 11.

[0116]

[0110] Clause 13: A system comprising memory having computer-executable instructions and one or more processors, wherein one or more processors are configured to execute computer-executable instructions and cause a processing system to carry out the method described in any one of Clauses 1 to 11.

[0117]

[0111] Clause 14: A non-temporary computer-readable medium comprising computer-executable instructions, wherein when the computer-executable instructions are executed by one or more processors of the processing system, the processing system causes the processing system to carry out the method described in any one of Clauses 1 to 11.

[0118]

[0112] Clause 15: A computer program product run on a computer-readable storage medium, comprising code for performing the method described in any one of Clauses 1 to 11.

[0119] Additional Considerations

[0113] The above description is provided to enable a person skilled in the art to carry out the various embodiments described herein. The examples described herein do not limit the scope, applicability, or embodiments described in the claims. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may apply to other embodiments. For example, changes may be made to the function and configuration of the elements described without departing from the scope of this disclosure. Various examples may, as appropriate, omit, replace, or add various procedures or components. For example, the methods described may be carried out in an order different from the order described, and various steps may be added, omitted, or combined. Also, features described in some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be carried out using any number of embodiments described herein. Furthermore, the scope of this disclosure shall cover, in addition to or in addition to, the various embodiments of this disclosure described herein, such apparatus or methods carried out using other structures, functions, or structures and functions. It should be understood that any embodiment of the disclosure disclosed herein may be carried out by one or more elements of the claims.

[0120]

[0114] As used herein, the term “exemplary” means “acting as an example, case, or illustration.” No embodiment described herein as “exemplary” should necessarily be construed as being preferable or advantageous to any other embodiment.

[0121]

[0115] As used herein, the phrase “at least one of” the list of items means any combination of those items that contains a single member. For example, “at least one of a, b, or c” shall include a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (for example, aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0122]

[0116] As used herein, the term “deciding” encompasses a wide variety of actions. For example, “deciding” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database or another data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0123]

[0117] As used herein, in the context of sharing electronic signals and data between the elements described herein, the term “connected” may generally mean data communication between each connected element. In some cases, elements may be directly connected to each other, such as via one or more conductive traces, lines, or other conductive carriers capable of carrying signals and / or data between each directly connected element. In other cases, elements may be indirectly connected to each other, such as via one or more data buses or similar shared circuits and / or integrated circuit elements for communicating signals and data between each indirectly connected element.

[0124]

[0118] The methods disclosed herein comprise one or more steps or actions for achieving the method. The steps and / or actions of the Method may be interchangeable with one another without departing from the claims. In other words, unless a particular order of steps or actions is specified, the order and / or use of any particular steps and / or actions may be modified without departing from the claims. Furthermore, various operations of the Method described above may be carried out by any preferred means capable of performing the corresponding function. These means may include, but are not limited to, a variety of hardware and / or software components and / or modules, including circuits, application-specific integrated circuits (ASICs), or processors. Generally, where there are operations shown in the figures, those operations may have corresponding means-plus-function components of a corresponding counterpart with a similar number.

[0125]

[0119] The following claims are not limited to the embodiments shown herein and should be given the full scope consistent with the language of the claims. In the claims, a singular reference to an element means "one or more" and not "one unique" unless otherwise explicitly stated. Unless otherwise explicitly stated, the term "several" means one or more. No claim element should be construed under Section 112(f) of the United States Patent Act unless it is explicitly stated using the phrase "means for" or, in the case of a method claim, the phrase "steps for". All structural and functional equivalents of the elements in various embodiments described throughout this disclosure, known to or to those skilled in the art, are expressly incorporated by reference herein and are encompassed by the claims. Furthermore, nothing disclosed herein is made public, whether such disclosure is expressly stated in the claims or not. The invention described in the original claims of this application is listed below. [C1] Receiving an input tensor having frequency dimensions and time dimensions, To generate a multidimensional intermediate feature map having the frequency dimension and the time dimension, the input tensor is processed using a first convolution operation, Converting the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension using frequency dimension reduction calculations, To generate a time feature map, the one-dimensional intermediate feature map is processed using a second convolution operation, To generate a multidimensional output feature map, a broadcasting operation is used to expand the time feature map to the frequency dimension, The multidimensional output feature map is extended using the multidimensional intermediate feature map via a first residual connection, Outputting the aforementioned extended multidimensional output feature map A computer implementation method comprising the following features. [C2] The computer implementation method according to C1, wherein the multidimensional intermediate feature map is a two-dimensional intermediate feature map, and converting the multidimensional intermediate feature map to the one-dimensional intermediate feature map reduces the number of calculations performed by the processor when generating the time feature map. [C3] The computer implementation method according to C1, further comprising extending the multidimensional output feature map using the input tensor via a second residual connection. [C4] The computer implementation method according to C1, wherein the first convolution operation uses one or more depth unit convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension. [C5] The computer implementation method according to C4, wherein the input tensor is output from a point-by-point convolution operation configured to change the number of channels in the input tensor. [C6] The computer implementation method according to C1, further comprising performing a subspectral normalization (SSN) operation on the multidimensional intermediate feature map before converting the multidimensional intermediate feature map to a one-dimensional intermediate feature map. [C7] The SSN calculation is performed as follows: The multidimensional intermediate feature map is divided into multiple subbands in the frequency dimension, Batch normalization is performed on each of the subbands of the aforementioned multiple subbands. A computer implementation method as described in C6, comprising: [C8] The computer implementation method according to C1, wherein the frequency dimensionality reduction operation comprises at least one of a maximum pooling operation, an average pooling operation, or a convolution operation. [C9] The computer implementation method according to C1, wherein the second convolution operation comprises a depth-separable convolution operation, wherein the depth-unit convolution of the depth-unit separable convolution operation is configured to use one or more depth-unit convolution kernels having a size equal to 1 in the frequency dimension and greater than 1 in the time dimension. [C10] The computer implementation method according to C9, wherein the point-by-point convolution of the depth-unit separable convolution operation is configured to use one or more point-by-point convolution kernels after the depth-unit convolution. [C11] The input tensor comprises input audio features, The computer implementation method according to C1, wherein the first and second convolution operations are part of a broadcast residual neural network configured to classify the input audio features. [C12] A non-temporary computer-readable medium having a computer-executable instruction, wherein when the computer-executable instruction is executed by one or more processors of a processing system, the processing system Receiving an input tensor having frequency and time dimensions, To generate a multidimensional intermediate feature map having the frequency dimension and the time dimension, the input tensor is processed using a first convolution operation, Converting the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension using frequency dimension reduction calculations, To generate a time feature map, the one-dimensional intermediate feature map is processed using a second convolution operation, To generate a multidimensional output feature map, a broadcasting operation is used to expand the time feature map to the frequency dimension, The multidimensional output feature map is extended using the multidimensional intermediate feature map via a first residual connection, Outputting the aforementioned extended multidimensional output feature map A non-temporary computer-readable medium that enables the execution of an operation comprising the following features. [C13] The non-temporal computer-readable medium according to C12, further comprising extending the multidimensional output feature map using the input tensor via a second residual connection. [C14] The non-temporal computer-readable medium according to C12, wherein the first convolution operation uses one or more depth unit convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension. [C15] A non-temporary computer-readable medium according to C14, wherein the input tensor is output from a point-by-point convolution operation configured to change the number of channels in the input tensor. [C16] The non-temporal computer-readable medium according to C12, further comprising performing a subspectral normalization (SSN) operation on the multidimensional intermediate feature map before converting the multidimensional intermediate feature map to a one-dimensional intermediate feature map. [C17] The SSN calculation is performed as follows: The multidimensional intermediate feature map is divided into multiple subbands in the frequency dimension, Batch normalization is performed on each of the subbands of the aforementioned multiple subbands. A non-temporary computer-readable medium as described in C16, comprising the following: [C18] The non-temporary computer-readable medium according to C12, wherein the frequency dimensionality reduction operation comprises at least one of (i) a maximum pooling operation, (ii) an average pooling operation, or (iii) a convolution operation. [C19] The non-temporary computer-readable medium according to C12, wherein the second convolution operation comprises a depth-separable convolution operation, wherein the depth-unit convolution of the depth-unit separable convolution operation is configured to use one or more depth-unit convolution kernels having a size equal to 1 in the frequency dimension and greater than 1 in the time dimension. [C20] The non-temporary computer-readable medium according to C19, wherein the point-by-point convolution of the depth-unit separable convolution operation is configured to use one or more point-by-point convolution kernels after the depth-unit convolution. [C21] The input tensor comprises input audio features, The non-transient computer-readable medium described in C12, wherein the first and second convolution operations are part of a broadcast residual neural network configured to classify the input audio features. [C22] Memory equipped with computer executable instructions, One or more processors and A processing system comprising, wherein one or more processors execute the computer executable instructions, and the processing system Receiving an input tensor having frequency and time dimensions, To generate a multidimensional intermediate feature map having the frequency dimension and the time dimension, the input tensor is processed using a first convolution operation, Converting the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension using frequency dimension reduction calculations, To generate a time feature map, the one-dimensional intermediate feature map is processed using a second convolution operation, To generate a multidimensional output feature map, a broadcasting operation is used to expand the time feature map to the frequency dimension, The multidimensional output feature map is extended using the multidimensional intermediate feature map via a first residual connection, Outputting the aforementioned extended multidimensional output feature map A processing system configured to perform an operation that includes the following: [C23] The processing system according to C22, wherein the operation further comprises extending the multidimensional output feature map using the input tensor via a second residual connection. [C24] The processing system according to C22, wherein the first convolution operation uses one or more depth unit convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension. [C25] The processing system according to C24, wherein the input tensor is output from a point-by-point convolution operation configured to change the number of channels in the input tensor. [C26] The processing system according to C22, further comprising performing a subspectral normalization (SSN) operation on the multidimensional intermediate feature map before converting the multidimensional intermediate feature map to a one-dimensional intermediate feature map. [C27] The SSN calculation is performed as follows: The multidimensional intermediate feature map is divided into multiple subbands in the frequency dimension, Batch normalization is performed on each of the subbands of the aforementioned multiple subbands. A processing system as described in C26, comprising: [C28] The processing system according to C22, wherein the frequency dimensionality reduction operation comprises at least one of (i) a maximum pooling operation, (ii) an average pooling operation, or (iii) a convolution operation. [C29] The processing system according to C22, wherein the second convolution operation comprises a depth-separable convolution operation, wherein the depth-unit convolution of the depth-separable convolution operation is configured to use one or more depth-unit convolution kernels having a size equal to 1 in the frequency dimension and greater than 1 in the time dimension. [C30] Means for receiving an input tensor having frequency dimensions and time dimensions, A means for processing the input tensor using a first convolution operation in order to generate a multidimensional intermediate feature map having the frequency dimension and the time dimension, Means for converting the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension using frequency dimension reduction calculations, Means for processing the one-dimensional intermediate feature map using a second convolution operation in order to generate a time feature map, Means for expanding the time feature map to the frequency dimension using a broadcasting operation in order to generate a multidimensional output feature map, Means for extending the multidimensional output feature map using the multidimensional intermediate feature map via a first residual connection, A processing system equipped with the following features.

Claims

1. Receiving an input tensor having a frequency dimension and a time dimension, wherein the input tensor comprises input audio data or input audio features. To generate a multidimensional intermediate feature map having the frequency dimension and the time dimension, the input tensor is processed using a first convolution operation, Converting the multidimensional intermediate feature map into a one-dimensional intermediate feature map in the time dimension using frequency dimension reduction calculations, To generate a time feature map, a second convolution operation is used to process the one-dimensional intermediate feature map, wherein the first and second convolution operations are part of a broadcast residual neural network configured to classify the input audio data or the input audio features. To generate a multidimensional output feature map, a broadcasting operation is used to expand the time feature map to the frequency dimension, The multidimensional output feature map is extended using the multidimensional intermediate feature map via a first residual connection, Outputting the aforementioned extended multidimensional output feature map A computer implementation method comprising the above.

2. The computer implementation method according to claim 1, wherein the multidimensional intermediate feature map is a two-dimensional intermediate feature map, and converting the multidimensional intermediate feature map to the one-dimensional intermediate feature map reduces the number of calculations performed by the processor when generating the time feature map.

3. The computer implementation method according to claim 1, further comprising extending the multidimensional output feature map using the input tensor via a second residual connection.

4. The computer implementation method according to claim 1, wherein the first convolution operation uses one or more depth unit convolution kernels having a size greater than 1 in the frequency dimension and equal to 1 in the time dimension.

5. The computer implementation method according to claim 4, wherein the input tensor is output from a point-by-point convolution operation configured to change the number of channels in the input tensor.

6. The computer implementation method according to claim 1, further comprising performing a subspectral normalization (SSN) operation on the multidimensional intermediate feature map before converting the multidimensional intermediate feature map to a one-dimensional intermediate feature map.

7. The aforementioned SSN calculation, The multidimensional intermediate feature map is divided into multiple subbands in the frequency dimension, Batch normalization is performed on each of the subbands of the aforementioned multiple subbands. The computer implementation method according to claim 6, comprising:

8. The computer implementation method according to claim 1, wherein the frequency dimensionality reduction operation comprises at least one of a maximum pooling operation, an average pooling operation, or a convolution operation.

9. The computer implementation method according to claim 1, wherein the second convolution operation comprises a depth-separable convolution operation, wherein the depth-unit convolution of the depth-unit separable convolution operation is configured to use one or more depth-unit convolution kernels having a size equal to 1 in the frequency dimension and greater than 1 in the time dimension.

10. The computer implementation method according to claim 9, wherein the point-by-point convolution of the depth-unit separable convolution operation is configured to use one or more point-by-point convolution kernels after the depth-unit convolution.

11. A non-temporary computer-readable medium comprising computer-executable instructions, wherein when the computer-executable instructions are executed by one or more processors of a processing system, the processing system causes the processing system to carry out the method according to any one of claims 1 to 10.

12. Memory equipped with computer executable instructions, One or more processors and A processing system comprising, wherein one or more processors are configured to execute computer executable instructions and cause the processing system to carry out the method according to any one of claims 1 to 10.

Images