Multi-dimensional spatial decomposition for transducer neural networks

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for processing multi-dimensional content using neural networks. One example method generally includes decomposing a multi-dimensional input into a plurality of two-dimensional subspaces, wherein the plurality of two-dimensional subspaces share a common dimension. Generating, via an attention block of the transformer neural network, a first attention matrix based on a projection of the marker in a first one of the plurality of two-dimensional subspaces, and generate, via an attention block of the transformer neural network, a second attention matrix based on a projection of the marker in a second one of the plurality of two-dimensional subspaces. An output of the transformer neural network is generated based on a combination of the first attention matrix and the second attention matrix.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. patent application No. 18 / 193,234, filed March 30, 2023, which is incorporated herein by reference in its entirety.

[0003] introduction

[0004] Aspects of the present disclosure relate to neural networks, and more particularly to using neural networks to process multidimensional content.

[0005] Various machine learning architectures have been used to provide solutions to a wide variety of computational problems. There are a variety of machine learning model architectures, such as artificial neural networks (which may include convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep neural networks, generative adversarial networks (GANs), etc.), random forest models, etc. Transformer neural networks have been increasingly widely used in a variety of image and video processing tasks, or other tasks that process multidimensional data to generate various inferences related to the multidimensional data.

[0006] However, transformer neural networks are often computationally expensive. For example, because visual transformers generally compute self-attention at each block, the computational and memory requirements may grow quadratically with respect to the size of the input data. While the computational complexity of these tasks can be addressed using high-performance processing units (e.g., graphics processing units, neural processing units, and / or other processing units that support a high degree of parallelism) and large amounts of memory, edge devices (e.g., user equipment (UE), such as mobile devices or autonomous vehicles, etc.) may not have sufficient computational resources to process multi-dimensional content (e.g., video content having a time dimension and two spatial dimensions (height and width)) at a performance level sufficient for the application in which the processing is to be performed. For example, these edge devices may not have sufficient computational resources to meet real-time or near real-time timing constraints for applications such as autonomous operations (e.g., self-driving cars, movement within constrained environments, etc.).

[0007] Therefore, there is a need for improved techniques for efficiently processing multi-dimensional content using neural networks.

[0008] Brief Overview

[0009] Certain aspects provide a processor-implemented method for processing multi-dimensional content using a neural network. The method generally includes decomposing a multi-dimensional input into a plurality of two-dimensional subspaces, where the plurality of two-dimensional subspaces share a common dimension. A first attention matrix is generated based on a projection of a label in a first two-dimensional subspace of the plurality of two-dimensional subspaces via an attention block of a transformer neural network, and a second attention matrix is generated based on a projection of a label in a second two-dimensional subspace of the plurality of two-dimensional subspaces via the attention block of the transformer neural network. An output of the transformer neural network is generated based on a combination of the first attention matrix and the second attention matrix.

[0010] Other aspects provide a processing system configured to perform the aforementioned methods and those described herein, a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the aforementioned methods and those described herein, a computer program product embodied on a computer readable storage medium comprising code for performing the aforementioned methods and those further described herein, and a processing system comprising means for performing the aforementioned methods and those further described herein.

[0011] The following description and the annexed drawings set forth certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings illustrate example features of certain aspects of the present disclosure and, thus, do not limit the scope of the disclosure.

[0014] Figure 1 An example transformer neural network architecture in which multi-dimensional data is processed is illustrated.

[0015] Figure 2 Various pipelines for processing multi-dimensional data in a transformer neural network are illustrated.

[0016] Figure 3 Example operations for processing a multi-dimensional input by a transformer neural network based on a decomposition of the multi-dimensional input into two-dimensional subspaces are illustrated in accordance with aspects of the present disclosure.

[0017] Figure 4 An example processing system configured to perform various aspects of the present disclosure is depicted.

[0018] To facilitate understanding, like reference numerals have been used, where possible, to designate like elements across the figures. It is contemplated that elements and features of one aspect can be beneficially incorporated into other aspects without further recitation.

[0019] DETAILED DESCRIPTION

[0020] Aspects of the disclosure provide apparatuses, methods, processing systems, and non-transitory computer-readable media for processing multi-dimensional input using a transformer neural network and decomposing the multi-dimensional input into multiple two-dimensional subspaces. As used herein, the term “multi-dimensional” generally refers to three or more dimensions (e.g., at least height, width, and time).

[0021] Various types of neural networks can be used to process visual content (e.g., to detect objects, to predict future motion of objects detected in the visual content, to segment visual content into different semantic groups, etc.) such as still images or visual content streams (e.g., video content captured as a series of images at a given frame rate such as 24 frames per second, 29.97 frames per second, 60 frames per second, etc.). However, these neural networks generally process visual content on a per-frame basis, which can be a computationally expensive process that increases in complexity as the frame size of each frame in the visual content increases.

[0022] Transformer neural networks (also referred to as “transformers”), and in particular visual transformers, have become increasingly prevalent in a wide variety of machine learning tasks. Transformer-based architectures are generally configured to generate an output based on a sequence of data (e.g., a sequence of frames in a video, a sequence of small patches from a frame or image, etc.). Generally, a machine learning model can use any number of transformer blocks (each providing self-attention) as well as any other components (e.g., one or more neural network layers).

[0023] For multi-dimensional data such as video data having a temporal dimension and two spatial dimensions, processing using a transformer neural network can be computationally expensive due to the amount of data to be processed. As the amount of data to be processed increases, this complexity can increase significantly; for example, in video content, the amount of data to be processed can scale quadratically in the spatial dimensions and the temporal dimension. Due to the structure of the transformer blocks in a transformer neural network, the computational cost involved in processing an input within any given transformer block can scale quadratically with the size of the input; for example, doubling the resolution of a video input while keeping the duration of the video input constant can result in a sixteen-fold increase in the computational expense of processing the video input through a transformer neural network. Moreover, because a transformer neural network can include multiple transformer blocks, each transformer can independently perform computations on the input data, resulting in significant computational cost.

[0024] Aspects of the present disclosure provide techniques for reducing computational cost of processing multi-dimensional input data in a transformer neural network. As discussed in further detail herein, to reduce computational complexity of processing multi-dimensional input, aspects of the present disclosure decompose the multi-dimensional input into multiple two-dimensional subspaces that can share a common dimension; for example, in a video input having height, width, and time dimensions, the two-dimensional subspaces can be height-time subspaces and width-time subspaces. By decomposing the multi-dimensional input into multiple two-dimensional subspaces that share a common dimension, the computational cost involved in processing the multi-dimensional input in a transformer neural network can be reduced and can scale sub-quadratically. As a result, various tasks using a transformer neural network, such as object detection or other computer vision tasks, can be accomplished with less computational resources. In turn, this can reduce the amount of power used by a computing device to perform these tasks and / or speed up the processing of the multi-dimensional input relative to the amount of power and / or time used when processing the multi-dimensional input without decomposing it into multiple two-dimensional subspaces.

[0025] Example Transformer Architecture

[0026] Figure 1 An example transformer architecture 100 is illustrated in which attention data propagates through a transformer block in a neural network (e.g., for other transformer block(s) in the network) in order to generate an output of the neural network.

[0027] As Figure 1 illustrated, input data 105 is accessed by a transformer 110. As used herein, accessing data can generally include receiving, retrieving, requesting, or otherwise obtaining access to the data. As discussed above, the input data 105 can correspond to an input to a first transformer block of a model (e.g., raw or pre-processed input data), an output of a previous transformer or other model component or block, etc. For example, the input data 105 can correspond to a multi-dimensional input, a tokenized version of the multi-dimensional input (which can optionally include position embeddings and / or learnable tokens), etc. The tokenized version of the multi-dimensional input can also be referred to as a set of features of the multi-dimensional input generated over different portions of the multi-dimensional input (e.g., different spatial portions or small pieces of the multi-dimensional input across multiple time points).

[0028] Generally, the transformer 110 includes a self-attention block 120 (labeled “SA”) and a feed-forward block 140 (labeled “FF”). In the self-attention block 120, the input data 105 can be linearly projected (e.g., multiplied using learned parameters) into three matrices: a query matrix Q 122 (also referred to in some aspects as a “query representation” or simply “queries”), a key matrix K 124 (also referred to in some aspects as a “key representation” or simply “keys”), and a value matrix V 126 (also referred to in some aspects as a “value representation” or simply “values”). For example, during training, one or more query weights, key weights, and value weights are learned based on training data, and the queries Q 122, keys K 124, and values V 126 can be generated by multiplying the input data by the learned weights.

[0029] In some aspects, an attention matrix A 130 (also referred to in some aspects as an “attention map” or simply “attention”) is then generated based on these queries and keys. For example, the self-attention block can compute the dot product of the query matrix and the transposed key matrix at operation 128 (e.g., Q · K T ). In some aspects, the self-attention block can apply one or more operations (e.g., a row-wise softmax operation) to the dot product to produce the attention matrix. That is, the attention matrix A 130 can be defined as A = σ(Q · K T ), where σ is a softmax function (or some other regularization function that can be used with transformer neural networks).

[0030] The resulting features f 134 generated by the self-attention block can then be computed at operation 132 as the dot product of the attention matrix A 130 and the value matrix V 126. These features can then be provided as input to the feed-forward block 140 (e.g., a neural network or subnetwork) to generate an output 150 from the transformer 110. The output 150 can be used as input to a subsequent transformer or other block in a neural network, or can be the final result of processing an input through the neural network. In some aspects, the feed-forward block 140 can be a multi-layer perceptron (MLP) that includes multiple layers separated by activation functions such as Gaussian error linear unit activation functions.

[0031] While Figure 1 not depicted in FIG. 1, in some aspects, the transformer 110 can include one or more skip or residual connections (with or without layer normalization). For example, the output 150 can be generated by adding the output of operation 132 to the input data 105, skipping the feed-forward block 140. As another example, the output 150 of the transformer 110 can be generated by adding the output of a final layer of the feed-forward block 140 to a previous version of the output 150.

[0032] Example subspace decomposition of multi-dimensional inputs for processing in transformer neural networks

[0033] As discussed, transformer neural networks can be used to process multi-dimensional inputs and generate inferences based on processing these multi-dimensional inputs. For example, these transformer neural networks can be used to perform various operations on video data, such as video enhancement (e.g., noise reduction, upsizing via super-resolution techniques that increase or otherwise enhance the resolution of an input), object detection, three-dimensional vision, medical imaging, etc. For example, in an object detection task, the output generated by a transformer neural network can be used to semantically segment an input into different segments associated with different levels of importance to the overall meaning of a scene, and select different portions of the scene (e.g., corresponding to different objects) to monitor. For example, the output generated by a transformer neural network can also be used to predict the motion of objects in a scene, which can then be used to apply various control inputs to an autonomous or semi-autonomous vehicle to ensure that the vehicle does not collide with the objects (or at least reduce the likelihood that the vehicle will collide with the objects). In a three-dimensional vision example, a transformer neural network can be used to reconstruct an environment in three-dimensional space based on truncated signed distance function (TSDF) data, etc. In a medical imaging example, a transformer neural network can be used for segmentation of three-dimensional data to identify various structures in captured medical imaging, such as blood vessels, tumors, etc.

[0034] In many such use cases, the information captured over time can be useful data that can be used to generate various inferences from the multi-dimensional input. For example, in computer vision tasks, motion information captured over time can be very useful data for processing video content. Due to the nature of transformer neural networks, temporal dependencies (or relationships) between different portions of the multi-dimensional input (e.g., different frames of a video having different timestamps) can be used, for example, to identify objects in motion, objects that are stationary, spatial relationships between different objects in a frame, etc. However, as discussed above, processing multi-dimensional inputs using transformer neural networks can be a computationally expensive task. In transformer neural networks, each token of an input - e.g., corresponding to a different discrete portion of the input, such as a small patch in an image (e.g., a pixel or contiguous block of pixels in an image) - can be compared to other tokens generated for the input. Due to these comparisons, the computational complexity involved in processing multi-dimensional inputs using transformer neural networks can scale quadratically with input resolution and with time (or temporal resolution, such as frame rate at which video frames are captured). Thus, while some devices can be capable of processing multi-dimensional inputs using transformer neural networks, it can not be practical to deploy transformer neural networks on other devices (e.g., user equipment (UE) in a wireless communication network, Internet of Things (IoT) devices, or other power-constrained devices that can have limited computational capabilities, or computational capabilities that are constrained by the amount of power that can be drawn from an energy storage device) to process these multi-dimensional inputs.

[0035] To improve the efficiency of transformer neural networks and reduce the computational complexity involved in processing inputs using transformer neural networks, various techniques can be used. In some examples, attention can be computed over a subset of tokens (rather than the entire set of tokens generated for an input) to achieve sub-quadratic scaling of computational complexity for inputs to a neural network. This can be performed, for example, based on sparsity constraints, block-wise processing, axial processing, linear decomposition of multi-dimensional inputs to one-dimensional inputs, other decompositions, etc. In one example, space and time can be independently decomposed in order to reduce the computational complexity of processing video inputs in a transformer neural network. In this example, for low-resolution inputs, the cost of decomposition can be relatively small; however, for high-resolution inputs, the computational complexity of processing the input can still be high. However, in decomposing space and time separately, temporal information can not be accessible during processing of the spatial content, although such temporal information includes a large amount of useful contextual data for processing video data by a transformer neural network and performing various actions with respect to outputs of processing the video data.

[0036] In order to further reduce the computational complexity of processing video data or other multidimensional data through transformer neural networks, aspects of the present disclosure decompose multidimensional data (e.g., having three or more dimensions) into multiple two-dimensional subspaces that share a common dimension. For example, video data having two spatial dimensions (height and width) and a time dimension can be decomposed into two two-dimensional subspaces: a first subspace having width and time dimensions and a second subspace having height and time dimensions, where time is the common dimension. By decomposing multidimensional data into multiple two-dimensional subspaces that share a common dimension, the computational complexity of processing such data can be evenly distributed, and the two-dimensional subspaces that share a common dimension can all have access to a large amount of useful contextual data in self-attention or other operations within the transformer neural network. In doing so, aspects of the present disclosure provide computationally efficient processing of multidimensional data relative to other decomposition techniques (e.g., as discussed in further detail below) whose efficiency increases with increasing input resolution. Furthermore, aspects of the present disclosure may provide similar inference accuracy (e.g., peak signal-to-noise ratio (PSNR)) using fewer computational resources than spatiotemporal decomposition, may provide increased inference accuracy using similar computational resources to other decomposition techniques, and may be computationally cheaper relative to other decomposition techniques as the number of samples used in a task (e.g., frames in video content) increases.

[0037] Figure 2 A pipeline for processing multidimensional data in a transformer neural network is explained. Figure 2 The pipeline in illustrates the transformer pipeline in the context of processing video data having height, width, and time dimensions, but it should be recognized that Figure 2 The transformer pipeline explained in

[15] can be adapted to process n-dimensional data (adding or removing attention heads in the pipeline). Figure 2 The pipelines illustrated in illustrate various sequential operations for processing multi-dimensional data in a transformer neural network, but it should be recognized that the various operations illustrated in these pipelines can be performed in parallel or substantially in parallel.

[0038] Pipeline 210 illustrates a pipeline for processing multidimensional data without decomposing the multidimensional input into different subspaces. As illustrated, the multidimensional input can be processed by the attention head 212 of the transformer neural network (e.g., Figure 1The output of the attention head 212 of the transformer neural network (in combination with the input) can be processed by a feedforward network 214. The output of the feedforward network 214, as well as the sum of the output and input of the attention head 212, can be combined to generate the output of the transformer neural network. Since the multi-dimensional data is not decomposed into smaller parts, there can be H*W*T tokens generated for the multi-dimensional input, where H and W correspond to the number of small patches in the height domain and the width domain into which the input is divided, respectively, and each token can be compared to all other tokens generated for the multi-dimensional input. Thus, the pipeline 210 can complete processing of the multi-dimensional input in O(H 2 W 2 T 2 ) time.

[0039] The pipeline 220 illustrates a pipeline for processing multi-dimensional data with a space-time decomposition of a multi-dimensional input that includes spatial and temporal components. In this example, the temporal component of the multi-dimensional input can be processed by a first attention head 222 of a transformer neural network, and the spatial component of the multi-dimensional input can be processed by a second attention head 224 of the transformer neural network. As illustrated, the input to the second attention head 224 of the transformer neural network can be the sum of the output and input of the first attention head 222. The sum of the output of the first attention head, the input, and the second attention head output of the second attention head 224 can then be processed by a feedforward network 226. The output of the feedforward network 226, as well as the sum of the output of the first attention head, the input, and the second attention head output of the second attention head 224, can be combined to generate the output of the transformer neural network. The computational complexity of processing the temporal component of the multi-dimensional input can be O(T 2 ), and the computational complexity of processing the spatial component of the multi-dimensional input can be O(H 2 W 2 ). Because the computational complexity involved in processing the temporal and spatial components of the multi-dimensional input are independent of each other (and thus additive, rather than multiplicative), the pipeline 220 can complete processing of the multi-dimensional input in O(HWT 2 +H 2 W 2 T) time, which represents a significant reduction in the computational complexity of processing the multi-dimensional input relative to the complexity of the pipeline 210 discussed above.

[0040] However, as discussed above, further improvements in both the computational complexity and accuracy of processing multi-dimensional inputs in transformer neural networks can be achieved by decomposing the multi-dimensional input into multiple two-dimensional subspaces that share a common dimension. Pipeline 230 illustrates a pipeline for processing multi-dimensional data based on decomposing the multi-dimensional input into multiple two-dimensional subspaces that share a common dimension, in accordance with aspects of the present disclosure. In this example, the multi-dimensional input can have height, width, and temporal components. Thus, decomposing the multi-dimensional input into multiple two-dimensional subspaces can be achieved by decomposing the multi-dimensional input into a first two-dimensional subspace that includes the height and temporal dimensions and a second two-dimensional subspace that includes the width and temporal dimensions.

[0041] As illustrated, the first attention head 232 can generate a first attention matrix based on the projection of the data in the first two-dimensional subspace (e.g., the tokens generated for individual elements in the first two-dimensional subspace). The second attention head 234 can similarly generate a second attention matrix based on the projection of the data in the second two-dimensional subspace. The outputs of the first and second attention heads 232 and 234 can be combined (e.g., via a summation operation) with the input and provided as input to the feedforward network 236.

[0042] The computational complexity of processing the data in the first two-dimensional subspace via the first attention head 232 can be O(H 2 T 2 ). Similarly, the computational complexity of processing the data in the second two-dimensional subspace via the second attention head 234 can be O(W 2 T 2 ). As with the pipeline 220, because processing the data in the attention heads 232 and 234 involves independent operations, the total computational complexity is additive rather than multiplicative. Thus, the pipeline 230 can complete processing of the multi-dimensional input in O(WH 2 T 2 + HW 2 T 2 ) time, which can provide further improvements in the computational complexity of processing multi-dimensional data (having three or more dimensions) in transformer neural networks. Moreover, unlike in the pipeline 220, the attention heads 232 and 234 in the pipeline 230 process data having a shared dimension, which allows for the use of useful contextual information (e.g., in the temporal domain) when generating the attention matrices for the two subspaces, rather than discarding such information in one of the attention heads, which in turn can allow for reduced computational complexity in various tasks involving multi-dimensional data (video enhancement (e.g., denoising, super-resolution, etc.), video classification, semantic segmentation, object detection, three-dimensional reconstruction, anatomical segmentation, etc.) by allowing for the use of small amounts of consecutive samples (e.g., frames) to perform various tasks. Although Figure 2It is illustrated that the height-time subspace is processed before the width-time subspace, but it should be recognized that data in different subspaces can be processed sequentially in any order (e.g., such that attention heads 232 and 234 are swapped in order) or can be processed in parallel or substantially in parallel.

[0043] As mentioned above, by decomposing data into multiple two-dimensional subspaces that share a common dimension, data in any multi-dimensional space can be processed using pipeline 230. For example, a four-dimensional (e.g., three spatial dimensions and one temporal dimension) input can be decomposed into three two-dimensional subspaces that share the temporal dimension as the common dimension, and the resulting transformer neural network that processes the four-dimensional input by way thereof can include three attention heads (e.g., one attention head for each two-dimensional subspace). Examples of data in multi-dimensional spaces can include, for example, audiovisual input that includes spatial, frequency, and temporal dimensions; input that includes different spatial environments, with operations performed along a common temporal dimension (e.g., input in a virtual reality environment, where each spatial environment corresponds to data displayed on one of a plurality of time-synchronized displays); and so on.

[0044] While the foregoing examples illustrate time as the common dimension shared by the multiple two-dimensional subspaces, it should be recognized that any other suitable dimension (e.g., frequency) can also or alternatively be used as the common dimension shared by the multiple two-dimensional subspaces.

[0045] For an input of T samples in a D-dimensional space decomposed into multiple two-dimensional subspaces that share a common dimension, a transformer neural network can complete processing of the input in O(D x S D+1 T 2 ) time, as compared to O(TS 2D +S D T 2 ) time using space-time decomposition (e.g., as in pipeline 220 discussed above) or O(S 2D T 2 ) time without decomposition. As discussed, processing multi-dimensional data based on decomposing such data into multiple two-dimensional subspaces that share a common dimension can be significantly less computationally expensive than processing multi-dimensional data using space-time decomposition, as processing data in multi-dimensional subspaces can reduce the amount of redundant data processed within the transformer neural network. These efficiencies can be seen to relate to the computational expense incurred in processing data, as it can be feasible to generate usable inferences from multi-dimensional data using a smaller number of samples (e.g., frames in an input video segment) relative to other decomposition techniques. Moreover, it can be seen that by processing fewer samples using a transformer neural network, a significant reduction in computational expense can be realized.

[0046] The reduction in computational complexity achieved by decomposing multi-dimensional data into multiple two-dimensional subspaces that share a common dimension relative to processing multi-dimensional data without decomposition (joint attention) can be represented by the following:

[0047]

[0048] where S represents the size of the non-time dimension in the multi-dimensional space, and D represents the number of spatial dimensions in the multi-dimensional space.

[0049] The reduction in computational complexity achieved by decomposing multi-dimensional data into multiple two-dimensional subspaces that share a common dimension relative to space-time decomposition of multi-dimensional data can be represented by the following:

[0050]

[0051] where T represents the number of samples in the multi-dimensional data.

[0052] The reduction in computational cost can scale based on the number of spatial dimensions included in the multi-dimensional input. For example, for a sequence of two-dimensional images, the computational complexity involved in processing the sequence by a transformer neural network can be reduced by time relative to joint attention, and can be reduced by time relative to space-time decomposition. For a sequence of three-dimensional volumetric data, the computational complexity involved in processing the data by a transformer neural network can be reduced by time relative to joint attention, and can be reduced by time relative to space-time decomposition.

[0053] Figure 3 An example operation 300 for processing a multi-dimensional input by a transformer neural network based on decomposition of the multi-dimensional input into two-dimensional subspaces is illustrated in accordance with aspects of the present disclosure. The operation 300 can be performed, for example, by a computing system having a transformer neural network deployed thereon for processing multi-dimensional data, such as a user equipment (UE), a smartphone, a tablet computer, an autonomous vehicle, an edge device, or other computing system (e.g., a processing system 400 as illustrated in and described in further detail below in Figure 4 ).

[0054] As illustrated, the operation 300 begins at block 310 with decomposing a multi-dimensional input into multiple two-dimensional subspaces. The multiple two-dimensional subspaces share a common dimension. Because the two-dimensional subspaces generally share a common dimension, the number of two-dimensional subspaces into which an n-dimensional input can be decomposed can be n-1. For example, a three-dimensional space in which an input resides can be decomposed into two two-dimensional subspaces.

[0055] In some aspects, the multi-dimensional input can include an input having a temporal dimension and a plurality of spatial dimensions. For example, the multi-dimensional input can be a video clip including a plurality of frames (temporal dimension) that includes two-dimensional spatial data (e.g., height and width). In such a case, the first two-dimensional subspace can be a subspace based on a first spatial dimension of the plurality of spatial dimensions and the temporal dimension, and the second two-dimensional subspace can be a subspace based on a second spatial dimension of the plurality of spatial dimensions and the temporal dimension.

[0056] At block 320, the operations 300 continue with generating, via the attention block of the transformer neural network, a first attention matrix based on a projection of the tokens in the first two-dimensional subspace of the plurality of two-dimensional subspaces.

[0057] In some aspects, generating the first attention matrix based on the projection of the tokens in the first two-dimensional subspace includes projecting the first two-dimensional subspace into query, key, and value data. The first attention matrix can be generated based on the query data, the key data, and a number of temporal components in the first two-dimensional subspace.

[0058] At block 330, the operations 300 continue with generating, via the attention block of the transformer neural network, a second attention matrix based on a projection of the tokens in the second two-dimensional subspace of the plurality of two-dimensional subspaces.

[0059] In some aspects, generating the second attention matrix based on the projection of the tokens in the second two-dimensional subspace includes projecting the second two-dimensional subspace into query, key, and value data. The second attention matrix can be generated based on the query data, the key data, and a number of temporal components in the second two-dimensional subspace.

[0060] At block 340, the operations 300 continue with generating an output of the transformer neural network based on the first attention matrix and the second attention matrix (e.g., a combination of the first attention matrix and the second attention matrix). For example, the combination can be a sum.

[0061] In some aspects, generating the output of the transformer neural network includes computing a first feature based on the first attention matrix and values projected from the tokens in the first two-dimensional subspace. A second feature can be computed based on the second attention matrix and values projected from the tokens in the second two-dimensional subspace. The first feature and the second feature can be combined into a combined feature representing the multi-dimensional input, and the output of the transformer neural network can be generated based on the combined feature.

[0062] In some aspects, generating the output of the transformer neural network based on the combined feature includes processing the combined feature through a feed-forward component of the transformer neural network to generate the output.

[0063] In some aspects, operation 300 further includes taking one or more actions based on the output generated by the transformer neural network. The output generated by the transformer neural network may include, for example, identification of different portions of the multidimensional input corresponding to different objects or object classes, and information identifying the importance of such objects to the scene for each object or object class. The one or more actions may include taking an action based at least in part on the importance of such objects to the scene relative to the importance of one or more objects in the scene. For example, different levels of compression may be applied to different portions of the multidimensional input, where a higher level of compression (with increased compression loss) is applied to less important portions of the multidimensional input, and a lower level of compression (with lower or no compression loss) is applied to more important portions of the multidimensional input. In another example, the output generated by the transformer neural network may include identification of objects in the multidimensional space and predictions of how at least some of the identified objects will move through the multidimensional space. The one or more actions may include generating one or more control inputs to manage the movement of an autonomous or semi-autonomous device through the multidimensional space. The autonomous or semi-autonomous device may include, for example, an autonomous vehicle, a robotic arm, or other device that can move within a multidimensional space with limited or no control. Of course, it should be recognized that these outputs and actions performed based on the above outputs are illustrative, and these outputs and actions performed based on the outputs generated by the transformer neural network can vary based on the environment from which the multi-dimensional input is captured, and the environment in which the device processes and / or operates with the multi-dimensional input.

[0064] Example processing system for processing multi-dimensional inputs in a transformer neural network based on subspace decomposition of the multi-dimensional inputs

[0065] Figure 4 An example processing system 400 is depicted that is configured to perform various aspects of the present disclosure, including, for example, reference Figures 1-3 In one aspect, the processing system 400 may use a converter-based architecture such as Figure 1 Although depicted as a single system for conceptual clarity, in at least some aspects, as discussed above, the operations described below with respect to processing system 400 can be distributed across any number of devices.

[0066] Processing system 400 includes a central processing unit (CPU) 402, which in some examples may be a multi-core CPU. Instructions executed at CPU 402 may be loaded, for example, from a program memory associated with CPU 402 or from a partition of memory 424.

[0067] The processing system 400 also includes additional processing components customized for particular functions, such as a graphics processing unit (GPU) 404, a digital signal processor (DSP) 406, a neural processing unit (NPU) 408, a multimedia processing unit 410, and a wireless connectivity component 412.

[0068] An NPU, such as the NPU 408, is generally a specialized circuit configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANN), deep neural networks (DNN), random forests (RF), and the like. An NPU is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligent processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0069] An NPU, such as the NPU 408, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs can be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, an NPU can be part of a specialized neural network accelerator.

[0070] An NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs capable of performing both training and inference, the two tasks can generally still be performed independently.

[0071] An NPU designed to accelerate training is generally configured to accelerate the optimization of a new model, which involves the highly computationally intensive operations of inputting an existing dataset (generally labeled or tagged), iterating over the dataset, and then adjusting model parameters (such as weights and biases) in order to improve model performance. Generally, optimization based on error prediction involves passing back through layers of the model and determining gradients to reduce prediction error.

[0072] An NPU designed to accelerate inference is generally configured to operate on a complete model. Such an NPU can thereby be configured to input a new piece of data and quickly process that new data through an already trained model to generate a model output (e.g., an inference).

[0073] In one implementation, the NPU 408 is part of one or more of the CPU 402, GPU 404, and / or DSP 406.

[0074] In some examples, the wireless connectivity component 412 can include subcomponents, e.g., for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless transmission standards. The wireless connectivity component 412 is further connected to one or more antennas 414.

[0075] The processing system 400 can also include one or more sensor processing units 416 associated with any manner of sensors, one or more image signal processors (ISPs) 418 associated with any manner of image sensors, and / or a navigation component 420, which can include satellite-based positioning system components (e.g., GPS or GLONASS) as well as inertial positioning system components.

[0076] The processing system 400 can also include one or more input and / or output devices 422, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0077] In some examples, one or more processors of the processing system 400 can be based on an ARM or RISC-V instruction set.

[0078] The processing system 400 further includes a memory 424, which represents one or more static and / or dynamic memories, such as dynamic random access memory, static memories based on flash, etc. In this example, the memory 424 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 400.

[0079] In particular, in this example, the memory 424 includes a multi-dimensional input decomposition component 424A, an attention matrix generation component 424B, and an output generation component 424C. While depicted as discrete components, the illustrated components (as well as other components not depicted) can be implemented collectively or individually in various aspects. Figure 4 The components illustrated in FIG. 4 are not meant to provide architectural services of some embodiments using a layered architecture, such as those illustrated in FIG. 3. Rather, the components illustrated in FIG. 4 are meant to provide a generalized architecture that can be used by any processing system 400, such as a server computer, desktop computer, laptop computer, tablet computer, home-entertainment computer, network server, cell phone, smart phone, web appliance, mobile appliance, game console, handheld game console, or other processing system 400.

[0080] In general, the processing system 400 and / or its components can be configured to perform the methods described herein.

[0081] Notably, in other aspects, aspects of the processing system 400 can be omitted, such as where the processing system 400 is a server computer, etc. For example, in other aspects, the multimedia processing unit 410, the wireless connectivity component 412, the sensor processing unit 416, the ISP 418, and / or the navigation component 420 can be omitted. Moreover, aspects of the processing system 400 can be distributed among multiple devices.

[0082] Example Clause

[0083] Implementation details of various aspects of the present disclosure are described in the following numbered clauses:

[0084] Clause 1 : A processor-implemented method comprising: decomposing a multi-dimensional input into a plurality of two-dimensional subspaces, wherein the plurality of two-dimensional subspaces share a common dimension; generating, via an attention block of a transformer neural network, a first attention matrix based on projections of tokens in a first two-dimensional subspace of the plurality of two-dimensional subspaces; generating, via the attention block of the transformer neural network, a second attention matrix based on projections of tokens in a second two-dimensional subspace of the plurality of two-dimensional subspaces; and generating an output of the transformer neural network based on the first attention matrix and the second attention matrix.

[0085] Clause 2: The method of clause 1, wherein the multi-dimensional input comprises an input having a temporal dimension and a plurality of spatial dimensions.

[0086] Clause 3: The method of clause 2, wherein the plurality of spatial dimensions comprises a width spatial dimension and a height spatial dimension, and wherein the common dimension comprises the temporal dimension, such that a computational complexity involved in generating the output of the transformer neural network is reduced relative to decomposing the multi-dimensional input into spatial components and temporal components.

[0087] Clause 4: The method of clause 2 or 3, wherein the multi-dimensional input comprises a video input.

[0088] Clause 5: The method of any of clauses 2-4, wherein: the first two-dimensional subspace comprises a subspace based on a first spatial dimension of the plurality of spatial dimensions and the temporal dimension, and the second two-dimensional subspace comprises a subspace based on a second spatial dimension of the plurality of spatial dimensions and the temporal dimension.

[0089] Clause 6: The method of any of clauses 1-5, wherein generating the output of the transformer neural network comprises: computing first features based on the first attention matrix and values projected from tokens in the first two-dimensional subspace; computing second features based on the second attention matrix and values projected from tokens in the second two-dimensional subspace; combining the first features and the second features into a combined feature representing the multi-dimensional input; and generating the output of the transformer neural network based on the combined feature.

[0090] Clause 7: The method of clause 6, wherein the combined feature comprises a sum of the first features and the second features.

[0091] Clause 8: The method of clause 6 or 7, wherein generating the output of the transformer neural network based on the combined feature comprises processing the combined feature through a feedforward component of the transformer neural network to generate the output.

[0092] Clause 9: The method of any of clauses 1-8, wherein generating the first attention matrix based on the projection of the tokens in the first two-dimensional subspace comprises: projecting the first two-dimensional subspace into query data, key data, and value data; and generating the first attention matrix based on the query data, the key data, and a number of components from a shared dimension in the first two-dimensional subspace.

[0093] Clause 10: The method of any of clauses 1-9, wherein generating the second attention matrix based on the projection of the tokens in the second two-dimensional subspace comprises: projecting the second two-dimensional subspace into query data, key data, and value data; and generating the second attention matrix based on the query data, the key data, and a number of components from a shared dimension in the second two-dimensional subspace.

[0094] Clause 11: A processing system comprising: memory comprising computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform the method of any of clauses 1-10.

[0095] Clause 12: A processing system comprising means for performing the method of any of clauses 1-10.

[0096] Clause 13: A non-transitory computer-readable medium comprising computer- executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any of clauses 1-10.

[0097] Clause 14: A computer program product embodied on a computer-readable storage medium comprising code for performing the method of any of clauses 1-10.

[0098] Additional Considerations

[0099] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not meant to limit the scope of the claims encompassing the disclosure, which is defined solely by the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects. For example, features described in the context of one example can be combined or eliminated in other examples. Numerous other changes can be made hereunder without departing from the scope of the disclosure. For example, various sequences, steps, or stages can be performed in an order different from that described. Various steps can be combined or omitted, and other steps can be added. Also, features described as part of one example can be combined in or excluded from another example. For example, any of the aspects described herein can be implemented using any number of the aspects described herein. In addition, the scope of the disclosure is intended to cover any resulting device or method encompassed by the claims, whether or not it is now claimed. It is understood that any aspect of the disclosure disclosed herein can be implemented by one or more elements of a claim.

[0100] As used herein, the term "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0101] As used herein, the phrase "at least one of" followed by a listing of two or more items means any of the listed items individually, or any combination of two or more of the listed items. As an example, "at least one of a, b, or c" means "a" or "b" or "c" or "a-b" or "a-c" or "b-c" or "a-b-c" or any combination thereof.

[0102] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, "determining" can include resolving, selecting, choosing, establishing and the like.

[0103] The methods disclosed herein include one or more steps or actions for achieving the methods. The method steps and / or actions can be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. Further, the various operations of methods described above can be performed by any suitable means capable of performing the corresponding functions. These means can include various hardware and / or software component(s) and / or module(s), including, but not limited to a circuit, an application specific integrated circuit (ASIC), or processor. Generally, where there are operations illustrated in figures, those operations can have corresponding counterpart means-plus-function components with similar numbering.

[0104] The following claims are not intended to be limited to the aspects shown herein but are to be afforded the full scope consistent with the language of the claims. Within the claims, the reference to an element in the singular is not intended to mean "one and only one" unless specifically so stated, but rather "one or more." Unless specifically stated otherwise, the term "some" refers to one or more. Any element expressed by the phrase "means for" is intended to functionally equate to the instances disclosed enabling that specific function, either by itself or in combination with other elements. All structural and functional equivalents to the aspects described throughout this disclosure that are enabled by this description are expressly incorporated by reference and intended to be encompassed by the claims. Moreover, any combination of the aspects described throughout this disclosure are expressly intended to be within the scope of the claims. Furthermore, any publication or other disclosure material presented herein that is incorporated by reference is expressly intended to be only for purposes of the claims and not to limit the aspects of the disclosure in any way.

Claims

1. A processor-implemented method comprising: decomposing a multi-dimensional input into a plurality of two-dimensional subspaces, wherein the plurality of two-dimensional subspaces share a common dimension; generating, via an attention block of a transformer neural network, a first attention matrix based on a projection of a token in a first two-dimensional subspace of the plurality of two-dimensional subspaces; generating, via the attention block of the transformer neural network, a second attention matrix based on a projection of a token in a second two-dimensional subspace of the plurality of two-dimensional subspaces; and generating an output of the transformer neural network based on the first attention matrix and the second attention matrix.

2. The method of claim 1, wherein the multi-dimensional input comprises an input having a temporal dimension and a plurality of spatial dimensions.

3. The method of claim 2, wherein the plurality of spatial dimensions comprises a width spatial dimension and a height spatial dimension, and wherein the common dimension comprises the temporal dimension, such that a computational complexity involved in generating the output of the transformer neural network is reduced relative to decomposing the multi-dimensional input into spatial components and temporal components.

4. The method of claim 2, wherein the multi-dimensional input comprises a video input.

5. The method of claim 2, wherein: the first two-dimensional subspace comprises a subspace based on a first spatial dimension of the plurality of spatial dimensions and the temporal dimension, and the second two-dimensional subspace comprises a subspace based on a second spatial dimension of the plurality of spatial dimensions and the temporal dimension.

6. The method of claim 1, wherein generating the output of the transformer neural network comprises: computing first features based on the first attention matrix and values projected from the token in the first two-dimensional subspace; computing second features based on the second attention matrix and values projected from the token in the second two-dimensional subspace; combining the first features and the second features into combined features representing the multi-dimensional input; and generating the output of the transformer neural network based on the combined features.

7. The method of claim 6, wherein the combined features comprise a sum of the first features and the second features. processing the combined features through a feed-forward component of the transformer neural network.

9. The method of claim 1, wherein generating the first attention matrix based on the projection of the token in the first two-dimensional subspace comprises:

8. The method of claim 6, wherein generating the output of the transformer neural network based on the combined features comprises: projecting the first two-dimensional subspace into query data, key data, and value data; and generating the first attention matrix based on the query data, the key data, and a number of components from the common dimension in the first two-dimensional subspace.

10. The method of claim 1, wherein generating the second attention matrix based on the projection of the token in the second two-dimensional subspace comprises: projecting the second two-dimensional subspace into query data, key data, and value data; and generating the second attention matrix based on the query data, the key data, and a number of components from the common dimension in the second two-dimensional subspace. ​ ​ ​ ​ 11. A system comprising: a memory having stored thereon executable instructions; and a processor configured to execute the executable instructions to cause the system to: decompose a multi-dimensional input into a plurality of two-dimensional subspaces, wherein the plurality of two-dimensional subspaces share a common dimension; generate, via an attention block of a transformer neural network, a first attention matrix based on a projection of a token in a first two-dimensional subspace of the plurality of two-dimensional subspaces; generate, via the attention block of the transformer neural network, a second attention matrix based on a projection of a token in a second two-dimensional subspace of the plurality of two-dimensional subspaces; and generate, based on the first attention matrix and the second attention matrix, an output of the transformer neural network.

12. The system of claim 11, wherein the multi-dimensional input comprises an input having a temporal dimension and a plurality of spatial dimensions.

13. The system of claim 12, wherein the plurality of spatial dimensions comprises a width spatial dimension and a height spatial dimension, and wherein the common dimension comprises the temporal dimension, such that a computational complexity involved in generating the output of the transformer neural network is reduced relative to decomposing the multi-dimensional input into spatial components and temporal components.

14. The system of claim 12, wherein the multi-dimensional input comprises a video input.

15. The system of claim 12, wherein: the first two-dimensional subspace comprises a subspace based on a first spatial dimension of the plurality of spatial dimensions and the temporal dimension, and the second two-dimensional subspace comprises a subspace based on a second spatial dimension of the plurality of spatial dimensions and the temporal dimension.

16. The system of claim 11, wherein to generate the output of the transformer neural network, the processor is configured to cause the system to: compute a first feature based on the first attention matrix and values projected from the token in the first two-dimensional subspace; compute a second feature based on the second attention matrix and values projected from the token in the second two-dimensional subspace; combine the first feature and the second feature into a combined feature representing the multi-dimensional input; and generate the output of the transformer neural network based on the combined feature.

17. The system of claim 16, wherein to generate the output of the transformer neural network based on the combined feature, the processor is configured to cause the system to process the combined feature through a feed-forward component of the transformer neural network.

18. The system of claim 11, wherein to generate the first attention matrix based on the projection of the token in the first two-dimensional subspace, the processor is configured to cause the system to: project the first two-dimensional subspace into query data, key data, and value data; and generate the first attention matrix based on the query data, the key data, and a number of components from the common dimension in the first two-dimensional subspace. ​ ​ 19. The system of claim 11, wherein to generate the second attention matrix based on the projection of the tokens in the second two-dimensional subspace, the processor is configured to cause the system to: project the second two-dimensional subspace into query data, key data, and value data; and generate the second attention matrix based on the query data, the key data, and a number of components from the shared dimension in the second two-dimensional subspace.

20. A system comprising: means for decomposing a multi-dimensional input into a plurality of two-dimensional subspaces, wherein the plurality of two-dimensional subspaces share a shared dimension; means for generating, via an attention block of a transformer neural network, a first attention matrix based on a projection of tokens in a first two-dimensional subspace of the plurality of two-dimensional subspaces; means for generating, via the attention block of the transformer neural network, a second attention matrix based on a projection of tokens in a second two-dimensional subspace of the plurality of two-dimensional subspaces; and means for generating an output of the transformer neural network based on the first attention matrix and the second attention matrix.

21. The system of claim 20, wherein the multi-dimensional input comprises an input having a temporal dimension and a plurality of spatial dimensions.

22. The system of claim 21, wherein the plurality of spatial dimensions comprises a width spatial dimension and a height spatial dimension, and wherein the shared dimension comprises the temporal dimension, such that a computational complexity involved in generating the output of the transformer neural network is reduced relative to decomposing the multi-dimensional input into spatial components and temporal components.

23. The system of claim 21, wherein: the first two-dimensional subspace comprises a subspace based on a first spatial dimension of the plurality of spatial dimensions and the temporal dimension, and the second two-dimensional subspace comprises a subspace based on a second spatial dimension of the plurality of spatial dimensions and the temporal dimension.

24. The system of claim 20, wherein the means for generating the output of the transformer neural network comprises: means for computing first features based on the first attention matrix and values projected from the tokens in the first two-dimensional subspace; means for computing second features based on the second attention matrix and values projected from the tokens in the second two-dimensional subspace; means for combining the first features and the second features into combined features representing the multi-dimensional input; and means for generating the output of the transformer neural network based on the combined features. means for processing the combined features by a feed-forward component of the transformer neural network.

25. The system of claim 24, wherein the means for generating the output of the transformer neural network based on the combined features comprises:

26. The system of claim 20, wherein the means for generating the first attention matrix based on the projection of the tokens in the first two-dimensional subspace comprises: means for projecting the first two-dimensional subspace into query data, key data, and value data; and ​ means for generating the first attention matrix based on the query data, the key data, and components from the shared dimension in the first two-dimensional subspace.

27. The system of claim 20, wherein the means for generating the second attention matrix based on the projection of the tokens in the second two-dimensional subspace comprises: means for projecting the second two-dimensional subspace into query data, key data, and value data; and means for generating the second attention matrix based on the query data, the key data, and components from the shared dimension in the second two-dimensional subspace.

28. A computer-readable medium having stored thereon executable instructions that, when executed by a processor, perform operations comprising: decomposing a multi-dimensional input into a plurality of two-dimensional subspaces, wherein the plurality of two-dimensional subspaces share a shared dimension; generating, via an attention block of a transformer neural network, a first attention matrix based on a projection of tokens in a first two-dimensional subspace of the plurality of two-dimensional subspaces; generating, via the attention block of the transformer neural network, a second attention matrix based on a projection of tokens in a second two-dimensional subspace of the plurality of two-dimensional subspaces; and generating an output of the transformer neural network based on the first attention matrix and the second attention matrix.

29. The computer-readable medium of claim 28, wherein the multi-dimensional input comprises an input having a temporal dimension and a plurality of spatial dimensions.

30. The computer-readable medium of claim 29, wherein the plurality of spatial dimensions comprises a width spatial dimension and a height spatial dimension, and wherein the shared dimension comprises the temporal dimension, such that a computational complexity involved in generating the output of the transformer neural network is reduced relative to decomposing the multi-dimensional input into spatial components and temporal components.

31. The computer-readable medium of claim 29, wherein: the first two-dimensional subspace comprises a subspace based on a first spatial dimension of the plurality of spatial dimensions and the temporal dimension, and the second two-dimensional subspace comprises a subspace based on a second spatial dimension of the plurality of spatial dimensions and the temporal dimension.

32. The computer-readable medium of claim 28, wherein generating the output of the transformer neural network comprises: computing first features based on the first attention matrix and values projected from the tokens in the first two-dimensional subspace; computing second features based on the second attention matrix and values projected from the tokens in the second two-dimensional subspace; combining the first features and the second features into combined features representing the multi-dimensional input; and generating the output of the transformer neural network based on the combined features.

33. The computer-readable medium of claim 32, wherein generating the output of the transformer neural network based on the combined features comprises: processing the combined features through a feed-forward component of the transformer neural network.

34. The computer-readable medium of claim 28, wherein generating the first attention matrix based on the projection of the tokens in the first two-dimensional subspace comprises: projecting the first two-dimensional subspace into query data, key data, and value data; and generating the first attention matrix based on the query data, the key data, and components from the shared dimension in the first two-dimensional subspace.

35. The computer-readable medium of claim 28, wherein generating the second attention matrix based on the projection of the label in the second two-dimensional subspace comprises: projecting the second two-dimensional subspace into query data, key data, and value data; and generating the second attention matrix based on the query data, the key data, and components from the shared dimension in the second two-dimensional subspace.

Images