Multidimensional spatial decomposition of transform neural networks

Abstract

Certain aspects of this disclosure provide techniques and apparatus for processing multidimensional content using a neural network. Exemplary methods generally involve decomposing a multidimensional input into a plurality of two-dimensional subspaces, the plurality of two-dimensional subspaces sharing a common dimension. A first attention matrix is ​​generated via an attention block of a transformer neural network based on the projection of tokens in the first two-dimensional subspace of the plurality of two-dimensional subspaces, and a second attention matrix is ​​generated via an attention block of a transformer neural network based on the projection of tokens in the second two-dimensional subspace of the plurality of two-dimensional subspaces. The output of the transformer neural network is generated based on a combination of the first and second attention matrices.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims the interests and priority of U.S. Patent Application No. 18 / 193,234, filed on March 30, 2023, which is incorporated herein by reference in its entirety.

[0002] introduction The aspects of this disclosure relate to neural networks, and more specifically, to processing multidimensional content using neural networks.

[0003] Various machine learning architectures have been used to provide solutions to a wide variety of computational problems. There are various machine learning model architectures, including artificial neural networks (which may include convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep neural networks, generative adversarial networks (GANs), etc.) and random forest models. The use of transformer neural networks is becoming increasingly widespread in various image and video processing tasks, or in other tasks where multidimensional data is processed to generate various inferences related to multidimensional data.

[0004] However, transformer neural networks tend to be computationally expensive. For example, vision transformers generally compute self-awareness in each block, so computational and memory requirements can increase proportionally to the square of the input data size. The computational complexity of these tasks can be addressed through the use of high-performance processing units (e.g., graphics processing units, neural processing units, and / or other processing units that support a high degree of parallelism) and large amounts of memory, but edge devices (e.g., user equipment (UEs) such as mobile devices or autonomous vehicles) may not have sufficient computational resources to process multidimensional content (e.g., video content with two spatial dimensions (height and width) and one temporal dimension) at a performance level sufficient for the application in which the processing will be performed. For example, these edge devices may not have sufficient computational resources to meet real-time or near-real-time timing constraints for applications such as autonomous driving (e.g., self-driving cars, movement in constrained environments, etc.).

[0005] Therefore, what is needed is improved technology for efficiently processing multidimensional content using neural networks. [Overview of the Initiative] [Means for solving the problem]

[0006] A particular embodiment provides a processor-driven method for processing multidimensional content using a neural network. This method generally involves decomposing a multidimensional input into multiple two-dimensional subspaces, each sharing a common dimension. A first attention matrix is ​​generated via the attention blocks of a transformer neural network based on the projection of tokens in the first two-dimensional subspace among the multiple two-dimensional subspaces, and a second attention matrix is ​​generated via the attention blocks of a transformer neural network based on the projection of tokens in the second two-dimensional subspace among the multiple two-dimensional subspaces. The output of the transformer neural network is generated based on a combination of the first and second attention matrices.

[0007] Other embodiments provide a processing system configured to perform the aforementioned methods and the methods described herein; a non-temporary computer-readable medium containing instructions, when executed by one or more processors of the processing system, that cause the processing system to perform the aforementioned methods and the methods described herein; a computer program product embodied on a computer-readable storage medium containing code for performing the aforementioned methods and the methods further described herein; and a processing system comprising means for performing the aforementioned methods and the methods further described herein.

[0008] The following description and related drawings provide a detailed representation of specific exemplary features of one or more embodiments.

[0009] The attached figures illustrate exemplary features of specific aspects of the disclosure and should therefore not be considered to limit the scope of the disclosure. [Brief explanation of the drawing]

[0010] [Figure 1] This shows an exemplary transform neural network architecture for processing multidimensional data. [Figure 2]This illustrates various pipelines for processing multidimensional data in transform neural networks. [Figure 3] This disclosure illustrates exemplary operations for processing multidimensional inputs through a transform neural network, based on the decomposition of multidimensional inputs into a two-dimensional subspace, according to various aspects of this disclosure. [Figure 4] An exemplary processing system configured to perform various aspects of this disclosure is shown.

[0011] For ease of understanding, the same reference numeral is used to designate identical elements common to the drawings, where possible. It has been conceived that elements and features of one embodiment can be usefully incorporated into other embodiments without further detail. [Modes for carrying out the invention]

[0012] Aspects of this disclosure provide apparatus, methods, processing systems, and non-temporal computer-readable media for processing multidimensional inputs using transformer-based neural networks and decomposition of multidimensional inputs into multiple two-dimensional subspaces. As used herein, the term “multidimensional” generally refers to three or more dimensions (e.g., at least height, width, and time).

[0013] Using various types of neural networks, it is possible to process visual content such as still images or a stream of visual content (e.g., video content captured as a series of images at a given frame rate such as 24 frames per second, 29.97 frames per second, 60 frames per second, etc.). For example, it is possible to detect objects, predict the future movement of objects detected within the visual content, segment the visual content into various semantic groups, etc. However, these neural networks generally process visual content frame by frame, which can be a computationally expensive process where the complexity increases as the frame size of each frame within the visual content increases.

[0014] Transformer neural networks (also referred to as "transformers"), particularly vision transformers, are becoming increasingly common in a wide variety of machine learning tasks. Transformer-based architectures are generally configured to generate an output based on a sequence of data (e.g., a sequence of frames within a video, a sequence of patches from a frame or image, etc.). Generally, a machine learning model can use any number of transformer blocks (each providing self-attention), as well as any other components (e.g., one or more neural network layers).

[0015] For multidimensional data, such as video data with two spatial dimensions and one temporal dimension, processing using transformer neural networks can be computationally expensive due to the amount of data being processed. This complexity can increase significantly as the amount of data being processed increases; for example, in video content, the amount of data being processed can increase or decrease proportionally to the square of the spatial and temporal dimensions. Due to the structure of transformer blocks in a transformer neural network, the computational cost of processing an input within any given transformer block can increase or decrease proportionally to the square of the input size. For example, doubling the resolution of a video input while keeping its duration constant can result in a 16-fold increase in the computational cost of processing this video input through the transformer neural network. Furthermore, because a transformer neural network can contain multiple transformer blocks, each transformer independently performs calculations on the input data, which can lead to significant computational costs.

[0016] Aspects of the present disclosure provide techniques for reducing the computational cost of processing multi-dimensional input data in a transformer neural network. As discussed in more detail herein, to reduce the computational complexity of processing multi-dimensional input, aspects of the present disclosure decompose multi-dimensional input into a plurality of two-dimensional subspaces that may share a common dimension. For example, in a video input having dimensions of height, width, and time, the two-dimensional subspaces can be height-time subspaces and width-time subspaces. By decomposing multi-dimensional input into a plurality of two-dimensional spaces that share a common dimension, the computational cost associated with processing multi-dimensional input in a transformer neural network can be reduced, and the computational cost can vary proportionally to the square. Therefore, for various tasks in which a transformer neural network is used, such as object detection or other computer vision tasks, less computational resources can be utilized. As a result, this reduces the amount of power used by a computing device to perform these tasks compared to the amount of power and / or time used when multi-dimensional input is not decomposed into a plurality of two-dimensional spaces for processing, and / or can accelerate the processing of multi-dimensional input. Exemplary Transformer Architecture

[0017] FIG. 1 shows an exemplary Transformer architecture 100 in which attention data is propagated through a Transformer block within a neural network (e.g., to other Transformer block(s) within the network) to generate an output of the neural network.

[0018] As shown in Figure 1, the input data 105 is accessed by the transformer 110. As used herein, accessing data can generally include receiving data, retrieving data, requesting data, or obtaining access to data by other means. As discussed above, the input data 105 may correspond to inputs to the first transformer block of the model (e.g., raw input data or pre-processed input data), outputs of preceding transformers or other model components or blocks, etc. For example, the input data 105 may correspond to a multidimensional input, a tokenized version of a multidimensional input (which may optionally include location embeddings (single or multiple) and / or learnable tokens (single or multiple)), etc. The tokenized version of a multidimensional input may also be referred to as a set of features of the multidimensional input generated over various parts of the multidimensional input (e.g., various spatial parts or patches of the multidimensional input over multiple time points).

[0019] Generally, the transformer 110 includes a self-attention block 120 (labeled "SA") and a feedforward block 140 (labeled "FF"). In the self-attention block 120, the input data 105 can be linearly projected (for example, multiplied using learned parameters) onto three matrices: a query matrix Q122 (also referred to in some embodiments as the "query representation" or simply "query"), a key matrix K124 (also referred to in some embodiments as the "key representation" or simply "key"), and a value matrix V126 (also referred to in some embodiments as the "value representation" or simply "value"). For example, during training, one or more query weights, key weights, and value weights can be learned based on the training data, and the query Q122, key K124, and value V126 can be generated by multiplying the input data by these learned weights.

[0020] In some embodiments, an attention matrix A130 (also referred to in some embodiments as an "attention map" or simply "attention") is then generated based on the query and key. For example, in operation 128, the self-attention block is generated by the dot product (e.g., Q·K) of the query matrix and the transposed key matrix. T ) can be calculated. In some embodiments, a self-attention block can produce its attention matrix by applying one or more operations (e.g., row-level softmax operation) to its dot product. That is, the attention matrix A130 is A = σ(Q·K T It can be defined as σ, where σ is the softmax function (or any other regularization function available in a transform neural network).

[0021] Next, the resulting features f134 generated by the self-attention block can be calculated in operation 132 as the dot product of the attention matrix A130 and the value matrix V126. These features can then be provided as input to the feedforward block 140 (e.g., a neural network or subnet) to generate the output 150 from the transformer 110. The output 150 can also be used as input to a subsequent transformer or other block in the neural network, or it can be the final result of processing the input through the neural network. In some embodiments, the feedforward block 140 can be a multilayer perceptron (MLP) containing multiple layers separated by an activation function such as a Gaussian error linear unit activation function.

[0022] Although not shown in Figure 1, in some embodiments, the transformer 110 may include one or more skip connections or residual connections (with or without layer normalization). For example, output 150 can be generated by skipping the feedforward block 140 and adding the output of operation 132 to the input data 105. In another embodiment, output 150 of transformer 110 can be generated by adding the output of the final layer of the feedforward block 140 to a previous version of output 150. Exemplary subspace decomposition of multidimensional inputs for processing in transform neural networks

[0023] As discussed, transformer neural networks can be used to process multidimensional inputs and generate inferences based on the processing of these multidimensional inputs. For example, these transformer neural networks can be used to perform various operations on video data, such as video enhancement (e.g., noise reduction, upsizing via super-resolution techniques to improve or enhance the resolution of the input), object detection, 3D vision, and medical imaging. In object detection tasks, for example, the output generated by the transformer neural network can be used to semantically segment the input into various segments associated with varying levels of importance to the overall meaning of the scene, and then select various parts of the scene (e.g., corresponding to various objects) to monitor. The output generated by the transformer neural network can also be used, for example, to predict the movement of objects in a scene, and then this prediction can be used to ensure (or at least reduce the likelihood of) a vehicle colliding with these objects by applying various control inputs to an autonomous or semi-autonomous vehicle. In 3D vision applications, transformer neural networks can be used to reconstruct environments in 3D space based on data such as truncated signed distance function (TSDF). In medical imaging applications, transformer neural networks can be used to segment 3D data to identify various structures within captured medical images, such as blood vessels and tumors.

[0024] In many of these use cases, information captured over time can be useful data that can be used to generate various inferences from multidimensional inputs. For example, in computer vision tasks, motion information captured over time can be extremely useful data for processing video content. Due to the nature of transformer neural networks, it is possible to use the temporal dependencies (or relationships) between different parts of a multidimensional input (e.g., different frames of a video with different timestamps) to identify, for example, moving objects, stationary objects, and spatial relationships between various objects within a frame. However, as discussed above, processing multidimensional inputs using transformer neural networks can be a computationally expensive task. In a transformer neural network, each token of the input, corresponding to various discrete parts of the input, such as patches in an image (e.g., pixels or blocks of consecutive pixels in an image), can be compared with other tokens generated for that input. These comparisons mean that the computational complexity involved in processing multidimensional inputs using transformer neural networks can increase or decrease in proportion to the input resolution and the square of the time (or temporal resolution, such as the frame rate at which the video frames are captured). Therefore, while some devices may be capable of processing multidimensional inputs using transformer neural networks, deploying transformer neural networks to process these multidimensional inputs on other devices (e.g., user equipment (UEs) in wireless communication networks, Internet of Things (IoT) devices, or other power-constrained devices that may have limited computing power or computing power limited by the amount of power that can be drawn from energy storage devices) may not be practical.

[0025] Various techniques can be used to improve the efficiency of transformer neural networks and reduce the computational complexity involved in processing inputs using transformer neural networks. In some embodiments, attention can be computed over a subset of tokens (rather than the entire set of tokens generated for the input) to achieve a sub-squared-proportional increase or decrease in computational complexity with respect to the input to the neural network. This can be done, for example, based on sparsity constraints, block-based processing, axial processing, linear decomposition of multidimensional inputs to one-dimensional inputs, and other decompositions. In one embodiment, space and time can be decomposed independently to reduce the computational complexity of processing video inputs in a transformer neural network. In this embodiment, the cost of decomposition can be relatively small for low-resolution inputs, however, the computational complexity of processing the input can still be high for high-resolution inputs. However, when decomposing space and time separately, while temporal information may contain a significant amount of useful contextual data for processing video data through the transformer neural network and performing various actions on the output of processing that video data, such temporal information may not be accessible during processing of spatial content.

[0026] To further reduce the computational complexity of processing video data or other multidimensional data through transform neural networks, aspects of the disclosure decompose multidimensional data (e.g., having three or more dimensions) into multiple two-dimensional subspaces that share a common dimension. For example, video data having two spatial dimensions (height and width) and one temporal dimension can be decomposed into two two-dimensional subspaces: a first subspace having width and time dimensions, and a second subspace having height and time dimensions, where time is the common dimension. By decomposing multidimensional data into multiple two-dimensional subspaces that share a common dimension, the computational complexity of processing such data can be evenly distributed, and both of these two-dimensional subspaces that share a common dimension can access a significant amount of useful contextual data for self-attention or other operations within the transform neural network. In doing so, aspects of the disclosure provide computationally efficient processing of multidimensional data that improves in efficiency as the input resolution increases compared to other decomposition techniques (e.g., those discussed in more detail below). Furthermore, aspects of this disclosure can provide similar inference accuracy (e.g., peak signal-to-noise ratio, PSNR) using fewer computational resources than spatial-temporal resolution, can provide improved inference accuracy using similar computational resources to other resolution techniques, and may become computationally less expensive compared to other resolution techniques as the number of samples (e.g., frames in video content) used in the task increases.

[0027] Figure 2 shows a pipeline for processing multidimensional data in a transformer neural network. While the pipeline in Figure 2 illustrates a transformer pipeline in the context of processing video data with height, width, and time dimensions, it should be noted that the transformer pipeline shown in Figure 2 may be applicable to processing n-dimensional data (with the addition or removal of attention heads within the pipeline). Furthermore, while the pipeline in Figure 2 illustrates various sequential operations for processing multidimensional data in a transformer neural network, it should be noted that the various operations shown within these pipelines can be executed in parallel or substantially in parallel.

[0028] Pipeline 210 represents a pipeline for processing multidimensional data without decomposing the multidimensional input into various subspaces. As shown in the figure, the multidimensional input can be processed through the attention head 212 of the transformer neural network (e.g., the self-attention block 120 shown in Figure 1). The output of the attention head 212 of the transformer neural network (combined with the input) can be processed through the feedforward network 214. By combining the output of the feedforward network 214 with the sum of the output and input of the attention head 212, the output of the transformer neural network can be generated. Since the multidimensional data is not decomposed into smaller parts, there may be H*W*T tokens generated for the multidimensional input, where H and W correspond to the number of patches in the height domain and the number of patches in the width domain, respectively, and each token can be compared with all other tokens generated for the multidimensional input. Therefore, pipeline 210 processes the multidimensional input in O(H) time. 2 W 2 T 2 It can be completed in the following time.

[0029] Pipeline 220 represents a pipeline for processing multi-dimensional data using spatio-temporal decomposition of a multi-dimensional input that includes a spatial component and a temporal component. In this example, the temporal component of the multi-dimensional input can be processed through the first attention head 222 of the transformer neural network, and the spatial component(s) of the multi-dimensional input can be processed through the second attention head 224 of the transformer neural network. As shown, the input to the second attention head 224 of the transformer neural network can be the sum of the output and the input of the first attention head 222. The sum of the output of the first attention head, the input, and the second attention head output of the second attention head 224 can then be processed through a feed-forward network 226. By combining the output of the feed-forward network 226 with the sum of the output of the first attention head, the input, and the second attention head output of the second attention head 224, the output of the transformer neural network can be generated. The computational complexity of processing the temporal component of the multi-dimensional input can be O(T 2 ), and the computational complexity of processing the spatial component of the multi-dimensional input can be O(H 2 W 2 ). Since the computational complexity involved in processing the temporal and spatial components of the multi-dimensional input is independent of each other (and hence additive, not multiplicative), pipeline 220 can complete the processing of the multi-dimensional input in O(H W T 2 + H 2 W 2 T) time, which represents a significant reduction in the computational complexity of processing the multi-dimensional input compared to the complexity of pipeline 210 discussed above.

[0030] However, as discussed above, by decomposing a multidimensional input into multiple two-dimensional subspaces that share a common dimension, it is possible to achieve further improvements in both computational complexity and accuracy of processing multidimensional inputs in transform neural networks. Pipeline 230 illustrates a pipeline for processing multidimensional data based on decomposing a multidimensional input into multiple two-dimensional subspaces that share a common dimension, according to various embodiments of this disclosure. In this embodiment, the multidimensional input may have height, width, and time components. Therefore, the decomposition of this multidimensional input into multiple two-dimensional subspaces can be achieved by decomposing the multidimensional input into a first two-dimensional subspace containing height and time dimensions and a second two-dimensional subspace containing width and time dimensions.

[0031] As shown in the figure, the first attention head 232 can generate a first attention matrix based on the projection of data in a first two-dimensional subspace (e.g., tokens generated for each element in the first two-dimensional subspace). Similarly, the second attention head 234 can generate a second attention matrix based on the projection of data in a second two-dimensional subspace. The outputs of the first attention head 232 and the second attention head 234 can be combined with the inputs (e.g., via addition) and provided as input to the feedforward network 236.

[0032] The computational complexity of processing data in the first two-dimensional subspace via the first attention head 232 is O(H 2 T 2 Similarly, the computational complexity of processing data in the second two-dimensional subspace via the second attention head 234 is O(W). 2 T 2 ) can be expressed as follows. Similar to pipeline 220, the data processing in attention heads 232 and 234 involves independent operations, so the sum of computational complexity is additive, not multiplicative. Therefore, pipeline 230 processes multidimensional inputs in O(WH). 2 T 2 +HW2 T 2 This can be completed in the time required, which provides a further improvement in the computational complexity of processing multidimensional data (having three or more dimensions) in transform neural networks. Furthermore, unlike pipeline 220, attention heads 232 and 234 in pipeline 230 process data with shared dimensions, allowing useful contextual information (e.g., in the time domain) to be used when generating attention matrices for both subspaces, rather than discarding it in one of the attention heads. This can consequently reduce computational complexity in various tasks involving multidimensional data (e.g., video enhancement (e.g., denoising, super-resolution), video classification, semantic segmentation, object detection, 3D reconstruction, anatomical segmentation, etc.) by enabling the use of fewer consecutive samples (e.g., frames) to perform various tasks. Figure 2 shows that the height-time subspace is processed before the width-time subspace, but it should be noted that data in different subspaces can be processed sequentially in any order (for example, so that the order of attention head 232 and attention head 234 is swapped), or they can be processed in parallel or substantially in parallel.

[0033] As described above, pipeline 230 can be used to process data in an arbitrary multidimensional space by decomposing the data into multiple two-dimensional subspaces that share a common dimension. For example, a four-dimensional input (e.g., three spatial dimensions and one time dimension) can be decomposed into three two-dimensional subspaces that share the time dimension as a common dimension, and the resulting transform neural network on which the four-dimensional input is processed may contain three attention heads (e.g., one for each two-dimensional subspace). Examples of data in a multidimensional space include, for example, audiovisual inputs that include spatial, frequency, and time dimensions, and inputs that include various spatial environments in which operations are performed along a common time dimension (e.g., inputs in a virtual reality environment where each spatial environment corresponds to data displayed on one of several time-synchronized displays).

[0034] While the above example shows time as a common dimension shared by multiple two-dimensional subspaces, it should be noted that any other suitable dimension (e.g., frequency) can be used similarly or alternatively as a common dimension shared by multiple two-dimensional subspaces.

[0035] For inputs of T samples in a D-dimensional space that are decomposed into multiple 2-dimensional subspaces sharing a common dimension, a transform neural network processes these inputs using time-space decomposition (for example, as in pipeline 220 discussed above) in O(TS) time. 2D +S D T 2 O(S) time, or O(S) without decomposition 2D T 2 In contrast to the time of O(D×S D+1 T 2This can be completed in the time of ). As discussed, processing data in a multidimensional subspace can reduce the amount of redundant data processed within a transform neural network, so processing multidimensional data based on the decomposition of such data into multiple 2D subspaces sharing a common dimension can be computationally significantly less expensive than processing multidimensional data using time-space decomposition. These efficiencies can be understood in relation to the computational costs incurred when processing the data, as it may be possible to generate usable inferences from multidimensional data using a smaller number of samples (e.g., frames in an input video segment) compared to other decomposition techniques. Furthermore, it can be understood that significant reductions in computational costs can be achieved by processing fewer samples using transform neural networks.

[0036] Compared to processing multidimensional data without decomposition (joint attention), the reduction in computational complexity achieved by decomposing multidimensional data into multiple 2D subspaces sharing a common dimension can be expressed by the following equation:

[0037]

number

[0038] The reduction in computational complexity achieved by decomposing multidimensional data into multiple 2D subspaces sharing a common dimension, compared to time-space decomposition of multidimensional data, can be expressed by the following equation:

[0039]

number

[0040] The reduction in computational cost can increase or decrease based on the number of spatial dimensions contained within the multidimensional input. For example, in the case of a sequence of 2D images, the computational complexity involved in processing this sequence through a transform neural network is, compared to joint attention,

[0041]

number

[0042]

number

[0043]

number

[0044]

number

[0045] Figure 3 shows an exemplary operation 300 for processing a multidimensional input through a transformer neural network, based on the decomposition of the multidimensional input into a two-dimensional subspace, according to various aspects of the present disclosure. Operation 300 can be performed by a computing system in which a transformer neural network is deployed to process multidimensional data, such as a user device (UE), smartphone, tablet computer, autonomous vehicle, edge device, or other computing system (e.g., a processing system 400, shown in Figure 4 and described in more detail below).

[0046] As shown in the diagram, operation 300 begins with block 310, which decomposes a multidimensional input into multiple two-dimensional subspaces. These multiple two-dimensional subspaces share a common dimension. Since two-dimensional subspaces generally share one common dimension, the number of two-dimensional subspaces into which an n-dimensional input can be decomposed can be n-1. For example, a three-dimensional space in which the input resides can be decomposed into two two-dimensional subspaces.

[0047] In some embodiments, a multidimensional input may include an input having multiple spatial dimensions and one time dimension. For example, a multidimensional input may be a video clip containing multiple frames (time dimension) containing two spatial data (e.g., height and width). In such a case, the first two-dimensional subspace may be a subspace based on the first spatial dimension and the time dimension of the multiple spatial dimensions, and the second two-dimensional subspace may be a subspace based on the second spatial dimension and the time dimension of the multiple spatial dimensions.

[0048] In block 320, operation 300 proceeds to generate a first attention matrix based on the projection of tokens in a first two-dimensional subspace among multiple two-dimensional subspaces, via the attention block of the transform neural network.

[0049] In some embodiments, generating a first attention matrix based on the projection of tokens in a first two-dimensional subspace includes projecting the first two-dimensional subspace onto query data, key data, and value data. The first attention matrix can be generated based on the query data, key data, and several time components in the first two-dimensional subspace.

[0050] In block 330, operation 300 proceeds to generate a second attention matrix based on the projection of tokens in a second two-dimensional subspace among multiple two-dimensional subspaces, via the attention block of the transform neural network.

[0051] In some embodiments, generating a second attention matrix based on the projection of tokens in a second two-dimensional subspace involves projecting the second two-dimensional subspace onto query data, key data, and value data. The second attention matrix can be generated based on the query data, key data, and several time components in the second two-dimensional subspace.

[0052] In block 340, operation 300 proceeds to generate the output of the transform neural network based on a first attention matrix and a second attention matrix (for example, a combination of the first and second attention matrices). This combination can be, for example, a sum.

[0053] In some embodiments, generating the output of a transform neural network involves computing a first feature based on a first attention matrix and values ​​projected from tokens in a first two-dimensional subspace. A second feature can be computed based on a second attention matrix and values ​​projected from tokens in a second two-dimensional subspace. The first and second features can be combined into a combined feature representing a multidimensional input, and the output of the transform neural network can be generated based on this combined feature.

[0054] In some embodiments, generating an output of a transform neural network based on a combination of features includes generating the output by processing the combination of features through the feedforward components of the transform neural network.

[0055] In some embodiments, operation 300 further includes performing one or more actions based on the output of the generated transform neural network. The output of the generated transform neural network may include, for example, identification information for various parts of the multidimensional input corresponding to various objects or classes of objects, and information identifying the importance of each object or class of objects to the scene. One or more actions may include performing an action on one or more of such objects in the scene, at least in part, based on the importance of the objects to the scene. For example, different levels of compression may be applied to the various parts of the multidimensional input, with a higher level of compression (with increased compression loss) applied to the less important parts of the multidimensional input, and a lower level of compression (with lower or no compression loss) applied to the more important parts of the multidimensional input. In another embodiment, the output of the generated transform neural network may include identification information for objects in the multidimensional space and predictions of how at least some of those identified objects will move through the multidimensional space. One or more actions may include generating one or more control inputs for managing the movement of an autonomous or semi-autonomous device through a multidimensional space. Examples of autonomous or semi-autonomous devices include autonomous vehicles, robotic arms, or other devices capable of moving through a multidimensional space with limited or no human control. Naturally, these outputs, and the actions performed based on those outputs, are illustrative, and it should be recognized that the outputs generated by the transform neural network, and the actions performed based on such outputs, may vary depending on the environment in which the multidimensional input is captured, and the environment in which the device processing and / or using the multidimensional input operates. An exemplary processing system for handling multidimensional inputs in a transform neural network based on subspace decomposition of multidimensional inputs.

[0056] Figure 4 shows an exemplary processing system 400 configured to perform various aspects of the present disclosure, including, for example, the techniques and methods described with respect to Figures 1 to 3. In one aspect, the processing system 400 may train, implement, or deliver machine learning models using a transformer-based architecture such as architecture 100 in Figure 1. Although shown as a single system for clarity of concept, in at least some aspects, the operations described below with respect to the processing system 400, as discussed above, can be distributed across any number of devices.

[0057] The processing system 400 includes a CPU 402, which in some embodiments may be a multicore central processing unit (CPU). Instructions executed in the CPU 402 can be loaded, for example, from program memory associated with the CPU 402, or from a partition of memory 424.

[0058] The processing system 400 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 404, a digital signal processor (DSP) 406, a neural processing unit (NPU) 408, a multimedia processing unit 410, and a wireless connectivity component 412.

[0059] NPUs, such as the NPU408, are generally special circuits configured to execute control and arithmetic logic for running machine learning algorithms, including algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), and random forests (RFs). NPUs may also be referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0060] NPUs such as the NPU408 are configured to accelerate the execution of common machine learning tasks, including image classification, machine translation, object detection, and various other predictive models. In some embodiments, multiple NPUs can be instantiated on a single chip, such as a system on a chip (SoC), while in other embodiments, the NPU can be part of a dedicated neural network accelerator.

[0061] NPUs can be optimized for training or inference, or in some cases, configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still generally be performed independently.

[0062] NPUs, designed to accelerate training, are generally configured to accelerate the optimization of new models, a highly computationally intensive operation that involves inputting existing datasets (often labeled or tagged), iterating through those datasets, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through layers of the model to determine gradients that reduce prediction errors.

[0063] NPUs designed to accelerate inference are generally configured to operate on complete models. Therefore, such NPUs can be configured to take new data as input and process this new data rapidly through a model that has already been trained to generate model outputs (e.g., inferences).

[0064] In one implementation configuration, the NPU408 is part of one or more of the CPU402, GPU404, and / or DSP406.

[0065] In some embodiments, the wireless connectivity component 412 may include sub-components relating to, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless transmission standards. The wireless connectivity component 412 is further connected to one or more antennas 414.

[0066] The processing system 400 may also include a navigation component 420 which may include one or more sensor processing units 416 associated with any type of sensor, one or more image signal processors (ISPs) 418 associated with any type of image sensor, and / or satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0067] The processing system 400 may also include one or more input and / or output devices 422, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone.

[0068] In some embodiments, one or more of the processors in the processing system 400 may be based on an ARM or RISC-V instruction set.

[0069] The processing system 400 also includes memory 424, which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this embodiment, memory 424 includes computer executable components that can be executed by one or more of the aforementioned processors of the processing system 400.

[0070] In particular, in this embodiment, the memory 424 includes a multidimensional input decomposition component 424A, an attention matrix generation component 424B, and an output generation component 424C. In Figure 4, these are shown as separate components for clarity of concept, but the illustrated components (and other components not shown) can be implemented collectively or individually in various embodiments.

[0071] In general, the processing system 400 and / or its components can be configured to perform the methods described herein.

[0072] In particular, in other embodiments, such as when the processing system 400 is a server computer, various embodiments of the processing system 400 can be omitted. For example, the multimedia processing unit 410, wireless connectivity component 412, sensor processing unit 416, ISP 418, and / or navigation component 420 may be omitted in other embodiments. Furthermore, various embodiments of the processing system 400 can be distributed among multiple devices. Exemplary clause

[0073] Details of the various implementations of this disclosure are described in the following numbered clauses.

[0074] Clause 1: A method performed by a processor, comprising: decomposing a multidimensional input into a plurality of two-dimensional subspaces, the plurality of two-dimensional subspaces sharing a common dimension; generating a first attention matrix via an attention block of a transform neural network based on the projection of tokens in a first two-dimensional subspace among the plurality of two-dimensional subspaces; generating a second attention matrix via an attention block of a transform neural network based on the projection of tokens in a second two-dimensional subspace among the plurality of two-dimensional subspaces; and generating an output of a transform neural network based on the first and second attention matrices.

[0075] Clause 2: The method of Clause 1, wherein the multidimensional input includes an input having multiple spatial dimensions and one time dimension.

[0076] Clause 3: The method of Clause 2, wherein multiple spatial dimensions include width and height spatial dimensions, and a common dimension includes time, thereby reducing the computational complexity involved in generating the output of the transform neural network compared to decomposing the multidimensional input into spatial and time components.

[0077] Clause 4: Multidimensional inputs include video inputs, in the manner of Clause 2 or 3.

[0078] Clause 5: The first two-dimensional subspace includes a subspace based on the first spatial dimension and the time dimension of a plurality of spatial dimensions, and the second two-dimensional subspace includes a subspace based on the second spatial dimension and the time dimension of a plurality of spatial dimensions, in any way of Clauses 2 to 4.

[0079] Clause 6: Any method of Clauses 1 to 5, wherein generating the output of a transform neural network includes: computing a first feature based on a first attention matrix and values ​​projected from tokens in a first two-dimensional subspace; computing a second feature based on a second attention matrix and values ​​projected from tokens in a second two-dimensional subspace; combining the first and second features into a combination feature representing a multidimensional input; and generating the output of the transform neural network based on the combination feature.

[0080] Clause 7: The method of Clause 6, wherein the combined feature includes the sum of the first feature and the second feature.

[0081] Clause 8: The method of Clause 6 or 7, wherein generating an output of a transform neural network based on a combination of features includes generating an output by processing the combination of features through a feedforward component of the transform neural network.

[0082] Clause 9: Any method of Clauses 1 to 8, wherein generating a first attention matrix based on the projection of tokens in a first two-dimensional subspace includes projecting the first two-dimensional subspace onto query data, key data, and value data, and generating a first attention matrix based on the query data, key data, and several components from a common dimension in the first two-dimensional subspace.

[0083] Clause 10: Generating a second attention matrix based on the projection of tokens in the second two-dimensional subspace is equivalent to projecting the second two-dimensional subspace onto query data, key data, and value data, and generating a second attention matrix based on the query data, key data, and several components from a common dimension in the second two-dimensional subspace. Any method from clauses 1 to 9, including the above.

[0084] Clause 11: A processing system comprising memory containing computer executable instructions and one or more processors configured to execute computer executable instructions and cause the processing system to perform any of the methods in Clauses 1 to 10.

[0085] Clause 12: A processing system comprising means for carrying out any of the methods specified in Clauses 1 to 10.

[0086] Clause 13: Non-temporary computer-readable media which, when executed by one or more processors of a processing system, includes computer-executable instructions causing the processing system to perform any of the methods described in Clauses 1 to 10.

[0087] Clause 14: A computer program product embodied on a computer-readable storage medium, which includes code for performing any of the methods described in Clauses 1 through 10. Additional considerations

[0088] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The embodiments discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may also be applied to other embodiments. For example, changes may be made to the function and configuration of the elements discussed without departing from the scope of this disclosure. Various embodiments may omit, substitute, or add various procedures or components as needed. For example, the methods described may be performed in an order different from the order described, and various steps may be added, omitted, or combined. Also, features described in some embodiments may be combined in some other embodiments. For example, an apparatus may be implemented or a method may be performed using any number of embodiments described herein. Furthermore, the scope of this disclosure is intended to encompass apparatus or methods that are practiced using other structures, functions, or structures and functions in addition to, or other than, the various embodiments of this disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0089] As used herein, the term “exemplary” means “serving as an example, case, or illustration.” No embodiment described herein as “exemplary” should necessarily be construed as being preferable or advantageous to any other embodiment.

[0090] Where used herein, the phrase “at least one of” the list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to include a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other sequence of a, b, and c).

[0091] As used herein, the term “determining” encompasses a wide range of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0092] The methods disclosed herein include one or more steps or actions to achieve those methods. The steps and / or actions of those methods can be replaced with one another without departing from the claims. In other words, unless a particular order of steps or actions is specified, the order of any particular steps and / or actions, and / or the use of those steps and / or actions, can be modified without departing from the claims. Furthermore, various operations of the methods described above can be performed by any preferred means capable of performing the corresponding functions. These means may include, but are not limited to, various hardware components and / or software components, including circuits, application-specific integrated circuits (ASICs), or processors, and / or various hardware modules and / or software modules. Generally, where operations are shown in figures, those operations may have corresponding equivalent means-plus-function components with similar numbering.

[0093] The following claims are not intended to be limited to the embodiments shown herein, but rather the full scope consistent with the language of the claims should be recognized. Within the claims, a singular reference to an element is intended to mean "one or more" rather than "one and only one" unless explicitly stated otherwise. Unless otherwise stated, the term "several" refers to one or more. No element of a claim should be construed under Section 112(f) of the U.S. Patent Act unless it is explicitly enumerated using the phrase "means to..." or, in the case of a method claim, enumerated using the phrase "steps for...". All structural and functional equivalents of elements of various embodiments described throughout this disclosure, whether known or to those skilled in the art, are expressly incorporated by reference herein and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly enumerated in the claims.

Claims

1. A method that is executed by a processor, Decomposing a multidimensional input into multiple two-dimensional subspaces, wherein these multiple two-dimensional subspaces share a common dimension. A first attention matrix is ​​generated based on the projection of tokens in a first two-dimensional subspace among the plurality of two-dimensional subspaces, via the attention block of the transform neural network. The process involves generating a second attention matrix based on the projection of tokens in a second two-dimensional subspace among the plurality of two-dimensional subspaces, via the attention block of the transform neural network, A method comprising generating the output of the transform neural network based on the first attention matrix and the second attention matrix.

2. The method according to claim 1, wherein the multidimensional input includes an input having multiple spatial dimensions and one time dimension.

3. The method according to claim 2, wherein the plurality of spatial dimensions include a width spatial dimension and a height spatial dimension, and the common dimension includes the time dimension, thereby reducing the computational complexity involved in generating the output of the transform neural network compared to decomposing the multidimensional input into spatial and time components.

4. The method according to claim 2, wherein the multidimensional input includes a video input.

5. The first two-dimensional subspace includes a subspace based on the first spatial dimension among the plurality of spatial dimensions and the time dimension, The second two-dimensional subspace includes a subspace based on the second spatial dimension among the plurality of spatial dimensions and the time dimension. The method according to claim 2.

6. To generate the output of the transform neural network, Computing a first feature based on the first attention matrix and the value projected from the token in the first two-dimensional subspace, The second feature is calculated based on the second attention matrix and the value projected from the token in the second two-dimensional subspace, The first feature and the second feature are combined into a combined feature representing the multidimensional input, The method according to claim 1, comprising generating the output of the transform neural network based on the aforementioned combination features.

7. The method according to claim 6, wherein the combined feature includes the sum of the first feature and the second feature.

8. The method according to claim 6, wherein generating the output of the transform neural network based on the combined features includes processing the combined features through the feedforward components of the transform neural network.

9. The first attention matrix is ​​generated based on the projection of the tokens in the first two-dimensional subspace. Projecting the aforementioned first two-dimensional subspace onto query data, key data, and value data, The method according to claim 1, comprising generating a first attention matrix based on the query data, the key data, and several components from the common dimension in the first two-dimensional subspace.

10. The second attention matrix can be generated based on the projection of the tokens in the second two-dimensional subspace. Projecting the aforementioned second two-dimensional subspace onto query data, key data, and value data, The method according to claim 1, comprising generating a second attention matrix based on the query data, the key data, and several components from the common dimension in the second two-dimensional subspace.

11. It is a system, Memory where executable instructions are stored, A processor configured to execute the aforementioned executable instructions, wherein the executable instructions are configured to execute the aforementioned executable instructions on the system. A multidimensional input is decomposed into multiple two-dimensional subspaces, wherein the multiple two-dimensional subspaces share a common dimension. A first attention matrix is ​​generated based on the projection of tokens in a first two-dimensional subspace among the plurality of two-dimensional subspaces, via the attention block of the transform neural network. A second attention matrix is ​​generated based on the projection of tokens in a second two-dimensional subspace among the plurality of two-dimensional subspaces, via the attention block of the transform neural network. A system that generates the output of the transform neural network based on the first attention matrix and the second attention matrix.

12. The system according to claim 11, wherein the multidimensional input includes an input having multiple spatial dimensions and one time dimension.

13. The system according to claim 12, wherein the plurality of spatial dimensions include a width spatial dimension and a height spatial dimension, and the common dimension includes the time dimension, thereby reducing the computational complexity involved in generating the output of the transform neural network compared to decomposing the multidimensional input into spatial and time components.

14. The system according to claim 12, wherein the multidimensional input includes a video input.

15. The first two-dimensional subspace includes a subspace based on the first spatial dimension among the plurality of spatial dimensions and the time dimension, The second two-dimensional subspace includes a subspace based on the second spatial dimension among the plurality of spatial dimensions and the time dimension. The system according to claim 12.

16. In order to generate the output of the transform neural network, the processor provides the system Based on the first attention matrix and the values ​​projected from the tokens in the first two-dimensional subspace, the first feature is calculated. A second feature is calculated based on the second attention matrix and the value projected from the token in the second two-dimensional subspace. The first feature and the second feature are combined into a combination feature representing the multidimensional input. The system according to claim 11, configured to generate the output of the transform neural network based on the aforementioned combination features.

17. The system according to claim 16, wherein the processor is configured to cause the system to process the combination features through the feedforward components of the transform neural network in order to generate the output of the transform neural network based on the combination features.

18. To generate the first attention matrix based on the projection of the tokens in the first two-dimensional subspace, the processor provides the system with the following information: The first two-dimensional subspace is projected onto query data, key data, and value data. The system according to claim 11, configured to generate a first attention matrix based on the query data, the key data, and several components from the common dimension in the first two-dimensional subspace.

19. To generate the second attention matrix based on the projection of the tokens in the second two-dimensional subspace, the processor provides the system with the following information: The second two-dimensional subspace is projected onto query data, key data, and value data. The system according to claim 11, configured to generate a second attention matrix based on the query data, the key data, and several components from the common dimension in the second two-dimensional subspace.

20. It is a system, A means for decomposing a multidimensional input into multiple two-dimensional subspaces, wherein the multiple two-dimensional subspaces share a common dimension. A means for generating a first attention matrix based on the projection of tokens in a first two-dimensional subspace among the plurality of two-dimensional subspaces via an attention block of a transform neural network, A means for generating a second attention matrix based on the projection of tokens in a second two-dimensional subspace among the plurality of two-dimensional subspaces via the attention block of the transform neural network, A system comprising means for generating the output of the transform neural network based on the first attention matrix and the second attention matrix.

21. The system according to claim 20, wherein the multidimensional input includes an input having multiple spatial dimensions and one time dimension.

22. The system according to claim 21, wherein the plurality of spatial dimensions include a width spatial dimension and a height spatial dimension, and the common dimension includes the time dimension, thereby reducing the computational complexity involved in generating the output of the transform neural network compared to decomposing the multidimensional input into spatial and time components.

23. The first two-dimensional subspace includes a subspace based on the first spatial dimension among the plurality of spatial dimensions and the time dimension, The second two-dimensional subspace includes a subspace based on the second spatial dimension among the plurality of spatial dimensions and the time dimension. The system according to claim 21.

24. The means for generating the output of the transform neural network is A means for calculating a first feature based on the first attention matrix and the value projected from the token in the first two-dimensional subspace, A means for calculating a second feature based on the second attention matrix and the value projected from the token in the second two-dimensional subspace, A means for combining the first feature and the second feature to form a combined feature representing the multidimensional input, The system according to claim 20, comprising means for generating the output of the transform neural network based on the aforementioned combination features.

25. The system according to claim 24, wherein the means for generating the output of the transform neural network based on the combined features includes means for processing the combined features through the feedforward components of the transform neural network.

26. The means for generating the first attention matrix based on the projection of the tokens in the first two-dimensional subspace, Means for projecting the first two-dimensional subspace onto query data, key data, and value data, The system according to claim 20, comprising means for generating a first attention matrix based on the query data, the key data, and several components from the common dimension in the first two-dimensional subspace.

27. The means for generating the second attention matrix based on the projection of the tokens in the second two-dimensional subspace, Means for projecting the second two-dimensional subspace onto query data, key data, and value data, The system according to claim 20, comprising means for generating a second attention matrix based on the query data, the key data, and several components from the common dimension in the second two-dimensional subspace.

28. A computer-readable medium storing executable instructions, wherein when the executable instructions are executed by a processor, Decomposing a multidimensional input into multiple two-dimensional subspaces, wherein these multiple two-dimensional subspaces share a common dimension. A first attention matrix is ​​generated based on the projection of tokens in a first two-dimensional subspace among the plurality of two-dimensional subspaces, via the attention block of the transform neural network. The process involves generating a second attention matrix based on the projection of tokens in a second two-dimensional subspace among the plurality of two-dimensional subspaces, via the attention block of the transform neural network, A computer-readable medium that performs an operation including generating the output of the transform neural network based on the first attention matrix and the second attention matrix.

29. The computer-readable medium according to claim 28, wherein the multidimensional input includes an input having multiple spatial dimensions and one time dimension.

30. The computer-readable medium according to claim 29, wherein the plurality of spatial dimensions include a width spatial dimension and a height spatial dimension, and the common dimension includes the time dimension, thereby reducing the computational complexity involved in generating the output of the transform neural network compared to decomposing the multidimensional input into spatial and time components.

31. The first two-dimensional subspace includes a subspace based on the first spatial dimension among the plurality of spatial dimensions and the time dimension, The second two-dimensional subspace includes a subspace based on the second spatial dimension among the plurality of spatial dimensions and the time dimension. The computer-readable medium according to claim 29.

32. To generate the output of the transform neural network, Computing a first feature based on the first attention matrix and the value projected from the token in the first two-dimensional subspace, The second feature is calculated based on the second attention matrix and the value projected from the token in the second two-dimensional subspace, The first feature and the second feature are combined into a combined feature representing the multidimensional input, The computer-readable medium according to claim 28, comprising generating the output of the transform neural network based on the aforementioned combination features.

33. The computer-readable medium according to claim 32, wherein generating the output of the transformer neural network based on the combined features includes processing the combined features through the feedforward components of the transformer neural network.

34. The first attention matrix is ​​generated based on the projection of the tokens in the first two-dimensional subspace. Projecting the aforementioned first two-dimensional subspace onto query data, key data, and value data, The computer-readable medium according to claim 28, comprising generating a first attention matrix based on the query data, the key data, and several components from the common dimension in the first two-dimensional subspace.

35. The second attention matrix can be generated based on the projection of the tokens in the second two-dimensional subspace. Projecting the aforementioned second two-dimensional subspace onto query data, key data, and value data, The computer-readable medium according to claim 28, comprising generating a second attention matrix based on the query data, the key data, and several components from the common dimension in the second two-dimensional subspace.

Images