Configurable neural network kernels

Abstract

Techniques for manipulating generic tensors are provided, including receiving a first request to perform a first operation on a generic tensor descriptor associated with the generic tensor, performing the first operation on the generic tensor descriptor in response to the first request, receiving a second request to perform a second operation on generic tensor raw data associated with the generic tensor, and performing the second operation on the generic tensor raw data in response to the second request.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 62 / 867,766, entitled "COMPOSABLE NEURAL NETWORK KERNELS," filed June 27, 2019, the entire contents of which are incorporated herein by reference. This application claims the benefit of U.S. Provisional Patent Application No. 62 / 925,168, entitled "TENSOR COORDINATE SPACE TRANSFORMATION IN NEURAL NETWORKS," filed October 23, 2019, the entire contents of which are incorporated herein by reference. This application claims the benefit of U.S. Provisional Patent Application No. 62 / 927,603, entitled "TENSOR COORDINATE SPACE TRANSFORMATION IN NEURAL NETWORKS," filed October 29, 2019, the entire contents of which are incorporated herein by reference. This application claims the benefit of U.S. Patent Application No. 16 / 779,557, entitled "COMPOSABLE NEURAL NETWORK KERNELS," filed January 31, 2020, the entirety of which is incorporated herein by reference. [Background technology]

[0002] The ever-increasing demand for computational power by modern neural networks has spurred a race to develop faster and more efficient hardware and software stacks for high-performance computing for neural networks. However, different types of neural network computations make such development a challenging task. First, neural networks often consist of complex calculations on tensors, and different networks may use different types of tensor computations. Second, a vast variety of dimensions, sizes, and choices of data layouts exist in memory for these tensors. Third, the computations need to run efficiently on many types of existing and new hardware. Improvements to computation for neural networks are needed.

[0003] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of an example device capable of implementing one or more features of the present disclosure. [Figure 2] 2 is a block diagram of the device of FIG. 1 illustrating additional details associated with executing processing tasks on an accelerated processing device, according to an example. [Figure 2A] FIG. 1 illustrates an exemplary embedding operation. [Figure 2B] FIG. 1 illustrates an exemplary embedding operation. [Figure 3] FIG. 1 is a block diagram of a generic tensor manipulation system for manipulating generic tensors, according to an example. [Figure 4A] FIG. 1 illustrates some example embodiments of a general tensor manipulation system. [Figure 4B] FIG. 1 illustrates some example embodiments of a general tensor manipulation system. [Figure 4C]FIG. 1 illustrates some example embodiments of a general tensor manipulation system. [Figure 5] FIG. 1 illustrates an example of a convolutional neural network in which operations for performing computations through the neural network are performed using operations on general tensors as described herein. [Figure 5A] FIG. 1 illustrates an exemplary technique for converting a convolution operation into a GEMM operation using general tensor operations. [Figure 6] FIG. 1 is a flow diagram of a method for performing tensor operations, according to an example. DETAILED DESCRIPTION OF THE INVENTION

[0005] Techniques are provided for manipulating generic tensors, including receiving a first request to perform a first operation on a generic tensor descriptor associated with the generic tensor, performing the first operation on the generic tensor descriptor in response to the first request, receiving a second request to perform a second operation on generic tensor raw data associated with the generic tensor, and performing the second operation on the generic tensor raw data in response to the second request.

[0006] 1 is a block diagram of an example device 100 capable of implementing one or more features of the present disclosure. Device 100 may include, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. Device 100 includes a processor 102, a memory 104, a storage device 106, one or more input devices 108, and one or more output devices 110. Device 100 optionally includes an input driver 112 and an output driver 114. It should be understood that device 100 may include additional components not shown in FIG. 1 .

[0007] In various alternatives, processor 102 includes a central processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU located on the same die, or one or more processor cores, each of which may be a CPU or a GPU. In various alternatives, memory 104 is located on the same die as processor 102 or is located separately from processor 102. Memory 104 may include volatile or non-volatile memory (e.g., random access memory (RAM), dynamic RAM, cache).

[0008] The storage devices 106 include fixed or removable storage devices (e.g., hard disk drives, solid state drives, optical disks, flash drives). The input devices 108 include, but are not limited to, a keyboard, a keypad, a touch screen, a touch pad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals). The output devices 110 include, but are not limited to, a display device 118, a speaker, a printer, a haptic feedback device, one or more lights, an antenna, or a network connection (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals).

[0009] The input driver 112 communicates with the processor 102 and the input device 108, allowing the processor 102 to receive input from the input device 108. The output driver 114 communicates with the processor 102 and the output device 110, allowing the processor 102 to send output to the output device 110. Note that the input driver 112 and the output driver 114 are optional components, and that the device 100 operates similarly without the input driver 112 and the output driver 114. The output driver 114 includes an accelerated processing device (APD) 116 coupled to a display device 118. The APD 116 is configured to accept computational commands and graphics rendering commands from the processor 102, process these computational commands and graphics rendering commands, and provide pixel output to the display device 118 for display. As described in more detail below, the APD 116 includes one or more parallel processing units configured to perform computations according to a single instruction, multiple data (SIMD) paradigm. Thus, although various functions are described herein as being performed by or in conjunction with APD 116, in various alternatives, the functions described as being performed by APD 116 are additionally or alternatively performed by other computing devices having similar capabilities that are not driven by a host processor (e.g., processor 102), and that are configured to provide (graphical) output to display device 118. For example, it is contemplated that any processing system that performs processing tasks according to the SIMD paradigm may be configured to perform the functions described herein.

[0010] FIG. 2 is a block diagram of device 100 illustrating additional details related to the execution of processing tasks on APD 116, according to an example. Processor 102 maintains one or more control logic modules in system memory 104 for execution by processor 102. The control logic modules include operating system 120, drivers 122, and applications 126. These control logic modules control various aspects of the operation of processor 102 and APD 116. For example, operating system 120 communicates directly with hardware and provides an interface to the hardware for other software running on processor 102. Driver 122 controls the operation of APD 116, for example, by providing an application programming interface (API) for software running on processor 102 (e.g., applications 126) to access various features of APD 116. In some embodiments, driver 122 includes a just-in-time compiler that compiles programs executed by processing components of APD 116 (such as SIMD unit 138, described in more detail below). In other embodiments, a just-in-time compiler is not used to compile the programs, but rather a standard application compiler compiles the shader programs that run on the APD 116.

[0011] APD 116 executes commands or programs suitable for parallel and / or out-of-order processing, such as graphics and non-graphics operations. APD 116 is used to perform graphics pipeline operations, such as pixel operations, geometric calculations, and rendering of images to display device 118, based on commands received from processor 102. APD 116 performs computational operations not related or directly related to graphics operations, such as operations related to video, physics simulation, computational fluid dynamics, or other tasks, based on commands received from processor 102. APD 116 performs computational operations related to ray tracing-based graphics rendering.

[0012] The APD 116 includes a compute unit 132 that includes one or more SIMD units 138 that perform operations in parallel as required by the processor 102 according to the SIMD paradigm. The SIMD paradigm allows multiple processing elements to share a single program control flow unit and program counter, allowing them to execute the same program but with different data. In one example, each SIMD unit 138 includes 16 lanes, each of which executes the same instruction simultaneously with the other lanes of the SIMD unit 138, but with different data. Predication can be used to turn off lanes when not all lanes need to execute a given instruction. Prediction can also be used to execute programs with branching control flow. More specifically, for programs with instructions such as conditional branches where control flow is based on calculations performed in individual lanes, predicting the lane corresponding to the currently unexecuted control flow path and executing the different control flow paths serially allows for arbitrary control flow. In an embodiment, each compute unit 132 can have a local L1 cache. In an embodiment, multiple compute units 132 share an L2 cache.

[0013] The basic unit of execution in compute unit 132 is the work item. Each work item represents a single instantiation of a program executing in parallel on a particular lane. Work items may execute together as a "wavefront" on a single SIMD processing unit 138. The SIMD nature of SIMD processing unit 138 means that multiple work items typically execute simultaneously in parallel. Work items executing together on a single SIMD unit in this manner are part of the same wavefront. In some embodiments or modes of operation, SIMD unit 138 executes a wavefront by simultaneously executing each of the work items of the wavefront. In other embodiments or modes of operation, SIMD unit 138 executes different subsets of the work items within the wavefront in parallel. In one example, a wavefront includes 64 work items, and SIMD unit 138 has 16 lanes (each lane is a unit of hardware sized to execute a single work item). In this example, SIMD unit 138 executes a wavefront by simultaneously executing 16 work items four times.

[0014] One or more wavefronts are included in a "workgroup," which contains a collection of work items designed to execute the same program. An application or other entity (host) requests a shader program to be executed by the accelerated processing device 116 by specifying a "size" (number of work items), and the command processor 136 creates one or more workgroups to execute the work. In various embodiments, some or all of the number of workgroups, the number of wavefronts in each workgroup, and the number of work items in each wavefront depend on the size of the work requested by the host. In one example, several work items in a workgroup execute a shader program specified with the size of a particular request. In that example, the APD 116 spawns several workgroups based on the size and number of work items per workgroup. As described above, the command processor 136 dispatches the workgroups to one or more compute units 132, which execute the appropriate number of wavefronts to complete the workgroup. In some embodiments, each workgroup is limited to a single compute unit 132.

[0015] The parallelism provided by the computation units 132 is well suited to graphics-related operations such as pixel value calculations, vertex transformations, and other graphics operations. Thus, in some examples, the graphics pipeline 134, which accepts graphics processing commands from the processor 102, provides computational tasks to the computation units 132 for execution in parallel.

[0016] In various embodiments, any of the above systems are used to perform computations to implement an artificial neural network. A neural network is modeled as layers of interconnected neurons. Each layer has connectivity to the previous layer (or to the inputs to the network) and to the subsequent layer (or to the outputs of the network). Each neuron has one or more inputs. Each input is either an input value from a previous layer or an input to the neural network. Each input has an associated weight. Each neuron defines a neuron function. This function defines how the inputs and weights are combined to generate an output.

[0017] This disclosure presents several operations that can be combined to form an artificial neural network "kernel." Here, the term "kernel" refers to a program, a portion of code, or a hardware operation. A kernel may be executed multiple times on very large chunks of data. For example, a kernel may perform calculations to determine the output from neurons in a particular layer of a neural network. Some kernels run as individual programs or threads on a central processing unit or graphics processing unit. Other kernels are application programming interface (API) calls. Other kernels are hardware (fixed-function circuit) implemented operations that can be invoked by instructions in an instruction set architecture. Still other kernels are operations implemented by a compiler.

[0018] Individual operations may, in various embodiments, be performed by software code running on a programmable processor (e.g., a CPU or GPU), by a compiler, by hardware that performs operations in response to invocations such as API calls, by hardware that performs operations as hardware instructions (e.g., as part of an instruction set architecture), or by any other entity.

[0019] Some operations operate on general tensor descriptors rather than on the general tensor raw data itself. Such operations include slice, strided slice, reorder, unfold, merge, embed, and move slicing window. The slicing operator "cuts out" a given portion of a tensor and generates a new general tensor descriptor. Specifically, slicing modifies a general tensor descriptor by restricting the possible index values ​​in one or more dimensions. For a given dimension, the range of possible indices to which index generation is restricted in the sliced ​​tensor is called the "slicing window." The number of possible index values ​​in a given dimension is called the "slicing window length" for that dimension. In the above example, one slice restricts the possible index values ​​in the z dimension to 0 to 3 (out of 0 to 7). A different slice restricts the possible index values ​​in the x dimension to 0 and the possible index values ​​in the y dimension to 0 and 1.

[0020] To facilitate computational processing of data through one or more neurons in a particular layer, the relevant data is stored in memory in a specific data format. Specifically, inputs, weights, and outputs are stored as tensors with a specified data format. A tensor, also referred to in this document as a "standard tensor," contains a defined number of dimensions. Each dimension has a stride and a length. Addressing a tensor is done using one index for each dimension. The length of a dimension is the number of distinct index values ​​in that dimension. The stride of a dimension determines the memory address offset for each increment in that dimension. For example, a 3D tensor has three dimensions: x, y, and z, with lengths Lx, Ly, and Lz, respectively. In the x dimension, the stride is Sx; in the y dimension, the stride is Sy; and in the z dimension, the stride is Sz. The element indices for the x, y, and z dimensions are Ix, Iy, and Iz, respectively. This corresponds to a layout where incrementing the x-dimension index Ix by 1 corresponds to an increment in memory addresses of Sx*data_type_size, incrementing the y-dimension index Iy by 1 corresponds to an increment in memory addresses of Sy*data_type_size, and incrementing the z-dimension index Iz by 1 corresponds to an increment in memory addresses of Sz*data_type_size. Here, data_type_size is the size of the data type of the tensor elements, e.g., 4 bytes for single-precision floating-point values, 2 bytes for half-precision floating-point values, etc. Data organized in such a memory layout is referred to herein as "standard tensor raw data." More specifically, standard tensor raw data is simply the data elements of a tensor laid out at memory addresses as described above.

[0021] In this example 3D tensor, the tuple of indices for the x, y, and z dimensions (Ix, Iy, Iz) is called a multidimensional index, or "multi-index." The standard tensor in this example contains Lx*Ly*Lz elements, with multi-indexes ranging from (0,0,0) to (Lx-1,Ly-1,Lz-1). The element with multi-index (0,0,0) is called the "point of origin" of the tensor. The memory address distance from an element to the "point of origin" divided by data_size_type is called the "offset" of the element. The "offset" of an element with multi-index (Ix,Iy,Iz) can be calculated as Ix*Sx+Iy*Sy+Iz*Sz.

[0022] A "general tensor" is an extension of the definition of a "standard tensor." A "general tensor" has a defined number of dimensions. Each dimension has a length, although in some circumstances, one or more dimensions do not have a constant stride. A general tensor is described by both a general tensor descriptor and general tensor raw data. A general tensor descriptor indicates whether a general tensor is a standard tensor, a coupled tensor, or an embedded tensor. Furthermore, a general tensor descriptor indicates the number of dimensions of the general tensor, the length of each dimension, and the base address of the general tensor raw data with which the general tensor descriptor is associated. Thus, a general tensor descriptor indicates how multi-indexes map to the general tensor raw data stored in memory. The general tensor descriptor facilitates the appropriate mapping of elements of a multi-index to elements of the general tensor raw data. A general tensor type also indicates how multi-indexes of a general tensor descriptor map to the general tensor raw data. Standard tensors are described elsewhere in this specification. With general tensors stored in memory, the origin corresponds to the base address of the general tensor.

[0023] A combined tensor is a tensor that contains data from standard tensor raw data, but the mapping from the combined tensor's indices to the general tensor raw data is different from that of a standard tensor. More specifically, to generate a combined tensor, multiple dimensions of a standard tensor are selected and "combined" so that the combined dimension acts as if it were a single dimension. For a combined dimension, the multiple dimensions "combined" into the combined dimension are referenced by a single index value. The resulting tensor can be indexed using a multi-index, where at least one index value of the multi-index is a scalar value that applies to the combined dimension created from multiple individual dimensions of the general tensor raw data.

[0024] Continuing with the above example, the general tensor has x-length Lx=2, y-length Ly=4, z-length Lz=8, x-stride Sx=100, y-stride Sy=10, and z-stride Sz=1. This general tensor has Lx*Ly*Lz=64 elements, and its multi-index ranges from (0,0,0) to (1,3,7). In an exemplary join operation, the join operation selects the y and z dimensions and generates a join tensor in which the y and z dimensions act as a single dimension. This single dimension is referred to herein as the w-dimension, for example. Thus, a join tensor with two dimensions (x and w) is generated. This join tensor has x-length Lx=2, x-stride Sx=100, and x-index Ix ranging from 0 to Lx-1 (equal to 1). Note that the x-length and x-stride of this join tensor are identical to those of a standard tensor.

[0025] This combined tensor has a length of Lw = Ly * Lz = 32 for w and has an index Iw in the w dimension. The index in the w dimension is mapped from indices y and z (Iy, Iz) as Iw = Lz * Iy + Iz = 8 * Iy + Iz and takes values in the range 0 to 31. Conversely, the mapping from index Iw to indices (Iy, Iz) can be expressed as Iy = Iw / Lz = Iw / 8 and Iz = Iw % Lz = Iw % 8. Note that the w stride in the combined w dimension is not constant. The reason is that increasing the w dimension index Iw by 1 does not always increase the offset by a constant amount. Rather, the increment of the offset when Iw is incremented by 1 depends on the value of index Iz. More specifically, by increasing Iw by 1, when Iz < Lz - 1 = 7, the increment of the offset is Sz = 1, but when Iz = Lz - 1 = 7, the increment is Sy - (Lz - 1) * Sz = 3.

[0026] The selection of the combined dimensions is indicated in the general tensor descriptor of the combined tensor. Obtaining the offset to the general tensor raw data when accessing the data of the general tensor is done by converting the multi - index (Ix, Iw) of the combined tensor into the address space of the general tensor raw data. This can be done in any technically feasible way. In one example, the combined multi - index (Ix, Iw) for the combined tensor is converted into the corresponding multi - index (Ix, Iy, Iz) for the standard tensor raw data, and this multi - index is then converted into a memory offset. In another example, the increment of the combined multi - index (Ix, Iw) is converted into the increment of the multi - index (Ix, Iy, Iz), and this increment is then converted into the increment of the memory offset. The rules for these index conversions are determined based on the general tensor descriptor. Generally, combination reduces the dimensions of the source tensor.

[0027] An embedding tensor is a tensor that contains the data of standard tensor raw data, but the mapping from embedding tensor indices to the general tensor raw data for an embedding tensor is different from that for standard tensors. More specifically, to generate an embedding tensor, one dimension of the input standard tensor descriptor is selected, and that dimension is "embedded" into multiple dimensions of the embedding tensor descriptor, so that the selected dimension of the input standard tensor acts as if it contained multiple dimensions in the "embedding" tensor.

[0028] The embedding operation selects one of the dimensions of a standard tensor, exemplarily referred to herein as the x-dimension, with length Lx and stride Sx, and embeds w0, w1, ... w from the x-dimension. n-1 Create dimensions, where the w0 dimension has length Lw0, the w1 dimension has length Lw1, the w2 dimension has length Lw2, and so on. The values ​​of n are given by w0, w1, ... w n-1 Dimension lengths Lw0,Lw1,...Lw n-1 Similarly, w0, w1,... w are specified by embedding operations. n-1 Dimension indices (Iw0, Iw1,...Iw n-1 ) to the index of the x-dimension Ix is Ix=E0*Iw0+E1*Iw1+...+E n-1 *Iw n-1 +E n Here, E0,E1…E n are called "embedding coefficients." These coefficients are integers whose values ​​are also specified by the embedding operation.

[0029] Continuing with the description of the embedding tensor descriptor generated by the embedding operation above, the w0, w1, ... w generated by the embedding operation n-1 Note that dimensions have deterministic strides: the w0 dimension has stride Sw0 = E0 * Sx, the w1 dimension has stride Sw1 = E1 * Sx, and so on, and w n-1 Dimension is Stride Sw n-1 =E n-1*Sx. As a result, the embedded tensor descriptor produced by the embedding operation can be thought of as a special case of a standard tensor descriptor. Also, note that elements of an embedded tensor may overlap in memory. This is because the elements of an embedded tensor may overlap in memory, as indicated by the indices (Iw0, Iw1, ... Iw n-1 ) sometimes map to the same index value Ix, meaning that this value maps to the same memory location.

[0030] In the exemplary embedding operation shown in FIG. 2A, a 2D standard tensor has x and y dimensions, with the lengths of the x and y dimensions being Lx=6 and Ly=10, and the strides of the x and y dimensions being Sx=10 and Sy=1. This standard tensor has Lx*Ly=60 elements, and its multi-index ranges from (0,0) to (5,9). In the exemplary embedding operation, the embedding operation selects the x dimension of the input standard tensor and embeds the x dimension into two dimensions (w0 and w1 dimensions). The embedding operation sets the lengths of the w0 and w1 dimensions to Lw0=2 and Lw1=2, and sets the embedding coefficients to E0=2, E1=1, and E2=1. The resulting embedded tensor descriptor produced by the embedding operation is a 3D embedded tensor descriptor with the dimensions w0, w1, and y. The mapping from the w0 and w1 dimension indices (Iw0, Iw2) to the x dimension index Ix is Ix = E0 * Iw0 + E1 * Iw1 + E2 = 2 * Iw0 + Iw1 + 1. The length of the embedding tensor descriptor is Lw for the w0, w1, and y dimensions, respectively. 0=2, Lw1 = 2, and Ly = 10. The strides of the embedding tensor descriptor are Sw0 = E0 * Sx = 20, Sw1 = E1 * Sx = 10, and Sy = 1 for the w0, w1, and y dimensions, respectively. Note that the resulting embedding tensor has Lw0 * Lw1 * Ly = 40 elements, with its multi-index ranging from (0,0,0) to (1,1,9). The embedding tensor has fewer elements than the original standard tensor (40 vs. 60). Each element of the embedding tensor maps to a unique element of the source standard tensor, but some elements of the source standard tensor do not map to any element of the embedding tensor.

[0031] The second example embedding operation shown in FIG. 2B uses the same source standard tensor, but the embedding operation is different, resulting in a different embedded tensor with overlapping elements in memory. In this example, the embedding operation selects the x dimension of the input standard tensor and embeds the x dimension into two dimensions (w0 and w1 dimensions). The embedding operation sets the lengths of the w0 and w1 dimensions as Lw0 = 3 and Lw1 = 4, and sets the embedding coefficients E0 = 1, E1 = 1, and E2 = 0. The resulting embedded tensor descriptor produced by the embedding operation is a 3D embedded tensor descriptor with w0, w1, and y dimensions. The mapping from the indices (Iw0, Iw2) of the w0 and w1 dimensions to the index Ix of the x dimension is Ix = E0 * Iw0 + E1 * Iw1 + E2 = Iw0 + Iw1. The lengths of the embedding tensor descriptor are Lw0 = 3, Lw1 = 4, and Ly = 10 for the w0, w1, and y dimensions, respectively. The strides of the embedding tensor descriptor are Sw0 = E0 * Sx = 10, Sw1 = E1 * Sx = 10, and Sy = 1 for the w0, w1, and y dimensions, respectively. Note that the resulting embedding tensor has Lw0 * Lw1 * Ly = 120 elements, with its multi-index ranging from (0,0,0) to (2,3,9). Even though the resulting embedding tensor contains the same tensor raw data in memory as the original standard tensor, the embedding tensor has more elements than the original standard tensor (120 vs. 60). This is because several different elements of the embedding tensor map to the same elements of the source standard tensor, and the elements map to the same memory locations.

[0032] A strided slice is an extension of a slice. Like a slice, a strided slice limits the possible index values ​​in one or more dimensions, but the range of bounded indices is not necessarily contiguous. For example, in a tensor with length 7 in the z dimension, a slice with a stride equal to 2 and a slicing window length of 3 limits the range of indices in the z dimension to 0, 2, and 4.

[0033] The reordering operation changes the order of dimensions of a general tensor. For example, a 3D standard tensor "TensorA" with x, y, and z as its first, second, and third dimensions has lengths Lx, Ly, and Lz and strides Sx, Sy, and Sz for its first, second, and third dimensions. The reordering operation creates a new general tensor "TensorB" with the new order of dimensions, where z, y, and x are its first, second, and third dimensions. TensorB has lengths Lz, Ly, and Lx and strides Sz, Sy, and Sx for its first, second, and third dimensions. "TensorA" and "TensorB" share the same tensor raw data in memory. Elements of TensorA with multi-index (Ix, Iy, Iz) have the same memory addresses as elements of TensorB with multi-index (Iz, Iy, Ix).

[0034] The convolution operation increases the dimension of a general tensor descriptor by generating an additional dimension from a single dimension. The product of the lengths of the resulting dimensions is equal to the lengths of the dimensions being convolved. The lengths of the resulting dimensions can be chosen in any way. The strides of each such dimension are as follows: One of the new dimensions is given the stride of the dimension being convolved; the other dimension is assigned the product of the stride and the length of the other new dimension. In the example of x, y, z, dimension z is convolved into z1, z2, and z3, each with length 2. Z3 has stride 1 and length 2. Z2 has stride 2×1=2 and length 2. Z1 has stride 2×2=4.

[0035] The combine operation generates a combined tensor descriptor from other general tensor descriptors. Specifically, the combine operation reduces at least some of the dimensions of the general tensor descriptors to 1. The concept of combined tensors is described elsewhere in this specification.

[0036] An embedding operation embeds one dimension of an input tensor into multiple dimensions to produce an embedded tensor.

[0037] A moving slicing window operation establishes a window (i.e., a set of index values) over which the slicing operation is performed.

[0038] Note that a general tensor descriptor is essentially a means for addressing data within the general tensor raw data.

[0039] The following operations operate on the actual data itself: slicing, reordering, copying and combinations thereof, general matrix multiplication (GEMM) or batched GEMM on 2D or 3D general tensors, reduction of n-dimensional general tensors, algorithm-specified transformations of n-dimensional general tensors, and the following operations operate on the data in the general tensor raw data:

[0040] The slice, reorder and copy operations generate new or modify existing general tensor raw data in a similar manner to above, except that these operations operate on the actual general tensor raw data instead of on general tensor descriptors. For example, it is possible to have a sliced ​​and reordered copy, a sliced ​​copy, a reordered copy, etc. from one memory to another.

[0041] The GEMM (general matrix multiplication) and batched GEMM operations on general 2D and 3D tensors are similar to the traditional non-batched and batched matrix multiplication operations. These operations generate new general tensor raw data based on the matrix multiplication of two input general tensors (raw data and a general tensor descriptor). Note that both the input and the result of a GEMM or batched GEMM operation can be general tensors, not just "standard" tensors.

[0042] A general tensor reduction operation takes multiple input general tensors with the same number of dimensions and the same length for each dimension, and reduces these multiple input general tensors to a single output general tensor with the same number of dimensions and the same length for each dimension as the multiple input general tensors. Each element with multi-index (Ix, Iy, Iz) of the output general tensor is the result of the reduction of elements with multi-index (Ix, Iy, Iz) from all input general tensors. The reduction operation on an element can be max, min, summation, etc.

[0043] In any of the above operations, the resulting tensor may be written to any memory, including a memory different from the memory in which the operand tensors originated.

[0044] In some embodiments, the operations are API functions that can be called by software. In some embodiments, the operations are operations that are performed by a compiler at compile time. In some embodiments, operations that operate on general tensor descriptors are compile-time operations, and operations that operate on the data itself are run-time operations. In various embodiments, either operation is implemented as a hardware instruction as part of an instruction set architecture, or is invoked in any technically feasible manner.

[0045] Figure 3 is a block diagram of a general tensor manipulation system 300 for manipulating general tensors, according to one example. The general tensor manipulation system 300 includes a tensor manipulator requestor 302 and a tensor manipulator 304. The tensor manipulator requestor 302 requests that the tensor manipulator 304 perform a tensor operation. The tensor operation may include any of the operations on general tensor raw data 306 and / or general tensor descriptors 308 described elsewhere herein. The tensor manipulator 304 may be implemented as software running on a processor, hardware (e.g., circuitry), or a combination thereof. The tensor operation requestor 302 may also be implemented as software running on a processor, hardware (e.g., circuitry), or a combination thereof.

[0046] 4A-4C illustrate some example embodiments of a general tensor manipulation system 300. In FIG. 4A, a processor 402 executes a program 403 that contains tensor manipulation requests. In some embodiments, the program itself includes a tensor manipulator 404, while in other embodiments, the program 403 accesses other software (such as a library or a different thread or process) that includes a tensor manipulator 406. The tensor manipulator 404 within the program or a tensor manipulator 406 external to the program 403 manipulates the general tensor raw data 306 and the general tensor descriptor 308 according to the operations for manipulating general tensor raw data and general tensor descriptors described elsewhere herein.

[0047] 4B, the compilation system 418 includes a processor 410 that executes a compiler 412. The compiler 412 parses a program source 414. The program source 414 includes operations that invoke manipulation of general tensor descriptors. In some circumstances, the program source 414 includes other operation requests, such as requests to manipulate general tensor raw data 306.

[0048] In the compilation system 418, the compiler 412 compiles the program 414 to generate a compiled program 416. The compilation includes processing operations that invoke generic tensor descriptor manipulation. More specifically, the program source 414 includes at least one operation for manipulating generic tensor descriptors, examples of which are provided herein. The program source 414 includes operations that use the manipulated generic tensor descriptors. Instead of compiling the program 414 so that the program 414 performs operations for manipulating generic tensor descriptors, the compiler 412 itself performs the operations for manipulating generic tensor descriptors. The compiler 412 compiles operations that use the manipulated generic tensor descriptors to use the generic tensor descriptors as modified by the compiler 412. These compiler-based generic tensor descriptor manipulation operations are possible for generic tensor descriptors known at compile time, resulting in improved performance of the compiled program 416 compared to performing such operations by the compiled program 416. The compiler 412 compiles other operation requests, including requests to operate on the general tensor raw data 306 to produce a compiled program 416.

[0049] The execution system 420 includes a processor 422 that executes the compiled program 416. When executed, the compiled program performs operations that manipulate the general tensor raw data 306 as specified by the program source 414, and accesses the manipulated general tensor descriptors 308 and the manipulated general tensor raw data 306.

[0050] In some embodiments, compilation system 418 is the same system as execution system 420 , and in other embodiments, compilation system 418 is a different system than execution system 420 .

[0051] In FIG. 4C , the hardware assistance system 430 includes a processor 432 that includes a program 434. The program includes tensor manipulation instructions. When executed, the program 434 requests the tensor manipulation instruction circuit 436 to perform the requested tensor manipulation instructions. The tensor manipulation instructions include one or more of instructions for manipulating the general tensor raw data 306 and / or instructions for manipulating the general tensor descriptor 308. In some embodiments, these instructions are part of an instruction set architecture, and the tensor manipulation instruction circuit 436 executes such instructions. Execution of the instructions includes manipulating the general tensor raw data 306 and / or manipulating the general tensor descriptor 308.

[0052] 4A-4C, the general tensor raw data 306 and the general tensor descriptor 308 may be stored in any suitable location, such as memory. In the descriptions of FIGS. 4A-4C, phrases such as "operating on general tensor raw data" refer to performing one or more of the operations described herein as being performed on general tensor raw data, such as slicing, reordering, copying and combinations thereof, general matrix multiplication, batched general matrix multiplication, reduction of n-dimensional general tensors, algorithmically specified transformations of n-dimensional general tensors, etc. Phrases such as "operating on general tensor descriptors" refer to performing one or more of the operations described herein as being performed on general tensor descriptors, such as slicing, strided slicing, reordering, combining, embedding, convolution, and moving slicing windows, etc.

[0053] 4A to 4C are diagrams showing examples of the tensor operation request unit 302 and the tensor manipulator 304. In FIG. 4A, the tensor operation request unit 302 is a program 403, and the tensor manipulator 304 is a tensor manipulator 404. In FIG. 4B, the tensor operation request unit 302 is a program source 414 for manipulating the general tensor descriptor 308, and is a compiled program 416 of the general tensor raw data 306. The tensor manipulator 304 is a compiler 412 for the general tensor descriptor 308, and is a compiled program 416 of the general tensor raw data 306. In FIG. 4C, the tensor manipulator 304 is a tensor operation instruction circuit 436, and the tensor operation request unit 302 is a program 434.

[0054] Any neural network can be implemented by combining the above operations. Figure 5 shows a portion of an exemplary convolutional neural network in which the operations for implementing computations through the neural network are implemented using the operations on general tensors described herein. The neural network is modeled as layers of interconnected neurons. Each layer has connectivity to the previous layer (or to the inputs to the network) and to the subsequent layer (or to the output of the network). While a single convolution is shown, it should be noted that an entire neural network including one or more convolutions and other layers such as GEMM, activation, and shrinkage can be constructed using the techniques described herein. Within a layer, each neuron has one or more inputs. Each input is either an input value from a previous neuron or an input to the layer. Each input has an associated weight. Each neuron defines a neuron function. This function defines how the inputs and weights are combined to generate an output. In any particular layer, the weights are represented by the input tensor W, the inputs are represented by the input tensor I, and the output is represented by the output tensor O. In the notation below, each such tensor has four dimensions, which are:

[0055] For the weight tensor, C is the input channel, where a "channel" is a portion of the input image (such as color). K is the output channel, i.e., the channel the output of the weight goes into. X and Y are the height and width of the weight.

[0056] For input tensors, C is the channel and N is the number of batches, where a batch represents a different convolution, and H and W are the height and width. For output tensors, H^ and W^ are the height and width of the output, N is the number of batches, and K is the number of output channels.

[0057] In the example of Figure 5, a direct convolution operation is performed, in which a weight tensor W is convolved with an input tensor I to produce an output tensor

number

[0058] In FIG. 5, one bar under a dimension index means that the dimension is sliced ​​once, and two bars means that the dimension is sliced ​​twice. A "1" for the number of dimensions means that the dimension is reduced to one index. "Device Memory" refers to the GPU's general-purpose memory (which in one example is the APD 116). "LDS" is local data sharing, accessible by elements of the compute unit 132 and not local to the compute unit 132 that is identical to the LDS. As commonly known, registers are lower-level structures available to the GPU's functional units (e.g., SIMD units 138).

[0059] In the example operation of FIG. 5 , a processing entity, such as a work item or thread, executing on processor 102 performs a block unit slice on a weight tensor W in the K and C dimensions in operation 502 and stores the result in the LDS. A processing entity, such as a work item or thread, executing on processor 102 performs a block unit slice on an input tensor I in the N, C, H, and W dimensions in operation 504. A block unit slice is a slice operation performed by multiple threads executing in parallel, where each such thread performs a portion of the slice operation. In some embodiments, each such thread is a CPU thread executing on processor 102. In other embodiments, each such thread is a work item executing on the APD 116. Both the block unit slice on the weight tensor and the block unit slice on the input tensor are performed by copying from device memory to the LDS, and these operations are considered operations on the data itself. Because slicing by copying involves copying multiple items of data, block-unit slicing divides the copied data among blocks of threads, with each of these threads copying the data designated for that thread to a local data share. A processing entity, such as a work item or processor 102, performs a thread-unit slice in operation 506 on a weight tensor, resulting in a tensor with a single element in the Y and X dimensions and reduced length in the K and C dimensions, and stores the resulting data in a register. A thread-unit slice is a slice in which each thread copies a single element of the tensor being processed. In operation 508, the processing entity performs a thread-unit slice on the indicated input tensor and stores the resulting element in a register. In operation 510, the processing entity performs a thread-unit 1×1 convolution to produce an output tensor as shown. A thread-unit 1×1 convolution is one in which each thread performs a convolution on one element from the weight tensor and one element from the input tensor.In operation 512, if there are more elements in the sub-portion of W[K,C,Y,X] denoted as C,Y,X, then operations 506, 508 and subsequent operations are performed again; otherwise, operation 514 is performed. In operation 514, if there are more elements in the sub-portion of W[K,C,Y,X] denoted as C, then operation 516 is performed. In operation 516, the thread copies the output elements of the output tensor produced by operation 510 to device memory, resulting in the output tensor in device memory.

number

[0060] Note that Figure 5 illustrates a specific example of combining operations on general tensors to achieve a particular result. However, the operations on general tensors described herein can be used in any technically feasible manner to produce any result. As shown in Figure 5, these operations are used in various embodiments to "compose" operations for neural network computations by providing individual operations performed by different execution threads that produce results and combine those results to produce output. Furthermore, the operations enable copying of data by different execution threads to proceed through the memory hierarchy from more general-purpose memory (e.g., device memory) to more specialized memory (e.g., LDS and registers).

[0061] 5A illustrates an exemplary technique for converting a convolution operation into a GEMM operation using general tensor operations, in which a weight tensor W is convolved with an input tensor I to produce an output tensor

number

[0062] In Figure 5A, parallelograms represent general tensors. Rectangles represent operations on general tensor descriptors. The operation takes a general tensor descriptor and produces another general tensor descriptor, but does not operate on any general tensor raw data. One-way arrows indicate that a general tensor descriptor is passed to or produced by an operation on a general tensor descriptor. Ovals represent operations on general tensor raw data. The operation takes raw data from one or more general tensors and produces raw data for one or more general tensors. Thick arrows indicate that general tensor raw data is passed to or produced by an operation on general tensor raw data.

[0063] In the example operation of Figure 2, the C, Y, and X dimensions of the 4D weight tensor W[K,C,Y,X] are combined to generate the E dimension, resulting in a 2D combined weight tensor W[K,E].

[0064] The H dimension of the input tensor I[N,C,H,W] is

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0065] Input Tensor

number

number

number

number

number

number

number

number

number

number

[0066] Output Tensor

number

number

number

number

number

[0067] Note that all the above operations only operate on the general tensor descriptor, not the raw tensor data. All general tensors produced by the operations share the same raw tensor data in memory as the original tensor on which the operation was performed. In other words, there is no data movement or computation in memory for the raw tensor data during these operations.

[0068] The above operation takes a general weight tensor W[K,E] and a general input tensor

number

number

number

[0069] Note that operations on general tensor descriptors can be performed by a compiler at software compile time, by software running at runtime, by dedicated instructions available in hardware, or a combination of these. GEMM operations are implemented in the hardware used to perform the operations. However, any feasible type of hardware may be used.

[0070] 6 is a flow diagram of a method 600 for performing tensor operations, according to an example. Although described with respect to the systems of FIGS. 1-5, one skilled in the art will understand that any system configured to perform the steps of method 600 in any technically feasible order is within the scope of this disclosure.

[0071] In step 602, the tensor operation request unit 302 sends a request to the tensor operation unit 304 to operate on a generic tensor descriptor. As described elsewhere herein, some operations performed on generic tensors modify the generic tensor descriptor rather than the raw data of the generic tensor. The generic tensor descriptor is a component that indicates how to obtain data elements of the generic tensor raw data given an input multi-index. Manipulating the generic tensor descriptor to generate a new generic tensor descriptor allows the generic tensor raw data to be treated as if it were a different tensor without modifying it. Numerous operations on the generic tensor descriptor are provided herein. In step 604, the tensor operation unit 304 performs the requested operation to operate on the generic tensor descriptor.

[0072] In step 606, the tensor operation requester 302 requests the tensor operator 304 to perform an operation on the generic tensor raw data associated with the generic tensor descriptor. A wide variety of operations are described herein. In contrast to operations on generic tensor descriptors, operations on generic tensor raw data modify the contents of the generic tensor raw data or generate new generic tensor raw data, depending on the specific operation requested. In step 608, the tensor operator 304 performs the operation requested in step 606.

[0073] The tensor operation requester 302 that performs the operation of step 602 may be a different tensor operation requester 302 than the tensor operation requester that performs the operation of step 606. In one example, program source code 414 requests that compiler 412 perform at least some tensor descriptor operations, and in response, compiler 412 performs the tensor descriptor operations. When compiled, compiled program 416 includes requests to perform operations on general tensor raw data (e.g., via function calls, API calls, or explicitly via specified instructions) and is thus a tensor operation requester 302; compiled program 416 or other entity that performs operations on general tensor raw data is a tensor operator 304. As noted above, FIGS. 4A-4C illustrate some example systems for performing the operations of FIG. 6.

[0074] It should be understood that many variations are possible based on the disclosure herein, and although features and elements are described above in particular combinations, each feature or element can be used alone without other features and elements, or in various combinations with or without other features and elements.

[0075] The various functional units illustrated in the figures and / or described herein (including, but not limited to, processor 102, input driver 112, input device 108, output driver 114, output device 110, accelerated processing device 116, scheduler 136, graphics processing pipeline 134, computation unit 132, and SIMD unit 138) may, in various embodiments, be implemented as a general-purpose computer, processor, processor core, or as a program, software, or firmware stored in a non-transitory computer-readable storage medium or another medium and executable by the general-purpose computer, processor, or processor core. Provided methods may be implemented in a general-purpose computer, processor, or processor core. Suitable processors include, by way of example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or a state machine. Such a processor may be manufactured by configuring a manufacturing process using results of processing hardware description language (HDL) instructions and other intermediate data (instructions storable on a computer-readable storage medium) including a netlist. The result of such processing may be a maskwork used in a semiconductor manufacturing process to produce a processor that implements aspects of the embodiments.

[0076] The methods or flow charts provided herein may be implemented in a computer program, software, or firmware embodied in a non-transitory computer-readable storage medium for execution by a general purpose computer or processor. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).

Claims

1. 1. A method for a computing system including a memory and a processor for manipulating general tensors, comprising: receiving a first request to perform a first operation on a general tensor descriptor associated with the general tensor; performing the first operation on the generic tensor descriptor in response to the first request, the first operation generating a modified generic tensor descriptor based on the generic tensor descriptor and the first operation without modifying raw generic tensor data; receiving a second request to perform a second operation on the general tensor raw data associated with the general tensor; and performing the second operation on the general tensor raw data in response to the second request. method.

2. The first operation is Any of slice, stride slice, reorder, convolution, join, embedding, and moving slicing window operations.

10. The method of claim 1.

3. The second operation is including any of slicing, reordering, copying, general or batched general matrix multiplication, reduction, and algorithm-specified transformations; 10. The method of claim 1.

4. the general tensor descriptor includes an indication of how to translate one or more index values ​​into one or more memory addresses for the general tensor raw data.

10. The method of claim 1.

5. the general tensor raw data includes data elements of the general tensor, 10. The method of claim 1.

6. the first request, the first operation, the second request, and the second operation are specified by program instructions; 10. The method of claim 1.

7. the first request is specified by a program source; the first operation is performed by a compiler configured to compile the program source to generate a compiled program; the second request is specified by the compiled program; the second operation is performed by the compiled program.

10. The method of claim 1.

8. the first request and the second request are specified by a program; At least one of the first operation and the second operation is performed by a dedicated hardware circuit configured to perform at least one operation on a general tensor descriptor or general tensor raw data.

10. The method of claim 1.

9. 1. A system for manipulating general tensors, comprising: a memory for storing a general tensor descriptor; a processor, The processor: receiving a first request to perform a first operation on the generic tensor descriptor associated with the generic tensor, the first operation generating a modified generic tensor descriptor based on the generic tensor descriptor and the first operation without modifying raw generic tensor data; performing the first operation on the general tensor descriptor in response to the first request; receiving a second request to perform a second operation on the general tensor raw data associated with the general tensor; performing the second operation on the general tensor raw data in response to the second request; configured to: system.

10. The first operation is Any of slice, stride slice, reorder, convolution, join, embedding, and moving slicing window operations. The system of claim 9.

11. The second operation is including any of slicing, reordering, copying, general or batched general matrix multiplication, reduction, and algorithm-specified transformations; The system of claim 9.

12. the general tensor descriptor includes an indication of how to translate one or more index values ​​into one or more memory addresses for the general tensor raw data. The system of claim 9.

13. the general tensor raw data includes data elements of the general tensor, The system of claim 9.

14. the first request, the first operation, the second request, and the second operation are specified by program instructions; The system of claim 9.

15. the first request is specified by a program source; the first operation is performed by a compiler configured to compile the program source to generate a compiled program; the second request is specified by the compiled program; the second operation is performed by the compiled program. The system of claim 9.

16. the first request and the second request are specified by a program; At least one of the first operation and the second operation is performed by a dedicated hardware circuit configured to perform at least one operation on a general tensor descriptor or general tensor raw data. The system of claim 9.

17. A computer-readable storage medium storing instructions, comprising: The instructions, when executed by a processor, receiving a first request to perform a first operation on a generic tensor descriptor associated with a generic tensor, the first operation generating a modified generic tensor descriptor based on the generic tensor descriptor and the first operation without modifying raw generic tensor data; performing the first operation on the general tensor descriptor in response to the first request; receiving a second request to perform a second operation on the general tensor raw data associated with the general tensor; performing the second operation on the general tensor raw data in response to the second request; to operate on the general tensor; A computer-readable storage medium.

18. The first operation is Any of slice, stride slice, reorder, convolution, join, embedding, and moving slicing window operations.

20. The computer-readable storage medium of claim 17.

Images