A unified programming interface for regranulated tile execution.

Abstract

To provide a unified programming interface, a system, a device, and a method for regrained tile execution.SOLUTION: In an arrangement scenario 10, a tensor operation has an unspecified tensor input size, determines the input tensor size at runtime, and selects a partition configuration for the tensor operation based at least in part on the input tensor size and one or more runtime conditions. In one example, this technology searches a lookup table for the input tensor size and at least one of the one or more runtime conditions for selecting the partition configuration.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments relate generally to application programming interfaces (APIs) and, more particularly, to a unified programming interface for regulated tile execution. [Background technology]

[0002] An instruction set architecture (ISA) generally defines the data types, registers, and hardware support supported for processor operations such as data processing, memory operations, arithmetic operations, and control flow operations. Recent developments in artificial intelligence (AI) may have resulted in ISA extensions to more explicitly support neural network training and inference operations. Thus, software developers may customize the code of AI applications to take advantage of new computational capabilities and accelerated execution facilitated by the extended ISA. However, code customization can be time-consuming, costly, and inefficient, especially when applications are deployed across different ISAs and processors. [Brief description of the drawings]

[0003] Various advantages of the embodiments will become apparent to those skilled in the art upon reading the following specification and appended claims, and upon reference to the following drawings.

[0004] [Figure 1] FIG. 2 is a block diagram of an example of an application deployment according to an embodiment.

[0005] [Diagram 2] 1 is a flowchart of an example method of operating a performance enhancing computing system according to an embodiment.

[0006] [Diagram 3]FIG. 2 is a block diagram of an example partitioned configuration, according to an embodiment.

[0007] [Figure 4] FIG. 2 is a diagram of an example partitioning configuration for a matrix multiplication operation, according to an embodiment.

[0008] [Diagram 5] FIG. 1 is a block diagram of an example performance enhancing computing system according to an embodiment.

[0009] [Figure 6] 1 is a diagram of an example of a semiconductor device according to an embodiment;

[0010] [Figure 7] FIG. 2 is a block diagram of an example processor according to an embodiment.

[0011] [Figure 8] FIG. 1 is a block diagram of an example multi-processor based computing system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1, a deployment scenario 10 is shown in which an application developer 12 generates an application 14 for deployment to an execution environment 16 having a computing system 20 (e.g., a back-end platform including one or more processor cores, not shown). The application 14 may include training (e.g., iterative selection of network layer weights) of a neural network (e.g., Convolutional Neural Network / CNN, Deep Neural Network / DNN, etc.) and / or real-time operation of the neural network (e.g., to draw inferences for image recognition, natural language processing / NLP, etc.). In an embodiment, the application 14 is a portable application designed to read and write configuration settings for the application 14 to a folder accessible to the computing system 20.

[0013] In the illustrated example, the application 14 includes one or more common tensor operations 18 (18a-18b, e.g., matrix multiplication operations, convolution operations, normalization operations, normalized linear unit / relu operations, exponential linear unit / elu operations, and / or other complex instruction set computer / CISC operations). In general, tensors can be multidimensional data arrays that facilitate automatic classification of input data by neural networks. The multidimensional nature of tensors generally requires the use of matrix-based mathematical operations, where the size of the matrices (e.g., column and / or row lengths) varies. It may not be possible to specify the input tensor size when the application 14 is created by the application developer 12.

[0014] Accordingly, the illustrated tensor operation 18 has a tensor input size that is not specified when the application 14 is created by the application developer 12. Rather, the computing system 20 may determine the tensor input size 22 (e.g., the length of the input columns and / or rows) of the tensor operation 18 at runtime (e.g., during training and / or inference of the neural network). In an embodiment, the computing system 20 also determines one or more runtime conditions 24 (e.g., expected power consumption, sparsity of matrices, availability of hardware resources, etc.) and selects the partitioning configuration 26 of the tensor operation 18 based on the tensor input size 22.

[0015] In one example, the partitioning configuration 26 defines a first set of matrix shapes (e.g., a combination of columns and widths of “tiles”) for the first tensor operation 18a and a second set of matrix shapes for the second tensor operation 18b. The partitioning configuration 26 may also define a first set of hardware resources (e.g., a pool of compute cores) for the first tensor operation 18a and a second set of hardware resources for the second tensor operation 18b, where the first set of hardware resources and the second set of hardware resources are different types of hardware resources. For example, the first set of hardware resources may be a relatively lightweight (e.g., “light”) pool of compute cores including scalar cores. Meanwhile, the second set of hardware resources may be a relatively heavyweight pool of compute cores. The computing system 20 may use the partitioning configuration 26 to generate output 28 (e.g., optimized code to perform training or inference based on runtime tensor sizes and available compute resources) from the application 14.

[0016] The presented solution is therefore less time-intensive, less expensive and more efficient from the application developer 12 point of view. Indeed, the same application 14 can be much more easily deployed across different ISAs and processors, since optimizations and transformations are not statically tied by the application developer 12 to specific tensor sizes or tensor cores. Furthermore, performance is improved by taking into account runtime conditions 24 when the partitioning configuration 26 is generated. For example, utilizing knowledge about the sparsity of a matrix (e.g., the distribution of zero values ​​in the matrix) may allow selecting a relatively light computation core pool and / or a different floating-point format for the operation.

[0017] 2 illustrates a method 30 for operating a performance-enhanced computing system. The method 30 may be implemented in one or more modules as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), firmware, flash memory, etc., in configurable logic such as a programmable logic array (PLA), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), in fixed-function logic hardware using circuit technologies such as an application specific integrated circuit (ASIC), complementary metal oxide semiconductor (CMOS) technology or transistor-transistor logic (TTL) technology, or any combination thereof.

[0018] For example, the computer program code for carrying out the operations illustrated in method 30 may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, or C++, traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. Furthermore, the logic instructions may include assembler instructions, ISA instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for integrated circuits, state information for personalizing electronic circuits, and / or other structural components specific to hardware (e.g., a host processor, a central processing unit / CPU, a microcontroller, etc.).

[0019] The illustrated processing block 32 detects tensor operations in the application, where the tensor operations have unspecified input tensor sizes. In an embodiment, the tensor operations are general and include matrix multiplication operations (e.g., matmul), convolution operations (e.g., conv2d, conv2d_transpose, conv3d), normalization operations (e.g., l2_normalize), normalized linear unit operations, exponential linear unit operations, etc., or any combination thereof. The tensor operations may be detected by analyzing and / or compiling the application 14 for execution. Block 34 determines the input tensor size at runtime. In an embodiment, the input tensor size is determined by analyzing input data to the neural network (e.g., input images, speech, etc.), analyzing data output from previous layers in the neural network, etc.

[0020] A partitioning configuration is selected for the tensor operations in block 36 based at least in part on the input tensor size and one or more runtime conditions. In one example, the runtime conditions include expected power consumption, matrix sparsity, and / or hardware resource availability. Additionally, the partitioning configuration may define a first set of matrix shapes (e.g., tile size) for the first operation, a second set of matrix shapes for the second operation, etc. In an embodiment, the partitioning configuration further defines a first set of hardware resources for the first operation, a second set of hardware resources for the second operation, etc., where the first set of hardware resources and the second set of hardware resources are different types of resources. Such an approach to tile size and resource selection allows the matrix computation granularity to be changed (e.g., re-granulated) on the fly and in real time.

[0021] Block 36 may include searching the input tensor size and at least one of the runtime conditions in a lookup table. In this regard, since each tensor operation is generally well understood in terms of computation, memory and communication patterns, it may be possible for the offline tuning / benchmarking process to capture near-optimal mappings for different tensor sizes and sets of available resources. Thus, at runtime, based on the detected tensor size and enumeration of available tensor cores, a table lookup may be performed by the runtime engine to obtain the most optimal / near-optimal execution / partitioning plan along with generation of optimized code (e.g., optimal tile size, etc.) in any optimized tensor cores. When the tuning process is performed offline, the runtime scheduling overhead may be minimal.

[0022] Thus, the illustrated method 30 provides a re-granulated tiled execution solution that improves performance by considering runtime conditions when generating partitioning configurations. For example, utilizing knowledge about expected power consumption may allow tensor operations to be mapped to more power-efficient core pools. The illustrated method 30 is also less time-intensive, less expensive, and more efficient from an application developer's perspective. Indeed, the same application may be much easier to deploy across different ISAs and processors, since optimizations tied to specific tensor input sizes are not built into the application by the application developer.

[0023] 3 illustrates a partitioning configuration 40 in which a unified dynamic dispatch 42 (e.g., a “granulator”) receives a computationally granular portable application 44. The dynamic dispatch 42 may be considered “unified” to the extent that the dispatch 42 configures the application 44 to run across a set of heterogeneous hardware resources (e.g., CPUs, graphics processing units / GPUs, FPGAs, dedicated accelerators, etc.) using a unified programming model such as, for example, ONEAPI. In an embodiment, the dispatch 42 includes a pre-compiled plan lookup table 46 that includes benchmark data that facilitates the selection of a partitioning configuration for the application 44 at runtime. Additionally, a set of pre-compiled granular optimization libraries (“libs”) 48 may include, for example, a vector performance library, a 16×16 (e.g., 16 element×16 element) performance library, a 32×32 (e.g., 32 element×32 element) performance library, etc.

[0024] In the illustrated example, the partitioning configuration defines / specifies the use of scalar cores 50 (e.g., selected from a unified light compute core pool), 16wide vector compute lanes 52 (e.g., selected from a unified heavy compute vector core pool), 16x16 tensor cores 54 (e.g., selected from a unified medium compute 2D / 2D tensor core pool), 32x32 tensor cores 56 (e.g., selected from a unified heavy compute GPU / 2D tensor core pool), etc. In an embodiment, dispatch 42 generates customized modules 58 (e.g., vector modules, 16x16 modules, 32x32 modules) at runtime for optimal execution of application 44 on heterogeneous tensor cores, potentially for cooperative execution.

[0025] FIG. 4 illustrates a partitioning configuration for a matrix multiplication (matmul) operation between an activation matrix 60 (e.g., matrix X representing activations of a neural network layer) and a weight matrix 62 (e.g., matrix W representing weights applied to the activations), where Y=XW. Loading the activation matrix 60 may put pressure on the memory bandwidth of a core, especially during training where the batch size is larger. This may be true for the weight matrix 62 as well. Based on the generated offline tuning plan, the runtime engine may decide to partition the XW operation, where X is a 52×32 element matrix and W is a 32×64 element matrix, into three parts.

[0026] Rows 0 to 31 of matrix X are read by a "heavy" 32x32 compute element tensor core, along with matrix W. The tiled matmul operation is performed twice to produce rows 0 to 31 of the output matrix 64.

[0027] Rows 32 through 47 and columns 0 through 31 of matrix X, and the top half of matrix W, are read by the 16×16 tensor core to produce a partial sum of the top half of the output matrix 64.

[0028] At the same time, rows 32 through 47 and columns 32 through 63 of matrix X, as well as the bottom half of matrix W, are read by another 16×16 tensor core to produce a partial sum of the top half of the output matrix 64.

[0029] During the reduction stage between the two cores, pairs of partial sums are added together to produce the final results in rows 32 to 47 of the output matrix 64.

[0030] Four 16wide vector units read the last four rows of matrix X, and each vector unit processes 1 / 4 of the columns of matrix W. Each vector unit produces 1 / 4 of the columns of the output matrix 64, outputting rows 48 through 51.

[0031] 5, a performance-enhancing computing system 151 is shown. The system 151 may generally be part of an electronic device / platform having a computing function (e.g., a personal digital assistant (PDA), a notebook computer, a tablet computer, a convertible tablet, a server), a communication function (e.g., a smartphone), an imaging function (e.g., a camera, a camcorder), a media playback function (e.g., a smart television (TV)), a wearable function (e.g., a watch, glasses, a hat, shoes, jewelry), a vehicle function (e.g., a car, a truck, a bike), a robotic function (e.g., an autonomous robot), or the like, or any combination thereof. In the illustrated example, the system 151 includes a host processor 153 (e.g., a CPU with multiple cores, not shown) having an integrated memory controller (IMC) 155 coupled to a system memory 157.

[0032] The illustrated system 151 further includes an input / output (IO) module 159 implemented along with the host processor 153 on a semiconductor die 163 as a system on chip (SoC), and a graphics processor 161. The illustrated IO module 159 communicates with, for example, a display 165 (e.g., touch screen, liquid crystal display / LCD, light emitting diode / LED display), a network controller 167 (e.g., wired and / or wireless), and mass storage 169 (e.g., hard disk drive / HDD, optical disk, solid state drive / SSD, flash memory).

[0033] In an embodiment, host processor 153, graphics processor 161 and / or IO module 159 execute program instructions 171 retrieved from system memory 157 and / or mass storage 169 to perform one or more aspects of method 30 (FIG. 2) previously described. Execution of illustrated instructions 171 may thus cause computing system 151 to detect tensor operations in an application, where the tensor operations have unspecified tensor input sizes, determine the input tensor sizes at runtime, and select a partitioning configuration for the tensor operations based at least in part on the input tensor sizes and one or more runtime conditions. In an embodiment, the partitioning configuration defines a first set of matrix shapes for a first operation, and a second set of matrix shapes for a second operation. The partitioning configuration may also define a first set of hardware resources for the first operation, and a second set of hardware resources for the second operation, etc., where the first set of hardware resources and the second set of hardware resources are different types of resources. In one example, to select a partitioning configuration, the instructions 171, when executed, cause the computing system 151 to search a lookup table for the input tensor size and at least one of the runtime conditions.

[0034] Thus, the illustrated system 151 is considered performance-enhanced at least to the extent that it provides a re-granulated tiled execution solution that takes into account runtime conditions when generating partition configurations. For example, utilizing knowledge about expected hardware resource availability may allow execution times to be reduced. Also, the illustrated system 151 saves application development time, reduces costs, and improves efficiency. Indeed, the same application may be much easier to deploy across different ISAs and processors, since optimizations tied to specific tensor input sizes or specific tensor cores are not built into the application by the application developer.

[0035] FIG. 6 illustrates a semiconductor packaged device 173. The illustrated device 173 includes one or more substrates 175 (e.g., silicon, sapphire, gallium arsenide) and logic 177 (e.g., transistor arrays and other integrated circuit (IC) components) coupled to the one or more substrates 175. The logic 177 may be implemented, at least in part, in configurable logic or fixed function logic hardware. In one example, the logic 177 implements one or more aspects of the method 30 (FIG. 2) previously described. Thus, the logic 177 may detect a tensor operation in an application, where the tensor operation has an unspecified tensor input size, determine the input tensor size at runtime, and select a partitioning configuration for the tensor operation based at least in part on the input tensor size and one or more runtime conditions. In one example, to select the partitioning configuration, the logic 177 searches a lookup table for the input tensor size and at least one of the runtime conditions.

[0036] Thus, the illustrated apparatus 173 is considered performance enhanced at least to the extent that it provides a re-granulated tiled execution solution that takes into account runtime conditions when generating partition configurations. For example, utilizing knowledge about expected hardware resource availability may allow execution times to be reduced. Also, the illustrated apparatus 173 saves application development time, reduces costs, and improves efficiency. Indeed, the same application may be much easier to deploy across different ISAs and processors since tensor input sizes are not hard-wired into the application by the application developer.

[0037] In one example, logic 177 includes transistor channel regions disposed (e.g., buried) within one or more substrates 175. Thus, the interface between logic 177 and one or more substrates 175 may not be a step bond. Logic 177 may further be considered to include an epitaxial layer grown on an initial wafer of one or more substrates 175.

[0038] FIG. 7 illustrates a processor core 200 according to one embodiment. Processor core 200 may be the core of any type of processor, such as a microprocessor, embedded processor, digital signal processor (DSP), network processor, or other device that executes code. Although only one processor core 200 is shown in FIG. 7, a processing element may alternatively include more than one of the processor cores 200 illustrated in FIG. 7. Processor core 200 may be a single-threaded core, or, for at least one embodiment, processor core 200 may be multi-threaded in that it may include more than one hardware thread context (or "logical processor") per core.

[0039] FIG. 7 also illustrates a memory 270 coupled to the processor core 200. The memory 270 may be any of a wide variety of memories (including various layers of a memory hierarchy) known or otherwise available to those skilled in the art. The memory 270 may include one or more code 213 instructions executed by the processor core 200, where the code 213 may implement one or more aspects of the method 30 (FIG. 2) previously described. The processor core 200 follows a program sequence of instructions represented by the code 213. Each instruction enters the front-end portion 210 and may be processed by one or more decoders 220. The decoders 220 may generate as their output micro-operations, such as fixed-width micro-operations, in a predefined format, or may generate other instructions, micro-instructions, or control signals that reflect the original code instructions. The illustrated front-end portion 210 also includes a register renaming logic 225 and a scheduling logic 230, which generally allocate resources and queue operations corresponding to the conversion instructions for execution.

[0040] Processor core 200 is shown to include execution logic 250 having a set of execution units 255-1 through 255-N. Some embodiments may include multiple execution units dedicated to a particular function or set of functions. Other embodiments may include only one execution unit, or may include one execution unit capable of performing a particular function. The execution logic 250 is shown to perform operations specified by code instructions.

[0041] After completing execution of the operations specified by the code instructions, back-end logic 260 retires the instructions of code 213. In one embodiment, processor core 200 allows out-of-order execution but requires in-order retirement of instructions. Retirement logic 265 may take a variety of forms known to those skilled in the art (e.g., a reorder buffer, etc.). Thus, processor core 200 is transformed during execution of code 213 with respect to at least the outputs generated by the decoders, the hardware registers and tables utilized by register renaming logic 225, and any registers (not shown) modified by execution logic 250.

[0042] 7, a processing element may include other elements on a chip with processor core 200. For example, a processing element may include memory control logic along with processor core 200. A processing element may include I / O control logic and / or I / O control logic integrated with memory control logic. A processing element may also include one or more caches.

[0043] Referring now to Figure 8, there is shown a block diagram of an embodiment of a computing system 1000 according to one embodiment. Shown in Figure 8 is a multiprocessor system 1000 comprising a first processing element 1070 and a second processing element 1080. Although two processing elements 1070 and 1080 are shown, it should be understood that an embodiment of the system 1000 may also include only one such processing element.

[0044] System 1000 is shown as a point-to-point interconnect system, where a first processing element 1070 and a second processing element 1080 are coupled via a point-to-point interconnect 1050. It should be understood that any or all of the interconnects shown in Figure 8 may be implemented as multi-drop buses rather than point-to-point interconnects.

[0045] 8, each of the processing elements 1070 and 1080 may be a multi-core processor including a first processor core and a second processor core (i.e., processor cores 1074a and 1074b, and processor cores 1084a and 1084b). Such cores 1074a, 1074b, 1084a, 1084b may be configured to execute instruction codes in a manner similar to that described above in connection with FIG.

[0046] Each of the processing elements 1070, 1080 may include at least one shared cache 1896a, 1896b. The shared caches 1896a, 1896b may store data (e.g., instructions) utilized by one or more components of the processor, such as the cores 1074a, 1074b and 1084a, 1084b, respectively. For example, the shared caches 1896a, 1896b may locally cache data stored in the memories 1032, 1034 for faster access by the components of the processor. In one or more embodiments, the shared caches 1896a, 1896b may include one or more mid-level caches, such as a level 2 (L2), level 3 (L3), level 4 (L4), or other level cache, a last level cache (LLC), and / or combinations thereof.

[0047] While only two processing elements 1070, 1080 are shown, it should be understood that the scope of the embodiments is not so limited. In other embodiments, one or more additional processing elements may be present in a given processor. Alternatively, one or more of the processing elements 1070, 1080 may be elements other than processors, such as accelerators or field programmable gate arrays. For example, the one or more additional processing elements may include one or more additional processors that are the same as the first processor 1070, one or more additional processors that are heterogeneous or asymmetric processors from the first processor 1070, accelerators (e.g., graphics accelerators or digital signal processing (DSP) units, etc.), field programmable gate arrays, or any other processing elements. There may be various differences between the processing elements 1070 and 1080 with respect to a wide range of value criteria, including architectural characteristics, microarchitectural characteristics, thermal characteristics, power consumption characteristics, and the like. These differences may effectively manifest as asymmetries and heterogeneities between the processing elements 1070, 1080. In at least one embodiment, the various processing elements 1070, 1080 may be in the same die package.

[0048] The first processing element 1070 may further include a memory controller logic (MC) 1072 and point-to-point (PP) interfaces 1076 and 1078. Similarly, the second processing element 1080 may include a MC 1082 and PP interfaces 1086 and 1088. As shown in FIG. 8, the MCs 1072 and 1082 couple the processors to respective memories, i.e., memory 1032 and memory 1034, which may be part of a main memory locally attached to the respective processors. Although the MCs 1072 and 1082 are shown as being integrated within the processing elements 1070 and 1080, in alternative embodiments, the MC logic may be separate logic external to the processing elements 1070, 1080, rather than being integrated therein.

[0049] The first processing element 1070 and the second processing element 1080 may be coupled to the I / O subsystem 1090 via PP interconnects 1076, 1086, respectively. As shown in FIG. 8, the I / O subsystem 1090 has PP interfaces 1094 and 1098. Additionally, the I / O subsystem 1090 has an interface 1092 that couples the I / O subsystem 1090 with a high performance graphics engine 1038. In one embodiment, a bus 1049 may be used to couple the graphics engine 1038 to the I / O subsystem 1090. Alternatively, a point-to-point interconnect may couple these components.

[0050] Similarly, I / O subsystem 1090 may be coupled to a first bus 1016 via an interface 1096. In one embodiment, first bus 1016 may be a bus such as a Peripheral Component Interconnect (PCI) bus, or a PCI Express bus or another third generation I / O interconnect bus, although the scope of the embodiment is not so limited.

[0051] As shown in FIG. 8, various I / O devices 1014 (e.g., biometric scanner, speaker, camera, sensor) may be coupled to the first bus 1016 along with a bus bridge 1018. The bus bridge 1018 may couple the first bus 1016 to a second bus 1020. In one embodiment, the second bus 1020 may be a low pin count (LPC) bus. In one embodiment, various devices may be coupled to the second bus 1020 including, for example, a keyboard / mouse 1012, one or more communication devices 1026, and a data storage unit 1019 such as a disk drive or other mass storage device that may include code 1030. The illustrated code 1030 may implement one or more aspects of the method 30 (FIG. 2) previously described. Additionally, an audio I / O 1024 may be coupled to the second bus 1020, and a battery 1010 may provide power to the computing system 1000.

[0052] It should be noted that other embodiments are contemplated. For example, instead of the point-to-point architecture of FIG. 8, the system may implement a multi-drop bus, or another such communication topology. Also, the elements of FIG. 8 may alternatively be split using more or fewer integrated chips than those shown in FIG. 8.

[0053] Additional Notes and Examples

[0054] Example 1 includes a performance-enhanced computing system including a network controller, a processor coupled to the network controller, and a memory coupled to the processor, where the memory includes a set of executable program instructions that, when executed by the processor, cause the computing system to detect tensor operations in an application, where the tensor application has an unspecified input tensor size, determine the input tensor size at runtime, and select a partitioning configuration for the tensor operations based at least in part on the input tensor size and one or more runtime conditions.

[0055] Example 2 includes the computing system of example 1, where the partitioning configuration defines a first set of matrix shapes for the first operation and a second set of matrix shapes for the second operation.

[0056] Example 3 includes the computing system of example 2, where the partitioning configuration further defines a first set of hardware resources for a first operation and a second set of hardware resources for a second operation, where the first set of hardware resources and the second set of hardware resources are different types of resources.

[0057] Example 4 includes the computing system of example 1, where the one or more runtime conditions include one or more of expected power consumption, sparsity of matrices, or availability of hardware resources.

[0058] Example 5 includes the computing system of example 1, where to select the partitioning configuration, the instructions, when executed, cause the computing system to search a lookup table for the input tensor size and at least one of the one or more runtime conditions.

[0059] Example 6 includes the computing system of any one of examples 1 to 5, where the tensor operation includes one or more of a matrix multiplication operation, a convolution operation, a normalization operation, a normalized linear unit operation, or an exponential linear unit operation.

[0060] Example 7 includes a semiconductor device including one or more substrates, and logic coupled to the one or more substrates, where the logic is implemented at least in part with one or more configurable logic or fixed function hardware logic, the logic coupled to the one or more substrates to detect a tensor operation in an application, where the tensor operation has an unspecified input tensor size, determine the input tensor size at runtime, and select a partitioning configuration for the tensor operation based at least in part on the input tensor size and one or more runtime conditions.

[0061] Example 8 includes the semiconductor device of example 7, where the partitioning configuration defines a first set of row and column configurations for a first operation and a second set of row and column configurations for a second operation.

[0062] Example 9 includes the semiconductor device of example 8, where the partitioning configuration further defines a first set of hardware resources for a first operation and a second set of hardware resources for a second operation, where the first set of hardware resources and the second set of hardware resources are different types of resources.

[0063] Example 10 includes the semiconductor device of example 7, where the one or more run-time conditions include one or more of expected power consumption, sparsity of matrices, or hardware resource availability.

[0064] Example 11 includes the semiconductor device of example 7, where to select the partitioning configuration, logic coupled to the one or more substrates searches a lookup table for the input tensor size and at least one of the one or more runtime conditions.

[0065] Example 12 includes the semiconductor device of any one of examples 7 to 11, where the tensor operation includes one or more of a matrix multiplication operation, a convolution operation, a normalization operation, a normalized linear unit operation, or an exponential linear unit operation.

[0066] Example 13 comprises at least one computer-readable storage medium including a set of executable program instructions that, when executed by a computing system, cause the computing system to detect a tensor operation in an application, where the tensor operation has an unspecified input tensor size, determine the input tensor size at runtime, and select a partitioning configuration for the tensor operation based at least in part on the input tensor size and one or more runtime conditions.

[0067] Example 14 includes at least one computer-readable storage medium of example 13, where the partitioning configuration defines a first set of matrix shapes for the first operation and a second set of matrix shapes for the second operation.

[0068] Example 15 includes at least one computer-readable storage medium of Example 14, where the partitioning configuration further defines a first set of hardware resources for the first operation and a second set of hardware resources for the second operation, where the first set of hardware resources and the second set of hardware resources are different types of resources.

[0069] Example 16 includes at least one computer-readable storage medium of Example 13, where the one or more runtime conditions include one or more of expected power consumption, sparsity of matrices, or availability of hardware resources.

[0070] Example 17 includes at least one computer-readable storage medium of Example 13, where to select the partitioning configuration, the instructions, when executed, cause the computing system to search a lookup table for the input tensor size and at least one of the one or more runtime conditions.

[0071] Example 18 includes at least one computer-readable storage medium of any one of Examples 13 to 17, where the tensor operation includes one or more of a matrix multiplication operation, a convolution operation, a normalization operation, a normalized linear unit operation, or an exponential linear unit operation.

[0072] Example 19 includes a method of operating a performance enhancing computing system, the method including detecting a tensor operation in an application, where the tensor operation has an unspecified input tensor size, determining the input tensor size at runtime, and selecting a partitioning configuration for the tensor operation based at least in part on the input tensor size and one or more runtime conditions.

[0073] Example 20 includes the method of example 19, where the partitioning configuration defines a first set of matrix shapes for the first operation and a second set of matrix shapes for the second operation.

[0074] Example 21 includes the method of example 20, where the partitioning configuration further defines a first set of hardware resources for the first operation and a second set of hardware resources for the second operation, where the first set of hardware resources and the second set of hardware resources are different types of resources.

[0075] Example 22 includes the method of example 19, where the one or more runtime conditions include one or more of expected power consumption, sparsity of matrices, or availability of hardware resources.

[0076] Example 23 includes the method of example 19, where selecting the partitioning configuration includes searching a lookup table for the input tensor size and at least one of the one or more runtime conditions.

[0077] Example 24 includes the method of any one of examples 19 to 23, where the tensor operation includes one or more of a matrix multiplication operation, a convolution operation, a normalization operation, a normalized linear unit operation, or an exponential linear unit operation.

[0078] Example 25 includes a means for carrying out the method of any one of Examples 19 to 24.

[0079] Thus, the techniques described herein provide a unified programming interface that targets generic tensor operations and a run-time procedure that re-granulates the computational architecture based on available computational resources. As a result, applications are appropriately mapped to available tensor cores and written only once.

[0080] The embodiments are applicable for use with all types of semiconductor integrated circuit ("IC") chips. Examples of these IC chips include, but are not limited to, processors, controllers, chipset components, programmable logic arrays (PLAs), memory chips, network chips, systems on chips (SoCs), SSD / NAND controller ASICs, and the like. Additionally, in some of the figures, signal lines are represented as lines. Some of these may be different to indicate more component signal paths, may have numerical labels to indicate multiple component signal paths, and / or may have arrows at one or more end points to indicate the primary direction of information flow. However, this should not be construed as limiting. Rather, such additional details may be used in conjunction with one or more exemplary embodiments to make the circuit easier to understand. Any signal lines shown, whether or not they have additional information, may actually include one or more signals that may have multiple directions of travel, and may be implemented with any suitable type of signaling, such as, for example, digital or analog lines implemented in differential pairs, fiber optic lines, and / or single-ended lines.

[0081] Although example sizes / models / values / ranges may be given, the embodiments are not limited thereto. It is expected that as manufacturing techniques (e.g., photolithography) mature over time, smaller devices may be manufactured. In addition, for simplicity of illustration and explanation, and so as not to obscure certain aspects of the embodiments, well-known power / ground connections to IC chips and other components may or may not be shown in the figures. Furthermore, to avoid obscuring the embodiments, configurations may be shown in block diagram form, and in view of the fact that the details regarding the implementation of such block diagram configurations are highly dependent on the computing system in which the embodiments are to be implemented, i.e., such details are within the knowledge of those skilled in the art. Where specific details (e.g., circuits) are described to explain exemplary embodiments, it is clear to one skilled in the art that the embodiments may be practiced without these specific details or with variations of these specific details. Thus, the detailed description should be regarded as illustrative and not limiting.

[0082] The term "coupled" may be used herein to refer to any type of direct or indirect relationship between components of interest and may apply to electrical, mechanical, fluid, optical, electromagnetic, electromechanical, or other connections. In addition, terms such as "first," "second," etc. may be used herein for ease of description only and do not imply any particular temporal or chronological meaning unless otherwise stated.

[0083] As used in this application and the claims, a list of items joined by the term "one or more of" may mean any combination of the listed terms. For example, the phrase "one or more of A, B, or C" may mean A; B; C; A and B; A and C; B and C; or A, B, and C.

[0084] Those skilled in the art will appreciate from the foregoing description that the broad techniques of the embodiments can be implemented in a variety of forms. Thus, although the embodiments have been described with reference to specific examples thereof, the true scope of the embodiments should not be so limited, as other modifications will become apparent to those skilled in the art upon review of the drawings, the specification, and the following claims. [Other possible items] (Item 1) A network controller; a processor coupled to the network controller; a memory coupled to the processor; A computing system comprising: The memory, when executed by the processor, provides the computing system with a set of executable program instructions for detecting tensor operations in an application; The above tensor operations are has an unspecified input tensor size, Determine the size of the input tensors at runtime. selecting a partitioning configuration for the tensor operation based at least in part on the input tensor size and one or more runtime conditions; Calculation system. (Item 2) 2. The computing system of claim 1, wherein the partitioning configuration defines a first set of matrix shapes for a first operation and a second set of matrix shapes for a second operation. (Item 3) 3. The computing system of claim 2, wherein the partitioning configuration further defines a first set of hardware resources for the first operation and a second set of hardware resources for the second operation, where the first set of hardware resources and the second set of hardware resources are different types of resources. (Item 4) 2. The computing system of claim 1, wherein the one or more runtime conditions include one or more of expected power consumption, matrix sparsity, or hardware resource availability. (Item 5) 2. The computing system of claim 1, wherein, to select the partitioning configuration, the instructions, when executed, cause the computing system to search a lookup table for the input tensor size and at least one of the one or more runtime conditions. (Item 6) 2. The computing system of claim 1, wherein the tensor operations include one or more of a matrix multiplication operation, a convolution operation, a normalization operation, a normalized linear unit operation, or an exponential linear unit operation. (Item 7) one or more substrates; A semiconductor device comprising logic coupled to the one or more substrates, the logic being at least partially implemented in one or more configurable logic or fixed function hardware logic; the logic is coupled to the one or more substrates; Detecting a tensor operation in the application, the tensor operation having an unspecified input tensor size; Determine the size of the input tensors at runtime. selecting a partitioning configuration for the tensor operation based at least in part on the input tensor size and one or more runtime conditions; Semiconductor device. (Item 8) 8. The semiconductor device of item 7, wherein the division configuration defines a first set of matrix shapes for a first operation and a second set of matrix shapes for a second operation. (Item 9) The division configuration further defines a first set of hardware resources for the first operation and a second set of hardware resources for the second operation, the first set of hardware resources and the second set of hardware resources being different types of resources. (Item 10) 8. The semiconductor device of item 7, wherein the one or more runtime conditions include one or more of expected power consumption, matrix sparsity, or hardware resource availability. (Item 11) 8. The semiconductor device of claim 7, wherein to select the partitioning configuration, the logic coupled to the one or more substrates searches a lookup table for the input tensor size and at least one of the one or more runtime conditions. (Item 12) 8. The semiconductor device according to item 7, wherein the tensor operation includes one or more of a matrix multiplication operation, a convolution operation, a normalization operation, a normalized linear unit operation, or an exponential linear unit operation. (Item 13) At least one computer-readable storage medium comprising a set of executable program instructions, the set of executable program instructions, when executed by a computing system, to cause the computing system to: Detecting a tensor operation in an application, the tensor operation having an unspecified input tensor size; determining a size of the input tensor at runtime; and selecting a partitioning configuration for the tensor operation based at least in part on the input tensor size and one or more runtime conditions; At least one computer-readable storage medium that causes the (Item 14) Item 14. At least one computer-readable storage medium according to item 13, wherein the partitioning configuration defines a first set of matrix shapes for a first operation and a second set of matrix shapes for a second operation. (Item 15) Item 15. The at least one computer-readable storage medium according to item 14, wherein the partitioning configuration further defines a first set of hardware resources for the first operation and a second set of hardware resources for the second operation, the first set of hardware resources and the second set of hardware resources being different types of resources. (Item 16) Item 14. At least one computer-readable storage medium according to item 13, wherein the one or more runtime conditions include one or more of expected power consumption, sparsity of matrices, or availability of hardware resources. (Item 17) Item 14. The at least one computer-readable storage medium of item 13, wherein the instructions, when executed, cause the computing system to search a lookup table for the input tensor size and at least one of the one or more runtime conditions to select the partitioning configuration. (Item 18) Item 14. At least one computer-readable storage medium according to item 13, wherein the tensor operation includes one or more of a matrix multiplication operation, a convolution operation, a normalization operation, a normalized linear unit operation, or an exponential linear unit operation. (Item 19) detecting a tensor operation in the application, the tensor operation having an unspecified input tensor size; determining the input tensor size at runtime; selecting a partitioning configuration for the tensor operation based at least in part on the input tensor size and one or more runtime conditions. (Item 20) 20. The method of claim 19, wherein the partitioning configuration defines a first set of matrix shapes for a first operation and a second set of matrix shapes for a second operation. (Item 21) 21. The method of claim 20, wherein the partitioning configuration further defines a first set of hardware resources for the first operation and a second set of hardware resources for the second operation, the first set of hardware resources and the second set of hardware resources being different types of resources. (Item 22) 20. The method of claim 19, wherein the one or more runtime conditions include one or more of expected power consumption, matrix sparsity, or hardware resource availability. (Item 23) 20. The method of claim 19, wherein selecting the partitioning configuration comprises searching a lookup table for the input tensor size and at least one of the one or more runtime conditions. (Item 24) 20. The method of claim 19, wherein the tensor operation includes one or more of a matrix multiplication operation, a convolution operation, a normalization operation, a normalized linear unit operation, or an exponential linear unit operation.

Claims

1. A method for generating a deep neural network comprising the steps of: obtaining information associated with tensor operations in a deep neural network; determining a size of an input tensor of the tensor operation based on the information, the input tensor having one or more dimensions, the size of the input tensor comprising a length of at least one of the one or more dimensions; and dividing the tensor operation based on the size of the input tensors and one or more hardware resources for performing the tensor operation. method.

2. The step of dividing the tensor operations comprises: partitioning the input tensor into a plurality of data tiles, each data tile comprising a portion of the input tensor; The method of claim 1.

3. The step of dividing the input tensor into the plurality of data tiles, comprising: determining one or more shapes of the plurality of data tiles; The method of claim 2.

4. The step of dividing the tensor operations comprises: and dividing a weight tensor of the tensor operation into a plurality of data tiles. The method of claim 2.

5. The one or more hardware resources include: a processing unit for performing the tensor operations; The method of claim 1.

6. The one or more hardware resources include: a memory for storing the input tensors or output tensors of the tensor operation; The method of claim 1.

7. The method of claim 1, wherein the tensor operations comprise matrix multiplication operations. The method of claim 1.

8. One or more machine-readable media for storing instructions, The instructions, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 7. Machine-readable medium.

9. An apparatus comprising means for performing a method according to any one of claims 1 to 7.

Images