Fusion of multilayer computation graphs

Abstract

A compilation system for compiling multilayer graphs that improves the optimization and extensibility of computation graphs used in machine learning systems. The system receives a multilayer computation graph containing a modularized computation graph that provides type system and device-independent rewrites. The system generates a modularized computation generation graph using a set of system kernels and a set of user-provided kernels, and performs the fusion of one or more operations to generate an optimized modularized computation generation graph with one or more fused operations. The system generates executable objects using the optimized modularized computation generation graph. By using a multilayer computation graph representation, the system provides improved integration with the compilation system and improves the user extensibility of the compilation process. Furthermore, the system treats user-provided kernels and operations as first-class objects and performs the same optimization processing as other first-class objects in the compilation system.

Description

Technical Field

[0001] This disclosure generally relates to compilers, and more particularly to compilers for high-computation-load code.

Background Art

[0002] Compilers are used to generate object code from high-level languages. It is desirable to generate optimized object code.

Brief Description of the Drawings

[0003] To easily identify the description of any particular element or operation, one or more of the leading digits of the reference number refer to the figure number in which the element was first introduced. [Figure 1] Data flow diagrams of kernel generation and usage processes, according to some examples. [Figure 2] Diagrams showing the collaboration of a compilation system, according to some examples. [Figure 3] Diagrams showing the activities of a kernel generation method, according to some examples. [Figure 4A] Diagrams of graphs used in a graph compiler, according to some examples. [Figure 4B] Diagrams showing the operations of a graph in a graph compiler where an operation includes input ports and output ports, according to some examples. [Figure 5A] Collaboration diagrams of a multi-layer graph compiler pipeline, according to some examples. [Figure 5B] Process flow diagrams of a compilation method, according to some examples. [Figure 6A] Diagrams showing a machine learning pipeline, according to some examples. [Figure 6B] Diagrams showing the training and use of a machine learning program, according to some examples. [Figure 7]This figure shows the deployment of a networked compilation environment with several examples. [Figure 8] This figure shows a machine architecture in which a set of instructions can be executed to cause the machine to perform any one or more of the methods described herein, as illustrated by several examples. [Modes for carrying out the invention]

[0004] Fusion is the process of taking a set of loop operations and folding them into a smaller set of operations. An example of fusion of two loops is shown below.

[0005]

number

[0006] Two loops can be merged into a single loop.

[0007]

number

[0008] If not used for any other purpose besides B, the B vector can be omitted as follows.

[0009]

number

[0010] In machine learning (ML) models, the model is composed of layers, and each layer corresponds to a series of for loops, resulting in fusion. This model, for example, includes two layers, op1 and op2.

[0011]

number

[0012] Two operators can be fused into a single operator.

[0013]

Number

[0014] When a neural network (NN) is trained, the NN often consists of many individual layers that are executed sequentially during inference. However, sequentially executing multiple individual layers can be computationally expensive and inefficient. Fusion helps optimize the execution of these layers by combining them into a single operation or layer that can execute more efficiently.

[0015] For example, if a neural network includes a convolutional layer followed by a batch normalization layer and a ReLU activation layer, these three layers can be fused into a single layer that executes all three operations in a single step. This can significantly reduce the overhead when executing each individual layer separately, enabling faster and more efficient inference.

[0016] ML systems often use models that can be represented as a series of computational graphs. These graphs can be optimized using methods such as eliminating redundant computations, performing device-specific conversions, and fusing kernels.

[0017] Fusion is a technique used in the compilation of ML models to optimize their performance on different hardware platforms (such as CPUs, GPUs, and dedicated accelerators).

[0018] The graph optimizer can set optimization conditions based on the device or kernel library specified when the graph is initialized, using the intermediate representation of a single layer. However, this has the drawback of increasing complexity in both the implementation of these paths and the expansion of the system.

[0019] In some examples, the methods described herein enable optimized execution by merging multiple operations into a single operation. This technique minimizes the overhead associated with performing multiple separate operations, resulting in improved performance, particularly in computationally intensive processes such as neural network inference.

[0020] In some examples, the compilation systems described herein are designed to be flexible and extensible, allowing user-provided kernels to be treated as first-class objects in the compilation process. This feature enables the integration of custom operations and optimizations without compromising the system's ability to effectively perform standard optimizations.

[0021] In some cases, compilation systems optimize performance across various hardware platforms by employing device-specific translations and kernel fusion. Compilation systems support diverse data types and operations, which enhances their usefulness across different deployment environments.

[0022] In some cases, advanced kernel generation processes are employed, using sophisticated parameterization and optimization techniques to determine the optimal settings for each kernel. This results in generated kernels that are highly optimized for performance.

[0023] In some cases, artificial intelligence (AI) components are introduced to enhance the optimization of the compilation process. This AI assistance helps in a more efficient search for the optimal kernel configuration, resulting in better optimization results compared to conventional methods.

[0024] In some cases, the system collects and utilizes execution metrics to continuously improve the compilation and execution processes. This adaptive approach ensures that the system learns from real-world applications and delivers the best possible performance.

[0025] In some examples, the compilation system provided by this disclosure creates a general-purpose kernel generator having a prologue function for loading input data and an epilogue function for writing data to output using a general-purpose user-provided function.

[0026] In some examples, the compilation system provided by this disclosure creates a general-purpose kernel generator having a prologue function for loading input data and an epilogue function for writing data to output using a general-purpose user-provided function.

[0027] In some examples, the compilation system receives a multilayer computation graph containing a modular operation graph, and the modular generation graph contains a set of modular operations. The compilation system uses the modular operation graph and one or more sets of kernels to generate a modular operation generator graph. The compilation system generates an optimized modular operation generator graph by fusing one or more of the operations of the modular operation generator graph, and uses the optimized modular operation generator graph to generate an executable object.

[0028] In some cases, the compilation system uses a set of modularized operation definitions in a modularized operation graph to select a kernel from a set of system kernels and a set of user-provided kernels.

[0029] In some cases, one kernel from a set of user-provided kernels is treated as a first-class object during the compilation process. In some cases, the compilation system uses metadata from a set of user-provided kernels to select a kernel.

[0030] In some examples, operations in a modularized operation generation graph include one or more prologue functions that define the loading of input data into the operation. In some examples, operations in a modularized operation generation graph include one or more epilogue functions that define the writing of the operation's output data.

[0031] Other technical features can be readily understood by those skilled in the art from the following drawings, description, and claims. Compilation system Figure 1 is a data flow diagram of the compilation and execution process 100, with several examples. The user creates a kernel definition 140 in the creation phase 120. The kernel definition 140 is used in the compilation phase 122 to construct a primitive-level representation of the kernel 112 by the compilation system 200a, which is described in more detail with reference to Figure 2, and in more detail with reference to Figure 3.

[0032] The kernel definition 140 consists of parameterization information 136 and one or more generators 132. The generators 132 comprise coding logic that defines one or more operations to be executed by the kernel according to the kernel definition 140, as shown by the code representation 130 of generator 1 and the code representation 134 of generator N. These operations can be organized into coding logic components that operate on one or more data buffers, including, but are not limited to, operations, operators, functions, objects, routines, subroutines, and modules. In some examples, one or more generators 132 include definitions of one or more data structures containing data buffers.

[0033] Parameterization information 136 includes a set of parameters that guide a compiler, such as the graph compiler 204 or kernel compiler 206 (in Figure 2), on how to generate a primitive-level representation of the kernel 112 using multiple generators 132 when compiling the kernel definition 140. In some examples, the generators 132 are written in a general-purpose programming language such as Python. In some examples, parameterization information 136 is a set of kernel parameters. In some examples, parameterization information 136 consists of a parsable scripting language or the like for generating the kernel parameters.

[0034] In compilation phase 122, the code for generator 132 (e.g., code representation 130 for generator 1 and code representation 134 for generator N) is lowered from a general-purpose programming language to lower-level representations of the generator (e.g., primitive-level representation 114 for generator 1 and primitive-level representation 116 for generator N) through a series of intermediate representations, such as the intermediate representation 110 for generator 1 and the intermediate representation 118 for generator N. In compilation phase 122, the optimal settings for generator 132 are determined in a search (e.g., search 1(106) and search N(108)) using parameterization information 136 and the intermediate representations of generator 132, in a process described in more detail with reference to Figure 3. The search results and other compilation metrics are stored in cache 128 for use in subsequent searches. The exploration may include a step of exploring a tree of possible configurations using a combination of static analysis of previous configurations and dynamic analysis of proposed configurations, which are performed during the refinement process, as will be explained in more detail with reference to Figure 3.

[0035] The primitive-level buffer semantic representation of generator 132 is combined with the primitive-level representation of kernel 112. The primitive-level representation of kernel 112 is stored for later use.

[0036] To generate an executable object, a kernel object 138 is created using the primitive-level representation of kernel 112, and this kernel object 138 is contained within the executable object 126. After being created, the executable object 126 is executed in execution phase 124, performing one or more computations using kernel object 138. In some examples, the kernel object 138 is extended with additional executable objects using library 202.

[0037] In some examples, kernel execution metrics are stored in the execution metrics 142 datastore. The execution metrics 142 are then used by a subsequent search component 102 to determine the optimal settings for one or more generators 132.

[0038] In some examples, an artificial intelligence (AI) component 104 is used to assist with exploration. In some examples, the AI ​​component 104 also assists in the creation phase 120, where the kernel is written within the software development environment (SDE) of an integrated development environment (IDE). The training of the AI ​​component 104 is described in more detail with reference to Figures 6A and 6B.

[0039] In some examples, one or more operations are defined by Generator 132 as a fusion of several other lower-level operations, such as broadcast operations and activation operations, and in some cases as an even larger fused amalgam, such as a Long Short-Term Memory (LSTM) operation. Describing generators at this level of abstraction simplifies, but is not limited to, higher-level optimizations such as shape operator extraction and operator gradient generation.

[0040] In some examples, one or more generators 132 are used to generate implementations of existing functions or operators in existing machine learning (ML) frameworks (such as TFLite, TF, ONNX, and PyTorch). Operators in existing ML frameworks have attributes such as, but are not limited to, broadcast and type promotion support, expert-selected handwritten operators known to be important for a particular class of model, activation operators fused into element-wise generators such as "add", support for architecture-specific DSP operations, and support for layouts, dynamic shapes, and dynamic dtypes assumed by the existing framework.

[0041] In some examples, the generator supports the following: • Dynamic shapes • Broadcasting, type promotion: For example, "mul" is a binary generator, and its two operands can have different shapes and dtypes. ML frameworks often improve usability by providing implicit promotion to common element types and support broadcasting of elements.

[0042] • Layout changes (munging): Some frameworks support multiple different layouts, such as row-major and column-major, tiled layouts, etc. Conversion may be necessary if the input is in a different format. Some libraries provide a common implementation that can work with many different layouts using strides, but strides are not common for tiled layouts.

[0043] • Type dispatch: Standard kernel libraries support multiple dtypes, but they are only known dynamically at kernel call time. This requires the kernel to dynamically dispatch for dtypes and dispatch to kernels specialized for many different dtypes. Some dtypes may have special cases; for example, "complex add" can be handled by the same code path as "scalar add" (since complex addition is element-wise), but "complex mul" is an entirely different algorithm from "scalar mul".

[0044] • Thread tiling: At a level outside of type-specific kernel algorithms, computations are divided into blocks that can be executed in parallel by multiple threads. The size of each subunit must be determined and is generally best evaluated based on hardware characteristics and the size of the input data (not on #available threads).

[0045] • Cache tiling: In per-thread computations, calculations are typically cached and blocked at, for example, the L2 level. The size of the L2 level is target-specific. This is important for algorithms that perform multiple passes through data, and may be less important for element-wise operations with little reuse.

[0046] • Tile-based algorithms: Within each L2 tile, there are many ways to implement the core algorithm, including the use of scalars, vectors, and prefetching. Also, when broadcasts are handled internally within the kernel, there are special cases that are interesting to handle, for example, when the fastest changing dimension of one operand is broadcast.

[0047] • Many microkernels: Algorithms like matrix multiplication rely on lower-level operations such as memset for clearing buffers, panel dot products, and reductions. These "microkernels" themselves can be implemented in many different ways.

[0048] • Macro-algorithms: Many generators have multiple completely different algorithms for computing the result; for example, in convolution, we see the im2col approach, direct convolution, and Winograd. Matmul has many implementations (especially when quantization and accelerators force special data layouts), including Strassen's algorithm, among others.

[0049] Hardware targets now frequently have spatial operations (such as Apple AMX or Intel AMX) that can accelerate multiple loop nests at once, for example, matrix multiplication and large element-unit blocks. They also have many architectural families that are register-blocked, pipelining, and require unrolling (deployment) in different ways.

[0050] In some examples, the primitive-level representation of kernel 112 is a component of a framework consisting of a set of code-generating kernels that operate on memory buffers, including, but not limited to, memory operators, 1D memory arrays, tensor buffers, and user-defined data structures. In some examples, the kernels directly use C / C++, assembly, and intrinsic for specific hardware features.

[0051] In some cases, a library of kernel components is used to generate additional kernels. For example, buffer-level operators are used to replace legacy kernels. Kernel components are modular and reusable, and include, but are not limited to, core algorithms such as memory fill, reduction, and element-wise operators, in addition to quantization kernels and more specialized primitives used in other domains.

[0052] In some cases, the generator is a parametric generator. It is difficult for humans to manually create and maintain all permutations of kernels (e.g., for all dtypes, all target machines, etc.), and therefore, metaprogramming is widely used. This metaprogramming takes various forms, such as C macros and ifdefs, Python generator frameworks, and "emitters" written in C++ for the "IRBuilder" compiler API, but the most widely used is C++ templates.

[0053] In some examples, a kernel is defined as a declarative generator that takes kernel parameters and has arbitrary imperative logic coded for those parameters, and that imperative logic is "burned into" the code generated for the kernel. This can be used to specialize in things like dtype, unroll coefficient, vector length, cache size, etc. Most parameters should have an integer type, be constrained by a range (e.g., unroll <= 8 times), a list of valid values ​​(e.g., vector length = 2, 4, 8, 16, 32), and support an enum (enumeration type) to make them searchable (e.g., consider dtype). Using generators still allows the use of concrete kernels (e.g., fixed blobs of assembly) because they are valid generators that have no parameters (or, similarly, have fully constrained parameters).

[0054] An example of the code is shown below. A kernel may have parameters bound to its invocation site; for example, after a dynamic switch on dtype, the next lower-level microkernel is called with a dtype parameter bound to a constant value.

[0055]

number

[0056] Figure 2 is a block diagram of the compilation system 200a, with some examples. The compilation system 200a generates software objects such as a kernel using a kernel definition 140. The kernel definition 140 includes parameterization information 136 and one or more generators 132. The compilation system 200a uses the generated kernel 208 and a handwritten kernel 210 to generate an executable object 216 in a binary executable format (BEF) BEF218 using the generated kernel 208 and the handwritten kernel 210. The compilation system 200a uses a kernel generation method 300 to generate a kernel using the kernel definition 140.

[0057] In some examples, an executable object 216 in a binary executable format 218 containing one or more kernel objects 138 is executed during runtime 212 on a group of hardware devices 214 to generate execution metrics 142 used to optimize the kernel by optimizing the generator settings.

[0058] While various examples of the processing of the compilation system 200a are described with reference to the generation of processing, kernels, and models in computational processes for artificial intelligence (AI) applications, it should be understood that the compilation system 200a and methods and systems of this disclosure may be applied to the compilation and generation of programs in an interpretable or executable format for all kinds of computational processes, including, but not limited to, database processes, real-time processes, networking processes, and client-server processes.

[0059] Figure 3 shows the activity of a kernel generation method in several examples. The compilation system 200a compiles or generates a kernel using the kernel generation method 300. The kernel generation method 300 shows a specific sequence of operations, but this sequence can be changed without departing from the scope of this disclosure. For example, some of the operations shown may be performed in parallel, in different orders, or by different components of a composable kernel compilation system, and such implementations do not substantially affect the generator of the detailing process. In other examples, different components of the exemplary device or implementation of the compilation system 200a may perform operations substantially simultaneously or in a specific order.

[0060] In process 302, the compilation system 200a receives parameterization information 136 and a kernel definition 140 which includes one or more generators 132 containing coding logic that defines the kernel. In some examples, the code includes generator code written in a general-purpose programming language.

[0061] In processes 304 and 306, the compilation system 200a determines the optimal settings for each generator using parameterization information 136. For example, the compilation system 200a searches for the optimal settings for a generator using an evaluator associated with the generator. The kernel compiler 206 can perform static and dynamic analysis searches for the optimal settings of a generator. In a static analysis search, the kernel compiler 206 uses a search component 102 to search several different types of datastores. One type of datastore is a cache 128 containing optimal generator settings that the kernel compiler 206 can reuse to determine the optimal settings of a generator when downgrading the generator during the compilation process. The cache 128 can be a local cache or a distributed cache distributed across remote storage nodes on one or more servers. For example, the compilation system 200a maintains a datastore of optimal settings within the cache 128. The search component 102 searches for the optimal settings for the generator using an evaluator, which is an indicator used by the search component 102 to determine if the generator settings are optimal.

[0062] In response to determining that an optimal generator configuration could not be found during static analysis search, the kernel compiler 206 performs a search using dynamic analysis of the generator. To do this, the kernel compiler 206 generates a set of configurations. For example, the compilation system 200a generates an intermediate representation of the generator. Using the intermediate representation of the generator and parameterization information 136, the compilation system 200a generates one or more configurations of the generator as one or more test intermediate representations of the generator.

[0063] The compilation system 200a uses one or more test intermediate representations to generate a set of executable test functions. For example, for each test intermediate representation, the compilation system 200a reduces the test intermediate representation to an executable object in the BEF to generate an executable test function.

[0064] The compilation system 200a executes a set of test functions to determine a set of individual performance scores. For example, a composable kernel compilation system executes each test function and monitors the performance of the test function as it operates on the data in the test suite. In some examples, the performance score includes an initialization score, indicating the amount of time the test function spent during its initialization. In some examples, the performance score includes an execution score, indicating the amount of time the test function spent processing the test dataset. In some examples, the performance score includes the amount of time the test function spent communicating with other generators in the kernel during execution.

[0065] The compilation system 200a uses a set of individual performance scores to select the optimal setting from a set of settings. For example, the kernel compiler 206 assigns weights to each set of generators, settings, and performance data. During selection, the compilation system 200a uses the set of functions, settings, and performance evaluation data, and their associated weights, to select the generator settings.

[0066] The compilation system 200a generates an intermediate representation of the generator using the optimal settings and stores the optimal settings of the generator in a cache for later exploration processes. In process 308, the compilation system 200a generates a primitive-level buffer semantic representation of the generator using the optimal settings for the generator. For example, the compilation system 200a uses one or more intermediate representations to reduce the generator to a primitive-level buffer semantic representation through a series of compilation passes.

[0067] In process 310, the compilation system 200a adds the primitive-level buffered semantic representation of the generator to the set of primitive-level buffered semantic representations of the generator. The set of primitive-level buffered semantic representations of the generator is used to construct the primitive-level buffered semantic representation of the kernel.

[0068] In process 312, the compilation system 200a uses a set of primitive-level buffer semantic representations of generators to construct primitive-level buffer semantic representations of kernels corresponding to input generators. For example, the compilation system 200a obtains a set of primitive-level buffer semantic representations of generators and code slices the primitive-level buffer semantic representations of generators and their dependencies into a single module or kernel.

[0069] In process 314, the compilation system 200a breaks down a single module into one or more object (.o) files and stores one or more object files of the kernel in a datastore, such as the CAS of the generated kernel 208, though not limited to these. In some examples, the object files have the format of object files generated by a standard C-style toolchain and therefore work seamlessly with a stack that implements a C / C++ External Generator Interface (FFI).

[0070] fusion Compilation system 200a uses several types of fusion. Examples, though not limited to, include element-wise fusion (processing one element at a time), reduction fusion (reducing a set of elements into a smaller set of elements), and matrix multiplication (matmul) fusion (enabling fusion to matmul operations).

[0071] Element-level operations operate on an element-by-element basis. For example, for a one-dimensional tensor, this corresponds to the following loop:

[0072]

number

[0073] Here, "shape" is the shape of the input tensor, and "func" depends on the definition of the operation (for example, the Tanh element-wise operation has Tanh as its innermost function). In some examples, the compile system 200a defines an element-level generator as follows:

[0074]

number

[0075] Such generators are parameterized with respect to rank for performance reasons and accept both the shape of the element-wise operation and the function called for each index. They are then parallelized with respect to the outermost dimension and vectorized with respect to the innermost dimension. In some examples, the compile system 200a allows kernel writers to specify these optimization patterns and provide their own functions.

[0076] The element-level layer operates on each element of the input tensor. Figure 4A shows an exemplary element-level sequence commonly found in generative models. The first binary operator binop1(406) receives two inputs, input 0(402) and input 1(404). The second binary operator binop2(408) receives the output of binop1(406) as input and input1(416) as another input. The first unary operator unaryop1(410) receives the output of binop2(408) as input. The second unary operator unaryop2(412) receives the output of unaryop1(410) as input and generates output 414.

[0077] Each of these layers operates on each element of the input tensor, transforming that element in the process. The graph in Figure 4A can be represented as follows using the generators described above.

[0078]

number

[0079] In some examples, each of the above operations can take a broadcasted value. Broadcasting allows element-wise operations to be performed on two tensors that do not match in rank or dimension. There are two types of broadcasts: explicit and implicit. In an explicit broadcast, the broadcasted value is materialized. For example, when shape(A)=d_0×d_1 and shape(B)=1×1 (i.e., a scalar), during the calculation of A+B, a new matrix containing the value of B, i.e., matrix B'=d_0×d_1, is created. All calculations using A+B are replaced with A+B'. Alternatively, one can perform an implicit broadcast. This means that the operator includes logic to handle the case where matrix B can be broadcast onto A. This makes kernel description potentially more difficult but does not sacrifice speed. For example,

[0080]

number

[0081] That is the case. In general, implicit broadcasting is preferred over explicit broadcasting because it eliminates the need to materialize large tensors, improves locality, and reduces memory pressure.

[0082] Note that broadcast fusion is obtained from the above. For example, when adding a scalar B to an N-dimensional tensor A, the conventional framework evaluates this by broadcasting B onto the dimension of A (creating a temporary tensor), as illustrated below.

[0083]

number

[0084] However, the following formula can be used in the system described herein.

[0085]

number

[0086] In some examples, a user can define a matmul that has insertion points to leverage the internal matmul algorithm, as shown in the following example of a typical naive matmul algorithm.

[0087] The system provided in this disclosure provides an element-level epilogue and an element-level prologue. Without the prologue, some optimizations, but not limited to, cannot be represented, such as quantization fusion within the matmul function.

[0088]

number

[0089] The code above takes five functions as input. • load_a_prologue: Defines how to load an A element given a row / column index. This function returns an element of type InputType.

[0090] • load_b_prologue: Defines how to load the B element given a row / column index. This function returns an element of type InputType. • load_c_prologue: Defines how to load a C element given a row / column index. This function returns an element of type InputType.

[0091] • elem_wise_epilogue: Defines how to handle a single computed element. This function takes a computed value of type OutputType along with the row / column index.

[0092] · row_wise_epilogue: Defines how to handle a single computed row. This function takes the row index along with the computed row value of type OutputType. This function stores the row value in the output buffer.

[0093] Using the above, a specialized version of the algorithm can be created. The conventional matmul may have the following signature:

[0094]

number

[0095] This depends on the transposeA and transposeB flags,

[0096]

number

[0097] The system provided in this disclosure calculates one of the following. The system provides a definition of matmul as exemplified below.

[0098]

number

[0099] One limitation of conventional General Matrix Multiplication (GEMM) implementations is that they cannot represent $A.B+bias$, where $bias$ is a vector. This is not a limitation of the generalized matmul described herein. In one example, the loading of the $C$ matrix above is modified so that the value of $bias$ is broadcast to $C$ as illustrated below.

[0100]

number

[0101] In one example, the element-level epilogue function is modified to accumulate bias in the resulting value.

[0102]

number

[0103] Both of the above examples are equivalent, but theoretically, the second implementation should be faster because it avoids using memcpy. Based on matmul + bias processing, a fully connected (FC) layer can be defined. The FC layer executes the function $activation(A.B+bias)$, as exemplified below.

[0104]

number

[0105] In some cases, fusion patterns can be fused with each other. For example, element-level fusion, reduction fusion, and matmul fusion patterns can be fused together to generate a single-head attention block.

[0106] The equation for single-head attention (also known as the scaled dot product) is shown below.

[0107]

number

[0108] Here Q = Query vector representing the current word or token being processed by the attention mechanism.

[0109] K = A key vector representing other words or tokens in the input sequence that may be potentially related to the current word or token. V = A value vector representing the values ​​associated with the key vector, for example, the word embeddings of the corresponding words or tokens.

[0110] T = the number of dimensions of the input vector, which includes the query vector "Q", the key vector "K", and the value vector "V". d k = Number of dimensions for attention weights.

[0111] The softmax function can be defined as follows:

[0112]

number

[0113] Note that the first step is an element-level operation, and the second and third steps are row-level operations. Therefore, the fused attention block to be implemented can be defined as follows:

[0114]

number

[0115] A fusion pattern is declaratively defined by the user through annotations that describe the types of fusions that are allowed. These annotations communicate with the graph compiler to describe what types of fusions are permitted and how the places where fusions are permitted should be integrated. Referring to Figure 4B, process 422, generated by the kernel generator, has a pair of input ports of arbitrary size, as indicated by input ports 1(418) to N(420), and a pair of output ports of arbitrary size, as indicated by output ports 1(424) to N(426). At runtime, process 422 receives input from the pair of input ports and generates output that is output through the output ports. These ports are functions that can perform arbitrary operations, such as load / store, activation calculation, and reduction, but are not limited to these.

[0116] The user communicates this information to the graph compiler by declaring these ports and functions. For example, in the generic matmul example above, the registration is illustrated as follows:

[0117]

number

[0118] Figure 5A is a collaboration diagram of the multilayer graph compiler pipeline 500a with several examples, and Figure 5B is an activity diagram of the compilation method 500b that uses a multilayer computation graph. The compilation system 200a uses the multilayer graph compiler pipeline 500a to generate an executable object (e.g., executable object 526) using the multilayer computation graph 530 and a set of kernels (e.g., a set of system kernels 528 and a set of user-provided kernels 522).

[0119] In many ML systems, models are represented as a series of computation graphs. These graphs can be optimized to eliminate redundant computations, perform device-specific transformations, and fuse kernels. Traditional graph optimizers use a single-layer intermediate representation to set optimization conditions based on a specified device or kernel library when the graph is initialized. Such processes introduce complexity both in the implementation of the optimization path and in extending the system that generates the model.

[0120] To mitigate some of the computation graph-related problems in other systems and to provide better integration with compilation system 200a, a multilayer computation graph representation 530 is employed.

[0121] In process 502, the compilation system 200a receives a multilayer computation graph 530, which includes a modularized operations graph 510. For example, the modularized operations graph 510 includes one or more modularized operations that form a Modular Operations (MO) layer. The modularized operations graph 510 provides a type system, device-independent rewriting, and a canonical set of operations for performing the rewriting.

[0122] In process 504, the compilation system 200a generates a modular operation generation graph 524 using the modular operation graph 510 and one or more sets of kernels (but not limited to one set of system kernels 528 and one set of user-provided kernels 522). In the Modular Operation Generator Graph (MOGG) layer of the multilayer computation graph 530, the operations of the modular operation generation graph 524 are kernel-aware. During the transition 534 from the modular operation graph 510 to the modular operation generation graph 524, the kernel is selected by the kernel selection component 512 in a selection process from one set of system kernels 528 and one set of user-provided kernels 522. The selected kernel is chosen based on the definition of a set of modular operations in the modular operation graph 510. The kernel selection component 512 generates a set of MOGG operations for the modular operation generation graph 524, which have properties that reflect how the selected kernel from the set of user-provided kernels 522 is configured. In some examples, the user provides one or more user kernel registrations 514 that contain metadata for a set of user-provided kernels 522, which the kernel selection component 512 uses to select a kernel from a set of user-provided kernels 522.

[0123] In some examples, for kernels written for compilation system 200a, the MOGG operations in modularized operation generation graph 524 directly represent the underlying structure of the kernel generator on which the kernel is based. For example, the above example of generalized_matmul is translated into the following multilayer intermediate representation (type specifications are omitted for brevity):

[0124]

number

[0125] In the example above, all operations are labeled with a fusion type, and identity implementations are created for all prologue and epilogue functions. These processes are automatically performed via declarations and registration decorators added to the kernel when the ML model is converted to binary format.

[0126] In process 506, the fusion pipeline 516 of the compilation system 200a generates an optimized modular arithmetic generation graph 532 using the modular arithmetic generation graph 524 by fusing one or more of the two or more arithmetic operations in the modular arithmetic generation graph 524. As a result, the optimized modular arithmetic generation graph 532 contains one or more fusing operations. For example, after the modular arithmetic generation graph 524 is generated by the compilation system 200a, the fusion pipeline 516 has specific information on both device targets and kernel definitions for all operations in the modular arithmetic generation graph 524. Therefore, the fusion pipeline 516 can perform device and kernel-specific optimizations 536 on the computation modular arithmetic generation graph 524.

[0127] In some examples, all kernels, and all prologue and epilogue functions within them, have fusion classifications such as element-wise, row-wise, reduction, and opaque, though these are not limited. The fusion pipeline 516 traverses the modularized arithmetic generation graph 524 and finds where the body of a kernel matches a suitable input lambda of another kernel. For kernels labeled as element-wise, row-wise, and reduction, these bodies can be directly inlined into suitable input or output lambdas of other kernels. In some examples, kernels labeled as opaque are not fused into other kernels but can accept the fusion of many other kernels into their own inputs and outputs.

[0128] In process 508, the kernel compiler of the compilation system 200a generates an executable object using the optimized modularized arithmetic generation graph 532. For example, the generated lambda function is inlined into the user's kernel definition by copying the definition of the lambda function into the kernel definition when the lambda function is called, during compilation by the kernel compiler 518, and is executed as a new native kernel in the final executable object 526 executed by the runtime system 520.

[0129] In some examples, the compilation system 200a provides user extensibility of the compilation process by implementing a multilayer graph compiler pipeline 500a having a fusion pipeline 516 as described herein. Since the fusion pipeline 516 operates on a modular arithmetic generation graph 524 after device-specific kernel selection by the kernel selection component 512, the interface is generated from the kernel definition, and the user-provided kernel only needs to satisfy the application programming interface (API) expected by the fusion pipeline 516. Thus, the user-provided kernel and arithmetic are treated as first-class objects by the compilation system 200a at compile time and can undergo the same optimization processing as other first-class objects within the compilation system 200a.

[0130] Machine learning pipeline Figure 6B is a flowchart of a machine learning pipeline 616 with several examples. The machine learning pipeline 616 may be used to generate a trained machine learning model 618, for example, a machine learning model used by the AI ​​component 104 in Figure 1 to perform kernel search and compiler optimization.

[0131] overview Generally speaking, machine learning can involve using computer algorithms to automatically learn patterns and relationships in data, potentially without requiring explicit programming. Machine learning algorithms can be divided into three main categories: supervised learning, unsupervised learning, and reinforcement learning.

[0132] Supervised learning involves training a model using labeled data to predict the output for new, unknown inputs. Examples of supervised learning algorithms include linear regression, decision trees, and neural networks.

[0133] Unsupervised learning involves training models on unlabeled data to find hidden patterns and relationships within the data. Examples of unsupervised learning algorithms include clustering, principal component analysis, and generative models such as autoencoders.

[0134] Reinforcement learning involves training models to make decisions in dynamic environments by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms include Q-learning and policy gradient methods.

[0135] Examples of specific machine learning algorithms that can be developed include logistic regression, a type of supervised learning algorithm used for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another exemplary type of machine learning algorithm is naive Bayes, another supervised learning algorithm used for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that predictor variables are independent of each other. Random forest is another type of supervised learning algorithm used for classification, regression, and other tasks. Random forest builds a set of decision trees and combines their outputs to make predictions. Further examples include neural networks, which consist of interconnected layers of nodes (or neurons) that process information and make predictions based on input data. Matrix factorization is another type of machine learning algorithm used for recommendation systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to reveal hidden patterns or relationships in the data. Support vector machines (SVMs) are a type of supervised learning algorithm used for classification, regression, and other tasks. SVM finds a hyperplane that separates different classes within the data. Other types of machine learning algorithms include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer models. The choice of algorithm depends on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0136] The performance of a machine learning model is typically evaluated against a separate test set of data not used during training, in order to ensure that the model can generalize to new, unknown data.

[0137] While several specific examples of machine learning algorithms are described herein, the principles described herein can also be applied to other machine learning algorithms. Deep learning algorithms such as convolutional neural networks, recurrent neural networks, and transformers, as well as more traditional machine learning algorithms such as decision trees, random forests, and gradient boosting, can be used in a variety of machine learning applications.

[0138] Three exemplary types of problems in machine learning are classification problems, regression problems, and generative problems. Classification problems, also called categorization problems, aim to classify an item into one of several categorical values ​​(e.g., Is this object an apple or an orange?). Regression algorithms aim to quantify several items (e.g., by providing values ​​that are real numbers). Generative algorithms aim to generate new examples similar to the examples provided for training. For example, a text generation algorithm is trained on many text documents and configured to generate new, consistent text with similar statistical properties to the training data.

[0139] Training Phase Generating a trained machine learning model 618 may involve several phases that form part of a machine learning pipeline 616, including, for example, the following phases shown in Figure 6A.

[0140] • Data acquisition and preprocessing 602: This phase may include acquiring and cleaning the data to ensure that it is suitable for use in machine learning models. This phase may also include removing duplicates, handling missing values, and converting the data to an appropriate format.

[0141] Feature Engineering 604: This phase may include selecting and transforming training data 622 to create features that are useful for predicting the target variable. Feature engineering may include (1) receiving the features 624 (e.g., as structured or labeled data in supervised learning) and / or (2) identifying the features 624 (e.g., unstructured or unlabeled data for unsupervised learning) in the training data 622.

[0142] • Model Selection and Training 606: This phase may include selecting an appropriate machine learning algorithm and training it on preprocessed data. This phase may further include splitting the data into training and test sets, evaluating the model using cross-validation, and tuning hyperparameters to improve performance.

[0143] • Model Evaluation 608: This phase may involve evaluating the performance of the trained model (e.g., trained machine learning model 618) on a separate test dataset. This phase can help determine whether the model is overfitting or underfitting and whether the model is suitable for deployment.

[0144] • Prediction 610: This phase involves using a trained model (e.g., a trained machine learning model 618) to generate predictions for new, unknown data. • Validation, refinement, or retraining 612: This phase may include updating the model based on feedback generated from the prediction phase, such as new data or user feedback.

[0145] Deployment 614: This phase may involve integrating the trained model (e.g., trained machine learning model 618) into a broader system or application such as a web service, mobile app, or IoT device. This phase may involve setting up APIs, building user interfaces, and ensuring that the model is scalable and capable of handling large amounts of data.

[0146] Figure 6B shows further details of two exemplary phases, namely, the training phase 620 (e.g., part of model selection and training 606) and the prediction phase 626 (part of prediction 610). Prior to the training phase 620, feature engineering 604 is used to identify features 624. This may involve identifying features that are informative, discriminative, and independent in order for the trained machine learning model 618 to work effectively in pattern recognition, classification, and regression. In some examples, the training data 622 includes pre-identified features 624 and labeled data known about one or more outcomes. Each of the features 624 may be a variable or attribute, such as an individually measurable characteristic of a process, item, system, or phenomenon represented by the dataset (e.g., the training data 622). The features 624 may be of different types, such as numerical features, strings, and graphs, and may include, for example, one or more of the content 628, concepts 630, attributes 632, historical data 634, and / or user data 636.

[0147] In training phase 620, the machine learning pipeline 616 uses the training data 622 to find correlations between features 624 that influence the predicted results or predicted / inference data 638.

[0148] Using the training data 622 and the identified features 624, the trained machine learning model 618 is trained in training phase 620 during machine learning program training 640. The machine learning program training 640 evaluates the values ​​of the features 624 so that they correlate with the training data 622. The result of the training is the trained machine learning model 618 (e.g., the trained or learned model).

[0149] Furthermore, the training phase 620 may include machine learning in which the training data 622 is structured (e.g., labeled during preprocessing operations). The trained machine learning model 618 implements a neural network 642 capable of performing classification and clustering operations, for example. In another example, the training phase 620 may include deep learning in which the training data 622 is unstructured, and the trained machine learning model 618 implements a deep neural network 642 capable of performing both feature extraction and classification / clustering operations.

[0150] In some examples, the neural network 642 may be generated during the training phase 620 and implemented within the trained machine learning model 618. The neural network 642 includes a hierarchical (e.g., layered) organization of neurons, where each layer consists of multiple neurons or nodes. Neurons in the input layer receive input data, and neurons in the output layer produce the network's final output. Between the input and output layers, there may be one or more hidden layers, each consisting of multiple neurons.

[0151] Each neuron in the neural network 642 operationally computes a function, such as an activation function, which takes a weighted sum of the outputs of neurons in the previous layer, along with a bias term, as input. The output of this function is passed as input to the neurons in the next layer. If the output of the activation function exceeds a certain threshold, the output is transmitted from that neuron (e.g., a transmitting neuron) to the connected neurons in the subsequent layer (e.g., a receiving neuron). The connections between neurons have associated weights, which define the influence of the input from the transmitting neuron to the receiving neuron. During the training phase, these weights are adjusted by a learning algorithm to optimize the network's performance. Different types of neural networks use different activation functions and learning algorithms, which can affect their performance on different tasks. The hierarchical organization of neurons, as well as the use of activation functions and weights, allows the neural network to model the complex relationships between inputs and outputs and generalize to new inputs not seen during training.

[0152] In some examples, neural network 642 may be, just as an example, one of several different types of neural networks, such as a single-layer feedforward network, a multilayer perceptron (MLP), an artificial neural network (ANN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a bidirectional neural network, a symmetric connected neural network, a deep belief network (DBN), a convolutional neural network (CNN), a generative adversarial network (GAN), an autoencoder neural network (AE), a restricted Boltzmann machine (RBM), a Hopfield network, a self-organizing map (SOM), a radial basis function network (RBFN), a spiking neural network (SNN), a liquid state machine (LSM), an echo state network (ESN), a neural Turing machine (NTM), or a transformer network.

[0153] In addition to the training phase, a validation phase may be performed on a separate dataset known as the validation dataset. The validation dataset is used to tune the model's hyperparameters, such as the learning rate and regularization parameters. The hyperparameters are tuned to improve the model's performance on the validation dataset.

[0154] Once a model is fully trained and validated, in the testing phase, it can be tested on a new dataset. The test dataset is used to evaluate the model's performance and ensure that the model has not overfitted to the training data.

[0155] In the prediction phase 626, the trained machine learning model 618 uses the features 624 to analyze the query data 644 and generates an inference, result, or prediction as an example of the prediction / inference data 638. For example, during the prediction phase 626, the trained machine learning model 618 generates an output. The query data 644 is provided as input to the trained machine learning model 618, and in response to receiving the query data 644, the trained machine learning model 618 generates the prediction / inference data 638 as an output.

[0156] In some examples, the types of training data included in the execution metrics 142 (Figure 2) collected by the compilation system 200a (Figure 2) during runtime 212 (Figure 2) to train the trained machine learning model 618 of the AI ​​component 104 include, but are not limited to, the following:

[0157] Execution metrics (execution phase data): • Performance scores for different kernel configurations, including execution time and load time.

[0158] • Data on the efficiency of kernel execution on various hardware devices. Metrics related to computing resources consumed by the kernel, such as CPU usage, memory usage, and I / O activity.

[0159] Kernel generation data (compile phase data): • Historical data on the success rates of different kernel configurations. • Information regarding the selection of parameter definitions made during kernel generation and their results.

[0160] • Data on the frequency and usage of various kernel parameters and configurations. Kernel creation data (kernel creation phase data): • Code structures and syntax patterns that result in more efficient kernels.

[0161] • Common errors or inefficiencies in kernel code that the AI ​​component can learn to identify and correct. User interaction with the SDE, such as the use of specific tools or features that assist in kernel creation.

[0162] Search data (search phase data): The results of the search for the optimal configuration, including the paths taken through the search space and the effectiveness of different search strategies.

[0163] • The impact of AI-assisted search on the quality and performance of the resulting kernel. Training and prediction data (training data for machine learning programs): • Features extracted from the kernel, and their performance metrics related to training the AI ​​model.

[0164] • Historical data on kernel performance that can be used to train predictive models within the AI ​​component. • Validation and refinement data from iterative training processes to improve the accuracy of AI models.

[0165] User data (software development environment data): • Developer feedback on suggestions provided by the AI ​​component. • Usage patterns of software development environments that can notify recommendations for AI components.

[0166] Expanded data: • Information about how the kernel behaves in a production environment, which can be used to further improve the AI ​​model.

[0167] By collecting and analyzing these types of execution data, the compilation generation system 200a can train the trained machine learning model 618 used within the AI ​​component 104 to better assist the exploration and kernel creation phases, ultimately leading to more efficient and effective kernel generation and deployment.

[0168] In some examples, the compilation system 200a collects kernel compilation data and generated data collected during the compilation phase, and uses the collected compilation and generated data, including but not limited to the following, to train a trained machine learning model 618 used in the AI ​​component 104.

[0169] Compilation time metrics: • The duration of the compilation process for each kernel or set of kernels. • Resources used during compilation, such as CPU and memory usage.

[0170] Intermediate representation data: • Characteristics of the intermediate representation generated during the decomposition from high-level code to machine code. • Transformations applied to the code during the compilation phase and their impact on performance.

[0171] Configuration and parameter definition data: • Specific parameter values ​​selected for kernel generation, and their impact on the performance of the compiled kernel.

[0172] • The frequency and effectiveness of different parameter combinations used during kernel generation. Optimization results: • The success rate of various optimization techniques applied during compilation, such as loop unrolling, vectorization, and inlining.

[0173] • Performance improvements achieved through specific optimizations. Error and warning logs: • Compilation errors and warnings that occur, which can be used to identify common problems and improve the robustness of the compilation process.

[0174] Search algorithm data: • The path taken through the search space when determining the optimal configuration. • The effectiveness of different search strategies and heuristics used by AI components.

[0175] Code generation patterns: • Common patterns or idioms in generated code that correlate with higher performance or efficiency.

[0176] • Variations of assembly or machine code generated for different target architectures. Feedback from runtime performance: Data on how well the kernel performs after deployment can be used to retrospectively adjust the compilation strategy.

[0177] Developer interaction data: • Inputs and adjustments made by developers during the kernel creation phase that affect the compilation result.

[0178] • Usage patterns of compilation flags and directives provided by the developer. By collecting and analyzing this kernel generation data, the AI ​​component can learn to predict the most effective compilation strategy for different scenarios, resulting in more efficient kernel generation and potentially reducing the time and resources required for the compilation phase. This data-driven approach can significantly enhance the capabilities of the AI ​​component in assisting kernel generation and optimization.

[0179] In some examples, the trained machine learning model 618 may be a generative AI model. Generative AI is a term that can refer to any type of artificial intelligence that can create new content from training data 622. For example, generative AI can generate text, images, videos, audio, code, or synthetic data that is similar to but not identical to the original data.

[0180] Some of the technologies that can be used in generative AI are as follows: • Convolutional Neural Networks (CNNs): CNNs can be used for image recognition and computer vision tasks. A CNN can be designed, for example, to extract features from an image by scanning the input image and using filters or kernels to highlight important patterns.

[0181] • Recurrent Neural Networks (RNNs): RNNs can be used to process sequential data such as speech, text, and time-series data. RNNs use a feedback loop that allows them to capture time dependencies and remember past inputs.

[0182] • Generative Adversarial Networks (GANs): GANs can consist of two neural networks: a generator and a discriminator. The generator network attempts to create realistic content that can "fool" the discriminator network, while the discriminator network attempts to distinguish between real and fake content. The generator and discriminator networks compete with each other and improve over time.

[0183] Variational Autoencoders (VAEs): VAEs can encode input data into a latent space (e.g., a compressed representation) and then decode it into output data. The latent space can be manipulated to generate new variations of the output data. VAEs can process input data using self-attention mechanisms, enabling them to handle long text sequences and capture complex dependencies.

[0184] • Transformer Models: Transformer models can use attention mechanisms to learn the relationships between different parts of input data (such as words or pixels) and generate output data based on these relationships. Transformer models can handle sequential data such as text or audio, as well as non-sequential data such as images or code.

[0185] In the example of generative AI, query data 644 may include text, audio, images, videos, numbers, or media content prompts, while output prediction / inference data 638 may include text, images, videos, audio, codes, or synthetic data.

[0186] In some examples, the training phase 620 and prediction phase 626 are executed on a distributed system, such as the compilation system 200a in Figure 2. In some examples, one or more of the operations in training phase 620 and prediction phase 626 are performed on the local device as part of the JIT compilation process.

[0187] Figure 7 shows a collaboration diagram of a networked compilation system 700, in several examples. The compilation system 700 includes one or more computing systems (e.g., computing system 1 (722) to computing system N (702)) communicating over one or more networks (e.g., network 714). In some examples, network 714 is a local area network (LAN). In some examples, network 714 is a wide area network (WAN), such as the Internet.

[0188] The computing system includes one or more computing machines, such as machine 800 in Figure 8. The computing system hosts one or more compilers, such as computing system 1(722) hosting compiler 1(724) and computing system N(702) hosting compiler N(720). Each compiler is communicably connected to other compilers of compilation system 700 (e.g., hosted on separate other computing systems) via one or more communication networks, including a local area network (LAN) 714. The compilers can also communicate with locally hosted applications on separate computing systems using application programming interfaces (APIs).

[0189] The compilation system 700 further includes a cache 716 that is connected to the compiler in a manner that allows communication via one or more networks, such as network 714. The compilation system 700 further includes an IDE server 718 that hosts an Integrated Development Environment (IDE) 704. The IDE 704 is communicated to the compiler via one or more communication networks, such as network 714.

[0190] The compiler interacts with other compilers and IDE 704 via network 714. The data exchanged between the compiler and IDE 704 includes functions (e.g., commands for calling functions) and payload data (e.g., coding logic for compilation).

[0191] The client computing system 708 hosts IDE clients 710 that are communicably connected to the IDE 704 via one or more communication networks, such as network 706. In some examples, network 706 is a LAN. In some examples, network 706 is a WAN, such as the Internet.

[0192] The user uses the IDE client 710 to communicate with the IDE 704 and write the coding logic to be compiled by the compiler. During compilation, the compiler may access the cache 716 to store an intermediate representation and / or executable object of the coding logic. In some examples, the IDE client 710 communicates with the client cache 712. During compilation, the compiler may access the client cache 712 via the IDE client 710 to store an intermediate representation and / or executable object of the coding logic.

[0193] The compilation system 700 provides server-side compilation functionality, as described herein, to the IDE client 710 via the network 706. While specific functions of the compilation system 700 are described herein as being performed by compilers (e.g., compiler 1(724) and compiler N(720)), the IDE 704, or one or more client-side APIs or applications, the location of specific functions within the compilation system 700 or the client computing system 708 can be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the client computing system 708 and then later migrate these technologies and functions to the compilation system 700 if the computing system of the compilation system 700 (e.g., computing system N(702)) has sufficient processing power.

[0194] The compilation system 700 supports various services and processes provided to the client computing system 708. Such processes include sending data to and receiving data from the compilation system 700 and the client computing system 708, and processing the data generated by them. This data may include, but is not limited to, coding logic, intermediate representations and / or executable objects of the coding logic, compilation metrics, and execution metrics for one or more executable objects. The IDE 704 provides one or more user interfaces (UIs) that a user uses to access the functions of the compilation system 700 via the IDE client 710.

[0195] In some examples, IDE704, cache716, and one or more compilers such as compiler1(724) and compilerN(720) are hosted on a single computing system. In some examples, IDE704, cache716, and one or more compilers such as compiler1(724) and compilerN(720) are hosted in a cloud-based computing environment.

[0196] Figure 8 is a schematic representation of machine 800 on which an instruction 810 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause machine 800 to perform any one or more of the methods described herein. For example, instruction 810 can cause machine 800 to perform any one or more of the methods or processes described herein. Instruction 810 transforms a general-purpose, unprogrammed machine 800 into a specific machine 800 programmed to execute the described and illustrated generator in the described manner. Machine 800 may operate as a standalone device or may be connected to other machines (e.g., network connection). In a networked deployment, machine 800 may operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 800, in combination with other components of the compiler system, may operate as a server, client, computer, personal computer (PC), tablet computer, laptop computer, or any other machine capable of sequentially or otherwise executing instructions 810 that specify actions to be performed by Machine 800. Furthermore, although a single Machine 800 is shown, the term “machine” may be interpreted to include a set of machines that individually or collectively execute instructions 810 to perform any one or more of the methods described herein.

[0197] Machine 800 may include one or more processors 802, memory 804, and I / O device interfaces 806, which can be configured to communicate with each other via bus 832. In one example, the processors 802 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a composite instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 808 and 812 that execute instruction 810. The term “processor” is intended to include multicore processors, which may include two or more independent processors (sometimes called “cores”) capable of executing instructions simultaneously. Figure 8 shows multiple processors 802, but machine 800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multicore processor), multiple processors with single cores, multiple processors with multiple cores, or any combination thereof.

[0198] Memory 804 includes main memory 814, static memory 816, and storage unit 818, all of which are accessible to processor 802 via bus 832. Main memory 804, static memory 816, and storage unit 818 store instructions 810 that embody any one or more of the methods or generators described herein. Instructions 810 may also reside, fully or partially, in main memory 814, static memory 816, non-temporary machine-readable media 820 in storage unit 818, in one or more of processors 802 (e.g., in the processor's cache memory), or in any suitable combination thereof during their execution by machine 800.

[0199] The I / O device interface 806 connects machine 800 to I / O device 834. One or more of the I / O devices 834 may be components of machine 800 or may be separate devices. The I / O device interface 806 may include a wide variety of interfaces to I / O devices 834 used by machine 800 for purposes such as receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, and capturing measurements. The specific I / O device interface 806 included in a particular machine depends on the type of machine. It will be understood that the I / O device interface 806 of I / O device 834 may include many other components not shown in Figure 8. In various examples, the I / O device interface 806 may include an output component interface 824 and an input component interface 828. The output component interface 824 may include interfaces to visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistors), and other signal generators. The input component interface 828 may include interfaces to alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing devices), haptic input components (e.g., physical buttons, touchscreens that provide the position and / or force of touch or touch gestures, or other haptic input components), and audio input components (e.g., microphones).

[0200] Communication can be implemented using a wide variety of technologies. The I / O device interface 806 further includes a communication component interface 830 that can operate to connect machine 800 to network 822 or one or more devices 836 via couplings 826 and 838, respectively. For example, the communication component interface 830 may include an interface to a network interface component or other suitable device for interface with network 822. In further examples, the communication component interface 830 may include an interface to a wired communication component, a wireless communication component, a cellular communication component, a near-field communication (NFC) component, a Bluetooth® component (e.g., Bluetooth® Low Energy), a Wi-Fi® component, and other communication components for communicating via other modalities. Device 836 may be another machine or one of a wide variety of peripheral devices (e.g., a peripheral device connected via USB).

[0201] Various memories (e.g., memory 804, main memory 814, static memory 816, and / or the memory of processor 802) and / or storage unit 818 may embody any one or more of the methods or generators described herein, or store one or more sets of instructions and data structures (e.g., software) used thereby. These instructions (e.g., instruction 810) cause processor 802 to perform various operations to implement the disclosed examples when executed.

[0202] Instruction 810 may be transmitted or received over network 822 using a transmission medium, via a network interface device (e.g., a network interface component included in communication component interface 830), and using one of several well-known transfer protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 810 may be transmitted or received over a transmission medium via coupling 838 to device 836 (e.g., peer-to-peer coupling).

[0203] To better illustrate the systems and methods described herein, a list of non-limiting examples is provided below. Example 1 is a method performed by a computer, comprising the steps of: receiving a multilayer computation graph including a modularized computation graph including a set of modularized operations; generating a modularized computation generation graph using the modularized computation graph and one or more sets of kernels; generating an optimized modularized computation generation graph by performing a fusion of one or more operations of two or more operations in the modularized computation generation graph; and generating an executable object using the optimized modularized computation generation graph.

[0204] In Example 2, the subject of Example 1 includes the step of generating a modularized operation generation graph, which involves selecting a kernel from a set of system kernels and a set of user-provided kernels using a set of modularized operation definitions for the modularized operation graph.

[0205] In Example 3, the subject matter of either Example 1 or 2 includes the fact that one of the user-provided kernels in a set is treated as a first-class object during the compilation process.

[0206] In Example 4, the subject of any of Examples 2-3 includes the step of selecting a kernel, which is further performed using metadata from a set of user-provided kernels. In Example 5, any of the themes from Examples 1-4 involves the operation of a modularized operation generation graph including one or more prologue functions that define the loading of input data into the operation.

[0207] In Example 6, any of the themes from Examples 1 to 5 involves the operation of a modularized operation generation graph including one or more epilogue functions that define the writing of the operation to the output data.

[0208] Example 7 is at least one machine-readable medium containing an instruction, the instruction causing the processing circuit to perform a process to carry out one of Examples 1 to 6 when the instruction is executed by the processing circuit. Example 8 is an apparatus that includes means for carrying out any of Examples 1 to 6.

[0209] Example 9 is a system that implements any of Examples 1 through 6. Example 10 is a method of implementing any of Examples 1 through 6. You may make changes and modifications to the disclosed examples without departing from the scope of this disclosure. These and other changes or modifications are intended to be included within the scope of this disclosure as expressed in the following claims.

[0210] Glossary "Carrier signal" means any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communication signals or other intangible medium for enabling the communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0211] "Client device" means any machine that interfaces to a communication network and obtains resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, desktop computer, laptop, personal digital assistant (PDA), smartphone, tablet, ultrabook, netbook, laptop, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communication device that a user may use to access the network.

[0212] "Communication Network" means one or more parts of a network that may be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), Internet, part of the Internet, part of the public switched telephone network (PSTN), conventional telephone service (POTS) network, mobile phone network, wireless network, Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or part of a network may include a wireless or cellular network, and the connection may be a code division multiple access (CDMA) connection, a Global System for Mobile Communications (GSM®) connection, or another type of cellular or wireless connection. In this example, the connection may implement any of the various types of data transfer technologies, including single-carrier radio transmission technology (1xRTT), Evolution Data Optimization (EVDO) technology, General-Purpose Packet Radio Service (GPRS) technology, Enhanced Data Rate for GSM® Evolution (EDGE) technology, 3G Partnership Projects (3GPP®) including 3G, 4G radio networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Interoperability for Microwave Access (WiMAX), Long-Term Evolution (LTE) standards, others defined by various standardization bodies, other long-range protocols, or other data transfer technologies.

[0213] "Machine-readable medium" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "machine-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0214] "Mechanical storage medium" means one or more storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions, routines and / or data. The term includes, but is not limited to, solid-state memory, optical media and magnetic media, including internal or external memory of a processor. Specific examples of mechanical storage medium, computer storage medium and / or device storage medium include non-volatile memory, including semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "mechanical storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "mechanical storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, some of which are covered under the term "signaling medium."

[0215] A "processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic running on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine code," etc.) and generates the associated output signals applied to operate a machine. A processor can be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a composite instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multicore processor having two or more independent processors (sometimes called "cores") that can execute instructions simultaneously.

[0216] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible mediums for enabling the communication of software or data. The term "signal medium" may be interpreted to include any form, such as modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal in which one or more of its properties are set or modified in order to encode information into a signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0217] A "kernel" or "microkernel" is an implementation of an algorithm that performs computations on memory objects, such as memory buffers, of a specific layout. While the two terms can be used interchangeably, "microkernel" tends to refer to smaller operations (e.g., memset, dot product, or reduction) within a larger generator kernel implementation. Algorithmically interchangeable / equivalent / substitutable kernels are sometimes referred to as "codelets" in the literature.

[0218] A "generator" is a metaprogram that is parameterized, executed, and produces a non-parametric implementation of a kernel or microkernel. A fixed kernel implementation (e.g., a panel-dot product implemented in assembly) is a degenerate case of a generator that does not have parameters.

[0219] A kernel interface declaration is a declaration of a kernel or microkernel that applies to multiple implementations of the kernel or microkernel. Kernels and microkernels can be implemented multiple times in several different ways. Interface declarations can exist independently of implementations and can allow clients and implementations to be type-checked.

[0220] "Generator parameter arguments" refer to values ​​that a kernel or microkernel can interact with. A generator is a metaprogram that generates kernels, and "parameters" are values ​​that this metaprogram can create.

[0221] The "kernel generator parameter result" is a value returned by the generator as a parameter to its source, allowing the generator to adapt to the behavior within the generated subkernel. For example, a panel dot product generator might return "processed a 3x5 panel of memory," causing the caller's for loop to advance by 3 and 5 in each dimension.

[0222] A "generator constraint" is a constraint that indicates a limitation on the parameters of a kernel or microkernel, such as "this implementation only works with dtype=float32", or "this only works on machines with the x86 VNNI extension", or "this works based on the remainder when the size is divided by 136". A generator may be a partial generator from an interface declaration to a concrete implementation. Constraints propagate upward from the kernel implementation to the generator graph.

[0223] "Kernel arguments" are static single assignment (SSA) argument values ​​used for buffers and other user-defined types to structured abstractions in memory, such as linear memory, N-dimensional tensors with layouts, trees and tables, and other higher-level data types; values ​​corresponding to op attributes at the tensor graph level (these values ​​may be modeled as constants there, but are dynamic in the kernel's runtime implementation); and very small microkernels at the bottom of the stack (e.g., adding two integers) that use arguments for their input.

[0224] A "kernel result" is an SSA result value, which is used for dynamically allocated result buffers, such as those with data-dependent shapes, and for very small microkernels at the bottom of the stack (e.g., adding two integers) that use the result for their output.

Claims

1. A method by which a computer performs an action. A step of receiving a multilayer computation graph including a modularized computation graph, wherein the modularized computation graph includes a set of modularized operations. A step of generating a first modularized operation generation graph using the modularized operation graph and one or more sets of kernels, The steps include generating a second modularized operation generation graph that includes one or more merged operations by performing the fusion of one or more operations of the aforementioned modularized operation generation graph, A method performed by a computer, comprising the steps of generating an executable object using the second modularized operation generation graph which includes one or more of the aforementioned fused operations.

2. The step of generating the second modularized operation generation graph is: A method performed by the computer according to claim 1, comprising selecting a kernel from a set of system kernels and a set of user-provided kernels using the definition of a set of modular operations in the modular operation graph.

3. The method performed by the computer according to claim 1, wherein the selection of a kernel is performed using metadata of a set of user-provided kernels.

4. A method for a computer to execute according to claim 1, wherein one of a set of user-provided kernels is treated as a first-class object during the compilation process.

5. The method performed by a computer according to claim 1, wherein the operation of the first modularized operation generation graph includes one or more prologue functions that define the loading of input data into the operation.

6. The method performed by a computer according to claim 1, wherein the calculation of the first modularized calculation generation graph includes one or more epilogue functions that define the writing of the calculation to the output data.

7. It is a machine, At least one processor, The system comprises at least one memory for storing instructions, and the instructions, when executed by the at least one processor, are sent to the machine. A process for receiving a multilayer computation graph including a modularized computation graph, wherein the modularized computation graph includes a set of modularized operations, and the process for receiving the multilayer computation graph includes a set of modularized operations. A process for generating a first modularized operation generation graph using the aforementioned modularized operation graph and one or more sets of kernels, A process to generate a second modularized operation generation graph containing one or more merged operations by performing the fusion of one or more operations of the aforementioned modularized operation generation graph, A machine that performs the process of generating executable objects using the second modularized operation generation graph which includes one or more fused operations.

8. The process for generating the first modularized operation generation graph is as follows: The machine according to claim 7, comprising selecting a kernel from a set of system kernels and a set of user-provided kernels using the set of modularized operation definitions of the modularized operation graph.

9. The machine according to claim 7, wherein kernel selection is performed using metadata of a set of user-provided kernels.

10. The machine according to claim 7, wherein one of a set of user-provided kernels is treated as a first-class object during the compilation process.

11. The machine according to claim 7, wherein the operations of the first modularized operation generation graph include one or more prologue functions that define the loading of input data into the operations.

12. The machine according to claim 7, wherein the operations of the first modularized operation generation graph include one or more epilogue functions that define the writing of the operations to the output data.

13. A machine storage medium for storing instructions, wherein, when the instructions are executed by the machine, the machine... A process for receiving a multilayer computation graph including a modularized computation graph, wherein the modularized computation graph includes a set of modularized operations, and the process for receiving the multilayer computation graph includes a set of modularized operations. A process for receiving a multilayer computation graph including a modularized computation graph, wherein the modularized computation graph includes a set of modularized operations, and the process for receiving the multilayer computation graph includes a set of modularized operations. A process for generating a first modularized operation generation graph using the aforementioned modularized operation graph and one or more sets of kernels, A process to generate a second modularized operation generation graph containing one or more merged operations by performing the fusion of one or more operations of the aforementioned modularized operation generation graph, A machine storage medium that performs the process of generating executable objects using the second modularized operation generation graph which includes one or more fused operations.

14. The process for generating the first modularized operation generation graph is as follows: The machine storage medium according to claim 13, comprising selecting a kernel from a set of system kernels and a set of user-provided kernels using the definition of a set of modular operations in the modular operation graph.

15. The machine storage medium according to claim 13, wherein one of a set of user-provided kernels is treated as a first-class object during the compilation process.

16. The machine storage medium according to claim 13, wherein kernel selection is performed using metadata of a set of user-provided kernels.

17. The machine storage medium according to claim 13, wherein the operations of the modularized operation generation graph include one or more prologue functions that define the loading of input data into the operations.

18. The machine storage medium according to claim 13, wherein the calculation of the modularized calculation generation graph includes one or more epilogue functions that define the writing of the calculation to the output data.

Images