Composable Kernel

Abstract

A compiler for generating composable kernels. The compiler receives a kernel generator having kernel parameter definitions and code for a set of operators written in a general-purpose programming language. For each operator in the set of operators, the compiler subclassifies the code for each operator into an intermediate representation of each operator in an intermediate language, determines a configuration for each operator based on the kernel parameter definition and the intermediate representation of each operator, and generates a binary object for each operator in a set of binary objects for the set of operators based on the configuration. The compiler composes a kernel corresponding to the kernel generator based on the set of binary objects for the set of operations and the kernel parameter definition.

Description

[Technical Field]

[0001] (Priority Claim) This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 442,742, filed February 1, 2023, which is incorporated herein by reference in its entirety.

[0002] (Technical field) The present disclosure relates generally to compilers, and more particularly to compilers for computationally intensive code. [Background technology]

[0003] A compiler is used to generate object code from a high-level language, preferably an optimized object code. [Brief explanation of the drawings]

[0004] To easily identify the description of any particular element or operation, the most significant digit(s) of a reference number refers to the number of the figure in which that element is first introduced. [Figure 1] FIG. 1 is a data flow diagram of a composable kernel generation and usage process, according to some examples. [Figure 2A] FIG. 2A is a block diagram of a composable kernel generation system 200a, according to some examples. [Figure 2B] FIG. 2B is a process flow diagram of a kernel generation method 200b, according to some examples. [Figure 2C] FIG. 2C is a data flow diagram of a generator generation method data flow, according to some examples. [Figure 2D] FIG. 2D is a process flow diagram of a generator generation method, according to some examples. [Figure 2E] FIG. 2E is a data flow diagram of a generator refinement method data flow, according to some examples. [Figure 2F]FIG. 2F is a process flow diagram of a generator refinement method 200f, according to some examples. [Figure 3A] FIG. 3A is a data flow diagram of a model generation data flow of a model generation method, according to some examples. [Figure 3B] FIG. 3B is a process flow diagram of a model compilation method 300b, according to some examples. [Figure 4] FIG. 4 is a block diagram of a production system, according to some examples. [Figure 5] Figure 5 shows a machine learning pipeline, with some examples. [Figure 6] FIG. 6 illustrates the training and use of a machine learning program, according to some examples. [Figure 7] FIG. 7 is a diagrammatic representation of a machine upon which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein, according to some examples. [Figure 8] FIG. 8 is a deployment diagram for a composable kernel generation and deployment system, according to some examples. DETAILED DESCRIPTION OF THE INVENTION

[0005] Other technical features may be readily apparent to those skilled in the art from the following figures, descriptions, and claims. Kernel libraries provide high-performance implementations of numerical and data processing algorithms optimized to take advantage of vector, multi-core specialized hardware blocks, and specialized accelerators available in modern computers. Modern machine learning frameworks are typically collections of hundreds to thousands of these kernels, integrating kernels from many different sources into a consistent API complemented by gradient computation and other higher-level functionality. These kernel libraries were a factor in the early success of frameworks like TensorFlow™ and PyTorch™, providing a "hackable" interface for extending these frameworks. This extensibility has enabled new research and integration into existing legacy applications (critical for data loading).

[0006] Kernel libraries have well-known scalability challenges: generating high-performance numerical kernels is difficult and requires significant time for rare expertise to build. Furthermore, these libraries are often insufficient to scale to thousands of operators. or function This makes it difficult to bring up new hardware. In fact, these libraries are usually not retuned for new generations of hardware.

[0007] Compiler engineers have risen to the challenge of solving this problem. Some systems focus on a small subset of the problem (dense linear algebraic operators with static shapes) using a closed set of operators through manual "intelligent design" of compiler heuristics. That said, some systems have successfully demonstrated that aggressive kernel fusion, data layout optimization, and highly challenging accelerators are within reach of modern machine learning (ML) model compilers. Some systems extend ideas from other systems to provide more flexible and extensible architectures, and many of these heuristics are now standardized. Adaptive Compilationto provide a different design point between the kernel writer and the computer.

[0008] These systems have limitations regarding specialization that make it difficult to support new hardware developments and to extend these systems to address a broader class of problems.

[0009] Examples of this disclosure break down these barriers and provide a next-generation system that establishes a new point in the design space, one that combines the ability of human experts to reason about kernel architecture and numerical precision while benefiting from computer attention to detail and from an open architecture.

[0010] In some examples, the computer-implemented method may involve one or more processors executing a kernel parameterization and a program written in a general-purpose programming language. generator Equipped with a set of codes Ruj The method includes receiving a generator. generator Each of the set generator To perform an action on generator The code in the intermediate language generator The kernel parameter definition and each generator Based on the intermediate representation of generator Determine the configuration of, generator Each of the set of binary objects in the set generator The method includes generating a binary object for the set of operations based on the set of binary objects and the kernel parameter definition. , Ji The method further includes configuring a kernel corresponding to the generator.

[0011] In some instances, generator Converting the code of generator This involves decomposing the code into a lower level intermediate language. In some examples, a computer-implemented method includes, by one or more processors, receiving an input model of a computation, the model being written in a high-level language; and converting code of the model into an intermediate representation in a first intermediate language, the intermediate representation converting a set of input variables, a set of output variables, and a set of input values ​​corresponding to the set of input variables into a set of output values ​​corresponding to the set of output variables. generator and converting the set of data into an intermediate representation, the intermediate representation including the set of data, by one or more processors based on a selection criterion. generator From the set of generator The first set of generator The method includes, by one or more processors, determining a second set of generator generating, by one or more processors, a set of optimized primitive-level buffer semantic representations of the first set of generator A second set of non-optimized primitive-level buffer semantic representations and fallbacks generator and constructing an executable model corresponding to the input model based on the set of optimized primitive-level buffer semantic representations and the set of non-optimized primitive-level buffer semantic representations, wherein the executable model is in a binary executable format.

[0012] In some instances, transforming the model involves decomposing the model's code into a lower-level intermediate language. Figure 1 is a dataflow diagram of a composable kernel generation and usage dataflow 100 of a composable kernel generation and usage process, according to some examples. A software developer creates a program 110 during a kernel authoring phase 122, a process that will be described in more detail with reference to Figure 4. The program 110 is used by a composable kernel generation system 200a, which will be described in more detail with reference to Figure 2A, to compose a primitive-level representation of a kernel 114 during a compilation phase 124, which will be described in more detail with reference to Figure 2B.

[0013] Program 110, as shown by generator 1 code representation 138 and generator N code representation 140, generator Parameterization 128 for the set of code 130. As used herein, generator "teeth, Code that generates the operator, or Performs an operation on one or more data buffers. Related Refers to a number.

[0014] Parameter definition 128 is The generator Operator based on code or function of To generate How Used Or the code is generator and how it is compiled into a composable primitive-level representation of the kernel114. generator In some examples, code 130 is written in a general-purpose programming language, such as Python. In some examples, parameter definition 128 is a set of kernel parameters. In some examples, parameter definition 128 comprises a parsable scripting language or the like that generates kernel parameters.

[0015] During the compilation phase 124, the generator 1 code representation 138 and the generator N code representation 140, etc. Writing a generator The code is translated from a general purpose programming language through a series of intermediate representations, such as generator 1 primitive level representation 116 and generator N primitive level representation 118, as shown by generator 1 intermediate representation 112 and generator N intermediate representation 120. generator During the compilation phase 124, generator The optimal configuration of is determined by parameter definition 128 and generator Based on the intermediate representation of Adaptive Compilation is determined by. Adaptive Compilation The results and other compilation metrics are available in subsequent Adaptive Compilation 104. Adaptive Compilation " refers to searching through a tree of possible configurations using a combination of static analysis of previous configurations and dynamic analysis of proposed configurations, performed during the elaboration process, as described in more detail with reference to Figure 2F.

[0016] generator The primitive-level buffer semantic representation of is combined into the primitive-level representation of kernel 114. The primitive-level representation of kernel 114 is stored for later use.

[0017] For execution, a kernel object 142 is created and included in the program object 132 . program During the execution phase 126, the kernel object 142 and the kernel generator Use the program Mu 1 32 in a process described in more detail with reference to FIG. program It is generated based on a set of logic and kernels. Mu 132 is executed during execution phase 126 and is a set of kernels to perform the computation. generator Use a set of

[0018] In some cases, the kernel and its generator The execution metrics are stored in an execution metrics 134 data store. The execution metrics 134 are used for subsequent executions to determine the optimal configuration. Adaptive Compilation Used by 104.

[0019] In some instances, Adaptive Compilation An artificial intelligence (AI) component 102 is used to assist in the search. The AI ​​component 102 comprises one or more machine learning models, as described in more detail with reference to FIGS. 5 and 6. For example, the AI ​​component 102 systematically evaluates various configurations and parameters during the search process to identify the most effective solution for kernel generation. It applies machine learning techniques to predict outcomes based on historical data, thereby reducing the required search space and computational resources. The involvement of the AI ​​component 102 improves the accuracy of the search, enabling a more targeted approach that can yield optimal or near-optimal configurations with greater speed and accuracy.

[0020] In some examples, the AI ​​component 102 also assists during the kernel creation phase 122, where the kernel is written within a software development environment (SDE), as described in more detail with reference to FIG. 4. For example, the AI ​​component 102 streamlines the kernel creation phase by providing intelligent code completion and advanced analysis within the SDE. The AI ​​component 102 analyzes the code being written in real time, provides recommendations for performance improvements, and identifies potential bottlenecks. The AI ​​component 102's deep understanding of the kernel architecture and its ability to learn from vast datasets of kernel performance metrics can guide developers toward best practices and more efficient design patterns. This proactive assistance not only enables kernels to be created with correctness in mind, but also enables the kernel to be optimized for the specific computational characteristics it will face in deployment.

[0021] In some instances, generator For example, broadcast generator , activation generator Some other lower level generator It is defined as a fusion of, and in some cases, Long Short-Term Memory (LSTM) generator At this level of abstraction, generator By writing a high level of optimization, shape generator Extraction of, and generator Gradient generation is simplified.

[0022] In some examples, kernel generators are used to generate implementations of existing operators in existing machine learning (ML) frameworks (e.g., TFLite, TF, ONNX, PyTorch, etc.) Operators in existing ML frameworks have attributes such as, but not limited to, broadcasting and type promotion support, expert-selected hand-written operators known to be important for particular classes of models, activation operators fused to element-wise operators such as "add", support for quantization algorithms that rely on architecture-specific DSP operations, layouts assumed by existing frameworks, and support for dynamic shapes and dynamic dtypes (data types).

[0023] In some instances, generator supports the following: Dynamic Shapes Broadcasting, type promotion: For example, "mul" is a binary operator, and the two operands can have different shapes and dtypes. ML frameworks often improve usability by providing implicit promotion to a common element type and support element broadcasting.

[0024] Layout munging: Some frameworks support multiple different layouts, e.g., row-major and column-major, tiled layouts, etc. If the input is in a different format, conversion may be required. Some libraries use strides to provide a common implementation that can work with many different layouts, but strides are not general for tiled layouts.

[0025] Type dispatch: Standard kernel libraries support multiple dtypes, but these are only known dynamically at kernel call time. This requires kernels to dynamically dispatch on dtypes, dispatching to many different dtype-specialized kernels. Some dtypes may have special cases; for example, "complex add" may be handled by the same code path as "scalar add" (because complex add is element-wise), but "complex mul" is a completely different algorithm from "scalar mul".

[0026] Thread Tiling: At a level outside of the type-specific kernel algorithm, the computation is divided into blocks that can be executed in parallel by multiple threads. The size of each subunit needs to be determined and is generally best evaluated based on the hardware characteristics and the size of the input data (not based on the number of available threads).

[0027] Cache tiling: For per-thread computation, computations are typically cache blocked, for example at the L2 level. The size of the L2 is target-specific. This is important for algorithms that make multiple passes over the data, and may be less important for element-wise operations with little reuse.

[0028] Tiled Algorithms: Within each L2 tile, there are many ways to implement the core algorithm, including using scalars, vectors, prefetching, etc. There are also special cases where broadcasting is handled inside the kernel, where it is interesting to handle, for example when the fastest varying dimension of one operand is broadcast.

[0029] Many microkernels: algorithms like matrix multiplication depend on lower level operations (operations) such as memset to clear a buffer, panel dot product, reduction, etc. These "microkernels" can themselves be implemented in many different ways.

[0030] Macro algorithms: Many operators have several completely different algorithms for computing their results, for example for convolution we see the im2col approach, direct convolution, Winograd, Matmul has many implementations (especially when quantization and accelerators force special data layouts), including Strassen's algorithm.

[0031] Hardware targets now frequently have spatial operations (like Apple AMX or Intel AMX) that can accelerate multiple loop nests at once, for example for matrix multiplication and large element-wise blocks. They also have many architecture families that require register blocking, pipelining, and unrolling in different ways.

[0032] In some examples, the primitive level representation of kernel 114 is a component of a framework consisting of a set of code-generated kernels that operate on memory buffers, such as, but not limited to, memory operators, 1D memory arrays, tensor buffers, user-defined data structures, etc. In some examples, the kernels directly use C / C++, assembly, and intrinsics for specific hardware capabilities.

[0033] In some examples, a library of kernel components is utilized to generate kernels, such as a buffer-level kernel generator that replaces legacy kernels. The kernel components are modular and reusable, and include core algorithms such as, but not limited to, memory fill, reduction, and element-wise operators, in addition to quantization kernels and more specialized primitives used in other domains.

[0034] In some instances, kernel generators are parametric kernel generators. It would be difficult for a human to manually create and maintain all permutations of kernels (e.g., for all dtypes, all target machines, etc.), and so metaprogramming is widely relied upon. This metaprogramming takes various forms, such as C macros and ifdefs, Python generator frameworks, "emitters" written in C++ against the "IRBuilder" compiler API, but the most widely used form is C++ templates.

[0035] In some examples, kernels are defined as declarative kernel generators that take kernel parameters, have arbitrary imperative logic coded against them, and have that imperative logic "burned into" the code generated for the kernel. This can be used to specialize on things like dtypes, unroll factors, vector lengths, cache sizes, etc. Most parameters should have integer types, be bounded by ranges (e.g., unroll <= 8 times), lists of valid values ​​(e.g., vector lengths = 2, 4, 8, 16, 32), and support enums to make them searchable (e.g., considering the dtype). Using kernel generators still allows the use of concrete kernels (e.g., fixed blobs of assembly) because they are valid generators with no parameters (or, equivalently, with fully constrained parameters).

[0036] Example code is shown below: Kernels may have parameters bound at their call site, e.g., after a dynamic switch on dtype, the next lower level microkernel is called with the dtype parameter bound to a constant value.

[0037]

number

[0038] In some instances, when a uniform representation is given for dynamic values ​​as well, the kernel generator provides a layering of value specialization when the attributes of input arguments are statically known to the kernel generator. For example, when generating a specialized version of a kernel for type f32, the "meta.buffer.dtype" and "scf.switch" operations can be constant propagated. If there is a reason, more sophisticated value propagation can be used to propagate the set (e.g., specializing to f32 and i8 and eliminating other dtypes).

[0039] In some instances, one aspect of parameterized generators is that they rely on types that are parameterized based on expressions derived from the generator parameters. This applies not only to types like "buffer", but also to things like SIMD vector lengths / dtypes and scalars with parametric types, as shown below:

[0040]

number

[0041] In some instances , Ji Generator is a part function and they are allowed to fail during generation time. This is simply Adaptive compilation search will be It simply removes candidates from the set of implementations. generator If there is no available implementation for, it will need to be solved at a higher level, for example by graph partitioning the accelerator vs. host computation.

[0042] In some examples, the parameter result is used to return a metaprogrammed value to the invoker. For example, the panel dot product microkernel is an element used in matrix multiplication implementations. Panel dot can be implemented in a target-specific manner using low-level vector register blocking, DSP instructions, and target-specific instructions, as shown by the following example:

[0043]

number

[0044] In the above example, the parameter result of the panelDotInner generator call is used lexically before the call itself. While the generated primitive-level representation of the kernel 114 is a traditional instruction program that is ultimately handed off to another system for code generation, the metaprogram is not. The metaprogram is interpreted by the compiler framework at kernel generation time and does not necessarily execute in the lexical order specified by the kernel. Instead, its location within the kernel is used to indicate where the generated code (or a call to it) should be inserted relative to other code in the kernel; it is the builder's insertion point.

[0045] This allows the order of evaluation of generators to be very flexible: if there is a valid topological order for the generator invocations, the generator is valid (and therefore cycles are invalid), for example:

[0046]

number

[0047] The final intermediate representation produced by @subKernel1 is executed before the final intermediate representation produced by @subKernel2, but the generator of @subKernel2 is executed before the generator of @subKernel1 due to its dependency on an intermediate parameter that needs to be computed.

[0048] Furthermore, in this example:

[0049]

number

[0050] Since there is no dependency between the two generators, the compiler can generate them in parallel. This structure (along with the general tree / forest / DAG structure of the computation) lends the compilation process for the kernel parallelism that can be exploited to speed up kernel generation on multi-core machines.

[0051] In some examples, kernels are defined using domain-specific and target-specific abstractions. One simple example is the "panel dot product" microkernel mentioned above. It is domain-specific (for matrix multiplication) and has many details specific to how it is used, and its implementations are often broadly target-specific; while a parametric intermediate representation generator can be used to generate them, it is also desirable to use implementations that use inline assembly and target-specific intrinsics. As mentioned above, ML operators are multi-level and have many complex implementations at many levels of abstraction.

[0052] In some instances, kernel authors declare their own abstractions, as in C++. To do so, the composable kernel generation system 200a provides for declaring interfaces to (micro)kernels and supports having many different implementations for each microkernel, each of which implements a common interface. Kernels may be recursively defined based on simpler, smaller kernels, which themselves can have multiple different implementations.

[0053] In some instances, interface declarations are implementation-independent, allowing clients to call them and type-check that the implementation follows the intended API. This not only provides type-checking, but also provides a framework within which the composable kernel generation system 200a can reason about many different implementations of the same algorithm (typically with different tradeoffs / constraints and architecture-specific).

[0054] In some examples, the composable kernel generation system 200a provides a natural way to abstract runtime interfaces and other concerns. For example, a "parallel for loop" kernel can be expressed and provided with implementations defined for different target runtimes (e.g., OpenMP instead of LLCL).

[0055] In some examples, each kernel, microkernel, Generators, functions , and there are multiple available implementations of the operator, and the composable kernel generation system 200a determines which one is best for a given target and scenario (dtype, size class, etc.). Thus, the (micro)kernel interface declaration is Adaptive CompilationDefine a cost model that is optimized by (e.g., find the implementation configuration with the "best achieved FLOPS"). For example, a microkernel implementation may include the use of scalar operations and may include implementations using multiple SIMD operators of different lengths, some implemented in inline assembly, and possibly some implemented using Apple AMX. The composable kernel generation system 200a selects the configuration for the implementation with the highest throughput by empirically measuring it for the current hardware (implementations for incompatible systems are ignored as infinite cost).

[0056] In some instances, Adaptive Compilation in a realistic way generator Use program This allows the composable kernel generation system 200a to collect execution metrics 134 data for appropriate tensor input sizes to measure (using realistic input dimensions instead of random input dimensions), similar to the mmperf "benchmark size" list. In some examples, profiles are collected and used, or depending on the goals of any particular product, "prioritize MLPerf performance" or "one size fits all." program To achieve a goal such as "generate the best possible code for...", certain dimensions are weighted more heavily.

[0057] In some cases, the top level generatorGiven the dimension weightings for a kernel, the composable kernel generation system 200a can propagate them down the tree of unfolding to the microkernels; for example, a microkernel that broadcasts tensor data to a buffer can be generated knowing all of the most frequent input dimensions from the kernels that use it. If the composable kernel generation system 200a chooses to output such a kernel out-of-line (versus inlining it to reduce code size), the composable kernel generation system 200a can aggregate the expected input dimensions from all the different kernels that call into it.

[0058] In some instances, parameters in parameter definition 128 are not specified. These parameters are Adaptive Compilation These low-level operators are explored and determined by the composable kernel generation system 200a during execution. For example, the composable kernel generation system 200a determines the number of iterations that fit into the cache, and the composable kernel generation system 200a returns that result as a parameter result, allowing the containing generator to tile or parallelize based on that. As another example, given an element-wise multiplication microkernel implemented on a vector over a 1D block of memory, a loop utilizing one of these low-level operators will increase FLOPS until it exceeds the L2 cache, at which point higher cache block algorithms are typically more efficient. Allowing a kernel to define a metric (e.g., FLOPS) can be a valuable tool for finding the correct implementation. Adaptive Compilation Allows the use of the top-level generator The kernel can use latency as its metric.

[0059] In some examples, some generator parameters (e.g., dtypes) are defined on the generator interface (and therefore common to all implementations), so the composable kernel generation system 200a may also have additional parameters. Niji Providing generator implementations (e.g., the same for different microarchitectures) generator (An ARM implementation of the kernel provides three implementations of the same microkernel.) This is just sugar to "flatten" these parameters as different individual implementations of the same microkernel.

[0060] In some instances, there are multiple implementations of each microkernel, which in turn are implemented based on other interfaces that may have many implementations. These deployments form a tree of possible deployments, with many top-level deployments within the framework. generator There are forests of unpackings that work at many levels of abstraction. For example, a matrix multiplication microkernel can be implemented with three levels of for loops, cache blocking, and internal L2 tiling. Also, 2D operators can be implemented to use target-specific dot product operations. or function and a common accelerator. Each of these may be implemented independently of the other, and all implement the same interface. Each "tree of deployment" is implemented using a single framework. generator We can have an exponential number of possible expansions for x, which makes it impractical to search the entire space for a single kernel, and even more difficult to support an entire ML framework, especially when a single framework may have hundreds / thousands of individual kernels.

[0061] In some examples, human-created constraints are defined on the kernel to cut off the search space or guide the search as fundamental bounds in parameter declarations. In some examples, conditional constraints are provided. In some examples, redundancy in tree-based structures is exploited by dynamic programming techniques. Dynamic programming uses subproblem storage / caching to algorithmically improve the performance of hierarchical tree-based algorithms. In some examples, each tree in the expansion has many common leaves, and the forest has many shared leaves, subtrees, and potentially the entire kernel. A cost model (top-level framework) generator By allowing the kernel to be defined at many levels (not just at the beginning), the composable kernel generation system 200a can exploit modularity for search and cache results. The use of dynamic programming collapses the "unfolding tree" into a directed acyclic graph (DAG).

[0062] In some examples, the cache is hosted on a cloud service and provides oracles to users so they can obtain searches offline. This allows users to avoid the full search algorithm on their device. In some examples, the composable kernel generation system 200a generates analytics for what the user is using the composable kernel generation system 200a for. In some examples, the installation size of a mobile framework can be very small, and instead of shipping a typical kernel library with many bloated kernels, the provider of the composable kernel generation system 200a ships a just-in-time (JIT) compiler that can generate kernels. Users may not want to perform searches on their devices, so the provider of the composable kernel generation system 200a can bundle a binary blob with the application or add logic to download kernel parameters appropriate for the target hardware, generate / cache machine code for the kernel at app install time, and utilize the compiler as a “compression scheme” to reduce the impact of the kernel library download size.

[0063] In some examples, the provider of the composable kernel generation system 200a takes the "most frequently used" results, compiles them into a binary blob, and ships it with the framework. This ensures that the most frequently used (e.g., all of BERT) always result in a cache hit. In some examples, offline exploration and using metrics to provide additional services to users is a generator It is extended to higher level problems such as fusion.

[0064] In some examples, each level of the kernel generator tree unfolding is functional (side-effect free), and the "key" used to look up the computation can be encoded in a way that the composable kernel generation system 200a can hash and look up (e.g., the key is a serialized MLIR blob), which is important for parallelizing tree compilation anyway (trees / DAGs have a lot of parallelism).

[0065] In some examples, kernel fusion is enabled, fusing arbitrary element-wise computations into matrix multiplications. The composable kernel generation system 200a supports this by allowing kernel generators to be parameterized by regions. Regions are simply different forms of parameter arguments, bodies of code passed, and metaprogramming constructs accessible. For example, by exposing regions as generic functions in the composable kernel generation system 200a, behaviors such as "switch based on dtype" and "statically unroll loops based on this parametric expression" can be defined in the system itself rather than hard-coded into the system. This allows the composable kernel generation system 200a to be user-extensible, since nothing in the stack is specialized for dense linear algebra; users can build their own libraries of generators that partition work against data tables or trees, interact with their own external storage (e.g., databases), and so on.

[0066] In some examples, parameterized generators also lead to a natural extension of the expressivity of ML operator graph abstractions. Instead of tfl.conv2d with an enum of activations, conv can take a region that performs element-wise computations on scalars, allowing arbitrary element-wise operators to be fused at the graph level. This allows the composable kernel generation system 200a to implement kernel fusion through graph rewriting that is subordinated to generators in a predictable way.

[0067] In some examples, the composable kernel generation system 200a utilizes algorithmic skeletons capable of describing high-order transformations that allow for encoding parallel patterns in a reusable manner, as the implementation task is simplified by the fact that each skeleton can be considered independently, as opposed to the monolithic programming interfaces of existing systems at a similar level of abstraction.

[0068] In some cases, the kernel generator performs partial translation from interface declarations to concrete implementations. function Constraints indicate restrictions on those parameters, such as "this implementation only works with dtype=float32", or "this only works on machines with the X86 VNNI extension", or "this works when based on the size modulo 128". In some cases, this is generator It is propagated upwards into the graph.

[0069] In some examples, the composable kernel generation system 200a can generate and capture large amounts of data and even have "importance weights" for the data. Given this data, an ML model is built for kernels that generalize from the data the composable kernel generation system 200a has seen and handle unknown situations that the composable kernel generation system 200a does not have. In some examples, the captured data is supplemented with randomly synthesized kernels (e.g., novel fusions) for directional learning. This allows the composable kernel generation system 200a to be highly efficient and excel at what it knows is important, while also generalizing to new hardware in an efficient manner.

[0070] In some examples, the composable kernel generation system 200a uses kernel descriptions in an intermediate representation format that is machine-parseable / translatable. In some examples, the composable kernel generation system 200a extracts shape operators for operators by extracting computations from the kernel description using code slicing. This ensures that the composable kernel generation system 200a has a single source of truth for the kernel + shape operators.

[0071] In some examples, the composable kernel generation system 200a statically derives the set of "ops+dtypes supported by this target" from the kernel library and encodes that data into tables used by the device graph partitioner. This maintains a single source of truth rather than redundantly encoding this in the graph partitioner. This allows users to incrementally implement several microkernels for new targets, generator The set can be gradually lit.

[0072] In some examples, the composable kernel generation system 200a may perform "invocation-independent computations," e.g., computations of known constants at the graph level. generator Detect lookup tables that depend only on attributes. This computation is "prepare-like" by computing the lookup table in a custom structure at initialization time, rather than computing it on every kernel invocation. generator can be automatically sliced ​​out from the main kernel computation.

[0073] In some examples, the composable kernel generation system 200a may use a multi-level intermediate representation (MLIR) compiler API to provide more complex structures than parameter-defined expansions. "Ji Implement the "Generator". generator are encoded as compiler transformations and provide a flexible programming model for users: they are generators that take a region of the intermediate representation as a parameter and produce a new one.

[0074] In some examples, the composable kernel generation system 200a automatically generates backward versions of the kernel. In some examples, the composable kernel generation system 200a extracts metadata about operations, such as whether they are associative, have side effects, etc.

[0075] In some examples, the composable kernel generation system 200a synthesizes versions of kernels for other considerations, such as code size. This can be achieved by using constant folding within the compiler. generator It may be useful for

[0076] In some examples, the composable kernel generation system 200a may: program kernel and generator For example, programIf only uses float32 or int8, the composable kernel generation system 200a removes all support for other dtypes, producing a much lighter-weight kernel library. This can be useful for deployment considerations and also to reduce instruction cache pressure (improving performance). 200a can also specialize when the shape is statically known.

[0077] In some examples, the composable kernel generation system 200a may generate complex framework-specific generator to a simpler framework-agnostic region parameter generator Normalize to.

[0078] In some examples, kernels generated by the composable kernel generation system 200a take output buffers as arguments that may not be exposed to the graph. The composable kernel generation system 200a provides a "buffer-exposing" graph-level representation that enables memory planning, in-place optimization for concatenation, and the like.

[0079] In some examples, the composable kernel generation system 200a may include a buffer level generator Get the implementation metadata and put it into generator Reflected at the graph level. In some examples, the composable kernel generation system 200a is target-agnostic and scales to CPUs and many accelerators. In some examples, the composable kernel generation system 200a is ML framework-agnostic, isolating all integration concerns and focusing only on kernel generation. In some examples, the composable kernel generation system 200a is not specialized to one memory layout or other limited set of assumptions. In some examples, the composable kernel generation system 200a is not specialized to ML or dense linear algebra and supports a wide range of data types and problem domains. For example, the composable kernel generation system 200a can be used to build high-performance audio kernels for processing audio signals or data kernels for use in database platforms.

[0080] In some examples, the composable kernel generation system 200a is extensible by users without access to the compiler's source code. In some examples, the composable kernel generation system 200a may be configured to generate any MLIR generator It employs a Python-like language, a user-extensible hybrid declarative / imperative programming language that allows graphs to be expressed in a usable way.

[0081] FIG. 2A is a block diagram of a composable kernel generation system 200a, according to some examples, and FIG. 2B is a process flow diagram of a kernel generation method 200b, according to some examples. The composable kernel generation system 200a , Ji Generé Ta The composable kernel generation system 200a also generates a program 232 in binary executable format (BEF) 234 based on the program 221 using the generated kernel 227 and the handwritten kernel 228. The composable kernel generation system 200a generates the kernel and its generatorusing a kernel generation method 200b to generate program To generate the program Use the compilation method 300b.

[0082] In some examples, a program 232 in binary executable format 233 is executed during runtime 229 on a set of hardware 230 devices to generate execution metrics 134 that are used to optimize the kernel in a process described in more detail with reference to FIG. 2D.

[0083] The kernel generation method 200b includes: generator While a particular sequence of operations is shown, the sequence may be modified without departing from the scope of this disclosure. For example, some of the operations shown may be performed in parallel, in a different sequence, or in different components of a composable kernel compilation system, and such implementations may be implemented in the refinement process. operation In other examples, different components of an example device or system implementing the composable kernel generation system 200a may perform operations substantially simultaneously or in a particular sequence.

[0084] In operation 202, the composable kernel generation system 200a generates a parameter definition 128 and generator for a set of code 130. In some examples, the code is written in a general-purpose programming language. generator Contains the code.

[0085] In operations 203 and 204, the composable kernel generation system 200a generates each generator , based on parameter definition 128 in a process that will be described in more detail with reference to FIG. 2D. generator During the process of determining the optimal configuration, the composable kernel generation system 200a generatorGenerate a primitive-level buffer semantic representation of

[0086] In operation 205, the composable kernel generation system 200a: generator We define the primitive-level buffer semantic representation of generator to the set of primitive-level buffer semantic representations. generator The set of primitive-level buffer semantic representations of is used to construct the primitive-level buffer semantic representation of the kernel.

[0087] In operation 206, the composable kernel generation system 200a: generator Based on a set of primitive-level buffer semantic representations, Power The composable kernel generation system 200a constructs a primitive-level buffer semantic representation of the kernel corresponding to the generator. For example, the composable kernel generation system 200a generator Obtain a set of primitive-level buffer semantic representations of generator Code slicing of primitive-level buffer semantic representations of and their dependencies into a single module or kernel.

[0088] In operation 207, the composable kernel generation system 200a decomposes a single module into an object (.o) file and stores the kernel's object file in a data store for the generated kernel 227. In some examples, the object file has the format of a standard C-style toolchain-generated object file, and therefore works seamlessly with a stack that implements a C / C++ external operator interface (FFI).

[0089] FIG. 2C shows, by way of example, generator Production method 200d Generator Creation FIG. 2D is a data flow diagram of data flow 200c. generatorFIG. 2E is a process flow diagram of production method 200d. generator Detailing method generator FIG. 2F is a data flow diagram of a refinement data flow 200e, according to some examples. generator 2 is a process flow diagram of a refining method 200f. generator Production method 200d and generator Using the refinement method 200f, generator Generate. generator Production method 200d and generator Although the refinement method 200f shows a particular sequence of operations, the sequence may be changed without departing from the scope of the present disclosure. For example, some of the operations shown may be performed in parallel, in a different sequence, or in different components of a composable kernel compilation system, and such implementations may be implemented in the refinement process. operation In other examples, different components of an exemplary device or implementation of the composable kernel generation system 200a may perform operations substantially simultaneously or in a particular sequence.

[0090] In operation 208, the composable kernel generation system 200a: kernel Parameter definitions 255 and generators 254 written in general-purpose programming languages Code Equipped with generator 236 or access to it.

[0091] In operation 209, the composable kernel generation system 200a: generator 254 codes in intermediate language generator For example, the composable kernel generation system 200a converts the , Ji Import the generator, generator 254 code based functions generator The graph compiler 225 generates an intermediate representation 234 of the generator The intermediate representation of 234 is generator In some examples, generator The intermediate representation of a function is in the library target format. In some cases, the intermediate representation of a function is program The calculation process of generator The composable kernel generation system 200a is in the form of a directed graph that represents the logic of generator By running a set of compiler passes on 254 pieces of code, generator 254 code, which includes: generator Parsing 254 codes, generator 254 code from a general-purpose programming language to an intermediate representation. In some instances, the intermediate representation may be a representation of the general-purpose programming language and an executable generator In some cases, the intermediate representation is a lower-level intermediate language between the object code of generator A graphical representation of the 254 code. In some examples: generator Converting 254 code involves substituting a general-purpose programming language into a lower-level intermediate language.

[0092] In operation 210, the composable kernel generation system 200a: kernel Parameter definition 255 and generator Based on the intermediate representation 234 of generator , and performs initial optimizations on the intermediate representation 234 of the function. For example, the composable kernel generation system 200a may use the graph compiler 225 to perform static analysis of the intermediate representation of the function and optimize it by unrolling loops, etc. generator The composable kernel generation system 200a determines a portion of the intermediate representation 234 of kernel Parameter definitions 255 are used to determine what types of optimizations can be performed, such as, but not limited to, the maximum number of loop iterations that can be unrolled.

[0093] In operation 211, the composable kernel generation system 200a uses the kernel compiler 226 to generate a kernel code in a process that will be described in more detail with reference to FIGS. 2E and 2F: kernel Parameter definition 255 and generator Based on the intermediate representation 234 of generator Based on the refinement of generator Determine the optimal configuration of

[0094] Based on the optimal configuration, the composable kernel generation system 200a generates a primitive-level buffer semantic representation 235 at operation 212. In some examples, the composable kernel generation system 200a caches the primitive-level buffer semantic representation 235 for later analysis.

[0095] In some examples, the composable kernel generation system 200a subclassifies primitive-level buffer semantic representations 235 into objects in binary executable format 237.

[0096] FIG. 2E shows, by way of example, generator Detailing method generator FIG. 2F shows a data flow diagram of the refined data flow, according to some examples. generator 1 is a process flow diagram of a refinement method. generator Using refinement method 200f, generator are evaluated by the composable kernel generation system 200a to determine the optimal configuration of generator Generate permutations of the configurations of . generator Although the refinement method 200f illustrates a particular sequence of operations, the sequence may be modified without departing from the scope of the present disclosure. For example, some of the operations illustrated may be performed in parallel, in a different sequence, or by different components of the composable kernel generation system 200a, and such implementations may be implemented in the refinement process. generatorIn other examples, different components of an exemplary device or implementation of the composable kernel generation system 200a may operate substantially simultaneously or in a particular sequence.

[0097] In operation 213, the composable kernel generation system 200a uses the kernel compiler 226 to search for an optimal configuration of the function 239 based on the evaluator associated with the generator 238. The kernel compiler 226: generator A static analysis search and a dynamic analysis search can be performed for the optimal configuration of the kernel. In a static analysis search, the kernel compiler 226 uses the kernel search 201 component to search several different types of data stores. One type of data store is a recently generated generator A kernel cache 224 containing Functions and The kernel cache 224 may be local 223 or may be distributed across remote storage nodes on one or more servers 222. For example, the composable kernel generation system 200a maintains a data store of optimal configurations in the kernel cache 224. The kernel exploration 201 component performs a kernel search based on an evaluator, which is a metric by which the composable kernel generation system 200a determines which implementation or configuration is optimal. generator Find the best configuration for you.

[0098] In operation 214, the composable kernel generation system 200a determines whether an optimal configuration was found during the search of the kernel cache 224. During static analysis exploration generator In response to determining that an optimal configuration of has not been found, the kernel compiler 226 generator To do so, in operation 215, the kernel compiler 226 performs a refinement phase 248 using dynamic analysis of generatorIntermediate representation of 234 and kernel Based on the parameter definition 255, a set of configurations such as configuration 0 241 and configuration N 250 is generated.

[0099] In operation 216, the composable kernel generation system 200a generates executable test sets based on one or more configurations. function Generate a set of In operation 217, in an evaluation phase 240, the composable kernel compilation system performs a test function , and determine a set of respective performance scores, as represented by performance score 0 242 for configuration 0 241 and performance score N 251 for configuration N 250. For example, the composable kernel compilation system may function Run the test function Test when it works against the test suite of data function In some instances, the performance score is function Test during initialization of function In some examples, the performance score may include an initialization score indicating the amount of time used by the test. function In some examples, the performance score includes an execution score indicating the amount of time it takes for the test function While the kernel is running, function This includes the amount of time spent communicating with the

[0100] In operation 218, in an aggregation phase 243, the kernel compiler 226 selects an optimal configuration from the set of configurations based on the respective sets of performance scores. function Assign a weight to each set of operator, configuration, and performance data, e.g., weight 0 245 assigned to set 0 244 of operator, configuration, and performance evaluation data; function, configurations, and weights N 253 assigned to performance evaluation data N 252. In a selection phase 246, the kernel compiler 226 selects an optimal configuration 247 configuration for the generator based on the set of functions, configurations, and performance evaluation data and their associated weights. Adaptive Compilation The optimal configuration for processing is cached in cache 249 .

[0101] In operation 219, the composable kernel generation system 200a generates a primitive-level buffer semantic representation of the function based on the optimal configuration. In some examples, generating the set of configurations is further based on parameter definitions of the target machine.

[0102] In some cases, the test function The set is run on multiple machines. In some cases, the test function The set is run on multiple machines. In some examples, the performance score includes a set of execution time and loading time.

[0103] In some cases, the test function Generating a set of libraries based on each configuration function Select a library and run tests based on the selected library and each configuration. function Testing a set of function This includes generating:

[0104] In some examples, the set of libraries includes a set of user-defined libraries and a set of system-defined libraries. In some examples, the generator is stored in an intermediate language.

[0105] In some instances, generatoris first defined in a programming language other than a general-purpose programming language and then subsumed into an intermediate language. In some examples, the optimal configuration is cached in a data store.

[0106] In some instances, the data store generator Searchable based on parameter definition and target machine parameter definition. In some instances, generator Determining the configuration of includes searching a data store based on kernel parameter definitions and the target machine configuration to find an optimal configuration.

[0107] In some examples, the data store is distributed across multiple storage nodes and the searching is performed on the distributed storage nodes. In some cases, during execution Generator Executed function The set of runtime performance data collected from the set of function is a known configuration and known generator Associated with a parameter definition.

[0108] In some examples, the performance data includes communication data of communications between a subset of operators that were executed. In some instances, generator Determining the configuration of the function determining a configuration using a machine learning model trained on a set of runtime performance data collected from a set of function is associated with a known configuration and known parameter definitions.

[0109] In some instances, generator Converting the code, determining the configuration, and generator Generating the kernel and configuring the kernel are performed on two or more machines.

[0110] In some examples, the kernel is stored in a data store that is accessible over a network. In some instances, kernels can be combined with other kernels in a kernel library.

[0111] FIG. 3A shows, by way of example, program During the compilation process program FIG. 3B shows a data flow diagram of the generated data flow 300a, according to some examples. program The composable kernel generation system 200a includes: program 3. Using the compilation method 300b, a modularized kernel is compiled in a binary executable format. program Generate. program Although the compilation method 300b is shown with a particular sequence of operations, the sequence may be changed without departing from the scope of the present example. For example, some of the operations shown may be performed in parallel or out of process. operation In other examples, program The different components of the composable kernel generation system 200a that implement the compilation method 300b may be compiled substantially simultaneously or in a particular order. operation may be executed.

[0112] In operation 302, the composable kernel generation system 200a receives an input program 314 of a computation written in a high-level language. For example, program is written in a general-purpose language, and each function is generator It can be composed of a set of functions including a set of

[0113] In operation 304, the composable kernel generation system 200a: programinto an intermediate representation in a first intermediate language (326), where the intermediate representation 334 has a set of input variables (represented by Input1 318, Input2 316, and InputN 320), a set of output variables (represented by Output 328), and a set of input values ​​corresponding to the set of input variables to a set of output values ​​corresponding to the set of output variables. generator (represented by generator 1 338, generator 2 322, and generator N 324). For example, the compilation system may compile program 314 as program into a graphical intermediate representation 334 of program Each expression in program as a set of nodes and vertices of a subgraph of the graphical intermediate representation 334 of program has a representation in intermediate representation 334 of

[0114] In operation 306, the composable kernel generation system 200a selects a kernel based on the selection criteria. generator From the set of generator The first set of generator For example, the compilation system may include a kernel library 336 of kernels, each kernel being a representation of the intermediate representation 334. generator may or may not support a subset of generator Contains generator In addition, the compilation system generator and intermediate representation 334 generator corresponds to a set of generator and fallbacks including generator The compilation system includes a fallback library 220 for the intermediate representation 334. generator Compile the system kernel library 336 generator Based on matching with the intermediate representation 334 generator A subset of generator The first set is separated into two sets of generator of intermediate representations 334 that do not belong to the first set of generator teeth, generator are assigned to the second set of

[0115] In operation 308, the composable kernel generation system 200a: generator Generates a set of optimized intermediate representations of generator teeth, generator This is based on the first set of

[0116] In operation 310, the composable kernel generation system 200a: generator Second set and fallback generator Based on the library of generator Generate a set of unoptimized intermediate representations of

[0117] For example, the compilation system may generator Based on generator In the first set of generator Select the second set of generator Within generator in the fallback library 220 generator In some cases, compilation systems are built in the open and new generator can be added to the kernel library 336, but any generator is found in the fallback library 220, so program All possible required by generator In some instances, the intermediate representation 334 does not need to have generator In the first set of generator teeth, generator are candidates for advanced optimization of the intermediate representation 334 using various methods, including but not limited to fusion, while generator is not subject to advanced optimization.

[0118] In operation 312, the composable kernel generation system 200a generates an optimized generator set and not optimized generator Based on the set of program Generate a primitive level representation of (332). For example, generator are sub-classified from a graph-level value-semantic intermediate representation 334 to a primitive-level buffer-semantic representation. In some examples, this transition makes lower-level optimizations such as memory planning easier. In some examples, part of the transition is to split the executable program 330 into an initialization phase and an execution phase, putting computationally expensive setup steps into the initialization phase, thereby keeping the execution phase computationally lightweight. This is program When you run a program, the initialization phase is usually run once and the execution phase is run many times.

[0119] The primitive-level representation is then encoded into a Binary Executable Format (BEF) file, an efficiently mappable binary format for execution in the graph runtime. From there, the runtime client loads the appropriate BEF and discovers the correct entry point ( program has a name and can be looked up), with the appropriate input program In some instances, the execution is fully asynchronous and multiple executions can be pipelined in a straightforward manner.

[0120] In some cases, intermediate generator The first set of expressions is generator Each of the first set of generator to each generator Between generator To convert the set of kernels into a representation Adaptive Compilation Based on intermediate generator determining a configuration of a set of representations, where each kernel in the set of kernels is an intermediate generator The second set of representations is composed of intermediate generator Determining the composition of a set of representations, intermediate generatorIt is generated by generating a set of optimized primitive-level buffer semantic representations based on the composition of the set of representations. generator To convert generator This involves decomposing the code into a lower level intermediate language.

[0121] In some examples, all or part of the composable kernel generation system 200a executes as part of a JIT process on a local device. For example, when a program requiring a particular kernel is executed, the JIT compiler is invoked. Unlike traditional ahead-of-time (AOT) compilation, in which all code is compiled before execution, JIT compilation defers the compilation of kernels until they are needed at runtime.

[0122] During the runtime analysis phase, the JIT compiler monitors the execution of the program to determine which kernels are frequently used or which are performance-critical. This runtime analysis helps the JIT compiler prioritize which kernels to compile and optimize first.

[0123] In the intermediate representation generation phase, for kernels identified for JIT compilation, the source code or bytecode is converted into IR, which is lower-level, platform-independent code that is easier for the JIT compiler to analyze and optimize.

[0124] In the on-demand compilation phase, the JIT compiler compiles the kernel's IR into native machine code on-demand, just before the kernel is executed for the first time. This step is performed at runtime, hence the term "Just In Time."

[0125] In the optimization phase, the JIT compiler applies various optimization techniques to the IR or directly to the machine code to improve performance. These optimizations may include inlining, loop unrolling, dead code elimination, and others informed by the runtime behavior of the program. In some examples, as described in more detail with reference to FIGS. 5 and 6, the AI ​​component 102 (of FIG. 1) using a model trained on execution metrics 134 (of FIG. 1) is used to optimize the compilation process. The JIT compiler can use feedback from the execution metrics to learn and improve its compilation and optimization strategies over time.

[0126] In the caching phase, once a kernel has been compiled to native code, it is cached in memory. Subsequent calls to the same kernel can use the cached version, eliminating the need to recompile the kernel each time it is called. As the program continues to run, the JIT compiler can collect more performance data and recompile and reoptimize the kernel to fit changing usage patterns or data sets. When a particular kernel is no longer in use or the system needs to free up resources, the JIT compiler can remove the compiled code from the cache; this process can be part of the system's garbage collection routine.

[0127] Using a JIT compilation process with the kernel allows for several advantages. Improved performance: JIT compilers are able to optimize kernels for the specific hardware they are running on, which can result in better performance compared to AOT compiled code.

[0128] · Flexibility: JIT compilation allows code to run on different hardware without the need for recompilation, since compilation is done on the target machine. Efficiency: Only kernels that are actually used are compiled, which can save time and resources compared to compiling all kernels upfront.

[0129] · Adaptability: The JIT process can adapt its optimizations based on actual data and workload, which can result in more efficient execution than static AOT optimizations.

[0130] 4 is a block diagram of an authoring system according to some examples. A software developer uses a kernel authoring system 400 to create a kernel that is used by the composable kernel generation system 200a. Ruj Describe the generator.

[0131] The kernel creation system 400 includes a software development environment 404 and a set of libraries 412. The software development environment 404 , Ji Generator 418 and related generator Used to edit kernel The software development environment 404 includes an editor 402. P The software development environment 404 further includes an interface 410 to L, a system constraints 406 component that provides information about system limitations of the system running the kernel, and a set of programming aids such as a debugger 408. generator Set 416 and Fallback Generator A set 412 of libraries containing a set 414 is accessed.

[0132] In some examples, the kernel creation system 400 may use value semantics generator including programming assistance that allows deriving shape operators by slicing the description; generatorA backward version of can often be generated from a buffer-level abstraction. For example, generator Many other simpler tables are also useful, such as determining whether a has side effects, what dtypes it supports, etc.

[0133] In some examples, the kernel creation system 400 may use formal methods to generator provides tools that allow comparing the equivalence between multiple implementations of In some instances , Ji The generator is an explicit Adaptive Compilation It is based on a declarative model with possible metaprogramming capabilities, supports low-level semantics, typing, and is written in a Python dialect that allows the development of additional functionality.

[0134] Machine Learning Pipeline 6 is a flowchart illustrating a machine learning pipeline 600, according to some examples, which can be used to generate a trained machine learning model 602, such as a machine learning model used by the AI ​​component 102 of FIG. 1 to perform kernel search and compiler optimization.

[0135] overview Generally, machine learning can involve the use of computer algorithms to automatically learn patterns and relationships in data, potentially without the need for explicit programming. Machine learning algorithms can be divided into three main categories: supervised learning, unsupervised learning, and reinforcement learning.

[0136] Supervised learning involves training a model using labeled data to predict outputs for new, unknown inputs. Examples of supervised learning algorithms include linear regression, decision trees, and neural networks.

[0137] Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships within the data. Examples of unsupervised learning algorithms include clustering, principal component analysis, and generative models such as autoencoders.

[0138] Reinforcement learning involves training a model to make decisions in a dynamic environment by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms include Q-learning and policy gradient methods.

[0139] According to some examples, examples of specific machine learning algorithms that may be deployed include logistic regression, a type of supervised learning algorithm used for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another exemplary type of machine learning algorithm is naive Bayes, another supervised learning algorithm used for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that predictor variables are independent of each other. Random forests are another type of supervised learning algorithm used for classification, regression, and other tasks. Random forests build a collection of decision trees and combine their outputs to make predictions. Further examples include neural networks composed of interconnected layers of nodes (or neurons) that process information and make predictions based on input data. Matrix factorization is another type of machine learning algorithm used for recommendation systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to reveal hidden patterns or relationships in the data. Support vector machines (SVMs) are a type of supervised learning algorithm used for classification, regression, and other tasks. SVM finds a hyperplane that separates different classes in the data. Other types of machine learning algorithms include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer models. The choice of algorithm depends on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0140] The performance of a machine learning model is typically evaluated against a separate test set of data that was not used during training to ensure that the model can generalize to new, unseen data.

[0141] Although some specific examples of machine learning algorithms are described herein, the principles described herein may be applied to other machine learning algorithms. Deep learning algorithms such as convolutional neural networks, recurrent neural networks, and transformers, as well as more traditional machine learning algorithms such as decision trees, random forests, and gradient boosting, may be used in various machine learning applications.

[0142] Three exemplary types of problems in machine learning are classification problems, regression problems, and generative problems. Classification problems, also called categorization problems, aim to classify items into one of several categorical values ​​(e.g., is this object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing a value that is a real number). Generative algorithms aim to generate new examples similar to the examples provided for training. For example, text generation algorithms are trained on many text documents and configured to generate new, coherent text with similar statistical properties as the training data.

[0143] Training Phase Generating the trained machine learning model 602 may include multiple phases that form part of a machine learning pipeline 600, including, for example, the following phases shown in FIG. 5:

[0144] Data Collection and Preprocessing 502: This phase may involve acquiring and cleaning data to ensure it is suitable for use in machine learning models. This phase may also include removing duplicates, handling missing values, and converting the data into an appropriate format.

[0145] Feature Engineering 504: This phase may involve selecting and transforming training data 606 to create features useful for predicting a target variable. Feature engineering may involve (1) receiving features 608 (e.g., as structured or labeled data in supervised learning) and / or (2) identifying features 608 in the training data 606 (e.g., as unstructured or unlabeled data for unsupervised learning).

[0146] Model Selection and Training 506: This phase may include selecting an appropriate machine learning algorithm and training it on the preprocessed data. This phase may further include splitting the data into a training set and a test set, evaluating the model using cross-validation, and tuning hyperparameters to improve performance.

[0147] Model Evaluation 508: This phase may involve evaluating the performance of the trained model (e.g., trained machine learning model 602) on a separate test dataset. This phase may help determine whether the model is overfitted or underfitted and whether the model is suitable for deployment.

[0148] Prediction 510: This phase involves using the trained model (e.g., trained machine learning model 602) to generate predictions for new, unseen data. Validation, refinement or retraining 512: This phase may involve updating the model based on feedback generated from the prediction phase, such as new data or user feedback.

[0149] Deployment 514: This phase may include integrating the trained model (e.g., Trained Machine Learning Model 602) into a broader system or application, such as a web service, a mobile app, or an IoT device. This phase may include setting up APIs, building user interfaces, and ensuring that the model is scalable and can handle large amounts of data.

[0150] FIG. 6 shows further details of two exemplary phases: training phase 604 (e.g., part of model selection and training 506) and prediction phase 610 (part of prediction 510). Prior to training phase 604, feature engineering 504 is used to identify features 608. This may include identifying informative, discriminative, and independent features to effectively operate the trained machine learning model 602 in pattern recognition, classification, and regression. In some examples, training data 606 includes labeled data for the pre-identified features 608 and one or more outcomes. Each of the features 608 may be a variable or attribute, such as a separately measurable characteristic of a process, article, system, or phenomenon represented by the dataset (e.g., training data 606). The features 608 may be of different types, for example, numerical features, strings, and graphs, and may include, by way of example only, one or more of content 612, concepts 614, attributes 616, historical data 618, and / or user data 620.

[0151] In the training phase 604 , the machine learning pipeline 600 uses training data 606 to find correlations between features 608 that influence predicted outcomes or prediction / inference data 622 .

[0152] Using the training data 606 and the identified features 608, the trained machine learning model 602 is trained in a training phase 604 during machine learning program training 624. The machine learning program training 624 evaluates the values ​​of the features 608 so that they correlate with the training data 606. The result of the training is a trained machine learning model 602 (e.g., a trained or learned model).

[0153] Additionally, the training phase 604 may include machine learning, where the training data 606 is structured (e.g., labeled during a preprocessing operation). The trained machine learning model 602 implements, for example, a neural network 626 capable of performing classification and clustering operations. In other examples, the training phase 604 may include deep learning, where the training data 606 is unstructured, and the trained machine learning model 602 implements a deep neural network 626 capable of performing both feature extraction and classification / clustering operations.

[0154] In some examples, the neural network 626 may be implemented within the trained machine learning model 602 created during the training phase 604. The neural network 626 includes a hierarchical (e.g., layered) organization of neurons, with each layer composed of multiple neurons or nodes. Neurons in the input layer receive input data, and neurons in the output layer generate the final output of the network. Between the input and output layers, there may be one or more hidden layers, each composed of multiple neurons.

[0155] Each neuron in the neural network 626 operationally computes a function, such as an activation function, that receives as input a weighted sum of the outputs of neurons in the previous layer and a bias term. The output of this function is passed as input to a neuron in the next layer. If the output of the activation function exceeds a certain threshold, the output is transmitted from that neuron (e.g., a sending neuron) to a connected neuron (e.g., a receiving neuron) in the subsequent layer. Connections between neurons have associated weights that define the influence of the input from the sending neuron to the receiving neuron. During the training phase, these weights are adjusted by a learning algorithm to optimize the network's performance. Different types of neural networks use different activation functions and learning algorithms, which can affect their performance for different tasks. The layered organization of neurons and the use of activation functions and weights enable neural networks to model complex relationships between inputs and outputs and generalize to new inputs not seen during training.

[0156] In some examples, the neural network 626 may be one of several different types of neural networks, such as, by way of example only, a single-layer feedforward network, a multi-layer perceptron (MLP), an artificial neural network (ANN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a bidirectional neural network, a symmetrically connected neural network, a deep belief network (DBN), a convolutional neural network (CNN), a generative adversarial network (GAN), an autoencoder neural network (AE), a restricted Boltzmann machine (RBM), a Hopfield network, a self-organizing map (SOM), a radial basis function network (RBFN), a spiking neural network (SNN), a liquid state machine (LSM), an echo state network (ESN), a neural Turing machine (NTM), or a transformer network.

[0157] In addition to the training phase 604, a validation phase may be performed on a separate data set known as the validation data set. The validation data set is used to tune the hyperparameters of the model, such as the learning rate and regularization parameters. The hyperparameters are tuned to improve the performance of the model on the validation data set.

[0158] Once a model is fully trained and validated, in a testing phase, the model can be tested against a new dataset. The test dataset is used to evaluate the model's performance and ensure that the model is not overfitting to the training data.

[0159] During the prediction phase 610, the trained machine learning model 602 uses the features 608 to analyze the query data 628 to generate inferences, results, or predictions, as examples of predicted / inferred data 622. For example, during the prediction phase 610, the trained machine learning model 602 generates an output. The query data 628 is provided as an input to the trained machine learning model 602, and the trained machine learning model 602 generates the predicted / inferred data 622 as an output in response to receiving the query data 628.

[0160] In some examples, the types of training data included in the execution metrics 134 (of FIG. 2A) collected by the composable kernel generation system 200a (of FIG. 2A) during runtime 229 (of FIG. 2A) to train the trained machine learning model 602 of the AI ​​component 102 include, but are not limited to:

[0161] Execution metrics (execution phase data): · Performance scores for different kernel configurations, including execution and load times. Data about the efficiency of kernel execution on various hardware devices.

[0162] Metrics related to the computational resources consumed by the kernel, such as CPU usage, memory usage, and I / O operations. Kernel generation data (compilation phase data): · Historical data on the success rates of different kernel configurations.

[0163] Information about parameter definition choices made during kernel generation and their results. · Data regarding the frequency and usage of various kernel parameters and configurations. Kernel Build Data (Kernel Build Phase Data): · Code structure and syntax patterns that result in more efficient kernels.

[0164] Common errors or inefficiencies in the kernel code that the AI ​​component can learn to identify and correct. User interaction with the SDE, such as the use of specific tools or features that aid in kernel creation.

[0165] Exploration Data (Exploration Phase Data): · The results of the search for the optimal configuration, including the paths taken through the search space and the effectiveness of different search strategies.

[0166] · The impact of AI-assisted search on the quality and performance of the resulting kernels. Training and prediction data (machine learning program training data): Features extracted from kernels and their performance metrics related to training AI models.

[0167] Historical data on kernel performance that can be used to train predictive models within the AI ​​component. Validation and refinement data from the iterative training process to improve the accuracy of AI models.

[0168] User data (software development environment data): Feedback from developers on suggestions provided by the AI ​​component. · Software development environment usage patterns that can inform AI component recommendations.

[0169] Deployment Data: Information about how the kernel performs in production, which can be used to further improve the AI ​​model.

[0170] By collecting and analyzing these types of execution data, the composable kernel generation system 200a can train the trained machine learning models 602 used within the AI ​​component 102 to better support the exploration and kernel creation phase, ultimately resulting in more efficient and effective kernel generation and deployment.

[0171] In some examples, the composable kernel generation system 200a collects kernel compilation data and generation data collected during the compilation phase and uses the collected compilation data and generation data to train the trained machine learning model 602 used in the AI ​​component 102, including but not limited to:

[0172] Compile time metrics: · Duration of the compilation process for each kernel or set of kernels. Resources used during compilation, such as CPU and memory usage.

[0173] Intermediate representation data: · Properties of the intermediate representation generated during the decomposition of high-level code into machine code. · Transformations applied to the code during the compilation stage and their impact on performance.

[0174] Configuration and parameter definition data: · The specific parameter values ​​chosen for kernel generation and their impact on the performance of the compiled kernel.

[0175] · The frequency and effectiveness of different parameter combinations used during kernel generation. Optimization results: The success rate of various optimization techniques applied during compilation, such as loop unrolling, vectorization, and inlining.

[0176] - Performance improvements achieved through specific optimizations. Error and Warning Log: · Compilation errors and warnings that occur, which can be used to identify common problems and improve the robustness of the compilation process.

[0177] Search Algorithm Data: · The path taken through the search space when determining the optimal configuration. · The effectiveness of different search strategies and heuristics used by the AI ​​component.

[0178] Code generation patterns: · Common patterns or idioms in the generated code that correlate with higher performance or efficiency.

[0179] Variations of the assembly or machine code generated for different target architectures. Runtime performance feedback: Data about how well the kernel performs after being deployed, which can be used to retroactively adjust compilation strategies.

[0180] Developer Interaction Data: · Inputs and adjustments made by developers during the kernel creation phase that affect the compilation result.

[0181] · Usage patterns of compilation flags and directives provided by the developer. By collecting and analyzing this kernel generation data, the AI ​​component can learn to predict the most effective compilation strategies for different scenarios, resulting in more efficient kernel generation and potentially reducing the time and resources required for the compilation phase. This data-driven approach can significantly improve the AI ​​component's capabilities in assisting with kernel generation and optimization.

[0182] In some examples, the trained machine learning model 602 may be a generative AI model, a term that may refer to any type of artificial intelligence that can create new content from training data 606. For example, a generative AI may generate text, images, video, audio, code, or synthetic data that is similar, but not identical, to the original data.

[0183] Some of the techniques that can be used in generative AI are: Convolutional Neural Networks (CNNs): CNNs can be used for image recognition and computer vision tasks. CNNs can be designed to extract features from images, for example, by scanning the input image and using filters or kernels that highlight important patterns.

[0184] Recurrent Neural Networks (RNNs): RNNs can be used to process sequential data, such as speech, text, and time-series data. RNNs use feedback loops that allow them to capture time dependencies and remember past inputs.

[0185] Generative Adversarial Networks (GANs): GANs can include two neural networks: a generator and a discriminator. The generator network attempts to create realistic content that can "fool" the discriminator network, while the discriminator network attempts to distinguish between real and fake content. The generator and discriminator networks compete with each other and improve over time.

[0186] Variational Autoencoders (VAEs): VAEs can encode input data into a latent space (e.g., a compressed representation) and then decode it into output data. The latent space can be manipulated to generate new variations of the output data. VAEs can process input data using a self-attention mechanism, allowing them to handle long text sequences and capture complex dependencies.

[0187] Transformer models: Transformer models may use attention mechanisms to learn relationships between different parts of input data (such as words or pixels) and generate output data based on these relationships. Transformer models can handle sequential data, such as text or speech, as well as non-sequential data, such as images or code.

[0188] In a generative AI example, query data 628 may include text, audio, images, video, numerical, or media content prompts, and output prediction / inference data 622 includes text, images, video, audio, code, or synthetic data.

[0189] In some examples, the training phase 604 and the prediction phase 610 are performed on a distributed system, such as the composable kernel generation system 200a of FIG. 2A. In some examples, one or more of the operations of the training phase 604 and prediction phase 610 are performed on a local device as part of a JIT compilation process, as described in more detail with reference to Figures 2E and 2F.

[0190] 7 is a diagrammatic representation of a machine 700 upon which instructions 710 (e.g., software, programs, applications, applets, apps, or other executable code) may be executed to cause the machine 700 to perform any one or more of the methods described herein. For example, the instructions 710 may cause the machine 700 to perform any one or more of the methods or processes described herein. The instructions 710 transform a general-purpose, unprogrammed machine 700 into a specific machine 700 that is programmed to perform the described and illustrated operators in the described manner. The machine 700 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 700 may operate as a server machine or a client machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 700, in combination with other components of a compiler system, may operate as, but is not limited to, a server, a client, a computer, a personal computer (PC), a tablet computer, a laptop computer, or any machine capable of sequentially or otherwise executing instructions 710 that specify actions to be taken by machine 700. Furthermore, although a single machine 700 is shown, the term "machine" may be interpreted to include a collection of multiple machines that individually or collectively execute instructions 710 to perform any one or more of the methodologies described herein.

[0191] Machine 700 may include one or more processors 702, memory 704, and I / O device interface 706, which may be configured to communicate with each other via bus 732. In one example, processor 702 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processor, or any suitable combination thereof) may include, for example, processor 708 and processor 712 that execute instructions 710. The term "processor" is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as "cores") capable of simultaneously executing instructions. While FIG. 7 shows multiple processors 702, machine 700 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0192] Memory 704 includes main memory 714, static memory 716, and storage unit 718, all accessible to processor 702 via bus 732. Main memory 704, static memory 716, and storage unit 718 store instructions 710 that embody any one or more of the methods or operators described herein. The instructions 710 may reside, completely or partially, within main memory 714, within static memory 716, within a non-transitory machine-readable medium 720 in storage unit 718, within one or more of processors 702 (e.g., within a processor's cache memory), or any suitable combination thereof during their execution by machine 700.

[0193] The I / O device interface 706 connects the machine 700 to I / O devices 734. One or more of the I / O devices 734 may be components of the machine 700 or may be separate devices. The I / O device interface 706 may include a wide variety of interfaces to the I / O devices 734 used by the machine 700 to receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The specific I / O device interface 706 included in a particular machine will depend on the type of machine. It will be understood that the I / O device interface 706 of an I / O device 734 may include many other components not shown in FIG. 7 . In various examples, the I / O device interface 706 may include an output component interface 724 and an input component interface 728. Output component interface 724 may include interfaces to visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistive mechanisms), other signal generators, etc. Input component interface 728 may include interfaces to alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or another pointing device), tactile input components (e.g., physical buttons, touchscreens that provide the position and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0194] Communications may be implemented using a wide variety of technologies. I / O device interface 706 further includes a communication component interface 730 operable to connect machine 700 to network 722 or one or more devices 736 via couplings 726 and 738, respectively. For example, communication component interface 730 may include a network interface component for interfacing with network 722 or an interface to other suitable devices. In further examples, communication component interface 730 may include interfaces to wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components for communicating via other modalities. Device 736 may be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0195] Various memories (e.g., memory 704, main memory 714, static memory 716, and / or memory of processor 702) and / or storage unit 718 may store one or more sets of instructions and data structures (e.g., software) that embody or are used by any one or more of the methods or operators described herein. These instructions (e.g., instructions 710), when executed by processor 702, cause various operations to be performed to implement the disclosed examples.

[0196] The instructions 710 may be transmitted or received over the network 722 using a transmission medium via a network interface device (e.g., a network interface component included in the communications component interface 730) and using any one of several well-known transfer protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 710 may be transmitted or received using a transmission medium via a coupling 738 (e.g., a peer-to-peer coupling) to the device 736.

[0197] The described implementations of the present subject matter can include one or more of the features, alone or in combination, as illustrated below, by way of example. FIG. 8 is a deployment diagram for a composable kernel generation and deployment system, according to some examples.

[0198] In some examples, the composable kernel generation and deployment system includes a composable kernel generation system server 802 that acts as a processing and management unit for kernels and objects in a binary executable format generation task. This server is designed to handle the computations and data processing required to create and optimize kernels, which are components of software applications that perform specific operations or calculations.

[0199] The system also includes an array of local devices that interact with the composable kernel generation system server 802. These local devices can vary greatly in their capabilities and purposes. For example, the local server 810 can act as an intermediary, providing additional computational resources or serving as a relay point for distributing kernels to other devices in the network. A standard computer 804, such as a desktop or laptop, can be used by developers to interface with the kernel generation system server 802 to write and test code and to compile and retrieve optimized kernels.

[0200] Wireless devices 808, which may range from smartphones to tablets, can utilize kernels generated by server 802 for a variety of applications requiring on-the-fly computation and benefit from a JIT compilation process that optimizes kernel performance based on the device's specific hardware characteristics.

[0201] Embedded computing systems 806 represent specialized hardware that often requires highly optimized kernels due to processing power or memory constraints. These systems can include IoT devices, automotive control systems, or industrial machine controllers, all of which may rely on the composable kernel generation system server 802 to provide efficient kernels tailored to their unique operating requirements.

[0202] A communications network 834 connects the composable kernel generation system server 802 with these various local devices. The composable kernel generation system server 802 facilitates the transfer of data, code, and instructions required for the kernel generation and deployment process. The communications network 834 can be comprised of a variety of technologies, including wired and wireless connections, and can span local and wide area networks to ensure seamless interaction between the composable kernel generation system server 802 and local devices, regardless of their geographic distribution.

[0203] Through this interconnected system, the composable kernel generation system server 802 can efficiently distribute workloads, manage kernel versions, and provide updates or optimizations to deployed kernels, ensuring that each device runs the most effective code for its particular use case. This architecture not only maximizes the performance of individual devices, but also increases the overall efficiency and adaptability of the kernel generation and deployment process across the ecosystem.

[0204] In some examples, the process of creating and deploying a kernel is divided into phases managed by a composable kernel generation system hosted on the composable kernel generation system server 802. During the generation phase, the composable kernel generation system is tasked with building executable objects in a binary executable format, which are the final executable form of the kernel. These executable objects are generated after a series of steps that may include receiving a kernel specification, performing optimizations, and compiling the kernel into a binary format suitable for execution on various hardware platforms.

[0205] In some examples, the generation phase involves not only translating high-level kernel code into machine-level instructions, but also applying various optimization techniques. These optimizations are tailored to improve performance, reduce resource consumption, and ensure compatibility with the target execution environment. The composable kernel generation system server 802 leverages its computational resources to efficiently handle these tasks and generate optimized kernels and executable objects that are ready for deployment.

[0206] Once the generation phase is complete, the system transitions to a runtime phase in which the executable objects are deployed to one or more local devices, which may include, but are not limited to, a computer 804, a wireless device 808, an embedded computing system 806, and a local server 810. Deployment involves transferring the executable objects from the composable kernel generation system server 802 to the local devices over a communications network 834, which may be comprised of a variety of network types and configurations.

[0207] During the runtime phase, local devices execute executable objects composed of kernels that were deployed as part of their software applications. Kernels perform the specific operations for which they were designed, such as data processing, numerical computation, or any other specialized task. In some examples, the deployment process is designed to be seamless and efficient, providing that local devices receive the correct version of the kernel that is compatible with their hardware and software environment.

[0208] In some examples, the composable kernel generation system server 802 may provide continuous support during the runtime phase, such as monitoring kernel performance, collecting execution metrics, and potentially delivering updates or further optimizations to the kernel as needed. This continuous support ensures that kernels remain efficient and effective throughout their operational lifespan, providing local devices with the computing power needed for their respective applications.

[0209] In some examples, executable objects deployed to local devices are not only self-contained execution units, but also include built-in analysis collection and transmission components. These analysis components serve two purposes: they collect data about how kernels behave under actual runtime conditions on the local device and ensure the safe and efficient transmission of these execution metrics to the composable kernel generation system server 802. These execution metrics provide insight into the real-world performance of kernels and enable a detailed understanding of their efficiency and stability across different devices and operating conditions.

[0210] For example, for local devices, the analysis component may track how quickly a kernel processes data, how much CPU time the kernel consumes, and whether the kernel operates within an expected memory footprint. On wireless devices, additional metrics such as battery consumption or network usage while the kernel is running may be particularly relevant. For embedded computing systems, where resources are often constrained, analysis may focus on real-time performance and reliability.

[0211] Once collected, these execution metrics are sent back to the composable kernel generation system server 802 via a communications network 834. This communication can be facilitated over a variety of protocols and network types, ensuring compatibility and security. The composable kernel generation system server 802 then uses this data to evaluate the kernel's performance and inform future optimizations.

[0212] The feedback loop generated by this process is extremely valuable for continuous improvement. By analyzing execution metrics, the AI ​​components within the composable kernel generation system can learn and adapt, refining the kernel generation process to generate ever more efficient and effective kernels. This continuous cycle of deployment, data collection, analysis, and refinement helps evolve the kernel generation system, facilitating its responsiveness to the changing demands of local devices and the environments in which they operate.

[0213] In some examples, the local device is equipped with the ability to perform kernel generation operations on the fly using a JIT compilation process that enables the local device to dynamically generate executable objects consisting of optimized executable versions of kernels at the point in time needed during program execution, rather than relying solely on pre-compiled kernels provided by the composable kernel generation system server 802.

[0214] By allowing local devices to generate their own executable objects using JIT compilation, the composable kernel generation and deployment system 832 enables a high degree of flexibility and responsiveness. Local devices can optimize the kernel for their current workload and operating conditions, potentially achieving better performance than if they were using pre-compiled executable objects. Furthermore, this approach can reduce the need for frequent communication with the composable kernel generation system server 802, which can be beneficial in scenarios where network connectivity is limited or low latency is desired.

[0215] Example 1 is a computer-implemented method, the method comprising: generating, by one or more processors, kernel parameter definitions and a program written in a general-purpose programming language; generatorreceiving a kernel generator comprising a set of code of generator Each of the set generator Regarding the above generator The code in the intermediate language generator and converting the kernel parameter definition and each of the generator Based on the intermediate representation of generator determining a configuration of the generator Set of binary objects of each of the sets generator and configuring a kernel corresponding to the kernel generator based on the set of binary objects for the set of operations and the kernel parameter definitions.

[0216] In Example 2, the subject matter of Example 1 is generator Converting the code of each generator This involves degrading the code to a lower level intermediate language.

[0217] In Example 3, the subject matter of any of Examples 1 and 2 is generator determining the configuration of the kernel parameter definition and each of the generator Based on the intermediate representation of generator generating a set of configurations for an executable test based on said set of configurations; generator generating a set of tests generator to determine a respective set of performance scores; selecting an optimal configuration from the set of configurations based on the respective sets of performance scores; generator determining said configuration of

[0218] In Example 4, the subject matter of any of Examples 1-3 includes generating the set of configurations further based on a parameter definition of a target machine. In Example 5, the subject matter of any of Examples 1 to 4 is generator , which includes a set of tasks running on multiple machines.

[0219] In Example 6, the subject matter of any of Examples 1-5 includes wherein the performance score includes a set of execution time and loading time. In Example 7, the subject matter of any of Examples 1 to 6 is generator generating a set of generator selecting a library for each of the configurations; and performing the test based on the selected library and each of the configurations. generator Testing a set of generator This includes generating a

[0220] In Example 8, the subject matter of any of Examples 1-7 includes wherein the set of libraries includes a set of user-defined libraries and a set of system-defined libraries. In Example 9, the subject matter of any of Examples 1 to 8 is a set of libraries defined in the library. generator However, this includes being stored in an intermediate language.

[0221] In Example 10, the subject matter of any of Examples 1 to 9 is generator is first defined in a programming language other than said general-purpose programming language and then sub-ordinated to said intermediate language.

[0222] In Example 11, the subject matter of any of Examples 1-10 includes caching the optimal configuration in a data store. In Example 12, the subject matter of any of Examples 1-11 is further modified such that the data store includes: generator The parameter definition and the target machine parameter definition can be searched for.

[0223] In Example 13, the subject matter of any of Examples 1 to 12 is generatordetermining the configuration further comprises searching the data store based on the kernel parameter definitions and the target machine configuration to find the optimal configuration.

[0224] In Example 14, the subject matter of any of Examples 1-13 includes the data store being distributed across multiple storage nodes, and the searching being performed on the distributed storage nodes.

[0225] In Example 15, the subject matter of any of Examples 1 to 14 is performed during generator A set of runtime performance data collected from the set of generator but with known configurations and known generator Including associated parameter definitions.

[0226] In Example 16, the subject matter of any of Examples 1 to 15 is further characterized in that the performance score is generator This includes based on communication between a subset of the In Example 17, the subject matter of any of Examples 1 to 16 is generator The configuration of the generator determining a configuration using a machine learning model trained on a set of runtime performance data collected from a set of generator is associated with a known configuration and known parameter definitions.

[0227] In Example 18, the subject matter of any of Examples 1 to 17 is generator converting the code of the generator The operations of generating the binary object and configuring the kernel are performed on two or more machines.

[0228] In Example 19, the subject matter of any of Examples 1-18 includes the kernel being stored in a data store accessible over a network. In Example 20, the subject matter of any of Examples 1-19 includes the kernel being combinable with other kernels in a kernel library.

[0229] In Example 21, the subject matter of any of Examples 1-20 includes using a just-in-time (JIT) compiler to perform one or more operations of the compilation phase on a local device.

[0230] Example 22 is at least one machine-readable medium containing instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1-21.

[0231] Example 23 is a device having means for implementing any of Examples 1 to 21. Example 24 is a system for implementing any of Examples 1 to 21. Example 25 is a method for implementing any of Examples 1-21.

[0232] Example 26 is a computer-implemented method including receiving, by one or more processors, an input model of a computation, the model being written in a high-level language; and converting, by the one or more processors, the model into an intermediate representation in a first intermediate language, the intermediate representation comprising a set of input variables, a set of output variables, and a set of input values ​​corresponding to the set of input variables into a set of output values ​​corresponding to the set of output variables. generator and converting, by one or more processors, the intermediate representation including a set of generator From the set of generator The first set of generator determining, by one or more processors, a second set of generatorgenerating, by one or more processors, a set of optimized primitive-level buffer semantic representations of the first set of generator a set of non-optimized primitive-level buffer semantic representations of the second set of generator and constructing an executable model corresponding to the input model based on the optimized set of primitive-level buffer semantic representations and the non-optimized set of primitive-level buffer semantic representations, the executable model being in a binary executable format.

[0233] In Example 27, the subject of Example 26 is optimized generator To generate a set of generator Each of the first set of generator to each of the above in the second intermediate language. generator based on a search for a set of kernels. generator determining a configuration of a set of intermediate representations of a set of kernels, generator determining a configuration of the set of intermediate representations, the set of intermediate representations being composed of a second set of intermediate representations of the intermediate generator generating a set of optimized primitive-level buffer semantic representations based on the configuration of the set of representations; generator generating a first set of representations.

[0234] In Example 28, the subject matter of any of Examples 26 to 27 is generator Each of the first set of generator Each of the above generator The intermediate representation of each of the generator This involves degrading the code to a lower level intermediate language.

[0235] In Example 29, the subject matter of any of Examples 26-28 includes where converting the model to an intermediate representation includes degrading code of the model to a lower-level intermediate language.

[0236] In Example 30, the subject matter of any of Examples 26 to 29 is the intermediate generator It includes the incorporation of two or more of the expressions. Example 31 is at least one machine-readable medium containing instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 26-29.

[0237] Example 32 is a device having means for implementing any of Examples 26 to 29. Example 33 is a system for implementing any of Examples 26 to 29. Example 34 is a method for implementing any of Examples 26-29. A "carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including a digital or analog communication signal or other intangible medium for facilitating communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0238] A "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, desktop computer, laptop, personal digital assistant (PDA), smartphone, tablet, ultrabook, netbook, laptop, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communications device that a user may use to access a network.

[0239] A "communications network" refers to one or more portions of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular network, a wireless network, a Wi-Fi network, other types of networks, or a combination of two or more such networks. For example, a network or portion of a network may include a wireless or cellular network, and the connection may be a code division multiple access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other type of cellular or wireless connection. In this example, the connection may implement any of various types of data transfer technologies, such as single-carrier radio transmission technology (1xRTT), Evolution Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data Rates for GSM Evolution (EDGE) technology, Third Generation Partnership Project (3GPP®) including 3G, Fourth Generation Wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standards, others defined by various standardization organizations, other long-range protocols, or other data transfer technologies.

[0240] "Machine-readable medium" refers to both machine storage media and transmission media. Thus, the term includes both storage devices / media and carrier wave / modulated data signals. The terms "machine-readable medium," "machine-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0241] A "machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory, optical media, and magnetic media, including memory internal or external to a processor. Specific examples of machine storage media, computer storage media, and / or device storage media include semiconductor memory devices, e.g., non-volatile memory including erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, some of which are covered under the term "signal media."

[0242] A "processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executing on an actual processor) that manipulates data values ​​in accordance with control signals (e.g., "commands," "opcodes," "machine code," etc.) and generates associated output signals that are applied to operate a machine. A processor may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") capable of simultaneously executing instructions.

[0243] "Signal medium" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communication signals or other intangible media for facilitating the communication of software or data. The term "signal medium" may be interpreted to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0244] A "kernel" or "microkernel" is an implementation of an algorithm that performs a computation on a memory object, such as a memory buffer with a particular layout. The two terms may be used interchangeably, but a "microkernel" is used to implement a larger generator It tends to mean small operations within a kernel implementation (e.g., memset, dot product, or reduction). Kernels that are algorithmically interchangeable / equivalent / substitutable are sometimes called "codelets" in the literature.

[0245] A "kernel generator" is a metaprogram that, when parameterized and run, generates a non-parametric implementation of a kernel or microkernel. A fixed kernel implementation (e.g., a panel dot product implemented in assembly) is a degenerate case of a parameterless generator.

[0246] A "kernel interface declaration" is a declaration of a kernel or microkernel that applies to multiple implementations of the kernel or microkernel. Kernels and microkernels may be implemented multiple times in multiple different ways. Interface declarations can exist independent of the implementation and can allow clients and implementations to be type-checked.

[0247] A "kernel generator parameter argument" refers to a value that a kernel or microkernel can act on. A kernel generator is a metaprogram that generates kernels, and "parameters" are values ​​that this metaprogram can act on.

[0248] "Kernel generator parameter results" are values ​​returned by a kernel generator as parameters to its caller, allowing the kernel generator to adapt behavior within the generated sub-kernel. For example, a panel dot product generator can return "processed a 3x5 panel of memory", which would cause the caller's for loop to step by 3 and 5 on each dimension.

[0249] A "kernel generator constraint" is a constraint that specifies restrictions on the parameters of a kernel or microkernel, such as "this implementation only works with dtype=float32", or "this only works on machines with the X86 VNNI extension", or "this works when based on the size modulo 128". A kernel generator is a part of the interface declaration that translates to a concrete implementation. generator The constraints are determined by the kernel implementation. generator It is propagated upwards into the graph.

[0250] "Kernel arguments" are static single assignment (SSA) argument values ​​used for buffers and other user-defined types for structured abstractions over memory, such as linear memory, N-dimensional tensors with layout, and other higher-level data types like trees and tables; for values ​​corresponding to op attributes at the tensor graph level (these values ​​may be modeled as constants there, but are dynamic in the kernel's runtime implementation); and when a very small microkernel at the bottom of the stack (e.g., adding two integers) uses an argument for its input.

[0251] A "kernel result" is an SSA result value that is used for dynamically allocated result buffers, such as those with data-dependent shapes, and when a very small microkernel at the bottom of the stack (e.g., adding two integers) uses the result for its output.

[0252] Changes and modifications can be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure, as expressed in the following claims.

Claims

1. 1. A computer-implemented method comprising: receiving, by one or more processors, a kernel generator comprising kernel parameter definitions and code for a set of operators written in a general-purpose programming language; For each operator in the set of operators: converting the code of each of said operators into an intermediate representation of each of said operators in an intermediate language; determining a configuration for each of the operators based on the kernel parameter definitions and the intermediate representation of each of the operators; generating a binary object for each operator of a set of binary objects for the set of operators based on the configuration; performing an action including constructing a kernel corresponding to the kernel generator based on the set of binary objects for a set of operations and the kernel parameter definitions; A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein converting the code for each operator comprises decomposing the code for each operator into a lower-level intermediate language.

3. Determining the configuration of each operator includes: generating a set of configurations for each of the operators based on the kernel parameter definitions and the intermediate representation of each of the operators; generating a set of executable test operators based on the set of configurations; executing said set of test operators to determine a set of respective performance scores; selecting an optimal configuration from the set of configurations based on the respective sets of performance scores; determining the configuration for each of the operators based on the optimal configuration; 2. The computer-implemented method of claim 1, comprising:

4. The computer-implemented method of claim 3 , wherein generating the set of configurations is further based on a target machine parameter definition.

5. The computer-implemented method of claim 3 , wherein the set of test operators executes on multiple machines.

6. The computer-implemented method of claim 3 , wherein the performance score comprises a set of execution time and loading time.

7. generating the set of test operators For each configuration of the set of configurations: selecting an operator library from the set of libraries based on each of said configurations; generating test operators of the set of test operators based on the selected library and each of the configurations; 4. The computer-implemented method of claim 3, comprising:

8. 1. A computing device comprising: one or more processors; a memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations; and the operation comprises: receiving, by one or more processors, a kernel generator comprising kernel parameter definitions and code for a set of operators written in a general-purpose programming language; For each operator in the set of operators: converting the code of each of said operators into an intermediate representation of each of said operators in an intermediate language; determining a configuration for each of the operators based on the kernel parameter definitions and the intermediate representation of each of the operators; generating a binary object for each operator of a set of binary objects for the set of operators based on the configuration; performing an action including constructing a kernel corresponding to the kernel generator based on the set of binary objects for a set of operations and the kernel parameter definitions; 1. A computing device comprising:

9. 9. The computing device of claim 8, wherein converting the code for each operator comprises decomposing the code for each operator into a lower level intermediate language.

10. Determining the configuration of each operator includes: generating a set of configurations for each of the operators based on the kernel parameter definitions and the intermediate representation of each of the operators; generating a set of executable test operators based on the set of configurations; executing said set of test operators to determine a set of respective performance scores; selecting an optimal configuration from the set of configurations based on the respective sets of performance scores; determining the configuration for each of the operators based on the optimal configuration; The computing device of claim 8 , comprising:

11. The computing device of claim 10 , wherein generating the set of configurations is further based on a target machine parameter definition.

12. The computing device of claim 10 , wherein the set of test operators executes on multiple machines.

13. The computing device of claim 10 , wherein the performance score comprises a set of execution time and loading time.

14. generating the set of test operators For each configuration of the set of configurations: selecting an operator library from the set of libraries based on each of said configurations; generating test operators of the set of test operators based on the selected library and each of the configurations; The computing device of claim 10 , further comprising:

15. A machine storage medium storing instructions that, when executed by a computer, cause the computer to: receiving, by one or more processors, a kernel generator comprising kernel parameter definitions and code for a set of operators written in a general-purpose programming language; For each operator in the set of operators: converting the code of each of said operators into an intermediate representation of each of said operators in an intermediate language; determining a configuration for each of the operators based on the kernel parameter definitions and the intermediate representation of each of the operators; generating a binary object for each operator of a set of binary objects for the set of operators based on the configuration; performing an action including constructing a kernel corresponding to the kernel generator based on the set of binary objects for a set of operations and the kernel parameter definitions; A mechanical storage medium that performs the above.

16. 16. The machine-storage medium of claim 15, wherein converting the code for each operator comprises degrading the code for each operator to a lower level intermediate language.

17. Determining the configuration of each operator includes: generating a set of configurations for each of the operators based on the kernel parameter definitions and the intermediate representation of each of the operators; generating a set of executable test operators based on the set of configurations; executing said set of test operators to determine a set of respective performance scores; selecting an optimal configuration from the set of configurations based on the respective sets of performance scores; determining the configuration for each of the operators based on the optimal configuration; 16. The machine storage medium of claim 15, comprising:

18. 20. The machine-storage medium of claim 17, wherein generating the set of configurations is further based on a parameter definition of a target machine.

19. 20. The machine-storage medium of claim 17, wherein the set of test operators executes on multiple machines.

20. The machine storage medium of claim 17 , wherein the performance score comprises a set of execution time and loading time.

Images