Compiler caching

Abstract

A system for caching compiler translations. The compilation system uses a parameterized hash in the form of a parameterized content-addressable store identifier (parameterized CAS ID) to store any operator region and any translation for any operation in the compiler intermediate representation (IR). The parameterized CAS ID includes a hash of the content of the operation region and a set of parameters containing a set of symbolic references to objects used and / or referenced within the operator region.

Description

Technical Field

[0001] This disclosure generally relates to compilers, and more particularly to compilers for code with high computational loads.

Background Art

[0002] Compilers are used to generate object code from high-level languages. It is desirable to generate optimized object code.

Brief Description of the Drawings

[0003] To easily identify the description of any particular element or operation, one or more of the leading digits of the reference number refer to the number of the figure in which the element was first introduced. [Figure 1] Data flow diagrams of kernel generation and usage processes, by way of several examples. [Figure 2] Diagrams showing the collaboration of compilation systems, by way of several examples. [Figure 3] Diagrams showing the activities of a kernel generation method, by way of several examples. [Figure 4] is a diagram showing the activities of a generator refinement method, by way of several examples. [Figure 5A] Diagrams showing the activities of a caching method, by way of several examples. [Figure 5B] Diagrams showing the intermediate representation of a program, by way of several examples. [Figure 5C] Diagrams showing a modified intermediate representation with a parameterized CAS ID replacing the cached region of the intermediate representation, by way of several examples. [Figure 5D] Diagrams showing the structure of a cache, by way of several examples. [Figure 6A] Diagrams showing the cached intermediate representation, by way of several examples. [Figure 6B]This figure shows a portion of the intermediate representation at time step 0, with several examples. [Figure 6C] This figure shows some of the intermediate representations in time step 1, with several examples. [Figure 6D] This figure shows some of the intermediate representations in time step 2, with a few examples. [Figure 6E] This figure shows the output intermediate representation with several examples. [Figure 7] This figure shows the detailed representation of the generator with several examples. [Figure 8] This figure shows the deployment of a networked compilation environment with several examples. [Figure 9] This figure shows a machine architecture in which a set of instructions can be executed to cause the machine to perform any one or more of the methods described herein, as illustrated by several examples. [Modes for carrying out the invention]

[0004] Compilers rely on tools like caches to provide compile-time memoization, where intermediate compilation results of source files are stored in a cache for later reuse. For distributed compilation and builds, various tools are available, such as Bazel, which rely on file-level hashes of file contents to determine which parts of the source code should be recompiled due to file changes. Using source files as the smallest level of granularity results in inefficient compilation, as even minor changes to the source file, such as reformatting for readability or changes to annotations, can trigger recompilation of the source file.

[0005] A compiler uses an amount of memory proportional to the number of operations in the program's intermediate representation (IR). For large programs, this can mean thousands of lines of code, which is more than twice the number of lines in the text-based IR. The compiler's role is to repeatedly scan the IR through multiple passes and perform the translation process. In languages ​​like C or C++, this process is performed on the entire translation unit (file) at once. This means that in large files with many private functions, even a single line change in the code, such as adding a comment, can result in the entire file being recompiled because the compiler does not have the necessary context to know what has actually changed and what actually needs to be recompiled.

[0006] By making symbolic operations (e.g., functions) the basic unit of computation, the compiler no longer needs to recompile all functions in a file when one of several functions changes. This dramatically improves the cache hit rate in the memoized cache of intermediate compilation results. Furthermore, IR (Investigative Reconstruction) for other functions in the file becomes unnecessary. By using a call graph, the compiler can statically determine which functions should be recompiled, avoiding the need to recompile the entire file.

[0007] The examples in this disclosure provide caching of regions associated with the IR of an operation, rather than at the source function level. In some examples, multiple levels within the function's IR are cached. In some examples, transformations are cached. Conceptually, by caching transformations, a Merkle tree is constructed from the code and the transformations to the code.

[0008] In some examples, parameterized hashes are provided in the form of parameterized content addressable store identifiers (parameterized CAS IDs). A parameterized CAS ID provides a structure where part of the code object or "blob" being hashed is not part of the content of the hash of the code object. This allows a series of operations to be performed on the cached data without actually retrieving the cached data. Illustrative operations, though not limited to, include updating weights in a static neural network architecture, and recompiling and relinking a single function from a large program.

[0009] In some cases, the use of parameterized CAS IDs can reduce the depth of the Merkle tree required to uniquely identify a cached function (thus increasing the number of cache hits), and the attribute can be brought up to the top level of the compiled program, making attribute metricing easier.

[0010] In some cases, changing the compilation paradigm from "generate files" to "generate parameterized CAS IDs" means that it becomes easier to implement distributed compilation by using a distributed hash table to store the compilation artifacts and using the parameterized CAS IDs as keys in the hash table.

[0011] While this document describes the compilation of kernels for performing many types of computations, it should be understood that the caching and compilation methods described herein can also be applied to the processing of all kinds of computer programs or code, or to the memoization of intermediate results in many types of multistep or distributed computations.

[0012] Other technical features can be readily understood by those skilled in the art from the following drawings, description, and claims. Figure 1 is a data flow diagram of the compilation and execution process 100, with several examples. The user creates a kernel definition 140 in the creation stage 120. The kernel definition 140 is used in the compilation stage 122 to construct a primitive-level representation of the kernel 112 by the compilation system 200a, which is described in more detail with reference to Figure 2, and in more detail with reference to Figure 3.

[0013] The kernel definition 140 consists of parameterization information 136 and one or more generators 132. The generators 132 comprise coding logic that defines one or more operations to be executed by the kernel according to the kernel definition 140, as shown by the code representation 130 of generator 1 and the code representation 134 of generator N. These operations can be organized into coding logic components that operate on one or more data buffers, including, but are not limited to, operations, operators, functions, objects, routines, subroutines, and modules. In some examples, one or more generators 132 include definitions of one or more data structures containing data buffers.

[0014] Parameterization information 136 includes a set of parameters that guide a compiler, such as the graph compiler 204 or kernel compiler 206 (in Figure 2), on how to generate a primitive-level representation of the kernel 112 using multiple generators 132 when compiling the kernel definition 140. In some examples, the generators 132 are written in a general-purpose programming language such as Python. In some examples, parameterization information 136 is a set of kernel parameters. In some examples, parameterization information 136 consists of a parsable scripting language or the like for generating the kernel parameters.

[0015] In the compilation stage 122, the code of the generator 132 (e.g., the code representation 130 of generator 1 and the code representation 134 of generator N) is lowered from a general - purpose programming language through a series of intermediate representations as shown by the intermediate representation 110 of generator 1 and the intermediate representation 118 of generator N to a lower - level representation of the generator (e.g., the primitive - level representation 114 of generator 1 and the primitive - level representation 116 of generator N). In the compilation stage 122, the optimal settings of the generator 132 are determined in a process described in more detail with reference to FIG. 4, in a search (e.g., search 1(106) and search N(108)) using the parameterization information 136 and the intermediate representation of the generator 132. The search results and other compilation metrics are stored in the cache 128 for use during subsequent searches. The search may include steps of exploring a tree of possible settings using a combination of static analysis of previous settings and dynamic analysis of proposed settings, which is done during the refinement process as described in more detail with reference to FIG. 4.

[0016] The primitive - level buffer semantic representation of the generator 132 is combined with the primitive - level representation of the kernel 112. The primitive - level representation of the kernel 112 is stored for later use.

[0017] To generate an executable object, a kernel object 138 is generated using the primitive - level representation of the kernel 112, and this kernel object 138 is included in the executable object 126. The executable object 126 is executed in the execution stage 124 after being generated, and uses the kernel object 138 to perform one or more calculations. In some examples, the library 202 is used to extend the kernel object 138 with additional executable objects.

[0018] In some examples, the execution metrics of the kernel are stored in the data store of execution metrics 142. Execution metrics 142 are used by subsequent search component 102 to determine the optimal settings of one or more generators 132.

[0019] In some examples, an artificial intelligence (AI) component 104 is used to assist in the search. In some examples, AI component 104 also assists in the creation stage 120 in which the kernel is described within the software development environment (SDE) of an integrated development environment (IDE).

[0020] In some examples, one or more operations are defined by generator 132 as a fusion of multiple other lower-level operations, such as broadcast operations, activation operations, etc., and in some cases, as an even larger fused amalgam such as a long short-term memory (LSTM) operation. By describing the generator at this abstract level, high-level optimizations such as, but not limited to, the extraction of shape operators and the generation of operator gradients are simplified.

[0021] In some examples, one or more generators 132 are used to generate implementations of existing functions or operators in existing machine learning (ML) frameworks (such as TFLite, TF, ONNX, PyTorch, etc.). Operators within existing ML frameworks have attributes such as, but not limited to, support for broadcast and type promotion, handwritten operators selected by experts known to be important for specific classes of models, such as activation operators fused into element-wise generators such as "add", support for quantization algorithms that depend on architecture-specific DSP operations, the layout assumed by the existing framework, support for dynamic shapes and dynamic dtypes.

[0022] In some examples, the generator supports the following: • Dynamic shapes • Broadcasting, type promotion: For example, "mul" is a binary generator, and its two operands can have different shapes and dtypes. ML frameworks often improve usability by providing implicit promotion to common element types and support broadcasting of elements.

[0023] • Layout changes (munging): Some frameworks support multiple different layouts, such as row-major and column-major, tiled layouts, etc. Conversion may be necessary if the input is in a different format. Some libraries provide a common implementation that can work with many different layouts using strides, but strides are not common for tiled layouts.

[0024] • Type dispatch: Standard kernel libraries support multiple dtypes, but they are only known dynamically at kernel call time. This requires the kernel to dynamically dispatch for dtypes and dispatch to kernels specialized for many different dtypes. Some dtypes may have special cases; for example, "complex add" can be handled by the same code path as "scalar add" (since complex addition is element-wise), but "complex mul" is an entirely different algorithm from "scalar mul".

[0025] • Thread tiling: At a level outside of type-specific kernel algorithms, computations are divided into blocks that can be executed in parallel by multiple threads. The size of each subunit must be determined and is generally best evaluated based on hardware characteristics and the size of the input data (not on #available threads).

[0026] • Cache tiling: In per-thread computations, calculations are typically cached and blocked at, for example, the L2 level. The size of the L2 level is target-specific. This is important for algorithms that perform multiple passes through data, and may be less important for element-wise operations with little reuse.

[0027] • Tile-based algorithms: Within each L2 tile, there are many ways to implement the core algorithm, including the use of scalars, vectors, and prefetching. Also, when broadcasts are handled internally within the kernel, there are special cases that are interesting to handle, for example, when the fastest changing dimension of one operand is broadcast.

[0028] • Many microkernels: Algorithms like matrix multiplication rely on lower-level operations such as memset for clearing buffers, panel dot products, and reductions. These "microkernels" themselves can be implemented in many different ways.

[0029] • Macro-algorithms: Many generators have multiple completely different algorithms for computing the result; for example, in convolution, we see the im2col approach, direct convolution, and Winograd. Matmul has many implementations (especially when quantization and accelerators force special data layouts), including Strassen's algorithm, among others.

[0030] Hardware targets now frequently have spatial operations (such as Apple AMX or Intel AMX) that can accelerate multiple loop nests at once, for example, matrix multiplication and large element-unit blocks. They also have many architectural families that are register-blocked, pipelining, and require unrolling (deployment) in different ways.

[0031] In some examples, the primitive-level representation of kernel 112 is a component of a framework consisting of a set of code-generating kernels that operate on memory buffers, including, but not limited to, memory operators, 1D memory arrays, tensor buffers, and user-defined data structures. In some examples, the kernels directly use C / C++, assembly, and intrinsic for specific hardware features.

[0032] In some cases, a library of kernel components is used to generate additional kernels. For example, buffer-level operators are used to replace legacy kernels. Kernel components are modular and reusable, and include, but are not limited to, core algorithms such as memory fill, reduction, and element-wise operators, in addition to quantization kernels and more specialized primitives used in other domains.

[0033] In some cases, the generator is a parametric generator. It is difficult for humans to manually create and maintain all permutations of kernels (e.g., for all dtypes, all target machines, etc.), and therefore, metaprogramming is widely used. This metaprogramming takes various forms, such as C macros and ifdefs, Python generator frameworks, and "emitters" written in C++ for the "IRBuilder" compiler API, but the most widely used is C++ templates.

[0034] In some examples, a kernel is defined as a declarative generator that takes kernel parameters and has arbitrary imperative logic coded for those parameters, and that imperative logic is "burned into" the code generated for the kernel. This can be used to specialize in things like dtype, unroll coefficient, vector length, cache size, etc. Most parameters should have an integer type, be constrained by a range (e.g., unroll <= 8 times), a list of valid values ​​(e.g., vector length = 2, 4, 8, 16, 32), and support an enum (enumeration type) to make them searchable (e.g., consider dtype). Using generators still allows the use of concrete kernels (e.g., fixed blobs of assembly) because they are valid generators that have no parameters (or, similarly, have fully constrained parameters).

[0035] An example of the code is shown below. A kernel may have parameters bound to its invocation site; for example, after a dynamic switch on dtype, the next lower-level microkernel is called with a dtype parameter bound to a constant value.

[0036]

number

[0037] Figure 2 is a block diagram of the compilation system 200a, with some examples. The compilation system 200a generates software objects such as a kernel using a kernel definition 140. The kernel definition 140 includes parameterization information 136 and one or more generators 132. The compilation system 200a uses the generated kernel 208 and a handwritten kernel 210 to generate an executable object 216 in a binary executable format (BEF) BEF218 using the generated kernel 208 and the handwritten kernel 210. The compilation system 200a uses a kernel generation method 300 to generate a kernel using the kernel definition 140.

[0038] In some examples, an executable object 216 in a binary executable format 218 containing one or more kernel objects 138 is executed during runtime 212 on a group of hardware devices 214 to generate execution metrics 142 used to optimize the kernel by optimizing the generator settings.

[0039] Figure 3 is an activity diagram of a kernel generation method in several examples, and Figure 4 is a generator refinement method 400. The compilation system 200a compiles or generates a kernel using the kernel generation method 300. The kernel generation method 300 and the generator refinement method 400 show a specific order of processing, but this order can be changed without departing from the scope of this disclosure. For example, some of the illustrated processing may be performed in parallel, in a different order, or by different components of a composable kernel compilation system, and such implementations do not substantially affect the generator of the refinement process. In other examples, different components of an exemplary device or implementation of the compilation system 200a may perform processing substantially simultaneously or in a specific order.

[0040] In process 302, the compilation system 200a receives parameterization information 136 and a kernel definition 140 which includes one or more generators 132 containing coding logic that defines the kernel. In some examples, the code includes generator code written in a general-purpose programming language.

[0041] In processes 304 and 306, the compilation system 200a determines the optimal settings for each generator using parameterization information 136 in a process that will be described in more detail with reference to Figure 4.

[0042] In process 308, the compilation system 200a generates a primitive-level buffer semantic representation of the generator using the optimal settings for the generator. For example, the compilation system 200a uses one or more intermediate representations to reduce the generator to a primitive-level buffer semantic representation through a series of compilation passes.

[0043] In process 310, the compilation system 200a adds the primitive-level buffered semantic representation of the generator to a set of primitive-level buffered semantic representations of the generator. The set of primitive-level buffered semantic representations of the generator is used to constitute the primitive-level buffered semantic representation of the kernel.

[0044] In process 312, the compilation system 200a uses a set of primitive-level buffer semantic representations of generators to construct primitive-level buffer semantic representations of kernels corresponding to input generators. For example, the compilation system 200a obtains a set of primitive-level buffer semantic representations of generators and code slices the primitive-level buffer semantic representations of generators and their dependencies into a single module or kernel.

[0045] In process 314, the compilation system 200a breaks down a single module into one or more object (.o) files and stores one or more object files of the kernel in a datastore, such as the CAS of the generated kernel 208, though not limited to these. In some examples, the object files have the format of object files generated by a standard C-style toolchain and therefore work seamlessly with a stack that implements a C / C++ External Generator Interface (FFI).

[0046] Figure 4 is an activity diagram of the generator refinement method 400, with some examples. The compile system 200a uses the generator refinement method 400 to generate a permutation of generator settings that are evaluated by the compile system 200a to determine the optimal settings for the generator. While the generator refinement method 400 shows a specific processing order, this order can be changed without departing from the scope of this disclosure. For example, some of the illustrated processing may be performed in parallel, in different orders, or by different components of the compile system 200a, and such implementations do not substantially affect the generator of the refinement process. In other examples, different components of the exemplary device or implementation of the compile system 200a may perform processing substantially simultaneously or in a specific order.

[0047] In process 402, the compilation system 200a searches for the optimal settings of the generator using an evaluator associated with the generator. The kernel compiler 206 can perform static and dynamic analysis searches for the optimal settings of the generator. In static analysis searches, the kernel compiler 206 uses a search component 102 to search several different types of datastores. One type of datastore is a cache 128 containing optimal generator settings that the kernel compiler 206 can reuse to determine the optimal settings of the generator when the generator is downgraded during the compilation process. The cache 128 can be a local cache or a distributed cache distributed across remote storage nodes on one or more servers. For example, the compilation system 200a maintains a datastore of optimal settings within the cache 128. The search component 102 searches for the optimal settings of the generator using an evaluator, which is an index for the search component 102 to determine that the generator settings are optimal.

[0048] In some examples, the cache includes a hash table. The hash table contains regions of intermediate representations of operations generated using a generator. The regions of intermediate representations are stored in the hash table using parameterized CAS IDs, as will be explained in more detail with reference to Figures 5A, 6A, and 7.

[0049] In process 404, the compilation system 200a determines whether an optimal setting was found during the search of cache 128. In response to determining that an optimal generator configuration could not be found during static analysis search, the kernel compiler 206 performs a search using dynamic analysis of the generator. To do this, in process 406, the kernel compiler 206 generates a set of configurations. For example, the compilation system 200a generates an intermediate representation of the generator. Using the intermediate representation of the generator and parameterization information 136, the compilation system 200a generates one or more configurations of the generator as one or more test intermediate representations of the generator.

[0050] In process 408, the compilation system 200a generates a set of executable test functions using one or more test intermediate representations. For example, for each test intermediate representation, the compilation system 200a generates executable test functions by lowering the test intermediate representation to an executable object in the BEF.

[0051] In process 410, the compilation system 200a executes a set of test functions to determine a set of individual performance scores. For example, a composable kernel compilation system executes each test function to monitor its performance as it operates on the data in the test suite. In some examples, the performance score includes an initialization score indicating the amount of time the test function spent during its initialization. In some examples, the performance score includes an execution score indicating the amount of time the test function spent processing the test dataset. In some examples, the performance score includes the amount of time the test function spent communicating with other generators in the kernel during execution.

[0052] In process 412, the compilation system 200a uses a set of individual performance scores to select the optimal setting from a set of settings. For example, the kernel compiler 206 assigns weights to each set of generator, setting, and performance data. When selecting, the compilation system 200a uses the set of function, setting, and performance evaluation data, and their associated weights, to select the generator setting.

[0053] In process 414, the compilation system 200a generates an intermediate representation of the generator using the optimal settings. In process 416, the compilation system 200a stores the optimal settings for the generator in a cache for the subsequent search process. For example, the cache includes a hash table. The hash table contains regions of intermediate representations of operations generated using the optimal settings for the generator. The regions of intermediate representations are stored in the hash table using parameterized CAS IDs, as will be explained in more detail with reference to Figures 5A, 6A, and 7.

[0054] In process 418, the compilation system 200a returns the optimal settings for the generator. In some examples, generating a set of configurations is done based on parameterization information of the target machine.

[0055] In some examples, a set of test functions is executed on multiple machines. In some examples, the performance score includes a set of execution time and loading time.

[0056] In some examples, generating a set of test functions involves using a setting to select a library for the generator settings from a set of libraries, and then using the selected library and setting to generate one of the test functions from the set.

[0057] In some examples, a library set includes a set of user-defined libraries and a set of system-defined libraries. In some cases, generator configurations within a set of libraries are stored in an intermediate language.

[0058] In some cases, generators are initially defined in a programming language other than a general-purpose programming language, and then lowered to an intermediate language. In some cases, cache 128 can be explored using the parameterization information of the generator and the parameterization information of the target machine.

[0059] In some examples, determining the generator settings involves searching cache 128 using parameterized information and target machine settings to find the optimal settings for the generator.

[0060] In some examples, cache 128 is distributed across multiple storage nodes, and exploration is performed on the distributed storage nodes. In some examples, a set of runtime performance data collected from a set of functions executed during runtime is stored in cache 128, and each executed function is associated with a known generator setting and known generator parameterization information.

[0061] In some examples, performance data includes communication data from the communication between some of the executed functions. In some examples, determining the generator settings involves using a machine learning model of an AI component 104 trained on a set of runtime performance data collected from a set of functions executed during runtime to determine the settings, with each executed function associated with known settings and known parameterization information.

[0062] In some examples, the process of translating generator code, determining settings, creating executable objects, and configuring the kernel is performed on two or more machines.

[0063] In some examples, the primitive-level representation of kernel 112 is stored in a datastore accessible over the network. In some cases, the primitive-level representation of kernel 112 can be coupled with other kernels in the kernel library.

[0064] In some cases, since there are no dependencies between one or more generators of a kernel, the compilation system 200a can process the generators in parallel. This structure (along with the common tree / forest / DAG structure of computation) contributes to a compilation process for parallel kernels that can be used to speed up kernel generation on one or more multicore machines.

[0065] In some cases, kernel creators declare their own abstractions, as in C++. To do this, compilation system 200a provides the ability to declare interfaces to (micro)kernels, supporting the possibility of having many different implementations for each microkernel, each of which implements a common interface. Each kernel can be defined recursively using simpler and smaller kernels, which themselves can have multiple different implementations.

[0066] In some examples, there are multiple available implementations of each kernel, microkernel, generator, and generator, and the compilation system 200a determines which is optimal for a given target and scenario (dtype, size class, etc.). Thus, the (micro)kernel interface declaration defines a cost model that is optimized by search (e.g., find the configuration of the implementation that has the "best achieved FLOPS"). For example, implementations of a microkernel may include using scalar arithmetic, being implemented with multiple SIMD generators of different lengths, some implemented in inline assembly, and possibly implemented using Apple AMX. The compilation system 200a empirically selects the configuration for the implementation with the highest throughput for the current hardware by measuring it (implementations for incompatible systems are ignored as infinite cost).

[0067] In some examples, exploration is enabled by constructing a large set of executable objects 126 that use a generator in a realistic way. This allows the compilation system 200a to collect data for one or more execution metrics 142. For example, metrics are collected for executable objects 126 that include a model. The metrics include tensor input size and execution time metrics (using realistic input dimensions instead of random input dimensions), similar to the mmperf "benchmark size" list. In some examples, profiles are collected and used, or certain dimensions are weighted more heavily, to achieve goals such as "prioritize MLPerf performance" or "generate the best possible code for a particular model," depending on the goals of any particular product.

[0068] In some examples, the parameters in parameterization information 136 are not specified. These parameters are explored and determined by the compilation system 200a during the search. For example, the compilation system 200a determines the number of iterations to fit the cache, returns the result as a parameter result, and allows the enclosing generator to tile or parallelize based on that result. As another example, given an element-wise multiplication microkernel implemented with respect to a vector across a 1D block of memory, a loop utilizing one of these low-level generators will increase its FLOPS until it exceeds the L2 cache, at which point the cache-blocking algorithm described above is usually more efficient. Allowing kernels to define a metric (e.g., FLOPS) enables the use of search to find the correct implementation. A top-level generator kernel may use latency as its metric.

[0069] In some cases, several generator parameters (e.g., dtype) are defined on the generator interface (and therefore common to all implementations), so the compilation system 200a provides an implementation of the generator that also has additional parameters (e.g., an ARM implementation of a kernel that provides three implementations of the same generator for different microarchitectures). This is sugar for "flattening" these parameters as different individual implementations of the same microkernel.

[0070] In some examples, multiple implementations exist for each microkernel, which are then implemented based on other interfaces that may have many implementations. These expansions form a tree of possible expansions, and since there can be many top-level generators within the framework, there is a forest of expansions that operate at many levels of abstraction. For example, a matrix multiplication microkernel can be implemented using three levels of for loops, cache blocking, and internal L2 tiling. It may also be implemented using a 2D generator and a common accelerator to use target-specific dot product operations. Each of these may be implemented independently of the others, all implementing the same interface. Each “expansion tree” can have an exponentially number of possible expansions for a single framework generator. This makes it impractical to explore the entire space for a single kernel, and even more difficult to support an entire ML framework, especially when a single framework may have hundreds / thousands of individual kernels.

[0071] In some examples, human-created constraints are defined on the kernel to cut off the search space or guide the search as a fundamental boundary in parameter declarations. In some examples, conditional constraints are provided. In some examples, redundancy in tree-based structures is leveraged by dynamic programming techniques. Dynamic programming algorithmically improves the performance of hierarchical tree-based algorithms by using subproblem storage / caching. In some examples, each tree in the expansion has many common leaves, and the forest has many shared leaves, subtrees, and potentially the entire kernel. By allowing the cost model to be defined at many levels (not just the top-level framework generator), the compilation system 200a can leverage modularity for the search and caching the results. The use of dynamic programming collapses the "expanded tree" into a directed acyclic graph (DAG).

[0072] In some examples, the cache is hosted on a cloud service, providing the user with an oracle so they can access previously explored algorithms offline. This allows the user to avoid performing the full exploration algorithm on their own device. In some examples, the compile system 200a generates an analysis of what the user is using the compile system 200a for. In some examples, the installation size of a mobile framework may be very small, and instead of shipping a typical kernel library with a lot of bloated kernels, the compile system 200a provider ships a just-in-time (JIT) compiler that can generate a kernel. Users may not want to perform the exploration on their own device, and therefore the compile system 200a provider may bundle a binary blob with the application, or download kernel parameters appropriate for the target hardware, generate / cached the machine code for the kernel during app installation, and add logic to use the compiler as a "compression scheme" to reduce the impact of the kernel library download size.

[0073] In some examples, each level of the generator tree expansion is functional (without side effects), and the "keys" used to look up the computations are encodeable in a way that allows the compilation system 200a to hash and look up the results of the transformations (for example, the key is a blob of serialized MLIR). This is useful for parallelizing tree compilation (trees / DAGs have a lot of parallelism).

[0074] In some cases, kernel fusion is enabled, which merges arbitrary element-wise calculations into matrix multiplication. Compilation system 200a supports this by allowing generators to be parameterized by regions. A region is simply a different form of parameter argument through which the code body is passed and which is accessible to metaprogramming structures. For example, by exposing regions as a general function in the system of compilation system 200a, it becomes possible to define actions such as "switch based on dtype" and "statically unroll loops using this parametric representation" in the system itself, rather than having them hardcoded into the system. This allows compilation system 200a to be user extensible, as nothing in the stack is specialized for dense linear algebra, allowing users to build their own libraries of generators that divide work over data tables or trees and interact with their own external storage (e.g., databases).

[0075] In some examples, the compilation system 200a leverages algorithmic skeletons that enable the description of higher-order transformations that allow parallel patterns to be encoded in a reusable manner, which simplifies the implementation task because each skeleton can be considered independently, in contrast to the monolithic programming interfaces of existing systems at a similar level of abstraction.

[0076] In some examples, a generator is allowed to be a partial generator, ranging from an interface declaration to a concrete implementation. Constraints indicate limitations on those parameters, such as "this implementation only works with dtype=float32," or "this only works on machines with the x86 VNNI extension," or "this works based on the remainder when the size is divided by 136." In some examples, this propagates upward from the kernel implementation to the generator graph.

[0077] In some examples, the compilation system 200a uses kernel descriptions in an intermediate representation format, which is a mechanically parseable / transformable format. In some examples, the compilation system 200a extracts shape generators for generators by extracting computations from the kernel description using code slicing. This ensures that the compilation system 200a has a single true source for kernel + shape generators.

[0078] In some examples, the compile system 200a implements generators with a multilevel intermediate representation (MLIR) compiler API to provide structures more complex than parameterized expansions. These generators are encoded as compiler transformations and provide users with a flexible programming model. They are generators that take regions of an intermediate representation as parameters and generate new ones.

[0079] In some cases, the compilation system 200a extracts metadata about the operation, such as whether they are coupled or whether they have side effects. In some cases, kernels generated by compilation system 200a take output buffers as arguments that may not be exposed to the graph. Compilation system 200a provides a “buffer-exposed” graph-level representation that enables memory planning, in-place optimizations for concatenation, and more.

[0080] In some cases, the compilation system 200a retrieves metadata from the buffer-level generator implementation and reflects it at the generator graph level. In some cases, the compile system 200a employs a Python-like language, which is a user-extensible hybrid declarative / imperative programming language that allows arbitrary MLIR generator graphs to be represented in a usable way.

[0081] Figure 5A is an activity diagram of caching method 500 with several examples. Figure 5B shows an intermediate representation 566 of the program with several examples, Figure 5C shows a modified intermediate representation 522 with parameterized CAS IDs that replace the cached regions of intermediate representation 566, and Figure 5D shows the structure of cache 544.

[0082] While the exemplary caching method 500 illustrates a specific order of processing, this order can be modified without departing from the scope of the disclosure. For example, some of the illustrated operations may be performed in parallel or in a different order that does not substantially affect the functionality of the caching method 500. In other examples, different components of the compilation system 200a may perform processing substantially simultaneously or in a specific order.

[0083] The intermediate representation 566 contains one or more components, namely the model "baz" 530 and a set of operations, namely the function "foo" 524, the function "bar" 526, and the operation "someop" 528. The memory footprint of the intermediate representation 566 is almost entirely contained within the bodies of the operations (such as the body 540), and the compilation system 200a caches only the regions of the modified intermediate representation 566 that are associated with these operations, with the purpose of the compilation system 200a to recompile only the parts of the program in the intermediate representation 522 that have been modified since the program was last compiled.

[0084] In process 502 of Figure 5A, the compilation system 200a receives the intermediate representation 566. In process 504, the compilation system 200a detects one or more operations in the intermediate representation 566. For example, the compilation system 200a parses the intermediate representation 566 and detects, but is not limited to, parts of the intermediate representation 566 that indicate the start of an operation definition, such as the strings "func", "%", or "model".

[0085] In process 506, for each component of the modified intermediate representation 522 (e.g., function "bar" 526), ​​the compilation system 200a separates the symbolic operator of the operation (e.g., symbolic operator 520) from its body (e.g., body 540) and replaces the symbolic operator itself with a substitution symbolic operator in the modified intermediate representation 522 (e.g., substitution operator 532) that represents the symbolic operator 520 in the intermediate representation 566.

[0086] In process 508, the compilation system 200a determines a set of regions in the body of the operation. For example, since the function "foo" 524 does not reference any other functions, the function "foo" 524 consists of a single region, i.e., the body of the function "foo" 524. Since the function "bar" 526 contains a single reference to the function "foo" 524, the function "bar" 526 consists of a single region, i.e., the body of the function "bar" 526. Similarly, the operation body "someop" 528 consists of a single region. The model "baz" 530 consists of two regions, indicated by the "init" region 548 which contains a reference to the function "foo" 524, and the "execute" region 550 which contains calls to the functions "foo" 524 and "bar" 526.

[0087] In process 510, the compilation system 200a detects calls in multiple regions. For example, the single region of function "foo" 524 contains no calls. The single region within function "bar" 526 contains a reference to function "foo" 524. The single region of operation "someop" 528 contains a reference to function "foo" 524. The "init" region 548 of model "baz" 530 contains a call 542 to function "foo" 524. The "execute" region 550 of model "baz" 530 contains a reference to function "foo" 524 and a reference to operation "someop" 528.

[0088] In process 512, the compilation system 200a uses the hash function, the region content, and the discovered call to generate a separate parameterized CAS ID for each region. For example, for the function "foo" 524, the compilation system 200a generates a parameterized CAS ID 552 by hashing the content of the single region of function "foo" 524 to create "HASHFOO". Since function "foo" 524 does not refer to any other functions, there are no symbolic references in the parameterized CAS ID 552.

[0089] For function "bar" 526, the compilation system 200a hashes the contents of a single region of function "bar" 526 to create a hash "HASHBAR" of the parameterized CAS ID 536. Since function "bar" 526 calls function "foo" 524, the compilation system 200a adds a reference to function "foo" 524 in symbolic reference 538.

[0090] For operation "someop" 528, the compilation system 200a hashes the contents of the only region of operation "someop" 528 to generate the hash "HASHSOMEOP" and appends it to the parameterized CAS ID 554. Since operation "someop" 528 calls function "foo" 524, the compilation system 200a adds a reference to function "foo" 524 to symbolic reference 556.

[0091] Model "baz" 530 contains two regions, "init" region 548 and "execute" region 550. Compilation system 200a generates a parameterized CAS ID 558 for "init" region 548 and a parameterized CAS ID 562 for "execute" region 550. Compilation system 200a hashes the contents of "init" region 548 to generate a hash "HASHBAZINIT" which is appended to parameterized CAS ID 558. Since "init" region 548 contains a reference to function "foo" 524, the reference to function "foo" 524 is appended to the symbolic reference 560 in parameterized CAS ID 558. Similarly, the compilation system 200a hashes the contents of the "execute" region 550 to generate a hash "HASHBAZEXEC" which is added to the parameterized CAS ID 562, and references to functions "foo" 524 and "bar" 526 are added to the symbolic reference 564 of the parameterized CAS ID 562.

[0092] In process 514, the compilation system 200a copies or moves the contents of multiple regions to separate containers (e.g., cache container 546) that are cached in a hash table, using the individual parameterized CAS IDs as keys in the hash table.

[0093] In process 516, the compilation system 200a replaces multiple regions of multiple operations with individual parameterized CAS IDs. For example, the region of function "foo" 524 is replaced with parameterized CAS ID 552, the region of function "bar" 526 is replaced with parameterized CAS ID 536, the region of operation "someop" 528 is replaced with parameterized CAS ID 554, and the two regions of model "baz" 530 are replaced with parameterized CAS ID 558 and parameterized CAS ID 562.

[0094] In process 518, the compilation system 200a returns the modified intermediate representation 522. In some examples, each region has a list of symbols it references (e.g., referenced symbol 534), and an index to this list is used in symbolic reference 538. In some examples, symbolic references can be used anywhere, including other attributes, so special attributes may be used to reference region metaparameters, which may include symbol bindings. In some examples, symbolic references may also be used for other attributes, though not limited to, such as constant data hashes.

[0095] In some examples, the user chooses how the parameterized CAS ID is parameterized at a finer granularity level. In some examples, a dialect interface is provided that the user can specialize to convert attributes into symbolic attributes, which the caching code translates from one arbitrary attribute to a top-level parameter list on the parameterized CAS ID.

[0096] The use of parameterized CAS IDs and parameterized hashes is a natural representation of maintaining a call graph. Symbols with calls (and therefore symbols with calls) are parameterized with respect to the callee, so if a change in the callee does not affect the caller, the caller does not need to be recompiled. This improves cache hit rates and effectively terminates traversals.

[0097] In some cases, performing a parsing or transformation involves inflating the modified intermediate representation 522 back to its original state. This process is easily reversible, and because the IR is cached in a hash table, the compilation system 200a only inflates the operations it uses to perform the parsing or transformation.

[0098] In some examples, operations without regions are not cached because the majority of operations in the intermediate representation consist of bodies of various regions. In some examples, the caching method 500 is entirely recursive. For example, if a SymbolTable operation involves symbols, the compilation system 200a can cache the region of the SymbolTable, and the parameterized CAS ID is parametric with respect to the symbols contained in the region. The cached object has a reference to the symbol rather than a call to the symbol, and expands to the cached symbol rather than a call operation.

[0099] Figures 6A to 6E illustrate the transformation of intermediate representations during the distributed adverse event analysis process in several examples. Figure 6A shows a cached intermediate representation 602 in several examples, Figure 6B shows a portion of the intermediate representation at time step 0 (604), Figure 6C shows a portion of the intermediate representation at time step 1 (606), Figure 6D shows a portion of the intermediate representation at time step 2 (608), and Figure 6E shows the output intermediate representation 610. The use of parameterized CAS IDs provides cache-aware transforms, such as the adverse event inference path or the strongly connected component (SCC) path in the call graph in LLVM.

[0100] Figure 6A shows an intermediate representation 602 having two functions, namely function "foo" 612 and function "bar" 614, and model "baz" 616. The regions of function "foo" 612, function "bar" 614, and model "baz" 616 are cached at cache locations pointed to by parameterized CAS IDs 618, 620, 622, and 624, respectively.

[0101] The side effect inference path is executed bottom-up against the call graph. Previous analyses of function "foo" 612 can be read from the cache by using the analysis to be performed as the key and the side effect attribute as the value in a hash table. Side effect inference is then performed against three regions: the body of function "bar" 614 (indicated by parameterized CAS ID 620), and two regions of model "baz" 616 (indicated by parameterized CAS IDs 622 and 624). In some examples, side effect inference path runs are parallelized and therefore have multiple processes performing inference, for example, task 1 (T1) 626 performs inference against the body of function "bar" 614, task 2 (T2) 628 performs inference against the first region "baz" 616 of the model, and task 3 (T3) 630 performs inference against the second region "baz" 616 of the model. T1 and T2 can run in parallel, and T3 depends on T1.

[0102] At time step 0 (604), the path has a cache hit for function "foo" 612, which has the side effect "read" 632. Task 1 (T1) 626 inflates function "bar" 614. Task 2 (T2) 628 partially inflates model "baz" 616.

[0103] Referring to Figure 6C, at time step 1(606), task 1(T1)626 finds that the function "bar" 614 has no additional side effects and only side effects from the function "foo" 612 (Figure 6B). Therefore, task 1(T1)626 removes the body of the function "bar" 614 and replaces it with the cache attribute 634.

[0104] Task 2(T2)628 finds that the initialization region of model "baz"616 (exemplified by parameterized CAS ID 636 "HASHBASINIT") has additional side effects. Therefore, Task 2(T2)628 deflates the first region by modifying its attributes. Once Task 1(T1)626 is complete, Task 3(T3)630 inflates the second region of model "baz"616.

[0105] Referring to Figure 6D, Task 3(T3)630 also finds additional side effects on the second region of model "baz"616, and therefore Task 3(T3)630 deflates the second region of model "baz"616 by modifying attribute 638, as exemplified by the parameterized CAS ID 640 "HASHBAZEXEC".

[0106] Referring to Figure 6E, the output intermediate representation 610 shows that all adverse event attributes can be cached by combining the performed analysis (adverse event inference) with the symbolic operations 644 of parameterized CAS ID 646 and parameterized CAS ID 648, and storing the result attributes as values ​​642.

[0107] In some cases, since caching transformations are completely reversible, integrating non-cache-aware transformations can be achieved by inflating the relevant sections before executing the path. In some cases, legacy paths can be treated as black boxes by simply caching the output of an operation that has the region attribute that the path processes, keyed in combination with the path name.

[0108] Figure 7 illustrates generator refinement using several examples. The elaborator component of a compilation system (e.g., compilation system 200a) can perform generator refinement while searching for the optimal settings for the generator, as described in more detail with reference to Figure 4.

[0109] In some examples, to perform refinement, the elaborator component creates a call graph SCC path in a bottom-up traverse of the call graph and expands the generators into functions. In some examples, the elaborator component is cache-aware and takes full advantage of the parallelism that can be obtained through the use of a distributed cache with parameterized CAS IDs. The cache-aware elaborator takes a deflated generator, inflates it back to the original generator as needed, performs the expansion, and deflates the resulting function. For example, generator 702 contains two generators, the "@foo" generator 710 and the "@bar" generator 712. These generators contain a parameterized CAS ID (e.g., parameterized CAS ID 730 and parameterized CAS ID 732) and a set of attributes containing a set of input metadata (e.g., input metadata 708 and input metadata 722). During refinement 706, the elaborator component uses attributes to generate a set of test functions 704, namely the "@foo" test function 714, the first "@bar" test function 716, the second "@bar" test function 718, and the third "@bar" test function 720.

[0110] Because the input metadata 708 for the "@foo" generator 710 contained an empty metadata set, the refinement of the "@foo" generator 710 generates a single test function, namely the "@foo" test function 714. Since there is only one test function for the "@foo" generator 710, the parameterized CAS ID 730 of the "@foo" generator 710 is copied to the "@foo" test function 714.

[0111] However, since the input metadata 722 of the "@bar" generator 712 included an indexable size parameter 740 as part of the input metadata 722, the elaborator component generates a set of test functions, each test function having unique size parameters such as size parameter 724, size parameter 726, and size parameter 728. Because the regions of the body of each test function are different, these regions are stored in separate caches accessed using parameterized CAS IDs, as indicated by parameterized CAS IDs 734, 736, and 738.

[0112] In some cases, the advantage of caching regions rather than operations that have regions is that the generator interface does not need to be cached. Since the generator interface does not have a body, it does not need to be cached. A cache-aware elaborator component can detect a deflated generator that implements the interface and, in the event of a cache miss, can inflate the deflated generator back to the original generator to perform the search and / or unfold.

[0113] Figure 8 shows a collaboration diagram of a networked compilation system 800, in several examples. The compilation system 800 includes one or more computing systems, such as computing system 1 (822) to computing system N (802), which communicate over one or more networks, such as network 814. In some examples, network 814 is a local area network (LAN). In some examples, network 814 is a wide area network (WAN), such as the Internet.

[0114] The computing system includes one or more computing machines, such as machine 900 in Figure 9. The one or more computing systems host one or more compilers, such as computing system 1(822) hosting compiler 1(824) and computing system N(802) hosting compiler N(820). Each compiler is communicably connected to other compilers of compilation system 800 (e.g., hosted on separate other computing systems) via one or more communication networks, including a local area network (LAN) 814. The compilers can also communicate with locally hosted applications on separate computing systems using application programming interfaces (APIs).

[0115] The compilation system 800 further includes a cache 816 that is connected to the compiler via one or more networks, such as network 814. The compilation system 800 further includes an IDE server 818 that hosts an Integrated Development Environment (IDE) 804. The IDE 804 is communicated to the compiler via one or more communication networks, such as network 814.

[0116] The compiler interacts with other compilers and IDE804 via network 814. The data exchanged between the compiler and IDE804 includes functions (e.g., commands for calling functions) and payload data (e.g., coding logic for compilation).

[0117] The client computing system 808 hosts IDE clients 810 that are communicably connected to the IDE 804 via one or more communication networks, such as network 806. In some examples, network 806 is a LAN. In some examples, network 806 is a WAN, such as the Internet.

[0118] The user uses the IDE client 810 to communicate with the IDE 804 and write the coding logic to be compiled by the compiler. During compilation, the compiler may access the cache 816 to store an intermediate representation and / or executable object of the coding logic. In some examples, the IDE client 810 communicates with the client cache 812. During compilation, the compiler may access the client cache 812 via the IDE client 810 to store an intermediate representation and / or executable object of the coding logic.

[0119] The compilation system 800 provides server-side compilation functionality, as described herein, to the IDE client 810 via the network 806. While specific functions of the compilation system 800 are described herein as being performed by compilers (e.g., compiler 1(824) and compiler N(820)), IDE 804, or one or more client-side APIs or applications, the location of specific functions within the compilation system 800 or client computing system 808 can be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the client computing system 808 and then later migrate these technologies and functions to the compilation system 800 if the computing system of the compilation system 800 (e.g., computing system N(802)) has sufficient processing power.

[0120] The compilation system 800 supports various services and processes provided to the client computing system 808. Such processes include sending data to and receiving data from the compilation system 800 and the client computing system 808, and processing the data generated by them. This data may include, but is not limited to, coding logic, intermediate representations and / or executable objects of the coding logic, compilation metrics, and execution metrics for one or more executable objects. The IDE 804 provides one or more user interfaces (UIs) that a user uses to access the functions of the compilation system 800 via the IDE client 810.

[0121] In some examples, IDE804, cache816, and one or more compilers such as compiler 1(824) and compiler N(820) are hosted on a single computing system. In some examples, IDE804, cache816, and one or more compilers such as compiler 1(824) and compiler N(820) are hosted in a cloud-based computing environment.

[0122] Figure 9 is a schematic representation of machine 900 on which an instruction 910 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause machine 900 to perform any one or more of the methods described herein. For example, instruction 910 can cause machine 900 to perform any one or more of the methods or processes described herein. Instruction 910 transforms a general-purpose, unprogrammed machine 900 into a specific machine 900 programmed to execute the described and illustrated generator in the described manner. Machine 900 may operate as a standalone device or may be connected to other machines (e.g., network connection). In a networked deployment, machine 900 may operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 900, in combination with other components of the compiler system, may operate as a server, client, computer, personal computer (PC), tablet computer, laptop computer, or any other machine capable of sequentially or otherwise executing instructions 910 that specify actions to be performed by Machine 900. Furthermore, although a single Machine 900 is shown, the term “machine” may be interpreted to include a set of machines that individually or collectively execute instructions 910 to perform any one or more of the methods described herein.

[0123] Machine 900 may include one or more processors 902, memory 904, and I / O device interfaces 906, which can be configured to communicate with each other via bus 932. In one example, the processors 902 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a composite instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 908 and 912 that execute instruction 910. The term “processor” is intended to include multicore processors, which may include two or more independent processors (sometimes called “cores”) capable of executing instructions simultaneously. Although Figure 9 shows multiple processors 902, machine 900 may include a single processor with a single core, a single processor with multiple cores (e.g., a multicore processor), multiple processors with single cores, multiple processors with multiple cores, or any combination thereof.

[0124] Memory 904 includes main memory 914, static memory 916, and storage unit 918, all of which are accessible to processor 902 via bus 932. Main memory 904, static memory 916, and storage unit 918 store instructions 910 that embody any one or more of the methods or generators described herein. Instructions 910 may also reside, fully or partially, in main memory 914, static memory 916, non-temporary machine-readable media 920 in storage unit 918, in one or more of processors 902 (e.g., in the processor's cache memory), or any suitable combination thereof during their execution by machine 900.

[0125] The I / O device interface 906 connects machine 900 to I / O devices 934. One or more of the I / O devices 934 may be components of machine 900 or may be separate devices. The I / O device interface 906 may include a wide variety of interfaces to I / O devices 934 used by machine 900 for purposes such as receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, and capturing measurements. The specific I / O device interface 906 included in a particular machine depends on the type of machine. It will be understood that the I / O device interface 906 of I / O devices 934 may include many other components not shown in Figure 9. In various examples, the I / O device interface 906 may include an output component interface 924 and an input component interface 928. The output component interface 924 may include interfaces to visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistors), and other signal generators. The input component interface 928 may include interfaces to alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing devices), tactile input components (e.g., physical buttons, touchscreens that provide the position and / or force of touch or touch gestures, or other tactile input components), and audio input components (e.g., microphones).

[0126] Communication can be implemented using a wide variety of technologies. The I / O device interface 906 further includes a communication component interface 930 that can operate to connect machine 900 to network 922 or one or more devices 936 via couplings 926 and 938, respectively. For example, the communication component interface 930 may include an interface to a network interface component or other suitable device for interface with network 922. In further examples, the communication component interface 930 may include an interface to a wired communication component, a wireless communication component, a cellular communication component, a near-field communication (NFC) component, a Bluetooth® component (e.g., Bluetooth® Low Energy), a Wi-Fi® component, and other communication components for communicating via other modalities. Device 936 may be another machine or one of a wide variety of peripheral devices (e.g., a peripheral device connected via USB).

[0127] Various memories (e.g., memory 904, main memory 914, static memory 916, and / or the memory of processor 902) and / or storage unit 918 may embody any one or more of the methods or generators described herein, or store one or more sets of instructions and data structures (e.g., software) used thereby. These instructions (e.g., instruction 910) cause processor 902 to perform various operations to implement the disclosed examples when executed.

[0128] Instruction 910 may be transmitted or received over network 922 using a transmission medium, via a network interface device (e.g., a network interface component included in communication component interface 930), and using one of several well-known transfer protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 910 may be transmitted or received over a transmission medium via coupling 938 to device 936 (e.g., peer-to-peer coupling).

[0129] Further examples include: Example 1 is a method of execution performed by a computer, which includes one or more processors receiving a kernel definition containing parameterization information and code for a set of generators written in a general-purpose programming language; one or more processors performing the following for each generator in the set of generators: translating the code for each generator into a first intermediate representation for each generator; one or more processors determining the settings for each generator using the parameterization information and the first intermediate representation; one or more processors generating a second intermediate representation using the settings; one or more processors caching the intermediate representations; one or more processors generating individual binary objects for a set of binary objects using the second intermediate representations; and one or more processors constructing a kernel corresponding to the kernel definition using the set of binary objects.

[0130] In Example 2, the subject of Example 1, caching the intermediate representation, includes detecting operations in the second intermediate representation, detecting regions within the body of the operations, generating parameterized content-addressable store identifiers (CAS IDs) using the content of the regions, copying the regions to a container, and caching the container in a hash table using the parameterized CAS IDs.

[0131] In Example 3, the subject of Example 2 involves determining the settings for each generator, which includes determining the region in the first intermediate representation, generating a parameterized CAS ID using the intermediate representation, and searching the hash table for the cached intermediate representation corresponding to the region using the parameterized CAS ID.

[0132] In Example 4, the subject of Examples 2-3, determining the settings for each generator, is performed across multiple machines. In Example 5, the subject of Examples 1-4 is that determining the settings for each generator includes generating a set of settings for each generator using parameterization information and an intermediate representation of each generator, generating a set of executable test functions using the set of settings, running the set of test functions to determine a set of individual performance scores, selecting the best setting from the set of settings using the set of individual performance scores, and determining the settings for each generator using the best setting.

[0133] In Example 6, the subject of Example 5 is further represented by the use of parameterized information of the target machine to generate a set of settings. In Example 7, the theme of Examples 5-6 is that a set of test functions is executed on multiple machines.

[0134] Example 8 is a machine comprising one or more processors and memory for storing instructions, wherein, when executed by one or more processors, the instructions cause the machine to perform the following actions: receive a kernel definition including parameterization information and code for a set of generators written in a general-purpose programming language; for each generator in the set of generators, perform the process of converting the code for each generator into a first intermediate representation of each generator; determine the settings for each generator using the parameterization information and the first intermediate representation; generate a second intermediate representation using the settings; cache the intermediate representations; generate individual binary objects for a set of binary objects using the second intermediate representations; and construct a kernel corresponding to the kernel definition using the set of binary objects.

[0135] In Example 9, the subject of Example 8 is caching an intermediate representation, which involves detecting operations in the second intermediate representation, detecting regions within the body of operations, generating parameterized content-addressable store identifiers (CAS IDs) using the content of the regions, copying the regions to a container, and caching the container in a hash table using the parameterized CAS IDs.

[0136] In Example 10, the subject of Example 9 involves determining the settings for each generator, which includes determining the region in the first intermediate representation, generating a parameterized CAS ID using the intermediate representation, and searching the hash table for the cached intermediate representation corresponding to the region using the parameterized CAS ID.

[0137] In Example 11, the subject of Examples 9-10, determining the settings for each generator, is performed across multiple machines. In Example 12, the subject of Examples 8-11 is that determining the settings for each generator includes generating a set of settings for each generator using parameterization information and an intermediate representation of each generator, generating a set of executable test functions using the set of settings, running the set of test functions to determine a set of individual performance scores, selecting the best setting from the set of settings using the set of individual performance scores, and determining the settings for each generator using the best setting.

[0138] In Example 13, the subject of Example 12 includes the fact that generating a set of settings is done using further parameterized information of the target machine. In Example 14, the theme of Examples 12-13 is that a set of test functions is executed on multiple machines.

[0139] Example 15 is a machine storage medium containing instructions, the instructions causing the machine, when executed by one or more processors of the machine, to perform the following operations: receive a kernel definition containing parameterization information and code for a set of generators written in a general-purpose programming language; for each generator in the set of generators, perform the following operations: convert the code for each generator into a first intermediate representation for each generator; determine the settings for each generator using the parameterization information and the first intermediate representation; generate a second intermediate representation using the settings; cache the intermediate representations; generate individual binary objects for a set of binary objects using the second intermediate representations; and construct a kernel corresponding to the kernel definition using the set of binary objects.

[0140] In Example 16, the subject of Example 15 is that caching an intermediate representation includes the steps of: detecting an operation in a second intermediate representation; detecting a region within the body of the operation; generating a parameterized content-addressable store identifier (CAS ID) using the content of the region; copying the region to a container; and caching the container in a hash table using the parameterized CAS ID.

[0141] In Example 17, the subject of Example 16 involves determining the settings for each generator, which includes determining the region in the first intermediate representation, generating a parameterized CAS ID using the intermediate representation, and searching in a hash table for the cached intermediate representation corresponding to the region using the parameterized CAS ID.

[0142] In Example 18, the subject of Examples 16-17, determining the settings for each generator, is performed across multiple machines. In Example 19, the subject of Examples 15-18 is that determining the settings for each generator includes generating a set of settings for each generator using parameterization information and an intermediate representation of each generator, generating a set of executable test functions using the set of settings, running the set of test functions to determine a set of individual performance scores, selecting the best setting from the set of settings using the set of individual performance scores, and determining the settings for each generator using the best setting.

[0143] In Example 20, the subject of Example 19 is further described as generating a set of settings using parameterized information of the target machine. In Example 21, the theme of Examples 19-20 is that a set of test functions is executed on multiple machines.

[0144] Example 22 is a machine-readable medium containing an instruction, the instruction causing the processing circuit to perform a process to carry out one of Examples 1 to 21 when executed by the processing circuit.

[0145] Example 23 is an apparatus that includes means for carrying out any of Examples 1 to 21. Example 24 is a system that implements one of Examples 1 through 21. Example 25 is a method of implementing any of Examples 1 through 21.

[0146] You may make changes and modifications to the disclosed examples without departing from the scope of this disclosure. These and other changes or modifications are intended to be included within the scope of this disclosure as expressed in the following claims.

[0147] Glossary "Carrier signal" means any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communication signals or other intangible medium for enabling the communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0148] "Client device" means any machine that interfaces to a communication network and obtains resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, desktop computer, laptop, personal digital assistant (PDA), smartphone, tablet, ultrabook, netbook, laptop, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communication device that a user may use to access the network.

[0149] "Communication Network" means one or more parts of a network that may be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), Internet, part of the Internet, part of the public switched telephone network (PSTN), conventional telephone service (POTS) network, mobile phone network, wireless network, Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or part of a network may include a wireless or cellular network, and the connection may be a code division multiple access (CDMA) connection, a Global System for Mobile Communications (GSM®) connection, or another type of cellular or wireless connection. In this example, the connection may implement any of the various types of data transfer technologies, including single-carrier radio transmission technology (1xRTT), Evolution Data Optimization (EVDO) technology, General-Purpose Packet Radio Service (GPRS) technology, Enhanced Data Rate for GSM® Evolution (EDGE) technology, 3G Partnership Projects (3GPP®) including 3G, 4G radio networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Interoperability for Microwave Access (WiMAX), Long-Term Evolution (LTE) standards, others defined by various standardization bodies, other long-range protocols, or other data transfer technologies.

[0150] "Machine-readable medium" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "machine-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0151] "Mechanical storage medium" means one or more storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions, routines and / or data. The term includes, but is not limited to, solid-state memory, optical media and magnetic media, including internal or external memory of a processor. Specific examples of mechanical storage medium, computer storage medium and / or device storage medium include non-volatile memory, including semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "mechanical storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "mechanical storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, some of which are covered under the term "signaling medium."

[0152] A "processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic running on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine code," etc.) and generates the associated output signals applied to operate a machine. A processor can be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a composite instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multicore processor having two or more independent processors (sometimes called "cores") that can execute instructions simultaneously.

[0153] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible mediums for enabling the communication of software or data. The term "signal medium" may be interpreted to include any form, such as modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal in which one or more of its properties are set or modified in order to encode information into a signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0154] A "kernel" or "microkernel" is an implementation of an algorithm that performs computations on memory objects, such as memory buffers, of a specific layout. While the two terms can be used interchangeably, "microkernel" tends to refer to smaller operations (e.g., memset, dot product, or reduction) within a larger generator kernel implementation. Algorithmically interchangeable / equivalent / substitutable kernels are sometimes referred to as "codelets" in the literature.

[0155] A "generator" is a metaprogram that is parameterized, executed, and produces a non-parametric implementation of a kernel or microkernel. A fixed kernel implementation (e.g., a panel-dot product implemented in assembly) is a degenerate case of a generator that does not have parameters.

[0156] A kernel interface declaration is a declaration of a kernel or microkernel that applies to multiple implementations of the kernel or microkernel. Kernels and microkernels can be implemented multiple times in several different ways. Interface declarations can exist independently of implementations and can allow clients and implementations to be type-checked.

[0157] "Generator parameter arguments" refer to values ​​that a kernel or microkernel can interact with. A generator is a metaprogram that generates kernels, and "parameters" are values ​​that this metaprogram can create.

[0158] The "kernel generator parameter result" is a value returned by the generator as a parameter to its source, allowing the generator to adapt to the behavior within the generated subkernel. For example, a panel dot product generator might return "processed a 3x5 panel of memory," causing the caller's for loop to advance by 3 and 5 in each dimension.

[0159] A "generator constraint" is a constraint that indicates a limitation on the parameters of a kernel or microkernel, such as "this implementation only works with dtype=float32", or "this only works on machines with the x86 VNNI extension", or "this works based on the remainder when the size is divided by 136". A generator may be a partial generator from an interface declaration to a concrete implementation. Constraints propagate upward from the kernel implementation to the generator graph.

[0160] "Kernel arguments" are static single assignment (SSA) argument values ​​used for buffers and other user-defined types to structured abstractions in memory, such as linear memory, N-dimensional tensors with layouts, trees and tables, and other higher-level data types; values ​​corresponding to op attributes at the tensor graph level (these values ​​may be modeled as constants there, but are dynamic in the kernel's runtime implementation); and very small microkernels at the bottom of the stack (e.g., adding two integers) that use arguments for their input.

[0161] A "kernel result" is an SSA result value, which is used for dynamically allocated result buffers, such as those with data-dependent shapes, and for very small microkernels at the bottom of the stack (e.g., adding two integers) that use the result for their output.

Claims

1. A method by which a computer performs an action. One or more processors receive a kernel definition that includes parameterization information and a set of generator code written in a general-purpose programming language. For each generator in the aforementioned set of generators, The process by which one or more processors convert the code of each generator into a first intermediate representation of each generator, The one or more processors perform a process of determining the settings of each generator using the parameterization information and the first intermediate representation, The process by which one or more processors generate a second intermediate representation using the settings, The one or more processors perform a process of caching the intermediate representation, The process by which one or more processors generate individual binary objects of a set of binary objects using the second intermediate representation, A method of execution by a computer, comprising the one or more processors constructing a kernel corresponding to the kernel definition using the set of binary objects.

2. Caching the aforementioned intermediate representation means To detect operations in the second intermediate representation, To detect a region within the main body of the aforementioned calculation, To generate a parameterized content addressable store identifier (hereinafter referred to as CAS ID) using the content of the aforementioned region, Copy the aforementioned region to the container. A method performed by a computer according to claim 1, comprising caching the container in a hash table using the parameterized CAS ID.

3. Determining the settings for each of the aforementioned generators is, Determining the region in the first intermediate representation, To generate a parameterized CAS ID using the aforementioned intermediate representation, The method performed by the computer according to claim 2, comprising searching the hash table for a cached intermediate representation corresponding to the region using the parameterized CAS ID.

4. The method described in claim 2, in which the settings of each of the generators are determined, is performed on multiple machines.

5. Determining the settings for each of the aforementioned generators is, Using the parameterized information and the intermediate representation of each generator, generate a set of settings for each generator. To generate a set of executable test functions using the aforementioned set of settings, The process involves executing the aforementioned set of test functions to determine a set of individual performance scores. Selecting the optimal setting from the set of settings using the set of individual performance scores, A method performed by a computer according to claim 1, comprising determining the settings for each of the generators using the aforementioned optimal settings.

6. The method performed by the computer according to claim 5, wherein generating the aforementioned set of settings is performed using parameterized information of the target machine.

7. The computer-based method according to claim 5, wherein the set of test functions is executed on multiple machines.

8. It is a machine, One or more processors, The system comprises one or more memories for storing instructions, and the instructions, when executed by the one or more processors, are sent to the machine. Receiving a kernel definition that includes parameterization information and a set of generator code written in a general-purpose programming language, For each generator in the aforementioned set of generators, A process of converting the code of each generator into a first intermediate representation of each generator, A process for determining the settings of each generator using the parameterized information and the first intermediate representation, A process to generate a second intermediate representation using the above settings, The process of caching the aforementioned intermediate representation, The process of generating individual binary objects of a set of binary objects using the second intermediate representation described above, A machine that uses the aforementioned set of binary objects to construct a kernel corresponding to the kernel definition.

9. Caching the aforementioned intermediate representation means To detect operations in the second intermediate representation, To detect a region within the main body of the aforementioned calculation, To generate a parameterized content addressable store identifier (hereinafter referred to as CAS ID) using the content of the aforementioned region, Copy the aforementioned region to the container. The machine according to claim 8, comprising caching the container in a hash table using the parameterized CAS ID.

10. Determining the settings for each of the aforementioned generators is, Determining the region in the first intermediate representation, To generate a parameterized CAS ID using the aforementioned intermediate representation, The machine according to claim 9, comprising searching the hash table for a cached intermediate representation corresponding to the region using the parameterized CAS ID.

11. The machine according to claim 9, wherein determining the settings of each of the generators is performed on multiple machines.

12. Determining the settings for each of the aforementioned generators is, Using the parameterized information and the intermediate representation of each generator, generate a set of settings for each generator. To generate a set of test functions that can be executed using the aforementioned set of settings, The process involves executing the aforementioned set of test functions to determine a set of individual performance scores. Selecting the optimal setting from the set of settings using the set of individual performance scores, The machine according to claim 8, comprising determining the settings for each of the generators using the aforementioned optimal settings.

13. The machine according to claim 12, wherein generating the set of settings is performed using parameterized information of the target machine.

14. The machine according to claim 12, wherein the set of test functions is executed on multiple machines.

15. A machine storage medium containing instructions, wherein the instructions, when executed by one or more processors of the machine, Receiving a kernel definition that includes parameterization information and a set of generator code written in a general-purpose programming language, For each generator in the aforementioned set of generators, A process of converting the code of each generator into a first intermediate representation of each generator, A process for determining the settings of each generator using the parameterized information and the first intermediate representation, A process to generate a second intermediate representation using the above settings, The process of caching the aforementioned intermediate representation, The process of generating individual binary objects of a set of binary objects using the second intermediate representation described above, A machine storage medium that allows the system to construct a kernel corresponding to the kernel definition using the aforementioned set of binary objects.

16. Caching the aforementioned intermediate representation means To detect operations in the second intermediate representation, To detect a region within the main body of the aforementioned calculation, To generate a parameterized content addressable store identifier (hereinafter referred to as CAS ID) using the content of the aforementioned region, Copy the aforementioned region to the container. The machine storage medium according to claim 15, comprising caching the container in a hash table using the parameterized CAS ID.

17. Determining the settings for each of the aforementioned generators is, Determining the region in the first intermediate representation, To generate a parameterized CAS ID using the aforementioned intermediate representation, The machine storage medium according to claim 16, comprising searching in the hash table for a cached intermediate representation corresponding to the region using the parameterized CAS ID.

18. The mechanical storage medium according to claim 16, wherein determining the settings of each generator is performed on multiple machines.

19. Determining the settings for each of the aforementioned generators is, Using the parameterized information and the intermediate representation of each generator, generate a set of settings for each generator. To generate a set of test functions that can be executed using the aforementioned set of settings, The process involves executing the aforementioned set of test functions to determine a set of individual performance scores. Selecting the optimal setting from the set of settings using the set of individual performance scores, The mechanical storage medium according to claim 15, comprising determining the settings of each generator using the aforementioned optimal settings.

20. The machine storage medium according to claim 19, wherein the generation of the aforementioned set of settings is performed using parameterized information of the target machine.

Images