Compiler cache

Abstract

The invention relates to a system for caching compiler transformations. A compilation system uses a parameterized hash in the form of a parameterized content addressable storage identifier (parameterized CAS ID) to store operator regions and any transformations to any operation of a compiler intermediate representation (IR). The parameterized CAS ID includes: a hash of the content of the region of operation; and a parameter set comprising a set of symbol references to objects used and / or referenced within the region of the operator.

Description

[0001] Priority requirements

[0002] This application claims the benefit of priority to U.S. Patent Application Serial No. 63 / 454,892, filed March 27, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to compilers, and more specifically to compilers for computationally intensive code. Background Technology

[0004] The compiler is used to generate object code from a high-level language. The goal is to generate optimized object code. Attached Figure Description

[0005] To facilitate the identification of any particular element or action in discussion, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced.

[0006] Figure 1 It is a data flow diagram based on some examples of kernel generation and usage processes.

[0007] Figure 2 It is a collaboration diagram of a compilation system based on some examples.

[0008] Figure 3 It is an activity graph based on some example kernel generation methods.

[0009] Figure 4 It is an activity graph that refines the generator methods based on some examples.

[0010] Figure 5A It is an activity graph based on some example caching methods.

[0011] Figure 5B An intermediate representation of the program based on some examples is shown.

[0012] Figure 5C The diagram shows a modified intermediate representation of the parameterized CAS ID of a cache region with a replacement intermediate representation, based on some examples.

[0013] Figure 5D The structure of a cache is shown based on some examples.

[0014] Figure 6A An intermediate representation of the cache is shown based on some examples.

[0015] Figure 6B The diagram shows a portion of the intermediate representation at time step 0, based on some examples.

[0016] Figure 6C The diagram shows a portion of the intermediate representation at time step 1, based on some examples.

[0017] Figure 6D The diagram shows a portion of the intermediate representation at time step 2, based on some examples.

[0018] Figure 6E The output intermediate representation is shown based on some examples.

[0019] Figure 7 The generator is shown in detail based on some examples.

[0020] Figure 8 This is a deployment diagram of a networked compilation environment based on some examples.

[0021] Figure 9 Based on some example machine architecture diagrams, within which instruction sets can be executed to enable the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0022] Compilers rely on tools like caching to provide compile-time memory operations, where intermediate compilation results of source files are stored in a cache for later reuse. For distributed compilation and builds, various tools (such as Bazel) are available, which rely on file-level hashes of file contents to determine which parts of the source code should be recompiled due to changes in the files. Using source files as the lowest granularity level leads to inefficient compilation because even trivial changes in source files (such as reformatting for readability or changes in comments) can trigger recompilation.

[0023] The compiler uses a proportional amount of memory to the operands in the program's intermediate representation (IR). For large programs, this could mean thousands of lines of code that are transformed into more than twice that amount of text in the IR. The compiler's job is then to make the IR traverse multiple stages and perform the transformations repeatedly. For languages ​​like C or C++, this is done at once for the entire transformation unit (file). This means that in a large file with many private functions, a change made in a single line of code, even just for a comment, will recompile the entire file because the compiler doesn't know what was actually changed or the context required for the recompilation.

[0024] By forming the basic atoms of computational symbolic operations (e.g., functions), the compiler no longer needs to recompile every function in a file if one of the functions changes. This significantly improves the cache hit rate of intermediate compiled results that are in-memory in the cache. Furthermore, IRs for other functions in the file are not required. By using a call graph, the compiler can statically determine which functions should be recompiled and avoid recompiling the entire file.

[0025] The examples in this disclosure provide examples of caching regions attached to the IR of an operation, rather than caching regions at the source function level. In some examples, multiple levels within the function IR are cached. In some examples, transformations are cached. Conceptually, caching transformations creates a Merkle tree based on the code and the transformations on the code.

[0026] In some examples, parameterized hashes are provided in the form of parameterized content-addressable storage identifiers (parameterized CAS IDs). A parameterized CAS ID provides a structure in which parts of the code object or "blob" to be hashed are not part of the hash content of the code object. This allows a set of operations to be performed on cached data without needing to retrieve the cached data from the cache at all. Example operations include, but are not limited to, weight updates for static neural network architectures, recompiling from large programs, and relinking individual functions.

[0027] In some examples, the use of parameterized CAS IDs reduces the depth of the Merkle tree required to uniquely identify cached functions (which results in more cache hits) and elevates the attribute to the top level of the program being compiled, where the attribute can be easily inserted.

[0028] In some examples, changing the compilation paradigm from "generating files" to "generating parameterized CAS IDs" means that distributed compilation becomes easier to implement by using parameterized CAS IDs as keys in a hash table to store the compilation artifacts in a distributed hash table.

[0029] Although described in relation to kernel compilation for performing various types of computations, it should be understood that the caching and compilation methods described herein can be applied to the processing of any type of computer program or code, or to the memorization of intermediate results for many types of multi-step or distributed computations.

[0030] Other technical features may be readily apparent to those skilled in the art based on the accompanying drawings, description, and claims.

[0031] Figure 1 This is a data flow diagram based on some examples of the compilation and execution process 100. The user creates the kernel definition 140 during the creation phase 120. (See reference...) Figure 3 A more comprehensive description is provided in compilation stage 122, as referenced. Figure 2 The more fully described compilation system 200a uses kernel definition 140 to combine the raw-level representation of the kernel 112.

[0032] Kernel definition 140 includes parameterization 136 and one or more generators 132. Generator 132 includes encoded logic, as shown by code representation 130 of generator 1 and code representation 134 of generator N, which, according to kernel definition 140, defines one or more operations to be executed by the kernel. Operations can be organized into encoded logic components, such as, but not limited to, operations, operators, functions, objects, routines, subroutines, modules, etc., that operate on one or more data buffers. In some examples, one or more generators 132 include definitions of one or more data structures containing data buffers.

[0033] Parameterization 136 includes a parameter set that instructs the compiler to compile kernel definition 140 (such as, ( Figure 2 The graph compiler 204 or kernel compiler 206 uses generator 132 to generate a primitive-level representation 112 of the kernel. In some examples, generator 132 is written in a general-purpose programming language (such as Python). In some examples, parameterization 136 is a set of kernel parameters. In some examples, parameterization 136 includes a parseable scripting language, etc., that generates kernel parameters.

[0034] During compilation phase 122, the code of generator 132 (such as code representation 130 of generator 1 and code representation 134 of generator N) is downgraded from the general programming language to lower-level representations of the generator, such as primitive-level representation 114 of generator 1 and primitive-level representation 116 of generator N, through a series of intermediate representations, as shown by intermediate representation 110 of generator 1 and intermediate representation 118 of generator N. During compilation phase 122, in reference to Figure 4 In a more comprehensive description, the optimal configuration of generator 132 is determined during searches (e.g., search 1 106 and search N 108) using intermediate representations of parameterization 136 and generator 132. Search results and other compilation metrics are stored in cache 128 for use during subsequent searches. A “search” may include static analysis using the previously configured parameters and, as referenced… Figure 4 A more comprehensive description of the refinement process involves a combination of dynamic analyses of the proposed configurations performed during the search for a tree of possible configurations.

[0035] The primitive-level buffer semantic representation of generator 132 is combined into the primitive-level representation 112 of the kernel. The primitive-level representation 112 of the kernel is stored for later use.

[0036] To generate an executable object, a kernel object 138 is generated using the kernel's primitive-level representation 112, and the kernel object 138 is included in the executable object 126. Once generated, the executable object 126 is executed during execution phase 124, and the kernel object 138 is used to perform one or more computations. In some examples, library 202 is used to enhance the kernel object 138 with an additional executable object.

[0037] In some examples, kernel execution metrics are stored in a data store for execution metrics 142. Execution metrics 142 are used by subsequent search components 102 to determine the optimal configuration of one or more generators 132.

[0038] In some examples, the artificial intelligence (AI) component 104 is used to assist in the search. In some examples, the AI ​​component 104 also assists during the authoring phase 120, during which the kernel is written within a software development environment (SDE) of an integrated development environment (IDE).

[0039] In some examples, one or more operations are defined by generator 132 as a fusion of several other lower-level operations (e.g., broadcast operations, activation operations), and sometimes as even larger fusion combinations such as Long Short-Term Memory (LSTM) operations. Describing the generator at this level of abstraction simplifies higher-level optimizations, such as, but not limited to, the extraction of shape operators and the generation of operator gradients.

[0040] In some examples, one or more generators 132 are used to generate implementations of existing functions or operators in existing machine learning (ML) frameworks (TFLite, TF, ONNX, PyTorch, etc.). Operators in existing ML frameworks have attributes such as, but not limited to, support for broadcasting and type generalization; handwritten operators known to be important for models of certain categories (e.g., activation operators fused into element-wise generators such as "addition"); support for quantization algorithms that depend on architecture-specific DSP operations; layouts assumed by the existing framework; support for dynamic shapes and dynamic data types, etc.

[0041] In some examples, the generator supports:

[0042] • Dynamic Shapes

[0043] • Broadcasting and type promotion: For example, "multiplication" is a binary generator, and two operands can have different shapes and data types. ML frameworks often improve usability by providing implicit promotion to common element types and supporting element broadcasting.

[0044] • Layout Modification: Some frameworks support multiple different layouts, such as row-major and column-major layouts, tiled layouts, etc. When input is in different formats, conversion may be necessary. Some libraries use stride to provide a common implementation that can work with many different layouts, but stride is not universal for tiled layouts.

[0045] • Type dispatch: The standard kernel library operates on multiple data types that are only dynamically known at kernel call time. This requires the kernel to dynamically dispatch on data types, and to dispatch to kernels specifically designed for many different data types. Some data types may have special cases; for example, "complex addition" can be handled by the same code path as "scalar addition" (because complex addition is element-wise), but "complex multiplication" is a completely different algorithm from "scalar multiplication".

[0046] • Thread partitioning: At the outer level of a type-specific kernel algorithm, computation is divided into blocks that can be executed in parallel by multiple threads. The size of each subunit needs to be determined and is typically optimally evaluated based on hardware characteristics and the size of the input data (rather than based on the number of available threads).

[0047] • Cache chunking: Within a per-thread computation, computation is often cache-blocked, for example, at the L2 level. The size of L2 is target-specific. It may be important for algorithms that pass data multiple times, but less important for element-wise operations with few reuses.

[0048] • Per-block algorithm: Within each L2 block, there are many ways to implement the kernel algorithm, including using scalars, vectors, prefetching, etc. There are also special cases of interest to handle when broadcasts are processed internally by the kernel (e.g., when the fastest changing dimension of an operand is broadcast).

[0049] • Many microkernels: Algorithms such as matrix multiplication depend on lower-level operations such as memset to clear the buffer, panel dot product, reduction, etc. These "microkernels" themselves can be implemented in many different ways.

[0050] • Macro algorithms: Many generators have a variety of completely different algorithms to compute the results. For example, in convolution, we see the im2col method, direct convolution, and Winograd. Matmul has many implementations, including the Strassen algorithm, etc. (especially when quantization and accelerators force strange data layouts).

[0051] • Hardware targets now typically feature space operations (such as Apple AMX or Intel AMX) that can accelerate multiple nested loops at once, for example, for matrix multiplication and large element-wise blocks. Hardware targets also have many architecture families that expect things to be register-blocked, pipelining, and unrolled in different ways.

[0052] In some examples, the raw-level representation of the kernel (112) is a component of the framework, which includes a code-generated kernel set that operates on memory buffers (such as, but not limited to, memory operators, 1D memory arrays, tensor buffers, user-defined data structures, etc.). In some examples, the kernel directly uses C / C++, assembly, and internal functions, depending on the specific hardware characteristics.

[0053] In some examples, kernel component libraries are used to generate additional kernels. For instance, buffer-level operators are used to replace those in the traditional kernel. Kernel components are modularized and reusable, including kernel algorithms such as, but not limited to, memory filling, reduction, element-wise operators, etc., in addition to more specialized primitives used in quantization kernels and other areas.

[0054] In some examples, the generator is a parameterized generator. Since it is difficult for humans to manually create and maintain all permutations of the kernel (e.g., for all data types, all target machines, etc.), they generally turn to metaprogramming. This metaprogramming takes various forms, such as C macros and ifdefs, Python generator frameworks, and "emitters" written in C++ for the "IRBuilder" compiler API, but the most widespread use is C++ templates.

[0055] In some examples, the kernel is defined as a declarative generator that takes kernel parameters and has arbitrary imperative logic encoded for those parameters, which is "burned" into the kernel's generated code. This can be used to specialize things like data types, expansion factors, vector lengths, cache sizes, etc. Most parameters are integer types and are constrained by ranges (e.g., expansion <= 8 times), lists of valid values ​​(e.g., vector lengths = 2, 4, 8, 16, 32), and should support enumeration (e.g., considering data types), making most parameters searchable. Using generators still allows for the use of specific kernels (e.g., fixed blobs in assembly) because they are efficient generators without parameters (or similarly fully constrained parameters).

[0056] The following is example code. The kernel may have bound parameters at its call point; for example, after a dynamic switch based on data type, the next-level microkernel is called, where the data type parameter is bound to a constant value:

[0057] / / This fills a 1D buffer with unknown length but known data type with ones.

[0058]

[0059] / / Fill a 1D buffer with unknown length and unknown data type using ones.

[0060]

[0061]

[0062] Figure 2 This is a block diagram of a compiler system 200a based on some examples. Compiler system 200a uses kernel definition 140 to generate software objects such as kernels. Kernel definition 140 includes parameterization 136 and one or more generators 132. Compiler system 200a uses the generated kernel 208 and the handwritten kernel 210 to generate an executable object 216 of Binary Executable Format (BEF) 218. Compiler system 200a uses kernel generation method 300 to generate the kernel using kernel definition 140.

[0063] In some examples, an executable object 216 comprising a binary executable format 218 of one or more kernel objects 138 is executed on a hardware device set 214 during runtime 212, and generates execution metrics 142 for optimizing generator configuration to optimize kernel performance.

[0064] Figure 3 It is an activity graph based on some example kernel generation methods, and Figure 4 This is generator refinement method 400. Compilation system 200a uses kernel generation method 300 to compile or generate a kernel. Although kernel generation method 300 and generator refinement method 400 describe a specific order of operations, the order can be changed without departing from the scope of this disclosure. For example, some of the described operations can be executed in parallel, in a different order, or in different components of a composable kernel compilation system, without substantially affecting the generator of the refinement process. In other examples, different components of the example apparatus or implementation of compilation system 200a can perform operations substantially simultaneously or in a specific order.

[0065] In operation 302, compilation system 200a receives kernel definition 140 including parameterization 136 and one or more generators 132, said generators 132 including coded logic defining the kernel. In some examples, the code includes generator code written in a general-purpose programming language.

[0066] In operations 304 and 306, refer to Figure 4 In a more comprehensive description, the compiler system 200a uses parameterization 136 for each generator to determine the optimal configuration for the generator.

[0067] In operation 308, compiler 200a generates a primitive-level buffer semantic representation of the generator using the generator's optimal configuration. For example, compiler 200a downgrades the generator to a primitive-level buffer semantic representation by using one or more successive compilation stages of intermediate representations.

[0068] In operation 310, the compilation system 200a adds the generator's primitive-level buffer semantic representation to the generator's primitive-level buffer semantic representation set. The generator's primitive-level buffer semantic representation set is used to combine the kernel's primitive-level buffer semantic representation.

[0069] In operation 312, the compilation system 200a uses the generator's primitive-level buffer semantic representation set to combine the primitive-level buffer semantic representation of the kernel corresponding to the input generator. For example, the compilation system 200a obtains the generator's primitive-level buffer semantic representation set and slices the generator's primitive-level buffer semantic representation and its dependent code into individual modules or kernels.

[0070] In operation 314, the compilation system 200a reduces a single module to one or more object (.o) files and stores one or more object files of the kernel in the data store (such as, but not limited to, CAS) of the generated kernel 208. In some examples, the object files have the object file format that a standard C-style toolchain would produce, and therefore work seamlessly with a stack that implements the C / C++ External Generator Interface (FFI).

[0071] Figure 4 This is an activity diagram based on some examples of the generator refinement method 400. Compiler system 200a uses generator refinement method 400 to generate permutations of generator configurations, which are evaluated by compiler system 200a to determine the optimal configuration of the generator. While generator refinement method 400 depicts a specific order of operations, the order can be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be executed in parallel, in a different order, or in different parts of compiler system 200a, without substantially affecting the generator of the refinement process. In other examples, different parts of the example apparatus or implementation of compiler system 200a may execute operations substantially simultaneously or in a specific order.

[0072] In operation 402, compilation system 200a uses an evaluator associated with the generator to search for the generator's optimal configuration. Kernel compiler 206 is capable of performing both static and dynamic analysis searches for the generator's optimal configuration. In the static analysis search, kernel compiler 206 uses search unit 102 to search several different types of data stores. One type of data store is a cache 128 containing the generator's optimal configuration, which can be reused by kernel compiler 206 to determine the generator's optimal configuration when the generator is degraded during the compilation process. Cache 128 can be a local cache or a distributed cache distributed across remote storage nodes on one or more servers. For example, compilation system 200a maintains the optimal configuration data store in cache 128. Search unit 102 uses an evaluator to find the generator's optimal configuration, which is the metric by which search unit 102 determines that the generator's configuration is optimal.

[0073] In some examples, the cache includes a hash table. The hash table comprises a region of intermediate representations of operations generated using a generator. This region of intermediate representations is stored in the hash table using parameterized CAS IDs, as shown in the reference. Figure 5A , Figure 6A and Figure 7 To describe it more comprehensively.

[0074] In operation 404, the compilation system 200a determines whether the optimal configuration was found during the search of cache 128.

[0075] In response to the determination that no optimal configuration for the generator was found during the static analysis search, kernel compiler 206 uses dynamic analysis of the generator to perform a search. To this end, in operation 406, kernel compiler 206 generates a set of configurations. For example, compilation system 200a generates an intermediate representation of the generator. Compilation system 200a uses the intermediate representation of the generator and parameterization 136 to generate one or more configurations of the generator as one or more test intermediate representations of the generator.

[0076] In operation 408, compiler 200a uses one or more intermediate test representations to generate a set of executable test functions. For example, for each intermediate test representation, compiler 200a downgrades the intermediate test representation into an executable object of BEF to generate executable test functions.

[0077] In operation 410, compilation system 200a executes a set of test functions to determine a corresponding set of performance scores. For example, a composable kernel compilation system executes each test function and monitors its performance as it operates on a test suite of data. In some examples, the performance scores include an initialization score, indicating the amount of time used by the test function during its initialization. In some examples, the performance scores include an execution score, indicating the amount of time spent by the test function operating on the test dataset. In some examples, the performance scores include the amount of time the test function spends communicating with other generators in the kernel during execution.

[0078] In operation 412, compiler system 200a uses the corresponding performance score set to select the best configuration from the configuration set. For example, kernel compiler 206 assigns weights to each set of generator, configuration, and performance data. During the selection process, compiler system 200a uses the sets of function, configuration, and performance evaluation data and their associated weights to select the generator configuration.

[0079] In operation 414, compilation system 200a uses optimal configuration to generate an intermediate representation of the generator.

[0080] In operation 416, the compilation system 200a caches the generator's optimal configuration in a cache for later search processes. For example, the cache includes a hash table. The hash table contains regions of intermediate representations of operations generated using the generator's optimal configuration. These regions of intermediate representations are stored in the hash table using parameterized CAS IDs, as shown in reference [reference missing]. Figure 5A , Figure 6A and Figure 7 To describe it more comprehensively.

[0081] In operation 418, compilation system 200a returns the optimal configuration of the generator.

[0082] In some examples, the generated configuration set is also parameterized based on the target machine.

[0083] In some examples, the test function set is executed on multiple machines.

[0084] In some examples, the performance score includes a combination of execution time and load time.

[0085] In some examples, generating a test function set involves: selecting a generator configuration library from a set of libraries using the configuration; and generating test functions from the test function set using the selected library and configuration.

[0086] In some examples, library sets include user-defined library sets and system-defined library sets.

[0087] In some examples, the generator configurations in the library set are stored in an intermediate language.

[0088] In some examples, the generator is initially defined in a programming language other than a general-purpose programming language and then downgraded to an intermediate language.

[0089] In some examples, cache128 can be searched using both generator parameterization and target machine parameterization.

[0090] In some examples, determining the generator's configuration involves using parameterization and target machine configuration to search the cache 128 to find the optimal configuration for the generator.

[0091] In some examples, cache 128 is distributed across multiple storage nodes, and searches are performed on distributed storage nodes.

[0092] In some examples, the runtime performance dataset collected from the set of execution functions during execution is stored in cache 128, where each execution function is associated with a known configuration and known generator parameterization.

[0093] In some examples, performance data includes communication data between subsets of executed functions.

[0094] In some examples, determining the generator's configuration involves using a machine learning model of AI component 104 to determine the configuration, which is trained on a runtime performance dataset collected during execution from a set of execution functions, where each execution function is associated with a known configuration and known parameterization.

[0095] In some examples, the generator code is transformed, the configuration is determined, the executable object is generated, and the kernel is combined and executed on two or more machines.

[0096] In some examples, the kernel's raw level representation 112 is stored in a data store that can be accessed over the network.

[0097] In some examples, the raw level representation 112 of the kernel can be combined with other kernels in the kernel library.

[0098] In some examples, there are no dependencies between one or more generators in the kernel, so the compiler system 200a can process the generators in parallel. This structure (and the general tree / forest / DAG structure of computation) facilitates the parallelism of the kernel compilation process, which can be used to accelerate kernel generation on one or more multi-core machines.

[0099] In some examples, kernel authors declare their own abstractions, as in C++. To this end, compiler system 200a provides interface declarations for the (micro)kernel and supports many different implementations for each microkernel—each implementing a public interface. Each kernel can be recursively defined using simpler, smaller kernels, which themselves can have multiple different implementations.

[0100] In some examples, there are multiple available implementations for each kernel, microkernel, generator, and compiler. Compiler system 200a determines which is optimal for a given target and scenario (data type, size category, etc.). Therefore, the (micro)kernel interface declaration defines the cost model for optimization through a search (e.g., finding a configuration with the "best implementation FLOPS"). For example, microkernel implementations could include those using scalar operations, those implemented with multiple SIMD generators of varying lengths, some implemented in inline assembly, and possibly those implemented with Apple AMX. Compiler system 200a empirically selects the configuration with the highest throughput for the current hardware by measuring current hardware (implementations for incompatible systems are ignored due to their infinite cost).

[0101] In some examples, the search is achieved by constructing a large collection of executable objects 126 that realistically use the generator. This allows the build system 200a to collect data for one or more execution metrics 142. For example, metrics are collected for executable objects 126 that include the model. Metrics include tensor input size and execution time metrics similar to the mmperf “benchmark size” list (using real input dimensions instead of random input dimensions). In some examples, profiles are collected and used, or certain dimensions are more heavily weighted, to achieve goals such as “prioritizing MLPerf performance” or “generating the best possible code for a model,” depending on the goals of any particular product.

[0102] In some examples, the parameters for parameterization 136 are not specified. These parameters are explored and determined by compiler system 200a during the search. For example, compiler system 200a determines the number of iterations that will fit the cache, and compiler system 200a returns the result as a parameter result to allow the enclosed generator to be tiled or parallelized around that result. As another example, given an element-wise multiplication microkernel implemented on 1D memory blocks in terms of vectors, a loop utilizing one of these low-level generators will increase in terms of FLOPS until it exceeds the L2 cache, at which point the cache-blocking algorithm above is generally more efficient. Allowing the kernel to define metrics (e.g., FLOPS) will allow the search to find the correct implementation. Top-level generator kernels can use latency as their metric.

[0103] In some examples, because some generator parameters (e.g., data types) are defined on the generator interface (and are therefore shared across all implementations), compiler system 200a also provides implementations of the generator with additional parameters (e.g., the kernel's ARM implementation provides three implementations of the same generator for different microarchitectures). This is equivalent to a "simplification measure" used to "flatten" these parameters into different separate implementations of the same microkernel.

[0104] In some examples, there are multiple implementations of each microkernel, which are then implemented based on other interfaces that may have many implementations. These extensions form a tree of possible extensions, and, since there may be many top-level generators in the framework, there is a forest of extensions working at many levels of abstraction. For example, a matrix multiplication microkernel can be implemented using a three-level for loop, through cache blocking, and through internal L2 tiling. It can also be implemented using a target-specific dot product operation, as well as through a 2D generator and common accelerators. Each of these can be implemented independently of each other, all implementing the same interface. For a single framework generator, each “extension tree” may have an exponential number of possible extensions. This makes searching the entire space for a single kernel impractical, and supporting the entire ML framework even more challenging, especially when a single framework may have hundreds / thousands of individual kernels.

[0105] In some examples, human-created constraints are defined at the kernel level as fundamental limitations in parameter declarations to cut off the search space or guide exploration. In some examples, conditional constraints are provided. In some examples, redundancy in the tree-based structure is utilized through dynamic programming techniques. Dynamic programming uses memoization / caching of subproblems to improve the performance of hierarchical tree-based algorithms. In some examples, each extended tree will have many common leaves, and the forest will have many shared leaves, subtrees, and possibly the entire kernel. By allowing cost models to be defined at many levels (not just at the top-level framework generator), Compilation System 200a leverages modularity for searching and can cache results. The use of dynamic programming collapses the "extended tree" into a directed acyclic graph (DAG).

[0106] In some examples, the cache is hosted on a cloud service to provide users with the oracle, allowing them to access previously searched algorithms offline. This allows users to avoid searching the entire algorithm on their devices. In some examples, the build system 200a generates analytics about what users are using it for. In some examples, the mobile framework's install size might be very small, and instead of releasing the typical kernel library with a bloated kernel, the build system 200a provider rolls out a just-in-time (JIT) compiler that can generate the kernel. Users might not want to perform searches on their devices, so the build system 200a provider can bundle the binary blob with the application or add logic to download the correct kernel parameters for the target hardware and generate / cachise machine code against the kernel at app install time, using the compiler as a "compression scheme" to reduce the impact of kernel library download size.

[0107] In some examples, the generator tree expansion at each level is functional (without side effects), and the "key" used for looking up computations is coded in such a way that the compilation system 200a can hash and look up the results of the transformations (e.g., the key is a blob of serialized MLIR). This is useful for parallelizing tree compilation (trees / DAGs have a lot of parallelism).

[0108] In some examples, kernels capable of arbitrary element-wise computation are fused into matrix multiplication. Compiler 200a supports this by allowing generators to be parameterized by region. A region is simply a different form of parameter argument, which is passed down the code body and can be accessed by metaprogramming constructs. For example, exposing regions as general features in Compiler 200a allows operations that are defined in the system itself (such as "switching on data types" and "expressing statically unrolled loops using this parameter") to be hardcoded into the system. This allows Compiler 200a to be user-expandable because nothing on the stack is specific to dense linear algebra; users can build their own generator libraries that partition data tables or trees, interact with their own external storage devices (e.g., databases, etc.).

[0109] In some examples, compiler system 200a utilizes algorithm skeletons to allow the description of higher-order transformations, which enables the coding of parallel modes in a reusable manner. This simplifies the implementation task by the fact that each skeleton can be considered independently, contrasting with the monolithic programming interface of existing systems at similar levels of abstraction.

[0110] In some examples, generators are allowed as partial generators from interface declarations to concrete implementations. Constraints indicate restrictions on their parameters, such as "this implementation only works with data type=float32", or "this only works with machines with x86 VNNI extensions", or "this works with size modulo 136", etc. In some examples, the kernel implementation propagates upwards to the generator graph.

[0111] In some examples, compiler 200a uses an intermediate representation of the kernel description, which is a machine-analyzable / transformable format. In other examples, compiler 200a extracts the shape generator from the kernel description by extracting computations through code slicing. This ensures that compiler 200a has a single true source: kernel + shape generator.

[0112] In some examples, compiler system 200a implements generators using the Multi-Level Intermediate Representation (MLIR) compiler API to provide more complex structures than parameterized extensions. The generators are encoded as compiler transformations and provide users with a flexible programming model. These are generators that take regions of intermediate representations as parameters and produce new parameters.

[0113] In some examples, the compilation system 200a extracts metadata about operations, such as whether the operation is associated or whether it produces side effects.

[0114] In some examples, the kernel generated by build system 200a acquires the output buffer as an argument that may not be exposed to the graph. Build system 200a provides a graph-level representation of "buffer exposure," which allows for memory planning, in-situ optimizations for cascading, and more.

[0115] In some examples, the compilation system 200a retrieves metadata about the buffer-level generator implementation and reflects it back to the generator graph level.

[0116] In some examples, the compiler system 200a uses a Python-like language, which is a user-extensible hybrid declarative / imperative programming language that allows arbitrary MLIR generator graphs to be represented in a usable manner.

[0117] Figure 5A It is an activity graph based on some example caching methods 500. Figure 5B The intermediate representation 566 of the program, based on some examples, is shown. Figure 5C The modified intermediate representation 522, with the parameterized CAS ID of the cache region replacing intermediate representation 566, is shown. Figure 5D The structure of a 544-bit cache is shown.

[0118] Although example caching method 500 depicts a specific order of operations, the order can be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be executed in parallel or in a different order without substantially affecting the functionality of caching method 500. In other examples, different components of compiler system 200a may perform operations substantially simultaneously or in a specific order.

[0119] The intermediate representation 566 comprises one or more components, such as a model, i.e., model "baz" 530, and an operation set, i.e., function "foo" 524, function "bar" 526, and operation "someop" 528. The memory footprint of the intermediate representation 566 is almost entirely contained within the bodies of the operations (e.g., body 540), and the compiler system 200a only caches the region of the intermediate representation 522 that is suspending modifications to these operations, because the goal of the compiler system 200a is to recompile only the parts of the program in the intermediate representation 566 that have changed since the last time the program was compiled.

[0120] exist Figure 5A In operation 502, the compilation system 200a receives intermediate representation 566.

[0121] In operation 504, compiler system 200a detects one or more operations in intermediate representation 566. For example, compiler system 200a parses intermediate representation 566 to detect portions in intermediate representation 566 that indicate the beginning of the definition of an operation, such as, but not limited to, the strings “func”, “%”, “model”, etc.

[0122] In operation 506, for each of the components of the modified intermediate representation 522 (e.g., function “bar” 526), ​​the compilation system 200a separates the symbolic operation of the operation (e.g., symbolic operation 520) from its body (e.g., body 540) and replaces the symbolic operation itself with a replacement symbolic operation in the modified intermediate representation 522 (e.g., replacement operation 532), which represents symbolic operation 520 in the intermediate representation 566.

[0123] In operation 508, compiler system 200a determines a set of regions within the body of the operation. For example, function “foo” 524 does not reference any other functions, therefore function “foo” 524 comprises a single region, namely the body of function “foo” 524. Function “bar” 526 comprises a single reference to function “foo” 524, therefore function “bar” 526 comprises a region, namely the body of function “bar” 526. In a similar manner, the body of operation “someop” 528 comprises a single region. Model “baz” 530 comprises two regions as indicated by “init” region 548 and “execute” region 550, the “init” region 548 comprising a reference to function “foo” 524, and the “execute” region 550 comprising calls to function “foo” 524 and function “bar” 526.

[0124] In operation 510, the compiler system 200a detects calls in regions. For example, a single region of function "foo" 524 does not contain any calls. A single region of function "bar" 526 contains a reference to function "foo" 524. A single region of operation "someop" 528 contains a reference to function "foo" 524. The "init" region 548 of model "baz" 530 contains a call 542 to function "foo" 524. The "execute" region 550 of model "baz" 530 contains a reference to function "foo" 524 and a reference to operation "someop" 528.

[0125] In operation 512, compiler system 200a uses a hash function, the contents of a region, and the detected call to generate a corresponding parameterized CAS ID for each region. For example, for function "foo" 524, compiler system 200a generates parameterized CAS ID 552 by hashing the contents of a single region of function "foo" 524 to create "HASHFOO". Function "foo" 524 does not reference any other functions, therefore there are no symbolic references in parameterized CAS ID 552.

[0126] For function “bar” 526, compiler system 200a hashes the contents of a single region of function “bar” 526 to create a hash “HASHBAR” for parameterized CAS ID 536. Function “bar” 526 calls function “foo” 524, so compiler system 200a adds a reference to function “foo” 524 to symbolic reference 538.

[0127] For operation "someop" 528, compiler system 200a hashes the contents of the unique region of operation "someop" 528 to generate the hash "HASHSOMEOP" and adds it to the parameterized CAS ID 554. Operation "someop" 528 calls function "foo" 524, so compiler system 200a adds a reference to function "foo" 524 to symbolic reference 556.

[0128] Model “baz” 530 comprises two regions, an “init” region 548 and an “execute” region 550. Compiler system 200a generates a parameterized CAS ID 558 for the “init” region 548 and a parameterized CAS ID 562 for the “execute” region 550. Compiler system 200a hashes the contents of the “init” region 548 to generate a hash “HASHBAZINIT”, and adds this hash to the parameterized CAS ID 558. Since the “init” region 548 contains a reference to the function “foo” 524, the reference to the function “foo” 524 is added to the symbolic reference 560 of the parameterized CAS ID 558. In a similar manner, the compilation system 200a hashes the contents of the “execute” region 550 to generate the hash “HASHBAZEXEC”, which is added to the parameterized CAS ID 562, and references to the functions “foo” 524 and “bar” 526 are added to the symbolic reference 564 of the parameterized CAS ID 562.

[0129] In operation 514, the compilation system 200a copies or moves the contents of the region to the corresponding container (such as cache container 546), which uses the corresponding parameterized CAS ID as the key in the hash table to cache it.

[0130] In operation 516, compiler system 200a replaces the operator regions with their corresponding parameterized CAS IDs. For example, the region of function “foo” 524 is replaced with parameterized CAS ID 552, the region of function “bar” 526 is replaced with parameterized CAS ID 536, the region of operation “someop” 528 is replaced with parameterized CAS ID 554, and two regions of model “baz” 530 are replaced with parameterized CAS ID 558 and parameterized CAS ID 562.

[0131] In operation 518, the compilation system 200a returns the modified intermediate representation 522.

[0132] In some examples, each region has a list of symbols it references (such as referenced symbol 534), and an index into that list used in symbol reference 538. In some examples, because symbolic references can be used anywhere, including for other properties, a special property is used to reference the region meta-parameter, which could be a symbol binding, etc. In some examples, symbolic references can be used for other properties, such as, but not limited to, constant data hashes.

[0133] In some examples, the user chooses how to parameterize the parameterized CAS ID at a more granular level. In other examples, a dialect interface is provided that the user can specialize to convert attributes into symbolic attributes, which the caching code then converts from an arbitrary attribute into an index in the top-level parameter list on the parameterized CAS ID.

[0134] Parameterized CAS IDs and parameterized hash natural representations are used to preserve the call graph. Symbols with calls (and therefore callees) are parameterized based on the callee, so if the callee changes in a way that does not affect the caller, the caller does not need to be recompiled. This increases cache hit rate and effectively truncates traversal.

[0135] In some examples, performing analysis or transformation requires inflating the modified intermediate representation 522 back to its original state. This operation is easily reversible, and since the IR is cached in a hash table, the compiler 200a only inflates the operations used by the compiler 200a to perform analysis or transformation.

[0136] In some examples, operations without regions will not be cached because the main body of various regions will be most of the operations in the intermediate representation.

[0137] In some examples, the caching method 500 is fully recursive. For instance, a SymbolTable operation contains symbols, and compiler 200a can cache regions of the SymbolTable, with parameterized CAS IDs being parameterized about the symbols contained in the region. Cached objects will have references to symbols rather than calls to symbols—and will extend to cached symbols rather than calls to operations.

[0138] Figures 6A to 6E The transformation of intermediate representations during a distributed side effect analysis process, based on some examples, is illustrated. Figure 6A The intermediate representation of cache 602 is shown based on some examples. Figure 6B This shows a portion of the intermediate representation at time step 0.604. Figure 6C This shows a portion of the intermediate representation at time step 1606. Figure 6DThe intermediate representation at time step 2608 is shown, and Figure 6E The output intermediate representation 610 is shown. The use of parameterized CAS IDs provides cache-aware transformations. For example, in the side-effect inference phase of LLVM or invoking the strongly connected component (SCC) phase of a graph.

[0139] Figure 6A An intermediate representation 602 is shown, consisting of two functions (function "foo" 612 and function "bar" 614) and a model "baz" 616. The regions of function "foo" 612, function "bar" 614, and model "baz" 616 are cached using cache locations pointed to by parameterized CAS IDs 618, 620, 622, and 624.

[0140] The side effect inference phase runs from bottom to top on the call graph. The side effect inference phase is able to read the previous analysis of function “foo” 612 from the cache using the analysis to be performed as the key and the side effect attributes as values ​​in a hash table. After this, side effect inference is run for three regions: the body of function “bar” 614 as indicated by parameterized CAS ID 620, and two regions of model “baz” 616 as indicated by parameterized CAS ID 622 and parameterized CAS ID 624. In some examples, the side effect inference phase runs in parallel, and therefore has multiple processes performing inference, for example, task 1 (T1) 626 performs inference on the body of function “bar” 614, task 2 (T2) 628 performs inference on the first region of model “baz” 616, and task 3 (T3) 630 performs inference on the second region of model “baz” 616. T1 and T2 can run in parallel, and T3 depends on T1.

[0141] At time step 0 604, this stage has a cache hit on function “foo” 612 and a side effect “read” 632. Task 1 (T1) 626 bloats function “bar” 614. And Task 2 (T2) 628 partially bloats model “baz” 616.

[0142] Reference Figure 6C At time step 1606, task 1 (T1) 626 finds that function "bar" 614 has no additional side effects, only those from ( Figure 6B The side effects of function "foo" 612. Therefore, task 1 (T1) 626 removes the body of function "bar" 614 and replaces cache attribute 634.

[0143] Task 2 (T2) 628 discovers that the initialization region of model "baz" 616 (as exemplified by the parameterized CAS ID 636 "HASHBASINIT") has additional side effects. Therefore, Task 2 (T2) 628 modifies the properties and compresses the first region. Task 1 (T1) 626 has been solved, so Task 3 (T3) 630 expands the second region of model "baz" 616.

[0144] Reference Figure 6D Task 3 (T3) 630 also discovered additional side effects on the second region of model “baz” 616, so Task 3 (T3) 630 modified attribute 638 and compressed the second region of model “baz” 616 as illustrated by parameterized CAS ID 640 “HASHBAZEXEC”.

[0145] Reference Figure 6E The intermediate output 610 shows that all side effect attributes can be cached by combining the performed analysis (side effect inference) with the symbolic operation 644 name used for parameterization CAS ID 646 and parameterization CAS ID 648 and saving the resulting attribute as value 642.

[0146] In some examples, because cache transformations are fully reversible, integration of non-cache-aware transformations can be achieved by bloating the relevant parts before running the stage. In some examples, a traditional stage can be viewed as a black box: operations are keyed to have region attributes that operate in combination with the stage name, and its output is simply cached.

[0147] Figure 7 The refinement of the generator is illustrated based on some examples. The refiner component of a compilation system (such as Compiler 200a) can be found in references... Figure 4 A more comprehensive description of the search generator is performed during the optimal configuration of the generator.

[0148] In some examples, to perform refinement, the refiner component performs a call graph SCC phase during a bottom-up traversal of the call graph and expands the generator into a function. In some examples, the refiner component is viewed as cache-aware to fully leverage the parallelism that can be obtained by using a distributed cache with parameterized CAS IDs. A cache-aware refiner employs a compressed generator, which may inflate it back to the original generator, perform the expansion, and compress the resulting function. For example, generator 702 includes two generators, "@foo" generator 710 and "@bar" generator 712. The generator includes parameterized CAS IDs (such as parameterized CAS ID 730 and parameterized CAS ID 732) and a set of attributes that includes a collection of input metadata (such as input metadata 708 and input metadata 722). During refinement 706, the refiner component uses properties to generate a set of test functions 704 (such as "@foo" test function 714, first "@bar" test function 716, second "@bar" test function 718, and third "@bar" test function 720).

[0149] Since the input metadata 708 of the "@foo" generator 710 contains an empty metadata dataset, the refinement of the "@foo" generator 710 results in a single test function, namely the "@foo" test function 714. Because of the existence of this single test function for the "@foo" generator 710, the parameterized CAS ID 730 of the "@foo" generator 710 is copied into the "@foo" test function 714.

[0150] However, since the input metadata 722 of the "@bar" generator 712 includes an indexable size parameter 740 as part of the input metadata 722, the refiner component generates a set of test functions, where each test function has a unique size parameter such as size parameter 724, size parameter 726, and size parameter 728. Because the bodies of the test functions are in different regions, these regions are cached in separate caches accessed using parameterized CAS IDs, as indicated by parameterized CAS ID 734, parameterized CAS ID 736, and parameterized CAS ID 738.

[0151] In some examples, the advantage of caching regions instead of operations with regions is that the generator interface does not need to be cached. Generator interfaces do not have bodies and therefore do not need to be cached. A cache-aware refiner component can detect compressed generators that implement the interface and, in the event of a cache miss, can inflate the compressed generator back into the original generator to perform a search and / or expansion.

[0152] Figure 8This is a collaboration diagram based on some examples of a networked compiler system 800. Compiler system 800 includes one or more computing systems, such as computing system 1 822 to computing system N 802, communicating via one or more networks (such as network 814). In some examples, network 814 is a local area network (LAN). In some examples, network 814 is a wide area network (WAN) such as the Internet.

[0153] A computing system includes one or more computing machines such as Figure 9 Machine 900. One or more computing systems host one or more compilers, such as computing system 1 822 hosting compiler 1 824 and computing system N 802 hosting compiler N 820. Each of the compilers is communicatively coupled to other compilers of compiler system 800 (e.g., hosted on corresponding other computing systems) via one or more communication networks 814, including a local area network (LAN). The compilers may also communicate with locally hosted applications on their respective computing systems using application programming interfaces (APIs).

[0154] The compiler system 800 also includes a cache 816 communicatively coupled to the compiler via one or more networks (such as network 814).

[0155] The compilation system 800 also includes an integrated development environment server (IDE) 818 that hosts the IDE 804. The IDE 804 is communicatively coupled to the compiler via one or more communication networks (such as network 814).

[0156] The compiler interacts with other compilers and with the IDE 804 via Network 814. The data exchanged between the compiler and the IDE 804 includes functions (e.g., commands that call functions) and payload data (e.g., the coding logic used for compilation).

[0157] Client computing system 808 hosts IDE client 810, which is communicatively coupled to IDE 804 via one or more communication networks (such as network 806). In some examples, network 806 is a LAN. In some examples, network 806 is a WAN such as the Internet.

[0158] Users communicate with the IDE 804 using the IDE client 810 and write coded logic that is compiled by the compiler. During compilation, the compiler can access the cache 816 to store an intermediate representation of the coded logic and / or an executable object. In some examples, the IDE client 810 communicates with the client cache 812. The compiler can access the client cache 812 via the IDE client 810 during compilation to store an intermediate representation of the coded logic and / or an executable object.

[0159] Compiler system 800 provides server-side compilation functionality, as described herein, to IDE client 810 via network 806. While some functionality of compiler system 800 is described herein as being performed by compilers (such as compiler 1 824 and compiler N 820), IDE 804, or one or more client-side APIs or applications, the location of certain functionality within compiler system 800 or client computing system 808 may be a design choice. For example, it may be technically preferred that specific technologies and functions are initially deployed within client computing system 808, but that technology and functions are later migrated to compiler system 800, where the computing system of compiler system 800 (such as computing system N 802) has sufficient processing power.

[0160] The compiler system 800 supports various services and operations provided to the client computing system 808. Such operations include transmitting data to and from the compiler system 800 and the client computing system 808, receiving data from them, and processing data generated by them. This data may include, but is not limited to, coding logic, intermediate representations of the coding logic and / or executable objects, compilation metrics, execution metrics of one or more executable objects, etc. The IDE 804 provides one or more user interfaces (UIs) via the IDE client 810 for users to access the functionality of the compiler system 800.

[0161] In some examples, the IDE 804, cache 816, and one or more compilers (such as compiler 1 824 and compiler N 820) are hosted by a single computing system. In other examples, the IDE 804, cache 816, and one or more compilers (such as compiler 1 824 and compiler N 820) are hosted in a cloud-based computing environment.

[0162] Figure 9This is a schematic representation of machine 900, within which instructions 910 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 900 to perform any or more of the methods discussed herein. For example, instructions 910 can cause machine 900 to perform any or more of the methods or processes described herein. Instructions 910 transform a general, unprogrammed machine 900 into a specific machine 900 programmed to perform the described and illustrated generator in the described manner. Machine 900 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 900 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 900, in combination with other components of the compiler system, can act as, but is not limited to, a server, client, computer, personal computer (PC), tablet computer, laptop computer, or any machine capable of sequentially or otherwise executing instructions 910 that specify the actions to be taken by machine 900. Furthermore, while a single machine 900 is shown, the term "machine" can also be considered as a collection of machines that individually or jointly execute instructions 910 to perform any or more of the methods discussed herein.

[0163] Machine 900 may include one or more processors 902, memory 904, and I / O device interface 906 that can be configured to communicate with each other via bus 932. In the example, processor 902 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 908 and 912 that execute instruction 910. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 9 Multiple processors 902 are shown, but machine 900 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0164] Memory 904 includes main memory 914, static memory 916, and storage cells 918, all of which are accessible by processor 902 via bus 932. Main memory 904, static memory 916, and storage cells 918 store instructions 910 embodying any one or more of the methods or generators described herein. Instructions 910 may also reside wholly or partially in main memory 914, in static memory 916, in a non-transitory machine-readable medium 920 within storage cells 918, in one or more of processors 902 (e.g., in the processor's cache memory), or in any suitable combination thereof during execution by machine 900.

[0165] I / O device interface 906 couples machine 900 to I / O devices 934. One or more of the I / O devices 934 may be components of machine 900 or may be separate devices. I / O device interface 906 may include various interfaces of the I / O devices 934 used by machine 900 for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O device interface 906 included in a particular machine will depend on the type of machine. It will be understood that the I / O device interface 906 of I / O device 934 may include Figure 9 Many other components are not shown. In various examples, I / O device interface 906 may include output component interface 924 and input component interface 928. Output component interface 924 may include interfaces for visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRTs), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input component interface 928 may include interfaces for alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing position and / or force of touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0166] Various technologies can be used to achieve communication. The I / O device interface 906 also includes a communication component interface 930, which is operable to couple the machine 900 to a network 922 or one or more devices 936 via couplings 926 and 938. For example, the communication component interface 930 may include a network interface component or an interface to another suitable device that interfaces with the network 922. In further examples, the communication component interface 930 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, or a Bluetooth component. ® Components (e.g., Bluetooth) ® Low power consumption, Wi-Fi ® The device 936 may be an interface to other communication components, including other communication components that provide communication via other modes. The device 936 may be another machine or any of various peripheral devices (e.g., a peripheral device coupled via USB).

[0167] Various memories (e.g., memory 904, main memory 914, static memory 916, and / or the memory of processor 902) and / or storage units 918 may store one or more instruction sets and data structures (e.g., software) embodied or used by any one or more of the methods or generators described herein. These instructions (e.g., instruction 910), when executed by processor 902, cause various operations to implement the disclosed examples.

[0168] Instructions 910 can be sent or received over network 922 via a transmission medium using a network interface device (e.g., a network interface component included in communication component interface 930) and using any of a plurality of known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 910 can be sent or received via a transmission medium through a coupling 938 to device 936 (e.g., a peer-to-peer coupling).

[0169] Other examples include:

[0170] Example 1 is a computer-implemented method comprising: receiving a kernel definition by one or more processors, the kernel definition including parameterization and code for a set of generators written in a general-purpose programming language; for each generator in the set of generators, performing operations including: transforming the code of each generator into a first intermediate representation of each generator by one or more processors; determining a configuration for each generator by one or more processors using the parameterization and the first intermediate representation; generating a second intermediate representation by one or more processors using the configuration; caching the intermediate representation by one or more processors; generating corresponding binary objects in a set of binary objects by one or more processors using the second intermediate representation; and combining a kernel corresponding to the kernel definition by one or more processors using the set of binary objects.

[0171] In Example 2, the subject of Example 1 is followed, wherein caching the intermediate representation includes: detecting an operation in the second intermediate representation; detecting a region in the body of the operation; generating a parameterized Content Addressable Storage Identifier (CASID) using the content of the region; copying the region to a container; and caching the container in a hash table using the parameterized CASID.

[0172] In Example 3, the subject of Example 2 is followed, where determining the configuration for each generator includes: determining a region in a first intermediate representation; generating a parameterized CAS ID using the intermediate representation; and searching a cached intermediate representation corresponding to the region in a hash table using the parameterized CAS ID.

[0173] In Example 4, the topics of Examples 2 and 3 are discussed, where the configuration for each generator is determined on multiple machines.

[0174] In Example 5, the topics of Examples 1 through 4 are discussed, wherein determining the configuration of each generator includes: generating a configuration set for each generator using parameterization and an intermediate representation of each generator; generating an executable set of test functions using the configuration set; executing the set of test functions to determine a corresponding performance score set; selecting the best configuration from the configuration set using the corresponding performance score set; and determining the configuration of each generator using the best configuration.

[0175] In Example 6, the topic of Example 5 is used, where the generated configuration set is also parameterized using the target machine.

[0176] In Example 7, the topics of Examples 5 and 6 are discussed, where a set of test functions is executed on multiple machines.

[0177] Example 8 is a machine comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations including: receiving a kernel definition including parameterization and code for a set of generators written in a general programming language; for each generator in the generator set, performing operations including: transforming the code of each generator into a first intermediate representation of each generator; determining a configuration for each generator using the parameterization and the first intermediate representation; generating a second intermediate representation using the configuration; caching the intermediate representation; generating a corresponding binary object in a set of binary objects using the second intermediate representation; and combining the binary object set with a kernel corresponding to the kernel definition.

[0178] In Example 9, the topic of Example 8 is followed, where caching the intermediate representation includes: detecting an operation in the second intermediate representation; detecting a region in the body of the operation; generating a parameterized Content Addressable Storage Identifier (CASID) using the content of the region; copying the region into a container; and caching the container in a hash table using the parameterized CAS ID.

[0179] In Example 10, the topic of Example 9 is followed, where determining the configuration for each generator includes: determining a region in a first intermediate representation; generating a parameterized CAS ID using the intermediate representation; and searching a cached intermediate representation corresponding to the region in a hash table using the parameterized CAS ID.

[0180] In Example 11, the topics of Examples 9 and 10 are discussed, where the configuration for each generator is determined on multiple machines.

[0181] In Example 12, the topics of Examples 8 through 11 are discussed, wherein determining the configuration for each generator includes: generating a configuration set for each generator using parameterization and an intermediate representation of each generator; generating an executable set of test functions using the configuration set; executing the set of test functions to determine a corresponding performance score set; selecting the best configuration from the configuration set using the corresponding performance score set; and determining the configuration for each generator using the best configuration.

[0182] In Example 13, the subject of Example 12 includes the generation of a configuration set that also uses target machine parameterization.

[0183] In Example 14, the topics of Examples 12 and 13 are discussed, where a set of test functions is executed on multiple machines.

[0184] Example 15 is a machine storage medium (machine-readable storage medium) including instructions that, when executed by one or more processors of the machine, cause the machine to perform operations including: receiving a kernel definition including parameterization and code for a set of generators written in a general programming language; for each generator in the generator set, performing operations including: transforming the code of each generator into a first intermediate representation of each generator; determining a configuration for each generator using the parameterization and the first intermediate representation; generating a second intermediate representation using the configuration; caching the intermediate representation; generating a corresponding binary object in a set of binary objects using the second intermediate representation; and combining the binary object set with a kernel corresponding to the kernel definition.

[0185] In Example 16, the subject of Example 15, wherein caching the intermediate representation includes: detecting an operation in the second intermediate representation; detecting a region in the body of the operation; generating a parameterized Content Addressable Storage Identifier (CASID) using the content of the region; copying the region into a container; and caching the container in a hash table using the parameterized CAS ID.

[0186] In Example 17, the subject of Example 16 is followed, where determining the configuration for each generator includes: determining a region in a first intermediate representation; generating a parameterized CAS ID using the intermediate representation; and searching a cached intermediate representation corresponding to the region in a hash table using the parameterized CAS ID.

[0187] In Example 18, the topics of Examples 16 and 17 are discussed, where the configuration for each generator is determined on multiple machines.

[0188] In Example 19, the topics of Examples 15 through 18 are discussed, wherein determining the configuration for each generator includes: generating a configuration set for each generator using parameterization and an intermediate representation of each generator; generating an executable set of test functions using the configuration set; executing the set of test functions to determine a corresponding performance score set; selecting the best configuration from the configuration set using the corresponding performance score set; and determining the configuration for each generator using the best configuration.

[0189] In Example 20, the topic of Example 19 is discussed, where the generated configuration set is also parameterized using the target machine.

[0190] In Example 21, the topics of Examples 19 and 20 are discussed, where a set of test functions is executed on multiple machines.

[0191] Example 22 is at least one machine-readable medium including instructions that, when executed by a processing circuitry system, cause the processing circuitry system to perform operations for implementing any one of Examples 1 to 21.

[0192] Example 23 is a device that includes means for implementing any one of Examples 1 through 21.

[0193] Example 24 is a system for implementing any one of Examples 1 through 21.

[0194] Example 25 is a method for implementing any of Examples 1 through 21.

[0195] Changes and modifications may be made to the disclosed examples without departing from the scope of this disclosure. Such and other changes or modifications are intended to be included within the scope of this disclosure as set forth in the appended claims.

[0196] Glossary

[0197] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0198] "Client device" means any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptops, portable digital assistants (PDAs), smartphones, tablets, ultrabooks, netbooks, multiple laptops, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0199] "Communications network" refers to one or more parts of a network, which may be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can enable any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0200] "Machine-readable medium" refers to both machine storage media and transmission media. Therefore, these terms encompass both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "machine-readable medium," and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0201] "Machine storage medium" refers to one or more storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to a processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are covered by the term "signal medium."

[0202] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates associated output signals that are applied to operate a machine. A processor can be, for example, a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Radio Frequency Integrated Circuit (RFIC), or any combination thereof. A processor can also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously.

[0203] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" can be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and are used interchangeably in this disclosure.

[0204] A "kernel" or "microkernel" is an implementation of an algorithm that performs computations on memory objects with a specific layout, such as memory buffers. The two terms are used interchangeably, but a "microkernel" often implies a small operation (e.g., memset, dot product, or reduction) within a larger generator kernel implementation. In the literature, algorithmically interchangeable / equivalent / replaceable kernels are sometimes referred to as "small code."

[0205] A generator is a parameterized metaprogram that is executed to generate a non-parameterized implementation of a kernel or microkernel. A fixed kernel implementation (e.g., panel dot product implemented in assembly) is a degenerate form of the generator without parameters.

[0206] A "kernel interface declaration" is a kernel or microkernel declaration applicable to multiple implementations of the kernel or microkernel. Kernels and microkernels can be implemented multiple times in many different ways. Interface declarations can be independent of implementation to allow type checking for both clients and implementations.

[0207] "Generator parameter arguments" refers to the values ​​that the kernel or microkernel is allowed to apply to it. A generator is a metaprogram that generates the kernel, and "parameters" are the values ​​that the metaprogram is allowed to apply to it.

[0208] "Kernel generator parameter results" are values ​​returned by the generator to its caller as arguments, allowing them to adapt to the behavior within the generated subkernel. For example, a panel dot product generator could return "I processed a 3×5 memory panel," which allows calling a for loop to step on each dimension with a size of 3 and 5.

[0209] "Generator constraints" are constraints that indicate limitations on parameters of the kernel or microkernel, such as "This implementation only works for data type = float32", or "This only works for machines with x86 VNNI extensions", or "This works for size modulo 136", etc. Generators are allowed as partial generators from interface declarations to concrete implementations. Constraints propagate upwards from the kernel implementation to the generator graph.

[0210] "Internal kernel arguments" are statically single-assigned (SSA) argument values ​​used for: buffers and other user-defined types for structured abstractions within memory such as linear memory, N-dimensional tensors with layouts, and other higher-level data types such as trees and tables; values ​​corresponding to operational attributes at the tensor graph level, although these values ​​can be modeled as constants there, they are dynamic values ​​for how the kernel is implemented at runtime; and very small microkernels at the bottom of the stack (e.g., adding two integers) that use arguments as their inputs.

[0211] "Kernel result" is an SSA result value used for: dynamically allocated result buffers, such as those with data-dependent shapes; and very small microkernels at the bottom of the stack (e.g., adding two integers) to use the result as their output.

Claims

1. A computer-implemented method, comprising: A kernel definition is received by one or more processors, the kernel definition including parameterization and generator set code written in a general programming language; For each generator in the generator set, perform the following operations: The code of each generator is transformed into a first intermediate representation of each generator by one or more processors; The configuration of each generator is determined by one or more processors using the parameterization and the first intermediate representation; The second intermediate representation is generated by one or more processors using the configuration; The intermediate representation is cached by one or more processors; and The corresponding binary objects in the binary object set are generated by the one or more processors using the second intermediate representation; and The one or more processors use the set of binary objects to combine the kernels corresponding to the kernel definition.

2. The computer-implemented method according to claim 1, wherein, The intermediate representation cached includes: Detect the operation in the second intermediate representation; Detecting a region within the main body of the operation; Use the content of the region to generate a parameterized Content Addressable Storage Identifier (CAS ID); Copy the region to the container; and The container is cached in a hash table using the parameterized CAS ID.

3. The computer-implemented method according to claim 2, wherein, Determining the configuration for each generator includes: Determine the region in the first intermediate representation; Use the intermediate representation to generate a parameterized CAS ID; and The parameterized CAS ID is used to search the hash table for the cached intermediate representation corresponding to the region.

4. The computer-implemented method according to claim 2, wherein, The configuration for each generator is determined on multiple machines.

5. The computer-implemented method according to claim 1, wherein, Determining the configuration for each generator includes: The configuration set for each generator is generated using the parameterization and the intermediate representation of each generator; Use the configuration set to generate an executable set of test functions; Execute the set of test functions to determine the corresponding performance score set; Select the optimal configuration from the configuration set using the corresponding performance score set; and The optimal configuration is used to determine the configuration of each generator.

6. The computer-implemented method according to claim 5, wherein, The configuration set is also generated using target machine parameterization.

7. The computer-implemented method according to claim 5, wherein, The test function set is executed on multiple machines.

8. A machine comprising: One or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, cause the machine to perform operations, including: Receive kernel definition, which includes parameterization and generator set code written in a general programming language; For each generator in the generator set, perform the following operations: Transform the code of each generator into a first intermediate representation of each generator; The configuration of each generator is determined using the parameterization and the first intermediate representation; Use the configuration to generate a second intermediate representation; Cache the intermediate representation; and The corresponding binary object in the binary object set is generated using the second intermediate representation; and The binary object set is used to assemble the kernel corresponding to the kernel definition.

9. The machine according to claim 8, wherein, The intermediate representation cached includes: Detect the operation in the second intermediate representation; Detecting a region within the main body of the operation; Use the content of the region to generate a parameterized Content Addressable Storage Identifier (CAS ID); Copy the region to the container; and The container is cached in a hash table using the parameterized CAS ID.

10. The machine according to claim 9, wherein, Determining the configuration for each generator includes: Determine the region in the first intermediate representation; Use the intermediate representation to generate a parameterized CAS ID; and The parameterized CAS ID is used to search the hash table for the cached intermediate representation corresponding to the region.

11. The machine according to claim 9, wherein, The configuration for each generator is determined on multiple machines.

12. The machine according to claim 8, wherein, Determining the configuration for each generator includes: The configuration set for each generator is generated using the parameterization and the intermediate representation of each generator; Use the configuration set to generate an executable set of test functions; Execute the set of test functions to determine the corresponding performance score set; Select the optimal configuration from the configuration set using the corresponding performance score set; and The optimal configuration is used to determine the configuration of each generator.

13. The machine according to claim 12, wherein, The configuration set is also generated using target machine parameterization.

14. The machine according to claim 12, wherein, The test function set is executed on multiple machines.

15. A machine storage medium including instructions that, when executed by one or more processors of the machine, cause the machine to perform operations, the operations including: Receive kernel definition, which includes parameterization and generator set code written in a general programming language; For each generator in the generator set, perform the following operations: Transform the code of each generator into a first intermediate representation of each generator; The configuration of each generator is determined using the parameterization and the first intermediate representation; Use the configuration to generate a second intermediate representation; Cache the intermediate representation; as well as The corresponding binary object in the binary object set is generated using the second intermediate representation; and The binary object set is used to assemble the kernel corresponding to the kernel definition.

16. The machine storage medium according to claim 15, wherein, The intermediate representation cached includes: Detect the operation in the second intermediate representation; Detecting a region within the main body of the operation; Use the content of the region to generate a parameterized Content Addressable Storage Identifier (CAS ID); Copy the region to the container; and The container is cached in a hash table using the parameterized CAS ID.

17. The machine storage medium according to claim 16, wherein, Determining the configuration for each generator includes: Determine the region in the first intermediate representation; Use the intermediate representation to generate a parameterized CAS ID; and The parameterized CAS ID is used to search the hash table for the cached intermediate representation corresponding to the region.

18. The machine storage medium according to claim 16, wherein, The configuration for each generator is determined on multiple machines.

19. The machine storage medium according to claim 15, wherein, Determining the configuration for each generator includes: The configuration set for each generator is generated using the parameterization and the intermediate representation of each generator; Use the configuration set to generate an executable set of test functions; Execute the set of test functions to determine the corresponding performance score set; Select the optimal configuration from the configuration set using the corresponding performance score set; and The optimal configuration is used to determine the configuration of each generator.

20. The machine storage medium according to claim 19, wherein, The configuration set is also generated using target machine parameterization.

Images