Automatic tuning of heterogeneous computing systems

Abstract

The invention provides a method of configuring program parameters for computation during runtime of a computing program in a heterogeneous computing system, processing a compiled program in an auto-tuning system to optimize parameters of an application for processing in a heterogeneous system including, for example, CPU and GPU cores.

Description

Technical Field

[0001] This invention relates to computing systems with different types of processors, and a technique for arranging different tasks across processors to perform parallel computing in an efficient manner. Background Technology

[0002] This heterogeneous computing system includes supercomputers (also known as high-performance computers), which consist of both CPU (Central Processing Unit) and GPU (Graphics Processing Unit) cores. Additional dedicated computing units, such as FPGAs (Field-Programmable Gate Arrays), DSPs (Digital Signal Processors), and quantum computing units, can also be provided.

[0003] A single computational task may be better suited to be executed on one type of processor than another, and arranging different tasks for parallel processing can be a fairly large undertaking. Arranging tasks to be processed by different units to optimize the degree of parallelization is called tuning. Tuning can be done "offline," performed before runtime, or "online," performed during runtime. While program developers are able to implement tuning using their skills and knowledge, this manual tuning is time-consuming. Automatic tuning, or automated tuning, is the desired development.

[0004] For example, automatic tuning is described in “Automatic Application Tuning for HPC Architectures” by Siegfried Benkner et al., from Dagstuhl Workshop 13401, September 29–October 4, 2014, pp. 214–243, https: / / www.dagstuhl.de / 13401, which is incorporated by reference for all purposes.

[0005] The problem with online autotuning is that system users may experience increased runtime as the system implements different configurations while searching for the optimal configuration. Summary of the Invention

[0006] This invention provides a method for performing online automatic tuning, which is comparable to known online tuning techniques while reducing the required runtime.

[0007] This invention provides a method for configuring program parameters for computation during runtime in a heterogeneous computing system. The method includes: receiving a transformation of a computing program, the transformation including one or more computing applications; generating one or more tuning parameters for each computing application, the one or more tuning parameters being categorized; and for a computing application to be optimized, recursively performing the following operations during runtime: adjusting one or more tuning parameters of the computing application; executing the computing application using the adjusted one or more tuning parameters; obtaining performance metrics for executing the computing application using the adjusted one or more tuning parameters; and determining whether the adjusted one or more tuning parameters provide an improvement in the performance metrics and whether a termination criterion has been met to end the adjustment, wherein the metric characteristics of the dynamic state of the computing application being optimized are stored, the metric being used in optimization to determine whether a known optimization can be used for the dynamic state indicated by the stored metric.

[0008] In another aspect, the present invention provides a computing entity programmed to optimize a computing application by recursively performing the following operations during runtime: adjusting one or more tuning parameters of the computing application; executing the computing application using the adjusted one or more tuning parameters; obtaining performance metrics for executing the computing application using the adjusted one or more tuning parameters; and determining whether the adjusted one or more tuning parameters provide an improvement in the performance metrics and whether a termination criterion has been met to end the adjustment, wherein metric characteristics of the dynamic state of the computing application being optimized are stored, which are used in the optimization to determine whether a known optimization can be used for the dynamic state indicated by the stored metrics. Attached Figure Description

[0009] Preferred embodiments of the invention will now be described by way of example only with reference to the accompanying drawings, wherein:

[0010] Figure 1 A schematic diagram of the compiler and automatic tuning system according to the present invention is shown;

[0011] Figure 2a yes Figure 1 A schematic diagram of an automatic tuning system;

[0012] Figure 2b This is an example of computer code used to optimize the number of threads during autotuning;

[0013] Figure 2c This is a schematic diagram of the interaction between the automatic tuning system of the present invention and the application, wherein feedback is passed to the memory to update the prediction and search states;

[0014] Figure 3 This is a schematic diagram of the operation of the auto-tuning search strategy, where the integer value of the configuration C changes over time using a mirror axis;

[0015] Figure 4 This is an example of computer code that provides a search implementation interface;

[0016] Figure 5 The available state transitions in the Nelder-Mead search algorithm are shown;

[0017] Figure 6 A graphical representation of the decomposition of the global search space for simplified heterogeneous mappings according to parameter categories is shown;

[0018] Figure 7 An example of decomposing the global search space using constraints provided by the application developer is shown;

[0019] Figure 8 This is an example of a search space graph for the parameter set T = {A; B; C; D; X; Y; Z; W}, where {A; B; C; D} are categorical and {W; X; Y; Z} are non-categorical.

[0020] Figure 9 This is an example of a hierarchical search for parameters {A; B; C; D; X; Y; Z; W};

[0021] Figure 10 yes Figure 11 The example shown is a partitioning of the scheduling tree and a platform mapping.

[0022] Figure 11 This is an example of a scheduling tree for matrix multiplication;

[0023] Figure 12 This is an example of the loop control flow structure supported by the system of the present invention;

[0024] Figure 13 The code snippet generated by the target plugin is shown, where solid arrows represent synchronous operations and dashed arrows represent asynchronous operations; and

[0025] Figure 14 An example of the assignment mapping for three arrays is shown. Detailed Implementation

[0026] Figure 1 A schematic diagram illustrating the implementation of automatic tuning according to the present invention is shown.

[0027] The implementation includes a compiler 10 and a runtime autotuning system 20. The compiler is adapted to parallelize sequential applications on heterogeneous platforms, such as systems composed of CPUs and GPUs, and autotuning is performed during execution. The compiler 10 interacts with the runtime autotuning system 20 to provide online tuning, optimizing task allocation between CPU cores and GPU cores.

[0028] The compiler 10 and the automatic tuning system 20 will now be described separately.

[0029] Figure 2a The automatic tuning system 20 is shown in more detail in the figure. Figure 2a The logic units constituting the automatic tuning system 20 are shown. The main component of system 20 is the hybrid tuning unit 30. The hybrid tuning unit 30 implements the hybrid tuning method of the present invention and interoperates with the application 32 through units called parameter space 34 and index space 36. The hybrid tuning unit 30 can be extended by search and prediction model selections 38 and 40 provided by the application developer. The search extension provides hierarchical search functionality, which depends on the corresponding basic search algorithm used in the deployment. The model extension provides a prediction model to implement the prediction component of the hybrid tuning unit.

[0030] The application iteratively interacts with the automatic tuning system (also known as the "automatic tuner" or simply the "tuner"). For a previously registered set of application parameters, the application requests new configurations for those parameters and feeds back performance metrics for that configuration to the tuner.

[0031] Therefore, integrating an application with an autotune requires two steps: exposing and registering tunable parameters, and introducing update and feedback calls. Tunable application parameters can be of any type and have arbitrary semantics, as long as they are freely configurable by the autotune. The application parameters are updated with the next configuration, and performance feedback regarding the configuration is reported to the tuner, including the application's tuning kernel.

[0032] Therefore, the update-sample-feedback sequence forms the core of the tuning loop. Most of the time, feedback is a runtime measurement, although this is not mandatory. The autotuner attempts to find the parameter configuration that minimizes the metric, regardless of the metric's meaning.

[0033] Figure 2b This describes an OpenMP example that uses autotuning to optimize the number of threads. The `NumThreads` parameter is registered to the tuner by reference, using a valid value range from 1 to 16. `NumThreads` is automatically updated with new sample values ​​from this range each time `add(...)` is called, specifically on line nine when `OMPTuner.start()` is invoked. Therefore, the new configuration is read from this variable. A subsequent call to `OMPTuner.stop()` then reports runtime feedback to the tuner, using the clock time elapsed since the last call to `start()`. For convenience, the `start()` / `stop()` APIs are provided.

[0034] Alternative APIs exist to use arbitrary feedback values. Note that this example is not simplified and shows the actual call sequence required to set up and perform autotuning, but does not include the tuning loop. The autotuner is generic and extensible. The behavior of the tuning process is fully customizable: the search algorithm and online learning mechanism can be changed or replaced via plugins provided by the application. For complete genericity, the tuner has no restrictions on the type, range, or semantics of application parameters. At its core, the autotuner provides an abstraction between the application and the search engine.

[0035] The search engine drives any search algorithm, such as the Nelder-Mead algorithm, ε-Greedy search, or genetic algorithms. Application developers choose an algorithm from a small set of built-in search algorithms or implement their own. The search is isolated from the application by an adaptation layer that maps between an unconstrained real-valued search space and a typically constrained, arbitrarily valued application parameter space. The library provides reasonable default values ​​for this mapping for the most common parameter types: bounded or unbounded integer or floating-point values, and non-numeric values ​​describing the selection set. The mapping can be modified by the application developer.

[0036] So far, the features outlined make the autotune suitable for most tuning scenarios. Besides its usability, the main feature of the autotune is its improved amortization mechanism. The autotune reduces both the total number of configurations tried and the number of particularly bad configurations tried. It achieves this through three techniques.

[0037] The first technique is called hybrid tuning. The performance of a tuning kernel typically depends not only on the tuning parameter configuration but also on the system and application state. To leverage this fact, the autotuner implements a hybrid autotuning approach that combines search, memory, and model-based prediction. Applications can register arbitrary “metrics” with the tuner, which characterize the dynamic state of the application and system. An example is the abstract “input size.” Using online learning, the autotuner automatically detects significant changes in the dynamic state and reacts to them: based on a predictor built through online learning, hybrid tuning can immediately switch to the learned configuration. If the exact same dynamic state has been observed in the past, the search results for that state are used. Otherwise, the trained model is queried for a predictable, well-defined configuration. The application does not need to provide training samples. The predictor is automatically trained on all observed metric states from the training samples generated during the search.

[0038] The second technique involves introducing parameter constraints to limit the scope of autotuning. Specific application parameters often impose constraints on much of the parameter space. One example is the parameter controlling algorithm selection. Any tuning parameters included in one of these algorithms should only be updated when that particular algorithm is selected. Therefore, the algorithm selection parameter controls the dependency of the algorithm parameters. On the other hand, parameters controlled by selection parameters are generally not independent of parameters not controlled by selection parameters. They cannot be tuned individually. If application developers flag such dependency constraints, the autotuner constructs a hierarchical search space to reduce dimensionality as much as possible while preserving dependencies. Furthermore, this allows different search algorithms to be used for different parts of the search space. For example, introducing the "Parallel" parameter also allows computations to be performed sequentially, in which case the NumThreads parameter becomes irrelevant. Registering this dependency with the tuner allows it to leverage this fact, tuning the number of threads only when Parallel is set to "true".

[0039] The third technique is providing analytical feedback. In many practical applications, application developers possess domain knowledge of parameter configurations and can predict whether a configuration is appropriate. Several such predictions are possible in compiler examples. For instance, allocating two iterations of a parallel loop to the GPU is essentially guaranteed to result in suboptimal performance. With this in mind, applications can reject configurations without measuring them. In this way, the autotuner can reduce the number of sampled misconfigurations, thus improving amortization time.

[0040] Figure 2c An iterative tuning routine executed by the hybrid tuning unit 30 is shown.

[0041] Suitable search extensions include, for example, those based on the Nelder-Mead search algorithm and a search algorithm called "ε-Greedy". Predictive model extensions can be based on nearest neighbor predictions.

[0042] Each application 32 is characterized by one or more parameters. These parameters are further classified into different categories, such as "categorical", "ordinal", "interval", and "ratio", as shown in Table 1.

[0043] Table 1:

[0044]

[0045] For simplification, parameters can be further divided into "classified" and "non-classified" categories.

[0046] Autotuning optimizes parameters according to certain defined criteria. The properties of these parameters are entirely application-specific. So-called black-box autotuners ignore the semantics of the parameters. More importantly, parameters in practice have different domains. For example, some parameters can be integers, others can be floating-point numbers, or even not numbers at all. Some parameters might assume valid values ​​from the range [0:10], while others come from the range [0:0; 1:0]. Dealing with these different domains is particularly cumbersome for search and model implementation.

[0047] Therefore, in this invention, parameter attributes are isolated from those of these specific applications. This isolation is provided by parameter space unit 34, which acts as a configurator between the application and its parameters on one side and the search and prediction algorithms on the other. The transformation allows the search and prediction algorithms to operate in an n-dimensional real-valued unbounded space called the search domain, where n is the number of parameters. When the application reports feedback on parameter configuration to the adaptation layer, the state of the hybrid tuning is updated, and the next configuration sample is queried.

[0048] The sample is a real-valued vector in the search domain. Configurations are cached within the parameter space unit. Once the application requests the next parameter update from the hybrid tuner component, the cached configuration sample is transferred to the application domain and then passed to the application. This mechanism has two positive effects. First, it allows the development of search algorithms to be independent of the applications that use them. Second, it prevents several numerical problems in the search algorithm. When attempting optimization in integer space, the search algorithm may get stuck in suboptimal configurations. Due to rounding, the search will never converge but will instead flip between a small set of configurations. Through parameter space unit 34, responsibilities such as rounding and boundary handling are moved from the search implementation to the application domain.

[0049] When registering parameters with this invention, applications can define separate domain and boundary mapping behaviors as needed. Furthermore, this invention provides several reasonable default adapters for the most frequently used parameter types. These are integer and floating-point parameters, unconstrained or derived from ranges, and bag-of-label parameters. The latter are parameters that can derive any value from a set of unordered labels. Through the default adapters, these are embedded as range-constrained integer parameters, effectively numbering the unordered labels. Integer parameters are converted from the real-value search domain by default via rounding. To restrict valid configurations to ranges, the adapter virtually mirrors the measurement function at the range boundaries. The configuration values ​​returned by the search are then mapped to the range associated with the mirror. Figure 3 An example of a process that accepts parameters from the range [1, 6] is shown. The figure illustrates the search space when a single parameter appears in the search algorithm.

[0050] Although this parameter can only accept values ​​between 1 and 6, the search can freely sample any value. For example, sampling a configuration value of C = 1.3 will apply the parameter as a value of 1 by rounding. Sampling a value of C = 10.7 will first apply the mirroring, then round it to apply the parameter as a value of 3. Negative values ​​are also handled accordingly, but are not shown in the examples for simplicity.

[0051] This mirroring strategy has many interesting properties. Most importantly, it preserves all local and global optima: any newly introduced point cannot exceed the global optimum, and all previous local optima remain local. Any local extrema on interval boundaries remain local, but their neighborhoods are now symmetric. We cannot expect the mirroring strategy to be effective for every possible search algorithm. For example, it might mislead searches based on assumptions about distances between points in the search space. Points might appear far apart for a search, even if their measurements are obtained from points mapped to very close points in the application domain. In such cases, application developers are expected to provide appropriate adaptation mechanisms and their parameter definitions.

[0052] Indicator space unit 36 ​​is a register used to store system indicators.

[0053] Managing application and system metrics is less complex than tuning parameters. Metric space unit 36 ​​is a registry containing sets of metrics. Besides registration, this component also implements change detection. The content constituting a change can be overridden by applications that define their own metrics. By default, a change is detected whenever the metric's value changes. Because applications are forced to represent their metrics as numerical values, this is sufficient for hybrid tuning. Domain transformations are not necessary as in the parameter space unit.

[0054] In this invention, application and system metrics are provided by application developers. When used within this framework, the framework's compiler assumes the role of the application developer and automatically defines the metrics.

[0055] Metrics provided by developers can be as simple as any application-side variable. Therefore, using program or function inputs as metrics is not critical, but arbitrarily complex metrics are also possible. For example, a metric for monitoring system load could be defined, which would require querying operating system statistics each time its metric status is queried. Furthermore, depending on the nature of the metric, observing its values ​​may require downsampling.

[0056] Therefore, application developers can add target resolutions. For example, this can be useful for monitoring metrics of system load. Whether the load is 99% or 100% has no significant difference in impact on tuned kernel performance, but the difference between 10% and 100% can be significant. It is up to the application developers to define meaningful sampling resolutions, as this invention does not intend to assume any semantics for specific metrics.

[0057] Two search algorithms for use with the hybrid tuning unit 30 will now be described.

[0058] Due to the isolation and abstraction provided by parameter space unit 34, the implementation of the search algorithm only requires a minimal interface. Figure 4 The base class is shown. Specifically, the search algorithm doesn't care about the actual sampling. Instead, it returns a series of configurations to the caller's parameter space unit. The caller's responsibility is to obtain the measurement and pass it back to the search. In the following text, we use the phrase "sampling" to refer to the single round trip of returning a configuration to the caller and receiving the measurement in the next iteration. The mandatory operations that the search algorithm implementation must provide are `start` and `getNext`, which respectively restart the search and obtain the next configuration. When obtaining the next configuration, the caller also provides the measurement of the previous configuration as a parameter. The search implementation expects to update its internal state upon calling `getNext` and return the next configuration it wants to sample. Furthermore, the implementation can support rejecting individual configurations without measuring them. The caller uses this mechanism to filter out illegal configurations or ignore configurations they don't want to sample. This is likely if they are using a performance model and predict the configuration will be suboptimal.

[0059] When tuning an application, developers can provide their own implementation of the search algorithm or choose the default implementation from a small catalog of implementations. The library differentiates between algorithms based on categorical and non-categorical parameters. Unless otherwise requested by the application developers, a new hierarchical search algorithm is used for searching in the mixed space. In this section, we describe the two default basic search techniques used in hierarchical search. The hierarchical search techniques themselves will be discussed below.

[0060] The tuning library includes implementations of the Nelder-Mead downhill simplification algorithm and the ε-Greedy algorithm as basic search algorithms, as well as random and full-scale exploration. While the latter two are self-evident, variants of Nelder-Mead and ε-Greedy will be discussed below.

[0061] In this embodiment of the invention, the basic search strategy for nonclassifiable parameters is an implementation of the Nelder-Mead algorithm described by J.A.N. Elder and R. Mead in "A Simplex Method for Function Minimization" in The Computer Journal 7.4 (January 1965) (cit. on pp. 15, 91). This algorithm is a heuristic derivativeless numerical optimization technique. The algorithm is implemented as a small Mealy automaton, such as... Figure 5 The finite state machine is shown. This implementation maintains the simplex as an array χ of points in the search space and their measurements mx, x∈χ. The points in χ are ordered in ascending order relative to mx. The state machine changes its state in each tuning iteration.

[0062] State transitions are defined as triples M / C;U, where C is the next configuration, M is the measurement feedback mC' for the previous configuration C', and optionally includes constraints on the measurements. Finally, U defines an optional update to the simplex χ, which is applied before the next configuration is returned. The state machine is defined by five states starting from S. Transitions implement the Nelder-Mead update rule.

[0063] Starting from the initial state, the automaton transitions to the initialization state I by initializing a J+1-dimensional simplex from samples of the Latin hypercube. The first simplex point is returned to the application as the next configuration. The Latin hypercube is a technique for obtaining configurations spanning space and is frequently used in seed search algorithms. To obtain a simplex based on this technique, each dimension of the bounded J-dimensional space is divided into J+1 equal-sized intervals. Then, samples are randomly drawn such that no two samples in any given dimension fall into the same interval. Once the application returns measurement feedback for the last simplex point, the automaton performs a reflection operation by transitioning to the reflection state R and returning the reflection point r. The successor to the reflection state depends on the measurement feedback m' for the last reflection point r'. r If m′ rIf the value is less than mχ[0], i.e., the measurement of the best simplex point, the automaton performs an expansion operation, returns to the expansion point e, and transitions to the expansion state E. If m'r is between the best and worst simplex points, the worst simplex point χ[J] is replaced with the previous reflection point r', and another reflection is performed to produce the next reflection point r. Otherwise, if m'r is greater than or equal to the measurement of the worst point, a contraction operation is performed and the contraction point c is returned. After receiving feedback me for the expansion point e, the expansion state E transitions back to the reflection state R. This transition replaces the worst simplex point with the better one of e and r', and another reflection operation is performed to produce the next r. Similarly, if the feedback mc for the contraction point c is better than the value of χ[J], the automaton performs a reflection operation in the contraction state C. In this case, χ[J] is replaced with c. Otherwise, if mc is greater than or equal to mχ[J], a reduction operation is performed. This operation forms a new simplex by moving each point toward χ[0]. Subsequently, every simplex point except χ[0] needs to be sampled, so the automaton enters initialization state I starting from χ[1]. The reflection point r, contraction point c, and expansion point e are calculated using the default parameter values ​​suggested in the Nelder-Mead paper: α = 1; β = 0; γ = 2. Our implementation of the Nelder-Mead algorithm does not support rejection of individual configurations.

[0064] The second search algorithm is the so-called ε-Greedy algorithm.

[0065] For the categorical parameters, in this embodiment of the invention, the ε-Greedy algorithm is implemented, which achieves optimal performance in the exploratory case study. This algorithm keeps track of the best known configuration. At each step, it samples a random configuration with probability ε, or uses the best known configuration with probability 1-ε. Another hyperparameter is added to control the decay of ε over time. This parameter is called γ. ε ∈(0,1). Then, the probability of choosing a random configuration at step i becomes ε. i =ε.(1-γ) ε ) i Therefore, γ ε It provides direct control over the convergence rate. Setting this hyperparameter to zero reverts to the original ε-Greedy behavior. When the application rejects the configuration, the algorithm returns new random samples.

[0066] In real-world programs, parameters are not only frequently integers, but also often categorical. Applying search algorithms to optimize mixtures of these two categories is highly problematic because it limits the choice of suitable search algorithms. While this invention does not actively discourage application developers from choosing inappropriate search algorithms for parameter mixtures, it strongly discourages such choices. Search algorithms that draw conclusions based on, for example, configured neighborhoods will lead to purely coincidental information. Consider the algorithm selection for the tuned matrix multiplication algorithm as an example.

[0067] When the categorical tuning parameters A = {ijk; ikj; strassen} are embedded into the integer space using a simple enumeration algorithm, a hill-climbing algorithm can be applied.

[0068] If the climber now observes m(ijk) = 2 and m(ikj) = 1, it will conclude that the space looks promising in the direction of ikj. Whether Strassen is actually better than ikj is already a coincidence. The order of the algorithms is irrelevant. As we found in our exploratory case study, the vast majority of empirical search algorithms used today face this problem. They operate on the concept of direction or distance. A notable exception is the genetic algorithm, which solves combinatorial optimization problems well when the appropriate crossover and mutation operators are used.

[0069] In this invention, we handle searches in this hybrid space in a structured manner that is automatic and transparent to application developers. As an example, consider a simplified version of the heterogeneous parallelization approach. We generate parameterized parallel code for both CPU and GPU, and add tuning parameters to select the platform. In this example, we tune a total of five application parameters: P, binary platform selection; T... CPU The number of CPU threads; T GPU The number of GPU threads; SM, the flag that enables shared memory; and U SM This breaks the memory copy loop and moves the data into shared memory. Table 2 summarizes these parameters and their ranges. Traditional, state-of-the-art tuning schemes embed these parameters into a single five-dimensional space, containing 2.48.32.2.2 = 12,228 valid points. Our first improvement is to divide this single space into two smaller spaces by distinguishing between categorical and non-categorical parameters. Figure 6 The effect is illustrated graphically: the space on the left now contains only categorical parameters, and the space on the right contains only non-categorical parameters. Note that this separation does not actually reduce dimensionality. There are 2.2.2 = 8 points in the categorical space and 32.48 = 1536 points in the non-categorical space. However, it is illegal to independently search for configurations in separate spaces because the parameters are not independent. For example, suppose on a GPU, the kernel runtime is measured in milliseconds (m). GPU=0:5 definition. If shared memory is disabled (C SM If == false), then it is T. GPU ,otherwise T GPU Which value is optimal clearly depends on the selected value of SM. Therefore, when choosing the non-categorical parameter T... GPU Ignoring the selected value of the classifier parameter SM is not a feasible approach.

[0070] Conversely, the configuration must be sampled from the cross product of the two spaces, meaning the cardinality of the separating spaces remains unchanged. Nevertheless, through this initial decomposition, we are now able to explore the space using different search algorithms suited to their respective parameter classes (e.g., the ε-Greedy algorithm on one hand and the Nelder-Mead algorithm on the other). To preserve the dependencies between parameters, we hierarchically configure the parameters:

[0071] We first use the ε-Greedy algorithm to determine the configuration of the categorical parameters, and then use the Nelder-Mead algorithm to determine the configuration of the non-categorical parameters. Because we cannot choose the configuration independently, we maintain a Nelder-Mead search instance for each categorical configuration. In this example, the memory consumption of our Nelder-Mead implementation increases by 8 times. A Nelder-Mead instance consumes O(n^2) time. 2 ) space, where n is the dimension of the search space.

[0072] However, the actual increase in memory usage was only about 30%, or more accurately, an increase of... While Nelder-Mead increases space consumption by 8x, the number of parameters is reduced from 5 to 2. Although a 30% increase in space may seem small, the penalty can actually have a significant impact because it is exponential in the dimension of the categorical search space. On the other hand, at this cost, we implement a simpler search space for each search algorithm and are able to locate the parameter classes with appropriate algorithms. Furthermore, the instantiation of search instances is lazy. Therefore, we only incur the worst-case memory overhead when we sample each categorical configuration, which actually only occurs for a small categorical space.

[0073] Decomposing the search space into categorical and non-categorical subspaces opens the door to another type of optimization. In practice, the categorical parameter often refers to algorithm selection, thus leading to different control flow paths in the application. In this example, parameter P controls whether the parallel code implementation is executed by the CPU or GPU. Therefore, transitively, this parameter also controls parameter T. CPU and T GPUThe relevance: When running CPU-based code, for example, configuring the number of GPU threads has no impact. We can leverage this constraint by utilizing the application developer's contextual knowledge of the semantics of application parameters. In the example above, we can identify two additional constraints: the number of GPU threads is related to the choice of GPU, and the loop for moving data into shared memory is only unrolled when shared memory is enabled. Following these constraints produces... Figure 7 The search space decomposition is shown. Unlike the first-level decomposition, the second step does indeed reduce the cardinality of the space. We see that, for this example, we have achieved the highest degree of dimensionality reduction, with only a one-dimensional search space. This also reduces the size of the sample space because the constraints exclude irrelevant configurations. The total number of sampleable points is now 48 + 32 + 2.32 = 148: there exists C P There are 48 configurations for CPU, 32 configurations for CP==GPU and CSM==false, and 2.32 configurations for CP==GPU and CSM==true.

[0074] The considerations for this second-level decomposition are the same as above: we need to preserve parameter dependencies. This means that for both categorized and non-categorized spaces, we need to maintain a separate search state for each space-related configuration.

[0075] In summary, even for this small example, we have managed to reduce the size of the search space by two orders of magnitude, while allowing the tuner to explore portions of the search space using different, appropriate search mechanisms. This size reduction is achieved by characterizing the parameters as their corresponding categories and decomposing the search space based on relevance constraints. Below, we formalize the algorithm for decomposition and searching within the decomposed space. The overall idea is to compute the decomposition into a minimum number of subspaces, thus preserving both relevance and category constraints. The first level of decomposition is straightforward: we define the search space T = T N ∪T R T is classified as a categorical parameter N T and non-categorical parameters R Naturally disjoint subspaces.

[0076] Relevance constraints. The second level of decomposition is calculated based on the relevance constraints defined by the application developers. We write... Where, τ i ∈T N , τ j ∈T and v∈τ i Represents the tuning parameter τ j The correlation depends on the class-specific tuning parameter τ i Configuration value When a parameter is relevant, the search must provide a configuration value for it. Otherwise, the parameter should not be configured. Note that only categorical parameters can appear on the left-hand side of a constraint. Naturally, neither cyclic constraints nor contradictory constraints are allowed.

[0077] Local search space. When all elements in a subspace are subject to the same correlation constraint, we define T as... N and T R Both of their subspaces are called local search spaces. More precisely, subspaces... It is a local search space if and only if or and

[0078] Search space graph. The local search space and relevance constraints together form a directed acyclic graph, which we call the search space graph. Nodes in the diagram It is a local search space caused by relevance constraints, which forms the edge E in the graph. S :

[0079]

[0080] The root of the graph is formed by the local search space containing all unconstrained parameters.

[0081] Before we introduce the hierarchical search process, let's first discuss another, more complete example of graph decomposition in the search space. Consider the parameter set T = {A, B, C, D, X, Y, Z, W}, where A to D are class-specific and X to Z are non-class-specific. Then, the first-level decomposition is T N ={A,B,C,D},T R ={W,X,Y,Z}. Furthermore, let the dependency constraints be... as well as Effective search space graphs, such as Figure 8 As shown, there exist two subspaces without any input constraint edges. One contains categorical parameters A and D, and the other contains non-categorical parameters W. The two spaces are disjoint because they contain parameters of different classes. Although X and Y both depend on the same configuration value of C, they reside in different local spaces because X has another dependency on B, which Y does not share. To find the configuration in the graph decomposition of the search space, a hierarchical search traverses the graph along the edges that satisfy the constraints. This process is described below.

[0082] Local, partial, and maximum configurations. Configuring tuning parameters based on the graph means determining the local configuration for all nodes that satisfy the constraints. The local configuration of the local search space l is the configuration C of the parameters of that local search space. lWe call the set of local configurations in different local spaces a partial configuration C of the search space graph S. S When it does not contain contradictory local configurations. If there are two paths including l and to l' containing contradictory constraints, then the local configuration C is... l C l 'Contradictory.' If there is no possible local configuration that can be added to a partial configuration without contradicting the local configurations in that set, then that partial configuration is called the largest partial configuration.

[0083] As an illustration of contradictory configurations, consider the local spaces containing B and C in the example above. These two spaces cannot be part of a partial configuration because this would require A to be configured with both b and c values ​​simultaneously.

[0084] Active node. Given partial configuration C S If from l R All edges on all paths p to l Let's talk about nodes. For C S It is active. Specifically, any node without any incoming edges is active. There can be two such nodes in the graph, one containing all unconstrained classifiers and the other containing all unconstrained nonclassifiers. Generating the largest portion of the configuration means recursively generating local configurations for all active nodes. Hierarchical search generates the largest portion of the configuration by continuously searching active nodes until no node becomes active. After obtaining the local configuration of the current node, the search recursively goes to the active neighbors of the current node in a defined order. Local results are accumulated into the partial configurations until no active neighbor generates the largest portion of the configuration. For the classifier space, hierarchical search leverages the fact that finding classifier configurations is a combinatorial optimization problem. We use this fact to implement a separate search instance on isolated single active classifier nodes. Since the parameters are interdependent, the search instance must consider the current partial configuration. For this reason, hierarchical search maintains concurrent search state for all the partial configurations it generates. On the other hand, for the nonclassifier space, such isolation of the active space is impractical. Instead, all active nonclassifier spaces are connected into a single space, which is then explored using Nelder-Mead instances. To account for the dependencies between categorical and non-categorical configurations, the instance is again maintained according to a partially categorical configuration. Because the non-categorical spaces are the leaves of the search space graph, they are the last to be activated and treated as a single unit.

[0085] consider Figure 8Here's a step-by-step example of the activation process. Initially, all search states are empty. For nodes {A, D}, let the ε-Greedy search select configuration {b, z}, which activates nodes {B} and {Z} according to constraints. Classified nodes are processed first, so let the ε-Greedy instance of the current partial configuration {b, z} select configuration {x'} for node {B}. This produces a partial configuration {b, z, x'}, which we use to finally activate the Nelder-Mead instance of node {Z} to produce the largest partial configuration. Let the application return a measurement of 0.7 for this configuration, then the search state after this first tuning iteration is:

[0086]

[0087] In the next tuning iteration, let the search instance of node {A,D} return configuration {c; z'}. This partial configuration only activates node {C}. Assume that the node returns {y'} for the search instance of this configuration, and the value of the measured maximum partial configuration is 0.9. Highlighting the updated value, the search state after the second tuning iteration becomes:

[0088]

[0089] For the third tuning iteration, suppose the instance {A, D} returns {c, z}, and {C} is activated. Let the ε-Greedy instance of {c, z} return {y}. Then, Figure 9 The current activation state for this partial configuration is shown. The hierarchical search yielded the partial configuration CA = c, C through two steps of applying the ε-Greedy algorithm. C =y,C D =z, and thus activates the local spaces {A, D}, {C}, {W}, {Y}, and {Z}, all highlighted in the figure. Although {W} is activated simultaneously with {A, D} due to its lack of constraints, it is a non-classified subspace and is therefore only activated when a partial configuration is maximized relative to the classified parameter. For a given partial configuration, the highlighted non-classified spaces {W}, {Y}, and {Z} are activated together for a single Nelder-Mead instance. Assuming the application reports a measurement of 0.6 for the largest obtained partial configuration, the search state after the third iteration becomes:

[0090]

[0091] Note that at this point, there are two concurrent search instances of {C}. Decomposing the search space in this way not only helps to effectively reduce its size but also allows for more efficient rejection of configurations. Configurations are rejected when the application heuristically determines that a configuration will result in poor performance. However, usually only a few combinations of parameters lead to this. Consider the heterogeneous offloading example ( Figure 7 If the parallelized code is trivial, or only performs a small number of iterations, the application can correctly determine that offloading the computation to the GPU due to overhead is highly likely to actually degrade performance. The only parameter relevant to this is P, which controls the platform selection. The remaining parameters do not contribute to the decision on offloading. By searching the space graph, we allow the application to also reject partial configurations. For example, once configuration C is built... P =Partial GPU configuration allows applications to reject certain configurations. Therefore, we can make rejection decisions early in the configuration process. Rejecting configurations early further reduces the number of activated nodes.

[0092] Two types of models for the prediction component of hybrid tuners have been explored. The first is the tabular nearest neighbor approach. The second uses generalized reinforcement learning, employing functional approximation to model the relationship between metrics, configuration, and runtime. The tabular predictor is simple: each observed state is recorded in a table along with the best known configuration for that state. When the tuner enters a new metric state, it queries the table to obtain the geometrically closest known state. While nearest neighbor search can be efficiently implemented, this method has two main drawbacks. The motivation for nearest neighbor prediction is, of course, that the same configuration might be beneficial for similar states. However, the query results can be poor when the new state is large enough to be found in all states in the table. This problem can be mitigated by not using the query results when the distance is too large, but this requires defining a distance threshold. The second drawback is the size of the table: storing information for every possible state is impractical, especially when the states are not discrete. The table can only be populated through exploration, or its size could be limited. However, this would disable the possibility of learning during exploration, or it would require adding another hyperparameter to limit the size. In this invention, we provide application developers with size constraints and control over metric resolution.

[0093] Generalized reinforcement learning offers a way to circumvent the drawbacks of table-based prediction. While classical reinforcement learning (RL) techniques are inherently table-based, generalized methods approximate the table through a function. By approximating the relationship between metrics, configurations, and runtime as a function, RL eliminates the need to store every state or choose which states to store. To predict a good configuration for a state, this function is used to find the configuration that maximizes the estimate of the reward for the state-configuration pair. Unfortunately, maximizing the function is expensive, which can hinder the applicability of hybrid tuning to online tuning scenarios. Instead, we therefore retain a record of the best configurations found during exploration and only consider these configurations as candidates for prediction. While this can still be large if exploration in hybrid tuning is frequent enough, it is typically much smaller than a complete state table.

[0094] In this invention, we implement a Greedy-GQ variant of Q-Learning as our function approximation predictor. We approximate the table as a function. in These are features of the metrics defined by the application developers. By default, this invention uses the normalized radial basis function as a feature: it defines...

[0095]

[0096] Where x = (s, a) is the concatenation of the index state vector s and the configuration (or action) a. This is similar to a regular radial basis function (RBF). c (x)=exp(-ω c (x-μ c ) 2 Compared to the previous version, the normalized variant guarantees a smoother spatial boundary and eliminates the "gaps" between basis functions. The center μc and width ωc are configurable parameters. The standard update equation of the Greedy-GQ algorithm is controlled by three hyperparameters: α, β, and γ. α and β are both learning rates. The third parameter γ determines the effect of the expected future reward on the update. However, for our application, we assume this effect is always zero. Considering future rewards allows information about future states to be propagated into the update. However, we assume there is no correlation between our current action and any subsequent state changes. Of course, this assumption only holds true for applications where the context state is independent of the tuning parameters. Setting γ to zero eliminates β and simplifies the update equation to...

[0097]

[0098] The automatic tuning system implements a hybrid online automatic tuning method as a black-box tuner. Hybrid tuning combines online search and model-based prediction to provide the following three main features:

[0099] Context sensitivity: The runtime of the tuning kernel depends not only on the configuration of the tuning parameters but also on the dynamic tuning context. In this invention, a table memory records the configuration and tuning context state. When a previously seen state reappears, the previous configuration for that state is used. In addition to the table memory, a predictive model is constructed to predict a good configuration for unknown context states.

[0100] Effective Exploration: To explore the search space and guide the tables and predictive models, the tuner uses online search. The implemented set of default search algorithms can be extended by application developers.

[0101] Efficient Exploration: While online search techniques are known to find (locally) optimal configurations relatively quickly, there is still room for improvement. The search algorithm described here, known as hierarchical search, significantly accelerates search time by automatically utilizing the structure of the search space and avoiding known poor configurations.

[0102] Current autotuning designs fall into two directions: optimization is typically based on machine learning or models, or it's search-based. These two alternatives offer different advantages and disadvantages. When parameter configurations are generated by a machine learning or model-based predictor, they can react instantly to changes in the dynamic context: the predictor is queried for the updated context state and returns a single parameter configuration. On the other hand, with search-based tuning, each change in context incurs overhead: to find a new configuration, the search must restart, sampling for the new configuration until a new (local) optimum is found. However, a drawback of using prediction is that the quality of the configuration depends on the accuracy of the predictor.

[0103] If a predictor is handcrafted, its accuracy depends on how precisely its designer models the application and dynamic context. If a predictor is learned, its accuracy depends on how well the learned model generalizes from the training data and how well the data reflects reality. While machine learning eliminates the need for meticulous human model designers, it typically requires a large number of training samples to achieve accuracy. With offline training, the quality of the learned predictor is limited by the extent to which the provided training data represents the data that will be observed in production. On the other hand, online training can learn from data that actually occurs in production, but there is a bootstrapping problem: if the initial model is inaccurate, the initial predictions will be suboptimal, and performance will degrade. With search-based tuning, there is no bootstrapping problem, and the quality of the found configuration is independent of any prior sample data.

[0104] Through hybrid tuning, we propose a compromise between these two approaches. Hybrid tuning combines prediction and search, using online search to explore the search space and continuously feeding training data to update the prediction model. Because training is performed online, the prediction is insensitive to any prior sample bias. The hybrid tuning workflow is as follows: Figure 9 As shown: Whenever a change in dynamic context is detected, hybrid tuning can choose between using a search to explore the search space or leveraging a trained prediction model. Any performance feedback provided in either mode is used to improve the prediction model. The configuration table also stores the best known configuration for the observed context states.

[0105] To achieve hybrid tuning, we model the changing context and the response to change as a Markov decision process (MDP). This model represents context changes as process state transitions and uses MDP decision-making to choose between exploration and exploitation. In the next section, we briefly introduce our process model. We then discuss how we achieve the detection of context changes and our approach to balancing exploration and exploitation.

[0106] To represent the context-sensitive problem in automatic tuning systems, we construct an infinite-level MDP. The MDP is a quadruple (Σ, A, Φ, ρ).

[0107] In this model, Σ is the state space, and A is the space of possible actions. The state space represents the dynamic context of the tuner's operation. In the MDP literature, the tuner will be referred to as the "agent." Therefore, changes in the context constitute changes in the state space, causing the agent to select actions. We define the action space A = T as the set of possible configurations.

[0108] The agent selects a configuration based on a policy commonly referred to as π(s), s∈∑, which determines the action for each state. The action model Φ:∑x Ax∑→[0,1] describes the probability Φ(s,a,s')=p(s′|s,a)= when transitioning to state s' while in state s and performing action a. We cannot define a “general” action model because we do not know whether the chosen action has any practical effect on the state transition. Consider again the library parallelization example discussed earlier; actions affect the runtime of the called library functions, but we cannot reasonably assume that they have any effect on the context state reported by the client to the library. Therefore, the action model must be learned by observing state transitions.

[0109] Finally, the reward function ρ: ∑×A×∑→[0,1] determines the reward r=ρ(s,a,s′) for transitioning from state s to s′ under action a. We set r=-m, meaning that the immediate reward achieved after each action selection is only a performance measure. The negative sign is used to transform the minimization objective of autotuning into the maximization objective of the agent.

[0110] Before an autotuner can dynamically adapt to changes in context, it must first detect them. To achieve this, we must first model and quantify the context. Therefore, we need to determine the instantaneous states of both the application and the system. For this purpose, we introduce context metrics. As a "probe," it is used to monitor various static and dynamic properties of the context at each time point t. Application context metrics set and system context metrics set Together, models generate the true dynamic context K at a given point in time. In the remainder of this section, we will only refer to approximate contexts, so terms may be used interchangeably.

[0111] In hybrid tuning, the index can represent both discrete and continuous quantities. While this is necessary to support general applications, it strongly influences MDP models. Because the context-formed state space of the process model is transformed into a continuous Markov decision process by incorporating a continuous index, this process exhibits significantly different solutions and convergence guarantees.

[0112] The distinction between discrete and continuous metrics also impacts change detection. In a discrete context, detecting change simply means observing changes in the metric's value. Of course, the same applies to continuous metrics, but we define two additional scenarios. Continuous quantities often exhibit high frequency. For example, consider the system load metric again: measuring it essentially produces a different value each time because the system running the application is never truly at rest. Here, smoothing the response function with some low-pass filter is helpful, for example, by averaging multiple samples of the metric over time. While changes in metric values ​​may be high-frequency, they tend to be low-amplitude: for example, when the system is busy, the system load typically remains around 100%. Furthermore, it's easy to assume that small changes in metric values ​​have only negligible impacts on performance, thus affecting the optimization of the current configuration. For example, if the system load changes from 98% to 99%, the impact is likely to be unmeasurable. Both smoothing and downsampling can help address this impact. Alternatively, state changes can also be detected purely by observing the metric's effect: monitoring the measured function, we can detect state changes when the function value changes drastically.

[0113] With a quantized context and a device capable of detecting changes, the next step is to react to those changes. Within the framework of our model, the reaction to changes is implemented by a policy π. We borrow from the field of reinforcement learning to develop our policy, which is loosely based on ε-Greedy, a policy frequently used in this domain: with probability ε, we use a hierarchical search to explore the search space when a state change is encountered. With probability 1-ε, we utilize a predictor that is trained from a sampled configuration during exploration.

[0114] Because the application-tuner interaction is iterative, neither option is "atomic," as they are single responses to state changes. Instead, the hybrid tuner can be in either explore or exploit mode. When in the former, it produces a configuration as instructed by a hierarchical search. Being in exploit mode means there is only a single configuration update, i.e., upon entering that mode. On the other hand, in explore mode, multiple configurations are sampled. Therefore, another state change may occur while the search is still in progress. In this case, the current search state is saved, and the search continues when the previous state reappears.

[0115] Having described the automatic tuning system 20, an example implementation of the compiler 10 will now be described. The specific form of the implemented compiler is not limited to the described embodiments, which are described for illustrative purposes only. In principle, parallelizing a program to perform tasks on a heterogeneous system can be done manually and then compiled, or the compiler can perform parallelization. The number of tasks as parallel tasks is itself a tuning parameter. If the program is written in the form of parallel tasks, further parallelization is not required. Automatic parallelization is preferred in this invention because it can be advantageously used with the automatic tuning system, where parameters such as the number of threads and whether shared memory is used are identified and can be used for tuning.

[0116] In this example, the compiler parallelizes the loops in the input program across multiple platforms and checks the results via a runtime library for interoperability with the autotuner. To analyze and transform the program, we implement two techniques. First, the compiler attempts to analyze the program using a polyhedral model. If this fails due to the constraints of the polyhedral model on the representable program, we revert to classic dependency testing and more ad hoc transformations.

[0117] In the following sections, we describe parallelization based on the polyhedral model, followed by ad hoc methods. We then discuss how the compiler and the automatic tuning system interact to incorporate analysis information that the compiler can derive into the tuning decisions.

[0118] Generating code for collaborative heterogeneous execution from a polyhedral model is a three-stage process. It begins with computing the SCoP scheduler to maximize data parallelism. The next step is to partition the outermost parallelizable loops for the relevant platforms—here, CPU and GPU cores—and then map the partitions onto the platforms. Finally, the optimized and mapped scheduler is translated into actual parallel code for the platforms. We describe our partitioning and mapping methods, code generation approach, and interoperability with the autotuner below.

[0119] The heterogeneous parallelization strategy for the polyhedral model is based on Polly-ACC, thus making extensive use of the PPCG GPU mapper described by Sven Verdoolaege et al. in "Polyhedral Parallel Code Generation for CUDA" in ACM Transactions on Architecture and Code Optimization (TACO) - Special Issue on High-Performance Embedded Architectures and Compilers 9.4 (January 2013), 54:1-54:23. With its help, we first compute a scheduling tree optimized for data parallelism from SCoP. Before mapping the optimized schedule to the GPU using the PPCG code generator, we transform it by introducing a first tunable partition and a second tunable platform mapping.

[0120] Essentially, partitioning the loop requires copying it and then modifying the boundaries of the copy: one copy handles the lower part of the iteration, and another handles the higher part. To make it tunable, tuning parameters determine the iterations that separate the lower and higher parts. Mapping the partitions to platforms is essentially the same operation: generate a partition instance for each platform and add tuning parameters to dynamically select the instance. Note that these two steps are parallelized only for the outermost loop. The repeated process recursively processes all the data in parallel loops. In the following sections, we will describe in more detail how we implement this process within the polyhedral framework.

[0121] We recursively transform the scheduling tree from top to bottom, searching for the outermost data-parallel loop within nested loops. To insert a partition, we first find the outermost band with a leading parallel dimension. The set of leading parallel dimensions corresponds to the set of outermost data-parallel loops. Each loop in this set is split into two parts: one for high indices and one for low indices. Although the number two seems self-evident, as it targets two platforms, this method supports any number of partitions.

[0122] We partition the outermost parallel dimension of the band by introducing a set node between the band and its parent, which has two filter child nodes that implement the partitioning constraints. The newly introduced parameter R0 defines the partitioning of the outermost parallel dimension of the band. R0 is subject to the same constraints as the iterator for that dimension. In the partitioning filter, this parameter becomes the new upper and lower bounds for the iterator to select the high and low indices, respectively. The original band node is copied and attached to the filter.

[0123] This process is recursively repeated for the remaining leading parallel dimensions of the band. In summary, when dividing into d dimensions, we insert a total of 2... d -1 set nodes, producing 2 of the original band. d Each copy. To achieve platform selection, we follow the exact same path. For each strip node we create, between the strip node and the ensemble filter we created during partitioning, we insert a new sequence node as its direct parent node. We introduce a new parameter P for platform selection in the i-th cycle. i Low and P i High P i r ∈{0,1} is used for high and low index partitions. The two filters for the new sequence node implement the constraint P for selecting the CPU. i r =0 and the constraint P for selecting the GPU i r =1. Copy the node again, and we finally achieve this by modifying P. i r We use the PPCG mapping algorithm to parallelize it for the GPU. However, for its sibling nodes, we insert a marker node that we use as the starting point for generating OpenMP code for CPU parallelization during code generation.

[0124] Figure 10 It shows Figure 11 This process is illustrated in the example scheduling tree.

[0125] The first dimension is generated by partitioning the nodes in the outermost layer of the tree. Figure 10 The tree shown in 'a' illustrates two newly inserted filters, using the partition parameter R0 as the lower and upper bounds of the iterator parameters, respectively. Then, Figure 10 b illustrates the platform mapping for one of the two partitions. A new parameter P0 is introduced, and the inserted filter thus generates GPU and CPU paths. The PPCG mapping algorithm is then invoked for the GPU path.

[0126] After applying all scheduling transformations, we then transform the scheduling tree back to LLVM IR. This process is built on top of the Polly-ACC code generator. Therefore, most of the implementation for generating code for partitioning and mapping is complete.

[0127] The Polly-ACC code generator is extended in two main ways because it is designed to handle only GPU offloading. To incorporate CPU parallelism, we integrate the code generator with the existing, experimental Polly OpenMP backend. This backend emits code for the libgomp OpenMP runtime library. The second extension is the instrumentation of the parallelization program to interoperate with the autotuner. During mapping and code generation, we extract several tunable parameters from the parallelization program. First and foremost is partitioning, R... i ∈[0; 1), and platform mappings i∈[0; d) and r∈{high, low} for P i r ∈{CPU,GPU}, where d is the number of outer loops that are parallelized. For example, parallelizing two outer loops in the parallel source region results in six tunable parameters. Two partitions plus a platform map determiner for four partitions. Therefore, multiple partitions can be offloaded to the same platform at runtime. This is a potential source of additional overhead, but unfortunately, it is unavoidable because non-affine constraints that could circumvent this problem cannot be expressed in the polyhedral framework.

[0128] In addition to these job allocation parameters, we can also obtain additional parameters from the mapping and code generation process. CPU code generation exposes parameters that control the number of threads. It can be freely configured at runtime. On the other hand, unfortunately, tuning GPU mapping is not straightforward. Configuration options affecting the mapping must be embedded in a polyhedral framework. An example is the number of threads per GPU block. Scheduling threads is a tiling operation, i.e., the strip is tiled into blocks of the specified number of threads. Then, the tiling point loop describes the kernel computation for each thread, and the tiling loop implements the execution of different thread blocks. Parameterizing the number of threads turns it into a parameterized tiling problem, for which there is currently no efficient general solution. Since we are unwilling to give up tuning GPU platform parameters, we are forced to take another approach. To make the number of threads tunable, we enumerate all configurations and statically generate corresponding versions of the target region code. We express this directly as set nodes in the scheduling tree, with their filter child nodes representing individual configurations. Thus, we obtain a single large scheduling tree. Using the same approach, we also add parameters to determine the block height (B). i,r To enable or disable the promotion of local variables to shared memory (SM). i,rThis involves looping through shared or local memory to move data in and out. Block height parameter B i,r The shape of the GPU thread blocks is directly controlled. Therefore, they affect the locality of access to global memory and the ability of thread bundles to access each other jointly. Table 3 summarizes the complete list of parameters and their default ranges. The value ranges for the block size and height parameters are not exhaustive. The chosen ranges represent a trade-off between covering a sufficiently large possible memory area and the number of versions we need to generate.

[0129] Table 3

[0130]

[0131]

[0132] The polyhedral model cannot represent parallelizable loops in some cases. However, by relaxing its constraints, it is sometimes still possible to analyze loops and detect data parallelism. Two examples where this occurs are function calls and inner loops without static control. Function calls are not modeled by the polyhedral model because instructions within a function cannot be scheduled.

[0133] Because of this, polyhedral compilers typically inline functions first and then eliminate function calls. However, for us, this would require speculatively inlining all function calls to be able to detect parallelism, which would be very expensive. On the other hand, loops without static control simply cannot be represented in the polyhedral framework. However, such loops can be benign when parallelization is the sole objective: if such loops do not access memory or only access memory that can be proven to be local, then parallelization is unaffected. Therefore, when polyhedral modeling fails, we fall back to a second analysis and transformation pipeline, using classic dependency testing and arbitrary transformations.

[0134] In this section, we will introduce pipelines, analysis performed by the compiler to detect parallelism, and platform-specific program transformations.

[0135] The first step in parallelizing a given program is to detect the parallelizable parts of the program. In this invention, we focus solely on finding data-parallel loops during this stage. The scope can be extended to include loops that incorporate reduction, which can be viewed as a special case of data parallelism with less stringent constraints on data dependencies. In addition to the restrictions on data parallelism, we impose further constraints on the loops we analyze to simplify implementation.

[0136] Most importantly, we only consider top-level loops as candidates. Secondly, we assume that the loops have a specific structure in the control flow graph. The expected structure is as follows: Figure 12As shown: Loops must be defined by a single-in, single-out control flow subgraph, meaning there exists a single basic block outside the loop whose only successor is the loop header block, and vice versa, the exit block. Furthermore, there must be a unique exit block whose branches are the exit block and the header block. Therefore, the exit block cannot be the loop header block. Note that this structural restriction is generally weak, as most loops can be normalized to adopt this form. However, a notable counterexample is a loop with multiple exit blocks, which are basic blocks outside the loop to which the loop body can jump. There is no general way to merge these blocks. Finally, we require the outermost loop to exhibit static control. The run count (i.e., the number of iterations) must be statically computable. More precisely, this run count can be a constant value or a provably loop-invariant variable. For all loops that pass these checks, the compiler performs its parallelism detection analysis. This is classic dependency analysis, attempting to prove, by disproving, the data dependencies carried by the loop. The LLVM framework includes various such analyses. The compiler extends the existing facilities of the LLVM framework to support inter-procedural analysis. It performs dependency analysis in loops containing function calls. To describe the effect of the call, we summarize all functions called within the candidate loop. A function summary is essentially an exhaustive list of all memory accesses within a function that affect its pointer arguments or global variables. These accesses are precisely those that the caller might be able to observe. Generally, summaries need to over-approximate the scope of accesses because accurate aliasing analysis is not practically available: when analyzing memory accesses, it is often not entirely clear which argument or variable is being read or written. In the worst case, the summary must assume that the function reads or writes every byte of memory in the program. This might happen, for example, when accesses occur indirectly, i.e., through pointers loaded from arguments or global variables. With the help of summaries, function calls can now be analyzed by expanding the summary at the call site, effectively “inlining” the function. To disprove the existence of any circularly carried dependencies, the compiler extended the LLVM implementation of the dependency testing algorithm of Goff, Kennedy, and Tseng (Gina Goff, Ken Kennedy, and Chau-Wen Tseng. "Practical Dependence Testing" In: Proceedings of the ACM SIGPLAN 1991 Conference on Programming Language Design and Implementation. PLDI'91. New York, NY, USA: ACM, 1991, pp. 15-29) to support the conclusion.

[0137] To broaden its applicability, the compiler was extended to support parallelism schemes that actually exhibit cyclically carrying dependencies: parallel reduction. Parallel reduction differs from simple data parallelism because it allows for specific data dependencies, for which values ​​read from and written to conflicting memory locations depend only on associative operations. A typical example is adding all elements of an array to an output variable. Each iteration reads the output variable, increments the array element, and writes it back to the same location. If the increment operator is associative, this operation can be performed in parallel. Because reduction can be seen as a slightly degenerate form of data parallelism, it is also supported.

[0138] After detecting a parallelizable loop, the compiler then generates parallel code. Code generation is implemented by a scalable set of target plugins. The purpose of these target plugins is to generate platform-specific code as well as management code needed to offload computation. The compiler then essentially becomes the driver of the transformations implemented by the plugins. Therefore, each plugin must emit three pieces of code: a prologue, a target region, and a coda. The interactions between these parts and the target are as follows: Figure 13 As shown in the diagram, synchronous operations are represented by solid arrows, and asynchronous operations are represented by dashed arrows. In the introduction, the target is expected to set the target platform memory and execute the target region asynchronously.

[0139] This may require allocating space for data accessed within the target region, including both arrays and scalar variables, and initiating data transfers. However, this step can be omitted if the host and target platforms share the same physical memory. The compiler processes the prologues of each target sequentially, followed by all the tails. In the tail stage, targets await the completion of their target regions and perform post-processing on the results if necessary. For example, post-processing is necessary for parallelized reduction. They can be done asynchronously to overlap computation and all data transfers. When the transformation additionally includes a reduction loop, the tail additionally aggregates the reduction results for each target slice of the work.

[0140] Currently, we have implemented two main target plugins, one for CUDA and the other for OpenMP targets.

[0141] In the following sections, we briefly discuss the design and implementation. Finally, we describe the target-specific tuning parameters exposed by the plugin and how we model dynamic workload partitioning.

[0142] Regarding the OpenMP target plugin, locating OpenMP on the CPU is straightforward. Because of shared memory, complex allocation management is unnecessary. To interoperate with the OpenMP library API, the plugin needs to generalize the target region code into a separate function, which is called by the OpenMP runtime. Generalization primarily involves replacing raw boundary checks with boundary checks for iterative blocks per OpenMP thread, and passing scalar variables and array pointers to the target region function. Therefore, in the prologue, the target plugin copies the scalars and pointers and calls the OpenMP runtime. The epilogue simply waits for the thread to complete.

[0143] Regarding the CUDA target plugin, in most systems, the host and GPU do not share memory. Exceptions include mobile graphics units and system-on-a-chip platforms, which we will not consider further here. Therefore, the primary task of the CUDA target plugin is to manage the allocation and data transfer between the target platform and the host. Unlike OpenMP targets, we need to transfer both scalar variables and complete arrays to the GPU. The number of scalar variables is statically known, making copying easy. On the other hand, the size and location of arrays are dynamic properties. Worse still, both can change at runtime, and while the size of the array can be constant, the amount of data to be transferred is not. Firstly, changes in the program's input or parallelization region affect the array size, thus changing the amount of data required. Secondly, these changes are also caused by variations in workload distribution due to tuning, meaning they are frequent. Therefore, the CUDA target plugin dynamically manages memory through a runtime system, which is further detailed below.

[0144] After allocating and copying all the data to the GPU, the prologue then executes the target region, which is implemented as a CUDA kernel. Finally, the prologue also initializes the transfer that will move the results from the GPU device back to host memory. Except for allocation, all operations performed in the prologue are asynchronous with respect to the host, but strictly sequential with respect to each other. Therefore, CUDA API calls used for copying to the device, starting the CUDA kernel, and copying from the device do not wait for their respective operations to complete. The waiting then occurs in the tail, which synchronizes the host thread with the final copy completion.

[0145] For a given source region, the system creates a single tuner instance, which we call the tuning context for that region. For this instance, it inserts special prologues and codas, which execute before all target plugin prologues and after all codas, respectively. Within these sections, the runtime environment invokes the tuner. The prologue updates the configuration of all tuning parameters in the context and starts time measurement; the coda stops time measurement and updates the configuration.

[0146] The runtime system performs three main tasks: managing and asynchronously executing target regions, managing memory allocation and transfers, and interface connections between parallelized applications and tuners. The compiler automatically links these into the generated binaries. Currently, host-side target region management only needs to handle the CUDA kernel. The compiler translates the kernel into CUDA PTX (CUDA-specific assembly code) and embeds it directly into the parallelized binary.

[0147] To boot the kernel, the runtime system loads the assembly code and JIT compiles it for the CUDA driver. The result is cached for future execution. Once compiled, the kernel can be booted. Device-side memory is managed by a custom dynamic memory allocator. This allocator is part of the runtime system and maintains a single contiguous block of memory. In the prologue, the size of all required array slices is dynamically known, and the allocator is invoked to remap all arrays accessed in the target region into the managed block, increasing it if necessary. If the host pointer points to overlapping memory regions, the allocator automatically merges their mappings accordingly.

[0148] Figure 14 An example of this process is shown. The allocator has allocated a contiguous array of 40 bytes. Mapping arrays A, B, and C into this memory only takes up 34 bytes. Although the size of A is 32 bytes, and the size of C is unknown, the loop will only access 16 bytes of C and 20 bytes of A, that is, directly accessing 16 bytes through A at offset 0, and indirectly accessing 16 bytes through B at offset 4. To calculate the mapping, the allocator represents each array Ai as a triple (Addr... Ai Size Ai Offset Ai Offset Ai Initially 0. These triples are sorted in ascending order if and only if Addr Ai +Size Ai >Addr Aj At that time, A i and A j The subsequent triples are combined. If this inequality holds, then A i and A j They overlap in memory. After merging, we set the offset. Aj =Addr Aj -Addr Ai and Addr Aj =Addr Ai Then, the merged triples can be simply mapped onto a continuous space, with additional assignments if necessary.

[0149] Finally, the runtime system's tuning interface enables interoperability with the library. It maintains a registry of instantiated and running tuners and provides an interface to this registry for parallelized programs. Finally, it also implements an algorithm for calculating the iterative boundaries of the target region's execution.

[0150] Most automated parallelization or auto-unloading methods rely on heuristics, cost models, and machine and performance models to make tuning decisions. With our current compiler and auto-tuning system, we combine these two techniques. The auto-tuner allows clients to reject configurations without actually measuring them. In the compiler, we use this mechanism to bridge the gap between static performance models and auto-tuning. For this purpose, the compiler includes a small embedded DSL compiler that developers can use to implement rejection rules. An example of such a rule can be as simple as the following: Threshold.

[0151] Here, ComputeVolume is a performance model that predicts the number of dynamic instructions executed in the source region. Assuming each instruction takes the same amount of time to execute, ComputeVolume is an accurate estimate of the time required to execute the source region. On the other hand, when ComputeVolume models the target region, it additionally depends on the tuning parameters. This allows us to make a priori runtime predictions of the transformed and tuned program. The heuristic implemented by the simple rule above can be intuitively interpreted as "if it will perform enough computation in this configuration, then only execute that region on the GPU." As a proof of concept, we implemented the above rule in our compiler.

[0152] In addition to `ComputeVolume`, the compiler embeds several similar profiling models into the application during transformation. These include the number of load and store instructions executed, and arithmetic strength. The latter describes the ratio between the `ComputeVolume` of the source region and the amount of memory it accesses. Therefore, arithmetic strength is a measure of the number of operands performed per transferred memory byte. Typically, profiling models are parameterized. For example, the number of load instructions executed in the source region depends on the number of loop iterations. This number often depends on the program's input or the current region. Therefore, the compiler cannot statically evaluate the models; it must defer them to program execution. For each model, the compiler generates code to compute its value at runtime. The model is then evaluated at runtime as needed. Furthermore, because they are parameterized, we can use the models for a second important purpose. The combination of `ComputeVolume` with the number of loads and stores provides an approximation of the runtime of parallelized loops. This means that the parameters in these expressions directly affect the loop runtime. Because the constructed parameters are program variables, they are perfect candidates for use as mixed autotuning metrics. Once the models are computed, the compiler extracts all their parameters and registers them accordingly with the autotuner.

[0153] Two mechanisms are supported for deriving the analysis model from the input program: based on a polyhedral model, or, if the source region cannot be polyhedralized, based on classical inter-process dataflow analysis. In the latter case, we recursively compute the dynamic instructions of the region. For example, the instruction count of a loop is the instruction count of its body multiplied by the number of loop iterations. However, this number often does not satisfy the static control criterion. That is, it is neither a constant nor a provably invariant variable. In this case, we underestimate the instruction count. Therefore, evaluating the model does not produce an exact value, but rather a lower bound on the specific count, which is sufficient for our model users. We perform this approximation when an exact count is impossible, especially for for and while loops with indeterminate iteration counts, if conditions with different instruction counts in their branches, and calls to functions that cannot be inter-process analyzed. However, if the polyhedral model is applicable to the source, no approximate count is needed, and we can determine the accurate instruction count. Using libbarvinok, we obtain the analysis model directly from the polyhedral descriptions of the source and target regions. This library provides an implementation of the Barvinok algorithm, which efficiently counts the number of points in an integer polyhedron.

[0154] Generating code for the analysis model expression is straightforward. Both data flow analysis and polyhedral models generate piecewise polynomial expressions and parameters in the program parameters. If the target region is modeled, tuning parameters can also be generated.

Claims

1. A method for configuring program parameters for computation during the runtime of a computation program in a heterogeneous computing system, the method comprising: Receive the transformation of the computing program, the transformation including one or more computing applications; One or more tuning parameters are generated for each computing application, and these one or more tuning parameters are categorized. as well as For the computing application to be optimized, the following operations are recursively performed during runtime: adjusting one or more tuning parameters of the computing application; executing the computing application using the adjusted one or more tuning parameters; obtaining performance metrics of executing the computing application using the adjusted one or more tuning parameters; and determining whether the adjusted one or more tuning parameters provide an improvement in the performance metrics and whether a termination criterion has been met to end the tuning. The system stores index characteristics of the dynamic state of the computational application being optimized. These indexes are used during optimization to determine whether a known optimization can be applied to the dynamic state indicated by the stored indexes. Specifically, characterizing the one or more tuning parameters as their corresponding categories is used to decompose the search space into a categorical space and a non-categorical space, and Specifically, for both the categorized space and the uncategorized space, a separate search state is maintained for each space-related configuration.

2. The method according to claim 1, wherein, The method also includes generating the transformation by a compiler adapted to evaluate the computation program to determine computational applications that can be executed in parallel by different processing units.

3. The method according to claim 1 or claim 2, wherein, The optimization is performed using tuning routines selected from model-based prediction and online search processes.

4. The method according to claim 3, wherein, The online search process is used to provide training data to update the model-based prediction process.

5. The method according to claim 1, wherein, The one or more tuning parameters are used to determine whether the optimization process can be performed within a limited search space.

6. The method according to claim 2, wherein, The transformation was generated using a polyhedral parallelization compiler.

7. The method according to claim 6, wherein, The polyhedral parallelization compiler converts the source code into an intermediate representation, and further converts the intermediate representation into binary code suitable for execution on the computing platform constituting the heterogeneous computing system.

8. A computer program product programmed to optimize a computing application by recursively performing the following operations during runtime: adjusting one or more tuning parameters of the computing application; executing the computing application using the adjusted one or more tuning parameters; obtaining a performance metric for executing the computing application using the adjusted one or more tuning parameters; and determining whether the adjusted one or more tuning parameters provide an improvement in the performance metric and whether a termination criterion has been met to end the tuning. in, The system stores metric characteristics of the dynamic state of the computational application being optimized, these metrics being used during optimization to determine whether a known optimization can be applied to the dynamic state indicated by the stored metrics. Specifically, characterizing the one or more tuning parameters as their corresponding categories is used to decompose the search space into a categorical space and a non-categorical space, and Specifically, for both the categorized space and the uncategorized space, a separate search state is maintained for each space-related configuration.

9. The computer program product according to claim 8, wherein, The computer program product is adapted to perform the optimization using tuning routines selected from model-based prediction processes and online search processes.

10. The computer program product according to claim 9, wherein, The computer program product is adapted to use the online search process to provide training data to update the model-based prediction process.

11. The computer program product according to any one of claims 8 to 10, wherein, The computer program product is adapted to use one or more tuning parameters to determine whether the optimization process can be performed in a limited search space.

Images