Automatic Tuning of Heterogeneous Computing Systems

Abstract

The present invention provides a method for configuring program parameters during runtime of a computing program for computation on a heterogeneous computing system, where the compiled program is processed in an autotuning system to optimize parameters of the application for processing on a heterogeneous system having, for example, CPU and GPU cores.

Description

[Technical Field]

[0001] The present invention relates to computing systems having various types of processors and techniques for arranging the various tasks to be divided among the processors for parallel computation in an efficient manner. [Background technology]

[0002] Such heterogeneous computing systems include supercomputers (also called high-performance computers) that include both CPU (Central Processing Unit) and GPU (Graphics Processing Unit) cores, and can also support more specialized computing units such as FPGAs (Field Programmable Gate Arrays), DSPs (Digital Signal Processors), and quantum computing units.

[0003] Individual computational tasks may be better suited to execution by one type of processor than another, and configuring various tasks to be processed in parallel can be a substantial task. Configuring tasks for processing by different units to optimize the degree of parallelism is called tuning. Tuning can be either "offline," performed before runtime, or "online," performed during runtime. Program developers can implement tuning using their skill and knowledge, but such manual tuning is time-consuming. It is desirable to develop automatic tuning or autotuning.

[0004] Autotuning is described, for example, in Siegfried Benkner et al. "Automatic Application Tuning for HPC Architectures" (Report from Dagstuhl Seminar 13401 29 September - 4 October 2014, pp 214-243, https: / / www.dagstuhl.de / 13401), which is incorporated by reference for all purposes.

[0005] A problem with online autotuning is that it can increase the runtime for system users as different configurations are implemented while the system searches for the optimal configuration. Summary of the Invention [Means for solving the problem]

[0006] The present invention provides a method for performing online autotuning that is comparable to known online tuning techniques while reducing the required run time.

[0007] The present invention provides a method for configuring program parameters during runtime of a computing program for computation in a heterogeneous computing system, the method including: receiving a transformation of the computing program, the transformation including one or more computing applications; generating, for each computing application, one or more tuning parameters, the one or more tuning parameters being classified into classes; and iteratively adjusting, during runtime, the one or more tuning parameters of the computing application to be optimized, executing the computing application using the adjusted one or more tuning parameters, obtaining performance metrics for the execution of the computing application using the adjusted one or more tuning parameters, and determining whether the adjusted one or more tuning parameters result in an improvement in the performance metrics and whether termination criteria for terminating the tuning are met, wherein an indicator characteristic of a dynamic state of the computing application being optimized is stored, and the indicator is used in the optimization to determine whether a known optimization is available for the dynamic state indicated by the stored indicator.

[0008] In a further aspect, the present invention provides a computing entity programmed to optimize a computing application by iteratively adjusting, during runtime, one or more tuning parameters of the computing application, executing the computing application using the adjusted one or more tuning parameters, obtaining performance metrics for the execution of the computing application using the adjusted one or more tuning parameters, and determining whether the adjusted one or more tuning parameters result in an improvement in terms of the performance metrics and whether termination criteria for terminating the tuning are met, wherein an indicator characteristic of a dynamic state of the computing application being optimized is stored, and the indicator is used in the optimization to determine whether a known optimization is available for the dynamic state indicated by the stored indicator.

[0009] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 shows a schematic diagram of a compiler and auto-tuning system according to the present invention. [Figure 2] FIG. 2 is a schematic diagram of the autotuning system of FIG. 1. [Figure 2a] 1 is an example of computer code for optimizing the number of threads during autotuning. [Figure 2b] 1 is a schematic diagram of the autotuning system of the present invention, which interacts with the application using feedback passed to memory to update prediction and search state. [Figure 3] FIG. 10 illustrates the operation of a search strategy for autotuning that uses a mirror axis to vary the integer value of configuration C over time. [Figure 4]1 is an example of computer code that provides a search implementation interface. [Figure 5] FIG. 1 illustrates the available state transitions in the Nelder-Mead search algorithm. [Figure 6] FIG. 10 illustrates a graphical representation of decomposing the global search space of a simplified heterogeneous mapping according to parameter classes. [Figure 7] FIG. 1 illustrates an example of decomposing the global search space using application developer-supplied constraints. [Figure 8] An example of a search space graph for a parameter set T={A;B;C;D;X;Y;Z;W}, where {A;B;C;D} is nominal and {W;X;Y;Z} is non-nominal. [Figure 9] This is an example of a hierarchical search for parameters {A;B;C;D;X;Y;Z;W}. [Figure 10] FIG. 12 is an example of schedule tree division and platform mapping shown in FIG. [Figure 11] 1 is an example of a schedule tree for matrix multiplication. [Figure 12] 1 is an example of a loop control flow structure supported by the system of the present invention. [Figure 13] 1 illustrates code sections generated by a target plugin, where solid arrows indicate synchronous behavior and dashed arrows indicate asynchronous behavior. [Figure 14] FIG. 10 is a diagram showing an example of allocation mapping for three arrays. DETAILED DESCRIPTION OF THE INVENTION

[0011] FIG. 1 shows a schematic diagram illustrating an implementation of autotuning according to the present invention.

[0012] This implementation includes a compiler 10 and a runtime auto-tuning system 20. The compiler is configured to parallelize sequential applications for a heterogeneous platform (such as a system with a CPU and a GPU) and perform auto-tuning during execution. The compiler 10 interacts with the runtime auto-tuning system 20 to provide online tuning and optimize the distribution of tasks between CPU and GPU cores.

[0013] Next, the compiler 10 and the auto-tuning system 20 will be described separately.

[0014] FIG. 2 illustrates the autotuning system 20 in greater detail. It shows the logical units that make up the autotuning system 20. The primary component of the system 20 is the hybrid tuning unit 30, which implements the hybrid tuning technique of the present invention and interacts with the application 32 via units called ParameterSpace 34 and IndicatorSpace 36. The hybrid tuning unit 30 is extensible via search and predictive model selection sections 38 and 40, provided by the application developer. Search extensions provide a hierarchical search function that depends on each basic search algorithm configured for use. Model extensions provide predictive models that enable the predictive component of the hybrid tuning unit.

[0015] An application iteratively interacts with the autotuning system (also referred to as an "autotuner" or simply "tuner"). For a previously registered set of application parameters, the application requests a new configuration for these parameters and reports performance metrics for this configuration to the tuner.

[0016] Therefore, integrating an application with the autotuner requires two steps: publishing and registering tunable parameters, and introducing update and feedback calls. Tunable application parameters can be of any type and have any semantics, as long as they are freely configurable by the autotuner. Updating application parameters with the next configuration and reporting performance feedback about the configuration to the tuner encompasses the application's tuning kernel.

[0017] Thus, the update-sample-feedback sequence is the body of the tuning loop. In most cases, the feedback is a runtime measurement, but this is not required. The autotuner tries to find the parameter configuration that minimizes the metric, regardless of the meaning of the metric.

[0018] Figure 2a shows an OpenMP example using autotuning to optimize the number of threads. The NumThreads parameter is registered by referencing the Tuner, with a valid value range of 1 to 16. After each call to add(...), the call to OMPTuner.start() on line 9 automatically updates NumThreads with a new sample value from this range. Thus, any reads from the variable will use the new configuration. A call to OMPTuner.stop() is followed by reporting runtime feedback to the Tuner, using the wall clock time elapsed since the last start() call. start() / stop() APIs are provided for convenience.

[0019] Alternative APIs exist for using arbitrary feedback values. Note that this example is not simplified and shows the actual sequence of calls required to set up and run autotuning, excluding the tuning loop. The Autotuner is generic and extensible. The behavior of the tuning process is fully customizable, and the search algorithm and online learning mechanism can be modified or replaced by application-provided plugins. By being fully generic, the Tuner imposes no restrictions on the type, range, or semantics of application parameters. Fundamentally, the Autotuner provides an abstraction between the application and the search engine.

[0020] The search engine is a driver for any search algorithm, such as the Nelder-Mead algorithm, ε-greedy search, or genetic algorithm. Application developers choose an algorithm from a small selection of built-in searches or implement their own. The search is decoupled from the application through an adaptation layer, which maps between an unconstrained real-valued search space and an often-constrained, arbitrary-valued application parameter space. The library provides sensible default values ​​for this mapping for the most common parameter types: bounded or unbounded integers or floating-point values, as well as non-numeric values ​​that describe the set of choices. The mapping can be modified by the application developer.

[0021] The features outlined above make the Autotuner applicable to most tuning scenarios. In addition to its usefulness, an important feature of the Autotuner is its mechanism for improving amortization. The Autotuner reduces both the total number of configurations attempted and the number of extremely bad configurations attempted. This is achieved through three techniques:

[0022] The first technique is called hybrid tuning. The performance of a tuning kernel generally depends not only on the tuning parameter configuration but also on the system and application state. To take advantage of this fact, the autotuner implements a hybrid autotuning method that combines search, memorization, and model-based prediction. Applications can register any "metric" that characterizes the dynamic state of the application and system with the tuner. One example is abstract "input size." Using online learning, the autotuner automatically detects and responds to significant changes in the dynamic state. Based on the predictor built through online learning, hybrid tuning can immediately switch to the learned configuration. If the exact same dynamic state has been observed in the past, the memorized search results for that state are used. If the exact same dynamic state has not been observed in the past, the trained model is queried for a good predictive configuration. The application does not need to provide training samples. The predictor is automatically trained for all observed metric states from the training samples generated during the search.

[0023] The second technique is to limit the scope of autotuning by introducing parameter constraints. Specific application parameters often impose constraints on a large portion of the parameter space. One example is the parameters controlling algorithm selection. Any tuning parameters included in one of these algorithms need only be updated if this particular algorithm is selected. Thus, algorithm selection parameters control the relevance of the algorithm's parameters. On the other hand, parameters controlled by selection parameters are generally not independent of parameters not controlled by selection parameters; they cannot be tuned separately. When such relevance constraints are expressed by the application developer, the autotuner constructs a hierarchical search space to reduce the dimensionality as much as possible while preserving the dependencies. This further allows the use of different search algorithms for different parts of the search space. For example, introducing the “Parallel” parameter allows computations to be performed serially, in which case the NumThreads parameter is unnecessary. By registering this dependency with the tuner, we can exploit this fact and tune the number of threads only when Parallel is set to “true.”

[0024] A third technique is providing analytical feedback. In many real-world applications, application developers have domain knowledge of parameter configurations and can predict whether a configuration is reasonable. In the compiler example, there are several such predictions possible. For example, allocating two iterations of a parallel loop to the GPU is virtually guaranteed to result in suboptimal performance. With this in mind, the application can reject the configuration without measuring those two iterations. In this way, the autotuner can reduce the number of sampled bad configurations and improve amortized time.

[0025] FIG. 2b illustrates the iterative tuning routine performed by hybrid tuning unit 30.

[0026] Suitable search extensions include, for example, those based on the Nelder-Mead search algorithm and a search algorithm called “ε-greedy.” Predictive model extensions may be based on nearest neighbor prediction.

[0027] Each application 32 is characterized by one or more parameters, which are grouped into different classes, such as "nominal," "ordinal," "interval," and "ratio," as shown in Table 1.

[0028] [Table 1]

[0029] For simplification, parameters can be further divided into "nominal" and "non-nominal" classes.

[0030] Autotuning optimizes parameters according to some determined criteria. The nature of these parameters is completely specific to a particular application. So-called black-box autotuners are unaware of the semantics of the parameters. More importantly, parameters actually have different domains. For example, some parameters may be integers, others may be floating-point, or may not be numbers at all. Some parameters assume legal values ​​from the interval [0;10], while others assume legal values ​​from the interval [0:0;1:0]. The need to handle different domains makes search and model implementation particularly cumbersome.

[0031] Therefore, in the present invention, the implementation is decoupled from these application-specific parameter characteristics. This decoupling is provided by the ParameterSpace unit 34, which is an adapter between the application and its parameters on the one hand, and the search and prediction algorithms on the other hand. This transformation allows the search and prediction algorithms to operate on an infinite space of n-dimensional real values, called the search domain, where n is the number of parameters. As the application reports parameter configuration feedback to the adaptation layer, the state of the hybrid tuning is updated and the next configuration sample is queried.

[0032] The samples are real-valued vectors in the search domain. The configurations are cached in the ParameterSpace unit. When the application requests the next parameter update from the hybrid tuner component, the cached configuration samples are transformed to the application domain and then passed to the application. This mechanism has two beneficial effects. First, it allows the development of search algorithms that are independent of the application they are used in. Second, it prevents several numerical problems in the search algorithm. Search algorithms that attempt to optimize in integer space can fall into suboptimal configurations. The search never converges, flip-flopping between a small set of configurations due to rounding. Through the ParameterSpace unit 34, responsibility for, for example, rounding and bounding, is transferred from the search implementation to the application domain.

[0033] When registering parameters in the present invention, applications can define individual domain and bound mapping behavior as needed. Furthermore, the present invention provides several practical default adapters for the most frequently used parameter types. These are integer and floating-point parameters, either unconstrained or from an interval, as well as Bag-of-Labels nominal parameters. The latter are parameters that can take any value from an unordered set of labels. Through the default adapters, these are embedded as interval-constrained integer parameters, effectively numbering the unordered labels. Integer parameters are converted from the real-valued search domain by rounding by default. To restrict legal configurations to an interval, the adapter effectively mirrors the measurement function at the interval bounds. The configuration values ​​returned by the search are then mapped to the interval with respect to the mirroring. Figure 3 shows an example of the procedure for a parameter that accepts integer values ​​from the range [1, 6]. This diagram displays the search space for a single parameter in a format recognized by the search algorithm.

[0034] Although the parameter can only accept values ​​between 1 and 6, the search is free to sample any value. For example, sampling a configuration value of C=1.3 will, after rounding, set the application parameter to a value of 1. Sampling a value of C=10.7 will first apply mirroring, then round, and set the application parameter to a value of 3. Negative values ​​are handled similarly, but are not shown in this example for simplicity.

[0035] Such a mirroring strategy has several interesting properties. Most importantly, all local and global optima are preserved. Newly introduced points do not exceed the global optima, and all previous local optima remain local. Any local extrema on the boundary of the interval remain local, but their neighborhoods are symmetric. A mirroring strategy cannot be expected to be effective for all possible search algorithms. For example, searches that make assumptions based on the distance of points in the search space may produce false alarms. A point may appear far away to the search, even if its measurements are obtained from a closely mapped point in the application domain. In this case, application developers are expected to provide appropriate adaptation mechanisms along with parameter definitions.

[0036] The IndicatorSpace unit 36 ​​is a register that stores system indicators.

[0037] Compared to tuning parameters, managing application and system indicators is not very complex. The IndicatorSpace unit 36 ​​is a registry containing a set of indicators. Besides registration, this component also implements change detection. What constitutes a change can be overridden by applications that define their own indicators. By default, a change occurs whenever the numeric value of an indicator changes. Since applications are forced to represent indicators numerically, this is sufficient to realize hybrid tuning. No domain transformations are required, as in the ParameterSpace unit.

[0038] In the present invention, application and system metrics are provided by the application developer. When used within the framework, the framework's compiler takes on the role of the application developer and automatically defines the metrics.

[0039] Developer-provided metrics can be as simple as any application-side variable. Thus, while it is easy to use program or function inputs as metrics, arbitrarily complex metrics are possible. For example, a metric can be defined to monitor system load, but this would require querying an operating system statistic each time the metric state is queried. Also, depending on the nature of the metric, downsampling may be required to observe its value.

[0040] To this end, application developers can add a target resolution. This is useful, for example, as a metric to monitor system load. A load of 99% or 100% is unlikely to make a discernible difference in tuning kernel performance, but the difference between 10% and 100% is likely to be noticeable. The invention does not assume any particular metric semantics, so it is up to the application developer to define a meaningful sampling resolution.

[0041] Two search algorithms for use in hybrid tuning unit 30 are now described.

[0042] Because of the separation and abstraction provided by the ParameterSpace unit 34, a search algorithm implementation is required to provide only a minimal interface. Figure 4 shows the base classes. In particular, the search algorithm is independent of the actual sampling. Instead, it returns a sequence of configurations to the caller's ParameterSpace unit. It is the caller's responsibility to obtain measurements and return them to the search. In the following, we use the term "sampling" to mean a single round trip: returning a configuration to the caller and receiving measurements in the next iteration. The required operations that a search algorithm implementation must provide are start and getNext, which (re)start the search and obtain the next configuration, respectively. When obtaining the next configuration, the caller also provides the measurement value of the last configuration as an argument. A search implementation is expected to update its internal state upon a call to getNext and return the next configuration it wishes to sample. Furthermore, an implementation can support rejecting individual configurations without performing measurements. Callers can use this mechanism to filter out invalid configurations or ignore configurations they do not wish to sample. This is useful when a performance model is used to predict that a configuration is not optimal.

[0043] When tuning an application, developers can provide their own implementation of the search algorithm or select a default from a catalog of a small number of implementations. The library distinguishes between algorithms with nominal and non-nominal parameters. Unless the application developer requests otherwise, a novel hierarchical search algorithm for searches in mixed spaces is used. This section describes the two default basic search techniques used by the hierarchical search. The hierarchical search technique itself is described later.

[0044] The tuning library includes implementations of the basic search algorithms Nelder-Mead Downhill Simplex, the ε-Greedy algorithm, as well as random and exhaustive search. The latter two are trivial, but variants of Nelder-Mead and ε-Greedy are described below.

[0045] In this embodiment of the present invention, the basic search strategy for non-nominal parameters is an implementation of the Nelder-Mead algorithm described in J.A. Nelder and R. Mead, "A Simplex Method for Function Minimization," The Computer Journal 7.4 (Jan. 1965) (cit. on pp. 15, 91). The algorithm is a heuristic and derivative-free numerical optimization technique. The algorithm is implemented as a compact Mealy automaton, a finite state machine shown in FIG. 5. In this implementation, a simplex is maintained as an array of search space points χ and their measurements mx,x∈χ. The points in χ are sorted in ascending order with respect to mx. The state machine changes its state at each tuning iteration.

[0046] The state transitions are defined as triples M / C;U, where C is the next configuration and M is the measurement feedback mC' for the previous configuration C', optionally including constraints on the measurements. Finally, U defines an optional update to the simplex χ that is applied before the next configuration is returned. The state machine is defined by five states with a starting state S. The transitions implement the Nelder-Mead update rule.

[0047] From the start state, the automaton transitions to the initialization state I by initializing a J+1-dimensional simplex from Latin hypercube samples. The first simplex point is returned to the application as the next configuration. Latin hypercube is a technique for obtaining space-spanning configurations, often applied in seed search algorithms. To obtain a simplex based on this technique, all dimensions of a bounded J-dimensional space are split into J+1 equal-sized intervals. Samples are then drawn randomly such that no two samples fall in the same interval in any given dimension. When the application returns measurement feedback for the last simplex point, the automaton transitions to the reflection state R and performs the reflection operation by returning the reflection point r'. The transition to the successor of the reflection state is performed by returning the measurement feedback m' for the last reflection point r'. r Depends on m' rIf m'r is less than the measurement value of the best simplex point mχ[0], the automaton performs an expansion operation, returns the expansion point e, and transitions to the expansion state E. If m'r is between the best and worst simplex points, the worst simplex point χ[J] is substituted for the previous reflection point r' and another reflection is performed to generate the next reflection point r. Otherwise, if m'r is greater than or equal to the measurement value of the worst point, a contraction operation is performed and the contraction point c is returned. After receiving feedback me for the expansion point e, the expansion state E transitions back to the reflection state R. This transition replaces the worst simplex point with the better of e and r' and performs another reflection operation to generate the next r. Similarly, the automaton performs a reflection operation in the contraction state C if the feedback mc for the contraction point c is better than the value of χ[J]. In that case, χ[J] is substituted for c. Otherwise, if mc is greater than or equal to mχ[J], a contraction operation is performed. This operation forms a new simplex by moving all points toward χ[0]. Then, all simplex points except χ[0] need to be sampled, starting from χ[1], and the automaton enters the initialization state I. The reflection, contraction, and expansion points r, c, and e are computed using the default parameter values ​​α=1; β=0:5; γ=2 proposed in the original Nelder-Mead paper. Our implementation of the Nelder-Mead algorithm does not support rejection of individual configurations.

[0048] The second search algorithm is the so-called ε-greedy algorithm.

[0049] For nominal parameters, in this embodiment of the invention, an ε-greedy algorithm was implemented, which achieved the best performance in exploratory case studies. This algorithm tracks the best known configuration. At each step, it either samples a random configuration with probability ε or uses the best known configuration with probability 1-ε. Another additional hyperparameter controls the decay of ε over time. This parameter is gammaε ∈(0,1) The probability of choosing a random configuration at step i is ε i =ε.(1-γ ε )i Therefore, γ ε provides direct control over the convergence rate. Setting this hyperparameter to 0 reverts to the original ε-greedy behavior. If the application rejects a configuration, the algorithm returns a new random sample.

[0050] In real-world programs, parameters are often not only integers, but also often nominal values. Applying a search algorithm to optimize a mixture of two classes is highly problematic because it limits the selection of appropriate search algorithms. While the present invention does not actively prevent application developers from selecting an inappropriate search algorithm for a parameter mix, this is strongly discouraged. A search algorithm that draws conclusions based on, for example, the neighborhood of configurations will derive information purely by chance. As an illustration, consider tuning the algorithm selection for a matrix multiplication algorithm.

[0051] A hill-climbing algorithm can be applied when the nominal tuning parameters A={ijk;ikj;strassen} are embedded in the integer space by simply enumerating the algorithm.

[0052] If a climber now observes that m(ijk) = 2 and m(ikj) = 1, the climber concludes that the space looks promising in the direction of ikj. This conclusion is already a coincidence, regardless of whether Strassen is actually better than ikj. The order of the algorithm is irrelevant. As we have found in our exploratory case studies, the majority of empirical search algorithms in use today face the problem of operating with this notion of direction or distance. A notable exception is genetic algorithms, which perform well for combinatorial optimization problems when using appropriate crossover and mutation operators.

[0053] In this invention, we approach such mixed space search in a structured way that is automatic and transparent to the application developer. As an example, consider a simplified version of the heterogeneous parallelization approach: we generate parametric parallel code for CPU and GPU, and add tuning parameters for platform selection. In this example, we consider a total of five application parameters: P (binary platform selection), T CPU (Number of CPU threads), T GPU (number of GPU threads), SM (flag for enabling shared memory), and U SMWe tune α (a flag for unrolling the memory copy loop that moves data to shared memory). Table 2 summarizes these parameters along with their ranges. Conventional state-of-the-art tuning solutions embed these parameters in a single five-dimensional space containing 2 × 48 × 32 × 2 × 2 = 12,228 valid points. Our first improvement is to separate this single space into two smaller spaces by distinguishing between nominal and non-nominal parameters. Figure 6 graphically illustrates this effect, with the left space containing only nominal parameters and the right space containing only non-nominal parameters. Note that this separation does not actually reduce the dimensionality. There are 2 × 2 × 2 = 8 points in the nominal space and 32 × 48 = 1,536 points in the non-nominal space. However, finding configurations independently in each space is not reasonable because the parameters are not independent. For example, when shared memory is disabled (C SM == false), the kernel runtime on the GPU is m GPU =0:5 T GPU otherwise, m GPU =1 / T GPU Let us assume that T is defined by GPU Which value of is optimal obviously depends on the chosen value of SM. Therefore, the non-nominal parameter T GPU Ignoring the selected value of the nominal parameter SM when selecting the value of is not a viable path.

[0054] Instead, we need to sample the configuration from the cross product of both spaces, which means that the cardinality of the separated spaces remains unchanged. Nevertheless, this first decomposition allows us to explore the spaces using different search algorithms suited to each parameter class, for example, the ε-greedy algorithm on the one hand and the Nelder-Mead algorithm on the other. To preserve the dependencies between parameters, we organize the parameters hierarchically.

[0055] First, we use the ε-greedy algorithm to determine the configuration of the nominal parameters, and then we use the Nelder-Mead algorithm to determine the configuration of the non-nominal parameters. Because the configurations cannot be chosen independently, we maintain one instance of the Nelder-Mead search for each nominal configuration. In this example, the memory consumption of our Nelder-Mead implementation increases by a factor of 8. The Nelder-Mead instance runs in O(n 2 ) space, where n is the dimension of the search space.

[0056] However, the actual increase in memory footprint is only about 30%, or (8x2 2 ) / 5 2 2x. The space consumption of the Nelder-Mead algorithm increases by a factor of 8, but the number of parameters is reduced from 5 to 2. While a 30% space increase seems small, the penalty is exponential in the dimension of the nominal search space, so it can actually have a dramatic impact. On the other hand, this cost is paid in that the search space of each individual search algorithm becomes simpler, allowing appropriate algorithms to target individual parameter classes. Furthermore, the instantiation of search instances occurs lazily. Therefore, the worst-case memory consumption overhead occurs only when sampling all nominal configurations, and this only occurs in practice for small nominal spaces.

[0057] Decomposing the search space into nominal and non-nominal subspaces opens up additional optimization avenues. In practice, nominal parameters often represent algorithm choices, resulting in different paths of control flow in the application. In this example, parameter P controls whether the CPU or GPU implementation of the parallel code should be run. This parameter therefore interactively influences parameter T. CPU and T GPUThese constraints control the relevance of the GPU threads. Configuring the GPU thread count, for example, has no effect when running the CPU version of the code. Using the application developer's contextual knowledge of the semantics of the application parameters, such constraints can be exploited. In the above example, two further constraints can be identified: the number of GPU threads is relevant to the GPU selection, and unrolling the loop that moves data to shared memory is only relevant when shared memory is enabled. Adhering to these constraints produces the search space decomposition shown in Figure 7. Unlike the first level decomposition, this second step reduces the cardinality of the space. In this example, we only have a one-dimensional search space, achieving the highest possible dimensionality reduction. The constraints also reduce the size of the sample space, since irrelevant configurations are filtered out. The total number of points that can be sampled is now 48 + 32 + 2 × 32 = 148, or C P There are 48 configurations where CP==GPU, CSM==false, 32 configurations where CP==GPU, CSM==true, and 2.32 configurations where CP==GPU, CSM==true.

[0058] The caveat of the second level decomposition is that, as above, we must preserve parameter dependencies, which means that we must maintain separate search states for all configurations of the spaces involved, for both nominal and non-nominal spaces.

[0059] In summary, even for such a small example, we were able to reduce the size of the search space by two orders of magnitude, while allowing the tuner to explore portions of the search space using different appropriate search mechanisms. This size reduction is achieved by classifying the parameters into their respective classes and decomposing the search space based on relevance constraints. In the following, we formulate an algorithm that implements the decomposition and search in the decomposed space. The general idea is to compute a decomposition into the minimum number of subspaces such that the relevance and class constraints are preserved. The first level of decomposition is straightforward: the search space T = T N ∪TR , the nominal parameter T N and the non-nominal parameter T R It divides the space into naturally separated subspaces.

[0060] Relevance Constraints. The second level of decomposition is computed based on relevance constraints defined by the application developer.

number

number

[0061] Local search space. When all elements of a subspace are subject to the same relevance constraints, T N and T R We call both subspaces of the local search space. To be precise, the subspace £⊆T is the subspace £⊆T. N or £⊆T R and

number

[0062] Search space graph. The local search space and the relevance constraints together form a directed acyclic graph, which we call the search space graph S = (£ S ,E S ) The nodes of the graph are called Sis the local search space induced by the relevance constraints, and the edges E of the graph S Forming

number

[0063] Before proceeding with the introduction of the hierarchical search procedure, we first provide another, more complete example of a search space graph decomposition. Consider a parameter set T = {A, B, C, D, X, Y, Z, W}, where A through D are nominal values ​​and X through Z are non-nominal values. The first level of decomposition is T N ={A,B,C,D}, T R ={W,X,Y,Z}. Furthermore, the dependency constraints are

number

[0064] Local configuration, partial configuration, and maximal partial configuration. Configuring tuning parameters based on a graph means determining local configurations for all nodes that satisfy the constraints. A local configuration of a local search space l is a configuration C of parameters within that local search space. l A set of local configurations of different local spaces is defined as a partial configuration C of the search space graph S that does not contain any contradictory local configurations. S If there are two paths containing l and l' that contain conflicting constraints, then there is a local configuration C l , Cl A partial configuration is called a maximal partial configuration if there is no possible local configuration that can be added to it without contradicting any local configuration in the set.

[0065] As an illustration of a contradictory composition, consider the local space containing B and C in the example above. The two spaces cannot be part of a partial composition, since this requires that A be simultaneously composed of values ​​b and c.

[0066] Active node. Partial configuration C S Given l R All edges on all paths p from to l

number

number

[0067] Consider Figure 8 for a step-by-step example of the activation process. Initially, all search states are empty. For nodes {A,D}, the ε-greedy search selects a configuration {b,z} that activates nodes {B} and {Z} according to the constraints. The nominal node is processed first, which causes the ε-greedy instance of the current partial configuration {b,z} to select configuration {x'} for node {B}. This results in the partial configuration {b,z,x'}, which is used to finally activate the Nelder-Mead instance of node {Z}, generating the maximal partial configuration. Suppose the application returns a measure of 0.7 for this configuration, then the search state after this first tuning iteration is [Table 2] This becomes:

[0068] In the next tuning iteration, suppose the search instance of the {A,D} node returns the configuration {c;z'}. This partial configuration activates only node {C}. Assume that the search instance of this node for this configuration returns {y'}, and the application measures a value of 0.9 for the resulting maximum partial configuration. With the updated values ​​highlighted, the search state after the second tuning iteration is: [Table 3] This becomes:

[0069] In the third tuning iteration, assume that the {A,D} instance returns {c,z} and activates {C}. Let the ε-greedy instance for {c,z} return {y}. Then, Figure 9 shows the current activation state of this partial configuration. By applying the two steps of the ε-greedy algorithm, the hierarchical search finds the partial configuration C. A =c,C C =y,C D=z, activating the local spaces {A,D}, {C}, {W}, {Y}, and {Z} highlighted in the figure. Because there are no constraints, {W} is activated simultaneously with {A,D}, but because it is a nonnominal subspace, it is activated only when the partial configuration is maximal with respect to the nominal parameters. For a given partial configuration, the highlighted nonnominal spaces {W}, {Y}, and {Z} are jointly activated for a single Nelder-Mead instance. Assuming the application reports a measure of 0.6 for the resulting maximal partial configuration, the search state after the third iteration is [Table 4] This becomes:

[0070] Note that at this point, there are two concurrent search instances for {C}. Decomposing the search space in this way not only helps effectively reduce the size of the space, but also allows for more efficient rejection of configurations. If an application heuristically determines that a configuration will produce poor performance, it will reject it. However, often, a combination of only a few parameters contributes to this fact. Consider the heterogeneous offloading example (Figure 7). If the parallelized code is trivial or runs for only a few iterations, the application can legitimately determine that offloading this computation to the GPU is likely to actually degrade performance due to overhead. The only parameter relevant here is P, which controls platform selection; none of the rest contribute to the offload decision. The search space graph also allows the application to reject partial configurations. For example, an application may reject a partial configuration setting C. P = Once a GPU is constructed, it can be rejected immediately. This allows the rejection decision to be made earlier in the configuration process. Rejecting a configuration earlier further reduces the number of previously active nodes.

[0071] We explored two types of models for the prediction component of hybrid tuning. The first model is a tabular nearest-neighbor approach. The second model uses generalized reinforcement learning and uses function approximation to model the relationship between index, configuration, and runtime. Tabular predictors are simple: all observed states are recorded in a table, along with the best-known configuration for that state. When the tuner enters a new index state, the geometrically closest known state is queried from the table. While nearest-neighbor search can be implemented efficiently, this approach has two major drawbacks. The motivation for nearest-neighbor prediction is, of course, that the same configuration is likely to be good for similar states. However, if the new state is far from all states in the table, the query results can be arbitrarily bad. This problem can be mitigated by not using the query results if the distance is too large, but this requires defining a distance threshold. The second drawback is the table size: storing information for all possible states is infeasible, especially if the states are not discrete. Of course, the table could be filled solely by search, or its size could be limited. However, this would eliminate the possibility of learning during exploitation or add another hyperparameter to limit the size.The present invention provides application developers with control over the size limit and metric resolution.

[0072] Generalized reinforcement learning (RL) offers a way to avoid the drawbacks of tabular prediction. While classical reinforcement learning (RL) techniques are themselves table-based, generalized methods approximate the table through a function. By approximating the relationship between metrics, configurations, and runtime with a function, RL can eliminate the need to memorize all states or select which states to memorize. To predict a good configuration for a state, the function is used to find a configuration that maximizes the estimated reward for the state-configuration pair. Unfortunately, maximizing the function is costly, so hybrid tuning may not be applicable to online tuning scenarios. Therefore, we keep a record of the best configurations found during the search and consider only these as prediction candidates. If hybrid tuning searches frequently enough, this may still be large, but it is generally much smaller than the full state table.

[0073] We implement a version of the greedy-GQ variant of Q-learning as a function approximation predictor. We approximate the table as a function Qθ(s,a)=θ·φ(s,a), where φ is a metric feature defined by the application developer. By default, we use normalized radial basis functions as features. Φ c (x)=exp(-ω c (x-μ c ) 2 ) / Σ i exp(-ω i (-μ i ) 2 ) where x = (s, a) is the concatenation of the index state vector s and the configuration (or action) a. c (x)=exp(-ω c (x-μ c ) 2Compared to regular radial basis functions, which are ∑μc, μc, and ωc, the regularized variant promises smoother spatial boundaries and smoother "gaps" between basis functions. The center μc and width ωc are configurable parameters. The standard update equation for the greedy-GQ algorithm is controlled by three hyperparameters α, β, and γ. α and β are both learning rates. The third parameter γ determines the influence of expected future rewards on the update. However, in this application, we assume that the influence is always zero. Considering future rewards allows us to reflect information about future states in the update. However, we assume here that there is no correlation between the selected action and subsequent state changes. This assumption is, of course, only valid for applications where the context state does not depend on the tuning parameters. Setting γ to 0 removes β and the update equation becomes θ t+1 =θ t +α(R t+1 -θ t Φ(s t ,a t ))Φ(s t ,a t ) The autotuning system implements a hybrid online autotuning approach as a black-box tuner. Hybrid tuning combines online search and model-based prediction to provide three main features:

[0074] Context Sensitivity: The runtime of a tuning kernel depends not only on the configuration of tuning parameters but also on the dynamic tuning context. In this invention, a tabular memory records the configuration and tuning context state. When a previously seen state recurs, the previous configuration of that state is used. In addition to the tabular memory, a predictive model is built to predict good configurations for unknown context states.

[0075] Effective Search: To explore the search space and bootstrap tables and predictive models, the Tuner uses online search. The set of implemented default search algorithms can be extended by application developers.

[0076] Efficient Search: Known online search techniques find (locally) optimal configurations relatively quickly, but there is room for improvement. The search algorithm described herein, called hierarchical search, automatically exploits structure in the search space and avoids known bad configurations, thereby substantially accelerating search time.

[0077] Current autotuning designs fall into two categories: optimization is generally either machine learning or model-based, or search-based. Each of the two alternatives has different advantages and disadvantages. When parameter configurations are generated by a machine learning or model-based predictor, it is possible to react immediately to dynamic context changes. The predictor is queried for an updated context state and returns a single parameter configuration. In contrast, in search-based tuning, every change of context incurs overhead. The search is restarted to find a new configuration, and new configurations are sampled until a new (local) optimum is found. Note that a disadvantage of using prediction is that the quality of the configuration depends on the accuracy of the predictor.

[0078] If a predictor is handcrafted, its accuracy is determined by how accurately the designer models the application and dynamic context. If a predictor is trained, its accuracy depends on how well the trained model generalizes from the training data and how well the data reflects reality. Machine learning eliminates the need for meticulous human model design, but typically requires a large number of training samples to achieve high accuracy. In offline training, the quality of the trained predictor depends on how well the training data represents the data observed in the production environment. On the other hand, online training can learn from actual production data, but it suffers from the problem of bootstrapping. If the initial model is inaccurate, initial predictions will be suboptimal, resulting in poor performance. Search-based tuning does not have the problem of bootstrapping, and the quality of the discovered configurations does not depend on prior sample data.

[0079] Hybrid tuning offers a compromise between these two approaches. It combines prediction and search, exploring the search space using online search and continuously providing training data to update the predictive model. Because training is done online, predictions are not affected by prior sample bias. The hybrid tuning workflow is shown in Figure 9. Whenever a dynamic context change is detected, hybrid tuning can choose between exploring the search space using search or leveraging the trained predictive model. Any performance feedback provided by the application in either mode is used to refine the predictive model. The configuration table additionally stores the best-known configuration for the observed context state.

[0080] To implement hybrid tuning, we model the changing context and the reaction to the changes as a Markov decision process (MDP). This model represents context changes as process state transitions and uses the MDP's decision-making to choose between exploration and exploitation. In the following sections, we briefly introduce our process model. We then discuss how to implement the detection of context changes and the technique for balancing exploration and exploitation.

[0081] To represent the context sensitivity problem in autotuning systems, we construct an infinite horizon MDP, which is a quadruple (Σ, A, Φ, ρ).

[0082] In the model, Σ is the state space and A is the space of possible actions. The state space represents the dynamic context in which the tuner operates. In MDP literature, the tuner is referred to as an "agent". Thus, changes in the context constitute changes in the state space, causing the agent to choose an action. We define the action space A=T as the set of possible configurations.

[0083] An agent selects a configuration according to a policy, usually referred to as π(s),s∈Σ, which determines the behavior for every state. A behavioral model Φ:Σ×A×Σ→[0,1] describes the probability of transitioning to state s' when in state s and executing action a: Φ(s,a,s')=p(s'|s,a). Since it is unknown whether the selected action has any actual effect on the state transition, it is not possible to define an "on-size-fits-all" behavioral model. Reconsidering the library parallelization example mentioned above, actions affect the runtime of the called library function, but cannot reasonably be assumed to have any effect on the context state reported to the library by the client. Therefore, the behavioral model must be learned by observing state transitions.

[0084] Finally, the reward function ρ:Σ×A×Σ→[0,1] determines the reward r=ρ(s,a,s') for transitioning from state s to s' under action a. We set r=-m. That is, the immediate reward achieved after every action choice is simply a performance measure. The negative sign serves to transform the minimization goal of autotuning into a maximization goal of the agent.

[0085] Before the autotuner can dynamically adapt to context changes, it must first detect them. To accomplish this, the context must first be modeled and quantified. Therefore, the instantaneous state of both the application and the system must be determined. For this reason, context metrics are used.

number

number

number

number

[0086] Indicators in hybrid tuning can represent both discrete and continuous quantities. This is necessary to support general applications, but it has strong implications for MDP models. Because the context shapes the state space of the process model, incorporating continuous indicators results in a continuous Markov decision process. Such processes have significantly different solution and convergence guarantees.

[0087] The distinction between discrete and continuous indicators also impacts change detection. In a discrete context, detecting a change simply means monitoring the change in the indicator's value. The same is certainly possible for continuous indicators, but we define two additional methods here. Continuous quantities often exhibit high frequency. Because the system on which an application runs never actually pauses, measuring a system load indicator will essentially produce a different value each time. Here, it is useful to smooth the response function with some kind of low-pass filter, for example, by averaging multiple indicator samples over time. While indicator value changes can be frequent, they are often also low-amplitude. When a system is busy, for example, the system load typically stays within the 100% range. Furthermore, it is easy to see that small changes in indicator value have negligible effects on performance and thus on the optimality of the current configuration. For example, if the system load changes from 98% to 99%, the impact is unlikely to be measurable. Both smoothing and downsampling are effective in this regard. Alternatively, it is possible to detect state changes purely by observing the effect of the indicator. A measurement function can be monitored to detect state changes when the function's value changes suddenly.

[0088] With the context quantified and the device capable of detecting change, the next step is to react to this change. Within the framework of our model, reacting to change is implemented by policy π. This policy is borrowed from the field of reinforcement learning and is loosely based on the ε-greedy policy commonly used in this field. When a state change is encountered, the search space is explored using a hierarchical search with probability ε. With probability 1-ε, we leverage a predictor trained during the search from the sampled configuration.

[0089] Because the interaction between the application and the tuner is iterative, both options are not "atomic" in the sense of being a single reaction to a state change. Instead, a hybrid tuner can be in either exploration mode or exploitation mode. In the former, it generates configurations as dictated by the hierarchical search. Being in exploitation mode means that the configuration is updated only once, when entering that mode. In exploration mode, on the other hand, multiple configurations are sampled. As a result, it is possible that another state change occurs while the search is still in progress. In that case, the current search state is saved, and the search continues when the previous state is repeated.

[0090] Having described the auto-tuning system 20, an example implementation of the compiler 10 will now be described. The particular form of the implemented compiler is not limited with respect to the described embodiment, which is provided for illustrative purposes only. In principle, the task of parallelizing a program for execution on a heterogeneous system can be done manually and then compiled, or the compiler can perform the parallelization. The number of tasks that are parallel is itself a tuning parameter. If a program is written in a form that includes parallel tasks, no further parallelization is required. Automatic parallelization is preferred in the present invention, as it can beneficially utilize an auto-tuning system that identifies and makes available for tuning parameters such as the number of threads and whether or not to use shared memory.

[0091] In this example, the compiler parallelizes loops in the input program for multiple platforms and devices and interacts with the autotuner through a runtime library to analyze and transform the program. Two techniques are implemented to analyze and transform the program. First, the compiler attempts to analyze the program using a polyhedral model. If this fails due to constraints that the polyhedral model imposes on representable programs, it reverts to classical dependency testing and more ad-hoc transformations.

[0092] Below, we describe polyhedral model-based parallelization, followed by ad-hoc approaches, and then we explain how compilers and autotuning systems can interact to incorporate analytical information that the compiler may derive into tuning decisions.

[0093] Generating code for cooperative heterogeneous execution from a polyhedral model is a three-step process. The process begins by computing a schedule for the SCoP that maximizes data parallelism. The next step is to partition the outermost parallelizable loop for the target platform, here CPU and GPU cores, and then map the partition onto the platform. Finally, the optimized and mapped schedule is translated into actual parallel code for the platform. Our approach to partitioning and mapping and code generation, and finally, our interaction with the autotuner, are described below.

[0094] Our strategy for heterogeneous parallelism using the polyhedral model is based on Polly-ACC and therefore makes extensive use of the PPCG GPU mapper described in Sven Verdoolaege et al., "Polyhedral Parallel Code Generation for CUDA," ACM Transactions on Architecture and Code Optimization (TACO)-Special Issue on High-Performance Embedded Architectures and Compilers 9.4 (Jan. 2013), 54:1-54:23. We use it to first compute a schedule tree optimized for data parallelism from the SCoP. Before mapping the optimized schedule to the GPU using the PPCG code generator, we transform the schedule by introducing a first tunable partition and a second tunable platform mapping.

[0095] Essentially, splitting a loop involves duplicating the loop and then modifying its bounds, with one copy processing the lower portion of the iterations and the other processing the upper portion. To enable this tuning, tuning parameters can be used to separate the lower and upper iterations. Mapping a partition to a platform is essentially the same operation: creating one instance of the partition per platform and adding tuning parameters to dynamically select the instance. Note that these two steps only parallelize the outermost loop. This process is repeated recursively to process all data-parallel loops. Below, we describe in more detail how this process is realized within the Polyhedron framework.

[0096] The schedule tree is transformed recursively from the top down to find the outermost data-parallel loop in the loop nest. To insert a partition, we start by finding the outermost band with a leading parallel dimension. The set of leading parallel dimensions corresponds to the set of outermost data-parallel loops. Every loop in this set is split into two parts, one for high indexes and one for low indexes. The number 2 seems self-suggestive because we are targeting two platforms, but this approach allows for any number of splits.

[0097] We split the outermost parallel dimension of a band by introducing a set node between the band and its parent with two filter children that implement the constraints for the split. A newly introduced parameter R0 specifies the split of the outermost parallel dimension of the band. R0 is subject to the same constraints as the iterator for that dimension. In a split filter, this parameter becomes the new upper and lower bounds of that iterator for selecting high and low indices, respectively. The original band node is duplicated and appended to the filter.

[0098] The process is repeated recursively for the remaining preceding parallel dimensions of the band, i.e., if we divide d dimensions, we get a total of 2 d - Insert one set node and add two copies of the original band d The implementation of platform selection is done in exactly the same way: for every band node we create, we insert a new sequence node as its direct parent between the band node and the aggregation filter we created during the split. A new parameter P for platform selection for the i-th loop is i Low and P i High and for both high and low index partitions, P i r ∈{0,1}. The two filters of the new sequence node are subject to the constraint P i r =0 and the constraint P i r = 1. By duplicating the band nodes again, we finally implement the PPCG mapping algorithm as P i r By applying it to the band, it is parallelized for GPU. However, a mark node is inserted in the sibling node, which is used as the starting point for OpenMP code generation for CPU parallelization during code generation.

[0099] Figure 10 illustrates the process for the example schedule tree in Figure 11. Splitting the first dimension at the outermost band node of the tree produces the tree shown in Figure 10a. This shows two newly inserted filters using the splitting parameter R0 as the lower and upper bounds of the iterator parameters, respectively. Figure 10b shows the platform mapping for one of the two partitions. Introducing the new parameter P0, the inserted filters generate GPU and CPU paths. The PPCG mapping algorithm is then invoked for the GPU path.

[0100] After all schedule transformations have been applied, we convert the schedule tree into LLVM IR. This procedure is built on top of the Polly-ACC code generator, so most of the implementation of generating code for partitioning and mapping is already done for us.

[0101] The Polly-ACC code generator, which was designed to handle GPU offloading only, has been extended in two major ways. To incorporate CPU parallelism, the code generator has been integrated with the existing experimental Polly OpenMP backend. This backend emits code for the libgomp OpenMP runtime library. The second extension is instrumentation of parallel programs to interact with the autotuner. During mapping and code generation, multiple tunable parameters are extracted from the parallel program. First, and most importantly, partition splitting R i ∈[0;1], and the platform mapping P i r∈{CPU,GPU}(i∈[0;d] and r∈{High,Low}), where d is the number of outer loops being parallelized. For example, when parallelizing two outer loops of a parallel source region, this results in six tunable parameters: two partition splits and four platform mapping determiners for the partitions. As a result, at runtime, multiple partitions may be offloaded to the same platform. This can cause extra overhead, but is unfortunately unavoidable because the non-affine constraints that avoid this cannot be expressed in the polyhedral framework.

[0102] In addition to these work distribution parameters, additional parameters can be derived from the mapping and code generation process: for CPU code generation, a parameter controlling the number of threads that is freely configurable at runtime

number

number

[0103] [Table 5]

[0104] The polyhedral model may not be able to represent parallelizable loops. However, by relaxing its constraints, it may be possible to analyze the loop and detect data parallelism. Two examples of such cases are function calls and inner loops without static control. Function calls cannot be modeled with the polyhedral model because instructions within the function cannot be scheduled.

[0105] Polyhedron compilers typically first inline functions and eliminate function calls. However, this requires speculative inlining of all function calls just to detect parallelism, which is very expensive. On the other hand, loops without static control cannot be expressed at all in the Polyhedron framework. However, when parallelization is the only goal, such loops can be benign. If such loops do not access memory or only access provably local memory, they have no impact on parallelization. Therefore, if Polyhedron modeling fails, we fall back to a second analysis and transformation pipeline using classical dependency testing and ad-hoc transformations.

[0106] This section introduces pipelining, the analysis performed by compilers to detect parallelism, and platform-specific program transformations.

[0107] The first step in parallelizing a given program is to find the parallelizable portions of the program. In this phase, we focus on finding only data-parallel loops. This scope can be extended to support loops with reductions, which can be viewed as a special case of data parallelism with relaxed constraints on data dependencies. In addition to the data-parallel constraints, we impose further constraints on the analyzed loops to simplify the implementation.

[0108] Most importantly, only top-level loops are considered as candidates. Second, we assume that loops have a specific structure in the control flow graph. The expected structure is shown in Figure 12. Loops must be defined by a single-entry, single-exit control flow subgraph. This means that there is a single basic block outside the loop whose only successor is the loop header block, and vice versa for the exit block. Furthermore, there must be a unique exit block that branches to the exit block and header block. Therefore, the existing block must not be a loop header block. Note that the structural constraints are generally weak, since most loops can be normalized to this form. However, a notable counterexample is a loop with multiple exit blocks—basic blocks outside the loop that the loop body can jump to. There is no universal technique for merging these blocks. Finally, the outermost loop must exhibit static control. The trip count, i.e., the number of iterations, must be statically computable. More precisely, this trip count can be a constant or a provably loop-invariant variable. For all loops that pass these checks, the compiler performs parallelism detection analysis. This is a classic dependency analysis, which attempts to disprove loop-carried data dependencies. The LLVM framework includes a variety of such analyses. The compiler extends the existing capabilities of the LLVM framework to support interprocedural analysis, which performs dependency analysis on loops containing function calls. It summarizes all functions called within a candidate loop to explain the effect of the calls. A function's summary is effectively an exhaustive list of all memory accesses within the function that affect its pointer arguments or global variables. These accesses are those potentially observable by the caller. In general, exact alias analysis is not possible in practice, so summarizing the ranges accessed requires over-approximations. When analyzing memory accesses, it is often not known exactly which arguments or variables are read or written.In the worst case, summarization must assume that a function reads and writes every byte of memory in the program. This can happen, for example, if the function is accessed indirectly through a pointer read from an argument or a global variable. With summarization, function calls can be analyzed by extending the summary at the call site, which virtually "inlines" the function. To disprove the existence of any loop-carried dependencies, the compiler supports summarization by extending LLVM's implementation of the dependency testing algorithm by Goff, Kennedy, and Tseng (Gina Goff, Ken Kennedy, and Chau-Wen Tseng. "Practical Dependence Testing", Proceedings of the ACM SIGPLAN 1991 Conference on Programming Language Design and Implementation. PLDI'91. New York, NY, USA: ACM, 1991, pp. 15-29).

[0109] To broaden its scope, the compiler has been extended to support parallel reduction, a parallelization technique that actually exhibits loop-carried dependencies. Parallel reduction differs from simple data parallelism in that it allows specific data dependencies to exist and allows reads and writes to conflicting memory locations to depend only by associative operations. A typical example of this is the sum of all elements of an array to an output variable: the output variable is read on each iteration, incremented by an array element, and written back to the same location. If the increment operator is associative, then this operation can be performed in parallel. Since reduction can be viewed as a slightly degenerate form of data parallelism, it is also supported.

[0110] When the compiler detects a parallelizable loop, it generates parallel code. Code generation is implemented by an extensible collection of target plugins. The purpose of a target plugin is to generate target platform-specific code as well as the management code needed to offload the computation. The compiler then essentially becomes a driver for the transformations implemented by the plugins. Each plugin therefore needs to emit three sections of code: a prologue, a target region, and an epilogue. The interaction between these sections and the target is shown in Figure 13. Synchronous operations are represented by solid arrows, and asynchronous operations are represented by dashed arrows. In the prologue, the target is expected to set up target platform memory and execute the target region asynchronously.

[0111] This potentially involves allocating space for data accessed within the target region, including both arrays and scalar variables, and initiating data transfers. However, if the host and target platforms share the same physical memory, this step may be omitted. The compiler processes each target's prologue in order, followed by all epilogues. During the epilogue phase, the targets wait for the target region to complete and post-process the results as needed. Post-processing may be required, for example, for parallel reductions. These may also be done asynchronously to overlap the computation and all data transfers. When transforming a loop that also contains reductions, the epilogue further aggregates the reduction results for each target's slice of work.

[0112] Currently, we have implemented two main target plugins: CUDA and OpenMP targets.

[0113] Below we briefly describe our design and implementation, and finally we describe the target-specific tuning parameters that the plugin exposes, as well as how we model dynamic workload partitioning.

[0114] Regarding the OpenMP target plugin, targeting OpenMP on a CPU is straightforward. Because of shared memory, no complex management of allocations is required. To interoperate with the OpenMP library API, the plugin needs to outline the target region's code into separate functions that are called by the OpenMP runtime. Outlining mainly involves replacing checks against bounds for chunks of iterations per OpenMP thread with the original bounds checks, and passing scalar variables and array pointers to the target region function. Thus, in the prologue, the target plugin copies scalars and pointers and calls the OpenMP runtime. In the epilogue, it simply waits for the threads to finish.

[0115] Regarding the CUDA target plugin, in most systems, the host and GPU do not share memory. Exceptions are mobile graphics units and system-on-chip platforms, which we will not consider further here. Therefore, the primary task of the CUDA target plugin is to manage allocation and data transfer between the target platform and the host. Unlike the OpenMP target, both scalar variables and full arrays must be transferred to the GPU. The number of scalar variables is statically known and therefore easy to copy. On the other hand, both the size and location of arrays exhibit dynamic characteristics. To make matters worse, both can change at runtime. Even if the size of an array is constant, the amount of data that needs to be transferred is not. On the one hand, the amount of data required changes as the input to the program changes or as parallelized regions affect the size of the array. On the other hand, it also changes due to changes in the workload distribution caused by tuning, so these changes are frequent. Therefore, the CUDA target plugin manages memory dynamically through a runtime system, which is described in more detail below.

[0116] After allocating and copying all data to the GPU, the prologue executes the target region, which is implemented as a CUDA kernel. Finally, the prologue also initializes the transfer of results from the GPU device to host memory. Besides the allocation, all operations performed in the prologue are asynchronous with respect to the host, but are strictly ordered with respect to each other. Thus, the CUDA API requests a copy to the device, launches a CUDA kernel, and copies from the device, without waiting for each operation to complete. A wait is then performed in the epilogue to synchronize the host thread with the completion of the final copy.

[0117] For a given source region, the system creates a single tuner instance, which we call the tuning context for that region. For that instance, the system inserts special prologues and epilogues that run before all target plug-in prologues and after all epilogues, respectively. These sections are where the runtime environment invokes the tuner. The prologue updates the configuration of all tuning parameters in this context and starts timing, and the epilogue stops timing and updates the configuration.

[0118] The runtime system performs three main tasks: managing target regions to execute asynchronously, managing memory allocation and transfers, and forming the interface between the parallelized application and the tuner. The compiler automatically links these tasks into the generated binary. The host-side target region management currently only needs to handle CUDA kernels. The compiler translates the kernels into CUDA PTX (CUDA-specific assembly code) and embeds it directly into the parallelized binary.

[0119] To launch a kernel, the runtime system loads the assembly code and the JIT compiles it for the CUDA driver. The result is cached for future execution. Once compiled, the kernel can be launched. Device-side memory is managed by a custom dynamic memory allocator. The allocator is part of the runtime system and maintains a single contiguous chunk of memory. In the prologue, the sizes of all required array slices are dynamically known, and the allocator is called to remap all arrays accessed in the target region into the managed chunk, and to enlarge it if necessary. If host pointers point to overlapping regions of memory, the allocator automatically merges these mappings accordingly.

[0120] Figure 14 shows an example of such a process. The allocator allocates a contiguous array of 40 bytes. Mapping arrays A, B, and C into this memory occupies only 34 bytes. Even though A's size is 32 bytes and C's size is unknown, the loop will only access 16 bytes of C and 20 bytes of A. That is, it will access 16 bytes once directly through A at offset 0, and once indirectly through B at offset 4. To compute the mapping, the allocator maps every array Ai into a triple (Addr Ai ,Size Ai ,Offset Ai ) is expressed as Offset Ai is initially 0. If we sort these triples in ascending lexicographic order, A i and A j The subsequent tuples of Addr Ai +Size Ai >Addr Aj If the inequality holds, then A i and A j overlap in memory. When merging, Offset Aj =Addr Aj -Addr Ai and Addr Aj =AddrAi In this case, the merged triples can be trivially mapped into a contiguous space, with allocations increasing as needed.

[0121] Finally, the runtime system's tuning interface implements the interaction with the library, maintaining a registry of instantiated and running tuners and providing an interface to that registry to parallelized programs. Finally, it implements an algorithm for computing loop iteration bounds for the target region execution.

[0122] Most approaches for automatic parallelization or automatic offloading rely on heuristics, cost models, and machine and performance models to make tuning decisions. Our compiler and autotuning system combine both techniques. The autotuner allows clients to reject configurations without actually measuring them. The compiler uses this mechanism to bridge the gap between static performance models and autotuning. To this end, the compiler includes a small built-in DSL compiler that developers can use to implement rejection rules. Examples of such rules are:

number

[0123] Here, ComputeVolume is a performance model that predicts the number of dynamic instructions executed in a source region. Assuming that all instructions take the same amount of time to execute, ComputeVolume is an accurate estimate of the time required to execute the source region. On the other hand, if ComputeVolume models the target region, it further depends on tuning parameters. This allows us to make a run-time prediction in advance for the translated and tuned program. The heuristic implemented by the above simple rule can be intuitively interpreted as "execute this region on the GPU only if it performs enough computation in this configuration." As a proof-of-concept, we implemented the above rule in a compiler.

[0124] In addition to ComputeVolume, the compiler embeds several similar analytical models into the application during the transformation. These include the number of executed load and store instructions and ArithmeticIntensity. The latter describes the ratio between the ComputeVolume of a source region and the amount of memory it accesses. ArithmeticIntensity is thus a metric of the number of operations executed per byte of memory transferred. Generally, the analytical models are parametric. For example, the number of load instructions executed by a source region depends on the number of loop iterations. This number generally depends on the program's input or the current region. As a result, the compiler cannot statically evaluate the models and must rely on program execution for this evaluation. For all models, the compiler generates code to calculate their values ​​at program runtime. The models are then evaluated at runtime as needed. Furthermore, because they are parametric, the models can be leveraged for a second important purpose: Combining ComputeVolume with the number of loads and stores provides an approximation of the runtime of a parallelized loop. This means that the parameters of these expressions directly affect the loop's runtime. Because the parameters are program variables, they are ideal candidates as metrics for hybrid autotuning. Once the model is calculated, the compiler extracts all parameters and registers them in the autotuner accordingly.

[0125] Two mechanisms for deriving an analytical model from an input program are supported: one based on a polyhedral model, or, if the source region is not polyhedral representable, one based on classical interprocedural dataflow analysis. In this case, the dynamic instructions of the region are counted recursively. For example, the instruction count of a loop is the instruction count of its body multiplied by the number of loop iterations. However, this count often does not satisfy static control criteria—that is, it is neither a constant nor a provably invariant variable—leading to an underestimation of the instruction count. Therefore, evaluating the model does not yield an exact value, but rather a lower bound on the specific count, which is sufficient for the model user. Whenever an exact count is not possible, approximations are made, particularly for for and while loops with indeterminable iteration counts, if-conditional expressions with different branch instruction counts, and calls to functions that cannot be analyzed between procedures. However, if a polyhedral model is applicable to the source, approximating the count is unnecessary, and the exact instruction count can be determined. Using libbarvinok, an analytical model can be obtained directly from polyhedral descriptions of the source and target regions. This library provides an implementation of Barvinok's algorithm for efficiently counting the number of points in an integer polyhedron.

[0126] It is easy to generate code for analytical model expressions: both dataflow analysis and polyhedral models generate parametric piecewise polynomial expressions in the program arguments and, if they model the target domain, also tuning parameters.

Claims

1. 1. A computer-implemented method for configuring program parameters during runtime of a computing program for computation in a heterogeneous computing system, the method comprising: receiving, by a computer, a transformation of the computing program, the transformation comprising one or more computing applications; generating, by the computer, for each computing application, one or more tuning parameters, the one or more tuning parameters being categorized into classes; For optimized computing applications, the computer iteratively adjusting one or more tuning parameters of the computing application during runtime; executing the computing application using the adjusted one or more tuning parameters; obtaining, by the computer, performance metrics for the execution of the computing application using the adjusted one or more tuning parameters; the computer determining whether the adjusted one or more tuning parameters result in an improvement in the performance metric and whether termination criteria for terminating the adjustment are met; Including, Indicative characteristics of a dynamic state of the computing application being optimized are stored in a memory of the computer, and the computer determines to utilize a known optimization for the dynamic state indicated by the stored indicator characteristics. method.

2. the method further comprising generating the transformation by a compiler configured to evaluate the computing program to determine computing applications that can be executed in parallel by different processing units; The method of claim 1.

3. The computer optimizes the computing application using tuning routines selected from a model-based prediction process and an online search process.

3. The method according to claim 1 or 2.

4. the online search process is used to provide training data for updating the model-based prediction process; The method of claim 3.

5. the transformation is generated using a polyhedral parallelizing compiler; The method of claim 2.

6. the polyhedral parallelizing compiler converts source code into an intermediate representation and further converts the intermediate representation into binary code suitable for execution on computing platforms forming the heterogeneous computing system; The method of claim 5.

7. A computing entity, the computing entity comprising: the computing application is programmed to optimize the computing application by iteratively adjusting, during runtime, one or more tuning parameters of the computing application; executing the computing application using the adjusted one or more tuning parameters; obtaining performance metrics for the execution of the computing application using the adjusted one or more tuning parameters; and determining whether the adjusted one or more tuning parameters result in an improvement with respect to the performance metrics and whether termination criteria for terminating the adjustment are met; Indicative characteristics of a dynamic state of the computing application being optimized are stored in a memory of the computing entity, and the computing entity determines to utilize a known optimization for the dynamic state indicated by the stored indicator characteristics. Computing entities.

8. the computing entity is configured to optimize the computing application using tuning routines selected from a model-based prediction process and an online search process; A computing entity according to claim 7.

9. the computing entity is configured to use the online search process to provide training data for updating the model-based prediction process. A computing entity according to claim 8.

Images