Method and device for generating program code for graphics processing unit (GPU) and computer readable medium

Abstract

A method, a calculator readable medium, and an apparatus are provided. The apparatus identifies load and store operations in a programme code accessing a structure matrix. The apparatus specifies relay data for each identified load and store operation. The relay data associated with each operation represents a structure of the matrix. The apparatus generates oracle conditions for the program code based on the relay data to facilitate buffer lifting. One or more indices of the matrix accessed by the load and store operations are unknown at the time of compilation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to communication systems, and more particularly, to techniques for proactive cache promotion in graphics processing units. BACKGROUND

[0002] Single instruction, multiple threads (SIMT) is a parallel execution model employed by some modern graphics processing units (GPUs). Such GPUs can execute a single instruction on multiple threads in synchronization by exploiting their parallel datapaths. Single-program multiple-data (SPMD) accelerator languages such as and have been developed to enhance the computational performance of GPUs with SIMT architectures.

[0003] Some modern GPUs can execute a single instruction on more threads than the number of their parallel datapaths. For example, a processor with 32 parallel datapaths can execute an instruction on 128 threads in 4 consecutive cycles. These 128 threads are hereinafter referred to as a thread block. All threads in a thread block share a program counter and instruction fetch and are executed in a synchronized manner, e.g., 32 threads in each consecutive cycle.

[0004] SIMT reduces the overhead of program counter and instruction fetch, but in some cases, the utilization of computational resources is low due to the synchronized execution model. For example, when individual threads of a processor follow different control flow paths to process an if-else block, threads following the "else" path are disabled (wait) while threads following the "if" path are executing, and vice versa. Only one control flow path is executed at a time, even though the execution is not useful for some threads. In addition, low utilization comes from redundant bookkeeping across threads. For example, in a while loop, all threads of a processor are synchronized to execute the loop count increment even though the increment is consistent (i.e., the same) across all threads. In addition to the redundant loop count computation, threads often compute the same branch condition, repeat the same base address, and perform similar address computations to retrieve data from data arrays. Therefore, there is a need to reduce the redundancy in SIMT computations to improve system performance. SUMMARY

[0005] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0006] In one aspect of the disclosure, a method, a computer readable medium, and an apparatus are provided. The apparatus identifies load and store operations in program code that access a structured matrix. The apparatus specifies relay data for each identified load and store operation. The relay data associated with each operation represents a structure of the matrix. The apparatus generates oracle conditions for the program code based on the relay data to facilitate cache promotion. One or more indices of the matrix accessed by the load and store operations are unknown at compile time.

[0007] To the accomplishment of the foregoing and related aspects, the one or more aspects comprise the features recited in the following description and the appended claims, both of which are incorporated herein by reference in their entirety. The following description and the annexed drawings set forth in detail certain illustrative aspects of the one or more aspects. These aspects are indicative, however, of but a few of the various ways in which principles of the aspects can be employed and the present description is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF DRAWINGS

[0008] The present application is illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar components, and in which:

[0009] Figure 1 is a schematic diagram illustrating a GPU.

[0010] Figure 2 is a schematic diagram of a first technique for cache promotion.

[0011] FIG. 3(A) is a schematic diagram of a function program code that utilizes an advanced data structure.

[0012] FIG. 3(B) is a schematic diagram illustrating fields of the structure of FIG. 3(A) linearized.

[0013] FIG. 4(A), (B), (C), (D) are schematic diagrams showing program code that performs non-constant stores and loads.

[0014] FIG. 5(A) shows program code for cache promotion in a first technique.

[0015] Figure 5(B) shows example program code for the buffer promotion in the second technique.

[0016] Figure 6(A) shows example program code for a two-dimensional matrix.

[0017] Figure 6(B) shows example program code generated.

[0018] Figure 7(A) shows example program code illustrating Algorithm 1.

[0019] Figure 7(B) shows example program code illustrating Algorithm 2.

[0020] Figure 8 Example program code showing the stored oracle insertion algorithm.

[0021] Figure 9 Example program code showing the loaded oracle insertion algorithm.

[0022] Figure 10 is a flowchart illustrating the oracle insertion phase of the second technique.

[0023] Figure 11 is a flowchart of a method (process) of generating program code for a graphics processing unit (GPU).

[0024] Figure 12 is a schematic diagram of a hardware embodiment of an apparatus for implementing a processing system. DETAILED DESCRIPTION

[0025] The following detailed description and attached drawings describe and illustrate various configurations by which concepts described herein can be practised. The detailed description and specific examples discussed below are intended as examples of various configurations and are not intended to limit the concepts described herein to the specific configurations described in the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts can be practised without these specific details. In some instances, well-known structures and components are shown in block diagram form in order to avoid obscuring the concepts being described. Figure 1 The detailed description is presented primarily for the purpose of enabling others to understand the concepts described herein. The detailed description provides an example of one or more implementations of the concepts described herein. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts can be practised without these specific details. In some instances, well-known structures and components are shown in block diagram form in order to avoid obscuring the concepts being described.

[0026] Several aspects of telecommunication systems will now be presented with reference to various apparatus and methods. These apparatus and methods will be described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented as hardware or software depends on the particular application and design constraints imposed on the overall system.

[0027] For example, an element, or any portion of an element, or any combination of elements can be implemented as a "processing system" that includes one or more processors. Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on a chip (SoC), baseband processors, field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described herein. One or more processors in the processing system can execute software. Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0028] Accordingly, in one or more example aspects, the functions described can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, combinations of the aforementioned types of computer-readable media, or any other medium that can be used to store computer executable instructions in the form of instructions or data structures that can be accessed by a computer.

[0029] FIG. 1 is a diagram illustrating a GPU. In this example, GPU 110 can directly schedule the next uplink transmission using SPS without waiting for the decoding result of the previous transmission, thereby reducing the overall latency and improving system efficiency.

[0030] SIMT computing system 100 includes a plurality of processors 160. Processing tasks enter SIMT computing system 100 in the form of thread blocks. Each processor 160 can process multiple thread blocks simultaneously. Each thread block contains a group of threads that share the same instructions. Each thread has its own register file 140 and shares data with other threads in the thread block through local shared memory 185.

[0031] The SIMT computing system 100 also includes a thread block allocator 110 that schedules and allocates thread blocks to the processors 160 through the processor controller 165. When a thread block is scheduled to a processor 160, the processor's instruction fetch and decode unit 120 fetches and decodes an instruction, and the thread scheduler 130 schedules the instruction and threads in the thread block to execute in the appropriate cycle; for example, when the source operands of the threads' instructions become available. The source operands can include a constant, or can be fetched from the register file 140 or memory, such as system memory 180, local shared memory 185, cache memory, or other register locations. The instruction is then sent to the processor's execution unit 150 along with the source operands for execution. The execution unit 150 can include arithmetic logic units (ALUs), floating point units (FPUs), special function units (SFUs), and load / store units (LSUs) to execute different types of instructions. After execution, the result data is written to the register file 140 or register locations.

[0032] Memory accesses, such as loads and stores, can be expensive operations in terms of latency and energy consumption. On the other hand, registers provide faster data access. Scalarization is a compiler optimization technique aimed at reducing memory accesses by promoting data from memory to registers. This can significantly improve the performance of GPU workloads, especially when dealing with memory-intensive operations.

[0033] Register promotion, also known as scalarization, is an optimization technique in modern compilers aimed at reducing memory accesses by promoting data from memory to registers. This transformation is particularly important for GPUs, as GPUs often face the "memory wall" problem, where memory accesses become a bottleneck in terms of latency and energy consumption. By processing data in registers instead of loading and storing data to and from memory, register promotion can significantly improve the performance of GPU workloads.

[0034] However, register promotion is not straightforward without knowing all the accessed pointers in advance. Current compiler solutions have limitations in this regard. For example, the memory addresses must be known at compile time, and the index of the stack pointer, which is an integer value, must also be known at compile time. This requirement for static addresses poses a significant challenge to register promotion.

[0035] In the first technique, support for certain "dynamic" pointer cases is provided by using "oracle conditions." An oracle condition is a way of adding program code to a program that specifies possible choices of addresses. When a particular choice is met, the corresponding value is used. For example, if the index is zero, the value stored in the first location is used; if the index is one, the value stored in the second location is used, and so on. This approach allows for scalarization even when the exact addresses are not known at compile time.

[0036] Figure 2 is a diagram illustrating the first technique for cache promotion. More specifically, Figure 2 shows certain program code before and after cache promotion.

[0037] The program code before cache promotion relies on memory accesses to store and retrieve values in the 'temp' array. The pointer used to access the array elements ('Index') is not known at compile time, making it a dynamic access pattern. This poses a problem for optimization techniques like scalarization, which aim to reduce memory operations by exploiting caches for data storage and manipulation.

[0038] To perform cache promotion, the compiler needs to know the possible values of the index to determine which array elements can be promoted to a cache. The first technique handles this case by using conditional statements. The compiler can generate program code that checks the value of the index and selects the appropriate cache based on the pointer value.

[0039] In this transformed program code, the elements of the temp array are promoted to caches R0 through R3. Then, conditional statements are used to determine the value of T based on the value of the index. This approach allows the compiler to perform cache promotion by generating an "oracle" that selects the appropriate cache based on the pointer value even when the index is not known at compile time.

[0040] In this example, the values 5.0 and 10.0 are stored at indices 0 and 1, respectively. When loaded from the unknown index (variable "Index") at compile time, the compiler generates an oracle condition. If the index is zero, R0 is used directly; if the index is one, R1 is used, and so on. This approach enables scalarization by using caches instead of memory accesses, albeit with some additional operations.

[0041] By using oracle conditions, the compiler can avoid memory accesses and perform scalarization even when the exact addresses are not known at compile time. This technique is particularly useful for GPU workloads, as memory accesses can be a significant performance bottleneck.

[0042] However, this approach has its limitations. It requires the compiler to generate additional conditional statements, which can increase program code size and run time. Furthermore, for large matrices or complex access patterns, it can be infeasible because the number of conditional statements grows exponentially.

[0043] Figure 3(A) is a diagram illustrating a function program code 300 using a high-level data structure. In this example, FCocTileSample is a structure containing six floating point fields, and t10 is a matrix of three such structures created in the function scope. The program code stores a value (16384.0) into the BgdMinCoc field of a structure in the matrix dynamically based on the value of u15.

[0044] Current compilers implementing the first technique described above operate on flattened memory representations when processing structure matrices. This means that the type information of the structure is lost during the compilation process, and the compiler only sees a linear layout of the matrix elements in memory.

[0045] Figure 3(B) is a diagram 350 illustrating the structure of Figure 3(A) linearized. The compiler sees only the byte offsets from the start of the matrix and does not have the field number information in the last row. This lack of structure information results in the generation of inefficient program code.

[0046] Since the compiler works on the linear layout, it cannot distinguish between field accesses within the structure. As a result, the generated program code is suboptimal because the compiler cannot utilize the type information of the structure to optimize the program code.

[0047] To compute the pointer to any field at a matrix index, the compiler can use the following formula:

[0048] Base_pointer_array + Stride x Index + field_offset

[0049] where Stride is the size of each matrix element (24 bytes in this case), and field_offset is the offset of the desired field within the structure.

[0050] The loss of structure information in the compiler intermediate representation hinders its ability to generate efficient program code, especially for high-level data structures like structure matrices. This limitation necessitates techniques that preserve and utilize the type information of the structure during the compilation process to enable more effective optimization.

[0051] When processing structure matrices, current compilers implementing the first technique described above operate on flattened memory representations. This means that the type information of the structures is lost during the compilation process, and the compiler only sees a linear layout of the matrix elements in memory. As a result, the compiler cannot distinguish between field accesses within a structure, leading to the generation of inefficient program code.

[0052] Consider the example shown in Figure 3(A), where FCocTileSample is a structure containing six floating-point fields, and t10 is a matrix of three such structures. The program code stores a value (16384.0) into the BgdMinCoc field of one structure in the matrix dynamically depending on the value of u15.

[0053] In this program code, the BgdMinCoc field is the third element of the structure, written at an unknown index at compile time. If the structure layout information is lost due to flattening, the compiler will not know that BgdMinCoc is the third element of the structure in the matrix. Instead, it will treat the matrix as a series of base values, requiring more conditional statements to handle all possible offsets.

[0054] Ideally, since there are only three instances of BgdMinCoc in the matrix, the compiler should generate only three conditional statements to handle the possible indices. However, when the structure information is flattened, the compiler loses the ability to track which offsets correspond to the BgdMinCoc field, resulting in the generation of more conditional statements to cover all possible byte offsets in the matrix.

[0055] For example, the compiler might generate program code with 18 conditional statements to cover all possible byte offsets, even though only three are actually needed to handle the BgdMinCoc field. The loss of structure information due to flattening can result in suboptimal program code generation, as the compiler cannot take advantage of the knowledge that only certain offsets correspond to the desired field. This leads to unnecessary conditional statements and decreased performance.

[0056] However, if the structure information is preserved, the compiler only needs to generate three conditional statements, one for each structure in the matrix. By preserving the type information of the structures, the compiler can more efficiently optimize the program code, reducing the number of conditional statements and improving overall performance.

[0057] The loss of structure information in the compiler intermediate representation hinders its ability to generate efficient program code, particularly for advanced data structures like structure matrices. This limitation highlights the need for techniques that preserve and utilize the type information of structures during the compilation process to enable more efficient optimization and the generation of more efficient program code.

[0058] To address the limitations of current compilers in handling structure matrices, a second technique preserves and utilizes the type information of the structure during the compilation process. This technique involves storing a data structure called "access vector relay data" in the intermediate representation of the compiler.

[0059] The access vector relay data is a triple for each dimension of the matrix containing (offset, length, stride). In the given example, the offset represents the field offset within the structure, the length represents the dimension of the matrix, and the stride represents the size of each matrix element or the size of the structure.

[0060] For example, in the program code of Figure 3(A), the structure FCocTileSample contains six float values, each occupying four bytes. The stride of the structure is 24 bytes, which means that each structure instance occupies 24 bytes in the matrix. The field BgdMinCoc is located at an offset of eight bytes from the start of the structure.

[0061] The access vector relay data for the store operation t10[int(u15)].BgdMinCoc = 16384.0; is represented as access_vector(8, 3, 24). The offset of 8 indicates that BgdMinCoc is the third element in the structure (after FgdMinCoc offset 0 and FgdMaxCoc offset 4), the length of 3 indicates the size of the matrix t10, and the stride of 24 indicates the size of each FCocTileSample instance.

[0062] These access vector relay data are added by the compiler front-end to each load and store operation in the intermediate representation of the compiler. For example, the intermediate representation of the store operation t10[int(u15)].BgdMinCoc = 16384.0; is:

[0063] store pointer, 16384.0, access_vector(8, 3, 24)

[0064] By preserving the type information of the structure through the access vector relay data, the compiler can generate more precise and efficient conditional statements for high-level data types like structure matrices. The compiler can use the access vector relay data to determine the exact offset and generate only the necessary conditional statements, instead of generating a large number of conditional statements based on the flattened memory layout.

[0065] For example, the program code in Figure 3(A) includes the following line, which dynamically assigns a value to the BgdMinCoc field based on the runtime value of u15, which is limited to be less than 3.

[0066] t10[int(u15)].BgdMinCoc = 16384.0;

[0067] If there is no access vector relay data as in the first technique, the compiler will have difficulty optimizing this operation because of the dynamic nature of the pointers and the complex data structure.

[0068] In the second technique, by knowing the exact layout and size of the data structure involved, the compiler can effectively convert the high-level program code into optimized machine-level instructions, better utilizing the GPU's cache. The oracle condition generated from the access vector relay data can be as follows:

[0069]

[0070] Here, Base_pointer represents the starting address of the matrix t10. The offset and stride values in the access vector are used to calculate the exact memory address of the BgdMinCoc field within each structure in the matrix.

[0071] Furthermore, this technique naturally extends to multi-dimensional matrices. Although the simplified example only demonstrates a one-dimensional matrix, the access vector relay data can be extended to support multiple dimensions by using a list of access vectors. Each dimension of the matrix will have its own access vector tuple, allowing the compiler to effectively handle multi-dimensional matrices.

[0072] The introduction of access vector relay data in the compiler's intermediate representation makes it possible to handle structured matrices and multi-dimensional matrices more precisely. By preserving the type information of the structure, the compiler can generate more efficient program code, reducing conditional statements, thus improving performance and reducing redundancy in SIMT computation.

[0073] Figure 4 shows program code 420 that performs non-constant stores and loads. The cache promotion of the above-mentioned first technique can require each address to be stored only once, and the stored value must be a constant or a consistent value at compile time. A consistent value is a value that remains unchanged across all threads of the shader program, even if it is unknown at compile time.

[0074] However, the second technique described here can extend the ability of cache promotion to support non-constant stores and loads. This means that this technique can handle cases where the address storing and loading values are unknown at compile time, and these values can vary across threads (i.e., non-consistent).

[0075] In the program code shown in Figure 4(A), the value T is stored at an unknown index i in the matrix Temp, and later the value is loaded from the same index i to T. In addition, another value R is later stored at the same unknown index i in the program, and then the value is again loaded from index i to T.

[0076] The second technique can handle these cases by generating oracle program code around stores and loads and using a compiler algorithm to propagate the dataflow of values. Oracle program code is a conditional statement that determines which register to use based on the value of an unknown address i.

[0077] Figure 4(B) shows the oracle program code 440 generated for the store operation Temp[i] = T. Figure 4(C) shows the oracle program code 460 generated for the load operation T = Temp[i].

[0078] To handle the dataflow between these multiple non-constant stores and loads, the compiler can use a dataflow propagation algorithm. This algorithm tracks the values stored in each register and propagates them to the appropriate load operations. By doing so, the correct values are loaded from the registers even though there are multiple non-constant stores and loads.

[0079] The dataflow propagation algorithm allows the compiler to handle complex cases where the value stored in a register can be overwritten by a subsequent store operation. By correctly tracking the dataflow and propagating the values correctly, the compiler can generate valid program code that correctly handles multiple non-constant stores and loads.

[0080] The access vector relay data described above preserves the type information of the structure and provides precise details about the offset, length, and stride of the matrix elements, which further enables the compiler to generate valid and accurate oracle program code to handle unknown addresses and non-constant values. Therefore, the second technique supports unknown accesses to locations in Temp[], whether it is a load or a store.

[0081] Figure 4(D) shows the program code 480 for the conditional select operation. In addition to using conditional branches (e.g., if-else statements) to handle oracle conditions, the second technique can also support using conditional select operations (e.g., csel instructions) when available in the target processor architecture. Conditional select operations provide an alternative method to implement oracle conditions without explicit branches.

[0082] In the program code of Figure 4(D), the csel.eg instruction conditionally selects between a new value T and an existing value in a register (r0, r1, r2, or r3) based on the value of the pointer I. The ".eg" suffix denotes an "equal to" condition, which means that if I is equal to the specified constant (0, 1, 2, or 3), the new value T is selected, otherwise the existing value in the register is preserved.

[0083] There are several advantages to using conditional select operations over conditional branches, such as reducing branch overhead, improving instruction-level parallelism, and simplifying program code generation.

[0084] The second technique support condition branching and condition selection operation provides flexibility to select the most appropriate method according to the specific requirements and characteristics of the program code optimization.

[0085] Figure 5(A) shows the register promotion program code 520 in the first technique. This program code utilizes the structure in Figure 3(A). The first technique handles the store operation to an unknown location in the structure matrix, which has limitations due to the loss of structure information during the compilation process. When the compiler operates on the flattened memory representation, it treats the matrix as a series of base values, ignoring the type information of the structure. This results in the generation of a large number of conditional statements to cover all possible byte offsets in the matrix, generating inefficient program code.

[0086] In the program code shown in Figure 5(A), t10[int(u15)].BgdMinCoc = 16384.0, where t10 is the structure matrix and BgdMinCoc is the field within the structure. The program code generated by the first technique uses 18 floating-point variables (e.g., R0 to R17) to handle all possible byte offsets of the index u15. This method does not utilize the type information of the structure and generates unnecessary conditional statements.

[0087] Figure 5(B) shows the register promotion program code 540 in the second technique. This program code utilizes the structure in Figure 3(A). The second technique uses the access vector relay data to generate more efficient program code. By preserving the type information of the structure through the access vector relay data, the compiler can detect which field of the structure each store or load operation accesses. This enables the compiler to precisely adjust the conditions, reducing the number of oracle conditions for a given store operation from 18 to just 3.

[0088] The access vector relay data provides the necessary information about the field offset, matrix length, and structure size, enabling the compiler to generate optimized program code. In this example, the compiler uses only three floating-point variables (R0, R1, and R2) to handle the store operation, corresponding to the three possible offsets of the BgdMinCoc field in the structure. The access vector relay data provides the necessary information of the field offset (8), matrix length (3), and structure stride (24) required to generate these precise oracle conditions. By using the access vector relay data, the proposed solution reduces the number of oracle conditions for this specific store operation from 18 to 3, improving the efficiency of program code generation. This method demonstrates the benefits of preserving and utilizing the type information of the structure during the compilation process, enabling more efficient optimization for complex data types like structure matrices.

[0089] The first cache buffer promotion technique described above has several limitations when handling arbitrary control flow and multiple stores to the same location. These limitations stem from the first technique's assumption of a simplified control flow structure and performing transformations based on certain restrictions. One of the main limitations is that the first technique expects stores to be performed only once, while loads can be performed multiple times. This assumption simplifies the conditional based scalar promotion process as it eliminates the need to consider the control flow effects due to multiple writes to the same location. When the store address is known and the store operation is performed only once, the first technique can easily convert the store instruction to an oracle condition for a specific location in the program code.

[0090] However, this limitation becomes problematic when handling more complex control flow structures and multiple stores to the same location. Consider the following code segment:

[0091]

[0092] In this example, the first technique struggles to handle the store operation `t832[i388+1] = x;` as it involves storing the value of a variable (x) to a variable location (`t832[i388+1]`). Additionally, there can be multiple stores to the same location within the loop, as can be seen from the statement `t832[i388+1] += t382[i388];`.

[0093] To handle such cases, the compiler needs to perform data flow analysis to determine how the value of x propagates and to which addresses it can be stored. This analysis becomes more complex when considering multiple stores and different control flow paths. For example, there can be another variable y that is stored to an unknown location, and the values of x and y can be stored in different orders depending on the control flow path taken.

[0094] The first technique does not perform this necessary data flow analysis, which limits its ability to handle arbitrary control flow and multiple stores to the same location. Therefore, it can only handle simple cases where stores are performed only once and control flow is limited to certain types of structures.

[0095] To overcome these limitations, a more advanced solution is needed to handle arbitrary control flow and perform data flow analysis to determine the propagation of values in multiple stores and different control flow paths. Such a solution would enable more comprehensive cache buffer promotion optimizations, resulting in improved performance and reduced memory access overhead in GPU workloads.

[0096] The second technique introduces a Static Single Assignment (SSA) algorithm to support scalarization in the presence of arbitrary control flow. This algorithm uses the access vector type information provided by the compiler front-end to generate precise oracle conditions, enabling efficient scalarization in complex control flow structures.

[0097] SSA is an intermediate representation used in compilers where each variable is assigned only once, and each variable is defined before it is used. This property makes SSA particularly suitable for various compiler optimizations, including scalarization. By using the SSA form, compilers can easily track the definition and use of variables, simplifying the process of converting memory accesses to register accesses.

[0098] The SSA-based scalarization algorithm utilizes the access vector relay data to generate precise oracle conditions. As previously mentioned, the access vector relay data provides information about field offsets, matrix lengths, and structure sizes, enabling the compiler to accurately determine the register location accessed by each load and store operation.

[0099] The SSA-based scalarization algorithm has the ability to handle arbitrary control flow while utilizing access vector information. Traditional scalarization techniques often struggle when dealing with complex control flow structures, such as loops and conditional statements, as it is difficult to track the flow of data across different paths. However, by combining the SSA form with access vector relay data, the proposed algorithm can effectively handle scalarization in the presence of arbitrary control flow.

[0100] The algorithm first converts the program to SSA form, ensuring that each variable has only one definition. Then, it uses the access vector relay data to identify the register location accessed by each load and store operation. With this information, the algorithm can generate precise oracle conditions for each memory access, taking into account the specific field offsets, matrix lengths, and structure sizes.

[0101] Oracle conditions are generated based on the SSA form and access vector relay data. For each load or store operation, the algorithm creates a set of conditional statements that determine which register to use based on the value of the access vector. These conditions are precise because they take into account the exact register location accessed by each operation, which is determined by the access vector relay data.

[0102] Once the oracle conditions are generated, the algorithm performs a dataflow analysis to propagate values in different control flow paths. This analysis takes into account the SSA form; values are correctly tracked and propagated according to the single assignment property. The dataflow analysis also takes into account the oracle conditions; the correct registers are used for each memory access according to the specific condition determined by the access vector.

[0103] By combining the SSA form, the access vector relay data, and the dataflow analysis, the proposed algorithm can efficiently handle scalarization in the presence of arbitrary control flow. This approach allows more accurate and efficient scalarization than traditional techniques because it takes into account the specific register location accessed by each operation and correctly propagates values in different control flow paths.

[0104] In an example of a structured matrix containing multiple fields, the SSA and access vector relay data can be applied as follows:

[0105] Source code snippet

[0106]

[0107] In the second technique, the compiler determines the access vector for field 'b' in the structure 'MyStruct'. Assuming the offsets for 'a', 'b', and 'c' are 0, 4, and 8, respectively, and each float occupies 4 bytes, the access vector for 'b' is (4, 10, 12), where 4 is the offset, 10 is the length (number of structures), and 12 is the stride (size of each structure).

[0108] In the SSA form, the loop can be transformed so that each variable is assigned only once:

[0109]

[0110]

[0111] Here, the pointer calculation uses the stride in the access vector, and each location is only allocated once per loop iteration, following the SSA form. This transformation allows the compiler to optimize the memory access pattern more efficiently by reducing the overhead of accessing individual fields through the structure and possibly enhancing the use of the cache.

[0112] In GPU programming using the SIMT (Single Instruction, Multiple Threads) model, shader programs are executed by multiple threads in parallel. Each thread can have different values for certain variables, resulting in different execution paths and possibly divergent control flow.

[0113] Variables in a shader program can be classified as uniform or varying. A uniform variable contains the same value in all threads executing the shader, even if the value is not known at compile time. On the other hand, a varying variable can contain different values in different threads, resulting in divergent behavior.

[0114] A branch in a shader program code that depends on a uniform variable is called a uniform branch. In this case, all threads will take the same execution path, either taking the branch or not. This allows efficient execution because all threads can be executed synchronously without divergence.

[0115] However, a branch that depends on a varying variable is called a divergent branch. In these cases, some threads can take the branch while others can not, resulting in divergent control flow. Divergent control flow affects performance because threads that take different paths must be executed separately, possibly leading to underutilization of GPU resources.

[0116] An example is shown below:

[0117]

[0118]

[0119] In this program code, if the index variable I is considered uniform across all threads, the compiler can generate program code using a conditional branch. This is because all threads will take the same control flow path, and the branch predictor can efficiently predict the outcome of the branch. In this case, the generated program code will be more efficient because it avoids executing unnecessary instructions.

[0120] If the variable I takes different values (0, 1, 2, or 3) for different threads, it is considered a varying variable. Each thread can take a different branch depending on its I value, resulting in divergent control flow. In this case, the GPU must execute each branch separately, possibly leading to performance degradation.

[0121] To mitigate the effects of divergent control flow, a second technique uses a conditional select statement instead of a branch. A conditional select statement allows all possible paths to be executed and selects the appropriate result based on the condition.

[0122] An example is shown below:

[0123] float temp[4];

[0124] / / temp[i] = T

[0125] r0 = csel.eg I, 0, T, r0;

[0126] r1 = csel.eg I, 1, T, r1;

[0127] r2 = csel.eg I, 2, T, r2;

[0128] r3 = csel.eg I, 3, T, r3;

[0129] In this program code, the csel.eg instruction is used to conditionally select the value of T based on the value of I. If I is equal to the specified constant (0, 1, 2, or 3), the corresponding register (r0, r1, r2, or r3) will be updated with the value of T. This approach avoids divergent control flow, as all threads execute the same instruction and select the appropriate result based on the condition.

[0130] Converting control flow statements (such as if-else) into data flow statements (such as conditional selection) is known as if-conversion. If-conversion is a compiler transformation that helps to reduce the impact of divergent control flow in GPU programs.

[0131] By analyzing the properties of variables (consistent or varying) and control flow structures, a compiler can make informed decisions about program code generation. For consistent branches, traditional branching instructions can be used effectively. However, for divergent branches, techniques like if-conversion and the use of conditional selection statements are preferred to minimize the impact of divergence and improve overall performance.

[0132] In the second technique, when the index variable is considered consistent across all threads, the compiler can use a divide-and-conquer strategy to generate more efficient program code. This strategy involves dividing the range of possible index values into smaller sub-ranges and recursively checking each sub-range to determine the appropriate register to use.

[0133] An example is shown below:

[0134]

[0135] In this program code, the index variable I is used to determine which register (r0, r1, r2, or r3) should be assigned the value of T. If I is considered consistent across all threads, meaning all threads will have the same I value, the compiler can use a divide-and-conquer approach to optimize the program code.

[0136] The divide-and-conquer strategy works by dividing the range of possible index values into smaller sub-ranges and recursively checking each sub-range. For example, the compiler can generate the following program code:

[0137]

[0138]

[0139] In this optimized program code, the compiler first checks if I is less than 2. If so, the program code enters the first branch and further checks if I is less than 1. If I is less than 1, r0 is assigned the value of T; otherwise, r1 is assigned the value of T. If I is greater than or equal to 2, the program code enters the second branch and checks if I is less than 3. If I is less than 3, r2 is assigned the value of T; otherwise, r3 is assigned the value of T.

[0140] The divide-and-conquer approach reduces the number of comparisons needed to determine the appropriate register. In the source code, if I is 3, the program code needs to perform four comparisons (I == 0, I == 1, I == 2, and I == 3) to assign the value of T to r3. In the optimized program code, only two comparisons (I < 2 and I < 3) are needed to assign the value of T to r3.

[0141] The divide-and-conquer strategy is beneficial when the index variable is considered consistent across all threads. If the index variable is varying, meaning different threads can have different index variable values, the divide-and-conquer approach can not provide any performance advantage. In this case, the compiler can generate program code that executes all branches, and threads with different index values can take different paths, resulting in divergent control flow and potential performance degradation.

[0142] Therefore, the second technique determines whether the index variable is consistent or varying in order to make informed decisions for program code generation. When the index variable is consistent, the divide-and-conquer strategy can be used to optimize the program code and reduce the number of comparisons needed. However, when the index variable is varying, the compiler can choose to generate program code that executes all branches to handle divergent control flow.

[0143] The second technique can efficiently handle multi-dimensional matrices by leveraging the divide-and-conquer strategy along with the analysis of consistent and varying variables. This approach is particularly useful when dealing with expensive memory accesses, such as in low-power GPUs where cache memory is limited and each load operation can require a significant number of cycles.

[0144] FIG. 6(A) shows an example program code 620 for a two-dimensional matrix. In this program code, temp is a two-dimensional matrix, and the values of I and J are used to access elements of the matrix. If the analysis shows that I is a varying variable and J is a consistent variable, the second technique can generate efficient program code using the divide-and-conquer strategy and conditional select (CSEL) instructions.

[0145] Figure 6(B) shows an example generated program code 640. In the generated program code, V, W, U, and X represent the elements at temp[0][0], temp[1][0], temp[2][0], and temp[3][0], respectively.

[0146] The divide-and-conquer strategy is applied to the consistent variable J, recursively dividing the range of possible values and generating nested conditional statements. This approach reduces the number of comparisons needed to determine the appropriate row of the two-dimensional matrix.

[0147] For the varying variable I, a conditional select (CSEL) instruction is used to handle the divergent control flow. The CSEL instruction selects the appropriate element (V, W, U, or X) based on the value of I for each thread. This avoids the need for explicit branching and minimizes the impact of divergent control flow on performance.

[0148] To generate these efficient program codes, the second technique relies on several components:

[0149] 1. Relay data from the type frontend: The frontend of the compiler provides relay data about the matrix type and dimensions, which guides the program code generation process.

[0150] 2. Consistent variable analysis: This technique uses a consistent variable analysis algorithm, such as the one available in the LLVM compiler infrastructure, to determine which variables are consistent across all threads.

[0151] 3. Inductive variable analysis and scalar evolution analysis: These analyses help identify and optimize variables and expressions related to the loop, enabling more accurate program code generation.

[0152] By combining these components, the second technique can generate efficient program codes that use the divide-and-conquer strategy for consistent variables and conditional select instructions for varying variables, minimizing the cost of each scalarization operation (load or store).

[0153] This technique is particularly advantageous for multi-dimensional matrices, where one dynamic pointer is consistent across threads and the other index diverges. The divide-and-conquer strategy can be applied to the consistent index, while the divergent pointer is handled using conditional select instructions.

[0154] Additionally, this technique is well-suited for situations where memory access is expensive, such as in low-power GPUs with limited cache memory. In such cases, the branch nesting generated by the divide-and-conquer strategy can be cheaper than directly accessing stack memory, improving performance and reducing energy consumption.

[0155] As mentioned above, some features of the second technique are:

[0156] 1. Efficient support for different types of aggregates (e.g., structured matrices and multi-dimensional matrices in stack memory, as well as internal constant buffers). The second technique uses "access vector relay data" that preserves the type information of the structure in the intermediate representation of the compiler. This relay data consists of a triple (offset, length, stride) for each matrix dimension. By keeping this information, the compiler can generate precise conditions for the high-level data types, resulting in more efficient program code.

[0157] 2. Precise handling of nested aggregate data types. Access vector relay data enables the compiler to efficiently handle nested data structures, as it preserves the necessary information about field offsets, matrix lengths, and structure sizes. This allows the compiler to generate optimized program code for complex data types, reducing the number of unnecessary conditional statements.

[0158] 3. Support for multiple stores and loads. The second technique uses an SSA algorithm that supports any control flow. The SSA algorithm, combined with access vector relay data, enables the compiler to efficiently handle multiple stores and loads. It performs data flow analysis to correctly propagate values across different control flow paths.

[0159] 4. Support for multi-dimensional matrices and leveraging coherence behavior. The second technique extends access vector relay data to naturally handle multi-dimensional matrices. It also leverages the analysis of consistent and varying variables in GPU workloads to generate efficient program code. When index variables are consistent across all threads, the compiler can apply a divide-and-conquer strategy to reduce the number of required comparisons. For varying variables, the compiler generates program code using conditional selection instructions to minimize the impact of divergent control flow.

[0160] 5. A solution independent of shaders, hardware, and graphics APIs. The second technique is generic and applicable to various GPU workloads, regardless of the specific shader type (compute, vertex, fragment), hardware architecture (as long as it supports conditional selection instructions), or graphics API (OpenGL, DirectX, Vulkan, Metal).

[0161] The second technique can include two main stages:

[0162] Stage 1: Relay data insertion process by the compiler frontend

[0163] Stage 2: Generating oracle conditions on the compiler intermediate representation

[0164] In the first stage, relay data is inserted into the front-end of the compiler. This relay data insertion process can be performed on an Abstract Syntax Tree (AST) or equivalent high-level Intermediate Representation (IR). One example of such an IR is the SPIR-V Intermediate Representation, commonly used to represent Vulkan shaders in a compiler.

[0165] The relay data inserted in this stage provides information about the structure and layout of data types, such as structured matrices and multi-dimensional matrices. These relay data, referred to as "access vector relay data," can include a triple for each matrix dimension: (Offset, Length, Stride). The access vector relay data preserves the type information of the structure in the intermediate representation of the compiler. By maintaining this information, the compiler can generate precise conditions in subsequent stages of the invention to handle high-level data types.

[0166] The second stage of the second technique involves generating oracle conditions on the mid-end intermediate representation of the compiler. This stage utilizes the relay data inserted in the first stage to generate efficient and precise oracle conditions to handle multiple stores and loads.

[0167] The oracle conditions can be generated using an SSA algorithm that supports arbitrary control flow. By combining the SSA form with the access vector relay data, this algorithm can efficiently handle scalarization in complex control flow structures.

[0168] The SSA-based algorithm performs data flow analysis to correctly propagate values across different control flow paths. It takes into account the access vector relay data to generate precise oracle conditions for each memory access. These oracle conditions determine which buffer to use based on the specific buffer location determined by the access vector relay data.

[0169] The generation of oracle conditions in the second stage is performed on the mid-end intermediate representation of the compiler, such as LLVM IR. LLVM is a widely used compiler infrastructure that provides a powerful and flexible intermediate representation for optimization and code generation.

[0170] The first stage of the second technique involves inserting relay data into the front-end of the compiler. This relay data insertion process is performed on an Abstract Syntax Tree (AST) or equivalent high-level Intermediate Representation (IR). One example of such an IR is the SPIR-V Intermediate Representation, commonly used to represent Vulkan shaders in a compiler. Here is an example using OpenGL and SPIR-V:

[0171] A store operation in the OpenGL code:

[0172] t10 [int(u12)].BgdMinCoc = 16384.0;

[0173] In the front-end SPIR-V intermediate form, the representation of this store operation:

[0174] %77 = OpLoad %uint %u12

[0175] %78 = OpBitcast %int %77

[0176] %80 = OpAccessChain %_ptr_Function_float %t10 %78 %int_2

[0177] OpStore %80 float_16384

[0178] In the SPIR-V representation, the matrix access is represented using the OpAccessChain operation (%80 in the example). The data type is %_ptr_Function_float, the constant offset is %int_2 (just the number 2), and the matrix index is dynamic, held in %78, corresponding to u12 in the OpenGL code.

[0179] From the data type and constant offset held in the compiler's symbol table, the compiler can fully obtain the access vector relay data (offset, stride, and matrix length). The result of the first stage is an intermediate form instruction with the access vector relay data:

[0180] %106 = store align(4) <1 x f32> %105, %102, access_vector(8, 3, 24)

[0181] In this example, 8 is the offset, 3 is the length, and 24 is the stride. The value %105 is stored at address %102. The sequence of addresses corresponding to the access vector relay data is computed as:

[0182] Array_Base_Pointer + Offset + l x Stride, by taking l from [0, Length)

[0183] For the above example, the integers added to the matrix pointer are:

[0184] 8 + 0 x 24, 8 + 1 x 24, 8 + 2 x 24

[0185] These correspond to the possible legal addresses for the store operation:

[0186] 8 + Base, 32 + Base, 56 + Base, where Base is the starting address of the array Thus, from each load / store operation with access vector relay data, the compiler can obtain the complete set of possible runtime addresses at compile time.

[0187] The process of obtaining access vector relay data involves traversing the values in the OpAccessChain instruction to find the structure and extract the necessary information. The SPIR-V representation directly provides the structure type information, dynamic index, and constant offset. By inspecting these values, the compiler can compute the access vector (e.g., access_vector(8, 3, 24)) from the data type of t10 (giving the length), the dynamic index %78 (indicating dynamic access), and the constant offset %int_2 (specifying the field offset).

[0188] By attaching the relay data to the instruction itself, the necessary information is readily available in subsequent stages of the compilation process, enabling more efficient and accurate code generation.

[0189] While the example shown here uses the SPIR-V intermediate representation, the same idea applies to other front-end representations, such as the abstract syntax trees used in compilers like Clang or ARM. The specific method of computing the access vector may vary depending on the format of the intermediate representation, but the overall concept remains the same: inspect the data type, offset, and constant to determine the access vector relay data.

[0190] A symbol table in a compiler is a data structure that stores information about the identifiers used in a program, such as variable names, function names, and data types. In one example, the following code defines a structure called'myStruct' with two floating-point fields, 'Max' and 'Min'. An array called 't10' has three elements of type'myStruct'.

[0191]

[0192] Table 1 is a symbol table containing information about the myStruct data type and the t10 array.

[0193] Table 1 - Symbol table representation for the t10 array

[0194] Array-access and field name Matrix index Field offset Bytes from start of matrix t10[0].Max 0 0 0 t10[0].Min 0 1 4 t10[1].Max 1 0 8 t10[1].Min 1 1 12 t10[2].Max 2 0 16 t10[21.Min 2 1 20

[0195] Each row in the table represents a particular element of the t10 array, identified by its array index and field name. The field offset indicates the location of the field in the myStruct data type, and the byte position from the start of the array shows the actual memory address of the element relative to the start of the array.

[0196] To access the field of any array element at index x, one can use the following formula:

[0197] x * sizeof(myStruct) + field_offset_in_bytes

[0198] For example, to access t10[2].Max, the formula would be:

[0199] 2 * S + 4

[0200] Here, sizeof(myStruct) is 8 bytes (assuming 4 bytes per float), and the field_offset_in_bytes of Max is 4.

[0201] Access vector relay data, as described above, is added by the compiler front-end to each load and store operation. It contains three components: offset, stride, and length. The offset corresponds to the field offset within the structure. The stride corresponds to sizeof(myStruct). The length is the total length of the matrix.

[0202] These access vector relay data are not limited to one-dimensional matrices; it also extends to multi-dimensional matrices. By providing these relay data, the compiler can generate more efficient code to access matrix elements and perform optimizations, such as cache-bank promotion, even in the presence of complex data structures like structure-of-array matrices.

[0203] Figures 7(A) and (B) show example code for the oracle insertion phase of the second technique. The oracle insertion phase in the second technique is composed of two main algorithms, which are applied sequentially: Algorithm-1 and Algorithm-2. These algorithms work together to collect the load and store operations for each matrix and assign a scalar to each accessed address using access vector relay data.

[0204] Algorithm-1 is responsible for collecting the load and store operations that reference each test matrix. It takes as input the base address of the matrix (i.e., the offset of the stack pointer) and uses a define-use chain walker to find all load and store operations that access the matrix.

[0205] Figure 7(A) shows example program code 720 illustrating Algorithm-1. This algorithm maintains a taskbar table and a visited set, both of which are initially empty. It first appends the base pointer (A) of the matrix to the taskbar table and adds it to the visited set. Then, it enters a loop that continues until the taskbar table is empty.

[0206] In each iteration, the algorithm pops an element (elt) from the taskbar table. If the element is a load or store operation, it checks whether the operation has associated relay data. If relay data exists, the load or store operation is added to the set of collected operations (ld_st). If relay data is missing, the algorithm returns an empty set, indicating that no optimization can be performed for the matrix.

[0207] If the element is not a load or store operation, the algorithm iterates over each user (U) of the element. For each user that is not already in the visited set, the algorithm appends it to the taskbar table and adds it to the visited set.

[0208] Finally, the algorithm returns the set of collected load and store operations (ld_st) for the given matrix.

[0209] Algorithm-2 is executed immediately after Algorithm-1. Its purpose is to assign scalars to each address that the collected load and store operations in Algorithm-1 can access. It uses the access vector relay data to determine the number of scalars required and their corresponding addresses.

[0210] Figure 7(B) shows example program code 740 illustrating Algorithm-2. This algorithm takes as input the base pointer (A) of the matrix and the set of collected load and store operations (ldst_set). It returns two maps: Slot_map and Choices_map. A "slot" refers to a specific register location within the matrix, corresponding to a specific field of the structure at a given index. Each slot is identified by its offset from the start of the matrix and is assigned a unique scalar name (e.g., A1, A2, etc.).

[0211] For each load or store operation (ld_st) in ldst_set, the algorithm extracts the offset, stride, and length from the operation's relay data. Then, it initializes an empty list called choices.

[0212] The algorithm iterates over the range [0, len) where len is the length obtained from the relay data. For each index I in this range, it computes the slot by adding the offset plus the product of the stride and I. If the slot does not already exist in Slot_map, the algorithm generates a new name for the scalar using the generate_new_name() function and adds it to Slot_map with the slot as the key.

[0213] The newly generated scalar name is appended to the choices list of the current load or store operation. After processing all indices, the choices list is added to Choices_map with ld_st as the key.

[0214] Finally, the algorithm returns Slot_map and Choices_map. Slot_map contains the mapping between each slot and its corresponding scalar name, while Choices_map contains the list of scalar names that each load or store operation can access.

[0215] These two algorithms work together to collect the information needed to perform oracle insertion. Algorithm-1 identifies the relevant load and store operations for each matrix, while Algorithm-2 determines the scalars required for each address accessed by these operations. This information is then used in subsequent steps of the oracle insertion process to generate efficient code for handling struct matrices and other complex data types.

[0216] To illustrate how the second technique assigns a scalar to each "slot" in a matrix based on the access vector relay data, consider the following example:

[0217]

[0218]

[0219] In this example, myStruct is a struct containing two float fields Max and Min, and t10 is a matrix containing three elements of type myStruct.

[0220] Additionally, the structure used to store the specific struct matrix of t

[10] can differ in size from myStruct. For example, the structure in the specific matrix can be 12 bytes in size, containing 3 slots or scalars. The length of the specific matrix is 2.

[0221] Table 2 shows the assigned scalar names, their corresponding matrix elements, and their byte positions from the start of the matrix.

[0222] Assigned scalar names and their corresponding matrix elements

[0223] Scalar name Matrix element Byte position from start of matrix A1 (group 1) t10[0].Max 0 A2 (group 2) t10[0].Min 4 A3 (group 3) t10[1].Max 8 A4 (group 1) t10[1].Min 12 A5 (group 2) t10[2].Max 16 A6 (group 3) t10[2].Min 20

[0224] The scalar names A1 to A6 are assigned according to the access vector relay data of each load or store operation. The groups (group 1, group 2, group 3) represent different oracle selection sets corresponding to specific offset, stride and length values:

[0225] group 1: offset = 0, stride = 12, length = 2

[0226] group 2: offset = 4, stride = 12, length = 2

[0227] group 3: offset = 8, stride = 12, length = 2

[0228] By applying the formula derived from the access vector relay data, the second technique determines the number of scalars and their corresponding addresses required for the oracle check of each load or store operation. This process filters the selection and reduces the set of elements to specific fields offset by stride 12, instead of considering all elements in the matrix.

[0229] The access vector relay data allows the compiler to generate more efficient code by focusing on the relevant fields and offsets, reducing the number of oracle conditions required for each load or store operation. This approach results in more accurate and optimized code generation for structure matrix and other complex data types.

[0230] Figure 8 shows an example program code 820 of the oracle insertion algorithm for store operations. This algorithm is responsible for generating the necessary oracle conditions and creating new Static Single Assignment (SSA) definitions for each store operation. This algorithm assists the second technique in optimizing the processing of structure matrices and other complex data types.

[0231] The algorithm accepts four inputs:

[0232] A: Base pointer of the matrix being processed.

[0233] ldst_set: Set of load and store operations collected by Algorithm-1.

[0234] Choices_map: Mapping associating each load or store operation with its possible choices (i.e., the scalar names corresponding to the addresses it can access).

[0235] Slot_map: Mapping associating each slot (i.e., specific register location within the matrix) with its corresponding scalar name.

[0236] The algorithm returns a map called New_definitions that contains the new SSA definitions created for each store operation.

[0237] The algorithm iterates over each load or store operation (ld_st) in the ldst_set. If the operation is a load, it is skipped because the algorithm focuses on processing stores. For each store operation, the algorithm retrieves the list of slots (slots) associated with the store from the Choices_map.

[0238] Next, the algorithm computes the pointer difference (PtrDiff) between the pointer operand of the store operation (ld_st.PointerOperand) and the base pointer of the matrix (A). The pointer difference represents the offset of the store operation from the start of the matrix. The InsertIRInst function is used to insert a new intermediate representation (IR) instruction of type POINTERDIFF that computes this offset.

[0239] Then, the algorithm initializes the SelectInst variable with the constant value zero. This variable will be used to build the oracle condition for the store operation.

[0240] For each slot in the list of slots, the algorithm performs the following steps:

[0241] 1) It inserts a new IR instruction of type COMPARE to compare the current slot with PtrDiff. This comparison determines whether the store operation is accessing the register location corresponding to the current slot.

[0242] 2) It inserts a new IR instruction of type SELECT that conditionally selects between the stored value of the store operation (ld_st.stored_value) and the current value of SelectInst, depending on the result of the comparison. This step builds the oracle condition for the store operation.

[0243] 3) It updates the SelectInst variable with the newly created SELECT instruction, effectively chaining the oracle conditions for each slot.

[0244] 4) It adds a new entry to the New_definitions map that associates the scalar name corresponding to the current slot (Slot_map[slot]) with the updated SelectInst. This step creates a new SSA definition for the stored value of each relevant register location.

[0245] After processing all the slots for the current store operation, the algorithm moves to the next store operation in the ldst_set.

[0246] Finally, the algorithm returns the New_definitions map, which contains the new SSA definitions created for each store operation. These new definitions represent the oracle conditions that determine the cache location of each store update, based on the pointer difference and the relevant slot.

[0247] The oracle conditions generated by this algorithm are essentially conditional select instructions that choose between the new value and the existing value stored at each relevant cache location. By creating these oracle conditions and new SSA definitions, the algorithm enables the compiler to generate optimized code that efficiently handles the storage of structured matrices and other complex data types, even in the presence of unknown or dynamic access patterns.

[0248] The New_definitions map returned by this algorithm will be used in the subsequent step that handles the load operations, selecting the appropriate SSA definition based on the loaded pointer operand and the relevant slot. The correct value is read from memory, taking into account the updates made by the store operations.

[0249] Therefore, the oracle insertion algorithm for store operations enables the efficient and accurate handling of complex data structures, even in the presence of unknown or dynamic access patterns.

[0250] Figure 9 shows an example program code 920 for the oracle insertion algorithm for load operations. This algorithm is responsible for generating the necessary oracle conditions to select the appropriate value for each load operation based on the SSA definitions created by the store algorithm. This algorithm enables the efficient and accurate handling of load operations, even in the presence of complex data structures and unknown or dynamic access patterns.

[0251] The algorithm accepts five inputs:

[0252] 1) A: the base pointer of the matrix being processed.

[0253] 2) New_definitions: the map of new SSA definitions created by the store algorithm for each relevant cache location.

[0254] 3) I dst_set: the set of load and store operations collected by Algorithm-1.

[0255] 4) Choices_map: the map that associates each load or store operation with its possible choices (i.e., the scalar names corresponding to the addresses it can access).

[0256] 5) Slot_map: the map that associates each slot (i.e., a specific cache location within the matrix) with its corresponding scalar name.

[0257] The algorithm iterates over each load or store operation (ld_st) in ldst_set. If the operation is a store, it is skipped because the algorithm focuses on loading. For each load operation, the algorithm retrieves the list of slots associated with that load from the Choices_map.

[0258] Next, the algorithm calculates the pointer difference (PtrDiff) between the pointer operand of the load operation (ld_st.PointerOperand) and the base pointer (A) of the matrix. This pointer difference represents the offset of the load operation from the beginning of the matrix. The InsertIRInst function is used to insert a new intermediate representation (IR) instruction of type POINTERDIFF, which calculates this offset.

[0259] The algorithm then initializes the SelectInst variable with a constant value of zero. This variable will be used to construct the Oracle conditions for the load operation.

[0260] For each slot in the slot list, the algorithm performs the following steps:

[0261] 1) Insert a new IR instruction of type COMPARE to compare the current slot with PtrDiff. This comparison determines whether the load operation is accessing the cache location corresponding to the current slot.

[0262] 2) Insert a new IR instruction of type SELECT, and conditionally select the arrival definition of the current slot based on the comparison result (SSAUpdater.get(Slot_map[slot], New_definitions[Slot_map[slot])).

[0263] The current value of SelectInst and the current value of SelectInst. This step constructs the Oracle conditions for the load operation.

[0264] 3) Update the SelectInst variable using the newly created SELECT command, effectively linking the Oracle conditions for each slot.

[0265] The arrival definition for each slot is obtained using the `SSAUpdater.get` function, which accepts the scalar name of the slot (`Slot_map[slot]`) and the corresponding new definition from the `New_definitions` map. This function is typically provided by the compiler's SSA update API, such as `SSAUpdater` in LLVM.

[0266] The correct value for each load operation is selected according to the latest definition of the corresponding cache location by using the SSAUpdater.get function. This preserves the correctness of the program in the presence of multiple stores and complex control flow. After processing all the slots of the current load operation, the algorithm moves to the next load operation in the ldst_set.

[0267] The oracle conditions generated by the algorithm are essentially conditional select instructions that select the definition of the relevant cache location according to the pointer difference and the relevant slot. Thus, the correct value is read from memory, taking into account the updates made by the store operations.

[0268] By exploiting the SSA form and the access vector relay data, the algorithm can generate efficient and accurate code for the load operations, even in the presence of unknown or dynamic access patterns. The oracle insertion algorithm for load operations, combined with the store algorithm and the access vector relay data, enables the second technique to efficiently optimize code involving structured matrices and other complex data types.

[0269] Figure 10 is a flowchart 1000 illustrating the oracle insertion phase of the second technique. The oracle insertion phase includes two main phases: collecting the load and store operations for each matrix (Algorithm-1) and assigning scalars to each accessed address using the access vector relay data (Algorithm-2).

[0270] The flowchart begins with the compiler driver program checking whether there are functions to process (operation 1010). If there are functions to process, the driver program checks whether there are matrices to process in the function (operation 1020)

[0271] If a matrix is found, the compiler applies Algorithm-1 (Find_Loads_Stores) to the matrix (operation 1030). Algorithm-1 traverses the definitions of the matrix pointer - using the chain to locate all the load and store operations related to the matrix. It returns a set of load and store operations (ldst_set) that access the matrix.

[0272] Next, the compiler applies Algorithm-2 (Assign_Scalars) to the matrix using the ldst_set obtained from Algorithm-1 (operation 1040). Algorithm-2 exploits the access vector relay data to determine the number of scalars (slots) in the matrix and their corresponding addresses. It returns two maps: Slot_map, which maps each slot to its corresponding scalar name; and Choices_map, which maps each load or store operation to its possible choices (i.e., the scalar names of the addresses it can access).

[0273] After obtaining Slot_map and Choices_map, the compiler performs oracle insertion for store operations using Algorithm-3 (Oracle_Insertion_Stores) (operation 1050). This algorithm generates the necessary oracle conditions and creates new SSA definitions for each store operation based on ldst_set, Slot_map, and Choices_map. It returns a map called New_definitions containing the new SSA definitions created for each store operation.

[0274] Finally, the compiler performs oracle insertion for load operations using Algorithm-4 (Oracle_Insertion_Loads) (operation 1060). This algorithm generates the necessary oracle conditions to select the appropriate value for each load operation based on the New_definitions map, ldst_set, Slot_map, and Choices_map. It utilizes the SSA update compiler API to obtain the reaching definitions for each load operation.

[0275] The oracle insertion phase enables efficient and accurate handling of complex data structures, such as struct-of-matrix, in the presence of unknown or dynamic access patterns. By collecting load and store operations, assigning scalars to each accessed address, and generating oracle conditions, the second technique can optimize code involving these data structures and improve overall performance.

[0276] The following example code snippets demonstrate the oracle insertion pseudocode generated by the second technique for store and load operations. These code snippets demonstrate how to utilize access vector relay data to generate efficient and accurate oracle conditions for handling complex data structures, such as struct-of-matrix.

[0277] More specifically, a set of load and store operations has a base matrix pointer %Base:

[0278]

[0279] The access vector relay data for the store and load operations is (8, 3, 24), where 8 represents the offset, 3 represents the length of the matrix, and 24 represents the stride (size of each matrix element).

[0280] For the store operation, the oracle condition pseudocode using the access vector relay data is as follows:

[0281]

[0282]

[0283] In this code, the oracle condition checks whether the value of the pointer operand %102 according to the access vector relativity data matches one of the computed addresses. The base address %Base plus the offset (8) and the product of the stride (24) and the matrix index (0, 1, or 2). If the pointer operand matches one of the computed addresses, the corresponding scalar variable (v_0, v_1, or v_2) is assigned the stored value (%105).

[0284] Similarly, for a load operation, the pseudo code using an oracle condition is as follows:

[0285]

[0286] In this code, the oracle condition checks whether the value of the pointer operand %103 according to the access vector relativity data matches one of the computed addresses, similar to the store operation. If the pointer operand matches one of the computed addresses, the corresponding scalar variable (v_0, v_1, or v_2) is assigned the target operand %106.

[0287] The second technique supports generating efficient code using a divide-and-conquer strategy when the index variable is determined to be consistent across all threads. This strategy involves recursively dividing the range of possible index values into smaller sub-ranges and generating nested conditional statements to determine the appropriate cache to use.

[0288] An example of code generated using the divide-and-conquer strategy is as follows:

[0289]

[0290]

[0291] In this code, the index variable idx is compared to progressively smaller ranges using nested conditional statements. The ranges are divided in half at each level of nesting, allowing for more efficient determination of the appropriate cache to use. For example, the code first checks whether idx is less than 8. If so, the code enters the first branch and further checks whether idx is less than 4. If idx is less than 4, the code checks whether it is less than 2. If idx is less than 2, it checks whether idx is equal to 0, and if so, %105 is assigned to v_0; otherwise, %105 is assigned to v_1. If idx is greater than or equal to 2 but less than 4, the code checks whether idx is equal to 2, and if so, %105 is assigned to v_2; otherwise, %105 is assigned to v_3. The code continues using this divide-and-conquer approach for the remaining sub-ranges.

[0292] However, if the index variable is determined to be varying across threads, the second technique is more inclined to use conditional selection instructions instead of branch generating code. This approach avoids divergent control flow; all threads execute the same instruction, thereby improving performance on SIMT architectures.

[0293] An example of program code generated using conditional selection is shown below:

[0294] v_0 = idx == 0? %105 : v_0

[0295] v_1 = idx == 1? %105 : v_1

[0296] …

[0297] In this program code, the conditional selection instruction (? : ) is used to assign the value %105 to the appropriate buffer (v_0, v_1, etc.) depending on the value of the index variable idx. This approach allows efficient execution on SIMT architectures because all threads execute the same instruction and select the appropriate value depending on the pointer.

[0298] To determine whether to use the branching divide-and-conquer strategy or the conditional selection approach, the second technique relies on a consistency analysis. Consistency analysis is a technique used in compilers, such as LLVM, to determine whether a variable has the same value (is consistent) across all threads or whether it can have different values (is varying) across threads.

[0299] If the consistency analysis determines that the index variable is consistent, the second technique deploys the branching divide-and-conquer strategy. On the other hand, if the consistency analysis determines that the index variable is varying across threads, the second technique generates program code that uses conditional selection instructions.

[0300] The use of the divide-and-conquer strategy and conditional selection is not limited to one-dimensional matrices. The second technique naturally extends these approaches to multi-dimensional matrices. By combining the divide-and-conquer strategy, conditional selection, and consistency analysis, the second technique can generate efficient program code for a variety of matrix access patterns and data structures.

[0301] FIG. 11 is a flowchart 1100 of a method (process) of generating program code for a graphics processing unit (GPU). The method can be performed by a compiler. In operation 1102, the compiler identifies load and store operations in the program code that access a structured matrix. In operation 1104, the compiler specifies relay data for each identified load and store operation. The relay data associated with each operation represents the structure of the matrix. In certain configurations, the relay data includes an offset that indicates a field offset within the structure, a length that indicates a dimension of the matrix, and a stride that indicates a size of the structure.

[0302] In operation 1106, the compiler generates oracle conditions for the program code based on the relay data to facilitate cache promotion. One or more indices of the matrix accessed by the load and store operations are not known at compile time. In operation 1108, to generate the oracle conditions, the compiler determines a set of possible addresses accessed by each load or store operation from the relay data. In operation 1110, the compiler assigns a scalar to each unique address in the set of possible addresses. In operation 1112, the compiler generates a condition for each scalar that compares the address of the load or store operation to the address corresponding to the scalar.

[0303] In certain configurations, the oracle condition for a store operation creates a new definition for each scalar assigned to an address. The oracle condition for a load operation selects an arrival definition for each scalar based on the new definitions created by the store operations.

[0304] In operation 1114, for each store operation, the compiler computes a pointer difference between the pointer operand of the store operation and the base pointer of the matrix. In operation 1116, the compiler generates a select instruction that conditionally selects the store value of the store operation based on comparing the pointer difference to each slot accessed by the load and store operations. In operation 1118, the compiler creates a new definition for each scalar mapped to the store operation using the select instruction.

[0305] In operation 1120, for each load operation, the compiler computes a pointer difference between the pointer operand of the load operation and the base pointer of the matrix. In operation 1122, the compiler generates a select instruction that conditionally selects an arrival definition for each scalar mapped to the load operation based on comparing the pointer difference to each slot accessed by the load and store operations. The arrival definition is determined using the new definitions created for the store operations.

[0306] In operation 1124, the compiler performs a coherence analysis on the index variables used in the load and store operations. In operation 1126, the compiler generates different oracle conditions based on whether the index variables are determined to be coherent or varying across threads of the GPU. In certain configurations, the oracle condition for a coherent pointer variable uses a divide and conquer strategy to reduce the number of comparisons required. The oracle condition for a varying pointer variable uses a conditional select instruction to avoid divergent control flow.

[0307] In certain configurations, the oracle conditional handles multiple non-constant stores and loads to the same location in a matrix. The oracle condition is generated using a static single assignment (SSA) algorithm that supports scalarization in control flow. In certain configurations, the relay data is added to an intermediate representation of the program code by a compiler front end. The oracle condition is generated on the intermediate representation. In certain configurations, the structure matrix is a multi-dimensional matrix.

[0308] FIG. 12 is a diagram of an example hardware implementation for an apparatus 1278 employing a processing system 1214. The apparatus 1278 can implement a compiler 1209. The processing system 1214 can be implemented with a bus architecture, as represented generally by the bus 1224. The bus 1224 can include any number of interconnecting buses and bridges, depending on the specific application of the processing system 1214 and the overall design constraints. The bus 1224 links together various circuits including the processor(s) and / or hardware components, represented by the processor 1204, the network controller 1210, and the computer-readable medium / memory 1206. The bus 1224 can also link various other circuits such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further.

[0309] The processing system 1214 can be coupled to the network controller 1210. The network controller 1210 provides a means for the processing system 1214 to communicate with various other apparatus over a network. The network controller 1210 receives a signal from the network, extracts information from the received signal, and provides the extracted information to the processing system 1214, specifically the communication component 1220 of the apparatus 1278. In addition, the network controller 1210 receives information from the processing system 1214, specifically the communication component 1220, and based on the received information, generates a signal to be sent to the network. The processing system 1214 includes the processor 1204 coupled to the computer-readable medium / memory 1206. The processor 1204 is responsible for general processing, including the execution of software stored on the computer-readable medium / memory 1206. The software, when executed by the processor 1204, causes the processing system 1214 to perform the various functions described supra for any particular apparatus. The computer-readable medium / memory 1206 can also be used for storing data that is manipulated by the processor 1204 when executing software. The processing system further includes the compiler 1209. These components can be software components running in the processor 1204, resident / stored in the computer-readable medium / memory 1206, one or more hardware components, or some combination of

[0310] The apparatus 1278 can include means for performing the operations described above with regard to FIG. 11. The aforementioned means can be one or more of the aforementioned one or more components of the apparatus 1278 and / or the processing system 1214 of the apparatus 1278 configured to perform the functions recited by the aforementioned means.

[0311] It should be understood that the particular order in which the operations in the presently claimed flow / process diagrams have been presented is exemplary. The operations in the presently claimed flow / process diagrams can be rearranged in other orders according to design preferences. Moreover, some of the illustrated blocks can be omitted or combined. The accompanying method claims present elements of the various blocks in exemplary order only and are not meant to limit the particular order in which the various blocks are presented. It is therefore intended that the ordering be interpreted as an example only.

[0312] The preceding description is intended to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects. Thus, the claims are not intended to be limited to the aspects presented herein, but are to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean "one and only one" unless specifically so stated, but rather "one or more." The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term "some" refers to one or more. Combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof include any combination of A, B, and / or C, and can include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" can be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combination can contain one or more members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. The words "module," "mechanism," "element," "device," and the like can not be a substitute for the word "means." As such, no claim element is to be construed as a means plus function unless the element is expressly recited using the phrase "means for."

[0313] While the application has been described in terms of several embodiments, those skilled in the art will recognize that the application is not limited to the embodiments described and can be practiced with modification and alteration within the spirit and scope of the appended claims. Accordingly, the description is to be regarded as illustrative instead of restrictive on the application.

Claims

1. A method for generating program code for a graphics processing unit (GPU), the method comprising: Identify load and store operations that access the structure matrix in the program code; For each identified load and store operation, relay data is specified, which is associated with each operation representing a structure matrix; as well as Based on the relay data, oracle conditions are generated for the program code to facilitate register promotion, wherein one or more indices of the matrix accessed by the load and store operations are unknown at compile time.

2. The method according to claim 1, wherein the relay data comprises: An offset that indicates the field offset within the structure; A length, indicating the dimension of the matrix; as well as A stride indicates the size of the structure.

3. The method according to claim 1, wherein generating the Oracle conditions comprises: Based on this relay data, a set of possible addresses is determined for each load or store operation; Assign a scalar to each unique address in the set of possible addresses; as well as Generate a condition for each scalar that compares the address of the load or store operation with the address corresponding to that scalar.

4. The method of claim 3, wherein the oracle condition of the storage operation creates a new definition for each scalar assigned to the address.

5. The method of claim 4, wherein the Oracle conditions of the load operation select the arrival definition for each scalar based on the new definition created by the storage operation.

6. The method of claim 1, wherein generating the oracle condition further comprises: For each storage operation: Calculate the pointer difference between the pointer operand of the memory operation and the base pointer of the matrix; Generate a selection instruction that conditionally selects the storage value for the storage operation based on comparing the pointer difference with each slot accessed by the load and store operation; as well as Use the selection instruction to create a new definition for each scalar mapped to this storage operation.

7. The method of claim 6, wherein generating the oracle condition further comprises: For each load operation: Calculate the pointer difference between the pointer operand of the load operation and the base pointer of the matrix; Generate a selection instruction that conditionally selects the arrival definition of each scalar mapped to the load operation by comparing the pointer difference with each slot accessed by the load and store operations, wherein the arrival definition is determined using a new definition created for the store operation.

8. The method of claim 1, further comprising: Perform a consistency analysis on the indicator variables used in the load and store operations; as well as Different Oracle conditions are generated based on whether the indicator variable is determined to be consistent or variable across threads on the GPU.

9. The method of claim 8, wherein the oracle condition of the consistency indicator variable uses a divide-and-conquer strategy to reduce the number of comparisons required.

10. The method of claim 8, wherein the Oracle condition for the changing indicator variable uses a conditional selection instruction to avoid divergent control flow. Wait for instructions from the DLA driver to continue the execution.

11. The method of claim 1, wherein the oracle condition processing stores and loads multiple non-constants at the same position in the matrix.

12. The method of claim 1, wherein the oracle condition is generated using a Static Single Specify (SSA) algorithm that supports scalarization in control flow.

13. The method of claim 1, wherein the relay data is added by the compiler front-end to the intermediate representation of the code.

14. The method of claim 13, wherein the oracle condition is generated on the intermediate representation.

15. The method of claim 1, wherein the structure matrix is ​​a multidimensional matrix.

16. An apparatus for generating program code for a graphics processing unit (GPU), comprising: A memory; as well as At least one is coupled to the memory and configured as follows: Identify load and store operations that access the structure matrix in the program code; For each identified load and store operation, relay data is specified, which is associated with each operation and represents the structure of the matrix; as well as Based on the relay data, oracle conditions are generated for the program code to facilitate register promotion, wherein one or more indices of the matrix accessed by the load and store operations are unknown at compile time.

17. The apparatus of claim 16, wherein the relay data comprises: An offset that indicates the field offset within the structure; A length, indicating the dimension of the matrix; as well as A stride indicates the size of the structure.

18. The apparatus of claim 16, wherein at least one processor is further configured to generate Oracle conditions in such a manner as: Determine a set of possible addresses accessed by each load or store operation based on the metadata; assign a scalar to each unique address in the set of possible addresses; and generate a condition for each scalar that compares the address of the load or store operation with the address corresponding to the scalar.

19. The apparatus of claim 18, wherein the oracle condition of the storage operation creates a new definition for each scalar assigned to the address.

20. A calculator-readable medium storing calculator executable code for generating program code for a graphics processing unit (GPU), comprising: identifying load and store operations in program code that access a structure matrix; specifying metadata for each identified load and store operation, the metadata associated with each operation representing the structure of the matrix; and generating oracle conditions for the program code based on the metadata to facilitate register lift, wherein one or more indices of the matrix accessed by the load and store operations are unknown at compile time.

Images