A CUDA code generation method based on data flow analysis

Abstract

The application provides a CUDA code generation method based on data flow analysis, comprising: converting a hardware-aware algorithm into a multi-memory level data flow graph based on architecture information of a multi-memory level in target hardware, the data flow graph comprising a plurality of task nodes representing different operation tasks in the algorithm, a plurality of affine edges, and a plurality of cache nodes representing memories, the cache nodes comprising memory levels and memory data, the affine edges comprising memory access information between two cache nodes connected by the affine edges and representing memories of adjacent levels; mapping each task node to the target hardware to obtain a target hardware unit allocated to an operation task of each task node; adjusting a blocking strategy of memory data of each cache node to maximize the performance of all operation tasks as an optimization target to obtain a memory data blocking result; and generating CUDA code indicating execution of operation tasks by each target hardware unit based on the data flow graph, the memory data blocking result, and memory access information of each affine edge.

Description

Technical Field

[0001] This invention relates to the field of deep learning compilers, specifically to the field of data flow analysis and code generation, and more specifically, to a CUDA code generation method based on data flow analysis. Background Technology

[0002] Data-intensive applications such as machine learning, graphics processing, and image processing are becoming increasingly important in fields such as artificial intelligence, big data, and computer vision. These applications typically involve complex data structures, such as tensors, matrices, graphs, and images, and require high-performance and scalable computation and data movement. However, developing and optimizing such applications is challenging due to the diversity and complexity of data structures, algorithms, target architectures, and platforms. Traditional methods such as compilers, libraries, and frameworks often fail to capture the high-level semantics and low-level details of data-intensive applications, and therefore cannot achieve the high performance and scalability required in these fields.

[0003] One of the key challenges in optimizing data-intensive applications is performing fine-grained fusion and loop optimization. Fine-grained fusion refers to the process of combining multiple operations into a single operation, which reduces data movement and application memory footprint. Loop optimization refers to the process of transforming application loops, such as loop fusion, tiling, skewing, and parallelization, which can improve data locality, parallelism, and application performance. However, performing fine-grained fusion and loop optimization is difficult for the following reasons:

[0004] 1) In data-intensive applications, the granularity of operations is often too large, limiting opportunities for fusion and iterative transformations. For example, convolution operations in neural networks can be viewed as a single operation or as a combination of multiple operations, such as matrix multiplication, element-wise addition, and activation functions. The choice of granularity affects the performance and scalability of the application.

[0005] 2) Data dependencies and memory hierarchies in data-intensive applications are often very complex and dynamic, making it extremely difficult to analyze and optimize data movement and data locality. For example, graph processing applications can have different data dependencies and memory hierarchies depending on the input graph, algorithm, and platform. Due to the dynamic nature of graphs, data dependencies and memory hierarchies can also change during application execution.

[0006] 3) The cyclic access patterns in data-intensive applications are often irregular and non-affine, making it difficult to apply existing cyclic analysis and transformation techniques (such as polyhedral models). For example, sparse matrix multiplication can have irregular and non-affine cyclic access patterns, depending on the sparsity and structure of the matrix. Existing cyclic analysis and transformation techniques typically assume that the cyclic access patterns are regular and affine, making them unable to handle irregular and non-affine cases.

[0007] To address the difficulties in the above analysis, the academic and industrial communities have proposed the following two methods to attempt a solution:

[0008] The first approach involves fine-tuning performance using low-level loop synthesis techniques. For example, in TVM, a tensor expression language similar to Einstein's summation is used to detail how tensor operators perform numerical calculations. While this approach is easy for users to use, the compiler's automatic scheduling can lead to high complexity. High-level tensor expressions are quickly internally converted into low-level loop expressions, which are closer to the instructions that the computer hardware can directly execute. However, this conversion process may not adequately consider the user-level abstraction requirements, resulting in a rapid loss of abstraction.

[0009] The second approach involves assigning some performance optimization responsibilities to users through a programming model that directly corresponds to the underlying hardware. For example, in Triton, the need for issues such as shared memory synchronization and inter-thread communication is effectively eliminated, simplifying the Single Instruction Multiple Threads (SIMT) programming model for NVIDIA GPUs (Graphics Processing Units) that support Compute Unified Device Architecture (CUDA) technology. This approach greatly simplifies code generation by the compiler because the programming models of the front-end and back-end hardware are closely related, and code generation can be close to translation. However, Triton still cannot fully exploit the hardware's performance, thus failing to generate better-performing hardware executable code.

[0010] Furthermore, current methods often abstract DNN algorithms into Directed Acyclic Graphs (DAGs) in deep learning compilers and frameworks. However, DAGs describe the sequential connections between multiple operators rather than an efficient way to utilize computational resources, leading to the following problems with the DAG representation method:

[0011] As DNN algorithms have evolved, data flow and computational patterns have become increasingly complex. Various innovative DNN algorithms cannot be described as large-scale, regular linear algebraic computations, leading to numerous limitations in using DAG representations to describe current DNN algorithms. For example, when dealing with attention mechanisms involving complex interactions, the DAG representation is limited in its ability to intuitively represent these complex interactions and its optimization capabilities for parallel computing are also limited. This is particularly true for hardware-aware algorithms (DNNs), which consider hardware characteristics and are designed collaboratively with the model to improve resource utilization and algorithm efficiency. However, DAG descriptions do not involve hardware information and cannot describe efficient utilization of computational resources, making it even more difficult for existing deep learning compilers to describe and optimize these hardware-aware algorithms. To address the problems of using DAGs to describe current DNN algorithms, existing methods utilize the flexible command-line interface provided by PyTorch, which can solve some of the problems. However, this method is hampered by language boundary issues, hindering algorithm optimization and requiring a sacrifice in efficiency.

[0012] Therefore, as the data flow and computational patterns of DNN algorithms become more complex, existing methods for abstracting DNN algorithms into DAGs within deep learning compilers and frameworks do not involve hardware information, leading to difficulties in describing and optimizing hardware-aware algorithms that take into account hardware characteristics.

[0013] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0014] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a CUDA code generation method based on data flow analysis.

[0015] The objective of this invention is achieved through the following technical solution:

[0016] According to a first aspect of the present invention, a CUDA code generation method based on data flow analysis is provided, comprising: S1, acquiring a hardware-aware algorithm, and converting the algorithm into a multi-memory-level data flow graph based on multi-level memory at different levels in the target hardware, which includes multiple task nodes representing different operation tasks within the algorithm, multiple affine edges, and multiple cache nodes representing memory. Each cache node includes the memory level it represents and the memory data. Each affine edge includes memory access information between two cache nodes representing adjacent memory levels that it connects, and includes a nested loop pattern describing the data loop traversal process between the memory levels represented by the two cache nodes connected by the affine edge. S1. Describe the data access pattern of the memory data mapping process represented by the two cache nodes connected by the affine edge; S2. Map each task node of the data flow graph to the target hardware to obtain the target hardware unit allocated to the operation task of each task node; S3. Adjust the memory data partitioning strategy of each cache node with the optimization goal of maximizing the performance of all operation tasks to obtain the memory data partitioning result of each cache node; S4. Based on the data flow graph, the memory data partitioning result of each cache node, the nested loop pattern of each affine edge and the data access pattern, generate CUDA code that instructs each target hardware unit to execute the corresponding operation task.

[0017] In some embodiments of the present invention, the block segmentation strategy includes segmenting by shape and size, and S3 includes: adjusting the block shape and size of the memory data of each cache node according to the optimization objective to obtain the final block shape and size of each cache node; dividing the memory data of each cache node according to the final block shape and size of each cache node to obtain the memory data block segmentation result of each cache node, including multiple data blocks, the size and shape of the data blocks.

[0018] In some embodiments of the present invention, the nested loop pattern includes multiple iteration variables. When a cache node connected by an affine edge accesses each data block of another cache node connected by the affine edge, step S4 further includes: determining the value range and step size of the iteration variables of the nested loop pattern of the affine edge based on the shape and size of each data block in the memory data block partitioning result of the other cache node; and generating CUDA code that instructs the two cache nodes to perform memory access between the memory represented by the two cache nodes based on the nested loop pattern of the affine edge, the data access pattern, the value range, and the step size, so as to access the data block under the shape and size according to the logic of the nested loop pattern and the data access pattern of the affine edge.

[0019] In some embodiments of the present invention, the final block shape and size are obtained by: dividing the memory data of each cache node into blocks according to a preset variety of block shapes and sizes to obtain a variety of memory data block results for each cache node; evaluating the performance of all operation tasks corresponding to each block shape and size under the preset variety of block shapes and sizes; and selecting the block shape and size of each cache node that makes the performance of all operation tasks the best, as the final block shape and size of each cache node.

[0020] In some embodiments of the present invention, all operation tasks include memory access tasks and computation tasks. Memory access tasks include data loading tasks and / or data storage tasks. Multiple task nodes include computation nodes representing computation tasks and subgraph nodes representing memory access tasks. S4 further includes: generating the CUDA code according to the order of operation tasks represented by each task node in the data flow graph, including instructions for executing data loading tasks, instructions for executing data storage tasks, and instructions for executing computation tasks.

[0021] In some embodiments of the present invention, the multi-layer memory includes register memory, shared memory, and global memory, and the multiple cache nodes include cache nodes representing register memory, cache nodes representing shared memory, and cache nodes representing global memory. The data flow graph also includes multiple directed connection edges, each directed connection edge being used to represent data transmission between the cache node representing register memory and the computing node it connects to. S4 further includes: when executing a computing task, generating CUDA code that instructs the execution of data transmission between the register memory represented by the cache node and the target computing unit according to the shape and size of the data blocks in the memory data block results of the cache node representing register memory.

[0022] In some embodiments of the present invention, the performance of all operation tasks is determined by: using a performance evaluation function based on multiple performance indicators to evaluate the performance of all operation tasks under various preset block shapes and sizes, wherein the multiple performance indicators include the execution time of the operation task, memory bandwidth utilization and cache hit rate.

[0023] According to a second aspect of the present invention, a method for compiling and running a hardware awareness algorithm is provided, comprising: acquiring the hardware awareness algorithm; generating CUDA code that instructs a target hardware device to execute the hardware awareness algorithm based on the method of the first aspect of the present invention; and deploying the CUDA code to the target hardware device for execution to complete the operation task of the hardware awareness algorithm.

[0024] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the method steps of the first and second aspects of the present invention by executing the executable instructions.

[0025] Compared with the prior art, the advantages of the present invention are as follows:

[0026] First, this invention considers the memory characteristics of the hardware and constructs the algorithm as a multi-memory-level data flow graph. Memory-related information, such as memory hierarchy information, memory data, and memory access information, is introduced into the graph to more accurately describe the algorithm. Second, memory access information explicitly expresses nested loop patterns and data access patterns, enabling a better mapping of the memory access process to hardware operations. Finally, with the goal of optimizing the performance of each operation task within the algorithm, memory data is divided into blocks based on memory-related information to optimize the execution process of the operation tasks. When generating CUDA code subsequently, the hardware memory characteristics can be reasonably utilized based on the memory data block results for compilation, thus achieving a full representation of the algorithm. Attached Figure Description

[0027] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0028] Figure 1 This is a schematic diagram of the CUDA code generation method based on data flow analysis according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram illustrating the structure of converting the algorithm into a multi-memory-level data flow graph according to an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram showing the result of mapping each task node of the data flow graph to the target hardware according to an embodiment of the present invention.

[0031] Figure 4 This is a schematic diagram illustrating the process of loading shared memory data into register memory and performing calculations for performance evaluation according to an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the compilation and execution method for a hardware awareness algorithm according to an embodiment of the present invention;

[0033] Figure 6 This is a schematic diagram showing the performance evaluation comparison results of the method of the present invention and the existing method on the HGEMM algorithm according to embodiments of the present invention;

[0034] Figure 7This diagram illustrates the performance comparison between the method of the present invention and existing methods on the Fused Two HGEMMs algorithm according to embodiments of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0036] As mentioned in the background section, with the increasing complexity of data flow and computation patterns in DNN algorithms, existing methods for abstracting DNN algorithms into DAGs in deep learning compilers and frameworks do not involve hardware information, resulting in difficulties in describing and optimizing hardware-aware algorithms that take into account hardware characteristics.

[0037] To address the aforementioned problems, the inventors, while researching deep learning compilers, discovered that the root cause is that existing methods abstract DNN algorithms into DAG representations fail to express nested loops between operators, further losing nested loop information within memory data access information. Furthermore, they cannot represent the specific hardware memory information. Therefore, according to an embodiment of the present invention, a CUDA code generation method based on data flow analysis is proposed. First, the present invention considers the multi-layered memory architecture information in the hardware, constructing the hardware-aware algorithm into a multi-memory-level data flow graph. Memory-related information, such as memory hierarchy information, memory data, and memory access information, is introduced into the data flow graph to more accurately describe existing hardware-aware algorithms that consider hardware characteristics. Second, memory access information explicitly expresses nested loop patterns and data access patterns, enabling better mapping of memory access processes to hardware operations. Finally, with the goal of optimizing the performance of each operation task within the algorithm, memory data is partitioned into blocks to optimize the execution process of operation tasks. During subsequent compilation to generate CUDA code, the hardware memory architecture information can be reasonably utilized based on the memory data partitioning results, achieving a full representation of the algorithm. In other words, this invention achieves full compilation of the algorithm, generating more efficient hardware operation code and improving the algorithm's execution efficiency.

[0038] According to one embodiment of the present invention, see Figure 1 This is a flowchart illustrating a CUDA code generation method based on data flow analysis. The method includes steps S1, S2, S3, and S4. To better understand this invention, each step is described in detail below with reference to specific embodiments.

[0039] In step S1, the hardware awareness algorithm is obtained. Based on the architecture information of the multi-layer memory in the target hardware, the algorithm is converted into a multi-memory-level data flow graph, which includes multiple task nodes representing different operation tasks within the algorithm, multiple affine edges, and multiple cache nodes representing memory. The cache nodes include the memory level they represent and the memory data. The affine edge includes the memory access information between the two cache nodes representing adjacent memory levels that it connects. It includes a nested loop pattern for describing the data loop traversal process between the memory represented by the two cache nodes connected by the affine edge and a data access pattern for describing the data mapping process between the memory represented by the two cache nodes connected by the affine edge.

[0040] According to one embodiment of the present invention, hardware-aware algorithms are algorithms based on hardware-model co-design. These algorithms are more commonly used in transformers and their variants, as well as new architectures designed for large language models (LLMs). Their implementation fully considers hardware characteristics, such as hardware-aware algorithms like FlashAttention, FlashFFTConv, and Gated Linear Attention.

[0041] According to one embodiment of the present invention, the architecture information of multi-layer memory includes the memory levels of each multi-layer memory and the access relationships between the multi-layer memory, that is, whether memory access is possible between any two levels of memory in the multi-layer memory. The target hardware can have any number of memory levels, such as 3-layer, 4-layer, or 5-layer memory, and the architecture information for each level. Taking the architecture information of 3-layer memory as an example, the 3-layer memory includes, from highest to lowest level, the first level of register memory, the second level of shared memory, and the third level of global memory. Memory access is possible between any two adjacent levels. That is, memory access is possible between the first level of register memory and the second level of shared memory, and between the second level of shared memory and the third level of global memory. The multiple cache nodes include cache nodes representing register memory, cache nodes representing shared memory, and cache nodes representing global memory. The memory levels corresponding to the cache nodes representing register memory, shared memory, and global memory are the first, second, and third levels, respectively.

[0042] According to one embodiment of the present invention, the operation tasks of multiple task nodes include memory access tasks and computation tasks. Memory access tasks include data loading tasks and / or data storage tasks. The multiple task nodes include: computation nodes for representing computation tasks, and subgraph nodes for representing memory access tasks between two adjacent memory levels. A computation node is considered as a minimal task, and a subgraph node is considered as a nested data parallel task. Subgraph nodes are constructed by progressively nesting from high-level to low-level, with one subgraph node constructed for each memory access task between two adjacent memory levels. Taking a memory divided into three levels as an example, two subgraph nodes need to be constructed, including one subgraph node corresponding to the adjacent first and second levels of memory, and one subgraph node corresponding to the adjacent second and third levels of memory.

[0043] According to one embodiment of the present invention, affine edges can only be established between cache nodes corresponding to two memory levels that can be accessed, and each affine edge is directed. Specifically, if the direction of the affine edge is from the cache node representing the lower-level memory to the cache node representing the higher-level memory, it indicates that data needs to be loaded from the lower-level memory of the two adjacent levels into the higher-level memory; this process is a data loading task. Conversely, if the direction of the affine edge is not specified, it indicates that data needs to be stored in the higher-level memory of the two adjacent levels into the lower-level memory; this process is a data storage task.

[0044] According to one embodiment of the present invention, the data flow graph further includes multiple directed connection edges, each directed connection edge representing the direct data transfer process between the cache node representing the register memory and the computing node it connects.

[0045] According to one embodiment of the present invention, an affine edge includes memory access information between two cache nodes representing adjacent memory levels that it connects, including a nested loop pattern and a data access pattern (AffineMap). The nested loop pattern includes multiple iteration variables, an iteration variable start, an iteration variable end, and an iteration step size.

[0046] The memory access information when accessing matrix C, which participates in the matrix multiplication algorithm, is represented by a nested vector consisting of multiple iterative variables and the computational form between two matrices A and B. Matrix A has a dimension of M×P (M rows, P columns). Matrix B has a dimension of P×N (P rows, N columns). The resulting matrix C has a dimension of M×N. The following code example illustrates the memory access information when accessing matrix C:

[0047] Code example:

[0048] 1) for(i1=0; i1 <M; i1++)

[0049] 2) for(i2=0; i2 <N; i2++)

[0050] 3) for(i3=0; i3 <P; i3++)

[0051] 4) C[i1][i2] += A[i1][i3]×B[i3][i2];

[0052] The code above is an example of memory access information for accessing matrix C involved in the calculation. The first three lines of code form a nested loop pattern, which describes the data traversal process between the memory locations represented by the two cache nodes connected by the affine edge. The fourth line of code is a data access pattern (Affine Map), describing the data mapping process between the memory locations represented by the two cache nodes connected by the affine edge. This memory access information utilizes the three nested loops in the first three lines to traverse the elements of the two input matrices A and B, and adds their product to the corresponding positions in the result matrix C to achieve access to matrix C.

[0053] In the data iteration process described in the example above, i1, i2, and i3 are iteration variables. Specifically, i1 is the iteration variable for the outer loop, representing the iteration from 0 to M-1 through the rows of matrix A, with a step size of 1. i2 is the iteration variable for the middle loop, representing the iteration from 0 to N-1 through the columns of matrix B (which is also the resulting matrix C), with a step size of 1. i3 is the iteration variable for the inner loop, representing the iteration from 0 to P-1 through the columns of matrix A and the rows of matrix B, with a step size of 1.

[0054] The calculation logic of the data access pattern (Affine Map) in the above example is as follows: For each element C[i1][i2] in matrix C, it is composed of the sum of the products of all rows (i1) of A and the corresponding elements of all columns (i2) of B. The inner loop calculates the product of A[i1][i3] and B[i3][i2], and then accumulates it to C[i1][i2]. This process is repeated P times, completing the accumulation of the product of all elements of column (i2) of matrix B in row (i1) of matrix A.

[0055] According to one embodiment of the present invention, see Figure 2 This diagram illustrates the structure of converting an algorithm into a multi-memory-level data flow graph. The algorithm shown in the diagram is General Matrix Multiplication (GEMM), which is constructed as a multi-memory-level data flow graph.

[0056] in, Figure 2In this context, gA, gB, gC, sA, sB, sC, rA, rB, and rC are all cache nodes, representing global memory A, global memory B, global memory C, shared memory A, shared memory B, shared memory C, register memory A, register memory B, and register memory C, respectively. Each cache node carries memory information, including memory level, memory data, memory shape, and size. GEMM represents the computation node that performs matrix multiplication operations.

[0057] Figure 2 Subgraph nodes are represented by hexagons. Taking a three-level memory hierarchy as an example, the graph includes two nested subgraph nodes, representing memory access tasks between global memory and shared memory, and memory access tasks between shared memory and register memory, respectively. Each nested subgraph node contains a data flow subgraph representing a higher-level memory, thus forming a multi-memory hierarchical data flow graph. For example, the outermost nested subgraph node contains a data flow subgraph representing the second-level shared memory, while the subgraph nodes nested within it contain a data flow subgraph representing the first-level register memory. The data flow subgraph corresponding to the first level includes cache nodes and compute nodes representing register memory A, register memory B, and register memory C, and each cache node is connected to a compute node via directed edges to indicate direct data transfer.

[0058] Figure 2 In this context, the connection between two cache nodes representing different levels is called an affine edge. Specifically, affine edges connect cache nodes gA and sA, sA and rA, gB and sB, sB and rB, gC and sC, and sC and rC. When the arrow of an affine edge points to a cache node at a higher memory level, it indicates that data is loaded from the lower memory level into that higher-level memory, such as the affine edges between gA and sA, sA and rA, gB and sB, and sB and rB. Conversely, it indicates that data from a higher-level memory is stored in the lower memory level, such as the affine edges between gC and sC, and sC and rC. Each affine edge carries a nested loop pattern and a data access pattern (Affine Map).

[0059] Figure 2 The text describes how, during a single memory access process corresponding to the affine edge between gA and sA, data is accessed from the global memory A represented by gA using a data access pattern (Affine Map). An Affine Map is a data structure based on a polyhedral model, consisting of a matrix M, nested circular vectors i, and offset vector o. The matrix M is a matrix of size M. The matrix is ​​denoted by , and vector i represents a vector of size D. The following is an example of an Affine Map illustrated by a formula:

[0060] ,

[0061] in, Let matrix M represent the affine mapping data. This represents a nested vector consisting of multiple iteration variables. , and For iteration variables, This represents the offset vector. The Affine Map is obtained by multiplying the first row vector of matrix M by the nested circular vectors and then adding the result to the offset vector. The first row of the array is obtained by multiplying the second row vector of matrix M with the nested circular vector and adding it to the offset vector. The second row of the array.

[0062] The technical solution of the embodiment of step S1 described above can achieve at least the following beneficial technical effects: Algorithms described by existing methods do not include hardware memory information and nested memory access loop information, and cannot well represent hardware-aware algorithms that can fully utilize hardware information. This invention constructs a multi-memory-level data flow graph, explicitly introducing hardware memory information and memory access information with nested loops into the graph. This memory access information can explicitly express how data is loaded from lower-level memory to higher-level memory or how data is stored from higher-level memory to lower-level memory, so as to better map the memory access process to hardware operations, thereby compiling and representing algorithms that can fully utilize hardware information. Furthermore, by fully considering the hardware memory hierarchy, this invention achieves reasonable utilization of memory architecture information, which helps to generate more efficient hardware operation code subsequently. The data flow graph of this invention can better describe hardware-aware algorithms such as FlashAttention, FlashFFTConv, and Gated Linear Attention that consider hardware characteristics.

[0063] In step S2, each task node of the data flow graph is mapped to the target hardware to obtain the target hardware unit allocated to the operation tasks of each task node.

[0064] According to one embodiment of the present invention, Figure 2 Taking the constructed data flow graph as an example, each task node of the data flow graph is mapped to the target hardware, resulting in the target hardware unit allocated to the operation tasks of each task node. See also Figure 3This is a schematic diagram illustrating the mapping of each task node in the data flow graph to the target hardware. For example, mapping to the target hardware NVIDIA GPU, which has three memory levels: global memory, shared memory, and register memory, arranged in descending order of level. , and This represents register memory, shared memory, and global memory. It maps specific operational tasks to specific target hardware.

[0065] For memory access tasks, data is mapped to corresponding memory locations for loading or storing data, such as... Figure 3 In the process, after dividing matrix A and matrix B into blocks, the data from each block of matrix A and B is retrieved from the memory level. Global memory loading into memory hierarchy Shared memory, and then from Shared memory is loaded into the memory hierarchy. The register memory is used to perform computational tasks based on the loaded data. The result data obtained during the computational task execution is matrix C. After dividing matrix C into blocks, the data of these blocks is loaded from the memory level. Register memory stored to Shared memory, and then from the memory level Shared memory is stored in the memory hierarchy. The global memory is used for matrix multiplication. The computational task of matrix multiplication is mapped to the tensor core (i.e., hardware tensor core) of the NVIDIA GPU. The tensor core performs matrix multiplication and accumulation operations to obtain matrix C.

[0066] The technical solution of the above-described step S2 embodiment can achieve at least the following beneficial technical effects: Since each cache node corresponds to a different memory level, the data flow graph is constructed with full consideration of hardware characteristics, and each cache node is directly mapped to the memory level it represents. The memory access task between two cache nodes is mapped to the corresponding two memory levels, which can effectively and quickly perform mapping and kernel fusion on the hardware.

[0067] In step S3, the memory data partitioning strategy of each cache node is adjusted with the optimization goal of maximizing the performance of all operation tasks, so as to obtain the memory data partitioning result of each cache node.

[0068] According to an embodiment of the present invention, the block partitioning strategy includes partitioning by shape and size. Step S3 includes step S31: adjusting the block shape and size of the memory data of each cache node according to the optimization objective, to obtain the final block shape and size of each cache node. Step S32: partitioning the memory data of each cache node according to the final block shape and size of each cache node, to obtain the memory data partitioning result of each cache node, including multiple data blocks, the size and shape of the data blocks. The technical solution of this embodiment can achieve at least the following beneficial technical effects: In modern GPU architectures, partitioning plays an extremely important role in the performance of algorithm execution. The present invention formulates a partitioning strategy and adjusts the partitioning strategy with the goal of optimizing the performance of the operation task, evaluates and selects the partitioning strategy that is most conducive to the performance of algorithm execution, loads data based on the partitioning, and thereby generates high-efficiency and high-performance CUDA code that instructs the kernel to perform computation.

[0069] According to an embodiment of the present invention, in step S31, the final block shape and size of each cache node are obtained by the following steps S311, S312 and S313:

[0070] Step S311: Divide the memory data of each cache node into blocks according to the preset multiple block shapes and sizes to obtain multiple memory data block results for each cache node.

[0071] According to one embodiment of the present invention, multiple block shapes and sizes can be preset as alternative block partitioning strategies based on the computational task. That is, considering both the computational cost of a single block / warp and the overall parallelism, different block partitioning strategies can be preset. For example, for a matrix with a small number of rows and a large number of columns, the corresponding column dimensions are divided into larger partitions, and the row dimensions into smaller partitions, forming rectangular blocks. If the number of rows and columns are the same, the data can be divided into rows and columns using the same dimensions, forming square blocks. Furthermore, block partitioning also considers memory data layout. For example, if the matrix multiplication performed by a Tensor Core has a dimension of 16×8×16, then the blocks are partitioned in multiples of this 16×8×16 shape to facilitate data access, layout, and computation.

[0072] According to one embodiment of the present invention, multiple block sizes can be preset for each preset block shape. For example, taking a square shape as an example, the block size can be preset to 16×16, or it can be preset to 32×32. For example, taking a shape with dimensions of 16×8×16 as an example, the block size can be preset to 32×16×32, or it can be preset to 16×8×16. By presetting multiple block shapes and sizes, the memory data of each cache node is divided into blocks respectively, resulting in multiple memory data block results for each cache node.

[0073] Step S312: Evaluate the performance of all operation tasks corresponding to each block shape and size under various preset block shapes and sizes.

[0074] According to one embodiment of the present invention, the performance of all operational tasks can be determined in the following manner:

[0075] Based on a pre-defined performance evaluation function using multiple performance metrics, the performance of all operation tasks under various pre-defined block shapes and sizes is evaluated. These performance metrics include execution time, memory bandwidth utilization, and cache hit rate. The pre-defined performance evaluation function performs a weighted sum of execution time, memory bandwidth utilization, and cache hit rate to obtain the evaluated performance. The weight parameters of each metric can be set according to requirements; for higher execution efficiency, a higher weight parameter is set for execution time. This invention does not limit the selection of multiple performance metrics; appropriate evaluation metrics can be selected to measure the performance of different block partitioning strategies based on algorithm requirements. Specifically, the multiple performance metrics include: execution time, memory bandwidth utilization, cache hit rate, parallelism, and power consumption, or combinations thereof.

[0076] According to one embodiment of the present invention, the performance of all operational tasks can also be determined in the following manner:

[0077] The performance of all operational tasks is evaluated using a cost model. This embodiment achieves at least the following beneficial effects: by obtaining reasonable partitioning through evaluation, memory access latency can be reduced, memory bandwidth utilization improved, and computational task execution efficiency increased.

[0078] Step S313: Select the block shape and size of each cache node that best performs all operation tasks, and use them as the final block shape and size of each cache node.

[0079] According to one embodiment of the present invention, if dividing the memory data into square blocks of 16×16 squares yields the best performance for all operations, then the memory data is divided into blocks of this size. It should be noted that the block strategy for memory data at different levels may be the same or different.

[0080] According to one embodiment of the present invention, see Figure 4 This diagram illustrates the process of loading shared memory data into register memory and performing calculations for performance evaluation. It uses a block iterator to automatically evaluate different block shapes and sizes. For each block shape and size, the evaluation is performed according to the following process:

[0081] Based on the current block shape and size, the data in shared memory and register memory are block-wise divided. Multiple blocks of the shared memory data are loaded into register memory, and then the register memory data is block-wise transmitted to the kernel for computation. The computation results are then stored in shared memory. The performance of this complete process is evaluated through testing. By iteratively running this process and adjusting the block shape and size multiple times, the optimal block strategy is found, thus obtaining the final block shape and size for both shared memory and register memory data. This method is commonly used in deep learning and other applications requiring extensive parallel computation.

[0082] The technical solution of the embodiment in step S3 above can achieve at least the following beneficial technical effects: Reasonable partitioning allows computational tasks to better utilize the parallel computing capabilities of the GPU, reduce synchronization overhead between threads, and improve algorithm performance. For example, when processing matrix multiplication in a multi-threaded manner, dividing the matrix into appropriately sized sub-matrix blocks allows each thread to independently compute the multiplication of a sub-matrix block, thereby improving parallelism. Furthermore, this invention also optimizes the partitioning of memory data in cache nodes representing register memory and optimizes the partitioning of memory data in cache nodes representing shared memory, thus covering a wide range of optimizable hardware.

[0083] In step S4, based on the data flow graph, the memory data block results of each cache node, the nested loop pattern of each affine edge, and the data access pattern, CUDA code is generated to instruct each target hardware unit to perform the corresponding operation task.

[0084] According to an embodiment of the present invention, when a cache node connected by an affine edge accesses each data block of another cache node connected by the same affine edge, step S4 includes: determining the value range and step size of the iteration variable of the nested loop pattern of the affine edge connecting the two cache nodes based on the shape and size of each data block in the memory data blocking result of the other cache node; and generating CUDA code that instructs the two cache nodes to perform memory access based on the nested loop pattern, data access pattern, value range and step size of the iteration variable, so as to access the data block under the shape and size according to the logic of the nested loop pattern and data access pattern of the affine edge. The logic of the CUDA code that performs memory access is the same as the logic of the nested loop pattern and data access pattern of the affine edge connecting the two cache nodes. This embodiment achieves at least the following beneficial technical effects: existing methods lack access patterns and nested loop patterns in their data flow graphs, making it difficult for traditional deep learning compilers to compile and express them. However, the present invention introduces nested loop information and access patterns, directly generating code with the same logic as its nested loop pattern and data access pattern based on this information and the final blocking strategy, thus improving code generation efficiency.

[0085] According to an embodiment of the present invention, taking accessing a matrix as an example, the method of determining the value range and step size of the iteration variable in the nested loop pattern of the affine edge connecting two cache nodes is explained. If a 4×4 dimension matrix is ​​stored in memory, and the memory data is partitioned into blocks of 2×2 dimensions, the matrix is ​​divided into four sub-matrices, each of which is 2×2 dimensions. When accessing the four sub-matrices, the nested loop pattern is organized into four nested loops, namely the row dimension of the outer loop, the column dimension of the outer loop, the row dimension of the inner loop, and the column dimension of the inner loop. Each dimension of each loop corresponds to an iteration variable. The outer loop represents access based on the sub-matrix, and the memory loop represents access based on each element of the sub-matrix. The value range of the iteration variable corresponding to the row dimension and column dimension of the outer loop is 0 to 1, and the step size is 2. The value range of the iteration variable corresponding to the row dimension and column dimension of the inner loop is 0 to 1, and the step size is 1. Each sub-matrix is ​​accessed sequentially according to its shape and size.

[0086] According to one embodiment of the present invention, step S4 further includes: generating CUDA code that instructs each target hardware unit to execute corresponding operation tasks according to the chronological order of the operation tasks represented by each task node in the data flow graph, including instructions to instruct a target memory unit to execute a data loading task and / or an instruction to execute a data storage task, and instructions to instruct a target computing unit to execute a computing task. This can be achieved by topologically sorting the subgraph nodes and computing nodes, then sorting the operation tasks represented by the topologically sorted subgraph nodes and computing nodes according to their chronological order, and generating CUDA code for each operation task sequentially according to this chronological order.

[0087] According to an embodiment of the present invention, S4 further includes: when performing a computation task, generating CUDA code that instructs the execution of data transfer between the register memory represented by the cache node and the target computing unit according to the shape and size of the data blocks in the memory data block results of the cache node representing the register memory, so as to read or write the data block in the register memory according to the shape and size of each data block.

[0088] According to an embodiment of the present invention, a method for compiling and running a hardware awareness algorithm is provided, comprising: obtaining the hardware awareness algorithm; generating CUDA code for a target device based on the code generation method of the above embodiment, which instructs the target hardware device to execute the hardware awareness algorithm; and deploying the CUDA code to the target hardware device for execution to complete the operation task of the hardware awareness algorithm.

[0089] According to one embodiment of the present invention, see Figure 5 This is a schematic diagram of the compilation and execution method for the hardware awareness algorithm. First, the hardware awareness algorithm is obtained. Second, CUDA code instructing the target hardware device to execute the hardware awareness algorithm is generated using the code generation method described in the above embodiment. The CUDA code generation process includes: I: constructing a multi-memory-level data flow graph; II: mapping each operation task of the data flow graph to the target hardware; III: dividing the memory data at each level into blocks to obtain the results of each memory data block; IV: generating code instructing the target hardware to perform operations based on the results of each memory data block and the data flow graph. Finally, the code is deployed to the target hardware device for execution, for example, deployed to run on the target hardware NVIDIA GPU or AMD GPU.

[0090] To verify the beneficial effects of the present invention, the inventors conducted the following comparative experiments:

[0091] First, the inventors selected two existing methods to evaluate the performance of the method of this invention. The two existing methods are: Existing Method 1: Triton compiler and Existing Method 2: CuBLAS; Triton is a compiler designed by OpenAI based on a GPU programming model, and CuBLAS is a hardware library from NVIDIA. CuBLAS has outstanding performance in general matrix multiplication compared to many other existing methods.

[0092] Secondly, the proposed method and existing methods were tested on two algorithms: half-precision general matrix multiplication (half GEMM, or HGEMM for short) and half-precision fused matrix multiplication (Fused Two HGEMMs). Among them, Fused Two HGEMMs is the core operation of FlashAttention-2.

[0093] Finally, we obtain three matrices A, B, and C for matrix multiplication, where matrix A has dimensions M×K, matrix B has dimensions K×N, and matrix C has dimensions N×P. The testing process is as follows:

[0094] 1) The method of this invention and existing methods were tested using the HGEMM algorithm. The code for the method of this invention was generated on an NVIDIA A100 80G. A and B were obtained for different dimensions, and A×B was calculated for each dimension (see [link to documentation]). Figure 6 This diagram illustrates the performance comparison between the method of this invention and existing methods on the HGEMM algorithm. In the diagram, the vertical axis represents the average execution time, and the horizontal axis represents the matrix dimensions of A and B, respectively. The dimensions [M, N, K] correspond to [1024, 128, 64], [2048, 128, 64], [4096, 128, 64], and [8192, 128, 64], respectively. As can be seen from the diagram, when the dimensions [M, N, K] correspond to [1024, 128, 64] and [2048, 128, 64], the code generated by the method of this invention after compiling the algorithm achieves the shortest average execution time. Under the dimensions [4096, 128, 64] and [8192, 128, 64], compared to the existing method 2 which excels in general matrix multiplication, the method of this invention still achieves essentially the same high efficiency.

[0095] 2) The method of this invention and existing methods were tested using the Fused Two HGEMMs algorithm. The code for the method of this invention was generated on an NVIDIA A100 80G. A, B, and C were obtained for different dimensions, and (A×B)×C was calculated for each dimension (see [link to documentation]). Figure 7This figure illustrates a performance comparison between the method of this invention and existing methods on the Fused Two HGEMMs algorithm. In the figure, the vertical axis represents the average execution time, and the horizontal axis represents the matrix dimensions of A and B, respectively. The dimensions [M, N, K, P] take the values ​​[2048, 128, 64, 64], [4096, 128, 64, 64], [8192, 128, 64, 64], [2048, 256, 64, 64], [4096, 256, 64, 64], and [8192, 256, 64, 64]. As can be seen from the figure, the code generated by compiling the algorithm using the method of this invention achieves the shortest average execution time.

[0096] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required function can be achieved.

[0097] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0098] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0099] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A CUDA code generation method based on data flow analysis, characterized in that, The method comprises the following steps: S1, obtaining a hardware-aware algorithm, and converting the algorithm into a multi-memory level data flow graph based on architecture information of multiple memory levels in target hardware, the data flow graph comprising multiple task nodes representing different operation tasks in the algorithm, multiple affine edges, and multiple cache nodes representing memories, the cache nodes comprising memory levels and memory data represented by the cache nodes, and the affine edges comprising memory access information between two cache nodes connected by the affine edges, the memory access information comprising a nested loop pattern describing a data loop traversal process between memories represented by the two cache nodes connected by the affine edges, and a data access pattern describing a data mapping process between the memories represented by the two cache nodes connected by the affine edges; S2, mapping each task node of the data flow graph to target hardware to obtain target hardware units allocated to operation tasks of each task node; S3, adjusting a blocking strategy of memory data of each cache node to maximize performance of all operation tasks, to obtain a memory data blocking result of each cache node, the blocking strategy comprising blocking according to shapes and sizes, and the S3 comprising: adjusting shapes and sizes of the memory data of each cache node according to the optimization target to obtain final shapes and sizes of the memory data of each cache node; dividing the memory data of each cache node according to the final shapes and sizes of the memory data of each cache node to obtain the memory data blocking result of each cache node, the memory data blocking result comprising multiple data blocks, sizes, and shapes of the data blocks; S4, generating CUDA code indicating execution of corresponding operation tasks by each target hardware unit based on the data flow graph, the memory data blocking result of each cache node, and a nested loop pattern and a data access pattern of each affine edge, the nested loop pattern comprising multiple iteration variables, and the S4 further comprising: determining a value range and a step length of an iteration variable of the nested loop pattern of the affine edge according to shapes and sizes of each data block in the memory data blocking result of the other cache node; generating CUDA code indicating execution of memory access between memories represented by the two cache nodes according to the nested loop pattern and the data access pattern of the affine edge, the value range, and the step length, to access the data block in the shapes and sizes according to the logic of the nested loop pattern and the data access pattern of the affine edge.

2. The method of claim 1, wherein, The final shapes and sizes are obtained in the following manner: blocking the memory data of each cache node according to multiple preset shapes and sizes to obtain multiple memory data blocking results of each cache node; evaluating performance of all operation tasks corresponding to each shape and size in the multiple preset shapes and sizes; selecting shapes and sizes of each cache node corresponding to the best performance of all operation tasks as final shapes and sizes of each cache node.

3. The method of claim 2, wherein, All operation tasks include memory access tasks and computing tasks, the memory access tasks include data loading tasks and / or data storing tasks, the plurality of task nodes include computing nodes representing computing tasks and subgraph nodes representing memory access tasks, wherein the S4 further comprises: The CUDA code is generated in the order of operation tasks represented by each task node in the data flow graph, including instructions for executing data loading tasks, instructions for executing data storing tasks, and instructions for executing computing tasks.

4. The method of claim 3, wherein, The multi-layer memory includes register memory, shared memory, and global memory, and the plurality of cache nodes include cache nodes representing register memory, cache nodes representing shared memory, and cache nodes representing global memory, and the data flow graph further comprises a plurality of directed connection edges, each directed connection edge being used to represent data transmission between a cache node representing register memory and a computing node connected thereto, and the S4 further comprises: When executing computing tasks, the CUDA code indicating data transmission between the register memory represented by the cache node representing register memory and the target computing unit is generated according to the shape and size of the data block in the memory data block result of the cache node representing register memory.

5. The method of claim 2, wherein, The performance of all operation tasks is determined in the following manner: A performance evaluation function based on a plurality of performance indicators is used to evaluate the performance of all operation tasks under various preset block shapes and sizes, wherein the plurality of performance indicators include execution time, memory bandwidth utilization, and cache hit rate.

6. A method for compiling and running a hardware-aware algorithm, the method comprising: receiving a hardware-aware algorithm; and compiling the hardware-aware algorithm for execution on a hardware platform. It comprises: Obtaining a hardware-aware algorithm, generating CUDA code indicating the target hardware device to execute the hardware-aware algorithm based on the method of any one of claims 1-5; Deploying the CUDA code to the target hardware device for running to complete the operation tasks of the hardware-aware algorithm.

7. A computer readable storage medium characterized in that, A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method of any one of claims 1-6.

8. An electronic device, comprising: It comprises: One or more processors; And Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1-6 by executing the executable instructions.

Images