CUDA code generation method based on data flow analysis

Abstract

The invention provides a CUDA (Compute Unified Device Architecture) code generation method based on data flow analysis, which comprises the following steps: converting a hardware sensing algorithm into a multi-memory hierarchical data flow diagram based on architecture information of a multi-layer memory in target hardware, including a plurality of task nodes representing different operation tasks in the algorithm, a plurality of affine edges and a plurality of cache nodes representing memories, the cache nodes comprise memory hierarchies and memory data, and the affine edge comprises memory access information between two cache nodes which are connected with the affine edge and represent memories of adjacent hierarchies; mapping each task node into target hardware to obtain a target hardware unit allocated to the operation task of each task node; by taking maximization of the performance of all the operation tasks as an optimization target, adjusting a blocking strategy of the memory data of each cache node to obtain a memory data blocking result; and based on the data flow diagram, the memory data partitioning result and the memory access information of each affine edge, generating a CUDA code for indicating each target hardware unit to execute the operation task.

Description

Technical Field

[0001] The present invention relates to the field of deep learning compilers, specifically to the field of data flow analysis and code generation, and more specifically to a CUDA code generation method based on data flow analysis. Background Art

[0002] Data-intensive applications such as machine learning, graph processing, and image processing are becoming increasingly important in various fields such as artificial intelligence, big data, and computer vision. These applications often involve complex data structures such as tensors, matrices, graphs, and images, and require high-performance and scalable computation and data movement. However, developing and optimizing such applications is challenging due to the diversity and complexity of data structures, algorithms, target architectures, and platforms. Traditional approaches such as compilers, libraries, and frameworks often fail to capture the high-level semantics and low-level details of data-intensive applications, and therefore fail to achieve the high performance and scalability required in the above fields.

[0003] One of the key challenges in optimizing data-intensive applications is performing fine-grained fusion and loop optimization. Fine-grained fusion refers to the process of combining multiple operations into a single operation, which can reduce data movement and the memory footprint of the application. Loop optimization refers to the process of transforming the application loops, such as loop fusion, tiling, skewing, and parallelization, which can improve data locality, parallelism, and the performance of the application. However, performing fine-grained fusion and loop optimization is difficult for the following reasons:

[0004] 1) The granularity of operations in data-intensive applications is often too large, limiting the opportunities for fusion and loop transformations. For example, the convolution operation in a neural network can be viewed as a single operation or as a combination of multiple operations, such as matrix multiplication, element addition, activation function, etc. The choice of granularity affects the performance and scalability of the application.

[0005] 2) The data dependencies and memory hierarchies of data-intensive applications are often very complex and dynamic, which makes it very difficult to analyze and optimize the data movement and data locality of the application. For example, a graph processing application can have different data dependencies and memory hierarchies depending on the input graph, algorithm, and platform. Due to the dynamic nature of the graph, the data dependencies and memory hierarchy will also change during the execution of the application.

[0006] 3) The loop access patterns of data-intensive applications are often irregular and non-affine, which makes it difficult to apply existing loop analysis and transformation techniques (such as polyhedral models). For example, sparse matrix multiplication can have irregular and non-affine loop access patterns, depending on the sparsity and structure of the matrix. Existing loop analysis and transformation techniques usually assume that the loop access patterns are regular and affine, which makes them unable to handle irregular and non-affine cases.

[0007] In response to the difficulties of the above analysis, the current academic and industrial circles have also proposed the following two methods to try to solve them:

[0008] The first approach is to fine-tune performance using low-level loop program synthesis techniques, such as in TVM, where a tensor expression language similar to Einstein summation is used to detail how tensor operators perform numerical computations. Although this approach is easy for users to use, the compiler's automatic scheduling may result in high complexity. High-level tensor expressions are quickly converted to low-level loop expressions internally, which are closer to the instruction form that computer hardware can directly execute. However, this conversion process may not fully consider the abstraction requirements at the user level, resulting in the abstraction being quickly lost.

[0009] The second approach is to let users take on some of the responsibilities for performance optimization through a programming model that directly corresponds to the underlying hardware. For example, in Triton, the Single Instruction Multiple Threads (SIMT) programming model of NVIDIA GPUs (graphics processing units) that support the Compute Unified Device Architecture (CUDA) technology is simplified by effectively eliminating the need for shared memory synchronization and inter-thread communication. This approach greatly simplifies the compiler's code generation because the programming models of the front-end and back-end hardware are closely related, and code generation can be close to translation. However, Triton still cannot exploit the full performance of the hardware and thus cannot generate hardware executable code with better performance.

[0010] In addition, existing methods often abstract DNN algorithms into directed acyclic graphs (DAGs) in current deep learning compilers and deep learning frameworks. However, the algorithm described by DAG is the relationship between multiple operators connected in sequence, rather than an efficient way to utilize computing resources. This leads to the following problems in the DAG representation method:

[0011] With the development of DNN algorithms, the flow and computing patterns of data have become more and more complex. Various innovative DNN algorithms cannot be described as large, regular linear algebra calculations, resulting in many limitations in using DAG representation methods to describe current DNN algorithms. For example, when faced with attention mechanisms involving complex interactive relationships, it is difficult to intuitively represent such complex interactive relationships, and the optimization representation capabilities for parallel computing are limited. Among them, hardware-aware algorithms (Hardware-AwareAlgorithms), which are popular in recent years, consider hardware characteristics and are co-designed based on hardware and models to improve resource utilization and algorithm efficiency. However, the DAG description method does not involve hardware information, and it is even more impossible to describe the efficient use of computing resources, making it more difficult for existing deep learning compilers to use DAG to describe and optimize these hardware-aware algorithms. In response to the problems of DAG description of current DNN algorithms, existing methods also use a flexible command line interface in PyTorch to solve some of the problems of DAG, but this method will hinder algorithm optimization due to language boundary issues, resulting in the need to sacrifice efficiency.

[0012] Therefore, as the data flow and computing mode of DNN algorithms become more complicated, the existing methods of abstracting DNN algorithms into DAGs in deep learning compilers and deep learning frameworks do not involve hardware information, and there is a problem that it is difficult to describe and optimize hardware-aware algorithms that take hardware characteristics into account.

[0013] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. The relevant information is submitted and disclosed together with the present invention solution. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0014] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a CUDA code generation method based on data flow analysis.

[0015] The objective of the present invention is achieved through the following technical solutions:

[0016] According to a first aspect of the present invention, a CUDA code generation method based on data flow analysis is provided, comprising: S1, obtaining a hardware-aware algorithm, and based on multi-layer memories at different levels in the target hardware, converting the algorithm into a multi-memory level data flow graph, which includes multiple task nodes representing different operation tasks in the algorithm, multiple affine edges and multiple cache nodes representing memories, wherein the cache node includes the memory level and memory data it represents, and the affine edge includes memory access information between two cache nodes representing adjacent levels of memory connected thereto, and includes a nested loop pattern describing the data loop traversal process between the memories represented by the two cache nodes connected by the affine edge. and a data access pattern describing the data mapping process between memories represented by two cache nodes connected by affine edges; S2, mapping each task node of the data flow graph to the target hardware to obtain the target hardware unit assigned to the operation task of each task node; S3, adjusting the memory data blocking strategy of each cache node with the optimization goal of maximizing the performance of all operation tasks to obtain the memory data blocking result of each cache node; S4, based on the data flow graph, the memory data blocking result of each cache node, the nested loop pattern and the data access pattern of each affine edge, generating a CUDA code to instruct each target hardware unit to perform the corresponding operation task.

[0017] In some embodiments of the present invention, the blocking strategy includes blocking by shape and size, and S3 includes: adjusting the blocking shape and size of the memory data of each cache node according to the optimization target to obtain the final blocking shape and size of each cache node; dividing the memory data blocking results of each cache node according to the final blocking shape and size of each cache node, including multiple data blocks, the size and shape of the data blocks.

[0018] In some embodiments of the present invention, the nested loop pattern includes multiple iteration variables. When a cache node connected by an affine edge accesses each data block of another cache node connected by the affine edge, S4 also includes: determining the value range and step size of the iteration variables of the nested loop pattern of the affine edge according to the shape and size of each data block in the memory data blocking result of the other cache node; generating a CUDA code instructing to perform memory access between the memories represented by the two cache nodes according to the nested loop pattern of the affine edge, the data access pattern, the value range and the step size, so as to access the data blocks of the shape and size according to the logic of the nested loop pattern and the data access pattern of the affine edge.

[0019] In some embodiments of the present invention, the method for obtaining the final block shape and size includes: according to a plurality of preset block shapes and sizes, the memory data of each cache node is separately blocked to obtain a plurality of memory data blocking results of each cache node; the performance of all operation tasks corresponding to each block shape and size under the preset plurality of block shapes and sizes is evaluated; and the block shape and size of each cache node corresponding to the best performance of all operation tasks is selected as the final block shape and size of each cache node.

[0020] In some embodiments of the present invention, all operation tasks include memory access tasks and computing tasks, memory access tasks include data loading tasks and / or data storage tasks, and multiple task nodes include computing nodes representing computing tasks and subgraph nodes representing memory access tasks, wherein the S4 also includes: generating the CUDA code according to the order of the operation tasks represented by each task node in the data flow graph, including instructions for executing data loading tasks, instructions for executing data storage tasks, and instructions for executing computing tasks.

[0021] In some embodiments of the present invention, the multi-layer memory includes register memory, shared memory and global memory, the multiple cache nodes include cache nodes representing register memory, cache nodes representing shared memory and cache nodes representing global memory, the data flow graph also includes multiple directed connection edges, each directed connection edge is used to represent the data transmission between the cache node representing the register memory and the computing node connected to it, and the S4 also includes: when executing the computing task, according to the shape and size of the data block in the memory data blocking result of the cache node representing the register memory, generate CUDA code instructing the register memory represented by the cache node and the target computing unit to perform data transmission.

[0022] In some embodiments of the present invention, the method for determining the performance of all operation tasks includes: using a performance evaluation function preset based on multiple performance indicators to evaluate the performance of all operation tasks under various preset block shapes and sizes, wherein the multiple performance indicators include the execution time of the operation tasks, memory bandwidth utilization and cache hit rate.

[0023] According to the second aspect of the present invention, a compilation and running method for a hardware-aware algorithm is provided, comprising: obtaining a hardware-aware algorithm, generating CUDA code instructing a target hardware device to execute the hardware-aware algorithm based on the method of the first aspect of the present invention; and deploying the CUDA code to the target hardware device for execution to complete the operation task of the hardware-aware algorithm.

[0024] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the method steps of the first aspect and the second aspect of the present invention by executing the executable instructions.

[0025] Compared with the prior art, the advantages of the present invention are:

[0026] Firstly, the present invention takes into account the memory characteristics of the hardware, constructs the algorithm into a multi-memory level data flow graph, and introduces memory-related information such as memory level information, memory data and memory access information into the graph to more accurately describe the algorithm; secondly, the memory access information explicitly expresses the nested loop pattern and the data access pattern, so as to better map the memory access process to the hardware operation; finally, with the goal of optimizing the performance of each operation task in the algorithm, the memory data is blocked according to the memory-related information to optimize the execution process of the operation task. When the CUDA code is subsequently generated, it can reasonably utilize the hardware memory characteristics for compilation based on the memory data blocking results, thereby achieving a full description of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0028] Figure 1 A schematic diagram of a CUDA code generation method based on data flow analysis according to an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a structure of converting an algorithm into a multi-memory level data flow graph according to an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the result of mapping each task node of a data flow graph according to an embodiment of the present invention to target hardware;

[0031] Figure 4 A schematic diagram of a flow chart of loading data from a shared memory into a register memory and performing calculations to perform performance evaluation according to an embodiment of the present invention;

[0032] Figure 5 A schematic diagram of a flow chart of a method for compiling and running a hardware-aware algorithm according to an embodiment of the present invention;

[0033] Figure 6 A schematic diagram of a performance evaluation comparison result of the method of the present invention and the existing method on the HGEMM algorithm according to an embodiment of the present invention;

[0034] Figure 7It is a schematic diagram of the performance evaluation comparison results of the method of the present invention according to an embodiment of the present invention and the existing method on the Fused Two HGEMMs algorithm. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] As mentioned in the background technology section, with the increasing complexity of data flow and computing patterns of DNN algorithms, existing methods of abstracting DNN algorithms into DAGs in deep learning compilers and deep learning frameworks do not involve hardware information, and there is a problem that hardware-aware algorithms that take hardware characteristics into account are difficult to describe and optimize.

[0037] In view of the above problems, the inventor found that the cause of the above problems is that the existing DNN algorithm is abstracted into DAG representation, which cannot express the loop nesting between operators, further loses the nested loop information in the memory data access information, and at the same time, cannot express the memory information of the specific hardware. Therefore, according to one embodiment of the present invention, a CUDA code generation method based on data flow analysis is proposed. First, the present invention considers the architecture information of multi-layer memory in the hardware, constructs the hardware-aware algorithm into a multi-memory hierarchical data flow graph, and introduces memory hierarchical information, memory data and memory access information in the data flow graph. These memory-related information can more accurately describe the existing hardware-aware algorithm that considers the hardware characteristics; secondly, the memory access information explicitly expresses the nested loop mode and data access mode, so as to better map the memory access process to the hardware operation; finally, with the goal of optimizing the performance of each operation task in the algorithm, memory data is divided into blocks to optimize the execution process of the operation task. When the CUDA code is subsequently compiled and generated, the architecture information of the hardware memory can be reasonably used for compilation based on the memory data division result, so as to fully express the algorithm. That is, the present invention realizes full compilation of the algorithm, generates more efficient hardware operation codes, and improves the algorithm execution efficiency.

[0038] According to one embodiment of the present invention, see Figure 1 , which is a flow chart of a CUDA code generation method based on data flow analysis. The method comprises steps S1, S2, S3 and S4. In order to better understand the present invention, each step is described in detail below in conjunction with specific embodiments.

[0039] In step S1, a hardware-aware algorithm is obtained, and based on the architectural information of the multi-layer memory in the target hardware, the algorithm is converted into a multi-memory level data flow graph, which includes multiple task nodes representing different operation tasks in the algorithm, multiple affine edges and multiple cache nodes for representing memory. The cache node includes the memory level and memory data it represents, and the affine edge includes memory access information between two cache nodes representing adjacent levels of memory connected by it, including a nested loop pattern for describing the data loop traversal process between the memories represented by the two cache nodes connected by the affine edge and a data access pattern for describing the data mapping process between the memories represented by the two cache nodes connected by the affine edge.

[0040] According to one embodiment of the present invention, a hardware-aware algorithm is an algorithm based on hardware-model co-design. The algorithm is more applied to transformers and their variants as well as new architectures designed for large language models (LLM). Its implementation fully considers the hardware characteristics of the algorithm, such as FlashAttention, FlashFFTConv, Gated Linear Attention and other hardware-aware algorithms.

[0041] According to one embodiment of the present invention, the architecture information of the multi-layer memory includes the memory levels of each multi-layer memory and the access relationship between the multi-layer memory, that is, whether memory access can be performed between every two levels of memory of the multi-layer memory. The target hardware can be any number of levels of memory, such as 3-layer memory, 4-layer memory or 5-layer memory and the architecture information of each level of memory. Taking the architecture information of the 3-layer memory as an example, the 3-layer memory includes the register memory of the first level, the shared memory of the second level and the global memory of the third level in order from high to low level, and memory access can be performed between any two adjacent levels of memory. That is, memory access can be performed between the register memory of the first level and the shared memory of the second level, and access can be performed between the shared memory of the second level and the global memory of the third level. Among them, the multiple cache nodes include a cache node representing the register memory, a cache node representing the shared memory and a cache node representing the global memory. The cache nodes representing the register memory, the cache nodes representing the shared memory and the cache nodes representing the global memory respectively correspond to the first level, the second level and the third level.

[0042] According to one embodiment of the present invention, the operation tasks of multiple task nodes include memory access tasks and computing tasks, the memory access tasks include data loading tasks and / or data storage tasks, and the multiple task nodes include: computing nodes, which are used to represent computing tasks, and subgraph nodes, which are used to represent memory access tasks between two adjacent levels of memory. The computing node is regarded as a minimum task, and the subgraph node is regarded as a nested data parallel task. The subgraph nodes are gradually nested and constructed from the high level to the low level, and a subgraph node is constructed for each memory access task between two adjacent levels of memory. Taking the memory divided into three levels as an example, two subgraph nodes need to be constructed, including a subgraph node corresponding to the adjacent first and second levels of memory, and a subgraph node corresponding to the adjacent second and third levels of memory.

[0043] According to one embodiment of the present invention, an affine edge can only be established between cache nodes corresponding to two memory levels that can be accessed, and each affine edge is directed. If the direction of the affine edge is from a cache node representing a low-level memory to a cache node representing a high-level memory, it means that data needs to be loaded from the low-level memory of two adjacent levels to the high-level memory, and this process is a data loading task; conversely, it means that the high-level memory data of two adjacent levels needs to be stored in the low-level memory, and this process is a data storage task.

[0044] According to one embodiment of the present invention, the data flow graph further includes a plurality of directed connection edges, each directed connection edge being used to represent a direct data transmission process between a cache node representing a register memory and a computing node connected thereto.

[0045] According to one embodiment of the present invention, an affine edge includes memory access information between two cache nodes representing adjacent levels of memory connected thereto, including a nested loop pattern and a data access pattern (AffineMap). The nested loop pattern includes multiple iteration variables, an iteration variable start (start), an iteration variable end (end) and an iteration step (stride).

[0046] The memory access information when accessing the matrix C involved in the matrix multiplication algorithm is represented in the form of a loop nested vector consisting of multiple iteration variables and the calculation between two matrices A and B. Among them, the dimension of matrix A is M×P, that is, M rows and P columns. The dimension of matrix B is P×N, that is, P rows and N columns. The dimension of the result matrix C is M×N. The following is an introduction to the memory access information when accessing matrix C with a code example:

[0047] Code example:

[0048] 1) for(i1=0; i1 <M; i1++)

[0049] 2) for(i2=0; i2 <N; i2++)

[0050] 3) for(i3=0; i3 <P; i3++)

[0051] 4) C[i1][i2] += A[i1][i3]×B[i3][i2];

[0052] The above code is an example of memory access information for accessing the matrix C involved in the calculation. The first three lines of code are nested loop patterns, which describe the data loop traversal process between the memories represented by the two cache nodes connected by the affine edge. The fourth line of code is the data access pattern (Affine Map), which describes the data mapping process between the memories represented by the two cache nodes connected by the affine edge. This memory access information uses the three nested loops in the first three lines to traverse the elements of the two input matrices A and B, and accumulate their products to the corresponding positions of the result matrix C to achieve access to the matrix C.

[0053] In the above example, i1, i2 and i3 are iteration variables during the data loop traversal process. i1 is the iteration variable of the outer loop, which means traversing the rows of matrix A from 0 to M-1, and the step length of this iteration variable is 1. i2 is the iteration variable of the middle loop, which means traversing the columns of matrix B (also the result matrix C) from 0 to N-1, and the step length of this iteration variable is 1. i3 is the iteration variable of the inner loop, which means traversing the columns of matrix A and the rows of matrix B from 0 to P-1, and the step length of this iteration variable is 1.

[0054] The calculation logic of the data access mode (Affine Map) in the above example is: for each element C[i1][i2] in the matrix C, it is composed of the sum of the products of all rows (i1) of A and all columns (i2) of B. The inner loop calculates the product of A[i1][i3] and B[i3][i2], and then adds it to C[i1][i2]. This process is repeated P times to complete the product accumulation of all elements of the columns (i2) of the matrix B on the rows (i1) of the matrix A.

[0055] According to one embodiment of the present invention, see Figure 2 , which is a schematic diagram of the structure of converting an algorithm into a multi-memory level data flow graph. The algorithm represented in the figure is General Matrix Multiplication (GEMM), which is constructed as a multi-memory level data flow graph.

[0056] in, Figure 2gA, gB, gC, sA, sB, sC, rA, rB, and rC in the table are cache nodes, representing global memory A, global memory B, global memory C, shared memory A, shared memory B, shared memory C, register memory A, register memory B, and register memory C, respectively. Each cache node carries memory information, including memory hierarchy, memory data, memory shape, and size. GEMM represents a computing node that performs matrix multiplication operations.

[0057] Figure 2 The subgraph nodes in are represented by hexagons. Taking the memory divided into three levels as an example, the figure includes two nested subgraph nodes, which represent the memory access tasks between global memory and shared memory, and the memory access tasks between shared memory and register memory, respectively. Each nested subgraph node contains a data flow subgraph representing a higher level of memory, thus forming a multi-memory level data flow graph. For example, the subgraph node nested in the outermost layer contains a data flow subgraph representing the shared memory of the second level, and the subgraph node nested in the subgraph node contains a data flow subgraph of the register memory of the first level. The data flow subgraph corresponding to the first level includes cache nodes and computing nodes representing register memory A, register memory B, and register memory C, and each cache node is connected to the computing node through a directed connection edge to indicate direct data transmission.

[0058] Figure 2 In the example, the connection between two cache nodes representing different levels is an affine edge, that is, there are affine edges between cache nodes gA and sA, sA and rA, gB and sB, sB and rB, gC and sC, and sC and rC. When the arrow direction of the affine edge points to the cache node of the higher-level memory, it means loading data from the lower-level memory to the higher-level memory, such as the affine edges between gA and sA, sA and rA, gB and sB, and sB and rB. Conversely, it means storing high-level memory data into low-level memory, such as the affine edges between gC and sC, and sC and rC. Each affine edge carries a nested loop pattern and a data access pattern (Affine Map).

[0059] Figure 2 In the figure, the affine edge between gA and sA is given. In a memory access process, the data is accessed from the global memory A represented by gA through the data access mode (Affine Map). Affine Map is a data structure designed based on a polyhedron model. It consists of a matrix M, a loop nested vector i, and an offset vector o. The matrix M is a The vector i represents a vector of size D. The following is an example of Affine Map using a formula:

[0060] ,

[0061] in, Represents matrix M, which is used as affine mapping data. Represents a loop nested vector consisting of multiple iteration variables, , and is the iteration variable, Represents the offset vector. The Affine Map represents: multiply the first row vector of the matrix M by the loop nested vector and add it to the offset vector to get The first row of the array is obtained by multiplying the second row vector of the matrix M with the loop nested vector and adding it to the offset vector. The second row of the array.

[0062] The technical solution of the embodiment of the above step S1 can at least achieve the following beneficial technical effects: the algorithm described by the existing method does not contain hardware memory information and nested loop information of memory access, and cannot well describe those hardware-aware algorithms that can make full use of hardware information. The present invention constructs a multi-memory level data flow graph, and explicitly introduces hardware memory information and memory access information with nested loops in the graph. The memory access information can explicitly express how to load data from a low-level memory to a high-level memory or how to store data from a high-level memory to a low-level memory, so as to better map the memory access process to the hardware operation, so as to compile and describe those algorithms that can make full use of hardware information. In addition, the hardware memory level is fully considered, so that the present invention realizes the reasonable use of memory architecture information, thereby being able to help generate more efficient hardware operation codes later. The data flow graph of the present invention can better describe hardware-aware algorithms that take hardware characteristics into consideration, such as FlashAttention, FlashFFTConv, and Gated Linear Attention.

[0063] In step S2, each task node of the data flow graph is mapped to the target hardware to obtain a target hardware unit allocated to the operation task of each task node.

[0064] According to one embodiment of the present invention, Figure 2 Taking the constructed data flow graph as an example, each task node of the data flow graph is mapped to the target hardware to obtain the target hardware unit assigned to the operation task of each task node. Figure 3, which is a schematic diagram of the result of mapping each task node of the data flow graph to the target hardware. For example, when mapping on the target hardware NVIDIA GPU, the NVIDIA GPU has three memory levels: global memory, shared memory, and register memory. , and Indicates register memory, shared memory and global memory. Specific operation tasks are mapped to specific target hardware.

[0065] Among them, for memory access tasks, they are mapped to the corresponding memory for data loading or data storage, such as Figure 3 In the example, after cutting matrices A and B into blocks, the data of matrices A and B after cutting into blocks are transferred from the memory level to the The global memory is loaded into the memory hierarchy shared memory, and then from The shared memory is loaded into the memory hierarchy The register memory is used to perform the computing task based on the loaded data. The result data obtained during the computing task execution is the matrix C. After the matrix C is cut into pieces, the data after the matrix C is cut into pieces is transferred from the memory level The register memory is stored to Shared memory, and then from the memory level Shared memory is stored in the memory hierarchy The matrix multiplication calculation task is mapped to the tensor core (i.e., hardware Tensor Core) of the NVIDIA GPU. The tensor core performs matrix multiplication and accumulation operations to obtain the matrix C.

[0066] The technical solution of the embodiment of the above-mentioned step S2 can at least achieve the following beneficial technical effects: since each cache node corresponds to memories representing different memory levels, the hardware characteristics are fully considered to construct the data flow graph, and each cache node is directly mapped to the memory of the memory level it represents, and the memory access tasks between two cache nodes are mapped to the corresponding two layers of memory, which can realize effective and fast mapping and kernel fusion on the hardware.

[0067] In step S3, the partition strategy of the memory data of each cache node is adjusted with the optimization goal of maximizing the performance of all operation tasks, and the memory data partition result of each cache node is obtained.

[0068] According to one embodiment of the present invention, the blocking strategy includes blocking by shape and size. Step S3 includes step S31: adjusting the blocking shape and size of the memory data of each cache node according to the optimization target to obtain the final blocking shape and size of each cache node. Step S32: blocking the memory data of each cache node according to the final blocking shape and size of each cache node to obtain the memory data blocking results of each cache node, including multiple data blocks, the size and shape of the data blocks. The technical solution of this embodiment can at least achieve the following beneficial technical effects: in modern GPU architecture, blocking plays an extremely important role in the performance of algorithm execution. The present invention formulates a blocking strategy and adjusts the blocking strategy with the goal of optimizing the performance of the operation task, evaluates and selects the blocking strategy that is most conducive to the algorithm execution performance, and loads data based on the blocking, thereby generating a high-efficiency and high-performance CUDA code that instructs the kernel to perform calculations.

[0069] According to an embodiment of the present invention, in step S31, the final block shape and size of each cache node is obtained by the following steps S311, S312 and S313:

[0070] Step S311: according to a plurality of preset block shapes and sizes, the memory data of each cache node is divided into blocks respectively, and a plurality of memory data block results of each cache node are obtained.

[0071] According to one embodiment of the present invention, a variety of block shapes and sizes can be preset as alternatives for the block strategy according to the computing task. That is, if the computational amount of a single block / warp and the overall parallelism are considered, different block strategies are preset. For example, if it is a matrix with a small number of rows and a large number of columns, when dividing the blocks, the corresponding column dimension is divided larger, and the row dimension is divided smaller, forming rectangular blocks. If the number of rows and columns is the same, the division can use the same dimension to divide the row and column data to form square blocks. In addition, the block division will also consider the memory data layout. For example, if the dimension of the matrix multiplication performed by the Tensor Core is 16×8×16, the blocks are divided into multiples of the shape of 16×8×16 to facilitate data access, layout and calculation.

[0072] According to one embodiment of the present invention, in each preset block shape, multiple block sizes can be preset. For example, taking a square shape block as an example, the block size can be preset to 16×16, or the block size can be preset to 32×32. For example, taking a shape with dimensions of 16×8×16 as an example, the block size can be preset to 32×16×32, or the block size can be preset to 16×8×16. After presetting multiple block shapes and sizes, the memory data of each cache node is separately blocked, and multiple memory data blocking results of each cache node are obtained.

[0073] Step S312: Evaluate the performance of all operation tasks corresponding to each block shape and size under the preset multiple block shapes and sizes.

[0074] According to one embodiment of the present invention, the performance of all operation tasks can be determined in the following manner:

[0075] Based on a performance evaluation function preset by a plurality of performance indicators, the performance of all operation tasks under various preset block shapes and sizes is evaluated, wherein the plurality of performance indicators include the execution time, memory bandwidth utilization and cache hit rate of the operation task. The preset performance evaluation function performs weighted summation on the execution time, memory bandwidth utilization and cache hit rate of the operation task to obtain the evaluated performance, and the weight parameters of each indicator can be set according to the requirements. If higher execution efficiency is pursued, a higher weight parameter is set for the execution time of the operation task. Among them, the present invention is not limited to the selection of a plurality of performance indicators, and suitable evaluation indicators can be selected according to the algorithm requirements to measure the performance of different block strategies. That is, a plurality of performance indicators include: execution time, memory bandwidth utilization, cache hit rate, parallelism and power consumption or a combination thereof.

[0076] According to one embodiment of the present invention, the performance of all operation tasks may also be determined in the following manner:

[0077] The performance of all operation tasks is evaluated by calculating the cost model. The technical solution of this embodiment can at least achieve the following beneficial technical effects: by evaluating and obtaining reasonable blocks, memory access latency can be reduced, memory bandwidth utilization can be improved, and computing task execution efficiency can be improved.

[0078] Step S313: selecting the block shape and size of each cache node corresponding to the best performance of all operation tasks as the final block shape and size of each cache node.

[0079] According to an embodiment of the present invention, if the memory data is divided into blocks in a square shape and the block size is 16×16, the performance of all operation tasks can be the best, and the memory data is divided into blocks in a square shape of 16×16. It should be noted that the block strategies of memory data at different levels may be the same or different.

[0080] According to one embodiment of the present invention, see Figure 4 , which is a flow chart of loading data from shared memory into register memory and performing calculations for performance evaluation. The figure uses a block iterator to automatically evaluate different block shapes and sizes. For each block shape and size, the following process is used for test evaluation:

[0081] Based on the current block shape and size, the data in the shared memory and the data in the register memory are divided into blocks, and the multiple data blocks after the shared memory data is divided are loaded into the register memory. The register memory data is then divided into blocks and transferred to the kernel to perform the calculation task to obtain the calculation result data, and the calculation result data is stored in the shared memory. The test evaluates the performance of the complete process. Then, the process is run through multiple iterations and the block shape and size are adjusted to find the best block strategy, so as to obtain the final block shape and size of the shared memory data and the register memory data. This method is often used for deep learning and other applications that require a lot of parallel computing.

[0082] The technical solution of the embodiment of the above step S3 can at least achieve the following beneficial technical effects: reasonable block division can make the computing task better utilize the parallel computing capability of the GPU, reduce the synchronization overhead between threads, and improve the algorithm performance. For example, when multi-threaded matrix multiplication is processed, the matrix is ​​cut into sub-matrix blocks of appropriate size so that each thread can independently calculate the multiplication of a sub-matrix block, thereby improving parallelism. In addition, the present invention also realizes the optimization of the block division of the memory data of the cache node representing the register memory, and the optimization of the block division of the memory data of the cache node representing the shared memory, and the hardware range that can be optimized is wide.

[0083] In step S4, based on the data flow graph, the memory data blocking results of each cache node, the nested loop pattern and the data access pattern of each affine edge, a CUDA code is generated to instruct each target hardware unit to execute a corresponding operation task.

[0084] According to one embodiment of the present invention, when a cache node connected by an affine edge accesses each data block of another cache node connected by the affine edge, step S4 includes: determining the value range and step size of the iteration variable of the nested loop mode of the affine edge connecting the two cache nodes according to the shape and size of each data block in the memory data blocking result of the other cache node; generating a CUDA code indicating the memory access between the two cache nodes according to the nested loop mode, data access mode, and value range and step size of the iteration variable of the affine edge, so as to access the data block under the shape and size according to the logic of the nested loop mode and data access mode of the affine edge. Wherein, the logic of the CUDA code for executing memory access is the same as the logic of the nested loop mode and data access mode of the affine edge connecting the two cache nodes. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the data flow graph of the existing method has no access mode and nested loop mode, making it difficult for the traditional deep learning compiler to compile and express. However, after the nested loop information and access mode are introduced in the present invention, the code with the same logic as its nested loop mode and data access mode is directly generated according to the information and the final blocking strategy, thereby improving the code generation efficiency.

[0085] According to an embodiment of the present invention, taking access to a matrix as an example, a method for determining the value range and step size of the iteration variable of the nested loop pattern of the affine edge connecting two cache nodes is described. If a 4×4 dimensional matrix is ​​stored in the memory, and the memory data block result is obtained by block division into 2×2 dimensions according to the shape and size of each data block, then the matrix is ​​divided into 4 sub-matrices of 2×2 dimensions. When accessing the 4 sub-matrices, the nested loop pattern is organized into four nested loops, namely the row dimension of the outer loop, the column dimension of the outer loop, the row dimension of the inner loop, and the column dimension of the inner loop. Each dimension of each loop corresponds to an iteration variable. The outer loop represents access based on the sub-matrix, and the memory loop represents access based on each element of the sub-matrix. The value range of the iteration variable corresponding to the row dimension and column dimension of the outer loop is 0 to 1, and the step size is 2; the value range of the iteration variable corresponding to the row dimension and column dimension of the inner loop is 0 to 1, and the step size is 1, so as to access each sub-matrix in turn according to the shape and size of each sub-matrix.

[0086] According to an embodiment of the present invention, step S4 further includes: generating CUDA codes for instructing each target hardware unit to perform corresponding operation tasks according to the order of operation tasks represented by each task node in the data flow graph, including instructions for instructing the target memory unit to perform data loading tasks and / or instructions for performing data storage tasks, and instructions for instructing the target computing unit to perform computing tasks. The CUDA codes for each operation task can be generated sequentially according to the order of order by topologically sorting the subgraph nodes and computing nodes, and then sorting the operation tasks represented by the topologically sorted subgraph nodes and computing nodes in order.

[0087] According to one embodiment of the present invention, S4 also includes: when executing a computing task, according to the shape and size of the data blocks in the memory data blocking result of the cache node representing the register memory, generating a CUDA code instructing the register memory represented by the cache node to perform data transfer between the target computing unit, so as to read or write the data block in the register memory according to the shape and size of each data block.

[0088] According to one embodiment of the present invention, a compilation and running method for a hardware-aware algorithm is provided, comprising: obtaining a hardware-aware algorithm, generating CUDA code for a target device based on the code generation method of the above embodiment, instructing the target hardware device to execute the hardware-aware algorithm; deploying the CUDA code to the target hardware device for execution to complete the operation task of the hardware-aware algorithm.

[0089] According to one embodiment of the present invention, see Figure 5 , which is a flowchart of the compilation and running method for the hardware-aware algorithm. First, obtain the hardware-aware algorithm. Secondly, use the code generation method of the above embodiment to generate CUDA code that instructs the target hardware device to execute the hardware-aware algorithm. The CUDA code generation process includes Ⅰ: constructing a multi-memory level data flow graph; Ⅱ: mapping each operation task of the data flow graph to the target hardware; Ⅲ: blocking the memory data at each level to obtain each memory data blocking result; Ⅳ: based on each memory data blocking result and the data flow graph, generate code that instructs the target hardware to perform the operation. Finally, deploy the code to the target hardware device for execution, such as deploying it on the target hardware NVIDIA GPU or deploying it on the AMD GPU.

[0090] In order to verify the beneficial effects of the present invention, the inventors conducted the following comparative experiments:

[0091] First, the inventor selected two existing methods and the method of the present invention for performance evaluation. The two existing methods are existing method 1: Triton compiler and existing method 2: CuBLAS; Triton is a compiler based on the GPU programming model designed by OpenAI, and CuBLAS is a hardware library of NVIDIA. Compared with many other existing methods, CuBLAS has outstanding performance in general matrix multiplication.

[0092] Secondly, the method of the present invention and the existing method are tested on two algorithms: half GEMM (HGEMM) and fused two HGEMMs, where Fused Two HGEMMs is the core operation of FlashAttention-2.

[0093] Finally, three matrices A, B, and C are obtained for matrix multiplication, where the dimensions of the A matrix are M×K, the dimensions of the B matrix are K×N, and the dimensions of the C matrix are N×P. The test process is as follows:

[0094] 1) The method of the present invention and the existing method are tested under the HGEMM algorithm. The code generated by the method of the present invention on NVIDIA A100 80G. Get A and B in different dimensions, calculate A×B in each dimension, see Figure 6 , which is a schematic diagram of the performance evaluation comparison results of the method of the present invention and the existing method on the HGEMM algorithm. In the figure, the ordinate represents the average execution time, the abscissa represents the matrix dimension size of A and B respectively, and the corresponding dimension values ​​of [M, N, K] are [1024, 128, 64], [2048, 128, 64], [4096, 128, 64], and [8192, 128, 64] respectively. It can be seen from the figure that when the corresponding dimension values ​​of [M, N, K] are [1024, 128, 64] and [2048, 128, 64] respectively, the code generated by the method of the present invention after compiling the algorithm has the shortest average execution time. Under the dimensions of [4096, 128, 64] and [8192, 128, 64], the method of the present invention still achieves the same high efficiency as the existing method 2 which is outstanding in general matrix multiplication.

[0095] 2) The method of the present invention and the existing method are tested under the Fused Two HGEMMs algorithm. The code generated by the method of the present invention on NVIDIA A100 80G. Get A, B and C in different dimensions, calculate (A×B)×C in each dimension, see Figure 7, which is a schematic diagram of the performance evaluation comparison results of the method of the present invention and the existing method on the Fused Two HGEMMs algorithm. In the figure, the ordinate represents the average execution time, the abscissa represents the matrix dimension size of A and B respectively, and the corresponding dimension values ​​of [M, N, K, P] are [2048, 128, 64, 64], [4096, 128, 64, 64], [8192, 128, 64, 64], [2048, 256, 64, 64], [4096, 256, 64, 64], [8192, 256, 64, 64]. It can be seen from the figure that the code generated by the method of the present invention after compiling the algorithm has the shortest average execution time.

[0096] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0097] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0098] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0099] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A CUDA code generation method based on data flow analysis, characterized in that: include: S1. Obtain a hardware-aware algorithm, and based on the architecture information of multi-layer memory in the target hardware, convert the algorithm into a multi-memory hierarchical data flow graph, which includes multiple task nodes representing different operation tasks in the algorithm, multiple affine edges, and multiple cache nodes representing memory. The cache node includes the memory hierarchy and memory data it represents, and the affine edge includes the memory access information between the two cache nodes representing adjacent hierarchical memories connected by it, including a nested loop pattern describing the data loop traversal process between the memories represented by the two cache nodes connected by the affine edge and a data access pattern describing the data mapping process between the memories represented by the two cache nodes connected by the affine edge; S2, mapping each task node of the data flow graph to the target hardware, and obtaining the target hardware unit allocated to the operation task of each task node; S3, adjusting the memory data partitioning strategy of each cache node with the optimization goal of maximizing the performance of all operation tasks, and obtaining the memory data partitioning result of each cache node; S4. Based on the data flow graph, the memory data blocking results of each cache node, the nested loop pattern and data access pattern of each affine edge, generate a CUDA code that instructs each target hardware unit to perform a corresponding operation task.

2. The method according to claim 1, characterized in that The block strategy includes block division by shape and size, and S3 includes: Adjusting the block shape and size of the memory data of each cache node according to the optimization target to obtain the final block shape and size of each cache node; The memory data block results of each cache node are obtained according to the final block shape and size of each cache node, including multiple data blocks, the size and shape of the data blocks.

3. The method according to claim 2, characterized in that The nested loop mode includes a plurality of iteration variables, and when a cache node connected by an affine edge accesses each data block of another cache node connected by the affine edge, S4 further includes: Determine the value range and step length of the iteration variable of the nested loop pattern of the affine edge according to the shape and size of each data block in the memory data block result of another cache node; According to the nested loop pattern of the affine edge, the data access pattern, the value range and the step size, a CUDA code is generated to instruct memory access to be performed between the memories represented by the two cache nodes, so as to access the data block of the shape and size according to the logic of the nested loop pattern and the data access pattern of the affine edge.

4. The method according to claim 2, characterized in that: The final block shape and size are obtained by: According to the preset multiple block shapes and sizes, the memory data of each cache node is divided into blocks respectively, and multiple memory data block results of each cache node are obtained; Evaluate the performance of all manipulation tasks for each block shape and size under various preset block shapes and sizes; The block shape and size of each cache node corresponding to the best performance of all operation tasks are selected as the final block shape and size of each cache node.

5. The method according to claim 4, characterized in that All operation tasks include memory access tasks and computing tasks, memory access tasks include data loading tasks and / or data storage tasks, and multiple task nodes include computing nodes representing computing tasks and subgraph nodes representing memory access tasks, wherein S4 further includes: The CUDA code is generated according to the sequence of the operation tasks represented by each task node in the data flow graph, including instructions for executing data loading tasks, instructions for executing data storage tasks, and instructions for executing computing tasks.

6. The method according to claim 5, characterized in that The multi-layer memory includes a register memory, a shared memory and a global memory, the plurality of cache nodes include a cache node representing the register memory, a cache node representing the shared memory and a cache node representing the global memory, the data flow graph further includes a plurality of directed connection edges, each directed connection edge is used to represent data transmission between the cache node representing the register memory and the computing node connected thereto, and the S4 further includes: When executing a computing task, a CUDA code is generated according to the shape and size of a data block in a memory data blocking result of a cache node representing a register memory to instruct execution of data transmission between the register memory represented by the cache node and a target computing unit.

7. The method according to claim 4, characterized in that The performance of all operational tasks is determined by: A preset performance evaluation function based on multiple performance indicators is used to evaluate the performance of all operation tasks under various preset block shapes and sizes. The multiple performance indicators include the execution time of the operation tasks, memory bandwidth utilization, and cache hit rate.

8. A compiling and running method for a hardware-aware algorithm, characterized in that: include: Obtain a hardware-aware algorithm, and generate a CUDA code for instructing a target hardware device to execute the hardware-aware algorithm based on the method according to any one of claims 1 to 7; The CUDA code is deployed to the target hardware device for execution to complete the operation task of the hardware-aware algorithm.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1-8 by executing the executable instructions.

Images