Data processing method, electronic device, storage medium, and program product

Abstract

The application relates to the technical field of artificial intelligence chips, and provides a data processing method, electronic equipment, a storage medium and a program product, wherein the method comprises the following steps: acquiring input data stored in a first data format; converting the input data from the first data format into a second data format in the process of loading the input data into a main cache, the second data format being a hardware special data format supported by a tensor core; performing a matrix multiplication operation based on the tensor core and using the data stored in the second data format in the main cache to generate result data; converting the result data from the second data format into the first data format, and outputting the converted result data. By introducing an automatic format conversion mechanism in a key link of data flow conversion, the application successfully opens up a channel between an upper-layer application and a bottom-layer tensor core hardware, realizes effective utilization of the tensor core without sacrificing ease of use, and brings about significant performance improvement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence chips, and in particular to a data processing method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] In the field of artificial intelligence and high-performance computing, matrix multiply-accumulate (MMA for short) is a core computing task. In order to efficiently perform such computing tasks, modern high-performance computing devices such as graphics processing units (GPUs), general-purpose computing on graphics processing units (GPGPUs), tensor processing units (TPUs), etc. usually integrate hardware acceleration units designed specifically for matrix operations, such as tensor cores, to provide powerful computing acceleration capabilities.

[0003] The Triton framework is a popular framework for simplifying the programming of high-performance computing devices (such as GPUs), which enables developers to easily implement operations such as MMA. However, through research, it has been found that the existing matrix multiplication implementation scheme based on the Triton framework is by default executed by general-purpose vector cores (Vector Core), but its efficiency in performing matrix operations is much lower than that of dedicated tensor cores. Due to the inability to effectively utilize the specialized computing capabilities of tensor cores and the limitations of data format compatibility issues, the computing performance cannot reach the theoretical peak of the hardware, making it difficult to meet the growing demand for high-performance computing. Therefore, how to enable the Triton framework to fully utilize the hardware capabilities of tensor cores is a problem that needs to be solved. SUMMARY

[0004] The present application provides a data processing method, an electronic device, a storage medium and a program product to solve the defect that the matrix multiplication calculation based on the Triton framework in the related art cannot utilize the specialized computing capabilities of tensor cores, resulting in poor computing performance.

[0005] The present application provides a data processing method, comprising:

[0006] obtaining input data stored in a first data format;

[0007] In the process of loading the input data into the main cache, the input data is converted from the first data format to a second data format, and the second data format is a hardware-specific data format supported by the tensor core;

[0008] Based on the tensor core, a matrix multiplication operation is performed using data stored in the main cache in the second data format, to generate result data;

[0009] The result data is converted from the second data format to the first data format, and the converted result data is output.

[0010] According to the data processing method provided by the application, the input data is converted from the first data format to a second data format during the process of loading the input data into a main cache, which comprises:

[0011] A block pointer corresponding to the input data is calculated, and the input data is loaded from a global memory to a thread local register based on the block pointer, wherein the input data is stored in the first data format in both the global memory and the thread local register;

[0012] The data stored in the thread local register is converted from the first data format to the second data format, and the converted data is transferred to the main cache.

[0013] According to the data processing method provided by the application, the step of converting the data stored in the thread local register from the first data format to the second data format and transferring the converted data to the main cache is realized by inserting a first layout conversion operation and a second layout conversion operation in the compilation conversion process of performing the loading operation on the input data, wherein the first layout conversion operation is used for format conversion, and the second layout conversion operation is used for data transfer.

[0014] According to the data processing method provided by the application, the step of performing the matrix multiplication operation comprises:

[0015] The data in the main cache is loaded to an input cache of the tensor core through a first special instruction;

[0016] The matrix multiplication accumulation calculation is performed using the data in the input cache through a second special instruction, to generate the result data.

[0017] According to the data processing method provided by the application, the step of converting the result data from the second data format to the first data format comprises:

[0018] If the result data is stored in the main cache, the result data is transferred from the main cache to a thread local register;

[0019] The result data in the thread local register is converted from the second data format to the first data format.

[0020] According to the data processing method provided by the application, the step of carrying the result data from the main cache to the thread local register and converting the result data from the second data format to the first data format is implemented by inserting a third layout conversion operation and a fourth layout conversion operation in the compilation conversion process of performing the storage operation on the result data, the third layout conversion operation is used for data carrying, and the fourth layout conversion operation is used for format conversion.

[0021] According to the data processing method provided by the application, the input data stored in the first data format is obtained, which comprises:

[0022] According to the shape of the input matrix and the hardware characteristics of the computing device, the input matrix is split to obtain a plurality of matrix subblocks, and the plurality of matrix subblocks are used as the input data.

[0023] The application further provides a data processing device, which comprises:

[0024] The obtaining unit is used for obtaining input data stored in a first data format;

[0025] The converting unit is used for converting the input data from the first data format to a second data format in the process of loading the input data to the main cache, and the second data format is a hardware-specific data format supported by a tensor core;

[0026] The operation unit is used for performing a matrix multiplication operation based on the tensor core and using data stored in the second data format in the main cache to generate result data;

[0027] The output unit is used for converting the result data from the second data format to the first data format and outputting the converted result data.

[0028] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and running on the processor, and the processor implements the data processing method of any of the above when executing the computer program.

[0029] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data processing method of any of the above.

[0030] The application further provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the data processing method of any of the above.

[0031] The data processing method, the electronic device, the storage medium and the program product provided by the application, by converting the data from the general first data format to the hardware-specific second data format supported by the tensor core in the process of loading the input data to the main cache, ingeniously solves the fundamental problem that the tensor core cannot be used due to data format incompatibility. This key step enables the subsequent calculation to be smoothly executed based on the tensor core. Since the calculation throughput of the tensor core is much higher than that of the vector core, the application can fully utilize the peak computing power of the hardware, thereby greatly improving the speed and efficiency of matrix multiplication operation. In addition, the application seamlessly integrates the complex format conversion operation in the data loading and data output process. For the upper application or developer, the input data they process and the final result data they obtain are always in the standard first data format. The whole process of format conversion is transparent to the user, and the developer does not need to manually perform complex data format management and adaptation work. This realizes hardware acceleration, obtains great performance gain, and does not increase the complexity of programming, maintains good compatibility and ease of use with existing programming frameworks, and solves the contradiction between high performance and development efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0033] Figure 1 is a structural schematic diagram of the computing device provided by the application;

[0034] Figure 2 is a flowchart of the data processing method provided by the application;

[0035] Figure 3 is a flowchart of the Triton framework compilation conversion provided by the application;

[0036] Figure 4 is a data flow graph for performing matrix multiplication operation by using the tensor core provided by the application;

[0037] Figure 5 is a flowchart of the Triton framework matrix multiplication acceleration method based on the tensor core provided by the application;

[0038] Figure 6 is a structural schematic diagram of the data processing device provided by the application;

[0039] Figure 7FIG. 1 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0041] In the field of artificial intelligence, deep learning and other high-performance computing, matrix multiplication accumulation (MMA) operation is one of the most core and computationally intensive operations. The training and inference processes of neural networks, such as the computation of convolutional layers and fully connected layers, essentially rely on a large number of matrix multiplication operations. In order to efficiently perform such computing tasks, high-performance computing devices such as GPUs and TPUs usually integrate hardware acceleration units designed specifically for matrix operations, such as Tensor Cores.

[0042] Triton is a cutting-edge open-source programming language and compiler framework that aims to enable developers to conveniently write high-performance GPU computing kernel functions (Kernel) in a Python-like syntax, thereby lowering the threshold for GPU programming. In the Triton compiler framework (hereinafter referred to as “Triton framework”), developers usually use the tl.dot operation provided by Triton to implement matrix multiplication. A typical Triton framework matrix multiplication implementation process is as follows: first, the large matrix to be calculated is divided into multiple smaller sub-blocks through the Tiling strategy; then, the sub-blocks of the input matrix are loaded from the Global Memory to the Thread Local Register (TLR) through the tl.load instruction in the loop; then, the tl.dot operation is called to perform matrix multiplication on the loaded sub-blocks and accumulate the results; finally, the final calculation results are written back to the Global Memory through the tl.store instruction.

[0043] However, in practical applications, it is found that the above-mentioned matrix multiplication implementation scheme based on the Triton framework has a significant performance bottleneck. First, the computing resource utilization is insufficient. In a conventional implementation, the tl.dot operation of the Triton framework is usually converted into using a general vector core to perform matrix multiplication operation after compilation. Although the vector core is general, its efficiency of performing matrix operation is much lower than that of a dedicated tensor core. The computing throughput of the tensor core is usually several times or even dozens of times that of the vector core. Therefore, relying only on the vector core for calculation causes the powerful tensor core computing resources in the GPU to be idle, and the peak computing capability of the hardware cannot be fully utilized, thereby causing a significant performance bottleneck.

[0044] Secondly, the data format is incompatible, which causes the dedicated instruction to be unable to be used. In order to achieve the ultimate memory access efficiency and computing performance, the hardware dedicated MMA instruction relied on by the tensor core usually requires the input data to comply with a special optimized hardware memory layout (for example, the memory layout of the usharp format). However, in the standard Triton framework programming model, the input matrix data usually exists in an unoptimized general memory format (for example, plain buffer) when loaded through the tl.load instruction. This mismatch in data format directly hinders the compiler from generating and calling the efficient hardware dedicated instruction for the tensor core, so that the Triton framework cannot fundamentally utilize the performance advantage brought by the tensor core. It should be understood that the plain buffer and the usharp describe two different storage layouts of matrix data in the GPU memory, wherein the plain buffer refers to a format in which the data is stored in a linear and continuous manner in the memory, such as row-major or column-major storage; the usharp is a hardware-specific, optimized non-linear memory layout, which is a data rearrangement format designed for the tensor core of a specific GPU architecture. In this format, the data is no longer simply stored continuously by row or column, but the elements within a matrix block are rearranged and combined so that the data elements that need to be loaded together by the tensor core are adjacent to each other.

[0045] To this end, the present application provides a data processing method. By introducing an automatic format conversion mechanism at a key link in the data flow, and seamlessly integrating the format conversion operation in the data loading and data output process, not only the channel between the upper layer application and the bottom layer tensor core hardware is successfully opened, thereby realizing the full utilization of the computing resources of the tensor core and significantly improving the performance of the matrix multiplication operation, but also the good compatibility and ease of use with the existing programming framework (such as the Triton framework) are maintained, so that the user can use without making a large number of modifications to the existing code, thereby overcoming the above-mentioned defects.

[0046] It can be understood that the upper application is a relative concept, which refers to any code, program or system that directly calls the data processing method described in the application without needing to care about the internal implementation details. For example, the upper application can be a Triton kernel function written by a developer. When a developer writes a matrix multiplication program using the Triton language, they only need to write according to the standard syntax of the Triton language, such as calling tl.load to load data, tl.dot to perform calculation, and tl.store to save the result. In this scenario, the Triton kernel function written by the developer is the upper application. They expect to input a matrix in a normal format (i.e., the first data format) and obtain a result matrix in the same format. The technical solution of the application automatically completes the whole process of converting from the first data format to the second data format, calling tensor core calculation, and converting back to the first data format in the underlying (i.e., the process of compiling and running the Triton framework). For the developer who writes the upper application, this complex optimization process is transparent and unnoticeable, and they can enjoy the performance improvement without modifying the code.

[0047] For another example, the upper application can also be a larger program or framework that calls the Triton kernel function, such as a large artificial intelligence model, a deep learning framework (such as PyTorch, TensorFlow), or a specific business program (such as an image recognition service or a natural language processing system). These large programs can call a high-performance operator written in the Triton language to complete a certain calculation task (such as matrix multiplication) when executing. In this scenario, the artificial intelligence model or business program is the upper application, which is only responsible for passing data to the Triton operator and receiving the result, and does not need to know that the underlying is a GPU executing, and does not need to care about how the data inside the GPU is converted to adapt to the tensor core.

[0048] The application encapsulates the complex underlying optimization (i.e., format conversion) made for extreme performance, providing a stable, simple and efficient interface for the upper application, thereby perfectly balancing high performance and ease of use.

[0049] Figure 1 is a structural schematic diagram of a computing device provided by the application, as Figure 1 The execution subject of the data processing method provided by the application can be a computing device 100, which at least includes a plurality of processing units 101 and a video memory 102. Each processing unit 101 can be a Streaming Processor Cluster (SPC), which can include a tensor calculation unit 103, a vector calculation unit 104, an on-chip cache 105 and a register 106.

[0050] Tensor computation unit 103 and vector computation unit 104 are heterogeneous units. Tensor computation unit 103 is mainly used to perform tensor computations (such as matrix multiplication, convolution, etc.). Tensor computation unit 103 includes multiple tensor cores for performing specific computational tasks. Vector computation unit 104 is mainly used to perform arithmetic and logical operations, such as accumulation, reduction, and regular addition, subtraction, multiplication, and division. Vector computation unit 104 includes multiple vector cores for performing specific computational tasks.

[0051] The video memory 102 can be high-bandwidth memory (HBM) or other types of memory. The on-chip cache 105 is a temporary memory with a smaller capacity than the video memory 102, but a faster data exchange speed. The on-chip cache 105 includes a cache shared by the tensor computation unit 103 and the vector computation unit 104, as well as a cache exclusively used by either the tensor computation unit 103 or the vector computation unit 104. The on-chip cache 105 may include a general-purpose matrix main buffer (GEMM Main Buffer, GMB) and a general-purpose matrix input buffer (GEMM Input Buffer, GIB) for the tensor computation unit 103.

[0052] Register 106 includes registers shared by tensor computation unit 103 and vector computation unit 104, as well as registers exclusive to either tensor computation unit 103 or vector computation unit 104. For example, register 106 may include thread-local registers (TLRs). Tensor computation unit 103 and vector computation unit 104 can exchange data through register 106. Compared to on-chip cache 105, register 106 has a smaller capacity but a faster data exchange speed.

[0053] It should be noted that the computing device 100 of this invention may include other structures besides those described above, and this invention does not specifically limit these structures. The computing device 100 may be a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processor (TPU), etc.

[0054] based on Figure 1 The diagram shows the architecture of a computing device. This embodiment of the invention provides a data processing method that can be applied to, for example... Figure 1The illustrated computing device, in particular, utilizes the tensor cores in the tensor computing unit 103 contained therein to accelerate matrix multiplication operations. Specifically, the method can be applied to various scenarios such as text processing, image processing, speech processing, etc. In different application scenarios, the input data has different physical meanings. For example, in the text processing scenario, the input data can be text data related to text generation, text recognition, etc. For another example, in the image processing scenario, the input data can be image data used in image preprocessing, image segmentation, target detection, etc. For another example, in the speech processing scenario, the input data can be speech data used in speech recognition, speech synthesis, etc. The specific flow of the data processing method provided by the present application is introduced below.

[0055] Based on the above embodiment, Figure 2 is a flowchart of the data processing method provided by the present application, as Figure 2 shown, the method comprises:

[0056] Step S10, obtaining input data stored in a first data format.

[0057] Specifically, the input data is usually two input matrices for performing matrix multiplication (such as AxB=C), for example, matrix A and matrix B, or can be submatrices of at least two input matrices. The input data is usually stored in a global memory (such as the video memory 102 shown in Figure 1 Therefore, the process of obtaining the input data can specifically mean that when a Triton kernel function is executed, the function receives pointers to the input matrices A and B stored in the global memory. At this time, the matrix data in the global memory is organized in the first data format (such as plain buffer).

[0058] It can be understood that the first data format refers to a general, non-specific hardware optimized data storage format, such as plain buffer. The data in this format can be stored continuously in the memory in a standard manner such as row-major order or column-major order, which is convenient for general computing units (such as vector cores) in computing devices (such as GPUs) to access and process. However, this general format is not optimized for the memory access mode of specific hardware acceleration units (such as tensor cores), so the special hardware instructions of the tensor cores cannot use data in this general format. In order to be able to utilize the powerful computing power of the tensor cores to perform matrix multiplication operations, it is necessary to convert the input data in the first data format.

[0059] Step S20, in the process of loading the input data into the main cache, the input data is converted from the first data format to a second data format, which is a hardware-specific data format supported by the tensor core.

[0060] It should be noted that in order to enable the Triton framework to fully utilize the computing resources of the tensor core to perform matrix multiplication operations, it is necessary to use the supported special MMA instructions. However, there are some constraints on the special MMA instructions of the tensor core. For example, the tensor core has its own storage unit (such as the main cache), so it is necessary to first load the input data into the main cache and then read the data from the main cache for calculation.

[0061] Specifically, the main cache is an on-chip cache with smaller capacity than the global memory (such as HBM) but much faster access speed, which is used to temporarily store data to be used by the computing unit to reduce memory access delay. For example, the main cache can be a general matrix main cache (GMB), which is a part of the on-chip cache 105 as shown in the figure. Figure 1

[0062] In addition to the above constraints on the memory access path, the special MMA instructions of the tensor core also have certain constraints on the layout of the data in the memory, that is, the tensor core cannot use the general first data format (such as plain buffer). Therefore, in the process of loading the input data into the main cache, the input data is converted from the first data format to a hardware-specific data format supported by the tensor core, that is, the second data format, so that the tensor core can correctly read the data for MMA calculation.

[0063] Here, the second data format refers to a special data layout designed to adapt to a specific hardware (i.e. the tensor core). This format rearranges the data to match the needs of the tensor core for parallel loading and calculation, thereby maximizing memory bandwidth utilization and computational efficiency. It should be understood that the second data format in the embodiments of the present application can be the usharp format.

[0064] ​It can be understood that the embodiment of the present application performs data format conversion in the process of data loading, which means that the data format conversion is deeply coupled with the data loading process, rather than an independent and serial step. In other words, the system does not first load the data in the plain buffer format to an intermediate position, and then start a new task to convert it into the usharp format; instead, when executing a high-level loading instruction (such as tl.load in the Triton framework), the compiler automatically and transparently rearranges the data stream in the plain buffer format read from the global memory during the lowering (also known as compilation conversion) into underlying hardware instructions, and finally writes the rearranged data in the usharp format into the GMB. To the upper-layer users, they only perceive one loading operation, and the complex format conversion and data transfer are hidden from them.

[0065] In step S30, based on the tensor core, a matrix multiplication operation is performed using the data stored in the main cache in the second data format, to generate result data.

[0066] Specifically, after the input data is prepared in the efficient second data format (usharp) in the main cache (GMB), this step will call the core computing resources of the hardware, i.e., the tensor core, to perform the core matrix multiplication accumulation (MMA) operation. Here, the tensor core is a hardware acceleration unit specially designed for performing large-scale matrix operations, and its throughput far exceeds that of general-purpose computing units (such as vector cores).

[0067] Performing the MMA operation based on the tensor core means that the computing task will be scheduled to be executed on these special cores. The specific execution process is that the tensor core efficiently reads the data stored in the usharp format from the GMB through a special hardware instruction. Since the data format has been optimized, the tensor core can load an entire block of data required at a time, achieving highly parallel computation. By executing the matrix multiplication accumulation instruction, the tensor core performs operations on the input matrix sub-blocks and accumulates the results. This process continues until all relevant matrix sub-blocks are calculated.

[0068] The result data generated after the operation, such as the corresponding sub-block data of the matrix C, is stored in a temporary location. According to the design of the hardware instruction, this location may be an accumulator register inside the tensor core, or it may be directly written back to the GMB. The result data at this time also maintains the second data format (i.e., usharp) that is adapted to the computing process.

[0069] In step S40, the result data is converted from the second data format to the first data format, and the converted result data is output.

[0070] Specifically, after the result data is generated, it needs to be converted from the second data format to the first data format, which is a reverse conversion process. It rearranges the result data in usharp format stored in a temporary location (such as GMB) back to the general plain buffer format. It should be understood that the reason for this reverse conversion is that, although the usharp format is efficient for tensor cores, other parts of the system or subsequent calculation steps that do not use tensor cores generally expect to receive data in the standard, general plain buffer format. Therefore, this conversion ensures that the processing method of the application can be seamlessly embedded into a wider calculation graph without compromising the generality of the data.

[0071] Specifically, similar to step S20, this reverse conversion process can also be seamlessly integrated into the compilation conversion process of the advanced storage instruction (such as tl.store in the Triton framework). When the user calls tl.store to write the result back to global memory, the compiler automatically inserts the corresponding conversion logic and finally outputs the converted result data. Here, the output of the converted result data refers to the result data that has been converted to the first data format (plain buffer) and finally written back to the corresponding position of the output matrix C in the global memory specified by the user. At this point, the user obtains a correct result in the global memory that is consistent with the input format but has been significantly accelerated in the calculation process.

[0072] The method provided by the embodiment of the application ingeniously solves the fundamental problem of being unable to use tensor cores due to incompatible data formats by converting the data from the general first data format to the hardware-specific second data format supported by the tensor cores during the process of loading the input data into the main cache. This key step enables the subsequent calculation to be successfully performed based on the tensor cores. Since the calculation throughput of the tensor cores is much higher than that of the vector cores, the application can fully utilize the peak computing capacity of the hardware, thereby greatly improving the speed and efficiency of matrix multiplication. In addition, the application seamlessly integrates the complex format conversion operation into the data loading and data output process. For the upper-layer application or developer, the input data they handle and the result data they finally obtain are always in the standard first data format. The entire format conversion process is transparent to the user, and the developer does not need to manually perform complex data format management and adaptation work. This achieves hardware acceleration and obtains great performance gain without increasing the complexity of programming, maintains good compatibility and ease of use with existing programming frameworks, and solves the contradiction between high performance and development efficiency.

[0073] Based on any of the above embodiments, step S10 specifically includes:

[0074] According to the shape of the input matrix and the hardware characteristics of the computing device, the input matrix is split to obtain a plurality of matrix subblocks, and the plurality of matrix subblocks are taken as the input data.

[0075] It should be noted that the embodiment of the present application illustrates how to effectively organize and manage input data through the block strategy when processing large-scale matrix multiplication, laying the foundation for subsequent efficient processing.

[0076] Specifically, in actual application, the size of the input matrix to be calculated (such as matrix A and matrix B) can be very large, far exceeding the capacity of the cache (such as GMB) or even the thread local register (TLR) on the computing device. It is unrealistic to load the entire matrix into high-speed storage at one time. Therefore, a block technology can be used to implement the processing of large-scale matrix multiplication tasks.

[0077] It can be understood that the core of this step is data blocking. The purpose is to decompose a large-scale computing task into a series of small-scale computing tasks that are manageable on small data blocks. This can significantly improve data locality, so that data can be loaded into on-chip cache with fast access speed and reused, thereby minimizing the number of accesses to global memory with slow access speed, which is the key to high-performance computing.

[0078] The splitting of the input matrix is not arbitrary, but needs to consider the following two core factors, namely the shape of the input matrix and the hardware characteristics of the computing device (such as GPU). The shape of the input matrix A and B (for example, the shape of matrix A is M × K, and the shape of matrix B is K × N) directly affects the splitting strategy, and different shapes of matrices should use appropriate splitting strategies to balance the computing load and memory access.

[0079] The hardware characteristics of the computing device are the key to determining the splitting granularity (i.e. the size of the subblock). The hardware characteristics to be considered include the computing granularity of the tensor core, the size of the main cache (GMB), the capacity of the thread local register (TLR), the memory bandwidth and delay, etc. Specifically, the tensor core usually operates in fixed-size blocks (such as 64 × 64 × 64). Therefore, the size of the subblock (BLOCK_SIZE_M, BLOCK_SIZE_N, BLOCK_SIZE_K) should preferably be an integer multiple of the tensor core computing granularity to ensure that the hardware computing unit is fully utilized. Here, BLOCK_SIZE_M, BLOCK_SIZE_N, BLOCK_SIZE_K respectively refer to the size of the subblock in M, N, K dimensions.

[0080] The split matrix sub-blocks (such as sub_A and sub_B) must be able to be completely loaded into the GMB. If the sub-blocks are too large, the GMB will not be able to accommodate them, resulting in repeated data exchange between the global memory and the GMB, which seriously affects performance. Similarly, the data is temporarily stored in the TLR before being loaded into the GMB, and the capacity of the TLR also limits the amount of data processed at a time. Therefore, when selecting a suitable splitting strategy, the size of the GMB and the capacity of the TLR, etc. need to be considered comprehensively. In addition, a reasonable block size can enable memory access merging and maximize effective bandwidth.

[0081] By comprehensively considering the above factors, the optimal block sizes BLOCK_SIZE_M, BLOCK_SIZE_N and BLOCK_SIZE_K can be determined. Based on this, the original large matrices A and B are logically divided into multiple matrix sub-blocks. For example, the matrix A is divided into multiple BLOCK_SIZE_M×BLOCK_SIZE_K sub-blocks, and the matrix B is divided into multiple BLOCK_SIZE_K×BLOCK_SIZE_N sub-blocks.

[0082] During the entire matrix multiplication calculation process, the program iteratively processes these sub-blocks through a loop. In each iteration, a pair of corresponding matrix sub-blocks (such as sub_A and sub_B) is selected, which becomes the input data in the current iteration round. All subsequent steps, including data loading, format conversion, data transfer, and performing MMA operations, are performed on this pair of matrix sub-blocks.

[0083] The method provided by the embodiment of the application provides a basis for the entire high-performance data processing process by introducing a strategy of intelligent splitting based on matrix shape and hardware characteristics. This splitting method not only solves the physical limitation that a large matrix cannot be loaded into the cache at a time, but more importantly, it fundamentally reduces the dependence on slow global memory by optimizing data locality and implementing data reuse in the cache, greatly improving memory access efficiency.

[0084] Based on any of the above embodiments, step S20 specifically includes:

[0085] Step S21, calculate the block pointer corresponding to the input data, and load the input data from the global memory to the thread local register based on the block pointer, wherein the input data is stored in the first data format in the global memory and the thread local register.

[0086] Specifically, to achieve efficient parallel computation, a block algorithm is usually adopted. Before performing matrix multiplication, a reasonable splitting strategy needs to be selected according to the shape of input matrices A, B and the hardware characteristics (e.g. cache size, supported computation granularity of tensor core, etc.) of the computing device (such as GPU) to logically divide the large input matrices A and B into multiple smaller matrix sub-blocks. Subsequently, for each matrix sub-block (i.e. input data) to be processed, the starting address of the matrix sub-block in the global memory, i.e. the block pointer, is calculated. The block pointer is usually obtained by adding the base address of the input matrix in the global memory to the offset calculated according to the sub-block index.

[0087] In the Triton framework, the data loading is usually performed using the tl.load instruction. Based on the calculated block pointer, the computing device loads the corresponding matrix sub-block data from the global memory with low speed but large capacity to the thread local register (TLR) private to each computing thread, which has extremely fast access speed. It should be understood that in this initial loading stage, the data format does not change. In other words, the input data stored in the global memory adopts the first data format (i.e. plain buffer), and when they are loaded into the TLR, the memory layout of the data in the TLR still remains the plain buffer format. This is equivalent to a direct copy of the original data, which provides a data source for subsequent format conversion and data rearrangement.

[0088] In step S22, the data stored in the thread local register is converted from the first data format to the second data format, and the converted data is carried to the main cache.

[0089] Specifically, after the data is loaded into the TLR, the format conversion is performed immediately. This process reads the data in the TLR in the plain buffer format and rearranges the data according to the specific rules of the second data format (i.e. usharp). The usharp format is a hardware-specific optimized memory format, and the layout thereof is designed to maximize the data reading efficiency of the tensor core. This conversion process is essentially a reorganization and arrangement of the data elements in the TLR.

[0090] After the data is converted into the usharp format, they are not written back to the TLR, but are directly carried and written into the main cache (GMB). Through this step, the data prepared for subsequent tensor core computation and in the correct format is placed in the most efficient memory location.

[0091] In the embodiment of the present application, by dividing the loading process into two stages of loading to the TLR and converting and carrying from the TLR to the GMB, the complex data format conversion operation can be completed in the high-speed on-chip storage, avoiding multiple read and write between the global memory and the main cache, thereby improving the efficiency of the entire data preparation stage.

[0092] Based on any of the above embodiments, step S30 specifically includes:

[0093] Step S31, loading the data in the main cache to the input cache of the tensor core through a first special instruction.

[0094] Step S32, performing matrix multiplication accumulation calculation using the data in the input cache through a second special instruction to generate the result data.

[0095] Specifically, the embodiment of the present application describes the specific data flow and instruction calling process when the tensor core internally performs matrix multiplication operation. After the processing of the previous steps, the sub-block data of the input matrices A and B for matrix multiplication operation has been stored in the main cache (GMB) in an efficient second data format (usharp). GMB serves as the direct data source for tensor core calculation. The target of data loading is the input cache of the tensor core (i.e. GIB). Here, GIB is a special small-capacity cache inside or extremely close to the core, which is the last cache of data before performing calculation. Therefore, the data in GMB can be loaded into GIB through a first special instruction, so as to perform subsequent MMA calculation.

[0096] Here, the first special instruction refers to a specific machine instruction provided at the hardware level, which is specially used to load data from GMB to GIB. For example, the instruction can be a Matrix LDMMA instruction. Unlike the general loading instruction, the Matrix LDMMA instruction is specially designed to efficiently read the data block in the usharp format in GMB and fill it into GIB, perfectly matching the calculation needs of the tensor core. The execution of the instruction is a prerequisite for realizing the high throughput of the tensor core.

[0097] Subsequently, the data in the input cache GIB can be read and MMA calculation can be performed through a second special instruction. Here, the second special instruction refers to a machine instruction provided at the hardware level, which is used to trigger the tensor core to perform actual matrix multiplication accumulation operation. For example, the instruction can be a Matrix MMA instruction. The instruction drives the tensor core to obtain the sub-block data of the input matrices A and B from GIB, and performs a large-scale parallel dot product operation, and then accumulates the calculation result to the internal accumulator.

[0098] After a series of Matrix MMA instructions are executed, the final calculation result is saved in the accumulator of the tensor core, forming the result data. According to the hardware design and instruction parameters, these result data can be selectively written out to different target positions, for example, can be directly saved in the TLR, or written back to the GMB for subsequent processing.

[0099] It can be understood that in the embodiments of the present application, the tl.dot operation of the Triton framework will be intelligently converted into the above-mentioned sequence of first special instructions (such as Matrix LDMMA) and second special instructions (such as Matrix MMA) in the compilation conversion (lowering) process. By using the first special instruction and the second special instruction, efficient data loading from the GMB to the GIB and hardware-level matrix multiplication accumulation calculation based on the tensor core are realized. This way of directly using hardware special instructions bypasses the general calculation path and maximizes the peak computing performance of the tensor core.

[0100] Based on any of the above embodiments, in step S40, the conversion of the result data from the second data format to the first data format comprises:

[0101] In step S41, if the result data is stored in the main cache, the result data is moved from the main cache to the thread local register;

[0102] In step S42, the result data in the thread local register is converted from the second data format to the first data format.

[0103] Specifically, as described in the above embodiments, the second special instruction (such as Matrix MMA) for performing matrix multiplication accumulation calculation can output the generated result data to different hardware positions according to its parameter configuration. One possibility is to output directly to the TLR, and the other possibility is to output to the GMB. Therefore, before format conversion, it is necessary to determine the current storage position of the result data.

[0104] When it is determined that the result data is located in the GMB, a data moving operation needs to be performed. This operation reads the result data (which is still in usharp format at this time) stored in the GMB and writes it into the TLR. It should be understood that the purpose of this moving operation is to transfer the data to a hardware unit that is more suitable for fine operations (such as format rearrangement). The TLR is a thread-private high-speed register, and performing data format conversion in the TLR can avoid access conflicts to shared GMB resources, and has lower operation delay and higher efficiency.

[0105] After the above step, it can be ensured that the result data is located in the TLR at this time (whether it is directly written by the MatrixMMA instruction or moved from the GMB). In the TLR, the result data still maintains the second data format (i.e., usharp). Subsequently, in the process of writing the result data from the TLR to the global memory, the result data can be reversely converted in format, i.e., converted from the usharp format back to the plain buffer format. Specifically, the process reads the data in the usharp format and rearranges the data elements according to the layout rule (such as row-major order) of the plain buffer. After the conversion, the result data stored in the global memory becomes the general plain buffer format.

[0106] It can be understood that in the entire scheme, the above condition judgment, data moving and format conversion operations are transparent to the upper-layer developers. These operations are seamlessly integrated in the compilation conversion (lowering) process of the tl.store instruction of the Triton framework. When processing the tl.store instruction, the layout information of the result data is analyzed, and if it is found that the data is in the GMB, the moving instruction from the GMB to the TLR is automatically inserted, and then the format conversion instruction from the usharp to the plain buffer is inserted, and finally the instruction for writing the plain buffer data back to the global memory is generated.

[0107] Based on any of the above embodiments, the step of converting the data stored in the thread local register from the first data format to the second data format and moving the converted data to the main cache is implemented by inserting a first layout conversion operation and a second layout conversion operation in the compilation conversion process of the load operation on the input data, the first layout conversion operation is used for format conversion, and the second layout conversion operation is used for data moving.

[0108] Specifically, the process of loading the input data from the global memory to the GMB and completing the format conversion includes two key actions of converting the plain buffer in the TLR to the usharp and moving the usharp data to the GMB. The two actions are implemented by introducing specific intermediate representation (Intermediate Representation, abbreviated as IR) operations when the compiler processes the load operation (such as tl.load). In order to facilitate the understanding of the technical solutions provided in the embodiments of the present application, the lowering process of the Triton framework will be briefly described below.

[0109] Figure 3 is a flowchart of the compilation conversion of the Triton framework provided by the present application, asFigure 3 As shown, it demonstrates the core flow of Triton compiler framework, that is, how to compile and convert the Python-like high-level Triton source code written by users into instructions executable by underlying hardware step by step. This process is completed through a series of intermediate representations, and each step performs specific conversion and optimization.

[0110] Triton source code (i.e., Triton Src) is the starting point of the whole process, which is a Python function written by developers using Triton DSL (Domain-Specific Language). These functions can be captured and compiled by the Triton compiler framework. At this time, the source code is high-level and platform-independent. When the Triton compiler framework starts to compile the source code, the first step is to parse it into a standard Python Abstract Syntax Tree (AST). Then, the compiler traverses this AST and converts it into Triton Intermediate Representation (i.e., Triton IR). At this stage, the IR is still hardware-independent, and it mainly focuses on the logic and data flow of the algorithm, rather than how data blocks are specifically mapped to GPU threads, thread bundles or shared memory. Then, the platform-independent Triton IR is further compiled and converted into platform-dependent intermediate representation, that is, GPU-oriented intermediate representation (Triton GPU IR), and at this time, the IR contains very specific GPU execution information. Finally, the Triton GPU IR is further converted into LLVM Intermediate Representation (i.e., LLVM IR). Here, LLVM is a mature and widely used compiler backend framework in the industry. By generating LLVM IR, the Triton compiler framework can take advantage of the powerful backend optimization capabilities of LLVM to handle register allocation, instruction scheduling and other underlying optimizations. Finally, LLVM will compile the LLVM IR into executable code for a specific GPU.

[0111] In the above compilation and conversion process, a key stage is the conversion from platform-independent Triton IR to platform-dependent Triton GPU IR. It is at this stage that the compiler performs layout-dependent analysis and conversion. In the embodiment of the present application, in order to realize the function of converting from TLR to GMB with format conversion, the compiler introduces two new layout conversion operations, i.e., the first layout conversion operation (convert_layout_1) and the second layout conversion operation (convert_layout_2), on the lowering path of the tl.load instruction.

[0112] convert_layout_1 is used to convert data from plain buffer format to usharp format, and convert_layout_2 is used to carry data from TLR to GMB. Specifically, the convert_layout_1 operation receives a tensor with plain buffer layout in TLR as input and generates a logical tensor with usharp layout as output. The implementation of the operation will eventually be converted into a series of data rearrangement instructions, which read data in TLR and reorganize them according to the requirements of the usharp format.

[0113] The convert_layout_2 operation receives a tensor with logically usharp layout generated by the convert_layout_1 operation as input and maps its physical location from TLR to GMB. Its implementation will eventually be converted into a series of storage instructions to write the rearranged data from TLR to the corresponding address of GMB.

[0114] The method provided by the embodiment of the application realizes seamless integration of complex data preparation work into the compilation process by automatically inserting the first layout conversion operation for format conversion and the second layout conversion operation for data carrying in the compilation conversion process of the tl.load instruction. This automatic and transparent processing method greatly reduces the threshold for using hardware acceleration functions, so that the developer can continue to use high-level and abstract programming interfaces, while enjoying the huge performance benefits brought by underlying hardware optimization, and both high performance and high usability are taken into account.

[0115] Based on any of the above embodiments, the steps of carrying the result data from the main cache to the thread local register and converting the result data from the second data format to the first data format are implemented by inserting a third layout conversion operation and a fourth layout conversion operation in the compilation conversion process of the storage operation on the result data, the third layout conversion operation is used for data carrying, and the fourth layout conversion operation is used for format conversion.

[0116] Specifically, converting the result data from GMB and usharp format back to plain buffer format involves two core steps of carrying from GMB to TLR and converting usharp to plain buffer in TLR. In the embodiment of the application, the operation of carrying data from GMB to TLR and converting usharp to plain buffer will be hidden in the storage operation (i.e. tl.store).

[0117] Similar to the load operation, when the Triton compiler framework processes a tl.store instruction, it analyzes the layout information of the data to be stored and performs the necessary layout conversion during the conversion from Triton IR to Triton GPU IR. In order to realize the reverse process of safely writing the result data back to the global memory, the compiler introduces two new layout conversion operations, namely the third layout conversion operation (convert_layout_3) and the fourth layout conversion operation (convert_layout_4), in the lowering path of the tl.store instruction.

[0118] convert_layout_3 is used to transfer data from GMB to TLR. This operation is inserted when the compiler detects that the result data to be stored is located in GMB. It receives a tensor with usharp layout in GMB as input and transfers its physical location to TLR. Its implementation will eventually be converted into a series of load instructions to read data from GMB into TLR.

[0119] convert_layout_4 is used to convert usharp to plain buffer. This operation receives a tensor with usharp layout in TLR as input (which may come from the third layout conversion operation or be generated directly by the calculation unit) and converts its layout to plain buffer format. The implementation of this operation will eventually be converted into a series of data rearrangement instructions to complete the reverse conversion of the format in TLR.

[0120] Specifically, when processing the tl.store instruction, the layout of the data source will be checked first. If the data is in GMB, the convert_layout_3 and convert_layout_4 operations will be inserted in turn. If the data is already in TLR but is still in usharp format, only the convert_layout_4 operation needs to be inserted. After completing these conversions, the tl.store instruction will finally write the data in plain buffer format in TLR back to the global memory.

[0121] The method provided by the embodiment of the application ensures that the initial state of the calculation result can be correctly and efficiently converted back to a general format by automatically inserting the third layout conversion operation for data transfer and the fourth layout conversion operation for format conversion during the compilation and conversion process of the tl.store instruction.

[0122] Based on any of the above embodiments, this invention provides a method for accelerating matrix multiplication using the Triton framework based on a tensor core. To fully utilize the computational resources of the tensor core, its dedicated MMA instructions are required. However, the MMA instructions have some constraints:

[0123] First, the Tensor Core has its own storage unit, and its dedicated MMA instructions cannot directly use the data in the TLR for calculation. Therefore, in this embodiment of the invention, the data is first moved from the TLR to the GMB, and then the data is repeatedly read from the GMB and loaded into the GIB for calculation using the dedicated MMA instructions of the Tensor Core (such as Matrix LDMMA).

[0124] Secondly, tensor cores cannot use data in plain buffer format. Therefore, this embodiment of the invention converts the data from plain buffer format to usharp format before computation.

[0125] Based on the above constraints, this embodiment of the invention adds the following operations to the traditional Triton framework matrix multiplication implementation: converting the input data from plain buffer format to usharp format; transferring the data from TLR to GMB; connecting tl.dot to the MMA instructions dedicated to the tensor core; if the output result is stored in GMB, it is also necessary to transfer the data back to TLR; and converting the output result from usharp format to plain buffer format. In the above operations, the plain buffer to usharp conversion and the data transfer from TLR to GMB operations are hidden in the tl.load process. The usharp to plain buffer conversion and the data transfer from GMB to TLR operations are hidden in tl.store.

[0126] like Figure 3 As shown, the lowering process of the Triton framework converts the Triton source code into LLVM IR, which is a multi-stage process. The lowering process from Triton IR to Triton GPU IR incorporates layout analysis and layout transformation operations. In the Triton framework, layout describes the storage layout of tensors in memory. Specifically, layout defines how each dimension of a tensor is mapped to a memory address. This embodiment of the invention implements the following functions during the layout transformation process: mutual conversion between usharp and plain buffer, data transfer from TLR to GMB, and data transfer from GMB to TLR.

[0127] Figure 4A data flow graph for performing matrix multiplication operation by using a tensor core is provided by the present application, as shown in Figure 4 As shown in the figure, when the GPU executes the Triton kernel function (which is used to implement the MMA operation), the input matrices A and B and the output matrix C (corresponding to A, B, and C shown in Figure 4 During the execution of the tl.load instruction, the data of the input matrices A and B is loaded from the global memory into the TLR by default, and the loaded data is in the plain buffer format. During the lowering of the tl.load, the data in the plain buffer format is converted into the usharp format and is carried into the GMB. During the execution of the tl.dot, the data is loaded from the GMB into the GIB through the Matrix LDMMA instruction, and then the Matrix MMA instruction is called to perform matrix multiplication calculation. The Matrix MMA instruction can control the writing of the calculation result to the TLR or the GMB. During the tl.store, if the calculation result is written to the GMB, the data needs to be carried back to the TLR, and then the data in the TLR is converted from the usharp format into the plain buffer format, and finally is stored into the global memory.

[0128] Figure 5 A flowchart of a Triton framework matrix multiplication acceleration method based on a tensor core is provided by the present application, as shown in Figure 5 The method comprises the following steps:

[0129] Step 1, the input matrices A and B are stored in the global memory in the plain buffer format. First, according to the shape of the input matrices A and B and the hardware characteristics of the GPU, a reasonable splitting strategy is selected, and the A and B matrices are divided into blocks. For each sub-block sub_A and sub_B, the block pointer ptr_sub_A and ptr_sub_B needs to be calculated.

[0130] Step 2, the tl.load instruction is executed, and the sub-blocks sub_A and sub_B of the matrices A and B are loaded according to the block pointers ptr_sub_A and ptr_sub_B. By default, the tl.load will load the data from the global memory into the TLR, and the loaded data is in the plain buffer format.

[0131] Step 3, convert_layout_1 is added in the lowering of the tl.load, which is used to convert the data from the plain buffer format into the usharp format and rearrange the data of sub_A and sub_B.

[0132] Step 4, add convert_layout_2 in the lowering process of tl.load, which is used for address mapping of sub_A and sub_B to carry data from TLR to GMB.

[0133] Step 5, dock tl.dot to the MMA instructions dedicated to tensor cores, including Matrix LDMMA and Matrix MMA. The Matrix LDMMA instruction is used to load data from GMB to GIB, and the Matrix MMA instruction is used to perform matrix multiplication calculation. This process is implemented in the lowering process of tl.dot.

[0134] Step 6, in the lowering process of tl.store, if the calculation result is written out to GMB, add convert_layout_3, which is used to carry data from GMB to TLR. If the calculation result is directly written out to TLR or carried from GMB to TLR, add convert_layout_4 in the lowering process of tl.store, which is used to convert the result data from usharp format to plain buffer format to ensure that the result matrix C can be stored in the form of plain buffer.

[0135] Step 7, write the calculation result to the corresponding position of the output matrix C, and tl.store will store data from TLR to global memory by default.

[0136] The method provided by the embodiment of the application can automatically process data format conversion and data carrying in the compilation process without changing the user's programming habits (still using high-level load, dot product, and store instructions), so that the Triton framework matrix multiplication operation that cannot originally utilize tensor cores can fully utilize the powerful computing power of tensor cores. This solves the problem of idle tensor cores and limited performance caused by mismatch between data format and memory access path in related technologies, thereby significantly improving the performance and efficiency of matrix multiplication operation while maintaining good programming ease of use and framework compatibility.

[0137] The data processing apparatus provided by the application is described below. The data processing apparatus described below can be referred to in conjunction with the data processing method described above.

[0138] Based on any of the above embodiments, Figure 6 is a structural schematic diagram of the data processing apparatus provided by the application, as Figure 6 shown, the apparatus comprises:

[0139] The acquisition unit 610 is configured to acquire input data stored in a first data format.

[0140] a converting unit 620, configured to convert the input data from the first data format to a second data format supported by the tensor core in a process of loading the input data to the main cache;

[0141] an operating unit 630, configured to perform a matrix multiplication operation based on the tensor core and using the data stored in the main cache in the second data format to generate result data;

[0142] an output unit 640, configured to convert the result data from the second data format to the first data format and output the converted result data.

[0143] The device provided by the embodiment of the present application ingeniously solves the fundamental problem that the tensor core cannot be used due to incompatible data formats by converting the data from the general first data format to the hardware-specific second data format supported by the tensor core in the process of loading the input data to the main cache, and this key step enables the subsequent calculation to be performed based on the tensor core. Since the calculation throughput of the tensor core is much higher than that of the vector core, the present application can fully utilize the peak calculation capacity of the hardware, thereby greatly improving the speed and efficiency of the matrix multiplication operation. In addition, the present application seamlessly integrates the complex format conversion operation in the process of data loading and data output. For the upper-layer application or the developer, the input data processed by them and the result data finally obtained are always the standard first data format, and the whole process of format conversion is transparent to the user, and the developer does not need to manually perform complex data format management and adaptation work. This not only achieves hardware acceleration and obtains great performance gain, but also does not increase the complexity of programming, maintains good compatibility and ease of use with the existing programming framework, and solves the contradiction between high performance and development efficiency.

[0144] Based on any of the above embodiments, the converting unit 620 is specifically configured to:

[0145] calculate a block pointer corresponding to the input data, and load the input data from a global memory to a thread local register based on the block pointer, the input data being stored in the first data format in the global memory and the thread local register;

[0146] convert the data stored in the thread local register from the first data format to the second data format, and carry the converted data to the main cache.

[0147] Based on any of the above embodiments, the step of converting the data stored in the thread local register from the first data format to the second data format and transferring the converted data to the main cache is implemented by inserting a first layout conversion operation and a second layout conversion operation in a compilation conversion process of performing a load operation on the input data, the first layout conversion operation is used for format conversion, and the second layout conversion operation is used for data transfer.

[0148] Based on any of the above embodiments, the operation unit 630 is specifically configured to:

[0149] loading data in the main cache to the input cache of the tensor core through a first special instruction;

[0150] performing matrix multiplication accumulation calculation using data in the input cache through a second special instruction to generate the result data.

[0151] Based on any of the above embodiments, the output unit 640 is specifically configured to:

[0152] if the result data is stored in the main cache, transferring the result data from the main cache to the thread local register;

[0153] converting the result data in the thread local register from the second data format to the first data format.

[0154] Based on any of the above embodiments, the step of transferring the result data from the main cache to the thread local register and converting the result data from the second data format to the first data format is implemented by inserting a third layout conversion operation and a fourth layout conversion operation in a compilation conversion process of performing a store operation on the result data, the third layout conversion operation is used for data transfer, and the fourth layout conversion operation is used for format conversion.

[0155] Based on any of the above embodiments, the obtaining unit 610 is specifically configured to:

[0156] According to the shape of the input matrix and the hardware characteristics of the computing device, the input matrix is split to obtain a plurality of matrix subblocks, and the plurality of matrix subblocks are taken as the input data.

[0157] Figure 7 An example of an electronic device is shown in the schematic diagram of the physical structure of the electronic device, as shown in Figure 7As shown, the electronic device can include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 can communicate with each other through the communication bus 740. The processor 710 can invoke a logical instruction in the memory 730 to execute a data processing method, which includes: obtaining input data stored in a first data format; converting the input data from the first data format to a second data format in the process of loading the input data into a main cache, the second data format being a hardware-specific data format supported by a tensor core; performing a matrix multiplication operation based on the tensor core using data stored in the second data format in the main cache to generate result data; converting the result data from the second data format to the first data format, and outputting the converted result data.

[0158] In addition, the logical instruction in the memory 730 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the related art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0159] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the data processing method provided by the above-mentioned method, which includes: obtaining input data stored in a first data format; converting the input data from the first data format to a second data format in the process of loading the input data into a main cache, the second data format being a hardware-specific data format supported by a tensor core; performing a matrix multiplication operation based on the tensor core using data stored in the second data format in the main cache to generate result data; converting the result data from the second data format to the first data format, and outputting the converted result data.

[0160] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a data processing method provided by any of the above methods, the method comprising: obtaining input data stored in a first data format; converting the input data from the first data format to a second data format in a process of loading the input data into a main cache, the second data format being a hardware-specific data format supported by a tensor core; performing a matrix multiplication operation based on the tensor core using data stored in the second data format in the main cache to generate result data; converting the result data from the second data format to the first data format, and outputting the converted result data.

[0161] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0162] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0163] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that, include: Obtain the input data stored in the first data format; During the compiler's compilation and conversion of the loading instruction, the input data is automatically read from global memory and a first conversion logic is inserted. The first conversion logic is used to convert the input data from the first data format to the second data format and move the converted input data to the main cache. The second data format is a hardware-specific data format supported by the Tensor Core. The loading instruction refers to the instruction that loads the input data from the global memory. Based on the tensor core, matrix multiplication is performed using data stored in the main cache in the second data format to generate result data. During the compiler's compilation and conversion of storage instructions, a second conversion logic is inserted. The second conversion logic is used to convert the result data from the second data format to the first data format and output the converted result data. The storage instruction refers to the instruction that writes the result data back to the global memory.

2. The data processing method according to claim 1, characterized in that, The automatic reading of the input data from global memory includes: Calculate the block pointer corresponding to the input data, and load the input data from the global memory to the thread local register based on the block pointer. The input data is stored in the first data format in both the global memory and the thread local register.

3. The data processing method according to claim 2, characterized in that, The first conversion logic includes a first layout conversion operation and a second layout conversion operation. The first layout conversion operation is used to convert the data stored in the thread local register from the first data format to the second data format, and the second layout conversion operation is used to move the converted data to the main cache.

4. The data processing method according to claim 1, characterized in that, The steps for performing matrix multiplication include: The data in the main cache is loaded into the input cache of the tensor core using the first dedicated instruction; The second dedicated instruction performs matrix multiplication and accumulation calculations using the data in the input buffer to generate the result data.

5. The data processing method according to claim 1, characterized in that, The step of converting the result data from the second data format to the first data format includes: If the result data is stored in the main cache, then the result data is moved from the main cache to the thread-local register; Convert the result data in the thread-local register from the second data format to the first data format.

6. The data processing method according to claim 5, characterized in that, The second conversion logic includes a third layout conversion operation and a fourth layout conversion operation. The third layout conversion operation is used to move the result data from the main cache to the thread local register, and the fourth layout conversion operation is used to convert the result data from the second data format to the first data format.

7. The data processing method according to any one of claims 1 to 6, characterized in that, The acquisition of input data stored in the first data format includes: Based on the shape of the input matrix and the hardware characteristics of the computing device, the input matrix is ​​divided into multiple matrix sub-blocks, and these multiple matrix sub-blocks are used as the input data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 7.

Images