Data processing method, electronic device, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence chips, and provides a data processing method, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining input data stored in a first data format; in the process of loading the input data to the main cache, the input data is converted into a second data format from the first data format, and the second data format is a hardware dedicated data format supported by the tensor core; based on the tensor core, using the data stored in the main cache in the second data format to execute matrix multiplication to generate result data; and converting the result data from the second data format to the first data format, and outputting the converted result data. According to the method, an automatic format conversion mechanism is introduced in a key link of data circulation, a channel between an upper-layer application and bottom-layer tensor core hardware is successfully opened, on the premise that usability is not sacrificed, effective utilization of a tensor core is achieved, and remarkable performance improvement is brought.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence chip technology, and in particular to a data processing method, electronic device, storage medium, and program product. Background Technology

[0002] In the fields of artificial intelligence and high-performance computing, matrix multiply-accumulate (MMA) is a core computational task. To efficiently perform this type of computational task, modern high-performance computing devices such as graphics processing units (GPUs), general-purpose computing on graphics processing units (GPGPUs), and tensor processing units (TPUs) typically integrate hardware acceleration units specifically designed for matrix operations, such as tensor cores, to provide powerful computational acceleration capabilities.

[0003] The Triton framework, a popular framework for simplifying programming on high-performance computing devices such as GPUs, allows developers to easily implement operations like MMA. However, research has revealed that existing matrix multiplication implementations based on the Triton framework, by default executed by the general-purpose Vector Core, are far less efficient at performing matrix operations than the dedicated Tensor Core. This failure to effectively utilize the dedicated computing capabilities of the Tensor Core, coupled with limitations in data format compatibility, prevents computational performance from reaching the theoretical hardware peak, thus failing to meet the ever-increasing demands of high-performance computing. Therefore, enabling the Triton framework to fully leverage the hardware capabilities of the Tensor Core is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This invention provides a data processing method, electronic device, storage medium, and program product to address the shortcomings of matrix multiplication calculations based on the Triton framework in related technologies, which suffer from poor computational performance due to the inability to utilize the dedicated computing capabilities of the tensor core.

[0005] This invention provides a data processing method, comprising: Obtain the input data stored in the first data format; During the process of loading the input data into the main cache, the input data is converted from the first data format to the second data format, which is a hardware-specific data format supported by the Tensor Core. Based on the tensor core, matrix multiplication is performed using data stored in the main cache in the second data format to generate result data. The result data is converted from the second data format to the first data format, and the converted result data is output.

[0006] According to a data processing method provided by the present invention, the step of converting the input data from a first data format to a second data format during the process of loading the input data into the main cache includes: Calculate the block pointer corresponding to the input data, and load the input data from global memory to the thread local register based on the block pointer. The input data is stored in the first data format in both the global memory and the thread local register. The data stored in the thread-local register is converted from the first data format to the second data format, and the converted data is moved to the main cache.

[0007] According to a data processing method provided by the present invention, the step of converting the data stored in the thread local register from the first data format to the second data format and moving the converted data to the main cache is implemented by inserting a first layout conversion operation and a second layout conversion operation during the compilation conversion process of performing a loading operation on the input data. The first layout conversion operation is used for format conversion, and the second layout conversion operation is used for data movement.

[0008] According to a data processing method provided by the present invention, the step of performing matrix multiplication includes: The data in the main cache is loaded into the input cache of the tensor core using the first dedicated instruction; The second dedicated instruction performs matrix multiplication and accumulation calculations using the data in the input buffer to generate the result data.

[0009] According to a data processing method provided by the present invention, the step of converting the result data from the second data format to the first data format includes: If the result data is stored in the main cache, then the result data is moved from the main cache to the thread-local register; Convert the result data in the thread-local register from the second data format to the first data format.

[0010] According to a data processing method provided by the present invention, the steps of moving the result data from the main cache to the thread local register and converting the result data from the second data format to the first data format are implemented by inserting a third layout conversion operation and a fourth layout conversion operation during the compilation and conversion process of performing storage operations on the result data. The third layout conversion operation is used for data moving, and the fourth layout conversion operation is used for format conversion.

[0011] According to a data processing method provided by the present invention, the step of acquiring input data stored in a first data format includes: Based on the shape of the input matrix and the hardware characteristics of the computing device, the input matrix is ​​divided into multiple matrix sub-blocks, and these multiple matrix sub-blocks are used as the input data.

[0012] The present invention also provides a data processing apparatus, comprising: An acquisition unit is used to acquire input data stored in a first data format; The conversion unit is used to convert the input data from the first data format to the second data format during the process of loading the input data into the main cache. The second data format is a hardware-specific data format supported by the Tensor Core. The computation unit is used to perform matrix multiplication operations based on the tensor core and using data stored in the main cache in the second data format to generate result data. The output unit is used to convert the result data from the second data format to the first data format and output the converted result data.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the data processing method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data processing methods described above.

[0016] The data processing method, electronic device, storage medium, and program product provided by this invention cleverly solve the fundamental problem of being unable to utilize the Tensor Core due to data format incompatibility by converting the data from a general first data format to a hardware-specific second data format supported by the Tensor Core during the process of loading input data into the main cache. This crucial step allows subsequent calculations to be executed smoothly based on the Tensor Core. Since the computational throughput of the Tensor Core is much higher than that of the Vector Core, this invention can fully utilize the peak computing power of the hardware, thereby greatly improving the speed and efficiency of matrix multiplication operations. Furthermore, this invention seamlessly integrates complex format conversion operations into the data loading and output processes. For upper-layer applications or developers, the input data they process and the final result data they obtain are always in the standard first data format. The entire format conversion process is transparent to the user, eliminating the need for developers to manually perform complex data format management and adaptation work. This achieves hardware acceleration and obtains significant performance gains without increasing programming complexity, maintaining good compatibility and ease of use with existing programming frameworks, and resolving the contradiction between high performance and development efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of the computing device provided by the present invention; Figure 2 This is a flowchart illustrating the data processing method provided by the present invention; Figure 3 This is a schematic diagram of the compilation and conversion process of the Triton framework provided by this invention; Figure 4 This invention provides a data flow graph for performing matrix multiplication using a tensor core. Figure 5 This is a flowchart illustrating the matrix multiplication acceleration method based on the tensor core of the Triton framework provided by this invention. Figure 6 This is a schematic diagram of the data processing device provided by the present invention; Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] In artificial intelligence, deep learning, and other high-performance computing fields, matrix multiplication-accumulation (MMA) is one of the most critical and computationally intensive operations. The training and inference processes of neural networks, such as the computation of convolutional and fully connected layers, essentially rely on a large number of matrix multiplication operations. To efficiently perform these computational tasks, high-performance computing devices such as GPUs and TPUs typically integrate hardware acceleration units specifically designed for matrix operations, such as Tensor Cores.

[0021] Triton, a cutting-edge open-source programming language and compiler framework, aims to lower the barrier to entry for GPU programming by enabling developers to easily write high-performance GPU computing kernels using Python-like syntax. In the Triton compiler framework (hereinafter referred to as the "Triton framework"), developers typically use its provided `tl.dot` operation to implement matrix multiplication. A typical Triton framework matrix multiplication implementation process is as follows: First, the large matrix to be computed is divided into multiple smaller sub-blocks using a titling strategy; then, in a loop, the `tl.load` instruction loads the sub-blocks of the input matrix from global memory into the thread local register (TLR); next, the `tl.dot` operation is called to perform matrix multiplication on the loaded sub-blocks, and the results are accumulated; finally, the `tl.store` instruction writes the final calculation result back to global memory.

[0022] However, in practical applications, the aforementioned matrix multiplication implementation based on the Triton framework has been found to have significant performance bottlenecks. First, computational resources are underutilized. In conventional implementations, the tl.dot operation of the Triton framework is typically converted after compilation to use a general-purpose vector core to perform matrix multiplication. While vector cores are versatile, their efficiency in performing matrix operations is far lower than that of dedicated tensor cores. The computational throughput of tensor cores is usually several times, or even tens of times, that of vector cores. Therefore, relying solely on vector cores for computation results in the idle resources of the powerful tensor cores in the GPU, preventing the full utilization of the hardware's peak computing power and creating a significant performance bottleneck.

[0023] Secondly, data format incompatibility renders specialized instructions unusable. To achieve optimal memory access efficiency and computational performance, the hardware-specific MMA instructions upon which the Tensor Core relies typically require input data to follow a specially optimized hardware memory layout (e.g., usharp memory layout). However, in the standard Triton framework programming model, input matrix data, when loaded via the tl.load instruction, is usually in an unoptimized general memory format (e.g., plainbuffer). This data format mismatch directly hinders the compiler from generating and invoking efficient hardware-specific instructions for the Tensor Core, preventing the Triton framework from fundamentally leveraging the performance advantages offered by the Tensor Core. It should be understood that plain buffer and usharp describe two different storage layouts of matrix data in GPU memory. Plain buffer refers to a format in which data is stored in memory in a linear and contiguous manner, such as row-major or column-major order. usharp, on the other hand, is a hardware-specific, optimized non-linear memory layout. It is a data rearrangement format designed for Tensor Cores of a specific GPU architecture. In this format, data is no longer simply stored contiguously by row or column, but rather the elements within a matrix block are rearranged and combined so that the data elements that need to be loaded together by the Tensor Core are adjacent to each other.

[0024] To address this, the present invention provides a data processing method that introduces an automatic format conversion mechanism at key stages of data flow and seamlessly integrates the format conversion operation into the data loading and output processes. This not only successfully establishes a channel between upper-layer applications and the underlying tensor core hardware, thereby fully utilizing the computing resources of the tensor core and significantly improving the performance of matrix multiplication operations, but also maintains good compatibility and ease of use with existing programming frameworks (such as the Triton framework), allowing users to use it without making extensive modifications to existing code, thus overcoming the aforementioned shortcomings.

[0025] It is understandable that "upper-layer application" is a relative concept, referring to any code, program, or system that directly calls the data processing method described in this invention without needing to concern itself with its internal implementation details. For example, an upper-layer application could be a Triton kernel function written by a developer. When a developer writes a matrix multiplication program using the Triton language, they only need to follow the standard syntax of the Triton language, such as calling `tl.load` to load data, `tl.dot` to perform calculations, and `tl.store` to save the results. In this scenario, the Triton kernel function written by the developer is the upper-layer application. They expect to input a matrix in a normal format (i.e., the first data format) and obtain a result matrix in the same format. The technical solution of this invention automatically completes the entire process at the underlying level (i.e., during the compilation and execution of the Triton framework) of converting from the first data format to the second data format, calling the tensor kernel for calculation, and then converting back to the first data format. For developers writing upper-layer applications, this complex optimization process is transparent and imperceptible; they enjoy performance improvements without modifying their code.

[0026] For example, the upper-layer application can also be a larger-scale program or framework that calls Triton kernel functions, such as a large artificial intelligence model, a deep learning framework (such as PyTorch or TensorFlow), or a specific business application (such as an image recognition service or a natural language processing system). When these large programs execute, they can call a high-performance operator written in the Triton language to complete a computational task (such as matrix multiplication). In this scenario, the artificial intelligence model or business application is the upper-layer application; it is only responsible for passing data to the Triton operator and receiving the results. It doesn't need to know that the underlying execution is done by the GPU, nor does it need to care about how the GPU's internal data undergoes format conversion to adapt to the tensor kernel.

[0027] This invention encapsulates the complex low-level optimizations (i.e. format conversion) performed for ultimate performance, providing a stable, concise, and efficient interface for upper-layer applications, thus perfectly balancing high performance and ease of use.

[0028] Figure 1 This is a schematic diagram of the structure of the computing device provided by the present invention, as shown below. Figure 1 As shown, the execution subject of the data processing method provided by the present invention can be a computing device. The computing device 100 includes at least a plurality of processing units 101 and video memory 102. Each processing unit 101 can be a streaming processor cluster (SPC), which may include a tensor calculation unit 103, a vector calculation unit 104, an on-chip cache 105, and a register 106.

[0029] Tensor computation unit 103 and vector computation unit 104 are heterogeneous units. Tensor computation unit 103 is mainly used to perform tensor computations (such as matrix multiplication, convolution, etc.). Tensor computation unit 103 includes multiple tensor cores for performing specific computational tasks. Vector computation unit 104 is mainly used to perform arithmetic and logical operations, such as accumulation, reduction, and regular addition, subtraction, multiplication, and division. Vector computation unit 104 includes multiple vector cores for performing specific computational tasks.

[0030] The video memory 102 can be high-bandwidth memory (HBM) or other types of memory. The on-chip cache 105 is a temporary memory with a smaller capacity than the video memory 102, but a faster data exchange speed. The on-chip cache 105 includes a cache shared by the tensor computation unit 103 and the vector computation unit 104, as well as a cache exclusively used by either the tensor computation unit 103 or the vector computation unit 104. The on-chip cache 105 may include a general-purpose matrix main buffer (GEMM Main Buffer, GMB) and a general-purpose matrix input buffer (GEMM Input Buffer, GIB) for the tensor computation unit 103.

[0031] Register 106 includes registers shared by tensor computation unit 103 and vector computation unit 104, as well as registers exclusive to either tensor computation unit 103 or vector computation unit 104. For example, register 106 may include thread-local registers (TLRs). Tensor computation unit 103 and vector computation unit 104 can exchange data through register 106. Compared to on-chip cache 105, register 106 has a smaller capacity but a faster data exchange speed.

[0032] It should be noted that the computing device 100 of this invention may include other structures besides those described above, and this invention does not specifically limit these structures. The computing device 100 may be a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processor (TPU), etc.

[0033] based on Figure 1 The diagram shows the architecture of a computing device. This embodiment of the invention provides a data processing method that can be applied to, for example... Figure 1The computing device shown specifically utilizes the tensor core in its tensor computing unit 103 to accelerate matrix multiplication operations. Specifically, this method can be applied to various scenarios, such as text processing, image processing, and speech processing. In different application scenarios, the input data has different physical meanings. For example, in a text processing scenario, the input data can be text data related to text generation and text recognition. As another example, in an image processing scenario, the input data can be image data used in tasks such as image preprocessing, image segmentation, and object detection. As yet another example, in a speech processing scenario, the input data can be speech data used in tasks such as speech recognition and speech synthesis. The specific flow of the data processing method provided by this invention is described below.

[0034] Based on the above embodiments, Figure 2 This is a flowchart illustrating the data processing method provided by the present invention, as shown below. Figure 2 As shown, the method includes: Step S10: Obtain the input data stored in the first data format.

[0035] Specifically, the input data is typically two input matrices used to perform matrix multiplication operations (such as A×B=C), for example, matrix A and matrix B, or it can be a submatrix of at least two input matrices. The input data is typically stored in global memory in a first data format (e.g., ...). Figure 1 In the video memory 102 shown, the process of acquiring this input data can specifically refer to the execution of a Triton kernel function, which receives pointers to input matrices A and B stored in global memory. At this time, these matrix data located in global memory are organized in a first data format (such as plain buffer).

[0036] Understandably, the first data format refers to a general data storage format that is not specifically optimized for hardware, such as a plain buffer. Data in this format may be stored contiguously in memory in standard row-major or column-major order, facilitating access and processing by general-purpose computing units (such as vector cores) in computing devices (such as GPUs). However, this general format is not optimized for the memory access patterns of specific hardware acceleration units (such as Tensor Cores), therefore, Tensor Cores' dedicated hardware instructions cannot use this general-format data. To leverage the powerful computing capabilities of Tensor Cores to perform matrix multiplication operations, the input data in the first data format needs to be converted.

[0037] Step S20: During the process of loading the input data into the main cache, the input data is converted from the first data format to the second data format, where the second data format is a hardware-specific data format supported by the Tensor Core.

[0038] It should be noted that in order for the Triton framework to fully utilize the computational resources of the Tensor Core to perform matrix multiplication operations, it needs to use its supported dedicated MMA instructions. However, the Tensor Core's dedicated MMA instructions have some constraints. For example, the Tensor Core has its own storage units (such as the main cache), so the input data needs to be loaded into the main cache first, and then the data needs to be read from the main cache for calculation.

[0039] Specifically, a main cache is an on-chip cache that is smaller in capacity but much faster than global memory (such as HBM). It is used to temporarily store data that will soon be used by the computing unit to reduce memory access latency. For example, this main cache could specifically be a general-purpose matrix main cache (GMB), which is... Figure 1 This is a portion of the on-chip cache 105 shown.

[0040] In addition to imposing certain constraints on the memory access paths mentioned above, the Tensor Core's dedicated MMA instructions also impose certain constraints on the data layout in memory. Specifically, the Tensor Core cannot use the general first data format (such as the plain buffer). Therefore, in this embodiment of the invention, during the process of loading input data into the main cache, the data is converted from the first data format to a hardware-specific data format that the Tensor Core can support, namely the second data format, so that the Tensor Core can correctly read the data for MMA calculations.

[0041] Here, the second data format refers to a dedicated data layout designed to adapt to specific hardware (i.e., Tensor Core). This format maximizes memory bandwidth utilization and computational efficiency by rearranging the data so that its organization in memory matches the parallel loading and computation requirements of Tensor Core. It should be understood that, in embodiments of the present invention, the second data format may be the usharp format.

[0042] It is understood that the data format conversion in this embodiment of the invention is performed during the data loading process. This means that the data format conversion is deeply coupled with the data loading process, rather than being an independent, sequential step. In other words, the system does not first load the plain buffer format data completely into an intermediate location and then start a new task to convert it to USharp format; instead, when executing a high-level loading instruction (such as tl.load in the Triton framework), the compiler automatically and transparently rearranges the plain buffer format data stream read from global memory during the lowering (downgrading, also known as compilation conversion) process into low-level hardware instructions, and finally writes the rearranged USharp format data into GMB. For upper-level users, they only perceive one loading operation, while the complex format conversion and data movement are hidden from them.

[0043] Step S30: Based on the tensor core, matrix multiplication is performed using the data stored in the main cache in the second data format to generate result data.

[0044] Specifically, after the input data is prepared in the main cache (GMB) in an efficient second data format (usharp), this step calls upon the core computing resources of the hardware, namely the tensor core, to perform the core matrix multiplication-accumulation (MMA) operation. Here, the tensor core is a hardware acceleration unit specifically designed to perform large-scale matrix operations, and its throughput far exceeds that of general-purpose computing units (such as vector cores).

[0045] Performing MMA operations based on Tensor Cores means that computational tasks will be scheduled to run on these dedicated cores. Specifically, the Tensor Cores efficiently read data stored in usharp format from the GMB using dedicated hardware instructions. Because the data format is optimized, the Tensor Cores can load the entire block of data at once, enabling highly parallel computation. By executing matrix multiplication and accumulation instructions, the Tensor Cores perform operations on the input matrix sub-blocks and accumulate the results. This process continues until all relevant matrix sub-blocks have been computed.

[0046] The resulting data, such as the corresponding sub-block data of matrix C, is stored in a temporary location. Depending on the hardware instruction design, this location may be the accumulator register inside the tensor core, or it may be written directly back to the GMB. This resulting data also maintains a second data format (i.e., usharp) adapted to the computation process.

[0047] Step S40: Convert the result data from the second data format to the first data format, and output the converted result data.

[0048] Specifically, after generating the result data, it needs to be converted from the second data format to the first data format—a reverse conversion process. This involves rearranging the result data stored in a temporary location (such as the GMB) in usharp format back into the common plain buffer format. It should be understood that this reverse conversion is necessary because, while the usharp format is efficient for tensor kernels, other parts of the system or subsequent computational steps that do not use tensor kernels typically expect to receive data in the standard, common plain buffer format. Therefore, this conversion ensures that the processing method of this invention can be seamlessly embedded into a wider computational graph without compromising data universality.

[0049] Specifically, similar to step S20, this reverse conversion process can also be seamlessly integrated into the compilation conversion process of high-level storage instructions (such as tl.store in the Triton framework). When the user calls tl.store to write the result back to global memory, the compiler automatically inserts the corresponding conversion logic, ultimately outputting the converted result data. Here, outputting the converted result data means writing the result data, which has been converted to the first data format (plain buffer), back to the corresponding position of the output matrix C in global memory specified by the user. Thus, the user obtains a correct result in global memory that is consistent with the input format but with a significantly accelerated computation process.

[0050] The method provided in this invention cleverly solves the fundamental problem of being unable to utilize the Tensor Core due to data format incompatibility by converting the data from a general first data format to a hardware-specific second data format supported by the Tensor Core during the process of loading input data into the main cache. This crucial step allows subsequent calculations to be executed smoothly based on the Tensor Core. Since the computational throughput of the Tensor Core is much higher than that of the Vector Core, this invention can fully utilize the peak computing power of the hardware, thereby greatly improving the speed and efficiency of matrix multiplication operations. Furthermore, this invention seamlessly integrates complex format conversion operations into the data loading and output processes. For upper-layer applications or developers, the input data they process and the final result data they obtain are always in the standard first data format. The entire format conversion process is transparent to the user, eliminating the need for developers to manually perform complex data format management and adaptation work. This achieves hardware acceleration and obtains significant performance gains without increasing programming complexity, maintaining good compatibility and ease of use with existing programming frameworks, and resolving the contradiction between high performance and development efficiency.

[0051] Based on any of the above embodiments, step S10 specifically includes: Based on the shape of the input matrix and the hardware characteristics of the computing device, the input matrix is ​​divided into multiple matrix sub-blocks, and these multiple matrix sub-blocks are used as the input data.

[0052] It should be noted that the embodiments of the present invention illustrate how to effectively organize and manage input data through a block-based strategy when processing large-scale matrix multiplication, laying the foundation for efficient subsequent processing.

[0053] Specifically, in practical applications, the size of the input matrices to be computed (e.g., matrices A and B) may be very large, far exceeding the capacity of the cache (such as the GMB) or even the thread-local register (TLR) on the computing device. Loading the entire matrix into high-speed storage at once is impractical. Therefore, a block-based technique can be used to handle large-scale matrix multiplication tasks.

[0054] Understandably, the core of this step is data partitioning. Its purpose is to break down a large-scale computational task into a series of manageable, small-scale computational tasks performed on small blocks of data. This significantly improves data locality, allowing data to be loaded into fast on-chip caches and reused, thereby minimizing access to slow global memory—a key to achieving high-performance computing.

[0055] The partitioning of the input matrix is ​​not arbitrary, but requires consideration of two core factors: the shape of the input matrix and the hardware characteristics of the computing device (such as a GPU). The shapes of the input matrices A and B (e.g., matrix A is M×K and matrix B is K×N) directly affect the partitioning strategy. Different partitioning strategies should be used for matrices with different shapes to balance computational load and memory access.

[0056] The hardware characteristics of the computing device are crucial in determining the partitioning granularity (i.e., sub-block size). Hardware characteristics to consider include the computational granularity of the tensor core, the size of the main cache (GMB), the capacity of the thread local registers (TLR), memory bandwidth, and latency. Specifically, the tensor core typically operates in fixed-size blocks (e.g., 64×64×64). Therefore, the sub-block size (BLOCK_SIZE_M, BLOCK_SIZE_N, BLOCK_SIZE_K) should preferably be an integer multiple of the tensor core computational granularity to ensure full utilization of the hardware computing units. Here, BLOCK_SIZE_M, BLOCK_SIZE_N, and BLOCK_SIZE_K refer to the size of the sub-block in the M, N, and K dimensions, respectively.

[0057] The resulting matrix sub-blocks (such as sub_A and sub_B) must be able to be fully loaded into the Global Memory Block (GMB). If the sub-blocks are too large, the GMB will not be able to accommodate them, causing data to be repeatedly swapped between global memory and the GMB, severely impacting performance. Similarly, data is temporarily stored in the Time Redirect Memory (TLR) before being loaded into the GMB, and the TLR's capacity also limits the amount of data processed in a single operation. Therefore, when choosing a suitable partitioning strategy, both the size of the GMB and the capacity of the TLR need to be considered. Furthermore, a reasonable block size can facilitate memory access merging, maximizing effective bandwidth.

[0058] By comprehensively considering the above factors, the optimal block sizes BLOCK_SIZE_M, BLOCK_SIZE_N, and BLOCK_SIZE_K can be determined. Accordingly, the original large matrices A and B are logically divided into multiple matrix sub-blocks. For example, matrix A is divided into multiple BLOCK_SIZE_M × BLOCK_SIZE_K sub-blocks, and matrix B is divided into multiple BLOCK_SIZE_K × BLOCK_SIZE_N sub-blocks.

[0059] Throughout the matrix multiplication process, the program iteratively processes these sub-blocks. In each iteration, a pair of corresponding matrix sub-blocks (e.g., sub_A and sub_B) are selected, becoming the input data for the current iteration. All subsequent steps, including data loading, format conversion, data transfer, and performing MMA operations, are performed on this pair of matrix sub-blocks.

[0060] The method provided in this invention lays the foundation for the entire high-performance data processing flow by introducing a strategy of intelligent block partitioning based on matrix shape and hardware characteristics. This partitioning method not only solves the physical limitation that large matrices cannot be loaded into the cache at once, but more importantly, it fundamentally reduces the dependence on slow global memory and greatly improves memory access efficiency by optimizing data locality and enabling data reuse in the cache.

[0061] Based on any of the above embodiments, step S20 specifically includes: Step S21: Calculate the block pointer corresponding to the input data, and load the input data from global memory to the thread local register based on the block pointer. The input data is stored in the first data format in both the global memory and the thread local register.

[0062] Specifically, to achieve efficient parallel computing, a block-based algorithm is typically employed. Before performing matrix multiplication, a suitable partitioning strategy needs to be selected based on the shapes of the input matrices A and B and the hardware characteristics of the computing device (such as a GPU) (e.g., cache size, computational granularity supported by tensor cores). This strategy logically divides the large input matrices A and B into multiple smaller matrix sub-blocks. Subsequently, for each matrix sub-block to be processed (i.e., the input data), its starting address in global memory, i.e., the block pointer, is calculated. This block pointer is usually obtained by adding the offset calculated based on the sub-block index to the base address of the input matrix in global memory.

[0063] In the Triton framework, the `tl.load` instruction is typically used to perform data loading. Based on the block pointers calculated above, the computing device loads the corresponding matrix sub-blocks of data from the slow but large-capacity global memory into the thread-local registers (TLRs) that are private to each computing thread and have extremely fast access speeds. It should be understood that during this initial loading phase, the data format does not change. In other words, the input data stored in global memory uses the first data format (i.e., plain buffer), and when it is loaded into the TLR, its memory layout within the TLR remains in plain buffer format. This is equivalent to a direct copy of the original data, providing a data source for subsequent format conversion and data rearrangement.

[0064] Step S22: Convert the data stored in the thread local register from the first data format to the second data format, and move the converted data to the main cache.

[0065] Specifically, after data is loaded into the TLR, a format conversion process follows. This process reads data in the plain buffer format from the TLR and rearranges it according to specific rules of a second data format (i.e., usharp). usharp is a hardware-specific optimized memory format whose layout is designed to maximize the data read efficiency of the tensor core. This conversion process essentially reorganizes and rearranges the data elements in the TLR.

[0066] After the data is converted to usharp format, it is not written back to the TLR, but is directly moved and written to the main memory cache (GMB). Through this step, the correctly formatted data prepared for subsequent tensor core computations is placed in the most efficient memory access location.

[0067] In this embodiment of the invention, by decomposing the loading process into two stages—loading to the TLR and converting and transporting from the TLR to the GMB—complex data format conversion operations can be completed in high-speed on-chip storage, avoiding multiple reads and writes between global memory and main cache, thereby improving the efficiency of the entire data preparation stage.

[0068] Based on any of the above embodiments, step S30 specifically includes: Step S31: Load the data in the main cache into the input cache of the tensor core using the first dedicated instruction; Step S32: Using the second dedicated instruction, perform matrix multiplication and accumulation calculation using the data in the input buffer to generate the result data.

[0069] Specifically, this embodiment of the invention describes the specific data flow and instruction call process when performing matrix multiplication operations within the Tensor Core. After processing in the preceding steps, the sub-block data of the input matrices A and B used for matrix multiplication have been stored in the main cache (GMB) in an efficient second data format (usharp). The GMB serves as the direct data source for Tensor Core computation. The target of data loading is the input cache (GIB) of the Tensor Core. Here, the GIB is a dedicated small-capacity high-speed cache located inside the Tensor Core or very close to the core, serving as the last cache for data before computation. Therefore, data in the GMB can be loaded into the GIB using a first dedicated instruction for subsequent MMA computation.

[0070] Here, the first dedicated instruction refers to a specific machine instruction provided at the hardware level, specifically used to load data from the GMB to the GIB. For example, this instruction could be the Matrix LDMMA instruction. Unlike general loading instructions, the Matrix LDMMA instruction is specially designed to efficiently read usharp-formatted data blocks from the GMB and populate them into the GIB, perfectly matching the computational requirements of the Tensor Core. Execution of this instruction is a prerequisite for achieving the high throughput of the Tensor Core.

[0071] Subsequently, data from the input buffer GIB can be read and MMA calculations performed using a second dedicated instruction. Here, the second dedicated instruction refers to a hardware-level machine instruction that triggers the Tensor Core to perform the actual matrix multiplication and accumulation operation. For example, this instruction could be a Matrix MMA instruction. This instruction drives the Tensor Core to retrieve sub-blocks of input matrices A and B from the GIB, perform a large-scale parallel dot product operation, and then accumulate the results into an internal accumulator.

[0072] After a series of Matrix MMA instructions are executed, the final calculation result is stored in the accumulator of the tensor core, forming the result data. Depending on the hardware design and instruction parameters, this result data can be selectively written to different target locations, such as being directly stored in the TLR or written back to the GMB for subsequent processing.

[0073] It is understood that in this embodiment of the invention, the tl.dot operation of the Triton framework is intelligently converted into the aforementioned first dedicated instruction (such as Matrix LDMMA) and second dedicated instruction (such as Matrix MMA) sequence during the compilation and lowering process. By employing the first and second dedicated instructions, efficient data loading from GMB to GIB and hardware-level matrix multiplication and accumulation calculations based on the Tensor Core are achieved. This method of directly utilizing hardware-specific instructions bypasses the general computational path and maximizes the peak computational performance of the Tensor Core.

[0074] Based on any of the above embodiments, in step S40, converting the result data from the second data format to the first data format includes: Step S41: If the result data is stored in the main cache, then the result data is moved from the main cache to the thread local register. Step S42: Convert the result data in the thread local register from the second data format to the first data format.

[0075] Specifically, as described in the above embodiments, the second dedicated instruction (such as MatrixMMA) that performs matrix multiplication and accumulation calculations can output the generated result data to different hardware locations according to its parameter configuration. One possibility is direct output to the TLR, and another is output to the GMB. Therefore, before performing format conversion, it is necessary to determine the current storage location of the result data.

[0076] When the result data is determined to be located in the GMB, a data transfer operation needs to be performed. This operation reads the result data (still in USARP format) stored in the GMB and writes it to the TLR. It should be understood that the purpose of this transfer is to move the data to a hardware unit more suitable for fine-grained operations (such as format reflow). The TLR is a thread-private high-speed register; performing data format conversion in the TLR avoids access conflicts to the shared GMB resources, and also results in lower latency and higher efficiency.

[0077] After the previous step, it can be ensured that the result data is now located in the TLR (whether it was written directly by the MatrixMMA instruction or moved from the GMB). In the TLR, this result data still maintains the second data format (i.e., usharp). Subsequently, during the process of writing the result data from the TLR to global memory, a reverse format conversion can be performed, that is, converting it from usharp format back to plain buffer format. Specifically, this process reads the usharp format data and rearranges the data elements according to the plain buffer layout rules (such as row-major order). After the conversion is complete, the result data stored in global memory becomes the universal plain buffer format.

[0078] Understandably, the aforementioned conditional checks, data movement, and format conversion operations are transparent to upper-level developers throughout the entire solution. These operations are seamlessly integrated into the compilation and lowering process of the Triton framework's tl.store instruction. When processing the tl.store instruction, the layout information of the resulting data is analyzed. If the data is found in the GMB, a movement instruction from the GMB to the TLR is automatically inserted, followed by a format conversion instruction from usharp to the plain buffer. Finally, an instruction to write the plain buffer data back to global memory is generated.

[0079] Based on any of the above embodiments, the step of converting the data stored in the thread-local register from the first data format to the second data format and moving the converted data to the main cache is implemented by inserting a first layout conversion operation and a second layout conversion operation during the compilation conversion process of performing a loading operation on the input data. The first layout conversion operation is used for format conversion, and the second layout conversion operation is used for data movement.

[0080] Specifically, the process of loading input data from global memory into the GMB and performing format conversion includes two key actions: converting the plain buffer in the TLR to usharp and moving the usharp data to the GMB. These two actions are implemented by introducing specific intermediate representation (IR) operations when the compiler processes loading operations (such as tl.load). To facilitate understanding of the technical solution provided by the embodiments of this invention, the lowering process of the Triton framework will be briefly described below.

[0081] Figure 3 This is a schematic diagram of the compilation and conversion process of the Triton framework provided by this invention, as shown below. Figure 3 As shown, it illustrates the core flow of the Triton compiler framework, which is how user-written, Python-like high-level Triton source code is progressively compiled into instructions executable by the underlying hardware. This process is accomplished through a series of intermediate representations, with specific transformations and optimizations performed at each step.

[0082] The Triton source code (Triton Src) is the starting point of the entire process; it consists of Python functions written by developers using the Triton DSL (Domain-Specific Language). These functions can be captured and compiled by the Triton compiler framework. At this stage, the source code is high-level and platform-independent. When the Triton compiler framework begins compiling the source code, the first step is to parse it into a standard Python Abstract Syntax Tree (AST). The compiler then traverses this AST and converts it into a Triton intermediate representation (TritonIR). At this stage, the IR remains hardware-independent, focusing primarily on the algorithm's logic and data flow, without concern for how data blocks are specifically mapped to GPU threads, thread bundles, or shared memory. Next, the platform-independent Triton IR is further compiled into a platform-dependent intermediate representation, namely the GPU-oriented intermediate representation (Triton GPU IR), which now contains very specific GPU execution information. Finally, the Triton GPU IR is further converted into an LLVM intermediate representation (LLVM IR). Here, LLVM is a mature and widely used compiler backend framework. By generating LLVMIR, the Triton compiler framework can leverage LLVM's powerful backend optimization capabilities to handle low-level optimizations such as register allocation and instruction scheduling. Ultimately, LLVM compiles the LLVM IR into executable code specific to the GPU.

[0083] In the aforementioned compilation and conversion process, a key stage is the conversion from the platform-independent Triton IR to the platform-dependent Triton GPU IR. It is during this stage that the compiler performs layout-related analysis and conversion. In this embodiment of the invention, to achieve the function of converting from TLR to GMB with accompanying format conversion, the compiler introduces two new layout conversion operations on the lowering path of the tl.load instruction: the first layout conversion operation (convert_layout_1) and the second layout conversion operation (convert_layout_2).

[0084] Specifically, `convert_layout_1` converts data from plain buffer format to usharp format, while `convert_layout_2` moves data from the TLR to the GMB. Specifically, the `convert_layout_1` operation takes a tensor with a plain buffer layout in the TLR as input and generates a logical tensor with a usharp layout as output. The implementation of this operation is ultimately translated into a series of data rearrangement instructions that read data from the TLR and reorganize it according to the requirements of the usharp format.

[0085] The `convert_layout_2` operation takes a tensor logically with a USARP layout, generated by the `convert_layout_1` operation, as input and maps its physical location from the TLR to the GMB. Its implementation is ultimately translated into a series of store instructions that write the rearranged data from the TLR to the corresponding address in the GMB.

[0086] The method provided in this invention, by automatically inserting a first layout conversion operation for format conversion and a second layout conversion operation for data transfer during the compilation and conversion process of the tl.load instruction, seamlessly integrates complex data preparation work into the compilation process. This automated and transparent approach significantly lowers the barrier to using hardware acceleration features, allowing developers to continue using high-level, abstract programming interfaces while enjoying the substantial performance gains from underlying hardware optimizations, thus balancing high performance and high availability.

[0087] Based on any of the above embodiments, the steps of moving the result data from the main cache to the thread-local register and converting the result data from the second data format to the first data format are implemented by inserting a third layout conversion operation and a fourth layout conversion operation during the compilation and conversion process of performing storage operations on the result data. The third layout conversion operation is used for data moving, and the fourth layout conversion operation is used for format conversion.

[0088] Specifically, converting the resulting data from GMB and usharp formats back to plain buffer format involves two core steps: moving the data from GMB to TLR and converting usharp to plain buffer within TLR. In this embodiment of the invention, the operations of moving data from GMB to TLR and converting usharp to plain buffer are hidden within the storage operation (i.e., tl.store).

[0089] Similar to the loading operation, when the Triton compiler framework processes a tl.store instruction, it analyzes the layout information of the data to be stored and performs the necessary layout transformations during the conversion from Triton IR to Triton GPU IR. To achieve the reverse process of safely writing the result data back to global memory, the compiler introduces two new layout transformation operations on the lowering path of the tl.store instruction: the third layout transformation operation (convert_layout_3) and the fourth layout transformation operation (convert_layout_4).

[0090] The `convert_layout_3` function is used to move data from the GMB to the TLR. This operation is inserted when the compiler detects that the result data to be stored is located in the GMB. It takes a tensor with a usharp layout in the GMB as input and moves its physical location to the TLR. Its implementation is ultimately translated into a series of load instructions that read the data from the GMB into the TLR.

[0091] `convert_layout_4` is used to convert usharp to plain buffer. This operation takes a tensor with a usharp layout in the TLR as input (this tensor may come from a third layout conversion operation or may be generated directly by the computation unit), and converts its layout to plain buffer format. The implementation of this operation is ultimately converted into a series of data rearrangement instructions, and the reverse format conversion is performed in the TLR.

[0092] Specifically, when processing the `tl.store` command, the data source layout is checked first. If the data is in the GMB, the `convert_layout_3` and `convert_layout_4` operations are inserted sequentially. If the data is already in the TLR but still in usharp format, only the `convert_layout_4` operation needs to be inserted. After these transformations are completed, the `tl.store` command finally writes the plain buffer format data in the TLR back to global memory.

[0093] The method provided in this embodiment of the invention ensures that, regardless of the initial state of the calculation result, it can be correctly and efficiently converted back to the general format by automatically inserting a third layout conversion operation for data transfer and a fourth layout conversion operation for format conversion as needed during the compilation and conversion process of the tl.store instruction.

[0094] Based on any of the above embodiments, this invention provides a method for accelerating matrix multiplication using the Triton framework based on a tensor core. To fully utilize the computational resources of the tensor core, its dedicated MMA instructions are required. However, the MMA instructions have some constraints: First, the Tensor Core has its own storage unit, and its dedicated MMA instructions cannot directly use the data in the TLR for calculation. Therefore, in this embodiment of the invention, the data is first moved from the TLR to the GMB, and then the data is repeatedly read from the GMB and loaded into the GIB for calculation using the dedicated MMA instructions of the Tensor Core (such as Matrix LDMMA).

[0095] Secondly, tensor cores cannot use data in plain buffer format. Therefore, this embodiment of the invention converts the data from plain buffer format to usharp format before computation.

[0096] Based on the above constraints, this embodiment of the invention adds the following operations to the traditional Triton framework matrix multiplication implementation: converting the input data from plain buffer format to usharp format; transferring the data from TLR to GMB; connecting tl.dot to the MMA instructions dedicated to the tensor core; if the output result is stored in GMB, it is also necessary to transfer the data back to TLR; and converting the output result from usharp format to plain buffer format. In the above operations, the plain buffer to usharp conversion and the data transfer from TLR to GMB operations are hidden in the tl.load process. The usharp to plain buffer conversion and the data transfer from GMB to TLR operations are hidden in tl.store.

[0097] like Figure 3 As shown, the lowering process of the Triton framework converts the Triton source code into LLVM IR, which is a multi-stage process. The lowering process from Triton IR to Triton GPU IR incorporates layout analysis and layout transformation operations. In the Triton framework, layout describes the storage layout of tensors in memory. Specifically, layout defines how each dimension of a tensor is mapped to a memory address. This embodiment of the invention implements the following functions during the layout transformation process: mutual conversion between usharp and plain buffer, data transfer from TLR to GMB, and data transfer from GMB to TLR.

[0098] Figure 4 This invention provides a data flow graph for performing matrix multiplication using a tensor core, such as... Figure 4 As shown, when the GPU executes the Triton kernel function (which is used to implement MMA operations), the input matrices A and B, and the output matrix C (corresponding to...) Figure 4 In the diagram, A, B, and C are all stored in global memory in plain buffer format. During the execution of the `tl.load` instruction, the data for input matrices A and B is loaded from global memory into the TLR by default, and the loaded data is in plain buffer format. During the lowering phase of `tl.load`, the plain buffer data is converted to usharp format and moved to the GMB. During the execution of `tl.dot`, the data is loaded from the GMB to the GIB using the Matrix LDMMA instruction, and then the Matrix MMA instruction is called to perform matrix multiplication. The Matrix MMA instruction can control whether the calculation result is written to the TLR or the GMB. During the `tl.store` process, if the calculation result is written to the GMB, the data needs to be moved back to the TLR, then the data in the TLR is converted from usharp to plain buffer, and finally stored in global memory.

[0099] Figure 5 This is a flowchart illustrating the matrix multiplication acceleration method based on the tensor core of the Triton framework provided by this invention, as shown below. Figure 5 As shown, the method includes the following steps: Step 1: Input matrices A and B are stored in global memory in plain buffer format. First, based on the shape of input matrices A and B and the hardware characteristics of the GPU, a reasonable partitioning strategy is selected to divide matrices A and B into blocks. For each sub-block sub_A and sub_B, its block pointer ptr_sub_A and ptr_sub_B need to be calculated.

[0100] Step 2: Execute the `tl.load` instruction to load the sub-blocks `sub_A` and `sub_B` of matrices A and B based on the block pointers `ptr_sub_A` and `ptr_sub_B`. By default, `tl.load` loads data from global memory into the TLR, and the loaded data is in plainbuffer format.

[0101] Step 3: Add convert_layout_1 during the lowering process of tl.load. This operation is used to convert the data from plain buffer format to usharp format and rearrange the data of sub_A and sub_B.

[0102] Step 4: Add convert_layout_2 during the lowering process of tl.load. This operation is used to perform address mapping for sub_A and sub_B in order to move data from TLR to GMB.

[0103] Step 5 involves hooking tl.dot into the Tensor Core's dedicated MMA instructions, including Matrix LDMMA and MatrixMMA. Matrix LDMMA instructions are used to load data from the GMB into the GIB, while Matrix MMA instructions are used to perform matrix multiplication. This process is implemented during the lowering phase of tl.dot.

[0104] Step 6: During the lowering process of tl.store, if the computation result is written to GMB, add convert_layout_3 to move the data from GMB to TLR. If the computation result is written directly to TLR or moved from GMB to TLR, add convert_layout_4 during the lowering process of tl.store to convert the result data from usharp format to plain buffer format to ensure that the result matrix C can be stored in plain buffer format.

[0105] Step 7: Write the calculation results to the corresponding positions in the output matrix C. By default, tl.store will store the data from the TLR to global memory.

[0106] The method provided in this invention, without altering user programming habits (still using advanced load, dot product, and store instructions), automatically handles data format conversion and data transfer during compilation, enabling matrix multiplication operations in the Triton framework, which previously could not utilize the Tensor Core, to fully leverage the Tensor Core's powerful computing capabilities. This solves the problem of Tensor Core idleness and performance limitations caused by mismatched data formats and memory access paths in related technologies, thereby significantly improving the performance and efficiency of matrix multiplication operations while maintaining good programmability and framework compatibility.

[0107] The data processing apparatus provided by the present invention will be described below. The data processing apparatus described below can be referred to in correspondence with the data processing method described above.

[0108] Based on any of the above embodiments Figure 6 This is a schematic diagram of the data processing device provided by the present invention, as shown below. Figure 6 As shown, the device includes: Acquisition unit 610 is used to acquire input data stored in a first data format; The conversion unit 620 is used to convert the input data from the first data format to a second data format during the process of loading the input data into the main cache. The second data format is a hardware-specific data format supported by the Tensor Core. The computation unit 630 is used to perform matrix multiplication operations based on the tensor core and using data stored in the main cache in the second data format to generate result data; The output unit 640 is used to convert the result data from the second data format to the first data format and output the converted result data.

[0109] The apparatus provided in this invention cleverly solves the fundamental problem of being unable to utilize the Tensor Core due to data format incompatibility by converting the data from a general first data format to a hardware-specific second data format supported by the Tensor Core during the process of loading input data into the main cache. This crucial step allows subsequent calculations to be executed smoothly based on the Tensor Core. Since the computational throughput of the Tensor Core is much higher than that of the Vector Core, this invention can fully utilize the peak computing power of the hardware, thereby greatly improving the speed and efficiency of matrix multiplication operations. Furthermore, this invention seamlessly integrates complex format conversion operations into the data loading and output processes. For upper-layer applications or developers, the input data they process and the final result data they obtain are always in the standard first data format. The entire format conversion process is transparent to the user, eliminating the need for developers to manually perform complex data format management and adaptation work. This achieves hardware acceleration and obtains significant performance gains without increasing programming complexity, maintaining good compatibility and ease of use with existing programming frameworks, and resolving the contradiction between high performance and development efficiency.

[0110] Based on any of the above embodiments, the conversion unit 620 is specifically used for: Calculate the block pointer corresponding to the input data, and load the input data from global memory to the thread local register based on the block pointer. The input data is stored in the first data format in both the global memory and the thread local register. The data stored in the thread-local register is converted from the first data format to the second data format, and the converted data is moved to the main cache.

[0111] Based on any of the above embodiments, the step of converting the data stored in the thread-local register from the first data format to the second data format and moving the converted data to the main cache is implemented by inserting a first layout conversion operation and a second layout conversion operation during the compilation conversion process of performing a loading operation on the input data. The first layout conversion operation is used for format conversion, and the second layout conversion operation is used for data movement.

[0112] Based on any of the above embodiments, the arithmetic unit 630 is specifically used for: The data in the main cache is loaded into the input cache of the tensor core using the first dedicated instruction; The second dedicated instruction performs matrix multiplication and accumulation calculations using the data in the input buffer to generate the result data.

[0113] Based on any of the above embodiments, the output unit 640 is specifically used for: If the result data is stored in the main cache, then the result data is moved from the main cache to the thread-local register; Convert the result data in the thread-local register from the second data format to the first data format.

[0114] Based on any of the above embodiments, the steps of moving the result data from the main cache to the thread-local register and converting the result data from the second data format to the first data format are implemented by inserting a third layout conversion operation and a fourth layout conversion operation during the compilation and conversion process of performing storage operations on the result data. The third layout conversion operation is used for data moving, and the fourth layout conversion operation is used for format conversion.

[0115] Based on any of the above embodiments, the acquisition unit 610 is specifically used for: Based on the shape of the input matrix and the hardware characteristics of the computing device, the input matrix is ​​divided into multiple matrix sub-blocks, and these multiple matrix sub-blocks are used as the input data.

[0116] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a data processing method, which includes: acquiring input data stored in a first data format; converting the input data from the first data format to a second data format, wherein the second data format is a hardware-specific data format supported by the Tensor Core, during the process of loading the input data into the main cache; performing matrix multiplication operations based on the Tensor Core using data stored in the main cache in the second data format to generate result data; converting the result data from the second data format back to the first data format, and outputting the converted result data.

[0117] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the data processing methods provided by the above methods. The method includes: acquiring input data stored in a first data format; converting the input data from the first data format to a second data format during the process of loading the input data into the main cache, wherein the second data format is a hardware-specific data format supported by the Tensor Core; performing matrix multiplication operations based on the Tensor Core and using the data stored in the main cache in the second data format to generate result data; converting the result data from the second data format to the first data format and outputting the converted result data.

[0119] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data processing method provided by the above methods. The method includes: acquiring input data stored in a first data format; converting the input data from the first data format to a second data format during the process of loading the input data into a main cache, the second data format being a hardware-specific data format supported by the TensorCore; performing matrix multiplication operations based on the TensorCore and using the data stored in the main cache in the second data format to generate result data; converting the result data from the second data format to the first data format and outputting the converted result data.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that, include: Obtain the input data stored in the first data format; During the process of loading the input data into the main cache, the input data is converted from the first data format to the second data format, which is a hardware-specific data format supported by the Tensor Core. Based on the tensor core, matrix multiplication is performed using data stored in the main cache in the second data format to generate result data. The result data is converted from the second data format to the first data format, and the converted result data is output.

2. The data processing method according to claim 1, characterized in that, The process of loading the input data into the main cache, including converting the input data from the first data format to the second data format, includes: Calculate the block pointer corresponding to the input data, and load the input data from global memory to the thread local register based on the block pointer. The input data is stored in the first data format in both the global memory and the thread local register. The data stored in the thread-local register is converted from the first data format to the second data format, and the converted data is moved to the main cache.

3. The data processing method according to claim 2, characterized in that, The step of converting the data stored in the thread-local register from the first data format to the second data format and moving the converted data to the main cache is achieved by inserting a first layout conversion operation and a second layout conversion operation during the compilation conversion process of loading the input data. The first layout conversion operation is used for format conversion, and the second layout conversion operation is used for data movement.

4. The data processing method according to claim 1, characterized in that, The steps for performing matrix multiplication include: The data in the main cache is loaded into the input cache of the tensor core using the first dedicated instruction; The second dedicated instruction performs matrix multiplication and accumulation calculations using the data in the input buffer to generate the result data.

5. The data processing method according to claim 1, characterized in that, The step of converting the result data from the second data format to the first data format includes: If the result data is stored in the main cache, then the result data is moved from the main cache to the thread-local register; Convert the result data in the thread-local register from the second data format to the first data format.

6. The data processing method according to claim 5, characterized in that, The steps of moving the result data from the main cache to the thread-local register and converting the result data from the second data format to the first data format are implemented by inserting a third layout conversion operation and a fourth layout conversion operation during the compilation and conversion process of performing storage operations on the result data. The third layout conversion operation is used for data moving, and the fourth layout conversion operation is used for format conversion.

7. The data processing method according to any one of claims 1 to 6, characterized in that, The acquisition of input data stored in the first data format includes: Based on the shape of the input matrix and the hardware characteristics of the computing device, the input matrix is ​​divided into multiple matrix sub-blocks, and these multiple matrix sub-blocks are used as the input data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 7.

Images