Data processing method and device based on heterogeneous accelerator, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device based on a heterogeneous accelerator, computer equipment, a readable storage medium and a program product. The method comprises the following steps: obtaining a weight matrix of a second data type obtained by quantization compression of a first data type, determining a processing strategy according to a data matrix of the first data type and the size of the weight matrix, and carrying out inverse quantization compression on the weight matrix to obtain a target weight matrix of the first data type; and processing the data matrix and the target weight matrix through a tensor core in the heterogeneous accelerator according to a processing strategy to obtain a processing result. Compared with the traditional data processing according to the original form of the data, the scheme determines the processing strategy by combining the size of the data matrix and the size of the weight matrix, inversely quantizes the quantized and compressed weight matrix, and realizes the processing of the data matrix and the inversely quantized weight matrix by using the tensor core of the heterogeneous accelerator. The technical effect of reducing video memory occupation during data processing of the heterogeneous accelerator is achieved.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product based on a heterogeneous accelerator. Background Technology

[0002] In recent years, the rapid development of hardware technology has provided powerful computing support for data processing in heterogeneous accelerators. Data processing in heterogeneous accelerators involves a massive number of parameters and computational complexity, often reaching hundreds of millions or even billions of parameters, posing a significant challenge to storage and computing resources. Currently, data processing using heterogeneous accelerators typically follows the original data format. However, processing data in its original format requires handling large volumes of data, resulting in high memory consumption.

[0003] Therefore, heterogeneous accelerators currently suffer from high memory usage when processing data. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on heterogeneous accelerators that can reduce video memory usage in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a data processing method based on heterogeneous accelerators, including:

[0006] Obtain a data matrix of a first data type and a weight matrix corresponding to the data matrix; the weight matrix is ​​a weight matrix of a second data type obtained from the first data type through quantization and compression.

[0007] Based on the size information of the data matrix and the weight matrix, the corresponding processing strategy is determined;

[0008] The weight matrix of the second data type is dequantized and compressed to obtain the target weight matrix of the first data type;

[0009] The data matrix and the target weight matrix are processed by the tensor core in the heterogeneous accelerator according to the processing strategy to obtain the corresponding processing results.

[0010] In this embodiment, the control unit determines the processing strategy by combining the dimensions of the data matrix and the weight matrix, dequantizes the weight matrix after quantization and compression, and uses the tensor core of the heterogeneous accelerator to process the data matrix and the dequantized weight matrix, thereby achieving the technical effect of reducing the memory usage of the heterogeneous accelerator during data processing.

[0011] In one embodiment, the method further comprises:

[0012] Obtain the weight matrix of the first data type corresponding to the data matrix;

[0013] The weight matrix of the first data type is transformed into a row-column-major order and then quantized and compressed based on the second data type to obtain the weight matrix of the second data type.

[0014] In this embodiment, the weight matrix of the first data type is converted into the weight matrix of the second data type by quantization compression, thereby compressing the weight matrix and reducing the memory usage.

[0015] In one embodiment, the step of performing row-major order transformation and quantization compression based on the second data type on the weight matrix of the first data type to obtain the weight matrix of the second data type includes:

[0016] The weight matrix of the first data type is transformed into a row-major order to obtain a column-major order weight matrix;

[0017] Save the weight data of each row of the weight matrix in column-major order, which is set to a predetermined number of columns, as weight data of a third data type;

[0018] Based on a preset number of weighted data of the third data type, a weighted data combination is generated;

[0019] The weight matrix of the second data type is obtained based on each of the weight data combinations, the quantization zeros and scaling factors corresponding to each of the weight data combinations.

[0020] In this embodiment, the control unit performs row and column major order conversion and quantization compression on the weight matrix to obtain a weight matrix of the second data type, thereby reducing the memory usage of the weight matrix during data processing on the heterogeneous accelerator.

[0021] In one embodiment, determining the corresponding processing strategy based on the size information of the data matrix and the weight matrix includes:

[0022] Based on the size information of the data matrix and the weight matrix, it is determined whether the data matrix and the weight matrix should be processed in blocks, and the corresponding target kernel is determined to obtain the processing strategy.

[0023] In this embodiment, the control unit can determine whether to include blocks and determine the target kernel based on the size information of the data matrix and the weight matrix, thereby enabling data processing based on the processing strategy that best matches the data matrix and the weight matrix, thus improving the processing efficiency of heterogeneous accelerators when processing data.

[0024] In one embodiment, determining whether the data matrix and the weight matrix should be partitioned based on their size information includes:

[0025] Obtain the number of columns in the data matrix and the number of rows in the weight matrix;

[0026] Based on the comparison result of the ratio of the number of columns of the data matrix to the number of rows of the weight matrix and the ratio threshold, it is determined whether the data matrix and the weight matrix should be divided into blocks.

[0027] In this embodiment, the control unit can determine whether block processing is required based on the size information of the data matrix and the weight matrix, and thus adopt the corresponding processing method for data matrices and weight matrices of different sizes, which can make full use of the computing power of heterogeneous accelerators and improve the efficiency of data processing.

[0028] In one embodiment, determining whether the data matrix and the weight matrix should be partitioned based on a comparison of the ratio of the number of columns in the data matrix to the number of rows in the weight matrix with a ratio threshold includes:

[0029] If the comparison result is that the ratio is greater than the ratio threshold, then it is determined that the data matrix and the weight matrix are to be processed in blocks based on the block processing function; the block processing function is a function that divides the data matrix and the weight matrix into multiple sub-matrices for processing and calculation.

[0030] If the comparison result is that the ratio is less than or equal to the ratio threshold, then it is determined that the data matrix and the weight matrix are subjected to non-block processing based on the non-block processing function; the non-block processing function is a function that processes and calculates the data matrix and the weight matrix as a whole.

[0031] In this embodiment, the control unit can determine whether block processing is required based on the size information of the data matrix and the weight matrix, and thus adopt the corresponding processing method for data matrices and weight matrices of different sizes, which can make full use of the computing power of heterogeneous accelerators and improve the efficiency of data processing.

[0032] In one embodiment, the step of determining the target kernel includes:

[0033] Obtain the number of rows in the data matrix and the number of columns in the weight matrix;

[0034] Based on the number of rows in the data matrix, the number of columns in the weight matrix, and the preset process matching rules, a query is performed on the composable kernel library to obtain the query results; the preset process matching rules include the matching relationships of each processing process corresponding to each row and column range; the row and column range includes a combination of the row number range and the column number range; the processing processes are stored in the composable kernel library;

[0035] Based on the query results, the target kernel in the composable kernel library corresponding to the target row and column range of the data matrix and the weight matrix is ​​determined.

[0036] In this embodiment, the control unit can use performance analysis tools to evaluate the performance of different sizes and templates in order to select the best kernel for computation, thereby improving the efficiency of data processing in the heterogeneous accelerator.

[0037] Secondly, this application also provides a data processing apparatus based on a heterogeneous accelerator, comprising:

[0038] The acquisition module is used to acquire a data matrix of a first data type and a weight matrix corresponding to the data matrix; the weight matrix is ​​a weight matrix of a second data type obtained from the first data type through quantization and compression.

[0039] The determination module is used to determine the corresponding processing strategy based on the size information of the data matrix and the weight matrix;

[0040] The dequantization module is used to dequantize and compress the weight matrix of the second data type to obtain the target weight matrix of the first data type.

[0041] The processing module is used to process the data matrix and the target weight matrix according to the processing strategy through the tensor core in the heterogeneous accelerator to obtain the corresponding processing results.

[0042] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0043] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0044] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0045] The aforementioned data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on heterogeneous accelerators obtains a weight matrix of a second data type from a first data type through quantization and compression. Based on the dimensions of the data matrix and weight matrix of the first data type, a processing strategy is determined. The weight matrix is ​​then dequantized and compressed to obtain a target weight matrix of the first data type. The data matrix and target weight matrix are then processed by the tensor core in the heterogeneous accelerator according to the processing strategy to obtain the processing result. Compared to traditional data processing based on the original data format, this solution determines the processing strategy by combining the dimensions of the data matrix and weight matrix, dequantizes the quantized and compressed weight matrix, and utilizes the tensor core of the heterogeneous accelerator to process the data matrix and the dequantized weight matrix, thus achieving the technical effect of reducing the memory usage during heterogeneous accelerator data processing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating a data processing method based on heterogeneous accelerators in one embodiment;

[0048] Figure 2 This is a schematic diagram of the quantization compression steps in one embodiment;

[0049] Figure 3 This is a schematic diagram of the block processing steps in one embodiment;

[0050] Figure 4 This is a flowchart illustrating a data processing method based on heterogeneous accelerators in another embodiment;

[0051] Figure 5 This is a structural block diagram of a data processing device based on a heterogeneous accelerator in one embodiment;

[0052] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0055] In related technologies, the rapid development of hardware technology has provided powerful computing support for large-scale model inference. The emergence of graphics processing units (GPUs) and dedicated neural network accelerators has significantly improved the computational speed and efficiency of model inference, making large-scale model inference possible and widely adopted in practical applications.

[0056] As application scenarios become increasingly diverse and task complexity continues to rise, the demand for model performance is becoming more urgent. For example, in the field of computer vision, tasks such as object detection, image segmentation, and face recognition require higher precision. These tasks place higher demands on the perceptual capabilities, accuracy, and robustness of models, thereby driving the widespread application of large-scale model inference.

[0057] The widespread application of large-scale model inference is an inevitable result of the development of deep learning. The rapid growth of data, the enhancement of computing power, and the continuous pursuit of model performance have jointly driven the development and application of large-scale model inference. Large-scale model inference has demonstrated powerful capabilities in fields such as computer vision, natural language processing, speech recognition, and recommendation systems, providing more accurate and intelligent solutions for various industries. With further development of hardware technology and continuous optimization of algorithms, large-scale model inference will continue to play a vital role, providing strong support for future deep learning applications.

[0058] Quantization of deep learning models is the process of converting high-precision floating-point representations into low-precision integer representations. Traditional neural network models typically use 32-bit floating-point numbers to represent weights and activation values, but this consumes a significant amount of storage space and computational resources. Therefore, how to quantize the model to a lower bit depth to reduce storage and computational overhead has become a pressing issue.

[0059] In large-scale model inference, large neural network models have a massive number of parameters and computational complexity, often reaching hundreds of millions or even billions of parameters, posing a huge challenge to storage and computing resources. To address this issue, currently, fp16 (16-bit floating-point numbers) are used to store computations while maintaining accuracy, thereby reducing the use of video memory. However, even so, a model with 7B parameters still requires 14GB of video memory, and a model with 14B parameters requires 32GB of video memory, indicating that the demand on graphics card memory remains very high.

[0060] Therefore, int4 (4-bit integer) quantization has become a highly regarded method. int4 quantization is a method that represents weights and activation values ​​as 4-bit integers. Compared to quantization methods with higher bit counts, int4 quantization can further reduce storage requirements and computational complexity. This is particularly beneficial for inference with large models, as it significantly reduces storage requirements and speeds up computation.

[0061] Int4 dequantization has become an important direction for performance optimization in inference frameworks. The main method involves quantizing the weights using int4 and then dequantizing them to fp16 during computation. However, due to differences in underlying hardware architecture, the instruction size, data layout, and underlying assembly instructions of Tensor Cores vary significantly across different hardware platforms, making it impossible for some inference frameworks to implement int4 dequantization on heterogeneous accelerators.

[0062] The mathematical library used for GEMM (General Matrix to Matrix Multiplication) computation on heterogeneous accelerators does not support input data preprocessing. Furthermore, GEMM computation requires one input to be fp16 and the other to be int4. To achieve high-performance int4 computation on heterogeneous accelerators, int4 dequantization needs to be integrated with GEMM computation to improve computational efficiency and fully leverage the performance advantages of heterogeneous accelerators.

[0063] Based on this, this application determines the processing strategy by combining the size of the data matrix and the weight matrix, dequantizes the weight matrix after quantization and compression, and uses the tensor core of the heterogeneous accelerator to process the data matrix and the dequantized weight matrix, thereby achieving the technical effect of reducing the memory usage of the heterogeneous accelerator during data processing.

[0064] In one embodiment, such as Figure 1As shown, a data processing method based on a heterogeneous accelerator is provided. This embodiment illustrates the application of this method to a control unit of the heterogeneous accelerator. This control unit can be a unit within the heterogeneous accelerator or an external unit used to control the heterogeneous accelerator. It is understood that this method can also be applied to a server, and further to a system including a control unit and a server, and is implemented through the interaction between the control unit and the server, including the following steps S202 to S208. Wherein:

[0065] Step S202: Obtain the data matrix of the first data type and the weight matrix corresponding to the data matrix; the weight matrix is ​​the weight matrix of the second data type obtained from the first data type through quantization and compression.

[0066] Heterogeneous accelerators can be used to handle data processing tasks such as large model inference. The aforementioned data matrix can be a matrix of data related to large model inference. In large model inference data processing tasks, each data point corresponds to a weight matrix. This weight matrix can be obtained through pre-training. The weight matrix trained by the control unit can be a weight matrix of a first data type, which can be pre-compressed. For example, the control unit can quantize and compress the weight matrix of the first data type to obtain a weight matrix of a second data type. The second data type can be a data type with smaller memory footprint than the first data type. For example, the first data type can be fp16 (16-bit floating-point number), and the second data type can be int4 (4-bit integer). Thus, the control unit can store the weight matrix in the heterogeneous accelerator with a smaller size (e.g., int4 type weights), reducing the memory footprint of the weight matrix.

[0067] During large-scale model inference using a heterogeneous accelerator, the control unit needs to perform data processing such as matrix multiplication. Matrix multiplication involves data matrices and weight matrices. The control unit can obtain a data matrix of the first data type and the corresponding weight matrix. The data matrix can be a data matrix related to the large-scale model inference task. The weight matrix can be a weight matrix related to the data in the large-scale model inference task. The weight matrix is ​​used to combine with the data matrix to adjust behavior during large-scale model inference.

[0068] Step S204: Determine the corresponding processing strategy based on the size information of the above data matrix and the above weight matrix.

[0069] The data matrix and weight matrix mentioned above may have corresponding size information, such as the number of rows and columns, respectively. The number of rows and columns in the data matrix may be the same as or different from the number of rows and columns in the weight matrix. The control unit can determine the corresponding processing strategy based on the size information of the data matrix and the inverse weight matrix. The processing strategy represents how to process the data matrix and weight matrix. The processing strategy may include whether to perform block processing on the data matrix and weight matrix, matching the optimal processing template kernel, etc.

[0070] Here, "block processing" indicates whether the data matrix and weight matrix need to be divided into sub-blocks for processing; "kernel" represents a set of processing logic used for data computation and other processing. For example, the control unit can match the composable kernel library in the heterogeneous accelerator based on the above size information to achieve the optimal processing template matching. Through the composable kernel library, the control unit can dynamically assemble low-level operators (such as matrix multiplication and convolution) according to task requirements to achieve extreme efficiency on the heterogeneous accelerator.

[0071] Step S206: Perform dequantization and compression on the weight matrix of the second data type to obtain the target weight matrix of the first data type.

[0072] The aforementioned weight matrix can be stored using a second data type. Before processing, the control unit can dequantize and compress the weight matrix of the second data type, thereby converting it into a target weight matrix of the first data type. For example, the weight matrix of the second data type can be an int4 type weight matrix, and the target weight matrix of the first data type can be an fp16 type weight matrix. Dequantization compression refers to the process of dequantizing the quantized data to recover the original data. By converting the weight matrix of the second data type to the target weight matrix of the first data type, the heterogeneous accelerator can combine the weight matrix of the first data type with the data matrix of the first data type for data processing under the condition of the same data type. Specifically, during matrix multiplication, the control unit dequantizes the int4 weight matrix and merges this dequantized weight matrix with the matrix multiplication.

[0073] Step S208: The tensor core in the heterogeneous accelerator processes the data matrix and the target weight matrix according to the processing strategy described above to obtain the corresponding processing results.

[0074] The aforementioned heterogeneous accelerator incorporates a tensor core, which can be a tensor core within the heterogeneous accelerator, specifically a composable kernel-based tensor core. This tensor core enables mixed-precision computation and can dynamically adjust computational power based on a decrease in precision, improving throughput while maintaining accuracy. The control unit can utilize the tensor core in the heterogeneous accelerator to process the data matrix and the target weight matrix according to the aforementioned processing strategy, thereby obtaining the corresponding processing results. This processing may include matrix multiplication; that is, the control unit can use the tensor core, combined with the processing strategy, to perform matrix multiplication on the data matrix and the target weight matrix, obtaining the corresponding matrix multiplication result as the processing outcome. The control unit can then perform further large-scale model inference based on these matrix multiplication results.

[0075] In the aforementioned data processing method based on heterogeneous accelerators, a weight matrix of a second data type is obtained by quantizing and compressing a first data type. Based on the dimensions of the data matrix and weight matrix of the first data type, a processing strategy is determined. The weight matrix is ​​then dequantized and compressed to obtain the target weight matrix of the first data type. The data matrix and target weight matrix are then processed using the tensor core in the heterogeneous accelerator according to the processing strategy to obtain the processing result. Compared to traditional data processing based on the original data format, this scheme determines the processing strategy by combining the dimensions of the data matrix and weight matrix, dequantizes the quantized and compressed weight matrix, and utilizes the tensor core of the heterogeneous accelerator to process the data matrix and the dequantized weight matrix. This achieves the technical effect of reducing the memory usage during heterogeneous accelerator data processing.

[0076] In one embodiment, the method further includes: obtaining a weight matrix of a first data type corresponding to the data matrix; performing a row-column major order transformation and quantization compression based on the second data type on the weight matrix of the first data type to obtain the weight matrix of the second data type.

[0077] In this embodiment, the control unit can perform data type conversion of the aforementioned weight matrix. For example, converting it from a first data type to a second data type. This data type conversion can be performed after the weight matrix is ​​obtained. The weight matrix obtained by the control unit through training or other methods can be a weight matrix of the first data type. After obtaining the weight matrix of the first data type corresponding to the aforementioned data matrix, the control unit can perform a row-column major order conversion on the weight matrix of the first data type, and then perform quantization compression based on the aforementioned second data type after the row-column major order conversion to obtain the aforementioned weight matrix of the second data type.

[0078] The row-major-major order conversion described above can transform the weight matrix from row-major to column-major order, thereby enabling more efficient matrix multiplication calculations. Furthermore, quantization compression converts the weight matrix of the first data type to the weight matrix of the second data type, thus compressing the weight matrix and reducing memory usage. The row-major-major order conversion and quantization compression can be combined with a third data type. This third data type can be unit32 (a 32-bit, 4-byte unsigned integer).

[0079] In one embodiment, performing row-major-major conversion and quantization compression based on the second data type on the weight matrix of the first data type to obtain the weight matrix of the second data type includes: performing row-major-major conversion on the weight matrix of the first data type to obtain a column-major-major weight matrix; saving the weight data of a preset number of columns in each row of the column-major-major weight matrix as weight data of a third data type; generating weight data combinations based on a preset number of weight data of the third data type; and obtaining the weight matrix of the second data type based on each weight data combination, the quantization zero point corresponding to each weight data combination, and the scaling factor.

[0080] In this embodiment, the control unit can convert the row-major weight matrix into a column-major weight matrix. For example, the control unit can perform a row-major-column-major conversion on the weight matrix of the first data type, that is, convert the row-major weight matrix into a column-major weight matrix to obtain a column-major weight matrix.

[0081] For a column-major weight matrix, the control unit can also save the weight data of a preset number of columns in each row of the column-major weight matrix as weight data of a third data type. For example, for each row in the weight matrix, the control unit saves the weight data of a preset number of columns in that row as weight data of a third data type. If the weight matrix has multiple rows, the control unit can obtain a preset number of weight data of the third data type. The control unit can then generate a weight data combination based on this preset number of weight data of the third data type. That is, the weight data combination includes weight data of a preset number of rows. The control unit can obtain the aforementioned weight matrix of the second data type based on the aforementioned weight data combination, the quantization zero point corresponding to each of the aforementioned weight data combinations, and the scaling factor. Here, the quantization zero point is the base offset value of the integer field, and the scaling factor is a linear scale between the floating-point value and the integer value. The aforementioned quantization zero point and scaling factor can be stored in the weight matrix of the second data type.

[0082] Specifically, the above process of row and column major order conversion and quantization compression can be described as follows: Figure 2 As shown, Figure 2This is a schematic diagram of the quantization compression steps in one embodiment. The heterogeneous accelerator deploys a composable kernel library. Based on the matrix multiplication calculations of the composable kernel library of the heterogeneous accelerator, sorting the weight matrix in column-major order yields better performance in A×weight (matrix multiplication of the data matrix and the weight matrix). Therefore, the control unit can, based on the characteristics of the heterogeneous accelerator, convert the weight matrix to row-major order before execution, and then perform int4 quantization compression after the row-major order conversion to obtain a weight matrix of the second data type. The quantization compression is implemented by storing each row and 8 columns as a uint32 (third data type) weight data. Every 128 rows are treated as a group (weight data combination), and the quantization zero and scales are stored to achieve the conversion of the weight matrix to the second data type.

[0083] Specifically, such as Figure 2 As shown, the initial weight matrix is ​​an fp16 (first data type) matrix, comprising 128 × group_size rows and 12288 columns. Here, group_size represents a pre-defined combination of weight data; for example, every 128 rows form a group_size. The control unit stores each row of 8 columns as a unit32 data unit and treats every 128 rows as a weight data combination, forming a unit32 weight matrix. Simultaneously, it stores zeros (quantization zeros) and scales (scaling parameters), forming a scales_zeros (scaling factor and quantization zero) matrix. This scales_zeros matrix can be of type half2 (two fp16 data units packed together). Thus, the control unit obtains an int4 type weight matrix by performing 8:1 quantization compression.

[0084] Through the above embodiments, the control unit obtains a weight matrix of a second data type by performing row and column major order transformation and quantization compression on the weight matrix, thereby reducing the memory usage of the weight matrix during data processing in the heterogeneous accelerator.

[0085] In one embodiment, determining the corresponding processing strategy based on the size information of the data matrix and the weight matrix includes: determining whether the data matrix and the weight matrix should be processed in blocks based on the size information of the data matrix and the weight matrix, and determining the corresponding target kernel to obtain the processing strategy.

[0086] In this embodiment, the processing strategy can be determined based on size. For example, the control unit can determine the processing strategy based on the size information of the data matrix and the weight matrix. The processing strategy includes various methods, such as whether to perform block processing on the data matrix and the weight matrix, and determining the corresponding target kernel. The control unit can determine whether to perform block processing on the data matrix and the weight matrix, and determine the corresponding target kernel based on the size information of the data matrix and the weight matrix, thus obtaining the processing strategy.

[0087] The aforementioned block processing can be implemented using a split-k (dividing into k blocks) approach. Split-k divides the matrix into multiple parts and performs matrix multiplication on each part using separate threads. Alternatively, without block processing, the control unit performs a global matrix multiplication of the data matrix and weight matrix to achieve data processing. The kernel described above can be a kernel from the composable kernel library. Using the target kernel for data processing improves data processing efficiency.

[0088] Through this embodiment, the control unit can determine whether to include blocks and determine the target kernel based on the size information of the data matrix and the weight matrix, thereby enabling data processing based on the processing strategy that best matches the data matrix and the weight matrix, thus improving the processing efficiency of heterogeneous accelerators when processing data.

[0089] In one embodiment, determining whether the data matrix and the weight matrix should be segmented based on the size information of the data matrix and the weight matrix includes: obtaining the number of columns of the data matrix and the number of rows of the weight matrix; and determining whether the data matrix and the weight matrix should be segmented based on a comparison result between the ratio of the number of columns of the data matrix to the number of rows of the weight matrix and a ratio threshold.

[0090] In this embodiment, the aforementioned size information includes the number of rows and columns of the matrix. The control unit can determine whether block processing is required based on the size information of the data matrix and the weight matrix. For example, the control unit can obtain the number of columns in the data matrix and the number of rows in the weight matrix as the aforementioned size information. Thus, the control unit can obtain the ratio of the number of columns in the data matrix to the number of rows in the weight matrix, and determine whether the data matrix and the weight matrix should be block processed based on the comparison result of this ratio with a ratio threshold. The ratio threshold can be set according to actual conditions. Furthermore, the strategy for whether to perform block processing can differ for different comparison results.

[0091] In one embodiment, determining whether the data matrix and the weight matrix should be segmented based on a comparison between the ratio of the number of columns in the data matrix to the number of rows in the weight matrix and a ratio threshold includes: if the comparison result is that the ratio is greater than the ratio threshold, then determining that the data matrix and the weight matrix should be segmented based on a segmentation processing function; the segmentation processing function is a function that divides the data matrix and the weight matrix into multiple sub-matrices for processing and calculation; if the comparison result is that the ratio is less than or equal to the ratio threshold, then determining that the data matrix and the weight matrix should be non-segmented based on a non-segmentation processing function; the non-segmentation processing function is a function that processes and calculates the data matrix and the weight matrix as a whole.

[0092] In this embodiment, the control unit can determine whether to perform block processing on the data matrix and the weight matrix based on the comparison result of the ratio and the ratio threshold. The determination of whether to perform block processing includes whether to perform block processing on the data matrix and the weight matrix, and whether to process the data matrix and the weight matrix as a whole. If the control unit detects that the comparison result is that the ratio is greater than the ratio threshold, the control unit can determine to perform block processing on the data matrix and the weight matrix based on the block processing function. The block processing function represents a function that divides the data matrix and the weight matrix into multiple sub-matrices for processing and calculation; for example, it can be a function based on split-k.

[0093] If the control unit detects that the comparison result is that the ratio is less than or equal to the ratio threshold, the control unit can determine to perform non-block processing on the data matrix and the weight matrix based on a non-block processing function. Here, the non-block processing function represents a function that processes and calculates the data matrix and the weight matrix as a whole; for example, it can be a function based on non-split-k.

[0094] Specifically, the ratio of the number of columns in the data matrix to the number of rows in the weight matrix can be represented by K. The control unit can determine whether to use the split-k function for block processing based on the input data matrix and the value of K in the weight matrix. For example, if the ratio is greater than a threshold, split-k will not be used for block processing.

[0095] The principle behind split-k is to divide matrix multiplication into multiple smaller k_size operations along the k-axis, thus forming multiple [m, k_tile] and [k_tile, n] matrix multiplications. By shortening the k-loop size of each matrix multiplication, the computation time of each thread can be reduced, and more threads can be created to perform computations in parallel. Figure 3 As shown, Figure 3This is a schematic diagram of the block processing steps in one embodiment. To ensure better performance, when performing matrix multiplication calculations, larger matrix multiplications of [m_tile,k] and [k,n_tile] can be calculated using thread blocks. Here, [m_tile,k] represents a submatrix that divides the complete matrix into multiple rows, each submatrix having m_tile rows and k columns; [k,n_tile] represents a submatrix that divides the complete matrix into multiple columns, each submatrix having k rows and n_tile columns. The control unit can then distribute these calculations to each thread, allowing each thread to handle the smaller [m_tile,k] and [k,n_tile] submatrix multiplications. The control unit computes multiple smaller k matrix multiplications in parallel using a tensor core and adds an additional reducesum operator for accumulation calculations.

[0096] In this case, the split-k function encounters a performance bottleneck when M (number of rows) and N (number of columns) are small but k is large. Because only a limited number of threads and thread blocks can be allocated, the number of loops within each thread increases significantly, leading to underutilization of resources and thus limiting the performance of matrix multiplication. In other words, when N is large, the performance of split-k may be worse than not using split-k. In this situation, the control unit can use a profiler (performance analysis tool) to analyze matrices of different sizes to select the optimal implementation method.

[0097] Through the above embodiments, the control unit can determine whether block processing is required based on the size information of the data matrix and weight matrix, and thus adopt the corresponding processing method for data matrices and weight matrices of different sizes, which can make full use of the computing power of heterogeneous accelerators and improve the efficiency of data processing.

[0098] In related technologies, the mathematical library of heterogeneous accelerators will degrade in performance with each iteration update. If the optimal kernel cannot be matched, it needs to be retrained to give reasonable performance.

[0099] In one embodiment, the step of determining the target kernel includes: obtaining the number of rows in the data matrix and the number of columns in the weight matrix; querying the composable kernel library based on the number of rows in the data matrix, the number of columns in the weight matrix, and a preset process matching rule to obtain a query result; the preset process matching rule includes the matching relationship of each processing process corresponding to each row and column range; the row and column range includes a combination of the row range and the column range; the processing process is stored in the composable kernel library; and based on the query result, determining the target kernel in the composable kernel library corresponding to the target row and column range where the number of rows in the data matrix and the number of columns in the weight matrix are located.

[0100] In this embodiment, the control unit can determine the optimal kernel by combining the size information of the data matrix and the weight matrix. The heterogeneous accelerator includes a composable kernel library. This library contains multiple processing flows, each representing the processing logic for matrix multiplication of the data matrix and weight matrix. The control unit can also pre-build preset flow matching rules. These rules include matching relationships between different processing flows corresponding to various row and column ranges, where each row and column range corresponds to a processing flow.

[0101] The control unit can query the composable kernel library based on the number of rows in the data matrix, the number of columns in the weight matrix, and preset process matching rules to obtain query results. These query results can include the range of rows and columns that the data matrix and weight matrix fall into. The control unit can then determine the target range of rows and columns that the data matrix and weight matrix fall into based on the query results, and determine the corresponding target kernel in the composable kernel library based on this target range.

[0102] Specifically, the aforementioned composable kernel library can be a composable kernel library. Composable kernel libraries based on heterogeneous accelerators can use tensor cores to perform matrix multiplication, thereby improving the performance of the fused matrix multiplication. Here, the number of rows in the data matrix can be represented as M, and the number of columns in the weight matrix can be represented as N. Different values ​​of M and N result in significant performance differences across different composable kernel libraries. The control unit can perform a performance analysis to select the optimal template (kernel). For example, the control unit can concatenate a string Ma-b_Nc-d as an index based on the ranges of M and N to select the kernel. Here, Ma-b_Nc-d represents the string index used to match the range of rows and columns, Ma_b indicates the row range is between a and b, and Nc-d indicates the column range is between c and d. The control unit can use a map operation to determine the target kernel, reducing the selection time. If no target row and column range is matched, the control unit can choose a default template for computation.

[0103] Through this embodiment, the control unit can use performance analysis tools to evaluate the performance of different sizes and templates, select the best kernel for computation, and improve the efficiency of data processing in heterogeneous accelerators.

[0104] In one exemplary embodiment, such as Figure 4 As shown, Figure 4This is a flowchart illustrating a data processing method based on a heterogeneous accelerator in another embodiment. It includes steps S300 to S312. Wherein:

[0105] Step S300: Perform quantization conversion on the weight, converting row-major quantization to column-major quantization.

[0106] Here, weight represents the weight matrix.

[0107] Step S302: Input input(fp16) and weight(int4).

[0108] Here, input(fp16) represents a data matrix of type fp16, and weight(int4) represents a weight matrix of type int4.

[0109] Step S304: Determine whether to use split-k based on the input dimension information.

[0110] Step S306, Do not use split-k: Select a non-split function.

[0111] Step S308, use split-k: select the function for split.

[0112] Step S310: Perform performance analysis and select the computational template kernel with the best performance under different sizes.

[0113] Step S312: Dequantize weight(int4) to obtain data for fp16, and select the optimal template kernel for matrix multiplication.

[0114] Through the above embodiments, the control unit determines the processing strategy by combining the dimensions of the data matrix and the weight matrix, dequantizes the weight matrix after quantization and compression, and uses the tensor core of the heterogeneous accelerator to process the data matrix and the dequantized weight matrix, thereby achieving the technical effect of reducing the memory usage of the heterogeneous accelerator during data processing.

[0115] Furthermore, optimal performance is achieved by converting the weight quantization method to a tensor core computation layout suitable for heterogeneous accelerators. Based on a composable kernel library supported by heterogeneous accelerators, a fusion computation of int4 dequantization and matrix multiplication is implemented. The split-k method is employed to maximize the utilization of computational resources and fully leverage the performance of heterogeneous accelerators.

[0116] Furthermore, by changing the weight quantization method from row-major to column-major, this method, based on the ComposableKernel's Tensor Core technology, integrates int4 dequantization and GEMM computation on heterogeneous accelerators. In addition, by using a Profiler (performance analysis tool), the optimal computation template (kernel) can be selected based on data of different sizes, thereby further improving computational efficiency and achieving more efficient matrix multiplication processing.

[0117] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0118] Based on the same inventive concept, this application also provides a heterogeneous accelerator-based data processing apparatus for implementing the above-described heterogeneous accelerator-based data processing method. The solution provided by this apparatus is similar to the implementation described in the above-described method. Therefore, the specific limitations in one or more embodiments of the heterogeneous accelerator-based data processing apparatus provided below can be found in the limitations of the heterogeneous accelerator-based data processing method described above, and will not be repeated here.

[0119] In one exemplary embodiment, such as Figure 5 As shown, a data processing device based on a heterogeneous accelerator is provided, including: an acquisition module 500, a determination module 502, an inverse quantization module 504, and a processing module 506, wherein:

[0120] The acquisition module 500 is used to acquire the data matrix of the first data type and the weight matrix corresponding to the data matrix; the weight matrix is ​​the weight matrix of the second data type obtained from the first data type through quantization and compression.

[0121] The determination module 502 is used to determine the corresponding processing strategy based on the size information of the above data matrix and the above weight matrix.

[0122] The dequantization module 504 is used to dequantize and compress the weight matrix of the second data type to obtain the target weight matrix of the first data type.

[0123] The processing module 506 is used to process the data matrix and the target weight matrix through the tensor core in the heterogeneous accelerator according to the processing strategy to obtain the corresponding processing results.

[0124] In one embodiment, the apparatus further includes: a quantization module, configured to obtain a weight matrix of a first data type corresponding to the data matrix; perform row and column major order conversion on the weight matrix of the first data type and quantization compression based on the second data type to obtain the weight matrix of the second data type.

[0125] In one embodiment, the quantization module is used to perform a row-major-major conversion on the weight matrix of the first data type to obtain a column-major-major weight matrix; save the weight data of a preset number of columns in each row of the column-major-major weight matrix as weight data of the third data type; generate weight data combinations based on a preset number of weight data of the third data type; and obtain the weight matrix of the second data type based on each weight data combination, the quantization zero point and scaling factor corresponding to each weight data combination.

[0126] In one embodiment, the determining module 502 is used to determine whether the data matrix and the weight matrix should be processed in blocks based on the size information of the data matrix and the weight matrix, and to determine the corresponding target kernel to obtain the processing strategy.

[0127] In one embodiment, the determining module 502 is used to obtain the number of columns of the data matrix and the number of rows of the weight matrix; and to determine whether the data matrix and the weight matrix should be segmented based on the comparison result of the ratio of the number of columns of the data matrix to the number of rows of the weight matrix and the ratio threshold.

[0128] In one embodiment, the determining module 502 is configured to: if the comparison result is that the ratio is greater than the ratio threshold, determine to perform block processing on the data matrix and the weight matrix based on a block processing function; the block processing function is a function that divides the data matrix and the weight matrix into multiple sub-matrices for processing and calculation; if the comparison result is that the ratio is less than or equal to the ratio threshold, determine to perform non-block processing on the data matrix and the weight matrix based on a non-block processing function; the non-block processing function is a function that processes and calculates the data matrix and the weight matrix as a whole.

[0129] In one embodiment, the determining module 502 is used to obtain the number of rows in the data matrix and the number of columns in the weight matrix; query the composable kernel library according to the number of rows in the data matrix, the number of columns in the weight matrix, and a preset process matching rule to obtain a query result; the preset process matching rule includes the matching relationship of each processing process corresponding to each row and column range; the row and column range includes a combination of the row number range and the column number range; the processing process is stored in the composable kernel library; and based on the query result, determine the target kernel in the composable kernel library corresponding to the target row and column range where the number of rows in the data matrix and the number of columns in the weight matrix are located.

[0130] The modules in the aforementioned data processing device based on heterogeneous accelerators can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0131] In one exemplary embodiment, a computer device is provided, which may be a heterogeneous accelerator, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores kernel data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a data processing method based on a heterogeneous accelerator.

[0132] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0133] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described data processing method based on a heterogeneous accelerator.

[0134] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described data processing method based on a heterogeneous accelerator.

[0135] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described heterogeneous accelerator-based data processing method.

[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data processing method based on heterogeneous accelerators, characterized in that, The method includes: Obtain a data matrix of a first data type and a weight matrix corresponding to the data matrix; the weight matrix is ​​a weight matrix of a second data type obtained from the first data type through quantization and compression. Based on the size information of the data matrix and the weight matrix, the corresponding processing strategy is determined; The weight matrix of the second data type is dequantized and compressed to obtain the target weight matrix of the first data type; The data matrix and the target weight matrix are processed by the tensor core in the heterogeneous accelerator according to the processing strategy to obtain the corresponding processing results.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the weight matrix of the first data type corresponding to the data matrix; The weight matrix of the first data type is transformed into a row-column-major order and then quantized and compressed based on the second data type to obtain the weight matrix of the second data type.

3. The method according to claim 2, characterized in that, The step of performing row-major order transformation and quantization compression based on the second data type on the weight matrix of the first data type to obtain the weight matrix of the second data type includes: The weight matrix of the first data type is transformed into a row-major order to obtain a column-major order weight matrix; Save the weight data of each row of the weight matrix in column-major order, which is set to a predetermined number of columns, as weight data of a third data type; Based on a preset number of weighted data of the third data type, a weighted data combination is generated; The weight matrix of the second data type is obtained based on each of the weight data combinations, the quantization zeros and scaling factors corresponding to each of the weight data combinations.

4. The method according to claim 1, characterized in that, The step of determining the corresponding processing strategy based on the size information of the data matrix and the weight matrix includes: Based on the size information of the data matrix and the weight matrix, it is determined whether the data matrix and the weight matrix should be processed in blocks, and the corresponding target kernel is determined to obtain the processing strategy.

5. The method according to claim 4, characterized in that, The step of determining whether the data matrix and the weight matrix should be partitioned based on their size information includes: Obtain the number of columns in the data matrix and the number of rows in the weight matrix; Based on the comparison result of the ratio of the number of columns of the data matrix to the number of rows of the weight matrix and the ratio threshold, it is determined whether the data matrix and the weight matrix should be divided into blocks.

6. The method according to claim 5, characterized in that, The step of determining whether the data matrix and the weight matrix should be partitioned based on a comparison of the ratio of the number of columns in the data matrix to the number of rows in the weight matrix with a ratio threshold includes: If the comparison result is that the ratio is greater than the ratio threshold, then it is determined that the data matrix and the weight matrix are to be processed in blocks based on the block processing function; the block processing function is a function that divides the data matrix and the weight matrix into multiple sub-matrices for processing and calculation. If the comparison result is that the ratio is less than or equal to the ratio threshold, then it is determined that the data matrix and the weight matrix are subjected to non-block processing based on the non-block processing function; the non-block processing function is a function that processes and calculates the data matrix and the weight matrix as a whole.

7. The method according to claim 4, characterized in that, The steps for determining the target kernel include: Obtain the number of rows in the data matrix and the number of columns in the weight matrix; Based on the number of rows in the data matrix, the number of columns in the weight matrix, and the preset process matching rules, a query is performed on the composable kernel library to obtain the query results; the preset process matching rules include the matching relationships of each processing process corresponding to each row and column range; the row and column range includes a combination of the row number range and the column number range; the processing processes are stored in the composable kernel library; Based on the query results, the target kernel in the composable kernel library corresponding to the target row and column range of the data matrix and the weight matrix is ​​determined.

8. A data processing device based on a heterogeneous accelerator, characterized in that, The device includes: The acquisition module is used to acquire a data matrix of a first data type and a weight matrix corresponding to the data matrix; the weight matrix is ​​a weight matrix of a second data type obtained from the first data type through quantization and compression. The determination module is used to determine the corresponding processing strategy based on the size information of the data matrix and the weight matrix; The dequantization module is used to dequantize and compress the weight matrix of the second data type to obtain the target weight matrix of the first data type. The processing module is used to process the data matrix and the target weight matrix according to the processing strategy through the tensor core in the heterogeneous accelerator to obtain the corresponding processing results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images