Computing acceleration method, and GPU, acceleration operator, computer program product and storage medium

Abstract

Provided in the embodiments of the present disclosure are a computing acceleration method, and a GPU, an acceleration operator, a computer program product and a storage medium. In the embodiments, it is proposed that a sparse matrix and dense matrix multiplication function is called to process a GEMV computing task, such that a reordering capability of the function for a sparse matrix can be used so as to convert the GEMV computing task into a tensor computing task that can fully utilize the computing power of a tensor core. In this way, when the tensor core is used to process the GEMV computing task, the computing power of the tensor core is not wasted, and compared with a general computing core, the tensor core can be utilized in such a way that the processing time of the GEMV computing task can also be effectively shortened.

Description

[0001] A computing acceleration method, GPU, acceleration operator, computer program product and storage medium

[0002]

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a computing acceleration method, GPU, acceleration operator, computer program product and storage medium.

[0003]

[0002] General Matrix-Vector Multiplication (GEMV), that is, calculating the product of a matrix A and a vector x, is a computing operation in a parallel computing scenario.

[0004]

[0003] In a computing-intensive scenario such as model inference, relying only on a CPU to perform computing cannot meet the computing performance requirement, and a Graphics Processing Unit (GPU) is suitable for processing a computing-intensive task, so the GPU is often used for computing acceleration in a computing-intensive scenario. The GEMV computing described above can be processed by the GPU.

[0005]

[0004] At present, when processing a GEMV computing task based on the GPU, a general computing core in the GPU is usually used to complete the computing. However, because the general computing core can only perform one computation per clock cycle, and the number of general computing cores in the GPU and the computing speed of each general computing core are limited, the processing efficiency of the GPU for the GEMV computing task is not good.

[0006]

[0005] Aspects of the present disclosure provide a computing acceleration method, GPU, acceleration operator, computer program product and storage medium to improve the processing efficiency of the GPU for the GEMV computing task.

[0007]

[0006] An embodiment of the present disclosure provides a computing acceleration method suitable for a GPU, the GPU being equipped with a tensor core, and the method comprising: performing zero value padding on a vector indicated in a target computing task according to a matrix format required by the tensor core to obtain a first matrix, the target computing task being a matrix vector multiplication computing task; calling a sparse matrix and dense matrix multiplication function for the first matrix and a second matrix indicated in the target computing task to construct a tensor computing task; and processing the constructed tensor computing task by using the tensor core to complete the target computing task.

[0008]

[0007] Further, according to the matrix format required by the tensor core, zero-value filling is performed on the vector indicated in the target computation task, including: if the matrix format is n*n, when the vector is a row vector, the row vector is filled to n rows by zero values ​​to obtain the first matrix; when the vector is a column vector, the column vector is filled to n columns by zero values ​​to obtain the first matrix, where n is a positive integer.

[0009]

[0008] Further, calling the sparse matrix and dense matrix multiplication function to construct the tensor computation task includes: calling the sparse matrix and dense matrix multiplication function from the function library used for processing sparse matrix computation to construct the tensor computation task.

[0010]

[0009] Further, processing the constructed tensor computation task using the tensor core includes: scheduling the target computation task to a first stream processor group included in the GPU; and processing the constructed tensor computation task using the tensor core provided in the first stream processor group.

[0011]

[0010] Further, before filling the vector indicated in the target computation task according to the matrix format required by the tensor core, the method further includes: in response to receiving a matrix-vector multiplication computation task group, if it is determined that the matrix-vector multiplication computation task group meets the triggering condition for enabling the tensor core for acceleration, then dividing a first task flow from the matrix-vector multiplication computation task group as the target computation task.

[0012]

[0011] Further, the method also includes: dividing a second task flow from the matrix-vector multiplication calculation task group; calling a matrix-vector multiplication function for the matrix and vector indicated in the second task flow to construct a general computing task; and using the general computing core assembled in the GPU to process the constructed general computing task to complete the second task flow.

[0013]

[0012] Further, calling the matrix-vector multiplication function to construct a general computing task includes: calling the matrix-vector multiplication function from a function library for processing linear algebra calculations to construct a general computing task.

[0014]

[0013] Further, the general-purpose computing task constructed is processed by using the general-purpose computing core equipped in the GPU, including: scheduling the second task flow to a second group of stream processors included in the GPU; and processing the general-purpose computing task constructed by using the general-purpose computing core provided in the second group of stream processors.

[0015]

[0014] Further, the method further includes: if the first task flow and the second task flow are scheduled to the same group of stream processors, detecting register occupancy in the group of stream processors; and if it is detected that the register occupancy exceeds a preset occupancy threshold, rescheduling the first task flow and the second task flow.

[0016]

[0015] Further, the first task flow is divided from the group of matrix-vector multiplication computing tasks, and the second task flow is divided from the group of matrix-vector multiplication computing tasks, including: creating a first task flow, a second task flow, a first handle for identifying a tensor core, and a second handle for identifying a general-purpose computing core; assigning the first handle to the first task flow and the second handle to the second task flow; and dividing a plurality of pairs of matrices and vectors included in the group of matrix-vector multiplication computing tasks into the first task flow and the second task flow.

[0017]

[0016] Further, the trigger condition for enabling the tensor core to accelerate includes: a usage rate of a stream multiprocessor in the GPU being lower than a preset usage rate threshold; and / or a precision of matrices and vectors in the group of matrix-vector multiplication computing tasks meeting a precision requirement of the tensor core; and / or a size of the matrices and vectors in the group of matrix-vector multiplication computing tasks exceeding a preset size threshold.

[0018]

[0017] Embodiments of the present disclosure further provide an acceleration operator including one or more computer instructions, and when the GPU invokes the acceleration operator, the one or more computer instructions are executed to perform the aforementioned computing acceleration method to accelerate the matrix-vector multiplication computing task by using the tensor core in the GPU.

[0019]

[0018] Embodiments of the present disclosure further provide a GPU including a memory, a controller, and a tensor core; the memory is configured to store one or more computer instructions; the controller is coupled to the memory and is configured to execute the one or more computer instructions to perform the aforementioned computing acceleration method to accelerate the matrix-vector multiplication computing task by using the tensor core.

[0020]

[0019] This disclosure also provides a computer-readable storage medium for storing a computer program, which, when executed by one or more processors, causes the one or more processors to perform the aforementioned computation acceleration method.

[0021]

[0020] This disclosure also provides a computer program product, including a computer program that, when executed by one or more processors, causes the one or more processors to perform the aforementioned computation acceleration method.

[0022]

[0021] In this embodiment of the present disclosure, a scheme for accelerating GEMV computation tasks based on GPUs is proposed. The target computation task indicates vectors and matrices that need to be multiplied. For the indicated vectors, the vectors can be filled into matrices according to the matrix format required by the tensor cores installed in the GPU. Based on this, the sparse matrix multiplication function can be called to construct the original GEMV computation task into a parallel tensor computation task, and the tensor cores installed in the GPU are used to process the constructed tensor computation task to complete the target computation task. Accordingly, this embodiment proposes to call the sparse matrix multiplication function to process the GEMV computation task, thereby utilizing the function's ability to rearrange sparse matrices to convert the GEMV computation task into a tensor computation task that can fully utilize the computing power of the tensor cores. This means that when using tensor cores to process GEMV computation tasks, not only is the computing power of the tensor cores not wasted, but the processing time of GEMV computation tasks can also be effectively shortened compared to general-purpose computing cores. (See attached figures)

[0023]

[0022] The accompanying drawings, which are included to provide a further understanding of this disclosure and constitute a part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0024]

[0023] Figure 1 is a flowchart illustrating a computation acceleration method provided in an exemplary embodiment of the present disclosure;

[0024] Figure 2 is a logical schematic diagram illustrating a computation acceleration method provided in an exemplary embodiment of the present disclosure;

[0025]

[0025] Figure 3 is a flowchart illustrating another computation acceleration method provided in an exemplary embodiment of the present disclosure;

[0026]

[0026] Figure 4 provides a logical diagram of whether a matrix-vector multiplication calculation task group meets the triggering conditions for enabling tensor cores for acceleration;

[0027]

[0027] Figure 5 is a logical schematic diagram of an exemplary implementation scheme for dividing a first task flow and a second task flow according to an exemplary embodiment of the present disclosure;

[0028]

[0028] Figure 6 is a schematic diagram of the structure of a GPU provided in another exemplary embodiment of this disclosure. Detailed Description

[0029]

[0029] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0030]

[0030] Before proceeding with a detailed description of the technical solutions provided in the various embodiments of this disclosure, the following is a brief explanation of several technical concepts involved in this disclosure.

[0031]

[0031] GEMV: A term in linear algebra that stands for "General Matrix-Vector Multiplication". This is a computational operation involving a two-dimensional matrix (M x N) and a one-dimensional vector.

[0032] (N-dimensional) vectors, when multiplied, produce another one-dimensional vector (M-dimensional). The GEMV computation task will indicate the vectors and matrices that need to be multiplied, and of course, it can also indicate the specific multiplication rules, etc.

[0033]

[0032] A stream processor (SP) is a processing unit on a GPU used to perform computational tasks, similar to a core in a CPU. Stream processors can be classified into various types, including at least general-purpose computing cores and tensor cores.

[0034]

[0033] A general-purpose computing core is a general-purpose computing unit on a graphics processing unit (GPU). It has the ability to process various types of computing tasks and is a computing unit commonly deployed in GPUs.

[0035]

[0034] Tensor cores are a special type of computing unit on a GPU, specifically designed for efficient processing of tensor calculations in deep learning. Typically, tensor cores have a computing power that is 2-16 times higher than general-purpose computing cores, thus offering higher computational efficiency.

[0036]

[0035] As described in the background section, currently, general-purpose computing cores are typically used to handle GEMV computation tasks. The reason tensor cores are not used is that tensor core computation requires block-based input and output, and correspondingly, it also requires block-based hardware resource allocation. Therefore, using tensor cores to perform GEMV computation results in a waste of hardware resources. For example, the block size supported by a tensor core is 16*16, meaning it can compute 16 rows of results at a time. However, the effective number of rows in a vector involved in GEMV is only 1. Therefore, the effective number of rows in the input block generated after dividing the vector is 1, but each input block still requires 16 rows of hardware resources. Thus, for a tensor core, its actual hardware utilization is only 1 / 16. The other 15 / 16 of the computing power is used, but no meaningful computation is performed. Therefore, using tensor cores to handle GEMV computation tasks results in a waste of computing power. While general-purpose computing cores have slightly lower computing power, it's not by much. However, they don't suffer from the same wasteful hardware utilization. Therefore, considering both processing time and hardware utilization, general-purpose computing cores are more suitable for handling GEMV computing tasks compared to tensor cores.

[0037]

[0036] This embodiment proposes a computing acceleration scheme, which aims to overcome the above-mentioned technical bias. Instead, it uses tensor cores to process GEMV computing tasks, thereby utilizing the higher computing power of tensor cores to improve the processing efficiency of GEM computing tasks, but without wasting the computing power of tensor cores.

[0038]

[0037] Figure 1 is a schematic flowchart of a computation acceleration method provided in an exemplary embodiment of this disclosure. This method can be executed by an acceleration operator, which can be implemented as software, hardware, or a combination of both. The acceleration operator can be integrated into a GPU. In this embodiment, existing acceleration operators, specifically the GEMV acceleration operator, can be modified to implement the computation acceleration method provided in this embodiment. The GEMV acceleration operator can be understood as an acceleration operator that is called by the GPU and used to process GEMV computation tasks. Referring to Figure 1, the method may include the following steps.

[0039]

[0038] Step 100: Fill the vectors indicated in the target computation task with zero values ​​according to the matrix format required by the tensor core to obtain the first matrix. The target computation task is a matrix-vector multiplication computation task.

[0040]

[0039] Step 101: For the first matrix and the second matrix indicated in the target computation task, call the sparse matrix and dense matrix multiplication function to construct the tensor computation task.

[0041]

[0040] Step 102: Use the tensor core to process the constructed tensor computation task to complete the target computation task.

[0042]

[0041] FIG2 is a logical schematic diagram of a computing acceleration method provided by an exemplary embodiment of the present disclosure. Referring to FIG2, in this embodiment, the GPU is equipped with tensor cores.

[0043]

[0042] In this embodiment, the target computation task can be any matrix-vector multiplication computation task determined to require processing using a tensor kernel, hereinafter described as a GEMV computation task. Referring to the preceding explanation of the concept of a GEMV computation task, in this embodiment, the target computation task indicates vectors and matrices that require multiplication. It is worth noting that the vectors and matrices indicated in the target computation task in this embodiment can be one or more pairs. In the case of multiple pairs, this embodiment supports parallel processing; that is, it supports performing multiplication computations on multiple pairs of vectors and matrices contained in the target computation task in parallel.

[0044]

[0043] In this embodiment, a function library for handling sparse matrix calculations is proposed to process the GEMV calculation task and call the tensor kernel. Two technical concepts need explanation here: sparse matrix and dense matrix. When the proportion of non-zero elements in a matrix is ​​high (usually greater than 1 / 3), it can be called a dense matrix; while when the proportion of non-zero elements is low (usually less than 1%), it is called a sparse matrix. The function library for handling sparse matrix calculations provides various functions for this purpose. It should be understood that the function library for handling sparse matrix calculations is usually only used to handle calculation tasks related to sparse matrices. However, the vectors and matrices in the GEMV calculation task in this embodiment are dense; therefore, the GEMV calculation task will not currently be handled using the function library for handling sparse matrix calculations.

[0045]

[0044] In this embodiment, for the first time, a technical concept is proposed to introduce a function library for handling sparse matrix calculations to process GEMV calculation tasks and call the tensor core. It should be understood that here, the function library for handling sparse matrix calculations provides usage rights to the scheduling method of this embodiment, that is, the scheduling method in this embodiment has the permission to use this type of function library.

[0046]

[0045] Based on this, referring to Figure 1, it is proposed in step 100 that the vectors indicated in the target computation task can be filled with zeros according to the matrix format required by the tensor core to obtain the first matrix. As mentioned above, the computation of the tensor core requires block input and output, and the matrix format here is the block size required by the tensor core, for example, it can usually be 8*8 or 16*16, etc.

[0047]

[0046] Here, the first matrix obtained by filling the vector indicated in the target computation task with zero values ​​will be a sparse matrix. In step 100, the GEMV computation task can be converted into a computation task related to sparse matrices, which creates a foundation for introducing a function library for processing sparse matrix computations to handle the GEMV computation task.

[0048]

[0047] During the research process, the inventors discovered that the vector indicated in the target computation task may be a row vector or a column vector. In response to these two possibilities, this embodiment proposes that: if the matrix format required by the tensor core is n*n, when the vector indicated in the target computation task is a row vector, the row vector can be filled to n rows by zero values ​​to obtain the first matrix; when the vector indicated in the target computation task is a column vector, the column vector can be filled to n columns by zero values ​​to obtain the first matrix, where n is a positive integer.

[0049]

[0048] Continuing to refer to Figure 1, in step 101, the sparse matrix multiplication function can be called for the first matrix and the second matrix indicated in the target computation task to construct the tensor computation task.

[0050]

[0049] As mentioned above, the vectors and matrices indicated in the target computation task are dense. Therefore, the second matrix here will be a dense matrix. As mentioned earlier, the first matrix generated in step 100 is a sparse matrix. Therefore, in step 101, the sparse matrix and dense matrix multiplication function can be called to handle the multiplication calculation between the first matrix and the second matrix. Moreover, this function has the ability to arrange sparse matrices and call tensor cores. This allows this function to arrange the multiplication calculation between sparse and dense matrices into parallel tensor computation tasks. The arranged tensor computation tasks conform to the block specifications required by the tensor core (i.e., the matrix format mentioned above). The tensor computation tasks provided by this function to the tensor core usually do not contain rows or columns of all zeros in the input blocks.

[0051]

[0050] The internal principle of the sparse matrix and dense matrix multiplication function can be roughly understood as follows: Zero values ​​in the sparse matrix are skipped, and the multiplications between the non-zero values ​​in the sparse matrix and the dense matrix are arranged as multiplications between input blocks. Each pair of input blocks corresponds to a tensor computation task. It can be seen that the multiplication between input blocks is a re-representation of the multiplication between sparse and dense matrices. The element arrangement order in the input block may no longer be the same as the original element arrangement order in the sparse and dense matrices. However, as mentioned above, each input block conforms to the block specifications required by the tensor core, and there are usually no rows or columns of all zeros in the input block. Therefore, each tensor computation task arranged can fully utilize the computing power of the tensor core without wasting computing power. Here, the internal workings of the sparse and dense matrix multiplication functions are merely illustrative, and this embodiment is not limited thereto. The internal workings of sparse and dense matrix multiplication functions provided by different vendors may differ. This embodiment will not elaborate further on the internal workings of the sparse and dense matrix multiplication functions. This embodiment primarily utilizes the aforementioned arrangement capabilities and tensor core calling capabilities of the sparse and dense matrix multiplication functions.

[0052]

[0051] Thus, in step 101, a parallel tensor computation task can be constructed based on the first and second matrices by calling the sparse matrix and dense matrix multiplication function. This can convert the target computation task into a parallel tensor computation task, and the generated tensor computation tasks can fully utilize the computing power of the tensor core without wasting computing power.

[0053]

[0052] Preferably, in this embodiment, a sparse matrix multiplication function can be called from a function library for processing sparse matrix calculations to construct a tensor calculation task. The function library for processing sparse matrix calculations contains several functions related to sparse matrix calculations, including the sparse matrix multiplication function to be called in step 101 of this embodiment. Of course, this is only an example, and this embodiment is not limited thereto.

[0054]

[0053] Continuing to refer to Figure 1, in step 102, the tensor core can be used to process the constructed tensor computation task to complete the target computation task. As mentioned above, the sparse matrix and dense matrix multiplication function has the ability to call the tensor core. Therefore, the tensor computation task constructed by the sparse matrix and dense matrix multiplication function can reach the tensor core and be processed by the tensor core.

[0055]

[0054] In step 102, the constructed tensor computation task can be scheduled so that the tensor computation task reaches the tensor core. In an exemplary scheduling scheme: the target computation task can be scheduled to the first stream processor group included in the GPU; the constructed tensor computation task is processed using the tensor core provided in the first stream processor group.

[0056]

[0055] Herein, a stream processor group is a hardware unit in a GPU. A stream processor group can be composed of multiple stream processors (SPs) and some other resources (such as registers). A stream processor group can also be called a streaming multiprocessor (SM). A stream processor group can contain multiple general-purpose computing cores and multiple tensor cores. Within the stream processor group, the multiple general-purpose computing cores and multiple tensor cores usually share registers.

[0057]

[0056] In this exemplary scheduling scheme, the target computing task as a whole can be used as the scheduling unit, and the target computing task can be scheduled to a stream processor group in the GPU, described here as the first stream processor group. Based on this, in this exemplary scheduling scheme, the tensor computing task constructed for the target computing task can be scheduled to the same stream processor group and allocated to multiple tensor cores contained in the stream processor group, and the multiple tensor cores can process their allocated tensor computing tasks in parallel.

[0058]

[0057] It should be understood that the above scheduling scheme is exemplified, and this embodiment is not limited thereto. For example, a group of tensor computation tasks constructed from a pair of vectors and matrices that need to be multiplied in the target computation task can also be used as a scheduling unit, and each group of tensor computation tasks constructed for the target computation task can be scheduled separately. In this way, tensor computation tasks in the same group can be scheduled to the same stream processor group, while tensor computation tasks in different groups may be scheduled to different stream processor groups. No further examples of scheduling schemes will be given here.

[0059]

[0058] In step 102, since there are usually multiple tensor computation tasks constructed for the target computation task, referring to Figure 2, there are also multiple tensor cores used in this embodiment. That is, multiple tensor cores process the tensor computation tasks constructed for the target computation task in parallel. In addition, after multiple tensor cores complete their assigned tensor computation tasks, in this embodiment, the processing results provided by the multiple tensor cores can be combined to generate the processing result corresponding to the target computation task, thereby completing the target computation task.

[0060]

[0059] In summary, this embodiment proposes a scheme for accelerating GEMV computation tasks based on GPUs. The target computation task indicates vectors and matrices that require multiplication. For the indicated vectors, the vectors can be filled into matrices according to the matrix format required by the tensor cores installed in the GPU. Based on this, the sparse matrix multiplication function can be called to construct the original GEMV computation task into a parallel tensor computation task, and the tensor cores installed in the GPU can be used to process the constructed tensor computation task to complete the target computation task. Accordingly, this embodiment innovatively proposes to call the sparse matrix multiplication function to process the GEMV computation task, thereby utilizing the function's ability to rearrange sparse matrices to convert the GEMV computation task into a tensor computation task that can fully utilize the computing power of the tensor cores. This means that when using tensor cores to process GEMV computation tasks, not only is the computing power of the tensor cores not wasted, but the processing time of GEMV computation tasks can also be effectively shortened compared to general-purpose computing cores.

[0060] FIG3 is a schematic flowchart of another computation acceleration method provided by an exemplary embodiment of the present disclosure. Referring to FIG3, the method may include the following steps.

[0061]

[0061] Step 300: In response to receiving the matrix-vector multiplication calculation task group, if it is determined that the matrix-vector multiplication calculation task group meets the triggering condition for enabling the tensor core for acceleration, then the first task flow is divided from the matrix-vector multiplication calculation task group as the target calculation task.

[0062]

[0062] Step 301: Fill the vector indicated in the target calculation task according to the matrix format required by the tensor core to obtain the first matrix. The target calculation task is a matrix-vector multiplication calculation task.

[0063]

[0063] Step 302: For the first matrix and the second matrix indicated in the target computation task, call the sparse matrix and dense matrix multiplication function to construct the tensor computation task.

[0064]

[0064] Step 303: Use the tensor core to process the constructed tensor computation task to complete the target computation task.

[0065]

[0065] Steps 301-303 can be referred to the description in the foregoing embodiments, and will not be repeated here.

[0066]

[0066] During the research process, the inventors discovered that GEMV computation tasks are usually submitted to the GPU in groups. Therefore, in this embodiment, the GEMV computation tasks submitted to the GPU in groups are described as the matrix-vector multiplication computation task group in step 300. The matrix-vector multiplication computation task group in step 300 can be understood as a group of GEMV computation tasks that can be stream-processed. Among them, stream processing technology is a technology used to support asynchronous execution, which allows multiple computation and data transfer operations to be executed simultaneously on the GPU to improve GPU utilization and parallel performance.

[0067]

[0067] For example, a key and common computational task in large model inference is the matrix multiplication between the query and the key in the attention mechanism of the encoder module (decoder). This matrix multiplication computation task typically involves computation between matrices of size b*l*h and b*n*h, where b represents the batch size, l refers to the number of query tokens processed in the encoder module each time, h represents the dimension of the hidden layer, and n is the length of the sequence, and b, n, and h are all positive integers. The essence of this computational task is a set of GEMV computation tasks, which can be split according to the size of b. It should be understood that this is only an exemplary set of matrix-vector multiplication computation tasks, and this embodiment is not limited thereto.

[0068]

[0068] Based on this, in step 300, if it is determined that the matrix-vector multiplication calculation task group meets the triggering condition for enabling tensor core acceleration, then a first task flow is partitioned from the matrix-vector multiplication calculation task group as the target calculation task. That is, a tensor flow can be partitioned from the matrix-vector multiplication calculation task group as the first task flow. In this way, the target calculation task in this embodiment will continue to be processed in the form of a tensor flow, thereby being better processed in parallel.

[0069]

[0069] The triggering conditions for enabling Tensor Core for acceleration mentioned in step 300 may include, but are not limited to: the utilization rate of the stream processor group in the GPU is lower than a preset utilization rate threshold; and / or, the precision of the matrices and vectors in the matrix-vector multiplication calculation task group meets the precision requirements of the Tensor Core; and / or, the specifications of the matrices and vectors in the matrix-vector multiplication calculation task group exceed a preset specification threshold.

[0070]

[0070] Figure 4 provides a logical diagram illustrating whether a matrix-vector multiplication computation task group meets the triggering conditions for enabling tensor cores for acceleration. Referring to Figure 4, the following processing logic can be executed.

[0071]

[0071] The first step is to receive the matrix-vector multiplication calculation task group.

[0072]

[0072] The second step is to obtain the utilization rate of the stream processor group in the GPU. Before this, it is also possible to detect whether the current utilization rate of the stream processor group has expired (for example, 1 minute in Figure 4). If it has expired, it is necessary to request to view it again. If it has not expired, it can be obtained directly.

[0073]

[0073] In the third step, if it is determined that the utilization rate of the stream processor group is lower than the preset utilization rate threshold, the subsequent judgment can continue; if the utilization rate of the stream processor group has exceeded the preset utilization rate threshold, the GEMV computing task can be processed by only using the general computing core according to the traditional scheme.

[0074]

[0074] Fourth step: Check whether the precision of the matrix and vector in the matrix-vector multiplication calculation task group meets the precision requirements of the tensor core. If the precision does not meet the precision requirements of the tensor core, the traditional solution is adopted; otherwise, the subsequent judgment can be carried out.

[0075]

[0075] Fifth step, check the specifications of the matrix and vector in the matrix-vector multiplication calculation task group. If they do not exceed the preset specification threshold, it means that the specifications are too small, and the traditional solution is adopted; otherwise, it can be determined that the matrix-vector multiplication calculation task group meets the triggering conditions for enabling tensor core for acceleration, and the subsequent processing logic in the calculation acceleration method provided in this embodiment continues to be executed.

[0076]

[0076] In a further preferred embodiment, it is proposed that, based on the implemented capability of using Tensor Cores to process GEMV computation tasks, the matrix-vector multiplication computation task group can be flexibly distributed, thereby using Tensor Cores and general-purpose computing cores to share the matrix-vector multiplication computation task group. That is, the aforementioned first task flow can only cover a portion of the tasks in the matrix-vector multiplication computation task group, rather than all of them.

[0077]

[0077] Therefore, in this preferred implementation, after determining that the matrix-vector multiplication computation task group meets the triggering condition for enabling Tensor Core acceleration, in addition to dividing a first task flow from the matrix-vector multiplication computation task group as the target computation task as mentioned in step 300 above, and processing the first task flow using Tensor Core according to steps 301-302, a second task flow can also be divided from the matrix-vector multiplication computation task group; for the matrices and vectors indicated in the second task flow, the matrix-vector multiplication function is called to construct a general computation task; the general computation core assembled in the GPU is used to process the constructed general computation task to complete the second task flow. Here, the second task flow is the general computation task flow.

[0078] The matrix-vector multiplication function can be used to handle multiplication calculations between dense matrices and dense vectors and has the ability to call a general-purpose computing core. Optionally, in this embodiment, the matrix-vector multiplication function can be called from a function library for handling linear algebra calculations to construct a general-purpose computing task. Function libraries for handling linear algebra calculations focus on basic linear algebra operations and provide efficient matrix operation functions, such as matrix multiplication, matrix-vector multiplication, and matrix transpose functions. Here, the matrix-vector multiplication function provided by such a function library can be selected and applied to the second task flow. Of course, this is only exemplary, and this embodiment is not limited to this. It should be understood that the scheduling method provided in this embodiment has the authority to call such function libraries.

[0078]

[0079] Similarly, in this preferred implementation, the general computing tasks constructed for the second task flow can also be scheduled. An exemplary scheduling scheme could be: scheduling the second task flow to a second stream processor group included in the GPU; and using the stream processors provided in the second stream processor group to process the constructed stream processing tasks.

[0079]

[0080] During the research process, the inventors discovered that after scheduling the first and second task flows according to the above-described exemplary scheduling scheme, the first and second task flows might be scheduled to the same stream processor group. Therefore, this exemplary scheduling scheme further proposes: if the first and second task flows are scheduled to the same stream processor group, the register occupancy rate in that stream processor group is detected; if the register occupancy rate exceeds a preset threshold, the first and second task flows are rescheduled. As mentioned earlier, a stream processor group contains multiple tensor cores and multiple general-purpose computing cores that share registers. Therefore, when the register occupancy rate in streaming multiprocessing is already high, it will affect the parallelism effect in this embodiment. Therefore, in this exemplary scheduling scheme, when this situation occurs, the first and second task flows can be rescheduled to prioritize scheduling the two task flows to different stream processor groups, or to stream processor groups where the register occupancy rate does not exceed the preset threshold.

[0080]

[0081] It should be understood that the above scheduling scheme is merely exemplary, and this embodiment is not limited thereto.

[0081]

[0082] Thus, in this preferred implementation, the matrix-vector multiplication task group received by the GPU (specifically, the aforementioned acceleration operator within the GPU) can be split into a first task flow and a second task flow. Tensor cores are used to process the first task flow, while general-purpose computing cores are used to process the second task flow. This effectively balances the occupancy of tensor cores and general-purpose computing cores in the GPU, preventing excessive preemption of tensor cores by GEMV computation tasks and thus affecting other computational tasks that need to be carried on the tensor cores.

[0082]

[0083] In this implementation, the splitting ratio between the first and second task flows is not limited. Generally, the received matrix-vector multiplication calculation task groups can be evenly divided to create the first and second task flows. Of course, other splitting ratios can also be used to divide the first and second task flows.

[0083]

[0084] For example, following the calculation tasks involving b*l*h and b*n*h matrices in the large model inference process mentioned earlier, for this matrix-vector multiplication calculation task group, if the value of b is even, the first task flow and the second task flow can be divided equally. If the value of b is odd, the first task flow and the second task flow can be divided according to other distribution ratios instead of equal distribution.

[0084]

[0085] Figure 5 is a logical schematic diagram of an exemplary implementation scheme for dividing a first task flow and a second task flow according to an exemplary embodiment of this disclosure. Referring to Figure 5, in this exemplary implementation scheme: after determining that tensor cores are enabled for computational acceleration, a first task flow, a second task flow, a first handle for identifying tensor cores, and a second handle for identifying stream processors can be created for the received matrix-vector multiplication computation task group; the first handle is assigned to the first task flow and the second handle is assigned to the second task flow; the multiple pairs of matrices and vectors contained in the matrix-vector multiplication computation task group are divided into the first task flow and the second task flow.

[0085]

[0086] Referring to Figure 5, the data segmented into the first data stream can be prepared according to the data format requirements of the sparse matrix multiplication function and the dense matrix multiplication function; and the data segmented into the second data stream can be prepared according to the data format requirements of the matrix-vector multiplication function. This generates the final first task stream, which is processed using tensor cores; and the final second task stream is generated, which is processed using general-purpose computing cores. After the first and second task streams have completed processing, the processing results can be merged to obtain the final processing result. Then, the resources occupied in the GPU (tensor cores and general-purpose computing cores) can be released, thus ending this round of task processing.

[0086]

[0087] In summary, this embodiment allows for the parallel processing of GEMV computation tasks on both the general-purpose computing core and the tensor core using two task streams. This further improves the GPU's processing performance and resource utilization for GEMV computation tasks. Furthermore, the offloading process fully considers factors such as the current state of the stream processor group in the GPU (e.g., stream processor group utilization, register occupancy, etc.) and the data precision in the GEMV computation task to determine whether to enable the tensor core to share the GEMV computation task. It does not abandon traditional solutions but rather uses them as a backup. Therefore, the computational acceleration method provided in this embodiment has good versatility and applicability.

[0087]

[0088] It should be noted that some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear in this document, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should also be noted that the terms "first," "second," etc., used in this document are used to distinguish different matrices, stream processor groups, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0088]

[0089] Figure 6 is a schematic diagram of the structure of a GPU provided in another exemplary embodiment of this disclosure. As shown in Figure 6, the GPU may include: a memory 60, a controller 61, and a tensor core 62. The memory 60 stores at least the computer program corresponding to the acceleration operator.

[0089]

[0090] The controller 61, coupled to the memory 60 and the tensor core 62, is used to execute a computer program in the memory 60 for: filling a vector indicated in a target computation task with zeros according to the matrix format required by the tensor core to obtain a first matrix, wherein the target computation task is a matrix-vector multiplication computation task; calling a sparse matrix and dense matrix multiplication function for the first matrix and a second matrix indicated in the target computation task to construct a tensor computation task; and processing the constructed tensor computation task using the tensor core to complete the target computation task.

[0090]

[0091] In an optional embodiment, when the controller 61 fills the vector indicated in the target computation task with zero values ​​according to the matrix format required by the tensor core, it can specifically be used to: if the matrix format is n*n, when the vector is a row vector, fill the row vector with zero values ​​to n rows to obtain the first matrix; when the vector is a column vector, fill the column vector with zero values ​​to n columns to obtain the first matrix, where n is a positive integer.

[0091]

[0092] In an optional embodiment, when the controller 61 calls the sparse matrix multiplication function to construct the tensor computation task, it may specifically be used to: call the sparse matrix multiplication function from a function library for processing sparse matrix computations to construct the tensor computation task.

[0092]

[0093] In an optional embodiment, when the controller 61 uses the tensor core to process the constructed tensor computation task, it may specifically be used to: schedule the target computation task to a first stream processor group included in the GPU; and use the tensor core provided in the first stream processor group to process the constructed tensor computation task.

[0093]

[0094] In an alternative embodiment, before filling the vector indicated in the target computation task according to the matrix format required by the tensor core, the controller 61 may further be configured to: in response to receiving a matrix-vector multiplication computation task group, if it is determined that the matrix-vector multiplication computation task group meets the triggering conditions for enabling the tensor core for acceleration, then divide a first task flow from the matrix-vector multiplication computation task group as the target computation task.

[0094]

[0095] In an optional embodiment, the GPU further includes a general-purpose computing core 6, and the controller 61 can also be used to: divide a second task flow from the matrix-vector multiplication computation task group; call a matrix-vector multiplication function for the matrix and vector indicated in the second task flow to construct a general-purpose computing task; and use the general-purpose computing core 63 assembled in the GPU to process the constructed general-purpose computing task to complete the second task flow.

[0095]

[0096] In an optional embodiment, when the controller 61 calls the matrix-vector multiplication function to construct a general computing task, it may specifically be used to: call the matrix-vector multiplication function from a function library for processing linear algebraic calculations to construct the general computing task.

[0097] In an optional embodiment, when the controller 61 processes the constructed general-purpose computing task using the general-purpose computing cores assembled in the GPU, it may specifically be used to: schedule the second task flow to the second stream processor group included in the GPU; and process the constructed general-purpose computing task using the general-purpose computing cores provided in the second stream processor group.

[0096]

[0098] In an optional embodiment, the controller 61 may also be configured to: if the first task flow and the second task flow are scheduled to the same stream processor group, detect the register occupancy rate in the stream processor group; if the register occupancy rate is detected to exceed a preset occupancy rate threshold, reschedule the first task flow and the second task flow.

[0097]

[0099] In an optional embodiment, when the controller 61 divides a first task flow from the matrix-vector multiplication calculation task group and a second task flow from the matrix-vector multiplication calculation task group, it may specifically be used to: create a first task flow, a second task flow, a first handle for identifying a tensor core, and a second handle for identifying a general computing core; assign the first handle to the first task flow and the second handle to the second task flow; and divide the multiple pairs of matrices and vectors contained in the matrix-vector multiplication calculation task group into the first task flow and the second task flow.

[0098]

[0100] In an optional embodiment, the triggering condition for enabling the tensor core for acceleration may be specifically used when: the utilization rate of the stream processor group in the GPU is lower than a preset utilization rate threshold; and / or, the precision of the matrices and vectors in the matrix-vector multiplication calculation task group meets the precision requirements of the tensor core; and / or, the specifications of the matrices and vectors in the matrix-vector multiplication calculation task group exceed a preset specification threshold.

[0099]

[0101] Furthermore, only some components are shown schematically in Figure 6, which does not mean that the GPU only includes the components shown in Figure 6.

[0100]

[0102] It is worth noting that the technical details of the above-mentioned GPU embodiments can be found in the description of the acceleration operators in the foregoing method embodiments. To save space, they will not be repeated here, but this should not cause any loss of the scope of protection of this disclosure.

[0101]

[0103] Accordingly, this disclosure also provides an acceleration operator, which includes one or more computer instructions. When the GPU calls the acceleration operator, one or more computer instructions are executed to perform the steps in the above method embodiments, so as to accelerate the matrix-vector multiplication calculation task by utilizing the tensor core in the GPU.

[0102]

[0104] Accordingly, this disclosure also provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps in the above method embodiments.

[0103]

[0105] Accordingly, this disclosure also provides a computer program product, wherein the computer program contained herein, when executed, can implement the steps in the above method embodiments.

[0104]

[0106] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0105]

[0107] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0106]

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0107]

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0108]

[0110] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0109]

[0111] The above description is merely an embodiment of this disclosure and is not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

Claims 1. A computational acceleration method applicable to a GPU, wherein the GPU is equipped with a tensor core, the method comprising: According to the matrix format required by the tensor core, the vector indicated in the target computation task is filled with zero values ​​to obtain the first matrix, wherein the target computation task is a matrix-vector multiplication computation task. For the first matrix and the second matrix indicated in the target computation task, the sparse matrix multiplication function is invoked to construct the tensor computation task; The tensor core is used to process the constructed tensor computation task in order to complete the target computation task.

2. The method according to claim 1, wherein, According to the matrix format required by the tensor core, zero-value filling is performed on the vector indicated in the target computation task, including: if the matrix format is n*n, when the vector is a row vector, the row vector is filled to n rows with zero values ​​to obtain the first matrix; when the vector is a column vector, the column vector is filled to n columns with zero values ​​to obtain the first matrix, where n is a positive integer.

3. The method according to claim 1, wherein, Calling sparse and dense matrix multiplication functions to construct tensor computation tasks includes: calling sparse and dense matrix multiplication functions from a function library for handling sparse matrix computations to construct tensor computation tasks.

4. The method according to claim 1, wherein, The tensor core is used to process the constructed tensor computation task, including: scheduling the target computation task to a first stream processor group included in the GPU; and using the tensor core provided in the first stream processor group to process the constructed tensor computation task.

5. The method according to claim 1, wherein, Before filling the vector indicated in the target computation task according to the matrix format required by the Tensor Core, the method further includes: in response to receiving a matrix-vector multiplication computation task group, if it is determined that the matrix-vector multiplication computation task group meets the triggering condition for enabling Tensor Core for acceleration, then dividing a first task flow from the matrix-vector multiplication computation task group as the target computation task.

6. The method according to claim 5, further comprising: A second task flow is derived from the matrix-vector multiplication calculation task group; For the matrix and vector indicated in the second task flow, a matrix-vector multiplication function is invoked to construct a general-purpose computing task; the general-purpose computing cores assembled in the GPU are then used to process the constructed general-purpose computing task to complete the task. Describe the second task flow.

7. The method according to claim 6, wherein, Calling matrix-vector multiplication functions to construct general-purpose computation tasks includes: calling matrix-vector multiplication functions from a function library used for handling linear algebra computations to construct general-purpose computation tasks.

8. The method according to claim 6, wherein, The general-purpose computing task constructed is processed using the general-purpose computing cores assembled in the GPU, including: scheduling the second task flow to the second stream processor group contained in the GPU; and processing the general-purpose computing task constructed using the general-purpose computing cores provided in the second stream processor group.

9. The method according to claim 8, further comprising: If the first task flow and the second task flow are scheduled to the same stream processor group, then the register occupancy rate in the stream processor group is detected; If the register occupancy rate is detected to exceed the preset occupancy rate threshold, the first task flow and the second task flow will be rescheduled.

10. The method according to claim 6, wherein, The process of dividing a first task flow from the matrix-vector multiplication computation task group and a second task flow from the same group includes: creating a first task flow, a second task flow, a first handle for identifying a tensor core, and a second handle for identifying a general-purpose computation core; assigning the first handle to the first task flow and the second handle to the second task flow; and dividing the multiple pairs of matrices and vectors contained in the matrix-vector multiplication computation task group into the first task flow and the second task flow.

11. The method according to claim 5, wherein, The triggering conditions for enabling Tensor Core acceleration include: the utilization rate of the stream processor group in the GPU is lower than a preset utilization rate threshold; and / or, the precision of the matrices and vectors in the matrix-vector multiplication calculation task group meets the precision requirements of the Tensor Core; and / or, the specifications of the matrices and vectors in the matrix-vector multiplication calculation task group exceed a preset specification threshold. When the acceleration operator is invoked, one or more computer instructions are executed to perform the computation acceleration method according to any one of claims 1-11, thereby accelerating the matrix-vector multiplication computation task using the tensor cores in the GPU.

13. A GPU, comprising memory, a controller, and tensor cores; The memory is used to store one or more computer instructions; the controller is coupled to the memory and the tensor core, and is used to execute the one or more computer instructions to execute the computation acceleration method according to any one of claims 1-11, so as to accelerate the matrix-vector multiplication computation task using the tensor core.

14. A computer-readable storage medium for storing a computer program, wherein, When the computer program is executed by one or more processors, it causes the one or more processors to perform the computation acceleration method according to any one of claims 1-11.

15. A computer program product, comprising a computer program, wherein, When the computer program is executed by one or more processors, the one or more processors cause the computing acceleration method according to any one of claims 1-11 to be performed. 18

Images