Methods, apparatus, computing devices, and storage media for computing a matrix multiplication of vectors

Abstract

Embodiments of the present disclosure provide a method, apparatus, computing device and storage medium for calculating matrix-vector multiplication, where the matrix is an M*K matrix and the vector is a K*1 vector, M and K are positive integers. The method comprises: converting the M*K matrix into an M*(K / N)*N first tensor, where N is a positive integer, N>1, and K is a multiple of N; converting the K*1 vector into an N*(K / N) second tensor; using a tensor computing core to batch-process the general matrix multiplication of the M*(K / N)*N first tensor and the N*(K / N) second tensor to obtain M result matrices; and adding the elements on the diagonal of each of the M result matrices to obtain each element on the M*1 result vector of the M*K matrix multiplied by the K*1 vector. The above scheme uses a tensor computing core to implement high-throughput, low-latency high-speed matrix-vector multiplication (MMV) operation.

Description

Technical Field

[0001] One or more embodiments of this disclosure relate to methods, apparatus, computing devices, and non-transitory computer-readable storage media (where M and K are positive integers) for computing a matrix M*K multiplied by a vector K*1 using a tensor computing kernel. Background Technology

[0002] Matrix-vector multiplication (MMV) is a core operation in linear algebra, widely used in machine learning, computer graphics, physics simulations, and other fields. Given an m×n matrix A and an n-dimensional column vector v, their product Av is an m-dimensional column vector, where each component is the dot product of a row of matrix A and vector v.

[0003] In neural network computation in machine learning, the product of the weight matrix and the input vector is the core operation of forward propagation, which is a typical MMV operation. For example, the output of a fully connected layer is y = Wx + b, where W is the weight matrix and x is the input vector.

[0004] Therefore, there is a need to improve the speed of MMV operations. Summary of the Invention

[0005] According to one aspect of this disclosure, at least one embodiment provides a method for calculating the multiplication of an M*K matrix by a K*1 vector using a tensor computation kernel, wherein M and K are positive integers. The method includes: converting the M*K matrix into a first tensor of M*(K / N)*N, where N is a positive integer, N>1, and K is a multiple of N; converting the K*1 vector into a second tensor of N*(K / N); performing a batch general matrix multiplication of the first tensor of M*(K / N)*N and the second tensor of N*(K / N) using a tensor computation kernel to obtain M result matrices; and summing the diagonal elements of each of the M result matrices to obtain each element of the M*1 result vector of the M*K matrix multiplied by the K*1 vector.

[0006] In some embodiments, the first tensor of M*(K / N)*N is M (K / N)*N first matrices, wherein each of the M (K / N)*N first matrices is formed by arranging the elements of each row of the first tensor in row-major order in the (K / N)*N first matrix, and the second tensor of N*(K / N) is formed by arranging the elements of each of the K*1 vectors in column-major order in the second tensor of N*(K / N).

[0007] In some embodiments, the method further includes: obtaining N, wherein N is set based on the computing power of the tensor computing kernel and the computing power of the vector computing kernel that can be used to compute a matrix multiplied by a vector of size K*1.

[0008] In some embodiments, N ,in It refers to the computing power of a tensor computing kernel, measured in floating-point operations per second. It is the computing power of the vector computing kernel, measured in floating-point operations per second.

[0009] In some embodiments, the larger K is, the closer N is to ... .

[0010] In some embodiments, N is 32 when K is less than or equal to 8192; or N is 64 when K is greater than 8192.

[0011] In some embodiments, the diagonal elements of each of the M result matrices are added together to obtain each element of the M*1 result vector of the M*K matrix multiplied by the K*1 vector, including: adding the diagonal elements in a thread-local register; and storing the result of the addition in a high-bandwidth memory as a result vector.

[0012] According to another aspect of this disclosure, at least one embodiment provides an apparatus for calculating the multiplication of an M*K matrix by a K*1 vector using a tensor computation kernel, wherein M and K are positive integers. The apparatus includes: a matrix transformation module configured to convert an M*K matrix into a first tensor of M*(K / N)*N, where N is a positive integer, N>1, and K is a multiple of N; a vector transformation module configured to convert a K*1 vector into a second tensor of N*(K / N); a computation module configured to perform a general matrix multiplication of the first tensor of M*(K / N)*N and the second tensor of N*(K / N) in batches using the tensor computation kernel to obtain M result matrices; and an addition module configured to add the diagonal elements of each of the M result matrices to obtain each element of the M*1 result vector of the M*K matrix multiplied by the K*1 vector.

[0013] According to another aspect of this disclosure, at least one embodiment provides an apparatus for computing an M*K matrix multiplied by a K*1 vector using a tensor computation kernel, the apparatus comprising: at least one memory for storing computer instructions; and at least one processor for reading the computer instructions from the at least one memory and executing the method according to at least one embodiment of this disclosure.

[0014] According to another aspect of this disclosure, at least one embodiment provides a computing device including means for computing an M*K matrix multiplied by a K*1 vector using tensor computing kernels, and a plurality of tensor computing kernels.

[0015] According to another aspect of this disclosure, at least one embodiment provides a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by at least one processor, cause at least one processor to perform a method according to at least one embodiment of this disclosure.

[0016] According to another aspect of this disclosure, at least one embodiment provides a computer program product including computer instructions, wherein, when executed by at least one processor, the computer instructions cause the at least one processor to perform a method according to at least one embodiment of this disclosure. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic block diagram of a general-purpose graphics processing unit (GPGPU) is shown.

[0019] Figure 2 The diagram illustrates the multiplication of a 3x3 matrix A by a 1x3 vector B in MMV form.

[0020] Figure 3 A flowchart is shown of a method for calculating the multiplication of an M*K matrix by a K*1 vector using a tensor computation kernel according to at least one embodiment of the present disclosure.

[0021] Figure 4A A schematic diagram of the folding process of a matrix-vector multiplication that needs to be calculated according to at least one embodiment of the present disclosure is shown.

[0022] Figure 4B A schematic diagram is shown illustrating the calculation of the result vector of a matrix-vector multiplication that needs to be calculated according to at least one embodiment of the present disclosure.

[0023] Figure 4C A schematic diagram is shown illustrating the calculation of the result vector of another matrix-vector multiplication according to at least one embodiment of the present disclosure.

[0024] Figure 5An example diagram is shown illustrating the computational performance achieved using a tensor computing kernel with different values ​​for folding granularity in different dimensions K, according to at least one embodiment of the present disclosure.

[0025] Figure 6 A block diagram is shown of an apparatus for computing an M*K matrix multiplied by a K*1 vector using a tensor computation kernel according to at least one embodiment of the present disclosure.

[0026] Figure 7 A block diagram of another apparatus for computing a matrix M*K multiplied by a vector K*1 using a tensor computation kernel according to at least one embodiment of the present disclosure is shown.

[0027] Figure 8 A block diagram of a computing device for computing an M*K matrix multiplied by a K*1 vector using a tensor computing kernel according to at least one embodiment of the present disclosure is shown.

[0028] Figure 9 A schematic block diagram of an electronic device according to at least one embodiment of the present disclosure is shown. Detailed Implementation

[0029] Referring now to specific embodiments of this disclosure, examples of which are illustrated in the accompanying drawings. Although this application will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit this application to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of this disclosure. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0030] In this document, "multiple" refers to two or more. "Or" describes the relationship between related objects, indicating that three relationships can exist. For example, A or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Unless otherwise defined, the technical or scientific terms used in this disclosure should be understood in their ordinary sense by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar words used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Words such as "including" or "contains" mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," and "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described objects changes, the relative positional relationship may also change accordingly. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components have been omitted. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0031] Figure 1 A schematic block diagram of a general-purpose graphics processing unit (GPGPU) is shown.

[0032] like Figure 1 As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1 The diagram shows the 1st streaming processor cluster, ..., the Mth streaming processor cluster, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster processes one computational task, or multiple streaming processor clusters process one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0033] like Figure 1 As shown, taking the first streaming processor cluster as an example, a streaming processor cluster includes multiple computing units, for example... Figure 1The computation unit (CU) is structured as the 1st, 2nd, ..., Nth computation unit, where N is a positive integer. Each CU performs arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A CU contains multiple cores, each including an arithmetic logic unit (ALU), a floating-point unit, etc., which execute specific computational tasks. Furthermore, the CU also includes registers (e.g., ...). Figure 1 The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0034] like Figure 1 As shown, each computing unit also provides a tensor core, or simply tensor core, for performing tensor-related computations, such as tensor shrinking operations. Tensor cores can accelerate tensor operations such as matrix multiplication. Tensor cores in multiple computing units can be scheduled and controlled uniformly.

[0035] like Figure 1 As shown, each streaming processor cluster also provides a buffer for caching data across the N computing units within the streaming processor cluster.

[0036] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0037] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory-executed instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory (e.g., [example cache]). Figure 1High Bandwidth Memory (HBM) is used for read and write operations.

[0038] MMV operations can be performed using a dedicated vector computation kernel in a GPGPU. Assume matrix A is m×n, vector x is n×1, and the output vector C=Ax is m×1, where m and n are positive integers. The specific formula is as follows:

[0039] .

[0040] That is, each element of the result vector is the dot product of the corresponding row of the matrix and the input vector.

[0041] Vector computation kernels can perform dot products between each element of a matrix and each element of an input vector. These kernels can execute multiple dot product operations simultaneously using either a Single Instruction Multiple Data (SIMD) architecture (where each processing unit operates on independent data elements, such as single floating-point numbers in a vector) or a Single Instruction Multiple Thread (SIMT) architecture (where each thread processes independent data or tasks but shares the same instruction stream (executed via warp locksteps)). Thus, by multiplying each element of each row of matrix A with the corresponding element of vector x and summing the results, the output vector C is stored in the corresponding element.

[0042] However, these methods operate on multiple data elements simultaneously through a single instruction or drive multi-threaded parallel execution through a single instruction stream, but this parallelization is still insufficient, preventing further improvement in computational speed.

[0043] The defects in the above solutions were discovered by the inventors of this disclosure through careful research and creative labor. The discovery process of the above problems and the solutions proposed by at least one embodiment disclosed below for the above problems are all creative contributions.

[0044] According to at least one embodiment of this disclosure, by folding (transforming) the matrix and vector to be multiplied to generate two tensors, the tensor operation characteristics of the tensor computation kernel (such as matrix partitioning, sliding window of convolution, pipelined collaboration, etc.) are utilized to perform the product of the two tensors. Finally, the result of the MMV operation is easily obtained by adding the diagonal elements of the result matrix, thereby achieving high-throughput, low-latency high-speed MMV operation.

[0045] It should be noted that the matrices and vectors of the methods or apparatus provided in at least one embodiment of this disclosure may have different physical meanings depending on the application scenario. For example, the methods or apparatus provided in at least one embodiment of this disclosure can be applied in fields such as speech processing, image processing, text processing, and video processing.

[0046] For example, in the field of speech processing, matrices and vectors can be input, output parameters, intermediate calculation results, etc., for tasks involving neural networks such as feature extraction, speech enhancement, and speech recognition.

[0047] For example, in the field of image processing, matrices and vectors can be input, output parameters, and intermediate calculation results in tasks involving neural networks such as image recognition, feature extraction, image segmentation, object detection, image classification, and scene reconstruction.

[0048] For example, in the field of text processing, matrices and vectors can be input, output parameters, and intermediate calculation results in tasks involving neural networks such as text classification, sentiment analysis, and text generation.

[0049] For example, in the field of video processing, matrices and vectors can be relevant parameters in the field of image processing, or input, output parameters, and intermediate calculation results specific to the field of video processing, such as optical flow operators (used to estimate motion between video frames) and target tracking operators (used to track specific targets in video).

[0050] Of course, this disclosure is not limited to this. For other application scenarios or fields, as long as matrix and vector multiplication calculations are required, the method or apparatus of at least one embodiment of this disclosure can be applied, and will not be described in detail here.

[0051] Note that the method and apparatus provided in at least one embodiment of this disclosure can be applied to different systems or devices, such as... Figure 9 The illustrated electronic device 900 can be a terminal, such as a mobile phone, tablet computer, laptop computer, AR device, VR device, vehicle terminal, etc., or it can be a server. The method provided in at least one embodiment of this disclosure can be applied to scenarios involving matrix-vector multiplication in electronic devices, such as those related to High Performance Computing (HPC) and Artificial Intelligence (AI). Of course, this disclosure is not limited to this; any scenario, device, or apparatus involving matrix-vector multiplication can employ the method or apparatus provided in at least one embodiment of this disclosure.

[0052] In some embodiments, the apparatus provided in at least one embodiment of this disclosure can be a chip, such as a system-on-a-chip (SoC). A SoC includes a processor, which can be a single-core or multi-core processor, memory, and I / O interfaces. The processor can load data and applications from memory and then process the data, for example, by using a tensor computation kernel to compute an M*K matrix multiplied by a K*1 vector.

[0053] Figure 2 The diagram illustrates the multiplication of a 3x3 matrix A by a 1x3 vector B in MMV form.

[0054] like Figure 2 As shown on the left, the 3x3 matrix A is .like Figure 2 As shown on the right, the 3*1 vector B is The vector kernel is typically used to calculate AB=C, where C is a 3*1 vector. ,in, = * , = * , = * .

[0055] Figure 3 A flowchart is shown of a method 300 for calculating a matrix M*K multiplied by a vector K*1 using a tensor computation kernel according to at least one embodiment of the present disclosure, wherein M and K are positive integers.

[0056] The exemplary method 300 includes the following steps S310, S320, S330 and S340.

[0057] In step S310, the M*K matrix is ​​converted into the first tensor of M*(K / N)*N, where N is a positive integer, N>1, and K is a multiple of N.

[0058] In step S320, the K*1 vector is converted into a second tensor of N*(K / N).

[0059] In step S330, a tensor computation kernel is used to perform a batch general matrix multiplication of the first tensor M*(K / N)*N and the second tensor N*(K / N) to obtain M result matrices.

[0060] In step S240, the diagonal elements of each of the M result matrices are summed to obtain each element of the M*1 result vector obtained by multiplying the M*K matrix by the K*1 vector.

[0061] Thus, by folding both the M*K matrix and the K*1 vector N times, i.e. converting them into the first tensor of M*(K / N)*N and the second tensor of N*(K / N) respectively, the parallel and efficient tensor operation characteristics of the tensor computation kernel (such as matrix partitioning, sliding window of convolution, pipelined collaboration, etc.) can be fully utilized to perform the product of the two tensors. Finally, by adding the diagonal elements of the result matrix, the result of the MMV operation can be easily obtained, realizing high-throughput, low-latency high-speed MMV operation.

[0062] Figure 4A A schematic diagram of the folding process of a matrix-vector multiplication that needs to be calculated according to at least one embodiment of the present disclosure is shown. Figure 4B A schematic diagram is shown illustrating the calculation of the result vector of a matrix-vector multiplication that needs to be calculated according to at least one embodiment of the present disclosure.

[0063] Suppose the matrix to be multiplied by vector is M*K and the vector is K*1. Figure 4A In the example shown, M=3, K=9. The matrix M*K is: The vector K*1 is: The result vector C of matrix-vector multiplication (or General Matrix-Vector Multiplication (GEMMV)) should be... ,in, = * , = * , = * .

[0064] According to at least one embodiment of this disclosure, an M*K matrix is ​​converted into a first tensor of M*(K / N)*N, where M, K, and N are all positive integers, N>1, and K is a multiple of N. The first tensor of M*(K / N)*N is M (K / N)*N first matrices. N can also be referred to as the folding granularity.

[0065] like Figure 4A As shown, assuming N=3, that is, folding the 3*9 matrix three times, then the 3*9 matrix needs to be converted into the first tensor of 3*(9 / 3)*3, that is, three (9 / 3)*3 first matrices.

[0066] exist Figure 4AThe diagram shows three first matrices (a first matrix with m=0, a first matrix with m=1, and a first matrix with m=2). Each of the M (K / N)*N first matrices is formed by arranging the elements of each row of the M-dimensional first tensor in row-major order (i.e., within each row, elements are placed sequentially from left to right; when the remaining space in the current row is insufficient to accommodate the next element, the element is placed at the beginning of the next row, and the arrangement continues in left-to-right order) within the (K / N)*N first matrix.

[0067] exist Figure 4A In the diagram, the first matrix with m=0 is formed by arranging the elements of the first row of dimension M in row-major order within the first matrix with m=0. Specifically, the elements a00, a01, a02, a03, a04, a05, a06, a07, and a08 are arranged in row-major order within the 3x3 first matrix with m=0, resulting in the following first matrix with m=0:

[0068] .

[0069] exist Figure 4A In the diagram, the first matrix with m=1 is formed by arranging the elements of the second row of dimension M in row-major order within the first matrix with m=1. Specifically, the elements a10, a11, a12, a13, a14, a15, a16, a17, and a18 are arranged in row-major order within the 3x3 first matrix with m=1, resulting in the following first matrix with m=1:

[0070] .

[0071] exist Figure 4A In the above, the first matrix of m=2 is formed by arranging the elements of the third row of dimension M in row-major order within the first matrix of m=2. That is, arranging the elements of a20, a21, a22, a23, a24, a25, a26, a27, and a28 in row-major order within the first 3x3 matrix of m=2, resulting in the first matrix of m=2 as follows:

[0072] .

[0073] According to at least one embodiment of this disclosure, a vector of K*1 is converted into a second tensor of N*(K / N).

[0074] like Figure 4A As shown, assuming N=3, that is, folding the 9*1 vector three times, the 9*1 vector is converted into a second tensor of 3*(9 / 3).

[0075] The second tensor of N*(K / N) is formed by arranging each element of the K*1 vector in column-major order (i.e., within each column, the elements are placed in order from top to bottom, and when the remaining space in the current column is insufficient to accommodate the next element, the element is placed at the beginning of the next column and the order continues from top to bottom) in the second tensor of N*(K / N).

[0076] exist Figure 4A In the 9*1 vector, the elements are arranged in column-major order in the second tensor. That is, the elements b00, b10, b20, b30, b40, b50, b60, b70, and b80 are arranged in column-major order in the second 3*3 tensor, resulting in the following 3*3 second tensor:

[0077] .

[0078] According to at least one embodiment of this disclosure, a tensor computation kernel is used to perform a general matrix multiplication (GEMM) of a first tensor of M*(K / N)*N and a second tensor of N*(K / N) in batches to obtain M result matrices.

[0079] Specifically, such as Figure 4A As shown, multiplying the first 3x3 matrix (m=0) by the second 3x3 tensor yields... × This yields the result corresponding to m=0. .

[0080] Multiply the first 3x3 matrix (m=1) by the second 3x3 tensor, i.e. × This yields the result corresponding to m=1. .

[0081] Multiply the first 3x3 matrix (m=1) by the second 3x3 tensor, i.e. × This yields the result corresponding to m=1. .

[0082] According to at least one embodiment of the present disclosure, the diagonal elements of each of the M result matrices are added together to obtain each element of the M*1 result vector of the M*K matrix multiplied by the K*1 vector.

[0083] like Figure 4BAs shown, for m=0, d00 = a00*b00 + a01*b10 + a02*b20, d01 = a00*b30 + a01*b40 + a02*b50, ..., d11 = a03*b30 + a04*b40 + a05*b50, ..., d22 = a06*b60 + a07*b70 + a08*b80. Figure 4B As shown, C00 in the result vector C is the sum of the diagonal elements of the result matrix with m=0, i.e., c00=d00+d11+d22.

[0084] like Figure 4B As shown, for m=1, d00 = a10*b00 + a11*b10 + a12*b20, d01 = a10*b30 + a11*b40 + a12*b50, ..., d11 = a13*b30 + a14*b40 + a15*b50, ..., d22 = a16*b60 + a17*b70 + a18*b80. Figure 4B As shown, C10 in the result vector C is the sum of the diagonal elements of the result matrix with m=1, i.e., c10=d00+d11+d22.

[0085] like Figure 4B As shown, for m=2, d00 = a20*b00 + a21*b10 + a22*b20, d01 = a20*b30 + a21*b40 + a22*b50, ..., d11 = a23*b30 + a24*b40 + a25*b50, ..., d22 = a26*b60 + a27*b70 + a28*b80. Figure 4B As shown, C20 in the result vector C is the sum of the diagonal elements of the result matrix with m=2, i.e., c20=d00+d11+d22.

[0086] Each of the above computational processes can be performed in batches and in parallel using one or more streaming processor clusters of tensor computation kernels. Therefore, the above computational process of the general matrix multiplication (GEMM) of the first tensor M*(K / N)*N and the second tensor N*(K / N) can be performed in batches and in parallel using tensor computation kernels.

[0087] The aforementioned method of diagonal accumulation after folding can also be optimized in hardware as follows: When performing matrix operations, for example, using the Warp Group Matrix Multiply Accumulate (WGMMA) instruction (which allows 128 consecutive threads (4 thread bundles, spanning all sub-units of shared memory) to collaboratively execute larger-scale matrix multiplication operations and supports asynchronous startup), the left and right tensors to be multiplied are divided into multiple tiles for MMA execution. The smallest tensor operation tile size is 64*64*8 (Tile_m=64, Tile_n=64, Tile_k=8). After a Tile_m*Tile_n operation is completed, it is not necessary to store the entire GEMM result in high-bandwidth memory (HBM). Instead, after the Tile_MMA is completed, the diagonal elements are directly accumulated in the thread local register (TLR), and the accumulated result is directly stored in HBM. This reduces the space occupied by HBM.

[0088] That is, in some embodiments, step S330 may include adding the diagonal elements in the thread-local register; and storing the result of the addition in a high-bandwidth memory as a result vector.

[0089] This eliminates the need to store the complete result in shared memory first and then in global local memory. Instead, it frees up the shared memory occupied by the output matrix, allowing the left and right tensors to use more shared memory and thus reducing latency.

[0090] Additionally, since diagonal elements are valid data, the Tile_MMA operation can be ignored when a Tile_m*Tile_n does not contain diagonal elements.

[0091] In this way, by folding (transforming) the matrix and vector to be multiplied to generate two tensors, the tensor computation kernel's tensor operation features (such as matrix partitioning, sliding window of convolution, pipelined collaboration, etc.) and its computing power (such as high number of floating-point operations per second) are fully utilized to perform the product of the two tensors. Finally, the result of the MMV operation is easily obtained by adding the diagonal elements of the result matrix, thus achieving high throughput and low latency high-speed MMV operation.

[0092] Because the above calculation process actually redundantly calculates all elements except the diagonal in the three result matrices m=0, m=1, and m=2 (since only the diagonal elements need to be added to obtain the values ​​in the result vector), the above calculation process actually wastes these redundant computational resources. The number of truly needed result elements only accounts for 1 / (N*N) of the total number of calculated elements, that is, the tensor calculation kernel redundantly calculates 1-1 / (N*N). For example, if the folding granularity N is 3, using... Figure 4A In the example calculation, a total of 27 elements were calculated, but only a result vector of 3 elements was originally needed. This means that using a vector kernel only requires calculating 1 / 9 of the elements, resulting in 8 / 9 of the computation being wasted by the tensor kernel. A larger number of folds allows the tensor kernel to better utilize its parallel batch computing capabilities compared to a vector kernel originally intended for matrix-vector multiplication. However, a larger number of redundant calculations can lead to increased computational resource and time costs. When the folding granularity N is too large, invalid computations increase, leading to performance degradation. Conversely, when the folding granularity N is too small, the tensor kernel's computational power cannot be fully utilized, resulting in poor performance. Therefore, the number of folds, i.e., the folding granularity N, can be determined by a trade-off between cost and benefit.

[0093] In some embodiments, method 300 further includes obtaining a folding granularity N, which can be set according to the computational power of the tensor computation kernel and the vector computation kernel. The vector computation kernel is the computation kernel originally intended for calculating matrix-vector multiplication. It can be considered whether using a tensor computation kernel can achieve faster computation and obtain benefits exceeding those of using a vector computation kernel, even when there are many redundant calculations, by taking into account the computational power of the vector computation kernel originally intended for calculating matrix-vector multiplication and the computational power of the tensor computation kernel used to replace the vector computation kernel.

[0094] In some embodiments, based on repeated experiments and calculations, N was found to be... When using a tensor kernel, even with a large number of redundant calculations, fast computation is possible and the benefits surpass those of using a vector kernel. Therefore, setting N... ,in It is the computing power of the tensor computing kernel, measured in floating-point operations per second (FLOPS). It is the computing power of the vector computing kernel, measured in floating-point operations per second (FLOPS).

[0095] The larger the dimension K, the larger the desired folding granularity N should be, because more folds allow for greater utilization of the parallel computing power of the tensor kernel. It was found that the larger the dimension K to be folded, the closer N is to... When the dimension K is large enough, N can be equal to .

[0096] For example, in a common scenario, the computing power of a vector computation kernel... The computing power of the tensor computation kernel is 200G FLOPS. For 900T FLOPS, therefore ≈67. Therefore, using the above formula, it can be concluded that in order to maximize the computing power of the tensor kernel, when the dimension K is large, N=64 (that is, the largest power of 2 less than 67).

[0097] Figure 5 An example diagram is shown illustrating the computational performance achieved using a tensor computing kernel with different values ​​for folding granularity in different dimensions K, according to at least one embodiment of the present disclosure.

[0098] Figure 5 The horizontal axis shows the computing performance (giga floating-point operations per second, GFLOPS) obtained under different values ​​of K (e.g., K=8192, K=16384, K=32768, K=49152, K=65536, K=81920) and different folding granularities N (shown as FLOD_K in the figure, including FLOD_K=1 (i.e., no folding), FLOD_K=16, FLOD_K=32, FLOD_K=64, FLOD_K=128), represented by different patterned bars. It can be seen that in some embodiments, when K is greater than 8192, N (i.e., FLOD_K in the figure) is 64, and the tensor computation kernel can achieve maximum computing power.

[0099] Alternatively, in some embodiments, when K is less than or equal to 8192 ( Figure 5 The diagram only shows the case where K=8192, but the performance is worse when K<8192, so the case where K<8192 is not shown. When K is small, N is set to 32. The main reason is that when K is small, the size of the folded K-dimensional dimension is smaller, which cannot fully utilize the pipelined parallel performance of the tensor computation kernel. Therefore, making the folding granularity N smaller can increase the size of the folded K-dimensional dimension, thereby obtaining better pipelined parallel performance.

[0100] Thus, according to at least one embodiment of this disclosure, by folding (transforming) the matrix and vector to be multiplied to generate two tensors, the tensor operation characteristics of the tensor computation kernel (e.g., matrix partitioning, sliding window of convolution, pipelined collaboration, etc.) are utilized to perform the product of the two tensors. Finally, the result of the MMV operation is easily obtained by adding the diagonal elements of the resulting matrix, achieving high-throughput, low-latency high-speed MMV operation. Furthermore, the optimal folding granularity can be determined based on the computing power of the tensor computation kernel, the vector computation kernel, and / or the size of the dimension to be folded, in order to achieve maximum computing power utilization and the best cost-benefit ratio.

[0101] in addition, Figure 4C A schematic diagram of the result vector computation of another matrix-vector multiplication according to at least one embodiment of the present disclosure is shown. Alternatively, instead of folding the matrix, the vector can be expanded to the same dimension as matrix A; that is, the vector can be expanded from 1*K to M*K by column copying, and then the GEMM operation with the matrix is ​​performed again to obtain an M*M output matrix, with the diagonal elements being the final output. Of course, this scheme expands the computation by a factor of M, so its performance is 1 / (M*M) of the vector computation kernel. This scheme may benefit in certain scenarios (e.g., when the tile_mma granularity is small). Similarly, after the result is calculated, it is not necessary to store the entire GEMM result in high-bandwidth memory (HBM); only the diagonal elements are stored in HBM.

[0102] Figure 6 A block diagram is shown of an apparatus 600 for calculating a matrix M*K multiplied by a vector K*1 using a tensor computation kernel according to at least one embodiment of the present disclosure.

[0103] The apparatus 600 may include a processor 610 and a memory 620, the memory 620 being coupled to the processor 610 and storing computer instructions therein for performing the steps of various methods of at least one embodiment of the present disclosure when executed by the processor 610. The computer instructions may implement the steps of various methods of at least one embodiment of the present disclosure in the form of code.

[0104] The processor 610 may include, but is not limited to, one or more processors or microprocessors.

[0105] The memory 620 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0106] Figure 7A block diagram is shown of another apparatus 700 for calculating a matrix M*K multiplied by a vector K*1 using a tensor computation kernel according to at least one embodiment of the present disclosure.

[0107] Here, M and K are positive integers. The device 700 may include a matrix transformation module 710, a vector transformation module 720, a calculation module 730, and an addition module 740.

[0108] The matrix transformation module 710 is configured to transform an M*K matrix into the first tensor of M*(K / N)*N, where N is a positive integer, N>1, and K is a multiple of N.

[0109] The vector transformation module 720 is configured to transform a K*1 vector into a second tensor of N*(K / N).

[0110] The computation module 730 is configured to perform general matrix multiplication of a first tensor of M*(K / N)*N and a second tensor of N*(K / N) in batches using a tensor computation kernel to obtain M result matrices.

[0111] The addition module 740 is configured to add the diagonal elements of each of the M result matrices to obtain each element of the M*1 result vector of the M*K matrix multiplied by the K*1 vector.

[0112] In some embodiments, the first tensor of M*(K / N)*N is M (K / N)*N first matrices. Each of the M (K / N)*N first matrices is formed by arranging the elements of each row of the first tensor in row-major order in the (K / N)*N first matrix, and the second tensor of N*(K / N) is formed by arranging the elements of each of the K*1 vectors in column-major order in the second tensor of N*(K / N).

[0113] In some embodiments, the apparatus 700 further includes an acquisition module (not shown) configured to acquire N, wherein N is set based on the computing power of the tensor computing kernel and the computing power of the vector computing kernel capable of computing a matrix multiplied by a vector of size K*1 for a matrix of size M*K.

[0114] In some embodiments, N ,in It refers to the computing power of a tensor computing kernel, measured in floating-point operations per second. It is the computing power of the vector computing kernel, measured in floating-point operations per second.

[0115] In some embodiments, the larger K is, the closer N is to ... .

[0116] In some embodiments, N is 32 when K is less than or equal to 8192; or N is 64 when K is greater than 8192.

[0117] In some embodiments, the addition module 740 is configured to add diagonal elements in a thread-local register and store the result of the addition in a high-bandwidth memory as a result vector.

[0118] Figure 8 A block diagram is shown of a computing device 800 that uses a tensor computing kernel to compute an M*K matrix multiplied by a K*1 vector according to at least one embodiment of the present disclosure.

[0119] The computing device 800 may include, for example Figure 6 or Figure 7 The device shown is 600 or 700 and multiple tensor computation cores 810.

[0120] It should be noted that in at least one embodiment of this disclosure, device 600 or 700 or computing device 800 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0121] For example, device 600 or 700 or computing device 800 may be implemented in hardware, software or a combination of hardware and software, and this disclosure does not impose any specific limitations on this.

[0122] The apparatus 600 or 700 or computing device 800 provided in at least one embodiment of this disclosure can achieve similar technical effects to the aforementioned method, and will not be described again here.

[0123] Figure 9 A schematic block diagram of an electronic device according to at least one embodiment of the present disclosure is shown.

[0124] like Figure 9 As shown, the electronic device 900 is, for example, suitable for implementing the methods provided in the embodiments of this disclosure. It should be noted that... Figure 9 The components of the electronic device 900 shown are merely exemplary and not limiting. The electronic device 900 may have other components depending on the actual application requirements.

[0125] like Figure 9 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions.

[0126] For example, when the computer-readable instructions are executed by the processing device 901, one or more steps of the method described according to any of the above embodiments can be performed. It should be noted that a detailed description of the method's processing procedure can be found in the relevant descriptions in the embodiments of the above methods.

[0127] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 903 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 908 into RAM 903 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 902, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0128] For example, the processing device 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0129] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, flash memory, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although... Figure 9 An electronic device 900 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 900 may alternatively implement or possess more or fewer devices. For example, a processing device 901 can control other components in the electronic device 900 to perform desired functions. The processing device 901 may be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be an x86, ARM, RISC-V architecture, etc. The GPU may be directly integrated into the SOC, directly integrated into the motherboard, or built into the northbridge chip of the motherboard.

[0130] According to at least one embodiment of this disclosure, a non-transitory computer-readable storage medium is also disclosed. Instructions, such as computer instructions, are stored on the non-transitory computer-readable storage medium. When the computer instructions are executed by a processor, the steps of the various methods described above can be performed. Non-transitory computer-readable storage media include, but are not limited to, RAM, ROM, flash memory, EPROM, electrically erasable programmable read-only memory (EEPROM), registers, computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.). For example, the non-transitory computer-readable storage medium can be connected to a computing device such as a computer, and then, when the computing device executes the computer instructions stored on the computer-readable storage medium, the steps of the various methods described above can be performed.

[0131] According to at least one embodiment of this disclosure, a computer program product is also disclosed, which can perform the methods, steps, and operations given herein. For example, such a computer program product may be a computer software package, computer code instructions, or a computer-readable tangible medium having computer instructions tangibly stored (or encoded) thereon, which can be executed by a processor to perform the operations described herein. The computer program product may include packaging materials.

[0132] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit at least one embodiment of the present disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for calculating the multiplication of an M*K matrix by a K*1 vector using a tensor kernel, wherein, M and K are positive integers, characterized in that the method includes: Convert the M*K matrix into the first tensor of M*(K / N)*N, where N is a positive integer, N>1, and K is a multiple of N; Convert the K*1 vector into a second tensor of N*(K / N); The tensor computation kernel is used to perform general matrix multiplication of the first tensor of M*(K / N)*N and the second tensor of N*(K / N) in batches to obtain M result matrices; The diagonal elements of each of the M result matrices are summed to obtain each element of the M*1 result vector obtained by multiplying the M*K matrix by the K*1 vector. The summation of the diagonal elements of each of the M result matrices to obtain each element of the M*1 result vector obtained by multiplying the M*K matrix by the K*1 vector includes: The diagonal elements are added in the thread-local register; The result of the addition is stored in a high-bandwidth memory as the result vector. The first tensor of the M*(K / N)*N is M (K / N)*N first matrices. Each of the M (K / N)*N first matrices is formed by arranging the elements of each row of the M-dimensional first tensor in the (K / N)*N first matrix in row-major order, and the N*(K / N) second tensor is formed by arranging the elements of each K*1 vector in the N*(K / N) second tensor in column-major order.

2. The method according to claim 1, characterized in that, The method further includes: Obtain N, where N is set based on the computing power of the tensor computing kernel and the computing power of the vector computing kernel that can be used to calculate the M*K matrix multiplied by the K*1 vector.

3. The method according to claim 2, characterized in that, N ,in This refers to the computing power of the tensor computation kernel, measured in floating-point operations per second. It is the computing power of the vector computing kernel, measured in floating-point operations per second.

4. The method according to claim 3, characterized in that, The larger K is, the closer N is to .

5. The method according to any one of claims 1-4, characterized in that, When K is less than or equal to 8192, N is 32; or when K is greater than 8192, N is 64.

6. An apparatus for calculating the multiplication of an M*K matrix by a K*1 vector using a tensor kernel, wherein, M and K are positive integers, characterized in that the device comprises: The matrix transformation module is configured to convert the M*K matrix into the first tensor of M*(K / N)*N, where N is a positive integer, N>1, and K is a multiple of N; The vector transformation module is configured to convert the K*1 vector into a second tensor of N*(K / N); The calculation module is configured to perform general matrix multiplication of the first tensor M*(K / N)*N and the second tensor N*(K / N) in batches using the tensor calculation kernel to obtain M result matrices; The addition module is configured to add the diagonal elements of each of the M result matrices to obtain each element of the M*1 result vector obtained by multiplying the M*K matrix by the K*1 vector. The summation of the diagonal elements of each of the M result matrices to obtain each element of the M*1 result vector obtained by multiplying the M*K matrix by the K*1 vector includes: The diagonal elements are added in the thread-local register; The result of the addition is stored in a high-bandwidth memory as the result vector. The first tensor of the M*(K / N)*N is M (K / N)*N first matrices. Each of the M (K / N)*N first matrices is formed by arranging the elements of each row of the M-dimensional first tensor in the (K / N)*N first matrix in row-major order, and the N*(K / N) second tensor is formed by arranging the elements of each K*1 vector in the N*(K / N) second tensor in column-major order.

7. An apparatus for calculating the multiplication of an M*K matrix by a K*1 vector using a tensor kernel, wherein, M and K are positive integers, characterized in that the device comprises: At least one memory storing computer instructions; At least one processor is configured to execute the computer instructions in the at least one memory to perform the method according to any one of claims 1-5.

8. A computing device, characterized in that, The computing device includes: The apparatus for calculating the multiplication of an M*K matrix by a K*1 vector using a tensor kernel, as described in claim 6 or 7; and Multiple tensor computation kernels.

9. A non-transitory computer-readable storage medium, characterized in that, It stores computer instructions. When the computer instructions are executed by at least one processor, the at least one processor causes the at least one processor to perform the method according to any one of claims 1-5.

Images