Method for accelerating matrix multiplication through processor, computing device and storage medium

Abstract

The embodiment of the invention provides a method for accelerating matrix multiplication through a processor, the processor supports a target instruction, and the target instruction is used for performing dot product on a plurality of sub-vectors of predetermined dimensions included in each of two input vectors and accumulating dot product results. The method comprises the following steps: storing a target row in a first matrix into a first vector register; first operation is executed multiple times, and the first operation comprises the steps that sub-vectors are intercepted from the designated positions of all columns of the second matrix to form column data, and the column data are stored in a second vector register; executing the target instruction on vectors in the first vector register and the second vector register, and accumulating dot product results to a third vector register bit by bit; performing first cyclic movement on the vector in the third vector register for next accumulation; and obtaining a multiplication result row of the target row and the second matrix according to data in the third vector register after the first operation is executed for multiple times.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to matrix multiplication operations, and more particularly, to accelerating matrix multiplication using a processor. Background Art

[0002] Matrix multiplication is a core operation in many scientific computing and engineering applications, such as deep learning and high-performance computing. However, on traditional processors (such as CPUs), its implementation efficiency can be limited by multiple loop nests and memory access bottlenecks, especially when processing large amounts of data. Therefore, improved solutions are needed to accelerate matrix multiplication. Summary of the Invention

[0003] According to a first aspect, a method for accelerating matrix multiplication by a processor is provided, wherein the processor supports a target instruction, and the target instruction is used to perform bitwise dot products on a plurality of sub-vectors of predetermined dimensions respectively included in two input vectors, and accumulate the dot product results; the method comprises: storing a target row in a first matrix into a first vector register; performing a first operation multiple times, and the first operation comprises: intercepting sub-vectors from respective specified positions of each column of a second matrix to form column data, and storing the column data into a second vector register; executing the target instruction on the vectors in the first vector register and the second vector register, and accumulating the dot product results bitwise to a third vector register; performing a first circular shift on the vectors in the third vector register for the next accumulation; and obtaining a row of multiplication results of the target row and the second matrix based on the data in the third vector register after performing the first operation multiple times.

[0004] In one embodiment, the two input vectors each include n consecutive sub-vectors, the predetermined dimension of each sub-vector is m, and the dimensions of the first and second matrices are both n*n.

[0005] In one embodiment, storing the target row in the first matrix into the first vector register includes: copying the target row m times and storing the copies into the first vector register.

[0006] In one embodiment, sub-vectors are respectively intercepted from respective designated positions of respective columns of the second matrix to form column data, including: intercepting an m-dimensional sub-vector from each column of the second matrix to form the column data, wherein the position of the intercepted sub-vector of each column is circularly shifted by m bits relative to the previous column in a first predetermined direction, and is circularly shifted by m bits relative to the position of the sub-vector intercepted from the same column in the previous first operation in a second predetermined direction.

[0007] In one embodiment, the first predetermined direction is a downward direction, and the multiple times are n / m times.

[0008] In one embodiment, the first circular movement is a leftward circular movement; and the second predetermined direction is an upward direction.

[0009] In one embodiment, the first circular movement is a rightward circular movement; and the second predetermined direction is a downward direction.

[0010] In one embodiment, in any b-th first operation, the column data is arranged by cyclically extracting sub-vectors from each column of the second matrix starting from the b-th column and proceeding to the right, wherein the sub-vectors in the b-th column are extracted from the top position.

[0011] In one embodiment, in the first first operation, the column data is arranged by cyclically extracting subvectors from each column of the second matrix starting from the 1st column and proceeding to the right, wherein the subvectors in the 1st column are extracted from the top position, and in any subsequent b-th first operation, the column data is arranged by cyclically extracting subvectors from each column of the second matrix starting from the n+2-bth column and proceeding to the right, wherein the subvectors in the n+2-bth column are extracted from the top position.

[0012] In one embodiment, obtaining a row of multiplication results of the target row and the second matrix based on data in the third vector register after multiple executions of the first operation includes: performing a second circular shift on the vector of the third vector register obtained after multiple executions of the first operation for the multiple times to obtain a row of multiplication results of the target row and the second matrix, wherein the direction of the second circular shift is opposite to the direction of the first circular shift.

[0013] In one embodiment, n is 16 and m is 2.

[0014] In one embodiment, the size of each element in the first and second vector registers is 16 bits, and the size of each element in the third vector register is 32 bits.

[0015] In one embodiment, the processor is a general-purpose processor.

[0016] According to the second aspect, a computing device is provided, comprising: a processor supporting a target instruction, wherein the target instruction is used to perform dot products on the bits of sub-vectors of predetermined dimensions respectively included in two input vectors, and to accumulate the dot product results; and a memory storing executable code thereon, wherein when the executable code is executed by the processor, the processor executes the method of any one of the embodiments of the first aspect above.

[0017] According to a third aspect, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of a computing device, the processor is caused to execute the method of any embodiment of the first aspect.

[0018] In an embodiment of the present specification, a scheme is proposed for accelerating matrix multiplication using a processor that supports a target instruction, wherein the target instruction can implement dot accumulation and addition operations for multiple groups of data at a time. The scheme can fully utilize the dot accumulation and addition operations of the target instruction through matrix element layout setting and circular shift accumulation results, and convert the matrix multiplication task into multiple dot accumulation and addition operations, thereby reducing the use of additional other instructions, such as horizontal accumulation instructions, and thus accelerating the matrix multiplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1A and Figure 1B Schematic diagrams respectively showing the operations of two exemplary target instructions supported by the processor according to some embodiments;

[0021] Figure 2 An exemplary flow chart illustrating a method of accelerating matrix multiplication by a processor according to some embodiments;

[0022] Figure 3 A schematic diagram illustrating elements of a vector in a first vector register according to some embodiments;

[0023] Figure 4 A schematic diagram illustrating a relationship between column data stored in a second vector register and a second matrix according to some embodiments;

[0024] Figure 5 a schematic diagram illustrating elements of a vector in a third vector register according to some embodiments;

[0025] Figure 6 A schematic diagram illustrating an exemplary structure of a computing device according to some embodiments is shown. DETAILED DESCRIPTION

[0026] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0027] Matrix multiplication is a mathematical operation performed on two matrices. It multiplies the corresponding elements of the rows and columns of the two matrices and adds them together to produce an element in the new matrix. This process is repeated for all rows and columns until the complete result matrix is obtained.

[0028] Several schemes have been proposed to accelerate matrix multiplication, such as using dedicated hardware and / or specific instruction sets.

[0029] For example, Tensor Cores are specialized hardware designed for deep learning on NVIDIA GPUs, accelerating matrix multiplication and convolution operations. Google's TPU is custom hardware designed specifically for deep learning training and inference, supporting efficient matrix multiplication acceleration. However, GPUs and TPUs employ heterogeneous computing, resulting in more complex architectures and the introduction of additional data copies. Furthermore, TPUs are specifically tailored for deep learning, making them less suitable for general-purpose computing.

[0030] ARM NEON is a single-instruction, multiple-data instruction set on the ARM platform that supports vectorized computing and is used to accelerate tasks such as matrix operations. However, ARM chips are primarily used on mobile devices, and their vector bit width and register count are generationally different from those of the X86 architecture. They are primarily suitable for lightweight tasks and face hardware bottlenecks when handling large-scale matrix operations.

[0031] Therefore, it is desirable to have an improved scheme for accelerating matrix multiplication that has wider applicability.

[0032] The inventors have discovered that in the instruction set of a general-purpose processor, there is a class of instructions that integrates multiplication and addition operations into a single, efficient instruction, which can implement dot-accumulation operations on multiple sub-vectors at a time. This instruction may also be referred to as a target instruction in the following text. The "target instruction" in this disclosure is used to perform dot products on the bits of multiple sub-vectors of predetermined dimensions included in each of two input vectors, and to accumulate the dot product results, that is, to accumulate them bit by bit into an accumulation vector. It will be understood that in this disclosure, the target instruction is not limited to instructions supported by general-purpose processors, but includes instructions supported by all processors, as long as it meets the definition of the target instruction mentioned above.

[0033] For example, the AVX-VNNI (Advanced Vector Extensions-Vector Neural Network Instructions) instruction set was introduced into the X86 processor architecture, which is a part of the AVX-512 instruction set. The instruction set includes some of the above-mentioned target instructions.

[0034] Figure 1A and Figure 1BThe operation diagrams of two exemplary target instructions in AVX-VNNI, namely VPDPWSSD and VPDPBUSD instructions, are given respectively.

[0035] like Figure 1A As shown, the VPDPWSSD instruction has three operands SRC1, SRC2 and SRC3, which may also be referred to as a first input vector, a second input vector and an accumulation vector.

[0036] The three operands SRC1, SRC2 and SRC3 are all 512-bit, where SRC1 and SRC2 are both 32-dimensional vectors, i.e., they have 32 16-bit elements A0 to A1 respectively. 31 , and B0 to B 31 , and SRC3 is a 16-dimensional vector, that is, it has 16 32-bit elements C0 to C 15 For example, the three operands SRC1, SRC2, and SRC3 can all use __m512 i type operands.

[0037] exist Figure 1A In FIG, the accumulation vector SRC3 shows the accumulation value before the instruction is executed, and the bottom DEST represents the dot accumulation operation performed by the instruction, the result of which is the value of the accumulation vector SRC3 after the instruction is executed.

[0038] like Figure 1A As shown in DEST in , the operations performed by the VPDPWSSD instruction can be viewed as follows:

[0039] The two input vectors SRC1 and SRC2 are divided into 16 2-dimensional sub-vectors, namely, [A0, A1], [A2, A3], ..., [A 30 ,A 31 ], and [B0,B1], [B2,B3], ..., [B 30 ,B 31 ],

[0040] Perform dot products on the sub-vectors at corresponding positions in the two input vectors, and get 16 dot product results A0B0+A1B1, A2B2+A3B3, ..., A 30 B 30 +A 31 B 31 ,

[0041] The 16 dot product results are added bit by bit to the accumulation vector SRC3, that is, the original values ​​[C0, C1, ..., C 15 ] are added together to get the final accumulated value of the accumulated vector SRC3, that is, [A0B0+A1B1+C0,A2B2+A3B3+C1,…,A30 B 30 +A 31 B 31 +C 15 ].

[0042] For example, in actual use, three vectors a, b, and c can be defined as the three operands SRC1, SRC2, and SRC3, respectively. Then, the VNNI instruction _mm512_dpwssd_epi32(c,a,b) can be used to perform the operation of the target instruction, and the result is stored in the returned vector c. When executing this instruction, the three vectors are stored in the corresponding vector registers (such as ZMM registers).

[0043] Figure 1B Shown with Figure 1A Similar instruction operations, the only difference is that SRC1 and SRC2 are both 64-dimensional vectors, that is, they have 64 8-bit elements A0 to A1 respectively. 63 , and B0 to B 63 , and are divided into 16 4-dimensional sub-vectors.

[0044] Specifically, if Figure 1B As shown, the three operands SRC1, SRC2 and SRC3 of the VPDPBUSD instruction are all 512-bit, where SRC1 and SRC2 are both 64-dimensional vectors, that is, they have 64 8-bit elements A0 to A1 respectively. 63 , and B0 to B 63 , and SRC3 is a 16-dimensional vector, that is, it has 16 32-bit elements C0 to C 15 .

[0045] like Figure 1B As shown in DEST in , the operations performed by the VPDPBUSD instruction can be viewed as follows:

[0046] The two input vectors SRC1 and SRC2 are divided into 16 4-dimensional sub-vectors, namely, [A0,…,A3], [A4,…,A7], …, [A 60 ,…,A 63 ], and [B0,…,B3], [B4,…,B7], …, [B 60 ,…,B 63 ],

[0047] Perform dot products on the sub-vectors at corresponding positions in the two input vectors, and get 16 dot product results A0B0+A1B1+A2B2+A3B3, ..., A 60 B 60 +A 61 B 61 +A62 B 62 +A 63 B 63 ,

[0048] The 16 dot product results are added bit by bit to the accumulation vector SRC3, that is, the original values ​​[C0, C1, ..., C 15 ] are added together to get the final accumulated value of the accumulated vector SRC3, that is, [A0B0+A1B1+A2B2+A3B3+C0,…,A 60 B 60 +A 61 B 61 +A 62 B 62 +A 63 B 63 +C 15 ].

[0049] The aforementioned AVX-VNNI instructions significantly accelerate low-precision (such as INT8 and INT16) dot product operations through hardware support. These instructions fuse multiplication and addition operations into a single, efficient instruction while optimizing data parallel processing and memory bandwidth utilization, making them an effective tool for accelerating matrix multiplication, especially for scenarios such as deep learning inference.

[0050] It is understandable that the above Figure 1A and Figure 1B The target instructions shown are merely illustrative and not limiting upon the embodiments of the present disclosure. For example, the embodiments of the present disclosure do not limit the total bit size of the operands and the bit size of each element therein, nor do they limit the dimensions of the operands and the dimensions of their sub-vectors.

[0051] However, as discussed above, such target instructions cannot be used to directly implement matrix multiplication.

[0052] To this end, the embodiment of the present disclosure proposes a scheme for accelerating matrix multiplication by using a processor that supports the above-mentioned target instruction, which decomposes the multiplication of two matrices (i.e., the first and second matrices) into multiplication operations of each row of the first matrix with the second matrix, wherein for the target row of the first matrix, the elements in the second matrix can be divided into multiple sets in a predetermined manner, and the above-mentioned target instruction is executed once for each set and the target row, and the accumulated result obtained after each operation is cyclically moved to finally obtain the multiplication result row of the target row and the second matrix. Therefore, the scheme of the embodiment of the present disclosure can make full use of the dot accumulation and addition operation of the target instruction, convert the matrix multiplication task into multiple dot accumulation and addition operations, wherein multiple multiplications and additions can be performed simultaneously each time, and reduce the use of additional other instructions, such as horizontal accumulation instructions, thereby greatly improving the computational efficiency of matrix multiplication and accelerating the operation of matrix multiplication.

[0053] In addition, in some embodiments, a general-purpose processor that supports the above-mentioned target instructions can be used to accelerate matrix multiplication, thereby eliminating the need for additional dedicated hardware and supporting general computing tasks. Therefore, its solution has wider universality, can be applied to a wider range of application scenarios, and has high cost performance and flexible deployment.

[0054] Figure 2 An exemplary flow chart of a method for accelerating matrix multiplication by a processor according to some embodiments is shown, wherein the processor supports the aforementioned target instruction and performs multiplication of a first matrix and a second matrix.

[0055] like Figure 2 As shown, the method according to an embodiment of the present disclosure includes:

[0056] Step S210, storing the target row in the first matrix into the first vector register;

[0057] Step S220, performing the first operation multiple times (N times, N>1), wherein the first operation includes: step S221, intercepting sub-vectors from respective specified positions of each column of the second matrix to form column data, and storing the column data into the second vector register; step S222, executing the target instruction on the vectors in the first vector register and the second vector register, and accumulating the dot product results bit by bit to the third vector register; step S223, performing a first circular shift on the vectors in the third vector register for the next accumulation;

[0058] Step S230 , obtaining a multiplication result row of the target row and the second matrix according to the data in the third vector register after performing the first operation multiple times.

[0059] The following describes the execution of the above steps in detail with reference to specific examples.

[0060] In some examples, the multiplication of the first and second matrices can be decomposed into the multiplication of each row of the first matrix with the second matrix, that is, each row of the first matrix is taken as the target row in step S210 and the corresponding row in the multiplication result matrix (that is, the multiplication result row mentioned above) is obtained in step S230. Figure 2 After the operation, the complete multiplication result matrix is obtained, that is, the matrix multiplication is completed.

[0061] For example, the target rows in the first matrix in step S210 may be determined sequentially from top to bottom.

[0062] In some examples, the two input vectors of the target instruction used each include n consecutive subvectors, each of which has a predetermined dimension of m. Therefore, the two input vectors each have n*m elements, while the dimension of the accumulation vector of the target instruction is n. The dimensions of the first and second matrices to be multiplied are both n*n. In some cases, the dimensions of the two matrices actually multiplied are not n*n. In this case, the multiplication can be transformed into one or more n*n matrix multiplications through various splitting and / or padding methods.

[0063] The first and second vector registers respectively store two input vectors of the target instruction, and thus can store n*m elements. A row of the first matrix has n elements. Therefore, in step S210, the target row of the first matrix can be copied m times and stored in the first vector register.

[0064] Figures 3 to 5 Shows a Figure 1A The target instruction shown implements a specific example of matrix multiplication of a first matrix A and a second matrix B. In this example, the processor is a general-purpose processor, n is 16, m is 2, and the first and second vector registers (corresponding to Figure 1A Each element in SRC1 and SRC2 in the vector register is 16 bits in size, and the third vector register (corresponding to Figure 1A Each element in SRC3 in is 32 bits in size. Each element in matrix A and matrix B uses row index (i) and column index (j), i.e., A i,j and B i,j To express. Figures 3 to 5 The calculation of the product of the first row vector A0 of matrix A and matrix B is used as an example for explanation.

[0065] like Figure 3 As shown, in the above step S210, A0 is copied twice and placed into the first 256-bit and the last 256-bit of the first vector register SRC1 that accommodates __m512i type operands, respectively, to obtain the first input vector A'0[A 0,0 ,A0,1 ,…,A 0,14 ,A 0,15 ,A 0,0 ,A 0,1 ,…,A 0,14 ,A 0,15 ]; "'" is added here to distinguish it from the row or column vector of the original matrix.

[0066] Then, in step S220, according to the rules of matrix multiplication, the elements of the second matrix are divided into N groups, where no elements overlap with each other. Each time a group of elements is stored in the second vector register, and the processor executes a target instruction once with the first vector register that previously stored the target row of the first matrix. The execution result is stored as an accumulated vector in the third vector register. After the execution is completed, the accumulated vector in the third vector register is subjected to a first circular shift for the next accumulation. That is, before the next target instruction is executed, the accumulated vector after the circular shift is stored in the third vector register. Thus, the above-mentioned first operation is performed N times. It can be understood that when the first first operation starts, the accumulated vector in the third vector register is initialized to a zero vector.

[0067] In some examples, the number N is determined by the number n of sub-vectors included in each of the two input vectors of the target instruction and the predetermined dimension m of each sub-vector, that is, N=n / m.

[0068] The above-mentioned N groups of elements correspond to the column data formed by intercepting sub-vectors from the respective specified positions of each column of the second matrix in each first operation. The specified position of any column in the second matrix in each first operation is different, so that the column can be traversed. The dimension of the sub-vector intercepted by each column is also m. In each first operation, the specified position of the sub-vector intercepted by each column and the arrangement order of each sub-vector in the column data are set so that: the row number (row index) of each element in the column data is the same as the column number (column index) of the element at the corresponding position in the first vector register, and the column number of each sub-vector is the same as the column number of the element at the corresponding position in the accumulated vector in the current third vector register in the multiplication result matrix.

[0069] Some specific examples of what constitutes column data are given below.

[0070] In some examples, an m-dimensional sub-vector is respectively intercepted from each column of the second matrix to form the column data, wherein the position of the sub-vector intercepted from each column is circularly shifted by m bits relative to its previous column in a first predetermined direction, and is circularly shifted by m bits relative to the position of the sub-vector intercepted from the same column in the previous first operation in a second predetermined direction.

[0071] The previous column mentioned above refers to the previous column obtained by performing a circular left shift (i.e., circularly subtracting one) on the column index. That is, the previous columns of columns 2 to n (column indexes start from 0, corresponding to columns 1 to n-1) are columns 1 to n-1 (columns 0 to n-2), and the previous column of column 1 (column 0) is column n (column n-1).

[0072] The first and second predetermined directions are both along the direction of the column, that is, upward or downward. The first and second predetermined directions can be set as needed, and can be the same or different.

[0073] Because each subvector is composed of consecutive elements in the column, the position of each subvector can be determined by the position of its first element in the column (which can be represented by its row index). For example, a circular shift of a subvector's position can refer to a circular shift of the row index of its first element. If the subvector is circularly shifted upward by m bits, the row index of the first element is circularly shifted left by m bits (i.e., circularly decremented by m). If the subvector is circularly shifted downward by m bits, the row index of the first element is circularly shifted right by m bits (i.e., circularly incremented by m).

[0074] It will be appreciated that in the above example, by presetting the position of the sub-vector extracted from any column in the first first operation, the positions of the sub-vectors extracted from all columns in all N first operations can be determined based on the first and second predetermined directions. The arrangement of the elements in the first vector register and the order of the sub-vectors extracted from each column in the column data can be pre-set, thereby presetting the position of the sub-vector extracted from any column in the first first operation, so that the dot product operation performed thereafter complies with the rules of matrix multiplication.

[0075] For example, the first predetermined direction may be a downward direction. If the first circular movement of the vector in the third vector register in step S223 is a leftward circular movement, the second predetermined direction may be an upward direction, and if the first circular movement is a rightward circular movement, the second predetermined direction may be a downward direction.

[0076] In the case where the first circular movement is a left circular movement, in any b-th first operation, the column data is arranged by sequentially intercepting sub-vectors from each column starting from the b-th column of the second matrix and circulating rightward, wherein the sub-vectors in the b-th column are intercepted starting from the top position.

[0077] In the case where the first circular movement is a right circular movement, in the first first operation, the column data is arranged by starting from the 1st column of the second matrix and circulating rightward in sequence from sub-vectors intercepted from each column, wherein the sub-vectors in the 1st column are intercepted from the top position, and in any subsequent b-th first operation, the column data is arranged by starting from the n+2-bth column of the second matrix and circulating rightward in sequence from each column, wherein the sub-vectors in the n+2-bth column are intercepted from the top position.

[0078] Then Figures 3 to 5 The process of the above step S220 is described in detail with an example.

[0079] first, Figure 4 The upper part shows all the elements of the second matrix B, which are divided into 8 groups (N=n / m=16 / 2=8) and represented by 8 different colors, that is, the elements of each color constitute a column of data.

[0080] Figure 4 The lower part shows the column data used in the eight first operations respectively.

[0081] like Figure 4 As shown, the elements with the same color in the matrix B are combined into column data and read into the second vector register SRC2 storing the operand of type __m512i, wherein the elements of these eight colors starting from the white in the first column of the first two rows are taken to the right in sequence as the column data in the first to eighth operations respectively.

[0082] In this example, the first circular shift of the vector in the third vector register in step S223 is a circular shift to the left, so in any b-th (1≤b≤8) first operation, the column data is arranged by circularly starting from the b-th column of the matrix B and sequentially intercepting an m-dimensional subvector from each column, wherein the subvector in the b-th column is intercepted from the top position, and the position of the subvector intercepted in each column is circularly shifted downward by m positions relative to the previous column. Figure 4 It can also be seen that the position of the sub-vector intercepted by each column in any b-th first operation is cyclically shifted upward by m positions relative to the position of the sub-vector intercepted by the same column in the previous operation.

[0083] like Figure 4 As shown in the lower part, in the first operation, starting from the first column and circulating to the right, the white elements in each column are read in sequence as column data, that is, the second input vector in the second vector register SRC2, B'0[B 0,0 ,B 1,0 ,B 2,1 ,B 3,1 ,…,B 14,7 ,B15,7 ,B 0,8 ,B 1,8 ,B 2,9 ,B 3,9 ,…,B 14,15 ,B 15,15 In the second first operation, starting from the second column and circulating to the right, the blue elements in each column are read in turn as column data B'1[B 0,1 ,B 1,1 ,B 2,2 ,B 3,2 ,…,B 14,8 ,B 15,8 ,B 0,9 ,B 1,9 ,B 2,10 ,B 3,10 ,…,B 14,0 ,B 15,0 ]. This continues until, in the eighth first operation, starting from the eighth column and circulating to the right, the red elements in each column are read in sequence as column data B'7[B 0,7 ,B 1,7 ,B 2,8 ,B 3,8 ,…,B 14,14 ,B 15,14 ,B 0,15 ,B 1,15 ,B 2,0 ,B 3,0 ,…,B 14,6 ,B 15,6 ], at this time the entire matrix B has been read.

[0084] Then, in each first operation, Figure 3 The vector sum in the first vector register SRC1 is shown Figure 4 The corresponding vector in the second vector register SRC2 shown in the lower part of executes the target instruction, for example Figure 1A The VPDPWSSD instruction shown in FIG. 1 accumulates the dot product result bit by bit to the third vector register SRC3, and then circularly shifts the vector in the third vector register SRC3 to the left for the next accumulation, thereby obtaining Figure 5 The accumulated vector in the third vector register SRC3 is shown. In this example, the multiplication result matrix of the first matrix A and the second matrix B is represented by matrix C, whereby the multiplication result row of the first row vector A0 of matrix A and the multiplication result row of matrix B is the first row vector C0 of matrix C.

[0085] For example, in the first operation, the row vector C0 is initialized to a zero vector and stored in the third vector register SRC3 as an accumulation vector, and the VPDPWSSD instruction _mm512_dpwssd_epi32 (C0, A'0, B'0) is used to perform a dot accumulation operation, and finally the following is obtained: Figure 5 The accumulation vector [C 0,0 ,C 0,1 ,C 0,2 ,…,C 0,15 Then, the accumulated vector in the third vector register SRC3 is circularly shifted left to serve as the accumulated vector in the second first operation, as shown in FIG. Figure 5 This operation is repeated until the accumulated vector is circularly shifted left in the eighth first operation, and the result is as follows: Figure 5 The accumulation vector [C 0,8 ,C 0,9 ,C 0,10 ,…,C 0,7 ]; It can be seen from this that after executing the first operation 8 times (i.e., the above-mentioned step S220), the position of the elements in the final accumulated vector must be changed to obtain Figure 5 The final multiplication result row C0 shown in the last row in , that is, the above-mentioned step S230 is executed.

[0086] from Figure 5 It can be seen that in step S230, the final accumulated vector ( Figure 5 The positions of the first half of the elements (first 256-bit) and the second half of the elements (last 256-bit) in the accumulated vector shown in the 9th row of the accumulator are obtained. Figure 5 The last line of the multiplication result is shown in [C 0,0 ,C 0,1 ,C 0,2 ,…,C 0,15 ]. This position transformation can also be regarded as Figure 5 The accumulated vector shown in the 9th row in the figure is circularly shifted right 8 times to offset the previous 8 circular left shifts, thereby obtaining a multiplication result row; therefore, it can be regularly summarized as follows: in step S230, the vector of the third vector register obtained after performing the first operation in step S220 for the multiple times (N times) is subjected to a second circular shift for the multiple times (N times) to obtain a multiplication result row of the target row of the first matrix and the second matrix, wherein the direction of the second circular shift is opposite to the direction of the first circular shift in step S223.

[0087] It can be understood that according to the above description, those skilled in the art can easily think of Figures 3 to 5The examples shown can be modified in various ways, all of which are included in the scope of protection of the present disclosure; for example, a simple modification can be made to use Figure 1B The target instruction shown is replaced by Figure 1A The target instruction shown can be used to accelerate matrix multiplication, or the circular left shift in step S223 can be changed to a circular right shift, etc.; no further details will be given here.

[0088] Combine Figures 3 to 5 As can be seen from the example, the solution of the embodiment of the present disclosure effectively utilizes the ability of multiple groups of sub-vectors of related instructions in the AVX-VNNI instruction set to perform dot accumulation and addition simultaneously through technologies such as matrix element layout transformation and circular move accumulation, and converts the matrix multiplication task into multiple dot accumulation and addition operations, in which multiple multiplications and additions are performed simultaneously each time, and reduces the use of additional instructions such as horizontal accumulation instructions, thereby greatly improving the computational efficiency of matrix multiplication.

[0089] In addition, since the AVX-VNNI instruction set is part of the X86 architecture and can run on all general-purpose processors (such as CPUs) that support this instruction set, the solution of the embodiment of the present disclosure does not require the purchase of additional dedicated hardware and has low operating power consumption; thus, compared with the aforementioned Tensor Core and Google TPU, it has wider universality and lower power consumption. For example, the solution of the embodiment of the present disclosure is based on a general-purpose processor and the AVX-VNNI instruction set, has high cost performance and flexible deployment, and can be widely used in cloud reasoning, edge computing and other environments without relying on additional hardware accelerators.

[0090] In addition, since the AVX-VNNI instruction set supports general computing tasks and is suitable for a wide range of application scenarios such as scientific computing, graphics processing, and data analysis, the solution of the embodiment of the present disclosure is more applicable in general matrix multiplication scenarios compared to Google TPU.

[0091] In addition, because the AVX-VNNI instruction set supports wider vector registers and dot product instructions, a single instruction can complete the multiplication and addition of multiple low-precision data. Therefore, compared with the aforementioned NEON instruction set, the AVX-VNNI instruction set can process more data blocks in the same cycle, making the solution of the disclosed embodiment suitable for high-performance deep learning reasoning. In addition, the AVX-VNNI instruction set is a native instruction of the X86 architecture and has mature software and hardware support, making the application ecosystem of the solution of the disclosed embodiment more mature.

[0092] According to another embodiment, a computing device is also provided, such as the following Figure 6 shown.

[0093] Figure 6A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned method of accelerating matrix multiplication through a processor according to at least one embodiment of the present disclosure is shown.

[0094] See also Figure 6 The computing device 600 includes a memory 610 and a processor 620, wherein the processor 620 supports a target instruction, wherein the target instruction is used to perform dot product on the bits of sub-vectors of predetermined dimensions respectively included in two input vectors, and accumulate the dot product results.

[0095] The processor 620 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 620 may be a general-purpose processor, which may include one or more single-core or multi-core general-purpose processors. For example, the processor 620 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc.; in this case, the processor 620 may also be referred to as a general-purpose processor. In some embodiments, the processor 620 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0096] The memory 610 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 620 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that retains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 610 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 610 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0097] The memory 610 stores executable codes. When the executable codes are processed by the processor 620 , the processor 620 can execute the above-mentioned method of accelerating matrix multiplication by the processor.

[0098] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0099] Alternatively, the present disclosure may also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of a computing device (or electronic device, server, etc.), the processor is caused to perform the various steps of the above-mentioned method according to the present disclosure.

[0100] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0101] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems and methods according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0103] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for accelerating matrix multiplication using a processor, the processor supporting a target instruction for performing a dot product of a plurality of subvectors of a predetermined dimension included in two input vectors, and accumulating the dot product results; The method comprises: Store the target row in the first matrix into the first vector register; performing a first operation multiple times, the first operation comprising: intercepting sub-vectors from respective designated positions of respective columns of the second matrix to form column data, and storing the column data in a second vector register; executing the target instruction on the vectors in the first vector register and the second vector register, and accumulating dot product results bit by bit to a third vector register; and performing a first circular shift on the vectors in the third vector register for a next accumulation; A multiplication result row of the target row and the second matrix is obtained according to the data in the third vector register after performing the first operation multiple times.

2. The method according to claim 1, wherein The two input vectors each include n consecutive sub-vectors, the predetermined dimension of each sub-vector is m, and the dimensions of the first and second matrices are both n*n.

3. The method according to claim 2, wherein: Storing the target row in the first matrix into the first vector register includes: copying the target row m times and storing the copies into the first vector register.

4. The method according to claim 2, wherein: Subvectors are respectively intercepted from respective specified positions of each column of the second matrix to form column data, including: An m-dimensional sub-vector is respectively intercepted from each column of the second matrix to form the column data, wherein the position of the sub-vector intercepted in each column is circularly shifted by m bits relative to the previous column in a first predetermined direction, and is circularly shifted by m bits relative to the position of the sub-vector intercepted in the same column in the previous first operation in a second predetermined direction.

5. The method according to claim 4, wherein The first predetermined direction is a downward direction, and the multiple times are n / m times.

6. The method according to claim 4, wherein: The first circular movement is a leftward circular movement; and the second predetermined direction is an upward direction.

7. The method according to claim 4, wherein: The first circular movement is a rightward circular movement; and the second predetermined direction is a downward direction.

8. The method according to claim 6, wherein: In any b-th first operation, the column data is arranged by cyclically extracting subvectors from each column of the second matrix starting from the b-th column and proceeding rightward, wherein the subvectors in the b-th column are extracted from the top position.

9. The method according to claim 7, wherein: In the first first operation, the column data is arranged by cyclically extracting sub-vectors from each column starting from the 1st column of the second matrix and proceeding to the right, wherein the sub-vectors in the 1st column are extracted from the top position, and in any subsequent b-th first operation, the column data is arranged by cyclically extracting sub-vectors from each column starting from the n+2-bth column of the second matrix and proceeding to the right, wherein the sub-vectors in the n+2-bth column are extracted from the top position.

10. The method according to claim 1, wherein Obtaining, according to the data in the third vector register after performing the first operation multiple times, a multiplication result row of the target row and the second matrix, comprising: Perform a second circular shift on the vector of the third vector register obtained after performing the first operation multiple times, the multiple times, to obtain a multiplication result row of the target row and the second matrix, wherein the direction of the second circular shift is opposite to the direction of the first circular shift.

11. The method according to claim 2, wherein: n is 16, m is 2; the size of each element in the first and second vector registers is 16 bits, and the size of each element in the third vector register is 32 bits.

12. The method according to claim 1, wherein The processor is a general-purpose processor CPU.

13. A computing device comprising: A processor supporting a target instruction for performing a dot product on each of two input vectors of a predetermined dimension and accumulating the dot product results. as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 12.

14. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of a computing device, causes the processor to perform the method according to any one of claims 1 to 12.

Images