An incremental matrix multiplication accelerator for HPC / AI applications

Abstract

The present invention discloses an incremental matrix multiplication accelerator for HPC / AI applications, comprising an HPC core, a Maidong accelerator core MZ, high-bandwidth memory HBM, global shared memory GSM, and a data bus. The Maidong accelerator core MZ includes a systolic array SA, a B buffer, and a C buffer. The B buffer is used to cache the matrix B required for matrix multiplication input to the systolic array SA, the C buffer is used to cache the matrix C required for matrix multiplication input to the systolic array SA, and the global shared memory GSM includes an A buffer for storing the matrix A required for matrix multiplication in the systolic array SA. The present invention can save storage resources without affecting performance, meet the bandwidth requirements of the systolic array, and can complete high-performance GEMM tasks simultaneously with the original HPC core without changing the original HPC core ecosystem and is compatible with the original matrix function library.

Description

Technical Field

[0001] The present invention relates to high performance computing (HPC) and artificial intelligence (AI) computing technologies, and in particular to an incremental matrix multiplication accelerator for HPC / AI applications. Background Art

[0002] Since the emergence of deep learning, the convergence of high-performance computing (HPC) and artificial intelligence (AI) computing power has been a growing trend, both in terms of cost and computing power, combined with the traditional high-performance computing needs of data centers. Research shows that the core of both HPC and AI operations lies in general matrix multiplication (GEMM), differing only in precision. For example, the core operations of convolutional and fully connected layers in typical convolutional neural networks (CNNs), which account for the greatest computational demand, can be converted to matrix multiplication through im2col transformation or batch operations. Transformer, a mainstream model in natural language processing (NLP), also operates at the core of matrix multiplication. To improve the performance of HPC and AI computing, adding acceleration cores optimized for GEMM operations to HPC chips is a mainstream technical approach. Systolic arrays are widely adopted as one of the most efficient architectures for performing matrix multiplication, due to their relatively simple data flow control.

[0003] The structure of the systolic array (SA) is as follows Figure 1As shown in Figure 1, it consists of an array of processing elements (PEs) with r rows and c columns. Each PE is typically a multiplier and accumulator (MAC), capable of pipelined multiplication and addition operations. The GEMM calculation form of a systolic array is Y = AB + C. During AI calculations, matrices A and B represent the weight and feature map matrices, respectively, C represents the offset matrix, and Y is the calculation result. Therefore, three buffers, A, B, and C, are typically required. Buffer C is used to store the initial offset matrix C and intermediate calculation results. Systolic arrays typically have three data flow formats: weight stationary (WS), input stationary (IS), and output stationary (OS). These formats correspond to preloading A, matrix B, and matrix C into the PE, respectively. This article only discusses the IS and WS data flow systolic arrays. The preloaded matrix is ​​a fixed matrix, and the matrix being multiplied is a dynamic matrix, dynamically loaded into adjacent PEs in sequence along the row direction for calculation.

[0004] Typical processors based on systolic arrays include Google's Tensor Processing Unit (TPU), whose structure is as follows: Figure 2 As shown, it is manufactured using 28nm process and the final area is less than 330um 2 The TPU's systolic array contains 256*256=65536 PEs. Despite this large number of PEs, they only occupy a quarter of the chip's area. Even with the computational control unit, the area still accounts for less than half. To increase the computational density of the systolic array, the memory unit needs to continuously provide data to the computational units, increasing bandwidth. Therefore, the TPU's on-chip memory occupies nearly one-third of the chip's area.

[0005] The background of this paper is to design an incremental systolic array accelerator core within a chip that already has high-performance computing capabilities. The basic structure of an HPC chip includes HPC cores, interconnect buses, shared memory, and other resources. Therefore, the area budget for the incremental accelerator core is limited. While supporting GEMM (fp16, fp32) for AI applications, it also supports GEMM (fp64) with existing HPC cores without changing the existing system ecosystem. This has become a key technical challenge that needs to be solved urgently. Summary of the Invention

[0006] The technical problem to be solved by the present invention is as follows: In response to the above-mentioned problems of the prior art, an incremental matrix multiplication accelerator for HPC / AI applications is provided. The present invention can save storage resources without affecting performance, meet the bandwidth requirements of the systolic array, and complete high-performance GEMM tasks simultaneously with the original HPC core without changing the ecology of the original HPC core and is compatible with the original matrix function library.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] An incremental matrix multiplication accelerator for HPC / AI applications, comprising an HPC core, a Maidong accelerator core MZ, a high-bandwidth memory HBM, a global shared memory GSM, and a data bus. The HPC core, the Maidong accelerator core MZ, the high-bandwidth memory HBM, and the global shared memory GSM are respectively connected to the data bus. The Maidong accelerator core MZ includes a systolic array SA, a B buffer, and a C buffer. The systolic array SA includes processing units PE arranged in a grid shape with multiple rows and columns. The B buffer is used to cache input data in the systolic array SA. The matrix B required for matrix multiplication is used, and the C buffer is used to cache the matrix C required for matrix multiplication in the input systolic array SA. The global shared memory GSM includes an A buffer for storing the matrix A required for matrix multiplication in the systolic array SA. When the HPC core is not working, the Maidong accelerator core MZ independently uses the matrix A required for matrix multiplication in the systolic array SA stored in the A buffer. When both the HPC core and the Maidong accelerator core MZ are working, the HPC core and the Maidong accelerator core MZ share the matrix A required for matrix multiplication in the systolic array SA stored in the A buffer.

[0009] Optionally, when both the HPC core and the Maidong accelerator core MZ are working, the computing mode executed by the Maidong accelerator core MZ is the matrix multiplication computing task GEMM in the HPC task; when the HPC core is not working, the computing mode executed by the Maidong accelerator core MZ is the convolution computing task CONV or the fully connected computing task FC in the AI ​​task.

[0010] Optionally, when the calculation mode executed by the Maidong accelerator core MZ is the matrix multiplication calculation task GEMM in the HPC task, the calculation mode executed by the Maidong accelerator core MZ is Y T =B T A T +C T , where A buffer is used to cache matrix A, and B buffer is used to cache the transposed matrix B of matrix B T , C buffer is used to buffer the transposed matrix C of matrix C T Or the final matrix Y is the transposed matrix Y TWhen the Maidong accelerator core MZ reads matrices from the A buffer, B buffer, and C buffer, it performs a transpose operation on the matrix A read from the A buffer to obtain the transposed matrix A of the matrix A. T .

[0011] Optionally, when the calculation mode executed by the Maidong accelerator core MZ is the matrix multiplication calculation task GEMM in the HPC task, the transposed matrix A of the matrix A is converted to T As a fixed matrix pre-loaded into each processing element PE of the systolic array SA, the transposed matrix B of the matrix B T , the transpose matrix C of matrix C T It is stored in the on-chip cache of the systolic array SA as a dynamic matrix to participate in the calculation.

[0012] Optionally, when the computing mode executed by the Maidong accelerator core MZ is the convolution computing task CONV in the AI ​​task, the computing method executed by the Maidong accelerator core MZ is Y=AB+C, where the A buffer is used to cache the matrix A, the B buffer is used to cache the matrix B, and the C buffer is used to buffer the matrix C or the final matrix Y, and the Maidong accelerator core MZ does not perform a transposition operation when reading matrices from the A buffer, the B buffer, and the C buffer.

[0013] Optionally, when the computing mode executed by the Maidong accelerator core MZ is the convolution computing task CONV in the AI ​​task, it includes pre-loading the matrix A as a fixed matrix into each processing unit PE of the systolic array SA, and storing the matrix B and the matrix C into the on-chip cache of the systolic array SA as dynamic matrices to participate in the calculation.

[0014] Optionally, when the computing mode executed by the Maidong accelerator core MZ is the fully connected computing task FC in the AI ​​task, the computing mode executed by the Maidong accelerator core MZ is Y=AB T +C, where the A buffer is used to cache matrix A, and the B buffer is used to cache matrix B or the transposed matrix B of matrix B. T The C buffer is used to buffer the matrix C or the final matrix Y, and the Maidong accelerator core MZ does not perform a transpose operation when reading matrices from the A buffer and the C buffer, and does not perform a transpose operation or performs a transpose operation when reading matrices from the B buffer.

[0015] Optionally, when the computing mode executed by the Maidong accelerator core MZ is the fully connected computing task FC in the AI ​​task, the matrix B or the transposed matrix B of the matrix B is converted to T The matrix A and the matrix C are pre-loaded into each processing unit PE of the systolic array SA as fixed matrices, and are stored in the on-chip cache of the systolic array SA as dynamic matrices to participate in the calculation.

[0016] Optionally, the matrix B or the transposed matrix B of the matrix BT When pre-loading the fixed matrix into each processing unit PE of the systolic array SA, if the matrix B is pre-loaded into each processing unit PE of the systolic array SA as a fixed matrix, the process of loading the fixed matrix includes: automatically dividing the fixed matrix into blocks according to the set size, if the size of the end block is insufficient, filling it and setting the mask position to 0 so that the data does not participate in the calculation, and then injecting each block into the systolic array SA from the right side according to the block order to achieve invisible transposition, thereby completing the loading of the fixed matrix; if the transposed matrix B of the matrix B is pre-loaded into each processing unit PE of the systolic array SA, the fixed matrix is ​​automatically divided into blocks according to the set size, if the size of the end block is insufficient, filling it and setting the mask position to 0 so that the data does not participate in the calculation, and then injecting each block into the systolic array SA from the right side according to the block order to achieve invisible transposition, thereby completing the loading of the fixed matrix; T The fixed matrix is ​​pre-loaded into each processing element (PE) of the systolic array SA. The process of loading the fixed matrix includes: automatically dividing the fixed matrix into blocks of a set size; if the end block size is insufficient, it is padded and the mask position is set to 0 to exclude the data from the calculation; then, the blocks are injected into the systolic array SA from the top side in the block order, completing the loading of the fixed matrix.

[0017] Optionally, the incremental matrix multiplication accelerator further includes a configuration bus, and the HPC core and the Maidong accelerator core MZ are respectively connected to the configuration bus.

[0018] Compared with the prior art, the present invention mainly has the following advantages:

[0019] The incremental matrix multiplication accelerator of the present invention includes an HPC core, a Maidong accelerator core MZ, a high-bandwidth memory HBM, a global shared memory GSM, and a data bus. The HPC core, the Maidong accelerator core MZ, the high-bandwidth memory HBM, and the global shared memory GSM are respectively connected to the data bus. The Maidong accelerator core MZ includes a systolic array SA, a B buffer, and a C buffer. The systolic array SA includes multiple rows and columns of processing units PE arranged in a grid shape. The B buffer is used to cache a matrix B required for matrix multiplication input into the systolic array SA. The C buffer is used to cache a matrix C required for matrix multiplication input into the systolic array SA. The global shared memory GSM includes an A buffer for storing a matrix A required for matrix multiplication in the systolic array SA. When the HPC core is not working, the Maidong accelerator core MZ alone uses the matrix A required for matrix multiplication in the systolic array SA stored in the A buffer. When both the HPC core and the Maidong accelerator core MZ are working, the HPC core and the Maidong accelerator core MZ share the matrix A required for matrix multiplication in the systolic array SA stored in the A buffer. The present invention can integrate HPC and AI acceleration, be embedded in the system environment of the HPC core, share high-bandwidth memory (HBM, High Bandwidth Memory) and global shared memory (GSM, Global Shared Memory) with the HPC core, and cooperate to complete matrix multiplication GEMM acceleration. At the same time, the MZ core (Maidong accelerator core) can also complete the acceleration of AI tasks. Taking advantage of the insensitivity of the systolic array to the fixed matrix bandwidth, a semi-storage structure is implemented in the Maidong accelerator core, which is different from the usual structure for the systolic array that requires three matrix buffers to store A, B, C / Y matrix data. The semi-storage structure is formed by using the GSM on the HPC core bus and the B buffer and C buffer on the Maidong accelerator core chip. The use of the semi-storage structure can effectively reduce the area overhead of on-chip storage and improve the density of the Maidong accelerator core computing unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 FIG. 4 is a structural diagram of a systolic array SA in the prior art.

[0021] Figure 2 This is a TPU structure diagram in the prior art.

[0022] Figure 3 This is the principle of the weight-fixed and input-fixed data flow mode involved in the embodiments of the present invention.

[0023] Figure 4 This is an example of the principle of systolic array calculation time in an embodiment of the present invention.

[0024] Figure 5 4 is a curve showing the impact of dynamic matrix bandwidth on performance in an embodiment of the present invention.

[0025] Figure 6This is a curve showing the impact of fixed matrix bandwidth on performance in an embodiment of the present invention.

[0026] Figure 7 Schematic diagram of the structure of the incremental matrix multiplication accelerator in an embodiment of the present invention.

[0027] Figure 8 Schematic diagram of the top injection fixed matrix method in an embodiment of the present invention.

[0028] Figure 9 Schematic diagram of a fixed matrix injection method on the right side in an embodiment of the present invention.

[0029] Figure 10 Schematic diagram of the working process of the Maidong accelerator core MZ in an embodiment of the present invention.

[0030] Figure 11 Schematic diagram of automatic block partitioning of a fixed matrix in an embodiment of the present invention.

[0031] Figure 12 Schematic diagram of automatic block partitioning of a dynamic matrix in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Before describing the incremental matrix multiplication accelerator for HPC / AI applications of the present invention, the storage access bandwidth requirements of the systolic array when executing GEMM are first analyzed. Figure 3 The data flow of the systolic array for both fixed-weight (WS) and fixed-input (IS) scenarios is shown. In both the IS and WS dataflow modes, three memory banks are required to provide data for the GEMM computation. The systolic array's computation results can be written back to the bias matrix / partial sum buffer. The matrices must be divided into blocks and injected into the systolic array based on the size of the systolic array (i.e., the number of processing units (PEs) in the systolic array). The fixed matrix only needs to be loaded once during a computation, while the other two matrices are dynamic and must be injected into the systolic array simultaneously. The bandwidth of each memory bank must match the size of the systolic array. In particular, the dynamic matrix must provide data to the systolic array in a timely manner every cycle to maintain high resource utilization. Modeling analysis is then performed: the systolic array is set to be r rows and c columns, and the IS dataflow mode is used. Each PE requires x cycles to compute the result. The computation of Y = AB + C in the IS dataflow mode is mapped to Y = WI + Bias, where W is the weight, I is the input, and Bias is the bias / partial sum. The size of matrix A is m rows and k columns, the size of matrix B is k rows and n columns, and the size of matrix C is m rows and n columns. α is the bandwidth utilization of matrix A, β is the bandwidth utilization of matrix B, and matrix C is required to be injected into the systolic array at the same time as matrix A. Figure 4To calculate Y=AB+C time, according to the bandwidth utilization, the calculation time is obtained as shown in formula (1). Formula (2) shows the ratio between the best performance and the test performance under the same calculation scale.

[0033]

[0034]

[0035] In the above formula, t(α,β) is the computing time, r is the number of rows in the systolic array, β is the bandwidth utilization of matrix B, k is the number of rows in matrix B, n is the number of columns in matrix B, x is the number of beats, c is the number of columns in the systolic array, m is the number of rows in matrix A, α is the bandwidth utilization of matrix A, Pref_ is the ratio between the best performance and the test performance, and t(1.0,1.0) is the computing time when both α and β are 1.0. Based on the theoretical model (1)(2), the impact of different bandwidths and bandwidth utilization on performance is analyzed. In the Maidong accelerator core, the computing core is a systolic array composed of 16 rows and 16 columns of processing units PE, so when analyzing the performance, we link the variables with the actual situation. When r=c=16,x=1,m=k=n=16*i(i=1,2,3…,10), α and β change from 10% to 100% respectively. Figure 5 and Figure 6 It shows the performance changes under different matrix sizes (different curves) and different bandwidth utilization (horizontal axis variables). Figure 5 In , as the size of matrix A increases, the impact of the bandwidth of matrix A on performance gradually increases and becomes proportional. Figure 6 In

[15] , as the size of matrix B increases, the impact of matrix B bandwidth on performance gradually decreases, eventually becoming almost negligible. Therefore, the GEMM performance of systolic arrays is extremely sensitive to the bandwidth of the dynamic matrix, but not to the bandwidth of the fixed matrix.

[0036] like Figure 7As shown, the incremental matrix multiplication accelerator for HPC / AI applications in this embodiment includes an HPC core, a Maidong accelerator core MZ, a high-bandwidth memory HBM, a global shared memory GSM, and a data bus. The HPC core, the Maidong accelerator core MZ, the high-bandwidth memory HBM (HighBandwidthMemory), and the global shared memory GSM (GlobalSharedMemory) are respectively connected to the data bus. The Maidong accelerator core MZ includes a systolic array SA, a B buffer, and a C buffer. The systolic array SA includes processing units arranged in a grid shape in multiple rows and columns. PE, the B buffer is used to cache the matrix B required for matrix multiplication in the input systolic array SA, the C buffer is used to cache the matrix C required for matrix multiplication in the input systolic array SA, the global shared memory GSM contains an A buffer for storing the matrix A required for matrix multiplication in the systolic array SA, when the HPC core is not working, the Maidong accelerator core MZ alone uses the matrix A required for matrix multiplication in the systolic array SA stored in the A buffer, when the HPC core and the Maidong accelerator core MZ are both working, the HPC core and the Maidong accelerator core MZ share the matrix A required for matrix multiplication in the systolic array SA stored in the A buffer. The GEMM in HPC tasks and AI tasks use storage differently. For the GEMM in AI tasks, the Maidong accelerator core independently completes the computing task, and there is no memory access conflict. For the GEMM in HPC tasks, the HPC core and the Maidong accelerator core share the matrix A data in the GSM. When the matrix A data is in the GSM, there is a memory access conflict between the HPC core and the Maidong accelerator core.

[0037] In this embodiment, when both the HPC core and the Maidong accelerator core MZ are working, the computing mode executed by the Maidong accelerator core MZ is the matrix multiplication computing task GEMM in the HPC task; when the HPC core is not working, the computing mode executed by the Maidong accelerator core MZ is the convolution computing task CONV or the fully connected computing task FC in the AI ​​task, as shown in Table 1.

[0038] Table 1: Computational mode table of the Maidong accelerator core MZ.

[0039]

[0040] Refer to Table 1. When the calculation mode of the Maidong accelerator core MZ is the matrix multiplication calculation task GEMM in the HPC task, the calculation mode of the Maidong accelerator core MZ is Y T =B T A T +C T , where A buffer is used to cache matrix A, and B buffer is used to cache the transposed matrix B of matrix B T , C buffer is used to buffer the transposed matrix C of matrix C TOr the final matrix Y is the transposed matrix Y T When the Maidong accelerator core MZ reads matrices from the A buffer, B buffer, and C buffer, it performs a transpose operation on the matrix A read from the A buffer to obtain the transposed matrix A of the matrix A. T In this embodiment, when the calculation mode executed by the Maidong accelerator core MZ is the matrix multiplication calculation task GEMM in the HPC task, the transposed matrix A of the matrix A is converted to T As a fixed matrix pre-loaded into each processing element PE of the systolic array SA, the transposed matrix B of the matrix B T , the transpose matrix C of matrix C T It is stored in the on-chip cache of the systolic array SA as a dynamic matrix to participate in the calculation. The GEMM in HPC tasks and AI tasks use different storage methods. For GEMM in AI tasks, the Maidong accelerator core completes the computing task independently, and there is no memory access conflict. For GEMM in HPC tasks, the HPC core and the Maidong accelerator core share the matrix A data in the GSM. When the matrix A data is in the GSM, there is a memory access conflict between the HPC core and the Maidong accelerator core. In order to compensate for the delay of accessing the global shared memory GSM through the bus, fixed matrix data can be stored in the global shared memory GSM, because the systolic array is not sensitive to the bandwidth of the fixed matrix, and the bandwidth of the global shared memory GSM is sufficient to ensure the throughput of the systolic array. In order to solve the two problems of memory access conflict and storing fixed matrices in the global shared memory GSM to solve the delay, a dual-mode loading scheme for fixed matrices is proposed to realize the invisible transpose of the matrix. Using the matrix calculation principle, Y=AB+C can be converted into Y T =B T A T +C T , A T As a fixed matrix of the pulsation array in the core of the Maidong accelerator, reducing its bandwidth requirement does not affect the ecological environment of the entire system. According to the matrix calculation principle, Y=AB+C can be converted to Y T =B T A T +C T , Y T 、B T 、A T 、C T Represent the transposed matrices of the Y, B, A, and C matrices respectively. By comparing the two calculation formulas, it can be seen that in the formula after transposition, the positions of matrices A and B are interchanged, so the positions where they are injected into the systolic array also change accordingly. If the original formula is calculated in the systolic array, according to the IS data flow format, matrix B is a fixed matrix and matrix A is a dynamic matrix. Matrix A has a high bandwidth requirement, but matrix A is stored in the global shared storage GSM and is shared with the HPC core, resulting in high memory access conflicts. After transposition, A TIt becomes a fixed matrix with high reuse rate and low bandwidth requirement. At this time, fetching A from the global shared memory GSM can reduce memory access conflicts. Since the global shared memory GSM is shared by the HPC core and the Maidong accelerator core MZ, it cannot destroy the original system's ecological environment. Therefore, the transposed matrix cannot be stored in the global shared memory GSM. After fetching the original matrix, it is necessary to perform the transposition during the transmission process. After analyzing the structural characteristics of the systolic array, it can be found that, Figure 8 , the first row of the pulsating array in the core Z of the Maidong accelerator is at the bottom. The fixed matrix is ​​injected from the first row downwards, and the first row is injected to the bottom of the pulsating array. The overall matrix is ​​inverted, but the left and right order of the matrix remains unchanged. The dynamic matrix is ​​injected from the left after the fixed matrix is ​​injected, and the calculation is performed. If the fixed matrix is ​​injected into the pulsating array from the right, the matrix is ​​injected column by column, such as Figure 9 As shown in Figure 1, the fixed matrix in the systolic array has a transposed function compared to the fixed matrix originally injected from above. Injecting data in this way achieves the implicit transposition achieved by the systolic array under the condition of storing in the original format, and avoids destroying the storage environment of the original system's global shared storage GSM. When executing GEMM in HPC tasks, the calculation scheme is Y T =B T A T +C T , the Maidong accelerator core uses implicit transposition to convert A T The matrix is ​​injected into the systolic array as a fixed matrix, and the bandwidth insensitivity of the fixed matrix is ​​used to reduce the memory access conflict, so that the HPC core and the Maidong accelerator core can complete the HPC task together. The HPC core and the Maidong accelerator core share the data of matrix A in the global shared memory GSM, and B T 、C T The matrix is ​​stored in the on-chip dedicated B buffer and C buffer in transposed form, and the generated matrix Y can be stored in the high-bandwidth memory HBM or the global shared memory GSM.

[0041] When executing AI tasks, the Maidong accelerator core does not need to cooperate with the HPC core, that is, no data is shared.

[0042] Referring to Table 1, when the computation mode executed by the Maidong accelerator core MZ is the convolution computation task CONV in an AI task, the computation method executed by the Maidong accelerator core MZ is Y=AB+C, where the A buffer is used to cache matrix A, the B buffer is used to cache matrix B, and the C buffer is used to cache matrix C or the resulting matrix Y. Furthermore, the Maidong accelerator core MZ does not perform a transpose operation when reading matrices from the A buffer, B buffer, and C buffer. When the Maidong accelerator core MZ is executed in the convolution computation task CONV in an AI task, the computation mode includes preloading matrix A as a fixed matrix into each processing element PE of the systolic array SA, and storing matrices B and C in the on-chip cache of the systolic array SA as dynamic matrices for computation. When executing a convolution computation task, for example, the im2col algorithm is used to convert a multi-channel feature map into a B matrix, and the convolution operation is reduced to a matrix multiplication computation GEMM. Matrix A is the convolution kernel stored in GSM. Since the HPC core does not perform convolution operations, there is no memory access conflict. The input feature map is stored in the on-chip dedicated B buffer, the deviation is stored in the on-chip dedicated C buffer, and the generated matrix Y can be stored in high-bandwidth memory HBM or global shared memory GSM.

[0043] Refer to Table 1. When the computing mode executed by the Maidong accelerator core MZ is the fully connected computing task FC in the AI ​​task, the computing mode executed by the Maidong accelerator core MZ is Y=AB T +C, where the A buffer is used to cache matrix A, and the B buffer is used to cache matrix B or the transposed matrix B of matrix B. T , C buffer is used to buffer matrix C or the final matrix Y, and the Maidong accelerator core MZ does not perform transposition operation when reading matrices from A buffer and C buffer, and does not perform transposition operation or performs transposition operation when reading matrices from B buffer. Referring to Table 1, when the computing mode executed by the Maidong accelerator core MZ is the fully connected computing task FC in the AI ​​task, it includes converting matrix B or the transposed matrix B of matrix B into T As fixed matrices, they are pre-loaded into the processing units (PEs) of the systolic array SA. Matrix A and Matrix C are stored in the on-chip cache of the systolic array SA as dynamic matrices for computation. When executing the fully connected computation task FC, batch operations are used to concatenate vectors into a matrix graph, and matrix-vector multiplication is converted into a matrix multiplication computation task GEMM. The computation scheme is Y = AB T +C. The input concatenation graph is stored in the on-chip B-buffer. Since the B-buffer is dedicated to the chip, the BT matrix can be stored directly in the on-chip buffer or transposed during data injection using invisible transposition. Matrix A is stored in the GSM for the concatenation graph. Because the HPC core does not perform fully connected operations, there is no memory access conflict. The deviation is stored in the on-chip dedicated C-buffer. The generated Y matrix can be stored in high-bandwidth memory (HBM) or global shared memory (GSM).

[0044] In this embodiment, the matrix B or the transposed matrix B of the matrix B T When pre-loading the fixed matrix into each processing unit PE of the systolic array SA, if the matrix B is pre-loaded into each processing unit PE of the systolic array SA as a fixed matrix, the process of loading the fixed matrix includes: automatically dividing the fixed matrix into blocks according to the set size, if the size of the end block is insufficient, filling it and setting the mask position to 0 so that the data does not participate in the calculation, and then injecting each block into the systolic array SA from the right side according to the block order to achieve invisible transposition, thereby completing the loading of the fixed matrix; if the transposed matrix B of the matrix B is pre-loaded into each processing unit PE of the systolic array SA, the fixed matrix is ​​automatically divided into blocks according to the set size, if the size of the end block is insufficient, filling it and setting the mask position to 0 so that the data does not participate in the calculation, and then injecting each block into the systolic array SA from the right side according to the block order to achieve invisible transposition, thereby completing the loading of the fixed matrix; T The fixed matrix is ​​pre-loaded into each processing element (PE) of the systolic array SA. The process of loading the fixed matrix includes: automatically dividing the fixed matrix into blocks of a set size; if the end block size is insufficient, it is padded and the mask position is set to 0 to exclude the data from the calculation; then, the blocks are injected into the systolic array SA from the top side in the block order, completing the loading of the fixed matrix.

[0045] like Figure 7 As shown, the incremental matrix multiplication accelerator of this embodiment further includes a configuration bus, and the HPC core and the Maidong accelerator core MZ are respectively connected to the configuration bus to facilitate relevant parameter configuration of the HPC core and the Maidong accelerator core MZ.

[0046] In this embodiment, injecting matrix data into the systolic array from different directions is called dual-mode loading. Combined with the semi-memory architecture, it not only allows the Maidong accelerator core to complete HPC tasks simultaneously with the HPC core, but also reduces the on-chip memory area. By sharing the GSM, one on-chip memory unit is eliminated from the Maidong accelerator core, thereby increasing the density of its on-chip computing units.

[0047] like Figure 10 As shown, the workflow of the Maidong accelerator core MZ in this embodiment includes:

[0048] S1. Determine the calculation mode. If the calculation mode is the matrix multiplication calculation task GEMM, proceed to step S2; if the calculation mode is the convolution operation calculation task CONV, proceed to step S3; if the calculation mode is the fully connected calculation task FC, proceed to step S4.

[0049] S2. Calculate the mapping scheme as Y T =B T A T +C T Since the global shared memory GSM is shared with the HPC core, the matrix A is directly stored in the global shared memory GSM as a fixed matrix, and the matrices B and C are transformed in the transposed form, that is, B T 、CT The matrix is ​​stored in the on-chip cache as a dynamic matrix, and the process proceeds to step S5.

[0050] S3. Calculate the mapping solution Y = AB + C. Since the Maidong accelerator core MZ independently performs the convolution operation, the matrix A is stored in the GSM as a fixed matrix, and the matrices B and C are stored in the on-chip cache as dynamic matrices. Go to step S6.

[0051] S4. Calculate the mapping scheme as Y=AB T +C, since the Maidong accelerator core MZ independently completes the full connection operation, the matrix B can be directly stored in the global shared memory GSM as a fixed matrix or the matrix B can be transposed into the matrix B T Store it in the global shared memory GSM as a fixed matrix, and store matrices A and C in the on-chip cache as dynamic matrices. If matrix B is directly stored in the global shared memory GSM, then go to step S5; if B T The matrix is ​​stored in the global shared memory GSM, and then the process goes to step S6.

[0052] S5. Since the matrix needs to be transposed, but is stored in the original format, an invisible transposition needs to be implemented. Figure 9 The fixed matrix is ​​automatically divided into blocks in the form of 16*16 blocks. If the end size does not meet 16, it is completed and the mask (MASK, used to limit whether the relevant bit is involved in the calculation) is set to 0, that is, the data is not involved in the calculation. In the order of blocks, the blocks are injected into the systolic array from the right, such as Figure 11 As shown, invisible transposition is achieved and the dynamic matrix injection is awaited, and the process proceeds to step S7.

[0053] S6. Since the matrix does not need to be transposed, it can be loaded directly according to the storage format. Figure 12 The fixed matrix is ​​automatically divided into blocks in the form of 16*16 blocks. If the end size does not meet 16, it is filled and the MASK position is set to 0, which means that the data does not participate in the calculation. In the order of blocks, the blocks are injected into the systolic array from the top, such as Figure 8 As shown, wait for dynamic matrix injection and go to step S7.

[0054] S7. Since the dynamic matrix has already been converted to a different format and stored in the on-chip memory regardless of whether it needs to be transposed or not, no additional processing is required. Figure 10 Automatic block division, the block size is 16*M, if the matrix row direction is less than 16, fill it up and set the mask (MASK, used to limit whether the relevant bit participates in the calculation) to 0, that is, the data does not participate in the calculation. According to the block order, such as Figure 8 As shown, the blocks are injected into the systolic array from the right side to calculate the result. Then, the process proceeds to step S8.

[0055] S8. If the matrix calculation is not finished, then add the part and C / C T Store it in the on-chip buffer and wait for the next injection as a dynamic matrix. If the matrix calculation is completed, the generated matrix Y / Y T Store in high-bandwidth memory HBM or global shared memory GSM.

[0056] To verify the semi-storage structure and dual-mode loading of fixed matrices, this embodiment performs an area synthesis on the Maidong accelerator core MZ to obtain the on-chip computing and storage resource overhead. By comparing this with the computing and storage resource overhead in the TPU, it is found that this method optimizes resource overhead without affecting performance. To evaluate this method, the Maidong accelerator core MZ is embedded into the MT-3000 system environment using a bus and evaluated. The Maidong accelerator core MZ shares HBM and GSM with the MT-3000. Without affecting the original system environment and performance, the area within the Maidong accelerator core MZ is synthesized. After obtaining specific results, the area percentage data is formed and compared with the TPU. The results are shown in Table 2.

[0057] Table 2: Area cost comparison.

[0058]

[0059]

[0060] As shown in Table 2, since the Maidong accelerator core MZ shares global shared memory (GSM), the on-chip storage for the systolic array is optimized. On-chip memory only accounts for 32.00% of the entire Maidong accelerator core MZ, compared to 46.03% for the TPU. This optimization also increases the density of on-chip computing resources. The systolic array SA, as a computing unit, occupies 58.03% of the Maidong accelerator core MZ area, compared to 47.63% for the TPU. This higher density enables more efficient computational tasks.

[0061] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0062] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An incremental matrix multiplication accelerator for HPC / AI applications, characterized by: The system includes an HPC core, a Maidong accelerator core MZ, a high-bandwidth memory HBM, a global shared memory GSM and a data bus. The HPC core, the Maidong accelerator core MZ, the high-bandwidth memory HBM and the global shared memory GSM are respectively connected to the data bus. The Maidong accelerator core MZ includes a systolic array SA, a B buffer and a C buffer. The systolic array SA includes a plurality of rows and columns of processing units PE arranged in a grid shape. The B buffer is used to cache the matrix B required for matrix multiplication of the input systolic array SA. The C buffer is used to cache the matrix C required for matrix multiplication of the input systolic array SA. The global shared memory GSM includes an A buffer for storing the matrix A required for matrix multiplication of the systolic array SA. When the HPC core is not working, the Maidong accelerator core MZ alone uses the matrix A required for matrix multiplication of the systolic array SA stored in the A buffer. When both the HPC core and the Maidong accelerator core MZ are working, the HPC core and the Maidong accelerator core MZ share the matrix A required for matrix multiplication of the systolic array SA stored in the A buffer.

2. The incremental matrix multiplication accelerator for HPC / AI applications according to claim 1, characterized in that: When both the HPC core and the Maidong accelerator core MZ are working, the computing mode executed by the Maidong accelerator core MZ is the matrix multiplication computing task GEMM in the HPC task; when the HPC core is not working, the computing mode executed by the Maidong accelerator core MZ is the convolution computing task CONV or the fully connected computing task FC in the AI ​​task.

3. The incremental matrix multiplication accelerator for HPC / AI applications according to claim 2, characterized in that: When the calculation mode of the Maidong accelerator core MZ is the matrix multiplication calculation task GEMM in the HPC task, the calculation mode of the Maidong accelerator core MZ is Y T =B T A T +C T , where A buffer is used to cache matrix A, and B buffer is used to cache the transposed matrix B of matrix B T , C buffer is used to buffer the transposed matrix C of matrix C T Or the final matrix Y is the transposed matrix Y T When the Maidong accelerator core MZ reads matrices from the A buffer, B buffer, and C buffer, it performs a transpose operation on the matrix A read from the A buffer to obtain the transposed matrix A of the matrix A. T .

4. The incremental matrix multiplication accelerator for HPC / AI applications according to claim 3, characterized in that: When the calculation mode executed by the Maidong accelerator core MZ is the matrix multiplication calculation task GEMM in the HPC task, the transposed matrix A of the matrix A is converted to T As a fixed matrix pre-loaded into each processing element PE of the systolic array SA, the transposed matrix B of the matrix B T , the transpose matrix C of matrix C T It is stored in the on-chip cache of the systolic array SA as a dynamic matrix to participate in the calculation.

5. The incremental matrix multiplication accelerator for HPC / AI applications according to claim 2, characterized in that: When the calculation mode executed by the Maidong accelerator core MZ is the convolution calculation task CONV in the AI ​​task, the calculation method executed by the Maidong accelerator core MZ is Y=AB+C, where the A buffer is used to cache the matrix A, the B buffer is used to cache the matrix B, and the C buffer is used to buffer the matrix C or the final matrix Y, and the Maidong accelerator core MZ does not perform a transpose operation when reading matrices from the A buffer, the B buffer, and the C buffer.

6. The incremental matrix multiplication accelerator for HPC / AI applications according to claim 5, characterized in that: When the computing mode executed by the Maidong accelerator core MZ is the convolution computing task CONV in the AI ​​task, it includes pre-loading the matrix A as a fixed matrix into each processing unit PE of the systolic array SA, and storing the matrix B and the matrix C into the on-chip cache of the systolic array SA as dynamic matrices to participate in the calculation.

7. The incremental matrix multiplication accelerator for HPC / AI applications according to claim 2, characterized in that: When the calculation mode executed by the Maidong accelerator core MZ is the fully connected computing task FC in the AI ​​task, the calculation method executed by the Maidong accelerator core MZ is Y=AB T +C, where the A buffer is used to cache matrix A, and the B buffer is used to cache matrix B or the transposed matrix B of matrix B. T The C buffer is used to buffer the matrix C or the final matrix Y, and the Maidong accelerator core MZ does not perform a transpose operation when reading matrices from the A buffer and the C buffer, and does not perform a transpose operation or performs a transpose operation when reading matrices from the B buffer.

8. The incremental matrix multiplication accelerator for HPC / AI applications according to claim 7, characterized in that: When the computing mode executed by the Maidong accelerator core MZ is the fully connected computing task FC in the AI ​​task, it includes converting the matrix B or the transposed matrix B of the matrix B T The matrix A and the matrix C are pre-loaded into each processing unit PE of the systolic array SA as fixed matrices, and are stored in the on-chip cache of the systolic array SA as dynamic matrices to participate in the calculation.

9. The incremental matrix multiplication accelerator for HPC / AI applications according to claim 8, characterized in that: The matrix B or the transposed matrix B of the matrix B T When pre-loading the fixed matrix into each processing unit PE of the systolic array SA, if the matrix B is pre-loaded into each processing unit PE of the systolic array SA as a fixed matrix, the process of loading the fixed matrix includes: automatically dividing the fixed matrix into blocks according to the set size, if the size of the end block is insufficient, filling it and setting the mask position to 0 so that the data does not participate in the calculation, and then injecting each block into the systolic array SA from the right side according to the block order to achieve invisible transposition, thereby completing the loading of the fixed matrix; if the transposed matrix B of the matrix B is pre-loaded into each processing unit PE of the systolic array SA, the fixed matrix is ​​automatically divided into blocks according to the set size, if the size of the end block is insufficient, filling it and setting the mask position to 0 so that the data does not participate in the calculation, and then injecting each block into the systolic array SA from the right side according to the block order to achieve invisible transposition, thereby completing the loading of the fixed matrix; T The fixed matrix is ​​pre-loaded into each processing element (PE) of the systolic array SA. The process of loading the fixed matrix includes: automatically dividing the fixed matrix into blocks of a set size; if the end block size is insufficient, it is padded and the mask position is set to 0 to exclude the data from the calculation; then, the blocks are injected into the systolic array SA from the top side in the block order, completing the loading of the fixed matrix.

10. The incremental matrix multiplication accelerator for HPC / AI applications according to claim 9, characterized in that: The incremental matrix multiplication accelerator further includes a configuration bus, and the HPC core and the Maidong accelerator core MZ are respectively connected to the configuration bus.

Images