In-memory calculation method and in-memory calculation system for general matrix multiplication

Abstract

The present application relates to an in-memory computing method and an in-memory computing system for general matrix multiplication, the method being applied to a tensor computing unit array, the method comprising: performing block processing on a left matrix and a right matrix to obtain a plurality of block left matrices and a plurality of block right matrices; based on a preset timing, distributing the plurality of block left matrices to each tensor computing unit in the first column of the array, and transmitting them to the last column in sequence; distributing the plurality of block right matrices to each tensor computing unit in the last row of the array, and transmitting them to the first row in sequence; performing multiplication and accumulation calculations corresponding to each tensor computing unit during the transmission process to obtain the corresponding block partial sums; obtaining the calculation result of the general matrix multiplication of the left matrix and the right matrix based on the partial sums corresponding to each tensor computing unit, thereby solving the problem of low computational efficiency of the existing in-memory computing architecture for large-scale matrix operations.

Description

Technical Field

[0001] The present application relates to the field of hardware computing technology, and in particular to an in-memory computing method and an in-memory computing system for general matrix multiplication. Background Art

[0002] With the rapid development of neural network algorithms, the amount of computation and data of neural network algorithms have increased significantly. In the process of neural network training and reasoning, tensor computation accounts for a large proportion of the time, and is a key primitive operator that needs to be considered for neural network acceleration. However, with the rapid change of algorithm types and the rapid growth of tensor scale, tensor accelerators face the problem of mismatch between memory bandwidth and computing power growth, which makes the data transfer time much longer than the calculation time when calculating large-scale tensors. The industry adopts an in-memory computing architecture to integrate storage units and computing units to reduce or even eliminate the data transfer overhead from storage to calculation. However, as the size of tensors to be calculated becomes larger and larger, the in-memory computing array combines storage and computing units and adopts a circuit topology structure with tight coupling of storage / computation. While expanding computing power, due to the loss of storage density, the hardware storage capacity that can be provided is limited, and all weight parameters cannot be fully stored. Therefore, weight updates are required for large-scale matrix operations. In traditional in-memory computing architectures, reading, writing and computing cannot be carried out in parallel. When the data of the memory array needs to be updated, the calculation needs to wait, which increases time and energy costs and reduces the overall efficiency of in-memory computing accelerators.

[0003] There is currently no effective solution to the problem of low computational efficiency of existing in-memory computing architectures for large-scale matrix operations. Summary of the invention

[0004] In this embodiment, a general in-memory computing method and an in-memory computing system for matrix multiplication are provided to solve the problem of low computing efficiency of existing in-memory computing architecture for large-scale matrix operations.

[0005] In a first aspect, in this embodiment, a general in-memory calculation method for matrix multiplication is provided, the method is applied to a tensor calculation unit array, each tensor calculation unit in the tensor calculation unit array is connected to adjacent tensor calculation units in the same row and column, and the method includes:

[0006] Performing block processing on a left matrix and a right matrix for performing general matrix multiplication to obtain a plurality of block left matrices and a plurality of block right matrices;

[0007] Based on a preset timing, the multiple block left matrices are allocated to each tensor calculation unit in the first column of the tensor calculation unit array, and are sequentially transmitted to the tensor calculation units corresponding to the last column; the multiple block right matrices are allocated to each tensor calculation unit in the last row of the tensor calculation unit array, and are sequentially transmitted to the tensor calculation units corresponding to the first row; during the transmission process, the multiplication and accumulation calculations corresponding to each tensor calculation unit are performed to obtain the corresponding block partial sums;

[0008] Based on the partial sums corresponding to the tensor computing units, a calculation result of a general matrix multiplication of the left matrix and the right matrix is ​​obtained.

[0009] In some of the embodiments, the working cycle of the tensor computing unit includes a writing phase and a reading phase, and the multiple block right matrices are distributed to each tensor computing unit in the last row of the tensor computing unit array, and are sequentially transmitted to the tensor computing units corresponding to the first row, and the multiplication and accumulation calculations corresponding to each tensor computing unit are performed during the transmission process, including:

[0010] In the writing stage, the block right matrix is ​​written into the tensor computing unit based on the storage address, and multiplication and accumulation calculation is performed with the block left matrix transmitted to the tensor computing unit;

[0011] In the readout stage, the multiplication-accumulation calculation is continued, and the block right matrix is ​​read out and transmitted to the upper adjacent tensor calculation unit based on the storage address, and the readout stage overlaps with the write stage of the upper adjacent tensor calculation unit.

[0012] In some embodiments, the tensor computation unit includes an in-memory computation macro, and writing the block right matrix into the tensor computation unit and performing a multiplication-accumulation calculation with the block left matrix transmitted to the tensor computation unit includes:

[0013] Determine whether the current write action is an initialization write of the tensor computing unit;

[0014] If yes, write the block right matrix into the in-memory calculation macro, and then perform multiplication and accumulation calculation;

[0015] If not, the multiplication-accumulation calculation is performed while writing the block right matrix into the in-memory calculation macro.

[0016] In some of the embodiments, after reading out the block right matrix based on the received read address and transmitting it to the upper adjacent tensor computing unit, the method further includes:

[0017] Based on the number of the block right matrices and the number of columns of the tensor computation unit array, determining whether the block right matrix needs to be updated;

[0018] When updating is required, the updated block right matrix is ​​obtained based on allocation or based on the transmission of the adjacent tensor computing unit below, and the steps of the writing phase and the reading phase are repeated until all block right matrices are calculated.

[0019] In some embodiments, the step of allocating the plurality of block left matrices to the tensor computing units in the first column of the tensor computing unit array and sequentially transmitting the plurality of block left matrices to the tensor computing units corresponding to the last column includes:

[0020] Rearranging the block left matrix and converting it into a corresponding bit stream;

[0021] Based on a preset transmission cycle, the bit stream is transmitted in sequence from each tensor computing unit in the first column of the tensor computing unit array to the tensor computing unit adjacent to the right, until it is transmitted to the tensor computing unit corresponding to the last column.

[0022] In some embodiments, the tensor computing unit includes an accumulation multiplexer, a portion and a buffer, and the multiplication and accumulation calculation corresponding to each tensor computing unit is performed during the transmission process to obtain the corresponding block portion and includes:

[0023] receiving the single-bit data corresponding to the block left matrix in sequence, and performing multiplication and accumulation calculations with the block right matrix stored in the tensor calculation unit to obtain the corresponding single-bit partial sum;

[0024] Perform shift accumulation calculation on multiple single-bit partial sums corresponding to each element in the block left matrix through the accumulation multiplexer to obtain the element partial sum;

[0025] The element partial sums corresponding to the elements of the block left matrix are accumulated and calculated in situ through the accumulation multiplexer to obtain the block partial sums.

[0026] In some embodiments, the element partial sums corresponding to the elements of the block left matrix are accumulated in situ by the accumulation multiplexer to obtain the block partial sums, including:

[0027] Reading the accumulated element partial sum from the partial sum buffer and performing floating point preprocessing;

[0028] Perform floating point preprocessing on the partial sum of the elements obtained by shift accumulation;

[0029] Performing in-situ accumulation calculation on the element partial sum after floating point preprocessing and the accumulated element partial sum, and performing floating point postprocessing to obtain an updated accumulated element partial sum;

[0030] Repeat the above steps until the element partial sums corresponding to all elements of the block left matrix are accumulated to obtain the block partial sums.

[0031] In a second aspect, an in-memory computing system is provided in this embodiment, the in-memory computing system comprising a tensor computing unit array, each tensor computing unit in the tensor computing unit array being connected to adjacent tensor computing units in the same row and column through a neighboring data transmission path;

[0032] The adjacent data transmission path is used to sequentially transmit the block left matrices of each tensor computing unit assigned to the first column of the tensor computing unit array to the tensor computing unit corresponding to the last column, and to sequentially transmit the block right matrices of each tensor computing unit assigned to the last row of the tensor computing unit array to the tensor computing unit corresponding to the first row;

[0033] The tensor calculation unit is used to perform the multiplication and accumulation calculations corresponding to each tensor calculation unit during the transmission process to obtain the corresponding block partial sum.

[0034] In some embodiments, the tensor computing unit includes an in-memory computing macro, a data flow control module, an accumulator multiplexer, and a partial sum buffer.

[0035] The in-memory calculation macro is used to perform multiplication and accumulation calculations of the block left matrix and the block right matrix, and complete corresponding data reading and writing;

[0036] The data flow central control module is used to control the adjacent data transmission path and the data processing timing of the in-memory computing macro;

[0037] The accumulation multiplexer is used to perform shift accumulation calculation and in-situ accumulation calculation;

[0038] The partial sum buffer is used to store partial sum data for performing in-situ accumulation calculations.

[0039] In some of the embodiments, the adjacent data transfer path includes a left matrix transfer channel and a right matrix transfer channel, and the right matrix transfer channel is constructed based on the read and write channels of the tensor computing unit.

[0040] Compared with the related art, the in-memory calculation method of the general matrix multiplication provided in this embodiment is applied to the tensor calculation unit array, and each tensor calculation unit in the tensor calculation unit array is connected with the adjacent tensor calculation units on the same row and column to form a neighboring data transmission path; by performing block processing on the left matrix and the right matrix for executing the general matrix multiplication, a plurality of block left matrices and a plurality of block right matrices are obtained, and a large-scale matrix is ​​converted into a plurality of matrices with smaller data scales for calculation, thereby reducing the demand for in-memory computing resources; based on a preset timing, each block left matrix is ​​transmitted from left to right in the tensor calculation unit array in sequence, and each block right matrix is ​​transmitted in the tensor calculation unit array in sequence. Data is transmitted sequentially from bottom to top, and high-speed data transmission is achieved by utilizing adjacent data transmission pathways, which reduces the time and energy overhead of data handling; each tensor computing unit in the array performs matrix data transmission and calculation in parallel in a pulsating manner to obtain the block partial sum; the calculation result of the general matrix multiplication is obtained based on the block partial sum, which solves the problem of low computational efficiency of large-scale matrix operations and hides the update time of weight data; at the same time, through the matrix expansion of the tensor computing unit, the timing synchronization problem of small-grained and large-number tensor computing units is effectively solved, and the timing control process of data transmission is simplified by utilizing the adjacent data transmission pathway and the read and write functions of the memory itself.

[0041] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 is a hardware structure block diagram of an in-memory computing system in some embodiments of the present application;

[0044] Figure 2 is a flow chart of an in-memory calculation method of a general matrix multiplication in some embodiments of the present application;

[0045] Figure 3 is a transmission flow chart of a block left matrix in some embodiments of the present application;

[0046] Figure 4 is a parallel execution flow chart of data update and calculation of a tensor computing unit in some embodiments of the present application;

[0047] Figure 5 is a schematic diagram of data transmission between a block left matrix and a block right matrix in some embodiments of the present application;

[0048] Figure 6is a timing diagram of writing and reading a tensor computing unit in some embodiments of the present application;

[0049] Figure 7 is a writing flowchart of a tensor computing unit in some embodiments of the present application;

[0050] Figure 8 is a data update flow chart of a tensor computing unit in some embodiments of the present application;

[0051] Fig. 9 This is a flowchart of performing multiplication and accumulation of bit stream and matrix data to obtain calculation results in some embodiments of the present application;

[0052] Fig.10 is a flow chart of in-situ accumulation calculation of some embodiments of the present application;

[0053] Fig.11 It is a flowchart of an in-memory calculation method of a general matrix multiplication according to some specific embodiments of the present application. DETAILED DESCRIPTION

[0054] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] The in-memory calculation method of general matrix multiplication provided in the embodiment of the present application can be executed in an in-memory computing system of a server, a computer or a similar computing device. Figure 1 1 is a hardware structure block diagram of the in-memory computing system in some embodiments of the present application. Figure 1 As shown, the in-memory computing system includes a tensor computing unit array 10, which includes a plurality of tensor computing units 101 arranged in a matrix. Each tensor computing unit 101 is connected to adjacent tensor computing units 101 in the same row and column.

[0057] Specifically, each tensor calculation unit 101 is connected to adjacent tensor calculation units 101 in the same row and column through adjacent data transmission paths. The calculation of the tensor calculation unit 101 can be performed in a digital manner or in an analog manner.

[0058] Further, the adjacent data transmission path may include a left matrix transmission channel and a right matrix transmission channel. The left matrix transmission channel is used to transmit data from left to right in the same row of the tensor computing unit array 10, and the right matrix transmission channel is used to transmit data from bottom to top in the same column of the tensor computing unit array 10. Among them, the right matrix transmission channel can be constructed based on the read and write channels of the tensor computing unit 101 itself.

[0059] Specifically, the adjacent data transmission path can be constructed by on-chip connection.

[0060] Specifically, Figure 1 As shown, the tensor computing unit may include an in-memory computing macro 13, a data flow control module 18, an accumulator multiplexer 15, and a portion and buffer 16.

[0061] Specifically, the in-memory computing macro 13 is the computing core of the tensor computing unit, which can be implemented by a storage-computation integrated design circuit. The memory of the in-memory computing macro 13 can be any one of static random access memory (SRAM), dynamic random access memory (DRAM), resistive random access memory, flash memory (NAND Flash or NOR Flash), and magnetic random access memory (MRAM).

[0062] Furthermore, if Figure 1 As shown, the tensor computing unit 101 may further include an input preprocessing module 17 and a multi-precision extension post-processing module 14 .

[0063] It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned in-memory computing system. Figure 1More or fewer tensor computing units 101 as shown in FIG. 1 , or tensor computing units 101 having the same Figure 1 Different configurations shown.

[0064] In this embodiment, a general in-memory calculation method for matrix multiplication is provided, which is applied to a tensor computing unit array. Figure 2 is a flowchart of an in-memory calculation method for general matrix multiplication in some embodiments of the present application, such as Figure 2 As shown, the process includes the following steps:

[0065] Step S201 , performing block processing on a left matrix and a right matrix for performing general matrix multiplication to obtain a plurality of block left matrices and a plurality of block right matrices.

[0066] Due to hardware resource limitations, it is not possible to read a large-scale matrix into memory for calculation at one time. The left matrix and the right matrix for performing general matrix multiplication (GEMM) can be processed in blocks to obtain multiple block left matrices and multiple block right matrices. The number of rows and columns of the block left matrix and the block right matrix can be determined based on the hardware configuration of the tensor computing unit.

[0067] Step S202, based on a preset timing, multiple block left matrices are allocated to each tensor calculation unit in the first column of the tensor calculation unit array, and are transmitted in sequence to the tensor calculation units corresponding to the last column; multiple block right matrices are allocated to each tensor calculation unit in the last row of the tensor calculation unit array, and are transmitted in sequence to the tensor calculation units corresponding to the first row; during the transmission process, the corresponding multiplication and accumulation calculations of each tensor calculation unit are performed to obtain the corresponding block partial sums.

[0068] Each tensor computing unit in the tensor computing unit array uses a pulsating method to transmit the data of the block left matrix and the block right matrix. That is, the data of the block left matrix and the block right matrix flow through each tensor computing unit in sequence in the horizontal and vertical directions in a pipeline manner; during the flow process, each tensor computing unit processes the data flowing through it in parallel.

[0069] The transmission cycle of the pulsating transmission can be preset. Before transmission, multiple block left matrices are first assigned to each tensor calculation unit in the first column of the tensor calculation unit array, and multiple block right matrices are assigned to each tensor calculation unit in the last row of the tensor calculation unit array. Then each block left matrix is ​​transmitted to the right through the corresponding left matrix transmission channel until it reaches the tensor calculation unit corresponding to the last column. Each block right matrix is ​​transmitted upward through the corresponding right matrix transmission channel until it reaches the tensor calculation unit corresponding to the first row.

[0070] Each tensor computing unit in the array receives a block left matrix and a block right matrix and performs multiplication and accumulation calculations to obtain the corresponding block partial sum.

[0071] Step S203, based on the partial sums corresponding to each tensor calculation unit, obtain the calculation result of the general matrix multiplication of the left matrix and the right matrix.

[0072] After each tensor computing unit calculates the corresponding block partial sum, it can read the block partial sum based on its own read and write channel, and perform corresponding accumulation to obtain the calculation result of the general matrix multiplication of the left matrix and the right matrix.

[0073] Through steps S201 to S203, by performing block processing on the left matrix and the right matrix for performing general matrix multiplication, a plurality of block left matrices and a plurality of block right matrices are obtained, and a large-scale matrix is ​​converted into a plurality of matrices with smaller data scales for calculation, thereby reducing the demand for in-memory computing resources; based on a preset timing, each block left matrix is ​​transmitted from left to right in the tensor computing unit array, and each block right matrix is ​​transmitted from bottom to top in the tensor computing unit array, and high-speed data transmission is achieved by using an adjacent data transmission path, thereby reducing the time and energy cost of data handling; each tensor computing unit in the array performs matrix data transmission and calculation in parallel in a pulsating manner to obtain a block partial sum; based on the block partial sum, the calculation result of the general matrix multiplication is obtained, thereby solving the problem of low computational efficiency of large-scale matrix operations and hiding the update time of weight data; at the same time, through the matrix expansion of the tensor computing unit, the timing synchronization problem of small-granularity and large-number tensor computing units is effectively solved, and the timing control process of data transmission is simplified by using the adjacent data transmission path and the read-write function of the memory itself.

[0074] In a further embodiment, Figure 3 is a transmission flow chart of the block left matrix in some embodiments of the present application, such as Figure 3 As shown, the process includes the following steps:

[0075] Step S301, rearrange the block left matrix and convert it into a corresponding bit stream.

[0076] Specifically, the block left matrix assigned to the first column of the tensor computing unit array can be subjected to vector preprocessing, including floating point mantissa alignment and data rearrangement, and then converted into a corresponding bit stream. For example, a 4×4 block left matrix can be first converted into 4 1×4 vectors, each of which is converted into 8-bit data, and the bit stream corresponding to the block left matrix is ​​4×32 bits.

[0077] Step S302, based on a preset transmission period, the bit stream is transmitted in sequence from each tensor computing unit in the first column of the tensor computing unit array to the tensor computing unit adjacent to the right, until it is transmitted to the tensor computing unit corresponding to the last column.

[0078] A transmission period for transmitting a bit stream from each tensor computing unit in the tensor computing unit array to the tensor computing unit adjacent to the right is preset, and based on the transmission period, each block left matrix is ​​sequentially transmitted from each tensor computing unit in the first column to the tensor computing unit adjacent to the right in units of bits, so that the tensor computing unit receiving the bit stream can calculate the single-bit data in correspondence with the data of the received block right matrix. The calculation process of the tensor computing unit and the transmission process of the bit stream can be carried out simultaneously.

[0079] Through steps S301~S302, the block left matrix is ​​rearranged and converted into a corresponding bit stream, thereby reducing the hardware resources required for transmission; based on a preset transmission cycle, the bit stream is transmitted in sequence from each tensor calculation unit in the first column of the tensor calculation unit array to the adjacent tensor calculation unit on the right until it is transmitted to the tensor calculation unit corresponding to the last column, thereby realizing the pulsating transmission and parallel calculation of the block left matrix through the control of the transmission timing.

[0080] In some embodiments, a working cycle of the tensor computing unit includes a write phase and a read phase. Figure 4 is a parallel execution flow chart of data update and calculation of the tensor computing unit in some embodiments of the present application, such as Figure 4 As shown, the process includes the following steps:

[0081] Step S401, in the writing stage, the block right matrix is ​​written into the tensor computing unit based on the storage address, and a multiplication and accumulation calculation is performed with the block left matrix transmitted to the tensor computing unit.

[0082] Specifically, the tensor calculation unit includes an in-memory calculation macro. Different from the above embodiment in which the block left matrix is ​​transmitted in bits, the block right matrix is ​​transmitted in the form of row vectors. Figure 5 This is a schematic diagram of data transmission of a block left matrix and a block right matrix in some embodiments of the present application. Figure 5 The in-memory computing macros of four tensor computing units in the tensor computing unit array are shown, namely, in-memory computing macro #1, in-memory computing macro #2, in-memory computing macro #3, and in-memory computing macro #4. Each in-memory computing macro includes a computing unit, a local read and hold unit, and multiple memory units, as well as an address decoder and a control unit.

[0083] Among them, the control unit is used to perform write control based on the write control signal, or to perform read control based on the read control signal. The calculation unit is used to perform the multiplication calculation of the bit stream and the block right matrix. When the block right matrix is ​​transmitted, the storage address Addr_0 or Addr_1 is first decoded by the address decoder, the memory cell of the corresponding row of the in-memory calculation macro is read, and then the read data is written to the corresponding storage address of the upper adjacent in-memory calculation macro. That is, through the read and write functions of the memory and in conjunction with the address data, the data transmission of the block right matrix can be completed. The transmission process can be carried out simultaneously with the calculation process of the tensor calculation unit.

[0084] It should be noted that the data sources of the in-memory computing macros of the tensor computing units at different positions in the tensor computing unit array are different. For example, for the in-memory computing macro of the first column of tensor computing units in the array, its horizontal input bit stream comes directly from the input vector preprocessing module, and the data of the input vector preprocessing module comes directly from the block left matrix obtained by blocking the left matrix. The input bit stream of the in-memory computing macro inside the array comes from the adjacent in-memory computing macro on the left. Similarly, for the in-memory computing macro of the tensor computing unit in the last row of the array, its vertical transmission data comes directly from the block right matrix obtained by blocking the right matrix, and the transmission data of the in-memory computing macro inside the array comes from the adjacent in-memory computing macro below.

[0085] After the bit stream corresponding to the block left matrix is ​​transmitted to the calculation unit of the in-memory calculation macro through the adjacent data transmission path, it is multiplied with the block right matrix stored in the memory unit to obtain the corresponding partial sum. Furthermore, multiple groups of low-precision partial sums can be expanded into high-precision partial sums through the multi-precision expansion post-processing module in the tensor calculation unit, and then sent to the accumulator multiplexer for shift accumulation or in-situ accumulation. The partial sums that cannot be accumulated in the current cycle will be sent to the partial sum buffer for storage, and when the accumulation round is reached, they will be read to the accumulator multiplexer for accumulation.

[0086] Step S402, in the readout phase, continue to perform multiplication and accumulation calculations, and read out the block right matrix based on the storage address and transmit it to the upper adjacent tensor calculation unit. The readout phase overlaps with the write phase of the upper adjacent tensor calculation unit.

[0087] Figure 6 This is a write and read timing diagram of the tensor computing unit in some embodiments of the present application. Figure 6The figure shows the overlapping relationship of the working cycles of the four tensor calculation units that perform data updates in sequence when the data of the block right matrix is ​​updated. A working cycle of the tensor calculation unit includes a writing phase t1 and a reading phase t2. In a writing phase t1, the tensor calculation unit completes the writing of the data of the corresponding block right matrix, and the data of the block right matrix can be read from the tensor calculation unit adjacent to the bottom. At the same time, the multiplication and accumulation calculation of the block right matrix and the corresponding bit stream is performed to obtain the corresponding partial sum. In a reading phase t2, the corresponding partial sum is input to the multi-precision extension post-processing module and the accumulation multiplexer of the tensor calculation unit for the next step of processing, and when data update is required, the block right matrix is ​​read out and transmitted to the tensor calculation unit adjacent to the top based on the received read address. In the same t2, the tensor calculation unit adjacent to the top also completes the writing of the block right matrix and starts to perform the multiplication and accumulation calculation of the block right matrix and the corresponding bit stream. If the current round of calculation does not require data update, it is not necessary to perform the reading of the block right matrix and the transmission action of the block right matrix.

[0088] Through steps S401~S402, in the write phase, the block right matrix is ​​written into the tensor calculation unit, and the multiplication and accumulation calculation is performed with the block left matrix transmitted to the tensor calculation unit. In the read phase, the multiplication and accumulation calculation is continued, and the block right matrix is ​​read out and transmitted to the upper adjacent tensor calculation unit based on the received read address. The read phase overlaps with the write phase of the upper adjacent tensor calculation unit, and the data update transmission of the block right matrix is ​​executed in parallel with the multiplication and accumulation calculation, so that the data update time is hidden in the calculation process, thereby shortening the time overhead of in-memory calculation.

[0089] In some embodiments, Figure 7 is a flowchart of writing a tensor computing unit in some embodiments of the present application, such as Figure 7 As shown, the process includes the following steps:

[0090] Step S701, determine whether the current write action is the initialization write of the tensor computing unit.

[0091] The writing of the block right matrix can be divided into two cases: initialization writing of the tensor computing unit (i.e., first writing) and non-initialization writing (i.e., data updating).

[0092] Step S702: If yes, write the block right matrix into the in-memory calculation macro, and then perform the multiplication and accumulation calculation.

[0093] When the current write action is initialization write, each row vector of the block right matrix is ​​first written into the corresponding memory unit in the in-memory calculation macro according to the address, and no calculation is performed in the current working cycle.

[0094] Step S703: If not, perform multiplication and accumulation calculations while writing the block right matrix into the in-memory calculation macro.

[0095] When the current write action is not an initialization write, the multiplication and accumulation calculation is performed while writing each row vector of the block right matrix into the corresponding memory unit in the in-memory calculation macro according to the address.

[0096] Through steps S701~S703, it is determined whether the current write action is the initialization write of the tensor calculation unit. If so, the block right matrix is ​​written into the in-memory calculation macro, and then the multiplication and accumulation calculation is performed; if not, the multiplication and accumulation calculation is performed while writing the block right matrix into the in-memory calculation macro, and the calculation timing of the block right matrix written into the tensor calculation unit for the first time is controlled to ensure the smooth calculation of the bit stream and the block right matrix; for the tensor calculation unit that performs data update, data update and calculation are performed in parallel, which shortens the time overhead of in-memory calculation.

[0097] In some embodiments, Figure 8 is a data update flow chart of the tensor computing unit in some embodiments of the present application, such as Figure 8 As shown, the process includes the following steps:

[0098] Step S801, based on the number of block right matrices and the number of columns of the tensor calculation unit array, determine whether the block right matrix needs to be updated.

[0099] When the matrix size of the general matrix multiplication calculation is large, the number of block right matrices may be large. When the number of block right matrices is greater than the number of columns of the tensor calculation unit array, the tensor calculation unit array cannot store all the block right matrices at one time, and the block right matrices need to be updated.

[0100] Step S802, when updating is required, based on allocation or based on the transmission of the adjacent tensor computing unit below, the updated block right matrix is ​​obtained, and the steps of the writing phase and the reading phase are repeated until all block right matrices are calculated.

[0101] In the case where the block right matrix needs to be updated, the updated multiple block right matrices are distributed to the tensor calculation units in the last row of the tensor calculation unit array. Each tensor calculation unit in the last row writes the updated block right matrix into the in-memory calculation macro during the write phase of the working cycle, and performs multiplication and accumulation calculations at the same time. In the read phase, the tensor calculation unit in the last row reads out the updated block right matrix based on the received read address and transmits it to the adjacent tensor calculation unit above while performing multiplication and accumulation calculations. By analogy, the adjacent tensor calculation unit above also completes the writing of the updated block right matrix at this stage, and begins to perform multiplication and accumulation calculations of the block right matrix and the corresponding bit stream.

[0102] Step S802 may be repeatedly executed until all the block right matrices are allocated and calculated.

[0103] Through steps S801~S802, it is determined whether the block right matrix needs to be updated based on the number of block right matrices and the number of columns of the tensor computing unit array. If it needs to be updated, the updated block right matrix is ​​obtained based on allocation or transmission based on the adjacent tensor computing unit below, and the steps of the write phase and the read phase are repeated until all block right matrices are calculated. In the case of a large matrix scale for performing general matrix multiplication calculations, a method for data update and calculation parallelism is given, which hides the time for data update in the calculation process and shortens the time overhead of in-memory calculations.

[0104] In some embodiments, Fig. 9 is a flowchart of performing multiplication and accumulation of bit stream and matrix data to obtain calculation results in some embodiments of the present application, such as Fig. 9 As shown, the process includes the following steps:

[0105] Step S901, receiving the single-bit data corresponding to the block left matrix in sequence, performing multiplication and accumulation calculations with the block right matrix stored in the tensor calculation unit, and obtaining the corresponding single-bit partial sum.

[0106] For the data with input precision IN_W in the block left matrix, each bit number is transmitted to the calculation unit of the tensor calculation unit array and multiplied with the block right matrix. Specifically, each bit number is multiplied with each row vector of the block right matrix, and the multiplication results corresponding to multiple row vectors are accumulated and output as single-bit parts and PSUM1.

[0107] Step S902 , performing shift accumulation calculation on multiple single-bit partial sums corresponding to each element in the block left matrix through an accumulation multiplexer to obtain an element partial sum.

[0108] Each element in the blocked left matrix can be converted into a set of data with precision IN_W. For a set of data with precision IN_W, a total of IN_W sets of single-bit partial sums are output, which can be shifted and accumulated through the accumulator multiplexer to obtain the element partial sum PSUM2 with input precision IN_W.

[0109] Step S903, performing in-situ accumulation calculation on the element partial sums corresponding to the elements of the block left matrix through an accumulation multiplexer to obtain the block partial sums.

[0110] The element partial sum PSUM2 corresponding to each element of the block left matrix needs to be accumulated in situ to obtain the block partial sum. The in-situ accumulation is after the shift accumulation, so the accumulation multiplexer can be time-division multiplexed. When the shift accumulation is performed, the shift is performed before inputting the accumulation multiplexer. After the shift accumulation is completed, the current element partial sum PSUM2 and the element partial sum PSUM2 of the previous round of shift accumulation are accumulated in situ.

[0111] Through steps S901 to S903, by time-division multiplexing of the accumulation multiplexer, it is possible to avoid using two sets of adder resources for shift accumulation and in-situ accumulation, thereby avoiding waste of adder area.

[0112] In some embodiments, Fig.10 is a flow chart of in-situ accumulation calculation in some embodiments of the present application, such as Fig.10 As shown, the process includes the following steps:

[0113] Step S1001, read the accumulated element partial sum from the partial sum buffer and perform floating point preprocessing.

[0114] The partial sum buffer is used to store the partial sums of elements that cannot be accumulated in the current cycle. Specifically, the partial sums of the elements corresponding to each element can be accumulated and stored in the partial sum buffer. When the accumulation multiplexer obtains the current element partial sum, the accumulated element partial sum is read from the partial sum buffer and accumulated with the current element partial sum, and then the result is sent to the partial sum buffer until the block partial sum is obtained. In-situ accumulation can be expanded by floating point post-processing and floating point pre-processing.

[0115] Step S1002, performing floating point preprocessing on the element partial sum obtained by shift accumulation.

[0116] Before performing the in-place accumulation calculation, the accumulated element partial sum and the current element partial sum are subjected to floating point preprocessing respectively. The floating point preprocessing can be performed in the accumulation multiplexer.

[0117] Step S1003, performing in-situ accumulation calculation on the element partial sum after floating point preprocessing and the accumulated element partial sum, and performing floating point postprocessing to obtain an updated accumulated element partial sum.

[0118] In-situ accumulation is performed in the accumulation multiplexer to obtain an accumulated value, and then floating-point post-processing is performed on the accumulated value to obtain an updated cumulative element partial sum.

[0119] Step S1004, repeat the above steps until the element partial sums corresponding to all elements of the block left matrix are accumulated to obtain the block partial sums.

[0120] Through steps S1001 to S1004, a floating-point adder is implemented through floating-point preprocessing and floating-point postprocessing, so that the calculated data accuracy can include floating point and fixed point, expanding the data application range of general matrix multiplication calculation.

[0121] The present embodiment is described and illustrated by means of specific examples below. The method is applied to Figure 1 The in-memory computing system shown. Fig.11 is a flow chart of an in-memory calculation method for general matrix multiplication in some specific embodiments of the present application, such as Fig.11 As shown, the process includes the following steps:

[0122] Step S1101, performing block processing on a left matrix and a right matrix for performing general matrix multiplication to obtain a plurality of block left matrices and a plurality of block right matrices;

[0123] Step S1102, allocating multiple block left matrices to each tensor computing unit in the first column of the tensor computing unit array;

[0124] Step S1103, allocating multiple block right matrices to each tensor computing unit in the last row of the tensor computing unit array;

[0125] Step S1104, the input vector preprocessing module in each tensor calculation unit of the first column performs input vector preprocessing on the multiple block left matrices and converts them into corresponding bit streams;

[0126] Step S1105, based on a preset transmission period, the tensor computing unit in the first column transmits the bit stream to the tensor computing unit adjacent to the right in sequence, and so on until the bit stream is transmitted to the tensor computing unit corresponding to the last column;

[0127] Step S1106, the tensor calculation unit of the first column inputs the bit stream into the in-memory calculation macro;

[0128] Step S1107, the tensor calculation unit determines whether the current block right matrix is ​​written for initialization of the tensor calculation unit;

[0129] Step S1108, if the judgment in step S1107 is yes, then write the block right matrix into the in-memory calculation macro of the tensor calculation unit, and then perform multiplication and accumulation calculation on the single-bit data corresponding to the block left matrix;

[0130] Step S1109, read out the block right matrix and transmit it to the tensor calculation unit adjacent to the upper side, and continue to calculate the corresponding single-bit partial sum, and then return to step S1107;

[0131] Step S1110, if the judgment of step S1107 is no, then while writing the block right matrix into the in-memory calculation macro, perform multiplication and accumulation calculation on the single-bit data corresponding to the block left matrix;

[0132] Step S1111, determine whether the current block right matrix needs to be updated; if the block right matrix does not need to be updated, repeat S1110 to S1111; if the block right matrix needs to be updated, execute step S1109, and then return to step S1107;

[0133] Step S1112, performing shift accumulation calculation on multiple single-bit partial sums corresponding to each element in the block left matrix through an accumulation multiplexer to obtain an element partial sum;

[0134] Step S1113, performing floating point preprocessing and floating point postprocessing on the accumulated element partial sums, and storing them in a partial sum buffer;

[0135] Step S1114, performing in-situ accumulation calculation on the element partial sums corresponding to each element of the block left matrix through an accumulation multiplexer to obtain the block partial sums;

[0136] Step S1115, based on the partial sums corresponding to each tensor computing unit, obtain the calculation result of the general matrix multiplication of the left matrix and the right matrix.

[0137] Through steps S1101 to S1115, the multiplication and accumulation calculations in the memory are performed through multiple in-memory calculation macros in the tensor calculation unit array, and the block left matrix and the block right matrix are transmitted using the adjacent data transmission path in the tensor calculation unit array to alleviate the bandwidth bottleneck. Using the in-memory calculation macro that can support parallel reading, writing and calculation, the adjacent data transmission path and the data flow central control module are used between the in-memory calculation macros to transmit the weight data (i.e., the block right matrix) between the in-memory calculation macros during calculation, so as to achieve the update of the weight data and the hiding of the update time, improve the overall processing efficiency, and accelerate the processing of large-scale matrix multiplication. At the same time, the granularity of the in-memory calculation macro is expanded, which effectively alleviates the timing synchronization problem of small granularity and large number, and simplifies the timing control of data transmission by using the inter-macro data path and the read and write function of the memory itself. In addition, the data transmission path is also constructed by the original write channel and read channel, which can achieve plug-in scale expansion. Through the time-sharing multiplexing of the shift accumulation and in-situ accumulation of the accumulation multiplexer, it is possible to avoid using two sets of adder resources for shift accumulation and in-situ accumulation, which causes area waste; at the same time, the floating-point adder is realized through the floating-point processing extension, saving area overhead.

[0138] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0139] In some embodiments, the present application also provides an in-memory computing system, which includes a tensor computing unit array, and each tensor computing unit in the tensor computing unit array is connected to the adjacent tensor computing units in the same row and column through a neighboring data transmission path.

[0140] The adjacent data transmission path is used to sequentially transmit the block left matrices of each tensor computing unit assigned to the first column of the tensor computing unit array to the tensor computing unit corresponding to the last column, and to sequentially transmit the block right matrices of each tensor computing unit assigned to the last row of the tensor computing unit array to the tensor computing unit corresponding to the first row;

[0141] The tensor computing unit is used to perform the multiplication and accumulation calculations corresponding to each tensor computing unit during the transmission process to obtain the corresponding block partial sum.

[0142] The in-memory computing system is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been explained will not be repeated here.

[0143] The in-memory computing system of this embodiment performs multiplication and accumulation calculations in the memory through multiple in-memory computing macros in the tensor computing unit array, and utilizes the adjacent data transmission path in the tensor computing unit array to transmit the block left matrix and the block right matrix, thereby alleviating the bandwidth bottleneck.

[0144] In some embodiments, the tensor computing unit includes an in-memory computing macro, a data flow control module, an accumulator multiplexer, and a partial sum buffer.

[0145] The in-memory calculation macro is used to perform multiplication and accumulation calculations of the block left matrix and the block right matrix, and complete the corresponding data reading and writing;

[0146] The data flow control module is used to control the adjacent data transmission path and the data processing timing of the in-memory computing macro;

[0147] The accumulation multiplexer is used to perform shift-accumulate calculation and in-place accumulation calculation;

[0148] The partial sum buffer is used to store partial sum data for performing in-situ accumulation calculations.

[0149] The in-memory computing system of this embodiment, through the in-memory computing macros that can support reading, writing and computing in parallel, utilizes the adjacent data transmission path and the data flow central control module between the in-memory computing macros of multiple tensor computing units, transfers weight data between the in-memory computing macros while calculating, realizes the updating of weight data and hiding of update time, improves the overall processing efficiency, and accelerates the processing of larger-scale matrix multiplication.

[0150] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0151] In some embodiments, the adjacent data transfer path includes a left matrix transfer channel and a right matrix transfer channel, and the right matrix transfer channel is constructed based on the read and write channels of the tensor computing unit.

[0152] The in-memory computing system of this embodiment constructs a tensor computing unit array through a left matrix transmission channel and a right matrix transmission channel to realize data transmission of a block left matrix and a block right matrix, and effectively alleviates the timing synchronization problem of a small granularity and a large number of tensor computing units through the arrangement of the tensor computing unit array, and simplifies the timing control of data transmission by using the adjacent data transmission path and the read and write functions of the memory itself. In addition, the right matrix transmission channel is also constructed by the read and write channels of the tensor computing unit, which can realize plug-in scale expansion.

[0153] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0154] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0155] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0156] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A general in-memory calculation method for matrix multiplication, characterized in that: The method is applied to a tensor computing unit array, each tensor computing unit in the tensor computing unit array is connected to adjacent tensor computing units in the same row and column, and the method includes: Performing block processing on a left matrix and a right matrix for performing general matrix multiplication to obtain a plurality of block left matrices and a plurality of block right matrices; Based on a preset timing, the multiple block left matrices are allocated to each tensor calculation unit in the first column of the tensor calculation unit array, and are transmitted to the tensor calculation units corresponding to the last column in sequence; the multiple block right matrices are allocated to each tensor calculation unit in the last row of the tensor calculation unit array, and are transmitted to the tensor calculation units corresponding to the first row in sequence; the data of the block left matrix and the block right matrix flow through each tensor calculation unit in sequence in the horizontal and vertical directions in a pipeline manner; during the transmission process, the multiplication and accumulation calculations corresponding to each tensor calculation unit are performed to obtain the corresponding block partial sums; Obtaining a calculation result of a general matrix multiplication of the left matrix and the right matrix based on the partial sums corresponding to each tensor computing unit; The working cycle of the tensor computing unit includes a writing phase and a reading phase, wherein the plurality of block right matrices are distributed to each tensor computing unit in the last row of the tensor computing unit array, and are sequentially transmitted to the tensor computing units corresponding to the first row, and the multiplication and accumulation calculations corresponding to each tensor computing unit are performed during the transmission process, including: In the writing stage, the block right matrix is ​​written into the tensor computing unit based on the storage address, and multiplication and accumulation calculation is performed with the block left matrix transmitted to the tensor computing unit; In the readout stage, the multiplication-accumulation calculation is continued, and the block right matrix is ​​read out and transmitted to the upper adjacent tensor calculation unit based on the storage address, and the readout stage overlaps with the write stage of the upper adjacent tensor calculation unit.

2. The method according to claim 1, characterized in that The tensor calculation unit includes an in-memory calculation macro, and the writing of the block right matrix into the tensor calculation unit and performing multiplication and accumulation calculation with the block left matrix transmitted to the tensor calculation unit includes: Determine whether the current write action is an initialization write of the tensor computing unit; If yes, write the block right matrix into the in-memory calculation macro, and then perform multiplication and accumulation calculation; If not, the multiplication-accumulation calculation is performed while writing the block right matrix into the in-memory calculation macro.

3. The method according to claim 1, characterized in that After reading out the block right matrix based on the received read address and transmitting it to the upper adjacent tensor computing unit, the method further includes: Based on the number of the block right matrices and the number of columns of the tensor computation unit array, determining whether the block right matrix needs to be updated; When updating is required, the updated block right matrix is ​​obtained based on allocation or based on the transmission of the adjacent tensor computing unit below, and the steps of the writing phase and the reading phase are repeated until all block right matrices are calculated.

4. The method according to claim 1, characterized in that: The step of allocating the plurality of block left matrices to the tensor computing units in the first column of the tensor computing unit array and sequentially transmitting the matrixes to the tensor computing units corresponding to the last column includes: Rearranging the block left matrix and converting it into a corresponding bit stream; Based on a preset transmission cycle, the bit stream is transmitted in sequence from each tensor computing unit in the first column of the tensor computing unit array to the tensor computing unit adjacent to the right, until it is transmitted to the tensor computing unit corresponding to the last column.

5. The method according to claim 1, characterized in that The tensor calculation unit includes an accumulation multiplexer, a part and a buffer, and the multiplication and accumulation calculation corresponding to each tensor calculation unit is performed during the transmission process to obtain the corresponding block part and includes: receiving the single-bit data corresponding to the block left matrix in sequence, and performing multiplication and accumulation calculations with the block right matrix stored in the tensor calculation unit to obtain the corresponding single-bit partial sum; Perform shift accumulation calculation on multiple single-bit partial sums corresponding to each element in the block left matrix through the accumulation multiplexer to obtain the element partial sum; The element partial sums corresponding to the elements of the block left matrix are accumulated and calculated in situ through the accumulation multiplexer to obtain the block partial sums.

6. The method according to claim 5, characterized in that The element partial sums corresponding to the elements of the block left matrix are accumulated and calculated in situ by the accumulation multiplexer to obtain the block partial sums, including: Reading the accumulated element partial sum from the partial sum buffer and performing floating point preprocessing; Perform floating point preprocessing on the partial sum of the elements obtained by shift accumulation; Performing in-situ accumulation calculation on the element partial sum after floating point preprocessing and the accumulated element partial sum, and performing floating point postprocessing to obtain an updated accumulated element partial sum; Repeat the above steps until the element partial sums corresponding to all elements of the block left matrix are accumulated to obtain the block partial sums.

7. An in-memory computing system, characterized in that: The in-memory computing system includes a tensor computing unit array, each tensor computing unit in the tensor computing unit array is connected to adjacent tensor computing units in the same row and column through a neighboring data transmission path; The adjacent data transmission path is used to sequentially transmit the block left matrices of each tensor computing unit assigned to the first column of the tensor computing unit array to the tensor computing unit corresponding to the last column, and to sequentially transmit the block right matrices of each tensor computing unit assigned to the last row of the tensor computing unit array to the tensor computing unit corresponding to the first row; The data of the block left matrix and the block right matrix flow through each tensor computing unit in sequence in a pipeline manner from the horizontal direction and the vertical direction respectively; The tensor calculation unit is used to perform the multiplication and accumulation calculations corresponding to each tensor calculation unit during the transmission process to obtain the corresponding block partial sum; The working cycle of the tensor computing unit includes a writing phase and a reading phase, wherein the plurality of block right matrices are distributed to each tensor computing unit in the last row of the tensor computing unit array, and are sequentially transmitted to the tensor computing units corresponding to the first row, and the multiplication and accumulation calculations corresponding to each tensor computing unit are performed during the transmission process, including: In the writing stage, the block right matrix is ​​written into the tensor computing unit based on the storage address, and multiplication and accumulation calculation is performed with the block left matrix transmitted to the tensor computing unit; In the readout stage, the multiplication-accumulation calculation is continued, and the block right matrix is ​​read out and transmitted to the upper adjacent tensor calculation unit based on the storage address, and the readout stage overlaps with the write stage of the upper adjacent tensor calculation unit.

8. The in-memory computing system according to claim 7, characterized in that: The tensor computing unit includes an in-memory computing macro, a data flow control module, an accumulator multiplexer and a partial buffer. The in-memory calculation macro is used to perform multiplication and accumulation calculations of the block left matrix and the block right matrix, and complete corresponding data reading and writing; The data flow central control module is used to control the adjacent data transmission path and the data processing timing of the in-memory computing macro; The accumulation multiplexer is used to perform shift accumulation calculation and in-situ accumulation calculation; The partial sum buffer is used to store partial sum data for performing in-situ accumulation calculations.

9. The in-memory computing system according to claim 7, characterized in that: The adjacent data transmission path includes a left matrix transmission channel and a right matrix transmission channel, and the right matrix transmission channel is constructed based on the read and write channels of the tensor calculation unit.

Images