In-memory calculation matrix multiplication acceleration system for fine-grained structured sparseness
By optimizing the in-memory computing array structure and combining it with a ping-pong pulsed input transmission chain and a data selection module, the problem of traditional in-memory computing arrays being unable to handle sparse computing is solved, achieving efficient N:M sparse support and improved computing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PEKING UNIV
- Filing Date
- 2026-01-07
- Publication Date
- 2026-04-17
AI Technical Summary
Traditional digital in-memory computing arrays struggle to efficiently handle sparse computations, especially N:M sparse neural network models, resulting in a fixed data flow that makes it difficult to optimize model accuracy and reduce the number of non-zero values.
Design an accelerated in-memory computation matrix multiplication system for fine-grained structured sparse computation. Employ an M-row N-column in-memory computation array group and an addition tree, combined with a ping-pong systolic input transmission chain (PPSIC) and a data selection module to optimize the in-memory computation array structure and support N:M sparse pruned neural network models.
It improves the flexibility of in-memory computing arrays, enabling efficient support for N:M fine-grained structured sparsity, reducing data transmission latency and improving computational efficiency.
Smart Images
Figure CN121880262A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of in-memory computing technology, and in particular relates to an in-memory computing matrix multiplication acceleration system for fine-grained structured sparse computing. Background Technology
[0002] N:M fine-grained structured sparsity refers to grouping M adjacent weights, where at least N weights must be 0. NVIDIA's Ampere architecture supports 2:4 sparsity, meaning that at least two of four consecutive weights must be 0. Numerous studies by NVIDIA and other researchers in academia and industry have shown that, across various models such as ResNet, VGG, VIT, and LLaMa, N:M sparsity can improve sparsity and reduce the number of non-zero values while maintaining high model accuracy. Digital in-memory computing arrays integrate computational logic within the storage array, thereby reducing the overhead of data transfer from storage modules to computation modules. However, traditional digital in-memory computing arrays often have relatively fixed data flows, making it difficult to efficiently handle sparse computations. Summary of the Invention
[0003] The purpose of this invention is to provide an accelerated in-memory matrix multiplication system for fine-grained structured sparse computation, in order to solve the problems existing in the prior art.
[0004] To achieve the above objectives, this invention provides an accelerated system for fine-grained structured sparse in-memory computation matrix multiplication, comprising an M-row N-column in-memory computation array group composed of multiple in-memory computation array groups and an addertree, wherein each row is connected to a corresponding input data line, all columns in the same row share the same input data line, and M and N are natural numbers, M≥1, N≥1;
[0005] The in-memory computing array includes:
[0006] The storage module mem is used to store the weights of the neural network model after N:M fine-grained structured pruning.
[0007] A computing unit, wherein the computing unit is used to perform computing operations;
[0008] The Ping Pong Pulse Input Transmission Chain (PPSIC) is used to transmit input data to the computing module cmp and to the next Ping Pong Pulse Input Transmission Chain (PPSIC).
[0009] The computational units of each in-memory computation array on the same column are connected to the same adder tree.
[0010] Optionally, the ping-pong pulse input transmission chain PPSIC includes two register transmission chains sharing the same input. The first and second register transmission chains are used to transmit input data to the computing module cmp or the ping-pong pulse input transmission chain PPSIC in the same row and next column according to the drive signal. When the first register transmission chain transmits data to the computing module cmp according to the drive signal, the second register transmission chain transmits input data to the ping-pong pulse input transmission chain PPSIC in the same row and next column according to the drive signal. When the input data transmitted by the first register transmission chain is calculated by the computing module cmp, the data transmission by the second register transmission chain is also completed. At this time, the drive signal is adjusted so that the first register transmission chain transmits input data to the ping-pong pulse input transmission chain PPSIC in the same row and next column according to the drive signal, and the second register transmission chain transmits data to the computing module cmp according to the drive signal.
[0011] Optionally, the processing procedure of the in-memory computing array group specifically includes:
[0012] The weights of the neural network model after N:M fine-grained structured pruning are stored in the in-memory computing array. During the storage process, non-zero values are stored in the in-memory computing array, and zero data is not stored.
[0013] The input data is acquired, and the data selection module selects the data that needs to be multiplied with non-zero weights.
[0014] The ping-pong pulsed input transmission chain PPSIC is used to transmit input data to each column. After multiple clock cycles of transmission, when the input data of all columns has been loaded, the calculation begins. The calculation result is input into the adder tree and accumulated column by column. The accumulated data is then sent out to the in-memory computing array for shifting and accumulation of data in different clock cycles.
[0015] Optionally, the data selection module supports multiple selection modes, such as 2:4 and 1:3. The basic structure is a 3-to-1 selector. In 1:3 mode, each 3-to-1 selector can directly select one data point from three input data points based on the index. In 2:4 mode, the first three data points and the last three data points are respectively fed into two 3-to-1 selectors, thus selecting two valid data points from four original input data points.
[0016] Based on N:M sparsity, in each column of the weight matrix, N consecutive weights are 0. When deploying matrix multiplication on the in-memory computing array, one column of the weight matrix is deployed to one column of the in-memory computing array, but 0 is not stored.
[0017] Because zeros are not stored, the weights of rows that were originally different become the same row after the data is stored in the in-memory computing array. However, the weight data of rows that were originally different need to be multiplied by the input data of their respective columns. In other words, in the current in-memory computing array, each column requires different input data, unlike traditional in-memory computing arrays where all columns can share input. Therefore, PPSIC is needed to transmit input data for each column. Once the input data for all columns is loaded, computation can begin. At this point, the transmission chain of PPSIC that has completed data transmission can provide input data to the computing module, and another transmission chain can continue transmitting data. The subsequent data transmission delay can then be masked by the computation delay.
[0018] The technical effects of this invention are as follows:
[0019] This invention optimizes the internal structure of the digital in-memory computing array, improves the flexibility of the in-memory computing array, and can efficiently support N:M fine-grained structured sparsity. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0022] Figure 1 This is a schematic diagram of a traditional in-memory computing array architecture;
[0023] Figure 2 This is a schematic diagram of the in-memory computing array structure in an embodiment of the present invention;
[0024] Figure 3 This is a structural diagram of PPSIC in an embodiment of the present invention;
[0025] Figure 4 This is a schematic diagram of the reconfigurable input data selection module in an embodiment of the present invention.
[0026] Figure 5 This is a schematic diagram illustrating the storage of weights after N:M structured pruning in an embodiment of the present invention.
[0027] Figure 6 This is a schematic diagram of the transmission delay of PPSIC in an embodiment of the present invention. Detailed Implementation
[0028] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.
[0029] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.
[0030] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.
[0031] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.
[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0033] like Figure 1 - Figure 6 As shown, this embodiment provides an accelerated system for fine-grained structured sparse in-memory computation matrix multiplication, including an in-memory computation array group of M rows and N columns composed of multiple in-memory computation arrays and an addertree, wherein each row is connected to a corresponding input data line, all columns in the same row share the same input data line, M and N are natural numbers, M≥1, N≥1;
[0034] The in-memory computing array includes:
[0035] The storage module mem is used to store the weights of the neural network model after N:M fine-grained structured pruning.
[0036] A computing unit, wherein the computing unit is used to perform computing operations;
[0037] The Ping Pong Pulse Input Transmission Chain (PPSIC) is used to transmit input data to the computing module cmp and to the next Ping Pong Pulse Input Transmission Chain (PPSIC).
[0038] The computational units of each in-memory computation array on the same column are connected to the same adder tree.
[0039] This embodiment provides a sparse matrix multiplication acceleration core based on in-memory computation, including a novel in-memory computation and preprocessing module. By optimizing the internal structure of the digital in-memory computation array, the flexibility of the in-memory computation array is greatly improved, enabling efficient support for N:M fine-grained structured sparseness.
[0040] like Figure 1 As shown, a traditional in-memory computation array mainly consists of a storage module (mem), a computation module (cmp), and an adder tree. For matrix multiplication, one matrix is stored in the mem, while the other serves as input data from outside the in-memory computation array. For neural network models, the weight data is typically stored in the mem. Each row of the in-memory computation array has an input data line, and all columns in that row share the same input line. After computation, the results of each column are accumulated through the adjacent adder tree.
[0041] The schematic diagram of the in-memory computing array proposed in this embodiment is as follows: Figure 2 As shown, a ping-pong systolic input chain (PPSIC) has been added.
[0042] The basic computational process of the entire array is as follows: input data is first input to the PPSIC. After several clock cycles of transmission, each column of the in-memory computation array's PPSIC has received the input data. Then, the data in the memory (mem) and the data in the PPSIC are sent to the computation module cmp to complete the computation. Typically, the cmp module completes the computation of 1 bit of input data and n bits of weight data (n is the width of mem) within one clock cycle. Assuming n=8, to calculate int8 multiplied by int8, a complete int8 multiplication by int8 requires 8 clock cycles. In each clock cycle, the cmp module calculates 1 bit of input data. The computation result of each clock cycle is input into the adder tree and accumulated column by column. The accumulated data is then sent out of the in-memory computation array, where external circuitry performs shifting and accumulation of data for different clock cycles. For example, an in-memory computing module with M rows and N columns, each column mem having a bit width of 8, can perform the multiplication of an M-fold int8 input vector and an M-fold and N-fold int8 weight matrix every 8 clock cycles, where M and N are natural numbers, M≥1, N≥1.
[0043] A structural diagram of a PPSIC module is shown below. Figure 3As shown, PPSIC has two register transfer chains with DIN as the common input. The data passes through a 1-to-2 DEMUX and is distributed to one of the two registers. The specific register to which it is distributed is selected by the SEL1 signal.
[0044] PPSIC has two output ports. One input port is IN, which is the input to the computation module cmp of the in-memory compute array. The other output port is DOUT, which transmits data to the next column of the in-memory compute array. The data in the two registers of PPSIC is selected and allocated through SEL2. One channel of data is transmitted to IN as the input for matrix calculation; the other channel is transmitted to DOUT, which is then transmitted to the next column of PPSIC.
[0045] Taking a row of a CIM array as an example, there are two transmission chains. One chain provides input to the calculation module while the other chain transmits data normally. When the data in one chain has been calculated, the data transmission in the other chain will also be completed. At this point, SEL1 and SEL2 are adjusted, and the roles of the two chains are reversed. The transmission path that has completed the calculation will then transmit data, while the path that has completed the data transmission will perform the calculation.
[0046] Input data selection module (preprocessing module):
[0047] In this embodiment, when processing the neural network model that has undergone N:M fine-grained structured pruning, the pruned weights are stored in the in-memory computing array. The input data is preprocessed by the input data selection module and then sent to the in-memory computing array as input data.
[0048] The weight data is pruned so that in each column of M consecutive weight data, only a maximum of N non-zero values are retained, and all other weight data are set to 0. The significance of setting them to 0 is that 0 does not affect the calculation, so ideally these 0 data can be left unstored and not participate in the calculation, thereby reducing the model's storage, bandwidth, and computing power requirements. In this embodiment, after the weight matrix is pruned, the non-zero values are stored in the in-memory computation array, and the 0 data is not stored.
[0049] However, this leads to two problems: 1. Weight data in the same row of the in-memory computation array needs to be multiplied by different input data. 2. Input data needs to be selected, and only the input data that needs to participate in the computation is sent to the in-memory computation array.
[0050] The first point is Figure 4For example, according to the definition of matrix multiplication, elements in matrix A are multiplied by elements of the same color in matrix W. For instance, green elements in matrix A are multiplied by green elements in matrix W. However, after pruning and sparsifying the weight matrix, if zero elements are not stored, the weight data stored in the in-memory computation array will no longer have the same color in the same row. This means that the weight data in this row needs to be multiplied by different input data. The novel in-memory computation array given earlier is designed to meet the requirement that different columns need different input data.
[0051] Secondly, after removing the zero weights, the input data that would otherwise be multiplied by these weights no longer needs to be sent to the in-memory computing array. Therefore, it is necessary to preprocess the input data to filter out the data in the original input that need to be multiplied by non-zero weights.
[0052] We designed a reconfigurable input data selection module. This module supports multiple selection modes, such as 2:4 and 1:3. The basic structure is a 3-to-1 selector. In 1:3 mode, each 3-to-1 selector can directly select one data point from three input data points based on an index. In 2:4 mode, the first three data points and the last three data points are fed into two 3-to-1 selectors respectively, thus selecting two valid data points from four original input data points.
[0053] The workflow of this module is as follows:
[0054] Based on N:M sparsity, in each column of the weight matrix, N consecutive weights are 0. When deploying matrix multiplication on the in-memory computing array, one column of the weight matrix is deployed to one column of the in-memory computing array, but 0 is not stored.
[0055] like Figure 5 As shown, because 0 is not stored, the weights of previously different rows become the same row after the data is stored in the in-memory compute array. However, the weight data of the previously different rows needs to be multiplied by the input data of their respective columns. In other words, in the current in-memory compute array, each column requires different input data, unlike traditional in-memory compute arrays where all columns can share the same input.
[0056] Therefore, PPSIC is needed to transmit input data for each column. Once the input data for all columns is loaded, calculation can begin. At this point, the PPSIC transmission chain, which has already completed data transmission, can provide input data to the calculation module, and another transmission chain can continue transmitting data. The subsequent data transmission delay can then be masked by the calculation delay, such as... Figure 6 As shown, the transmission delay of PPSIC can be masked by the calculation after the calculation begins.
[0057] In summary, this embodiment provides an in-memory computing array that natively supports N:M sparse computing. By optimizing the internal structure of the digital in-memory computing array, the flexibility of the in-memory computing array is greatly improved. The latency of transmitting input data can be masked in the computation latency, and it can efficiently support N:M fine-grained structured sparse computing.
[0058] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A system for accelerating in-memory matrix multiplication in fine-grained structured sparse computation, characterized in that, It includes an M-row N-column in-memory computing array group consisting of multiple in-memory computing arrays and an adder tree, wherein each row is connected to a corresponding input data line, all columns in the same row share the same input data line, M and N are natural numbers, M≥1, N≥1; The in-memory computing array includes: The storage module mem is used to store the weights of the neural network model after N:M fine-grained structured pruning. A computing unit, wherein the computing unit is used to perform computing operations; The Ping Pong Pulse Input Transmission Chain (PPSIC) is used to transmit input data to the computing module cmp and to the next Ping Pong Pulse Input Transmission Chain (PPSIC). The computational units of each in-memory computation array on the same column are connected to the same adder tree.
2. The system according to claim 1, characterized in that, The ping-pong pulse input transmission chain (PPSIC) includes two register transmission chains sharing the same input. The first and second register transmission chains transmit input data to the computing module cmp or the ping-pong pulse input transmission chain PPSIC in the same row and next column according to a drive signal. When the first register transmission chain transmits data to the computing module cmp according to the drive signal, the second register transmission chain transmits input data to the ping-pong pulse input transmission chain PPSIC in the same row and next column according to the drive signal. When the input data transmitted by the first register transmission chain is calculated by the computing module cmp, the data transmission by the second register transmission chain is also complete. At this time, the drive signal is adjusted so that the first register transmission chain transmits input data to the ping-pong pulse input transmission chain PPSIC in the same row and next column according to the drive signal, and the second register transmission chain transmits data to the computing module cmp according to the drive signal.
3. The system according to claim 1, characterized in that, The processing procedure of the in-memory computing array group specifically includes: The weights of the neural network model after N:M fine-grained structured pruning are stored in the in-memory computing array. During the storage process, non-zero values are stored in the in-memory computing array, and zero data is not stored. The input data is acquired, and the input data is selected through the data selection module to select valid data. The valid data is the data in the input data that needs to be multiplied with non-zero weights. The ping-pong pulsed input transmission chain PPSIC is used to transmit input data to each column. After multiple clock cycles of transmission, when the input data of all columns has been loaded, the calculation begins. The calculation result is input into the adder tree and accumulated column by column. The accumulated data is then sent out to the in-memory computing array for shifting and accumulation of data in different clock cycles.
4. The system according to claim 3, characterized in that, The data selection module includes a 3-to-1 selector, supporting a first selection mode and a second selection mode; First selection mode: Use a 3-to-1 selector to filter out 1 valid data from 3 input data based on the index; Second selection mode: The first 3 data and the last 3 data are respectively fed into two 3-to-1 selectors to select 2 valid data from the 4 original input data.