A bit-level dynamic sparse in-memory computing architecture

Abstract

The application discloses a bit-level dynamic sparse in-memory computing architecture, relates to in-memory computing technology, and is proposed in view of the low circuit activity problem in the prior art. The application is composed of a software layer and a hardware layer. The software layer is arranged on a server end and is used for weight quantization, pruning and mapping optimization of a neural network model. The hardware layer is arranged on an edge end and is used for in-memory computing. The application has the advantages that the neural network weight bit sparsity can be deeply mined and utilized, the physical layout of the weight in an array is optimized, and conditions are created for fine power consumption control. The bit-level dynamic power consumption control mechanism can perform fine-grained switch control on a computing circuit according to a predicted bit sparsity mode, so that invalid power consumption is significantly eliminated, and better energy efficiency and performance are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to in-memory computing technology, and in particular to a bit-level dynamic sparse in-memory computing architecture. BACKGROUND

[0002] Deep neural networks (DNNs) have achieved great success in many fields, but their huge computational load and storage demand pose a serious challenge to traditional computing architectures such as CPUs and GPUs. When executing DNN computations, the frequent data transfer of processors based on the von Neumann architecture results in a significant "memory wall" problem, i.e., most of the energy and time is consumed in data access rather than actual computation.

[0003] To overcome the "memory wall", an in-memory computing architecture has emerged. This architecture embeds computing functions into memory units, enabling parallel computation at the data storage location, thereby greatly reducing data transfer overhead. Among various in-memory computing implementation schemes, SRAM-based in-memory computing technology has attracted much attention due to its high speed, high durability, and perfect compatibility with standard CMOS processes. A typical SRAM in-memory computing macrocell stores weight values in an SRAM cell array and uses word line input activation signals to complete the multiplication and addition operation by summing analog current or voltage on the bit line.

[0004] On the other hand, researchers have found that DNN models generally exhibit sparsity. To take advantage of sparsity to reduce computational load, existing technologies mainly focus on element-level sparsity, i.e., identifying and skipping entire zero-valued weights or activations. For example, pruning techniques are used to set unimportant weights to zero and skip these zero-value operations in computation.

[0005] However, for SRAM-based in-memory computing architectures, the above existing technologies still have the following outstanding defects and limitations in achieving high-performance computing:

[0006] 1. Neglect and underutilization of "bit-level sparsity":

[0007] Most existing optimization schemes stop at element-level sparsity and fail to exploit and utilize deeper bit-level sparsity. When a weight is stored in the form of multi-bit data (such as int8) in an SRAM array (each bit is stored by a different SRAM cell), its binary representation usually contains a large number of '0' bits.

[0008] In SRAM computing, the access of each bit (whether '0' or '1') will trigger the corresponding circuit action (such as bit line charging and discharging). A large number of SRAM cells storing '0' bits will still be activated during the calculation process, generating dynamic power consumption, although their numerical contribution to the final calculation result is zero or fixed. This invalid power consumption generated by '0' bits cannot be effectively eliminated in existing architectures, resulting in a huge waste of energy.

[0009] 2. Mismatch between weight mapping strategy and hardware characteristics:

[0010] When loading the trained network weights into the SRAM computing array, existing technologies usually use simple and fixed mapping strategies (such as sequential mapping or random mapping). This mapping method does not take into account the bit distribution characteristics of weight data at all.

[0011] Because the mapping is blind, it causes the weight bits with high '0' bit probability to be randomly scattered in various physical locations of the array. This disorder makes it impossible for the hardware to predict and manage the areas with high probability of '0' in a centralized manner, thereby losing the opportunity for large-scale, coordinated power optimization and limiting the further exploration of energy efficiency potential.

[0012] 3. "Computing efficiency imbalance" problem caused by uneven bit distribution:

[0013] In the SRAM computing array, due to the random bit pattern ('0' and '1' distribution) of weight values and the lack of optimization in traditional mapping strategies, there is a serious imbalance in the "effective computing efficiency" of SRAM cells in different physical locations during actual operation. Specifically, some columns (or rows) of SRAM cells may frequently store and participate in the calculation of '1' bits, causing their corresponding bit line (or word line) circuits to be in a high activity state for a long time, resulting in high dynamic power consumption and becoming the "bottleneck" of energy efficiency and the "hot spot" of reliability.

[0014] On the contrary, those columns (or rows) that happen to store a large number of '0' bits have very low circuit activity, and their computing resources are idle.

[0015] In summary, existing SRAM-based in-memory computing technology is fundamentally limited in energy efficiency, reliability, and performance due to its failure to effectively utilize bit-level sparsity, lack of intelligent weight mapping methods that match it, and serious computing efficiency imbalance. SUMMARY

[0016] The present application aims to provide a bit-level dynamic sparse in-memory computing architecture to solve the above-mentioned problems of existing technologies.

[0017] The bit-level dynamic sparse in-memory computing architecture in the application is composed of a software layer and a hardware layer; the software layer is deployed on a server end and is used for weight quantization, pruning and mapping optimization of a neural network model; and the hardware layer is deployed on an edge end and is used for in-memory computing;

[0018] The hardware layer comprises an in-memory computing core, a systolic array and a peripheral circuit; basic weights are mapped to the in-memory computing macro for multiplication and addition calculation, and repeated weights are calculated by the systolic array, and the in-memory computing macro and the systolic array cooperatively complete a hybrid inference task;

[0019] The software layer is provided with a quantization pruning screening algorithm, and the weight data after training is divided into two types of basic weights and repeated weights; the basic weights are used for retaining main effective numerical components; and the repeated weights are used for recording reusable parts.

[0020] The bit-level dynamic sparse in-memory computing architecture in the application has the advantages that the neural network weight bit sparsity can be deeply mined and utilized, the physical layout of the weights in the array is optimized, and conditions are created for fine power consumption control. The bit-level dynamic power consumption control mechanism can perform fine-grained switch control on the computing circuit according to the predicted bit sparse mode, thereby significantly eliminating invalid power consumption and achieving better energy efficiency and performance. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a structural schematic diagram of the in-memory computing architecture in the application.

[0022] Figure 2 is a data processing flow schematic diagram of the in-memory computing architecture in the application.

[0023] Figure 3 is an algorithm flow schematic diagram of the software layer of the in-memory computing architecture in the application.

[0024] Figure 4 is a structural schematic diagram of the in-memory computing macro in the application.

[0025] Figure 5 is a structural schematic diagram of the 8T-SRAM unit in the application.

[0026] Figure 6 is a computing flow schematic diagram of the hardware layer of the in-memory computing architecture in the application.

[0027] REFERENCE NUMERALS:

[0028] NMOS0-First NMOS tube, NMOS1-Second NMOS tube, NMOS2-Third NMOS tube, NMOS3-Fourth NMOS tube, NMOS4-Fifth NMOS tube, NMOS5-Sixth NMOS tube;

[0029] PMOS0 - First PMOS transistor, PMOS1 - Second PMOS transistor

[0030] index0 - First control signal, index1 - Second control signal. Detailed Implementation

[0031] The bit-level dynamically sparse in-memory computing architecture described in this invention has the following characteristics:

[0032] 1. Decompose the original weights using a special quantization pruning algorithm.

[0033] For the original weights used in training deep neural network models, in-memory computing hardware suffers from low storage efficiency, data redundancy, and lack of reusability when storing these weights. Therefore, the in-memory computing architecture described in this invention first quantizes the weights into structured data, and then uses pruning to filter out repetitive parts in the structured weight data, thereby decomposing the original weights into basic weights and repeating weights, facilitating subsequent hardware processing. The basic weights are characterized by high bit width and a large number of weights, while the repeating weights are characterized by low bit width and a small number of weights. Decomposing the original weights not only improves the hardware's storage efficiency but also effectively solves the problem of "uneven computational efficiency caused by uneven bit distribution" in this subsequent step.

[0034] 2. Use a hybrid hardware approach to balance computing efficiency

[0035] Based on the characteristics of the two different weighted data types, the in-memory computing architecture described in this invention employs two calculation methods to calculate the basic weight and the repeated weight respectively. For the basic weighted data, due to its high bit width and large quantity, it is suitable to use an in-memory computing approach. For the repeated weighted data, its low bit width and small quantity are suitable for calculation using ordinary multiply-accumulate units, i.e., systolic array computing. By using different calculation methods for different data types, a balanced computational efficiency is achieved, effectively solving the problem of "mismatch between weight mapping strategy and hardware characteristics."

[0036] 3. Dynamic zero-jump calculation is implemented using a complement 8T-SRAM cell.

[0037] The in-memory computing architecture described in the application uses an improved 8T-SRAM cell, which adds two MOS tubes to the original 6T-SRAM cell. By dynamically controlling the reading of Q and Q~ in the SRAM cell, the original code or the complement output of the stored data can be flexibly selected. At the same time, with the effective bit selection, dynamic bit skip zero can be realized, thereby reducing the calculation period and improving the computing power. The input data is dynamically adjusted, for example, when the number of bit 1 in the input data is greater than the number of bit 0, all bits are inverted, and the original bit 0 index is used for shift accumulation. Since the bit inversion will introduce subtraction operation, the high-efficiency complement mechanism of 8T-SRAM can reduce the additional hardware overhead and obtain shorter calculation period, which can effectively solve the "neglect and insufficient utilization of 'bit-level sparsity'".

[0038] Specifically, the bit-level dynamic sparse in-memory computing architecture described in the application is composed of a software layer and a hardware layer as shown in the figure. Figure 1

[0039] The software layer is deployed on the server side, and is used for weight quantization, pruning and mapping optimization of the neural network model; the hardware layer is deployed on the edge side, and realizes high energy-efficient memory computing integrated computing.

[0040] The hardware layer includes memory computing integrated cores CIM Cores, pulsatile array PEs Array and peripheral circuit. Among them, the basic weight is mapped to the memory computing integrated macro for multiplication and addition calculation, and the repeated weight is calculated by the pulsatile array, and the two cooperate to complete the hybrid inference task. The memory computing integrated core is a multi-core parallel structure, and each memory computing integrated core is composed of a plurality of input processing units IPUs, 8T-SRAM arrays, shift control units and accumulation units. The input processing unit pre-processes the input features, generates indexes and metadata, and then sends them to the 8T-SRAM array for calculation. The 8T-SRAM array is composed of a plurality of 8T-SRAM units, each of which has the reading characteristics of original code and complement switchable, and in combination with the bit-level identification mechanism of the input processing unit, dynamic bit skip zero can be realized, effectively reducing the redundant calculation period and bit line switching power consumption. Without increasing hardware overhead, the energy efficiency is significantly improved, the calculation time is shortened by about 30%, and the power consumption is reduced by about 50%.

[0041] As shown in the figure, Figure 2 ​As shown, the core function of the in-memory computing architecture described in this invention when executing neural network tasks is to achieve efficient multiplication and addition operations between weight data and input feature data. Addressing the mismatch between the weight storage structure and computational precision in traditional CIM architectures, this invention introduces a quantization pruning and filtering algorithm at the software layer, decomposing the trained weight data into two categories: basic weights and repeated weights. Basic weights are used to retain the main effective numerical components, and are numerous and have a high bit width; repeated weights are used to record reusable parts, and are fewer in number and have a lower bit width.

[0042] Based on the different characteristics of the two types of weights, this invention constructs a hybrid computing framework at the hardware layer: the basic weights and their corresponding feature data are calculated in parallel by an in-memory computing core, while repeated weights and feature data are calculated supplementarily by a systolic array. The 8T-SRAM array has the advantages of large area but high computing speed, making it suitable for high-precision weight calculation; the systolic array has small area and low latency, making it suitable for lightweight weight processing. The two arrays work together to achieve a balance between accuracy and speed in the computational task.

[0043] Taking a 16-bit fixed-point weight as an example, this invention decomposes it into a 12-bit basic weight and a 4-bit repeating weight. The 8T-SRAM array only needs to store the high 12 bits, saving approximately one-third of the storage space. After calculation, the two types of results are fused into the final output via an accumulator. This hybrid architecture effectively improves the storage utilization of the 8T-SRAM array, reduces redundant accesses, and achieves simultaneous optimization of computation time and energy consumption. Compared to traditional architectures, the overall computation speed is increased by approximately 25%, and the energy efficiency ratio is improved by approximately 40%.

[0044] like Figure 3 As shown, the software-layer quantization pruning and filtering algorithm of the in-memory computing architecture described in this invention is mainly used for structured quantization and hierarchical filtering of the initial weight array of a neural network model. The algorithm first splits the original weight data according to the weight bit width, dividing it into high-bit-width data and low-bit-width data, so that information at different precision levels can be processed separately in subsequent steps. Then, the algorithm uses a stepwise quantization adjustment mechanism to iteratively optimize the value range of the low-bit-width data.

[0045] After each adjustment, the mean squared error (MSE) and Kullback-Leibler divergence between the adjusted weights and the original weights are calculated. When the adjusted weight values ​​meet the preset error and divergence threshold conditions, it means that the quantization result can maintain the numerical distribution characteristics of the model and will not cause significant accuracy loss; if the conditions are not met, the algorithm will re-select values ​​through exhaustive search until the error converges.

[0046] Through the above iterative screening process, the algorithm can identify a set of low-bit-width bit columns shared in the weight and a differentiated high-bit-width bit column, thereby realizing the structured decomposition of the weight matrix. The low-bit-width part usually has stronger reusability and can be reused or shared at the hardware layer, while the high-bit-width part retains the main numerical information for the main calculation path. Finally, the screened weight data is output as the base weight and the repeated weight for the storage and calculation integrated macro and the systolic array of the architecture to process respectively. The algorithm realizes significant reduction of storage redundancy while maintaining model accuracy, providing an optimized data input basis for subsequent hardware acceleration calculation.

[0047] As shown in Figure 4 The storage and calculation integrated core in the hardware layer of the in-memory computing architecture in the application is composed of multiple storage and calculation integrated macros (CIM Macro) in parallel, which is used to realize large-scale weight-feature multiplication and addition calculation. Each CIM Macro is composed of an input processing unit, an 8T-SRAM array, a shift control unit, and an accumulator, and the modules work cooperatively through data paths and control signals.

[0048] At the beginning of the operation, the input processing unit performs effective bit detection on the input feature data to count the number of "1" and "0" in the input data. If the detection result shows that the number of valid bits bit1 is greater than that of invalid bits bit0, the input processing unit will perform bitwise inversion on the input feature data and generate a new valid bit index table, and send a control signal to the 8T-SRAM array to switch to the complement reading mode. At this time, the 8T-SRAM array reads the corresponding complement weight according to the new index, and the shift control unit performs the shift and accumulation operation according to the index. The accumulation result is collected by the first stage shift adder unit, and then the final summation is output by the post accumulator.

[0049] If the detection result shows that the number of bit1 is less than that of bit0, the original input data and the true code reading mode of the weight are maintained, and the calculation is completed according to the standard shift and accumulation path. Through this adaptive true code / complement code switching and bit zero skipping mechanism, the CIM Macro realizes the dynamic optimization of the operation path without additional hardware. This design effectively reduces the invalid bit operation and redundant shift process, reduces the dynamic power consumption, and improves the overall throughput and energy efficiency of the array, providing efficient hardware support for large-scale bit-level sparse computation.

[0050] As shown in Figure 5 The structure of the 8T-SRAM unit in the application is composed of two PMOS tubes and six NMOS tubes:

[0051] The first PMOS PMOS0 source is connected with Vdd, the first PMOS PMOS0 drain is connected with the storage node, and the first PMOS PMOS0 gate is connected with the reverse node;

[0052] The second PMOS PMOS1 source is connected with Vdd, the second PMOS PMOS1 drain is connected with the reverse node, and the second PMOS PMOS1 gate is connected with the storage node;

[0053] The first NMOS NMOS0 source is connected with the ground, the first NMOS NMOS0 drain is connected with the storage node, and the first NMOS NMOS0 gate is connected with the reverse node;

[0054] The second NMOS NMOS1 source is connected with the ground, the second NMOS NMOS1 drain is connected with the reverse node, and the second NMOS NMOS1 gate is connected with the storage node;

[0055] The third NMOS NMOS2 source is connected with the storage node, the third NMOS NMOS2 drain is connected with the bit line, and the third NMOS NMOS2 gate is connected with the word line;

[0056] The fourth NMOS NMOS3 source is connected with the reverse node, the fourth NMOS NMOS3 drain is connected with the reverse bit line, and the fourth NMOS NMOS3 gate is connected with the word line;

[0057] The fifth NMOS NMOS4 source is connected with the output end, the fifth NMOS NMOS4 drain is connected with the storage node, and the fifth NMOS NMOS4 gate is connected with the second control signal;

[0058] The sixth NMOS NMOS5 source is connected with the reverse output end, the sixth NMOS NMOS5 drain is connected with the reverse node, and the sixth NMOS NMOS5 gate is connected with the first control signal.

[0059] The word line WL is used for controlling the read-write state, and the bit line BL and the reverse bit line BLB are used for data writing and reading operation respectively. The fifth NMOS transistor NMOS4 and the sixth NMOS transistor NMOS5 are connected to the storage node Q and the reverse node Q~ respectively, and the selective access to the storage data is realized through the control signal. The 8T-SRAM unit can output the original data and the direct output of the complement, and the flexibility of data reading is greatly improved. Index1 and Index0 are key control signals, and correspond to the reading mode of the original code and the complement code respectively. When Index1 is high, the fifth NMOS transistor NMOS4 is turned on, and the output end Out is directly connected to the storage node Q, so that the original code data is outputted; when Index0 is high, the sixth NMOS transistor NMOS5 is turned on, and the reverse output end OutB is connected to the reverse node Q~, so that the complement data output is realized. The single SRAM unit can support the dual-mode output, and the complement operation can be completed without additional logic inverter.

[0060] Through the 8T-SRAM unit, the original code / complement code optional reading and dynamic bit switching function are realized while maintaining the stability of the traditional SRAM read-write, the complexity of external logic control is effectively reduced, and an efficient and low-power storage foundation is provided for bit-level dynamic sparse calculation.

[0061] As shown in the figure, Figure 6 As shown in the figure,

[0062] During the whole calculation process, the 8T-SRAM unit provides flexible true and complement reading capability, so that the data can be seamlessly switched in different modes without the need for external logic inverting unit. Finally, the accumulator collects all the intermediate results and outputs, realizing the multiplication and addition result of the feature input and weight data. Through the dynamic zero skipping mechanism based on bit distribution, the invention significantly reduces the number of redundant shifts and invalid operation periods, effectively shortens the calculation time without increasing additional hardware resources, and improves the overall computing power and energy efficiency ratio.

[0063] For those skilled in the art, other various corresponding changes and modifications can be made according to the above described technical solutions and concepts, and all these changes and modifications should belong to the protection scope of the claims of the present application.

Claims

1. A bit-level dynamically sparse in-memory computing architecture, characterized in that, It consists of two parts: a software layer and a hardware layer. The software layer is deployed on the server side and is used to perform weight quantization, pruning, and mapping optimization on the neural network model. The hardware layer is deployed at the edge and is used for in-memory computing. The hardware layer includes a memory computing core (CIM Core), a pulsating array, and peripheral circuits. Basic weights are mapped to the memory computing core for multiplication and addition calculations, while repeated weights are calculated by the pulsating array. The memory computing core and the pulsating array work together to complete the hybrid inference task. The software layer includes a quantitative pruning and screening algorithm that decomposes the trained weight data into two categories: basic weights and repeated weights. Basic weights are used to retain the main effective numerical components, while repeated weights are used to record the reusable parts. The quantization pruning and filtering algorithm is used to perform structured quantization and hierarchical filtering on the initial weight array of the neural network model; the original weight data is split into two parts, high-width data and low-width data, according to the weight bit width. The basic weights are characterized by high bit width and a large number of bits, while the repeated weights are characterized by low bit width and a small number of bits. Based on the different characteristics of the two types of weights, a hybrid computing framework is built at the hardware layer: the basic weights and their corresponding feature data are computed in parallel by the in-memory computing core, while the repeated weights and feature data are computed by the pulsating array.

2. The bit-level dynamically sparse in-memory computing architecture according to claim 1, characterized in that, The in-memory computing core is a multi-core parallel structure. Each in-memory computing core consists of multiple in-memory computing macros (CIMMacro), and each in-memory computing macro consists of an 8T-SRAM array, a shift control unit, and an accumulation unit. The input processing unit preprocesses the input features, generates an index and metadata, and then sends them to the 8T-SRAM array for computation. The 8T-SRAM features switchable reading of original code and complement code, and combined with the bit-level recognition mechanism of the input processing unit, it enables dynamic bit zero jumping.

3. The bit-level dynamically sparse in-memory computing architecture according to claim 2, characterized in that, At the start of the operation, the input processing unit performs valid bit detection on the input feature data to count the number of "1"s and "0"s in the input data. If the detection result shows that the number of valid bits (bit1) is greater than the number of invalid bits (bit0), the input processing unit inverts the input feature data bit by bit and generates a new valid bit index table. At the same time, it sends a control signal to the 8T-SRAM array to switch to the two's complement read mode. At this time, the 8T-SRAM array reads the corresponding two's complement weights according to the new index, and the shift control unit performs shift addition and subtraction operations according to the index. The accumulated result is summarized by the first-stage shift adder unit and then finally summed and output by the subsequent accumulator. If the detection result shows that the number of bits1 is less than that of bits0, the original code reading mode of the original input data and weights is maintained, and the calculation is completed according to the standard shift accumulation path.

4. The bit-level dynamically sparse in-memory computing architecture according to claim 3, characterized in that, Each cell structure of the 8T-SRAM array consists of two PMOS transistors and six NMOS transistors: The source of the first PMOS transistor (PMOS0) is connected to Vdd, the drain of the first PMOS transistor (PMOS0) is connected to the memory node, and the gate of the first PMOS transistor (PMOS0) is connected to the reverse node. The source of the second PMOS transistor (PMOS1) is connected to Vdd, the drain of the second PMOS transistor (PMOS1) is connected to the reverse node, and the gate of the second PMOS transistor (PMOS1) is connected to the memory node. The source of the first NMOS transistor (NMOS0) is grounded, the drain of the first NMOS transistor (NMOS0) is connected to the memory node, and the gate of the first NMOS transistor (NMOS0) is connected to the reverse node. The source of the second NMOS transistor (NMOS1) is grounded, the drain of the second NMOS transistor (NMOS1) is connected to the reverse node, and the gate of the second NMOS transistor (NMOS1) is connected to the memory node. The source of the third NMOS transistor (NMOS2) is connected to the memory node, the drain of the third NMOS transistor (NMOS2) is connected to the bit line, and the gate of the third NMOS transistor (NMOS2) is connected to the word line. The source of the fourth NMOS transistor (NMOS3) is connected to the reverse node, the drain of the fourth NMOS transistor (NMOS3) is connected to the reverse bit line, and the gate of the fourth NMOS transistor (NMOS3) is connected to the word line. The source of the fifth NMOS transistor (NMOS4) serves as the output terminal, the drain of the fifth NMOS transistor (NMOS4) is connected to the memory node, and the gate of the fifth NMOS transistor (NMOS4) is connected to the second control signal. The source of the sixth NMOS transistor (NMOS5) is used as the inverting output terminal, the drain of the sixth NMOS transistor (NMOS5) is connected to the inverting node, and the gate of the sixth NMOS transistor (NMOS5) is connected to the first control signal.

5. The bit-level dynamically sparse in-memory computing architecture according to claim 4, characterized in that, The quantization pruning and screening algorithm adopts a stepwise quantization adjustment mechanism to iteratively optimize the value range of low bit width data; After each adjustment, the mean square error and KL divergence between the adjusted result and the original weights are calculated. When the adjusted weight values ​​meet the preset error and divergence threshold conditions, it means that the quantization result can maintain the numerical distribution characteristics of the model. If the conditions are not met, the values ​​are re-evaluated by exhaustive search until the error converges.

Images