Hybrid RRAM and SRAM reconfigurable in-memory computing system

Abstract

The invention provides an RRAM and SRAM mixed reconfigurable in-memory computing system. The RRAM and SRAM mixed reconfigurable in-memory computing system comprises an RRAM memory calculation core module, an SRAM memory calculation core module, a matrix addition acceleration unit module, a special function unit module, a CPU module, a high-speed bus module and an on-chip shared memory module. Wherein the RRAM storage calculation kernel module is composed of a plurality of RRAM storage calculation arrays, and is used for realizing a convolution operator in neural network reasoning; and the SRAM memory calculation core module consists of a plurality of SRAM memory calculation arrays and is used for realizing full-connection operator and full-connection weight updating in neural network reasoning. According to the method, the storage density advantage of the RRAM and the read-write speed advantage of the SRAM are combined, reasoning and training data streams are optimized, and the performance is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of in-memory computing technology, and more specifically, to a reconfigurable in-memory computing system that combines RRAM and SRAM. Background Technology

[0002] Resistive random-access memory (RRAM or ReRAM) is a type of non-volatile memory whose core principle is based on the reversible conversion of the resistance of a non-conductive material between a high-resistance state and a low-resistance state.

[0003] SRAM (Static Random-Access Memory) is a semiconductor memory based on bistable flip-flops. It stores data through transistor states and can maintain storage while powered on without the need for refresh circuits. Data is lost when power is off.

[0004] Low Earth orbit (LEO) constellations carry numerous remote sensing payloads, requiring onboard computing platforms with powerful real-time processing capabilities to handle the massive amounts of remote sensing image data. Currently, deep learning-based intelligent algorithms are widely used in onboard remote sensing image processing, significantly reducing the amount of data that needs to be transmitted downlink through techniques such as feature extraction, thus greatly alleviating the transmission pressure on the satellite-to-ground link. However, when using the traditional von Neumann architecture for AI computation, the frequent data transfers result in significant latency and power consumption.

[0005] In-memory computing systems can solve the problems caused by data movement mentioned above. By performing calculations directly within the storage unit, they effectively reduce the energy consumption of data movement and significantly improve computational energy efficiency. Because in-memory computing has inherently high efficiency support for matrix multiplication and addition operations, it is particularly suitable for artificial intelligence algorithms and has demonstrated outstanding application value in satellite on-orbit missions. However, as neural network computing structures become more complex, single-architecture in-memory computing systems face challenges in terms of computational energy efficiency and area efficiency.

[0006] Currently, onboard in-memory computing systems still face the following challenges: For the two modes of in-memory computing inference and training, RRAM memory is suitable for the inference mode with high array density, while SRAM memory is suitable for the training mode with fast weight data updates. This requirement for reconfigurable in-memory computing makes data flow scheduling very complex. Therefore, it is necessary to design a reconfigurable in-memory computing system that combines RRAM and SRAM for real-time adjustment.

[0007] Patent application CN102820052A discloses an SRAM multiplexing device, including multiple local multiplexers and one global multiplexer. Each local multiplexer is connected to a memory bank. The global multiplexer has multiple inputs, each connected to a corresponding output of one of the local multiplexers. In response to a decoded address, during a read operation, the inputs of the local multiplexers are transferred to the corresponding input of the global multiplexer. However, this patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention. Summary of the Invention

[0008] In view of the deficiencies in the prior art, the purpose of this invention is to provide a reconfigurable in-memory computing system that combines RRAM and SRAM.

[0009] The reconfigurable in-memory computing system that combines RRAM and SRAM according to the present invention includes: an RRAM in-memory computing core module, an SRAM in-memory computing core module, a matrix addition acceleration unit module, a special function unit module, a CPU module, a high-speed bus module, and an on-chip shared memory module. The RRAM storage and computation kernel module is used to implement the convolution operator in neural network inference; The SRAM storage and computation core module is used to implement the fully connected operator and fully connected weight update in neural network inference; The matrix addition acceleration unit module is used to implement the matrix addition operator in the neural network; The special functional unit module is used to implement the activation function, loss function, and normalization function in the neural network; The CPU module is used to send instructions to control the RRAM storage core module, SRAM storage core module, matrix addition acceleration unit module and special function unit module, and to perform general calculations; The high-speed bus module is used to realize high-speed data interaction between the RRAM storage core module, SRAM storage core module, matrix addition acceleration unit module, special function unit module, CPU module and on-chip shared memory module; The on-chip shared memory module is used to store input data, intermediate results and output data, and realizes data sharing among the RRAM storage core module, SRAM storage core module, matrix addition acceleration unit module and special function unit module.

[0010] Preferably, the RRAM memory core module consists of multiple RRAM memory arrays, each RRAM memory array having a size of 128×128; The RRAM in-memory core module reads input feature map slices from the on-chip shared memory module and broadcasts the input feature map slices to multiple RRAM in-memory arrays; The RRAM memory module performs calculations:

[0011] in, It is the number of output channels. K is the number of input channels, and K is the convolutional layer size. Here, W is the input feature map, and W is the weight of the convolutional layer in the neural network. is the output feature map, h and w are the height and width of the feature map respectively, and i and j are the spatial positions on the two-dimensional plane inside the output channel.

[0012] Preferably, the SRAM memory core module consists of multiple SRAM memory arrays, each SRAM memory array having a size of 128×128; During the inference phase, the SRAM in-memory core module reads the input feature map from the on-chip shared memory module and broadcasts the input feature map to multiple SRAM in-memory arrays to perform calculations:

[0013] During the training phase, the SRAM in-memory core module reads the input error from the on-chip shared memory module. The error is then broadcast to multiple SRAM memory arrays for calculation:

[0014] And update the weights:

[0015] in, For output error, For the updated weights, For learning rate, It is the transpose symbol.

[0016] Preferably, the matrix addition acceleration unit module performs bias addition calculations: Slices of the output feature map of the convolutional layer The bias addition is:

[0017] For the output feature map of the fully connected layer The bias addition is:

[0018] Where b is the bias; The matrix addition acceleration unit module performs residual connection calculation: For the output feature map The residual connection is:

[0019] Where A is any input feature map.

[0020] Preferably, the special functional unit module uses a lookup table to implement the activation function, loss function, and normalization function; When the neural network has an 8-bit precision, each lookup table is 256 in size, and the values ​​in the lookup tables are pre-written.

[0021] Preferably, the CPU module sends instructions to the RRAM memory core module, SRAM memory core module, matrix addition acceleration unit module, and special function unit module; The CPU module performs general-purpose computing, including feature scaling and data normalization of neural network input data, as well as post-processing of neural network output data.

[0022] Preferably, the high-speed bus module connects the RRAM memory module, SRAM memory module, matrix addition acceleration unit module, special function unit module, CPU module and on-chip shared memory module, supporting high-speed data interaction between modules.

[0023] Preferably, the on-chip shared memory module stores input data, intermediate results, and output data; The on-chip shared memory module provides input feature map slices to the RRAM in-memory core module and input feature maps and errors to the SRAM in-memory core module.

[0024] Preferably, during the inference phase, both the RRAM memory module and the SRAM memory module are used for forward propagation; The RRAM memory core module and the SRAM memory core module adopt an adjustable precision architecture, supporting floating-point mode and fixed-point mode.

[0025] Preferably, during the training phase, the RRAM memory module is used for forward propagation with fixed weights; The SRAM storage and computation module is used for forward propagation, backward propagation, gradient calculation, and weight update for non-fixed weights. The SRAM storage and processing core module includes a transpose module and a normal module; The transpose module supports matrix transpose and matrix multiplication and addition, and is used for forward propagation and backward propagation. The general module supports matrix multiplication and addition for forward propagation and gradient calculation; After the ordinary module calculates the gradient result, it updates the weight parameters of the transpose module.

[0026] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a reconfigurable in-memory computing system that combines RRAM and SRAM. By combining the advantages of RRAM array in terms of storage density and SRAM array in terms of read and write speed, fixed weights that do not need to be updated are deployed in the RRAM array, while weights that need to be updated frequently are deployed in the SRAM array, thereby achieving optimal performance improvement. Attached Figure Description

[0027] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A schematic diagram of a reconfigurable in-memory computing system that combines RRAM and SRAM, provided for the present invention; Figure 2 A schematic diagram of a reconfigurable in-memory computing inference data flow scheme that combines RRAM and SRAM provided by the present invention; Figure 3 This is a schematic diagram of a reconfigurable in-memory computation training data flow scheme that combines RRAM and SRAM, as provided by the present invention.

[0028] Reference numerals: RRAM memory core module 101, RRAM memory array 1011, SRAM memory core module 102, SRAM memory array 1021, matrix addition acceleration unit module 103, special function unit module 104, CPU module 105, high-speed bus module 106, on-chip shared memory module 107. Detailed Implementation

[0029] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0030] Example Figure 1 A schematic diagram of a reconfigurable in-memory computing system combining RRAM and SRAM is provided for this invention, as shown below. Figure 1 As shown, the present invention provides a reconfigurable in-memory computing system that combines RRAM and SRAM, including: an RRAM in-memory computing core module 101, an SRAM in-memory computing core module 102, a matrix addition acceleration unit module 103, a special function unit module 104, a CPU module 105, a high-speed bus module 106, and an on-chip shared memory module 107.

[0031] The RRAM memory core module 101 consists of multiple RRAM memory arrays 1011, used to implement convolution operators in neural network inference. For example, the weight W of the current neural network convolutional layer has a size of... × ×K×K, where It is the number of output channels. Where K is the number of input channels and K is the convolutional layer size, and the current size of the RRAM memory array 1011 is 128×128, then the number of RRAM memory arrays 1011 required by the RRAM memory core module 101 is... / 128 × ×K×K / 128 An input feature map is read from the on-chip shared memory module 107. slice ×K×K and broadcast it to / 128 × ×K×K / 128 In the RRAM memory array 1011, in-memory computation is performed in the RRAM memory core module 101. The in-memory computation can be implemented as either analog in-memory computation based on an ADC or digital in-memory computation based on an addition tree.

[0032] The SRAM memory module 102 consists of multiple SRAM memory arrays 1021, used to implement fully connected operators and fully connected weight updates in neural network inference. For example, the current weight W of the fully connected layer of the neural network has a size of... × ,in It is the output size. The input size is 128×128. Therefore, the number of SRAM memory arrays 1021 required by the SRAM memory core module 102 is... / 128 × / 128 During inference, a memory module of size 107 is read from the on-chip shared memory module 107. Input feature map And broadcast it to / 128 × / 128 In the SRAM memory array 1021, in-memory computation is performed in the SRAM memory core module 102. The in-memory computation can be implemented as either analog in-memory computation based on an ADC or digital in-memory computation based on an addition tree. During training weight updates, a memory module with a size of [size missing] is read from the on-chip shared memory module 107. error And broadcast it to / 128 × / 128 In the SRAM memory array 1021, in-memory computation is performed in the SRAM memory core module 102. The calculated Lout is then passed to the next layer. Finally, based on... Update weights .

[0033] RRAM has an advantage in storage density but relatively slow read / write speed; SRAM has an advantage in read / write speed but lower storage density. Therefore, RRAM's high storage density is suitable for storing fixed weights that don't need updating during inference, while SRAM is suitable for storing weight parameters that need frequent updates during training. During neural network training, when performing transfer learning, the weight parameters of the first few convolutional layers are kept fixed to prevent overfitting on small datasets and preserve the model's general feature extraction capabilities. In this case, the RRAM storage kernel module 101 is used for fixed-weight neural network computation, while the SRAM storage kernel module 102 is used for non-fixed-weight neural network computation and weight updates, improving computational efficiency.

[0034] The matrix addition acceleration unit module 103 is used to implement matrix addition operators in neural networks, including bias addition, residual connections, and other algorithms. For example, bias addition involves slicing the output feature map of the current neural network convolutional layer. The bias addition is For example, the current size of the fully connected layer in a neural network is... Output feature map The bias addition is For residual connections, for example, the current convolutional layer size of the neural network is... Output feature map of ×H×W The residual connection is , where H and W are the height and width of the feature map, respectively, and A is the input feature map of a certain layer.

[0035] The special function unit module 104 is used to implement special functions such as activation layer operators in the neural network, including activation functions such as ReLU and Sigmoid, loss functions such as cross-entropy and Huber loss, and normalization functions such as Softmax and BatchNorm. These activation functions, loss functions, and normalization functions are implemented in hardware using lookup tables in the special function unit module 104. For example, when the neural network precision is 8 bits, the lookup table size for each special function is 256, and the values ​​in the lookup table are pre-written.

[0036] The CPU module 105 is used to send instructions to control other modules and perform general-purpose calculations. These general-purpose calculations include feature scaling and data normalization of neural network input data, as well as post-processing of neural network output data to decision results.

[0037] The high-speed bus module 106 is used to realize high-speed data interaction between modules and supports the interconnection of multiple heterogeneous computing resources such as the RRAM storage and computing core module 101, the SRAM storage and computing core module 102, the matrix addition acceleration unit module 103, the special function unit module 104, and the CPU module 105, thereby reducing the energy consumption of data transmission.

[0038] The on-chip shared memory module 107 is used to store input data, intermediate results, output data, etc., and enables low-latency data sharing among multiple modules such as the RRAM storage and processing core module 101, the SRAM storage and processing core module 102, the matrix addition acceleration unit module 103, and the special function unit module 104.

[0039] Figure 2 This invention provides a schematic diagram of a reconfigurable in-memory computing inference data flow scheme that combines RRAM and SRAM, as shown below. Figure 2 As shown, this invention provides a reconfigurable in-memory computation inference dataflow scheme that combines RRAM and SRAM, achieving optimal performance improvement by leveraging the advantages of hybrid storage devices. During the inference phase, only forward propagation is included, excluding backpropagation and gradient calculation. In this case, both the RRAM in-memory core module 101 and the SRAM in-memory core module 102 are used entirely for forward propagation. Both modules employ a precision-adjustable architecture, allowing for either floating-point mode to improve accuracy or fixed-point mode to reduce power consumption.

[0040] Figure 3 This invention provides a schematic diagram of a reconfigurable in-memory computation training data flow scheme that combines RRAM and SRAM, as shown below. Figure 3As shown, this invention provides a reconfigurable in-memory computation training data stream scheme that combines RRAM and SRAM, achieving optimal performance improvement by leveraging the advantages of hybrid storage devices. During the training phase, processes include forward propagation, backward propagation, gradient calculation, and weight updates. At this time, the RRAM in-memory computation core module 101 is used for forward propagation with fixed weights, while the SRAM in-memory computation core module 102 is used for forward propagation, backward propagation, gradient calculation, and weight updates with non-fixed weights. Figure 3 As shown, the SRAM in-memory computation module 102 is divided into two types: a transpose module that supports matrix transpose and matrix multiplication and addition, used for forward and backward propagation; and a regular module that only supports matrix multiplication and addition, without supporting matrix transpose, used for forward propagation and gradient calculation. After the gradient result is calculated by the regular module, the weight parameters of the transpose module are updated. This reconfigurable in-memory computation training data flow scheme optimizes the inference and training data flow by deploying fixed weights on the RRAM array, leveraging its low power consumption and high storage density; and deploying weights that need frequent updates on the SRAM array, leveraging its high read / write speed and high precision, thus achieving a significant performance improvement.

[0041] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0042] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A reconfigurable in-memory computing system hybridizing RRAM and SRAM, characterized in that, Comprise: a RRAM computing core module, an SRAM computing core module, a matrix addition acceleration unit module, a special function unit module, a CPU module, a high-speed bus module and an on-chip shared memory module; the RRAM computing core module is used for implementing a convolution operator in neural network inference; the SRAM computing core module is used for implementing a full connection operator and full connection weight update in neural network inference; the matrix addition acceleration unit module is used for implementing a matrix addition operator in a neural network; the special function unit module is used for implementing an activation function, a loss function and a normalization function in a neural network; the CPU module is used for sending instructions to control the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module and the special function unit module, and performing general calculation; the high-speed bus module is used for implementing high-speed data interaction between the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module, the special function unit module, the CPU module and the on-chip shared memory module; the on-chip shared memory module is used for storing input data, intermediate results and output data, and implementing data sharing between the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module and the special function unit module. 2.The RRAM and SRAM hybrid in-memory computing system of claim 1, wherein, The RRAM computing core module is composed of a plurality of RRAM computing arrays, and each RRAM computing array has a size of 128x128; The RRAM computing core module reads an input feature map slice from the on-chip shared memory module and broadcasts the input feature map slice to a plurality of RRAM computing arrays; The RRAM computing core module performs calculation: wherein, is the number of output channels, is the number of input channels, K is the convolution layer size, is the input feature map, W is the weight of the neural network convolution layer, is the output feature map, h and w are the height and width of the feature map, respectively, and i and j are the spatial positions on the two-dimensional plane within the output channel.

3. The RRAM and SRAM hybrid in-memory computing system of claim 2, wherein, The SRAM computing core module is composed of a plurality of SRAM computing arrays, and each SRAM computing array has a size of 128x128; In the inference stage, the SRAM computing core module reads an input feature map from the on-chip shared memory module and broadcasts the input feature map to a plurality of SRAM computing arrays, performs calculation: In a training phase, the SRAM compute core module reads input errors from the on-chip shared memory module and broadcasts the errors to a plurality of SRAM compute arrays, performing a computation: and updates the weight: wherein, is the output error, is the updated weight, is the learning rate, is the transpose symbol.

4. The RRAM and SRAM hybrid in-memory computing system of claim 3, wherein, The matrix addition acceleration unit module performs bias addition calculation: For the convolutional layer output feature map slice , the bias addition is: For the fully connected layer output feature map , the bias addition is: wherein b is a bias; The matrix addition acceleration unit module performs residual connection calculation: For the output feature map , the residual connection is: wherein A is any input feature map.

5. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, The special function unit module implements activation function, loss function and normalization function by using lookup table; When the neural network precision is 8 bits, the size of each lookup table is 256, and the values in the lookup table are pre-written.

6. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, The CPU module sends instructions to the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module and the special function unit module; The CPU module performs general calculation, including feature scaling and data normalization of neural network input data, and post-processing of neural network output data.

7. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, The high-speed bus module connects the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module, the special function unit module, the CPU module and the on-chip shared memory module, and supports high-speed data interaction between the modules.

8. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, The on-chip shared memory module stores input data, intermediate results and output data; The on-chip shared memory module provides the RRAM compute-in-memory module with an input feature map slice and provides the SRAM compute-in-memory module with an input feature map and an error.

9. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, In the inference phase, the RRAM compute-in-memory module and the SRAM compute-in-memory module are both used for forward propagation. The RRAM compute-in-memory module and the SRAM compute-in-memory module adopt a precision-adjustable architecture and support floating-point mode and fixed-point mode.

10. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, In the training phase, the RRAM compute-in-memory module is used for forward propagation of fixed weights. The SRAM compute-in-memory module is used for forward propagation of non-fixed weights, back propagation, gradient calculation and weight update. The SRAM compute-in-memory module includes a transpose module and a normal module. The transpose module supports matrix transpose and matrix multiplication and addition and is used for forward propagation and back propagation. The normal module supports matrix multiplication and addition and is used for forward propagation and gradient calculation. After the normal module calculates the gradient result, the weight parameters of the transpose module are updated.

Images