Reconfigurable in-memory computing system hybridizing rram and sram

Abstract

The application provides a reconfigurable in-memory computing system combining RRAM and SRAM, comprising a RRAM computing core module, an SRAM computing core module, a matrix addition acceleration unit module, a special function unit module, a CPU module, a high-speed bus module and an on-chip shared memory module. The RRAM computing core module is composed of a plurality of RRAM computing arrays and is used for implementing convolution operators in neural network inference; the SRAM computing core module is composed of a plurality of SRAM computing arrays and is used for implementing full connection operators and full connection weight updates in neural network inference. The application combines the advantages of RRAM in storage density and the advantages of SRAM in read-write speed, optimizes the data flow of inference and training, and realizes significant performance improvement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of in-memory computing, in particular to a reconfigurable in-memory computing system mixed with RRAM and SRAM. BACKGROUND

[0002] Resistive random-access memory (RRAM or ReRAM) is a non-volatile memory whose core principle is based on the reversible conversion of the resistance of non-conductive materials between high and low resistance states.

[0003] Static random-access memory (SRAM) is a semiconductor memory based on bistable flip-flop, which saves data through transistors and maintains storage in the power supply state without refreshing circuit, and loses data after power off.

[0004] Low Earth Orbit constellation system carries a large number of remote sensing payloads, which requires the on-board computing platform to have strong real-time processing capability to meet the processing needs of massive remote sensing image data. At present, intelligent algorithms based on deep learning are widely used in on-board remote sensing image processing. Through feature extraction and other technical means, the amount of data to be transmitted is greatly reduced, and the transmission pressure of the satellite-ground link is significantly alleviated. However, when AI computing is performed using the traditional von Neumann architecture, the data transfer needs to be performed frequently, resulting in large latency and power consumption overhead.

[0005] In-memory computing systems can solve the above problems caused by data transfer, effectively reducing the energy consumption overhead of data transfer, and greatly improving the computing energy efficiency ratio. Since in-memory computing has natural high efficiency support for matrix multiplication and addition operations, it is particularly suitable for artificial intelligence algorithms and has shown outstanding application value in satellite on-orbit missions. However, as the neural network computing structure becomes complex, single-architecture in-memory computing systems face problems in computing energy efficiency, area efficiency, etc.

[0006] Current on-board in-memory computing systems still face the following challenges: for in-memory computing inference and training modes, RRAM memory is suitable for high array density inference mode, and SRAM memory is suitable for weight data fast update training mode. The reconfigurable requirements of the inference and training make the data flow scheduling very complex, so it is necessary to design a reconfigurable in-memory computing system mixed with RRAM and SRAM for real-time adjustment.

[0007] Patent application document CN102820052A discloses a SRAM multiplexing device, comprising a plurality of local multiplexers and a global multiplexer. Each local multiplexer is connected with a memory bank. The global multiplexer has a plurality of input ends, each of which is connected with a corresponding output end of the plurality of local multiplexers. In response to a decoded address, during a read operation, the input of the local multiplexer is transmitted to the corresponding input end of the global multiplexer. However, the patent cannot completely solve the existing technical problems and cannot meet the needs of the present application. SUMMARY

[0008] In view of the defects in the prior art, the purpose of the present application is to provide a reconfigurable in-memory computing system combining RRAM and SRAM.

[0009] The reconfigurable in-memory computing system combining RRAM and SRAM provided by the present application comprises an RRAM computing core module, an SRAM computing core module, a matrix addition acceleration unit module, a special function unit module, a CPU module, a high-speed bus module and an on-chip shared memory module.

[0010] The RRAM computing core module is used to implement convolution operators in neural network inference.

[0011] The SRAM computing core module is used to implement fully connected operators and fully connected weight updates in neural network inference.

[0012] The matrix addition acceleration unit module is used to implement matrix addition operators in neural networks.

[0013] The special function unit module is used to implement activation functions, loss functions and normalization functions in neural networks.

[0014] The CPU module is used to send instructions to control the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module and the special function unit module, and to perform general-purpose computing.

[0015] The high-speed bus module is used to realize high-speed data interaction between the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module, the special function unit module, the CPU module and the on-chip shared memory module.

[0016] The on-chip shared memory module is used to store input data, intermediate results and output data, and to realize data sharing between the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module and the special function unit module.

[0017] Preferably, the RRAM compute core module is composed of multiple RRAM compute arrays, each RRAM compute array has a size of 128x128;

[0018] The RRAM compute core module reads input feature map slices from the on-chip shared memory module and broadcasts the input feature map slices to multiple RRAM compute arrays;

[0019] The RRAM compute core module performs computation:

[0020]

[0021] wherein, is the number of output channels, is the number of input channels, K is the size of convolution layer, is the input feature map, W is the weight of neural network convolution layer, is the output feature map, h and w are the height and width of the feature map respectively, i and j are the spatial positions on the two-dimensional plane within the output channel.

[0022] Preferably, the SRAM compute core module is composed of multiple SRAM compute arrays, each SRAM compute array has a size of 128x128;

[0023] In the inference phase, the SRAM compute core module reads input feature maps from the on-chip shared memory module and broadcasts the input feature maps to multiple SRAM compute arrays, and performs computation:

[0024]

[0025] In the training phase, the SRAM compute core module reads input error from the on-chip shared memory module and broadcasts the error to multiple SRAM compute arrays, and performs computation:

[0026]

[0027] and updates the weight:

[0028]

[0029] wherein, is the output error, is the updated weight, is the learning rate, is the transpose symbol.

[0030] Preferably, the matrix addition acceleration unit module performs bias addition computation:

[0031] For the output feature map slice of the convolution layer , the bias addition is:

[0032]

[0033] For the output feature map of the full connection layer , the bias addition is:

[0034]

[0035] Wherein, b is the bias;

[0036] The matrix addition acceleration unit module performs residual connection calculation:

[0037] For the output feature map , the residual connection is:

[0038]

[0039] Wherein, A is any input feature map.

[0040] Preferably, the special function unit module realizes the activation function, the loss function and the normalization function by using a lookup table.

[0041] When the precision of the neural network is 8 bits, the size of each lookup table is 256, and the values in the lookup table are pre-written.

[0042] Preferably, the CPU module sends instructions to the RRAM memory and computing core module, SRAM memory and computing core module, matrix addition acceleration unit module and special function unit module.

[0043] The CPU module performs general-purpose computing, including feature scaling and data normalization of neural network input data, and post-processing of neural network output data.

[0044] Preferably, the high-speed bus module connects the RRAM memory and computing core module, SRAM memory and computing core module, matrix addition acceleration unit module, special function unit module, CPU module and on-chip shared memory module, and supports high-speed data interaction between modules.

[0045] Preferably, the on-chip shared memory module stores input data, intermediate results and output data.

[0046] The on-chip shared memory module provides the RRAM memory and computing core module with input feature map slices, and provides the SRAM memory and computing core module with input feature maps and errors.

[0047] Preferably, in the inference stage, the RRAM memory and computing core module and the SRAM memory and computing core module are both used for forward propagation.

[0048] The RRAM memory and computing core module and the SRAM memory and computing core module adopt a precision-adjustable architecture, support floating point mode and fixed point mode.

[0049] Preferably, in the training phase, the RRAM memory and computing core module is used for forward propagation of fixed weights.

[0050] The SRAM memory and computing core module is used for forward propagation, backward propagation, gradient calculation and weight update of non-fixed weights.

[0051] The SRAM memory and computing core module comprises a transpose module and a normal module.

[0052] The transpose module supports matrix transpose and matrix multiplication and addition, and is used for forward propagation and backward propagation.

[0053] The normal module supports matrix multiplication and addition, and is used for forward propagation and gradient calculation.

[0054] After the normal module calculates the gradient result, the weight parameters of the transpose module are updated.

[0055] Compared with the prior art, the present application has the following beneficial effects:

[0056] The present application provides a reconfigurable in-memory computing system mixed with RRAM and SRAM, which combines the advantages of storage density of RRAM array and the advantages of read-write speed of SRAM array, deploys fixed weights that do not need to be updated in the RRAM array, and deploys weights that need to be frequently updated in the SRAM array, so as to achieve optimal performance improvement. BRIEF DESCRIPTION OF DRAWINGS

[0057] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, read in conjunction with the accompanying drawings:

[0058] Figure 1 A reconfigurable in-memory computing system mixed with RRAM and SRAM provided by the present application is shown in the figure;

[0059] Figure 2 A reconfigurable in-memory computing inference data flow scheme mixed with RRAM and SRAM provided by the present application is shown in the figure;

[0060] Figure 3 A reconfigurable in-memory computing training data flow scheme mixed with RRAM and SRAM provided by the present application is shown in the figure.

[0061] Figures: RRAM computing core module 101, RRAM computing array 1011, SRAM computing core module 102, SRAM computing array 1021, matrix addition acceleration unit module 103, special function unit module 104, CPU module 105, high-speed bus module 106, on-chip shared memory module 107. DETAILED DESCRIPTION

[0062] The present application will be described in detail below with specific embodiments. The following examples will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made. These are within the scope of the present application.

[0063] EMBODIMENT

[0064] Figure 1 A RRAM and SRAM hybrid reconfigurable in-memory computing system provided by the present application is shown in Figure Figure 1 The present application provides a RRAM and SRAM hybrid reconfigurable in-memory computing system, comprising: a RRAM computing core module 101, a SRAM computing core module 102, a matrix addition acceleration unit module 103, a special function unit module 104, a CPU module 105, a high-speed bus module 106, and an on-chip shared memory module 107.

[0065] The RRAM computing core module 101 is composed of a plurality of RRAM computing arrays 1011, which are used to implement convolution operators in neural network inference. For example, the weight W size of the current neural network convolution layer is × ×K×K, where is the number of output channels, is the number of input channels, and K is the size of the convolution layer. The size of the current RRAM computing array 1011 is 128x128, so the number of RRAM computing arrays 1011 required by the RRAM computing core module 101 is / 128 × ×K×K / 128 An input feature map slice ×K×K is read from the on-chip shared memory module 107 and broadcast to / 128 × ×K×K / 128 In the RRAM memory-computing core module 101, in-memory computing is performed The in-memory computing implementation can be ADC-based analog in-memory computing or addition tree-based digital in-memory computing.

[0066] The SRAM memory-computing core module 102 is composed of a plurality of SRAM memory-computing arrays 1021, used to implement full connection operators and full connection weight updates in neural network inference. For example, the weight W size of the current neural network full connection layer is × , where is the output size, is the input size, and the size of the current SRAM memory-computing array 1021 is 128x128. The number of SRAM memory-computing arrays 1021 required by the SRAM memory-computing core module 102 is / 128 × / 128 In inference, an input feature map with a size of is read from the on-chip shared memory module 107 and broadcast to / 128 × / 128 SRAM memory-computing arrays 1021. In the SRAM memory-computing core module 102, in-memory computing is performed The in-memory computing implementation can be ADC-based analog in-memory computing or addition tree-based digital in-memory computing. In weight update training, an error with a size of is read from the on-chip shared memory module 107 and broadcast to / 128 × / 128 SRAM memory-computing arrays 1021. In the SRAM memory-computing core module 102, in-memory computing is performed and the calculated Lout is passed to the next layer. Finally, the weight is updated. .

[0067] ​​RRAM has an advantage in storage density but relatively slow read / write speed; SRAM has an advantage in read / write speed but lower storage density. Therefore, RRAM's high storage density is suitable for storing fixed weights that don't need updating during inference, while SRAM is suitable for storing weight parameters that need frequent updates during training. During neural network training, when performing transfer learning, the weight parameters of the first few convolutional layers are kept fixed to prevent overfitting on small datasets and preserve the model's general feature extraction capabilities. In this case, the RRAM storage kernel module 101 is used for fixed-weight neural network computation, while the SRAM storage kernel module 102 is used for non-fixed-weight neural network computation and weight updates, improving computational efficiency.

[0068] The matrix addition acceleration unit module 103 is used to implement matrix addition operators in neural networks, including bias addition, residual connections, and other algorithms. For bias addition, for example, it involves slicing the output feature map of the current neural network convolutional layer. The bias addition is For example, the current size of the fully connected layer in a neural network is... Output feature map The bias addition is For residual connections, for example, the current convolutional layer size of the neural network is... Output feature map of ×H×W The residual connection is , where H and W are the height and width of the feature map, respectively, and A is the input feature map of a certain layer.

[0069] The special function unit module 104 is used to implement special functions such as activation layer operators in the neural network, including activation functions such as ReLU and Sigmoid, loss functions such as cross-entropy and Huber loss, and normalization functions such as Softmax and BatchNorm. These activation functions, loss functions, and normalization functions are implemented in hardware using lookup tables in the special function unit module 104. For example, when the neural network precision is 8 bits, the lookup table size for each special function is 256, and the values ​​in the lookup table are pre-written.

[0070] The CPU module 105 is used to send instructions to control other modules and perform general-purpose calculations. These general-purpose calculations include feature scaling and data normalization of neural network input data, as well as post-processing of neural network output data to decision results.

[0071] The high-speed bus module 106 is used to realize high-speed data interaction between modules, support the interconnection of the RRAM memory and computing core module 101, the SRAM memory and computing core module 102, the matrix addition acceleration unit module 103, the special function unit module 104, the CPU module 105 and other multi-heterogeneous computing resources, and reduce the energy consumption overhead of data transmission.

[0072] The on-chip shared memory module 107 is used to store input data, intermediate results, output data and the like, and realizes low-delay data sharing between the RRAM memory and computing core module 101, the SRAM memory and computing core module 102, the matrix addition acceleration unit module 103, the special function unit module 104 and other modules.

[0073] Figure 2 A RRAM and SRAM mixed reconfigurable in-memory computing inference data flow scheme provided by the present application is shown in FIG. 1. Figure 2 The present application provides a RRAM and SRAM mixed reconfigurable in-memory computing inference data flow scheme, which realizes optimal performance improvement by mixing the advantages of storage devices. In the inference stage, only forward propagation is included, and the reverse propagation and gradient calculation process are not included. At this time, the RRAM memory and computing core module 101 and the SRAM memory and computing core module 102 are all used for forward propagation. The RRAM memory and computing core module 101 and the SRAM memory and computing core module 102 both adopt a precision adjustable architecture, which can adopt a floating point mode to improve precision, or a fixed point mode to reduce power consumption.

[0074] Figure 3 A RRAM and SRAM mixed reconfigurable in-memory computing training data flow scheme provided by the present application is shown in FIG. 2. Figure 3 The present application provides a RRAM and SRAM mixed reconfigurable in-memory computing training data flow scheme, which realizes optimal performance improvement by mixing the advantages of storage devices. In the training stage, forward propagation, reverse propagation, gradient calculation, weight update and other processes are included. At this time, the RRAM memory and computing core module 101 is used for forward propagation of fixed weights, and the SRAM memory and computing core module 102 is used for forward propagation, reverse propagation, gradient calculation and weight update of non-fixed weights. Figure 3As shown, the SRAM memory and computing core module 102 is divided into two types, the transpose module supports matrix transpose and matrix multiplication and addition, which is used for forward propagation and back propagation; the ordinary module only supports matrix multiplication and addition, does not support matrix transpose, which is used for forward propagation and gradient calculation. After the gradient result is calculated by the ordinary module, the weight parameters of the transpose module are updated. The reconfigurable in-memory computing training data flow scheme deploys the fixed weight on the RRAM array, which takes advantage of its low energy consumption and high storage density; the weight that needs to be frequently updated is deployed on the SRAM array, which takes advantage of its high read-write speed and high precision, optimizes the inference and training data flow, and significantly improves the performance.

[0075] Those skilled in the art know that, in addition to implementing the system, device and each module thereof provided by the present application in the form of pure computer readable program code, the same program can also be realized in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and each module thereof provided by the present application can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing methods and structures within hardware components.

[0076] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other without conflict.

Claims

1. A reconfigurable in-memory computing system hybridizing RRAM and SRAM, characterized in that, Comprise: a RRAM computing core module, an SRAM computing core module, a matrix addition acceleration unit module, a special function unit module, a CPU module, a high-speed bus module and an on-chip shared memory module; the RRAM computing core module is used for implementing a convolution operator in neural network inference; the SRAM computing core module is used for implementing a full connection operator and full connection weight update in neural network inference; the matrix addition acceleration unit module is used for implementing a matrix addition operator in a neural network; the special function unit module is used for implementing an activation function, a loss function and a normalization function in a neural network; the CPU module is used for sending instructions to control the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module and the special function unit module, and performing general calculation; the high-speed bus module is used for implementing high-speed data interaction between the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module, the special function unit module, the CPU module and the on-chip shared memory module; the on-chip shared memory module is used for storing input data, intermediate results and output data, and implementing data sharing between the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module and the special function unit module; the RRAM computing core module is composed of a plurality of RRAM computing arrays, and each RRAM computing array has a size of 128x128; the RRAM computing core module reads an input feature map slice from the on-chip shared memory module and broadcasts the input feature map slice to a plurality of RRAM computing arrays; the RRAM computing core module performs calculation: wherein, is the number of output channels, is the number of input channels, K is the convolution layer size, is the input feature map, W is the weight of the neural network convolution layer, is the output feature map, h and w are the height and width of the feature map, respectively, and i and j are the spatial positions on the two-dimensional plane within the output channel.

2. The RRAM and SRAM hybrid in-memory computing system of claim 1, wherein, the SRAM computing core module is composed of a plurality of SRAM computing arrays, and each SRAM computing array has a size of 128x128; in an inference stage, the SRAM computing core module reads an input feature map from the on-chip shared memory module and broadcasts the input feature map to a plurality of SRAM computing arrays, performs calculation: In a training phase, the SRAM compute core module reads input errors from the on-chip shared memory module and broadcasts the errors to a plurality of SRAM compute arrays, performing a computation: and updates weights: wherein, is the output error, is the updated weight, is the learning rate, is the transpose symbol.

3. The RRAM and SRAM hybrid in-memory computing system of claim 2, wherein, the matrix addition acceleration unit module performs bias addition calculation: For the convolutional layer output feature map slice , the bias addition is: For the fully connected layer output feature map , the bias addition is: wherein b is a bias; the matrix addition acceleration unit module performs residual connection calculation: For the output feature map , the residual connection is: wherein A is any input feature map.

4. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, The special function unit module implements activation functions, loss functions and normalization functions using lookup tables; when the neural network precision is 8 bits, the size of each lookup table is 256, and the values in the lookup table are pre-written.

5. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, The CPU module sends instructions to the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module and the special function unit module; The CPU module performs general calculation, including feature scaling and data normalization of neural network input data, and post-processing of neural network output data.

6. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, The high-speed bus module connects the RRAM computing core module, the SRAM computing core module, the matrix addition acceleration unit module, the special function unit module, the CPU module and the on-chip shared memory module, and supports high-speed data interaction between the modules.

7. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, The on-chip shared memory module stores input data, intermediate results and output data; The on-chip shared memory module provides the RRAM compute-in-memory module with an input feature map slice and provides the SRAM compute-in-memory module with an input feature map and an error.

8. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, In the inference phase, the RRAM compute-in-memory module and the SRAM compute-in-memory module are both used for forward propagation. The RRAM compute-in-memory module and the SRAM compute-in-memory module adopt a precision-adjustable architecture and support floating-point mode and fixed-point mode.

9. The RRAM and SRAM hybrid in-memory reconfigurable computing system of claim 1, wherein, In the training phase, the RRAM compute-in-memory module is used for forward propagation of fixed weights. The SRAM compute-in-memory module is used for forward propagation of non-fixed weights, back propagation, gradient calculation and weight update. The SRAM compute-in-memory module includes a transpose module and a normal module. The transpose module supports matrix transpose and matrix multiplication and addition and is used for forward propagation and back propagation. The normal module supports matrix multiplication and addition and is used for forward propagation and gradient calculation. After the normal module calculates the gradient result, the weight parameters of the transpose module are updated.

Images