Stream control storage and computing architecture design method for neural network inference

Abstract

The application relates to a flow control storage and calculation architecture design method for neural network inference, wherein the method comprises the following steps: storing more weight parameters by utilizing the large storage density of an RRAM array, and configuring fewer storage calculation resources to perform data preprocessing such as weight quantization and inverse quantization with low accuracy requirements. The weight parameters are moved to the calculation unit by the storage and calculation integrated system, the calculation unit temporarily stores the reused weights, and receives feature map data or variable matrix data for multiplication and accumulation operation. The operation is directly controlled by data flow, without complex control instructions, so that the matching and completeness of data are guaranteed. The application supports various neural network inference requirements, improves the calculation power utilization rate and matrix calculation efficiency, and reduces the consumption of data movement. Therefore, the problems that the current heterogeneous computing architecture for inference tasks cannot meet the calculation requirements of neural networks with various parameter scales, the matrix calculation efficiency and calculation power utilization rate are low, and the consumption of data movement is large are solved.

Description

Technical Field

[0001] This application relates to the field of flow-controlled in-memory computing architecture technology, and in particular to a flow-controlled in-memory computing architecture design method for neural network inference. Background Technology

[0002] With the rapid development of information technology, the inference tasks of neural networks are no longer limited to traditional convolutional neural networks. Large language models are becoming increasingly important in applications, leading to a massive increase in the number of training parameters required for inference tasks and a sharp rise in data migration time. The memory wall has become one of the bottlenecks in the development of traditional von Neumann computing architectures. Among numerous solutions, in-memory computing (IMC) systems, which integrate computing and storage units, are considered a breakthrough direction for high-performance computing due to their characteristics of large-scale parallel operation, low energy consumption, and reduced data migration costs. In the traditional computer architecture, the storage module serves the computation, so the design considers the separation and prioritization of storage and computation. However, at present, storage and computation need to be considered holistically to serve data acquisition, transmission, and processing in the best possible way. In the past few years, researchers have proposed near-memory architectures that directly package storage and computing modules together, as well as in-memory computing architectures based on RRAM and other memories, where independent computing units located within memory chips or storage areas perform computations. Currently, IMC systems are heterogeneous systems, combining computing units using different types of instruction sets and architectures, with these different computing units sharing a unified memory system.

[0003] However, current in-memory computing systems using RRAM (Resistive Random Access Memory) exhibit significant errors in inference calculations for precision-sensitive neural network layers when performing simulations within memory. Furthermore, placing the computation units within the storage area effectively occupies the original storage space, reducing storage density and thus diminishing the high storage density advantage of RRAM. Additionally, the heterogeneity of modules in in-memory computing systems often necessitates complex instruction thread control for data matching.

[0004] Current in-memory computing systems are all heterogeneous computing systems. The computing unit architecture must not only meet the computational requirements of convolutional neural networks but also handle the numerous variable matrix multiplications in large language model inference. Storage units require precise control to match the computing units. In these variable matrix calculations, some neural network layers have high precision requirements, and some large models have a massive number of parameters, requiring significant storage resources. Some inference tasks can reuse weight parameters extensively, while others are almost unusable, requiring storage units to continuously provide new parameters. Although some researchers have implemented in-memory computing chips using RRAM, these chips exhibit larger errors in analog computation compared to traditional digital computing units, and the storage density of pure RRAM is reduced to achieve in-memory computing functionality. Other researchers have designed PIM (Processing-in-Memory) chips using DRAM (Dynamic Random Access Memory), placing some computing units directly next to the DRAM array. However, this architecture also requires a trade-off between storage density and computing power. These in-memory computing systems have limited computing power and often require external logic computing chips. Instructions and scheduling also require meticulous design to ensure data matching.

[0005] In summary, current heterogeneous computing architectures for inference tasks are insufficient to meet the computational needs of neural networks with various parameter scales. Matrix computation efficiency and computing power utilization are low, and data migration costs are high, which greatly affects the future development of neural network inference and urgently needs to be addressed. Summary of the Invention

[0006] This application provides a flow-controlled in-memory computing architecture design method for neural network inference, in order to solve the problems that current heterogeneous computing architectures for inference tasks cannot meet the computing needs of neural networks with various parameter scales, have low matrix computing efficiency and computing power utilization, and have high data migration costs.

[0007] The first aspect of this application provides a flow-controlled in-memory computing architecture design method for neural network inference, comprising the following steps: obtaining the task type of the target inference task, and performing a corresponding weight matrix partitioning operation on a preset weight matrix according to the task type to obtain multiple weight matrix partitioning data, performing a dequantization operation on the multiple weight matrix partitioning data to obtain multiple dequantized weight matrix partitioning data, caching the multiple dequantized weight matrix partitioning data in a target weight cache SRAM, and using the target weight cache SRAM to send the multiple dequantized weight matrix partitioning data to the multiple target multiply-accumulate units; obtaining feature map data or variable matrix data of the target inference task, and... The feature map data or the variable matrix data is transferred to the target shift register to divide the feature map data or the variable matrix data into multiple rows of feature map data or multiple rows of variable matrix data. The target variable cache SRAM is used to obtain the corresponding multiple rows of feature map data or multiple rows of variable matrix data from the target shift register, and the multiple rows of feature map data or multiple rows of variable matrix data are sent to multiple target multiply-accumulate units. The multiple inverse quantization weight matrices in the multiple target multiply-accumulate units are used to perform multiply-accumulate calculations with the feature map data blocks or the variable matrix data blocks to obtain the matrix multiplication result corresponding to the target inference task.

[0008] Optionally, in one embodiment of this application, the step of obtaining the task type of the target inference task, performing a corresponding weight matrix partitioning operation on a preset weight matrix according to the task type to obtain multiple weight matrix partitioning data, performing dequantization operation on the multiple weight matrix partitioning data to obtain multiple dequantized weight matrix partitioning data, caching the multiple dequantized weight matrix partitioning data in a target weight cache SRAM, and sending the multiple dequantized weight matrix partitioning data to the multiple target multiply-accumulate units using the target weight cache SRAM includes: obtaining the weight parameters of a single output channel of the target convolutional neural network kernel size or the single column weight parameters of the weight matrix in the target large language model, and constructing the weight matrix according to the single column weight parameters or the weight parameters of the single output channel; determining the task type of the target inference task, and performing a basis function on the weight matrix according to the task type. The system performs output channel partitioning or column partitioning operations to obtain the multiple weight matrix partitioning data. Based on the preset quantization coefficients in the target RRAM computing array, the multiple weight matrix partitioning data is quantized and compressed to obtain quantized compressed data. The quantized compressed data is stored in the target RRAM storage array, and when reading the quantized compressed data in the target RRAM storage array, the quantized compressed data is multiplied and added with the preset inverse quantization coefficients in the target RRAM computing array to obtain the multiple inverse quantization weight matrix partitioning data. The number of units in the multiple target multiply-accumulate units is determined, and the target weight cache SRAM is divided into blocks according to the number of units to obtain multiple target weight cache blocks corresponding to the target weight cache SRAM. Each inverse quantization weight matrix partitioning data in the multiple inverse quantization weight matrix partitioning data is cached in the corresponding target weight cache block.

[0009] Optionally, in one embodiment of this application, the step of acquiring feature map data or variable matrix data of the target inference task and transferring the feature map data or variable matrix data to a target shift register to divide the feature map data or variable matrix data through the target shift register to obtain multi-row feature map data or multi-row variable matrix data includes: caching the feature map data or variable matrix data through the target shift register, so that the target shift register splits the feature map data or variable matrix data according to the row data volume of the target variable cache SRAM to obtain the multi-row feature map data or multi-row variable matrix data.

[0010] Optionally, in one embodiment of this application, the step of obtaining the corresponding multi-row feature map data or multi-row variable matrix data in the target shift register through the target variable cache SRAM, and sending the multi-row feature map data or multi-row variable matrix data to multiple target multiply-accumulate units includes: obtaining the convolution kernel size corresponding to the feature map data or the number of rows of the variable matrix corresponding to the variable matrix data, and dividing the target variable cache SRAM according to the convolution kernel size or the number of rows of the variable matrix to obtain multiple target variable cache blocks corresponding to the target variable cache SRAM; sequentially writing each row of feature map data or each row of variable matrix data into the corresponding target variable cache block of the target variable cache SRAM through the target shift register, and determining the target The system checks whether the remaining feature map data or variable matrix data in the shift register meets the row data volume requirement of the target variable cache SRAM. If the remaining feature map data or variable matrix data in the target shift register meets the row data volume requirement, then each row of feature map data or each row of variable matrix data is continuously written into the target variable cache SRAM. If the remaining feature map data or variable matrix data in the target shift register does not meet the row data volume requirement, then new feature map data or new variable matrix data corresponding to the target inference task is received again, and the new feature map data or new variable matrix data is concatenated with the remaining feature map data or variable matrix data to send the concatenated data to the target variable cache SRAM.

[0011] Optionally, in one embodiment of this application, the step of obtaining the corresponding multi-row feature map data or multi-row variable matrix data in the target shift register through the target variable cache SRAM, and sending the multi-row feature map data or multi-row variable matrix data to multiple target multiply-accumulate units, further includes: recording the amount of cached data corresponding to each target variable cache block in the target variable cache SRAM; determining whether the target variable cache SRAM meets a preset full storage requirement based on the amount of cached data, wherein if the target variable cache SRAM meets the preset full storage requirement, then controlling the target shift register to stop data read / write operations, and outputting the multi-row feature map data or multi-row variable matrix data to the multiple target multiply-accumulate units.

[0012] Optionally, in one embodiment of this application, after sending the plurality of inverse quantization weight matrix partition data to the plurality of target multiply-accumulate units using the target weight cache SRAM, the method further includes: obtaining the real-time inference requirements of the target inference task, and updating the plurality of inverse quantization weight matrix partition data cached in the target weight cache SRAM according to the real-time inference requirements, so as to reuse the plurality of inverse quantization weight matrix partition data for corresponding matrix multiplication operations.

[0013] A second aspect of this application provides a flow control matrix operation on-chip system for neural network inference, comprising: a main control processor, configured to receive basic information corresponding to a target inference task, configure each functional module of the flow control matrix operation on-chip system according to the basic information, and update the internal cache capacity of each functional module in real time to determine the thread number of the data stream; a transceiver module, configured to receive weight parameters corresponding to the target inference task and write the weight parameters into a preset RRAM storage system; the RRAM storage system, configured to quantize the weight parameters to generate and store quantized weight parameters, and perform dequantization on the quantized weight parameters to generate and output dequantized weight parameters; and a memory module, configured to receive initial feature map data corresponding to the target inference task, cache the results of each intermediate layer of the neural network corresponding to the target inference task, and send the initial feature map data and the results of each intermediate layer of the neural network to a preset memory when the received data volume meets a preset data sufficiency requirement. A matrix calculation module is configured to receive the initial feature map data and the results of each intermediate layer of the neural network according to the thread number of the data stream, and determine whether the matrix calculation module meets the preset data backpressure requirements. If the matrix calculation module meets the preset data backpressure requirements, the initial feature map data and the results of each intermediate layer of the neural network are backpressured into the memory module; otherwise, matrix calculation is performed using the quantized weight parameters, the initial feature map data, and the results of each intermediate layer of the neural network to obtain the matrix calculation result. A special function calculation module is configured to determine whether the special function calculation module meets the data backpressure requirements. If the special function calculation module meets the data backpressure requirements, the matrix calculation result is backpressured into the memory module; otherwise, a special function calculation is performed on the matrix calculation result using a preset special function to obtain the neural network inference result corresponding to the target inference task, and the neural network inference result is stored in the memory module.

[0014] Therefore, the embodiments of this application have the following beneficial effects:

[0015] The embodiments of this application can store more weight parameters by utilizing the high storage density of RRAM arrays and configure fewer in-memory computing resources for data preprocessing such as weight quantization and dequantization, which do not require very high precision. The weight parameters are moved from the in-memory computing system to the computing unit, where the computing unit temporarily stores the reused weights and receives feature map data or variable matrix data for multiplication and accumulation operations. The operations are directly controlled by the data flow, without the need for complex control instructions, ensuring data matching and integrity. This application supports various neural network inference requirements, improves computing power utilization and matrix computation efficiency, and reduces data movement costs. Therefore, it solves the problems of current heterogeneous computing architectures for inference tasks being unable to meet the computational needs of neural networks with various parameter scales, having low matrix computation efficiency and computing power utilization, and high data movement costs.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0018] Figure 1 An overall architecture diagram of a matrix operation module is provided for one embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a flow-controlled in-memory computing architecture design method for neural network inference, according to an embodiment of this application.

[0020] Figure 3 A diagram of an RRAM in-memory computing system architecture is provided for one embodiment of this application;

[0021] Figure 4 A schematic diagram of the data flow for matrix multiplication of large language model variables is provided as an embodiment of this application;

[0022] Figure 5 A schematic diagram of the data flow for a convolution kernel matrix multiplication operation is provided as an embodiment of this application;

[0023] Figure 6 This is an example diagram of a flow control matrix operation on-chip system for neural network inference according to an embodiment of this application; Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0025] The following describes a flow-controlled in-memory computing architecture design method for neural network inference according to embodiments of this application, with reference to the accompanying drawings. Addressing the problems mentioned in the background art, this application provides a flow-controlled in-memory computing architecture design method for neural network inference. In this method, feature map data or variable matrix data of the target inference task is acquired, and the feature map data or variable matrix data is transferred to a target shift register to divide the feature map data or variable matrix data into multiple rows of feature map data or multiple rows of variable matrix data. The corresponding multiple rows of feature map data or multiple rows of variable matrix data in the target shift register are obtained through a target variable cache SRAM, and the multiple rows of feature map data or multiple rows of variable matrix data are sent to multiple target... The multiply-accumulate unit performs column-wise partitioning on a preset weight matrix to obtain multiple weight matrix partition data. It then performs dequantization on these partition data to obtain multiple dequantized weight matrix partition data. This dequantized weight matrix partition data is cached in a target weight cache SRAM. The target weight cache SRAM is then used to send these dequantized weight matrix partition data to multiple target multiply-accumulate units. These units then perform multiply-accumulate calculations with feature map data blocks or variable matrix data blocks to obtain the matrix multiplication result corresponding to the target inference task. This flow-controlled matrix computation architecture effectively supports various neural network inference requirements, thereby increasing computational power utilization, reducing data migration costs, and improving matrix computation efficiency. This solves the problems of current heterogeneous computing architectures for inference tasks, which struggle to meet the computational needs of neural networks with various parameter scales, exhibiting low matrix computation efficiency and computational power utilization, and incurring significant data migration costs.

[0026] To facilitate those skilled in the art to understand the execution logic of the flow control in-memory computing architecture design method for neural network inference of this application, the following describes the architecture of the flow control matrix operation module involved in the flow control in-memory computing architecture design method for neural network inference of this application.

[0027] To perform matrix calculations for neural network inference tasks, such as Figure 1As shown, the entire computing unit architecture in this application includes an RRAM storage system responsible for inputting the weight parameter matrix, an SRAM (Static Random-Access Memory) for temporarily storing and reusing the weight matrix parameters, an SRAM for caching feature maps and variable matrices, and a computing unit responsible for matrix row and column multiplication and accumulation.

[0028] The flow control matrix operation module has the function of dividing the SRAM according to the configuration parameters and outputting data to the multiply-accumulate unit and weight parameters in blocks for operation. The weight parameters can be received from the RRAM storage system or directly input from the outside. The entire architecture is directly controlled by the data flow. When the data flow has not completed the operation or the data cache has no valid space, back pressure is applied to temporarily store the data flow in the previous level. Before performing matrix operations, the configuration register receives the operation scale information from the outside, such as the convolution kernel size and sliding window stride. The configuration register divides the SRAM into blocks according to the configuration information and records the starting address and valid data volume of each block. This information is updated in real time according to the inflow and outflow of data after the operation starts.

[0029] Therefore, this application utilizes the large storage density of the RRAM array to store more weight parameters and configures fewer in-memory computing resources for data preprocessing such as weight quantization and dequantization, which do not require high precision. The weight parameters are moved from the in-memory computing system to the computing unit, which temporarily stores the reused weights and receives feature map data or variable matrix data for multiplication and accumulation operations. The operation is directly controlled by the data flow and does not require complex control instructions, thus ensuring data matching and integrity.

[0030] Specifically, Figure 2 This is a flowchart illustrating a flow-controlled in-memory computing architecture design method for neural network inference, provided in an embodiment of this application.

[0031] like Figure 2 As shown, the flow-controlled in-memory computing architecture design method for neural network inference includes the following steps:

[0032] In step S201, the task type of the target inference task is obtained, and the preset weight matrix is ​​partitioned according to the task type to obtain multiple weight matrix partition data. The multiple weight matrix partition data is dequantized to obtain multiple dequantized weight matrix partition data. The multiple dequantized weight matrix partition data is cached in the target weight cache SRAM, and the multiple dequantized weight matrix partition data is sent to multiple target multiply-accumulate units using the target weight cache SRAM.

[0033] The embodiments of this application first perform different partitioning operations on the weight matrix according to different types of inference tasks, and then perform dequantization operations on the partitioned weight matrix to obtain multiple dequantized weight matrix partition data, which are then cached in the target weight cache SRAM, and the multiple dequantized weight matrix partition data are sent to multiple multiply-accumulate units.

[0034] Optionally, in one embodiment of this application, the task type of the target inference task is obtained, and a corresponding weight matrix partitioning operation is performed on a preset weight matrix according to the task type to obtain multiple weight matrix partitioning data. The multiple weight matrix partitioning data are then dequantized to obtain multiple dequantized weight matrix partitioning data. The multiple dequantized weight matrix partitioning data are cached in a target weight cache SRAM, and the multiple dequantized weight matrix partitioning data are sent to multiple target multiply-accumulate units using the target weight cache SRAM. This includes: obtaining the weight parameters of a single output channel of the target convolutional neural network kernel size or the single-column weight parameters of the weight matrix in the target large language model, and constructing a weight matrix based on the single-column weight parameters or the weight parameters of a single output channel; determining the task type of the target inference task, and performing a basis function on the weight matrix according to the task type. The system performs output channel partitioning or column partitioning operations to obtain multiple weight matrix partitioning data. Based on the preset quantization coefficients in the target RRAM computing array, the multiple weight matrix partitioning data are quantized and compressed to obtain quantized compressed data. The quantized compressed data is stored in the target RRAM storage array, and when reading the quantized compressed data in the target RRAM storage array, the quantized compressed data is multiplied and added with the preset inverse quantization coefficients in the target RRAM computing array to obtain multiple inverse quantized weight matrix partitioning data. The number of cells in the multiple target multiply-accumulate units is determined, and the target weight cache SRAM is divided into blocks according to the number of cells to obtain multiple target weight cache blocks corresponding to the target weight cache SRAM. Each inverse quantized weight matrix partitioning data in the multiple inverse quantized weight matrix partitioning data is cached into the corresponding target weight cache block.

[0035] It should be noted that the embodiments of this application can determine the task type of the target inference task, and perform output channel-based partitioning or column-based partitioning operations on the weight matrix according to the task type, thereby obtaining multiple weight matrix partitioning data; for example, when the task type of the target inference task is a convolutional neural network inference task, the weight matrix is ​​partitioned according to the output channel of the large language model neural network; when the task type of the target inference task is a large language model neural network inference task, the weight matrix is ​​partitioned by column to obtain the corresponding multiple weight matrix partitioning data.

[0036] In the specific implementation process, the RRAM memory system in this embodiment is responsible for storing weight parameters and performing quantization and dequantization processing, such as... Figure 3 As shown (where, Figure 3 The left side represents the simulation calculation. Figure 3 (The right side represents digital computation). Weight parameters (i.e., data divided by multiple weight matrices) are quantized and compressed (i.e., quantized compressed data) and stored in the RRAM storage array. Quantization and dequantization coefficients are stored in the RRAM of the RRAM computation array. The computation array in the RRAM in-memory system can adopt a digital or analog multiply-accumulate computation architecture. Quantization or dequantization parameters are pre-stored in the storage units of the computation array. When external weight parameters are input, the parameters are quantized to low bits by the computation array and then stored in the storage array. When weight parameters are read out, the quantized data and dequantization coefficients are multiplied and added by the computation array to complete dequantization before output. Because the computational cost of quantization and dequantization is much smaller than that of matrix operations, the RRAM in-memory system can complete quantization and dequantization calculations with only a small-scale computation array, and can have a larger-scale storage array to meet the storage requirements of larger models with a greater number of weight parameters.

[0037] The SRAM for weight parameters is divided into blocks according to the number of multiply-accumulate computation units. When performing matrix operations, the weight parameters are transferred to the RRAM computation array for dequantization before being output to the SRAM for weight parameters. Each SRAM block stores the weight parameters of a single output channel of the convolutional neural network kernel or a single column of the weight parameters in the weight matrix of a large language model, thereby facilitating the reuse of these weight parameters and reducing the data movement of weight parameters.

[0038] Optionally, in one embodiment of this application, after sending the multiple inverse quantization weight matrix partition data to multiple target multiply-accumulate units using the target weight cache SRAM, the method further includes: obtaining the real-time inference requirements of the target inference task, and updating the multiple inverse quantization weight matrix partition data cached in the target weight cache SRAM according to the real-time inference requirements, so as to reuse the multiple inverse quantization weight matrix partition data for corresponding matrix multiplication operations.

[0039] As one feasible approach for convolutional neural network inference tasks, the RRAM storage system in this embodiment divides parameters according to the output channels of the convolutional kernel calculation. The parameters required for the convolutional kernel of each output channel are sequentially dequantized row-by-row and written into the parameter cache SRAM for reuse. In this SRAM (i.e., the target weight cache SRAM), the parameters are arranged according to the order in which the feature map convolutional kernel data is sent to the multiply-accumulate unit. The multiply-accumulate unit sequentially reads the corresponding parameters based on the amount of valid feature map data obtained and performs multiplication and accumulation. After each calculation, the parameter SRAM address is sequentially incremented. This process continues until all data for a convolutional kernel has been transmitted and multiplied and accumulated with the corresponding parameters. Then, the multiply-accumulate unit outputs the convolution calculation result for a corresponding output channel.

[0040] Furthermore, for the inference task of a large language model neural network, in the embodiments of this application, the input variable matrix data will be transmitted row by row alternately, and the weight matrix will be divided by column, dequantized, and written into the SRAM corresponding to each multiply-accumulate unit, such as... Figure 4 As shown.

[0041] The weight matrix will be divided by column, and the weight cache SRAM corresponding to each multiply-accumulate unit will cache part of the weight parameters of the corresponding column of the weight matrix; the SRAM for caching the variable matrix will be divided into several blocks, each block corresponding to one row of the variable matrix; the variable matrix data will also be transferred to the shift register alternately according to the number of blocks in the SRAM.

[0042] In actual implementation, this embodiment assumes that the variable matrix cache SRAM is divided into n blocks, and each weight cache SRAM can cache m weights. The variable matrix uses the m-th column as the last row. After transmitting m columns, rows n+1 to 2n of the variable matrix are transmitted to maximize weight reuse and reduce data movement. The variable matrix cache SRAM also alternately sends data to the multiply-accumulate unit for each row. Calculation begins when there is valid data in the block corresponding to the current read address. After each batch of data is calculated, it is temporarily stored in the accumulate cache register. After one alternation cycle is completed, the multiply-accumulate unit changes the weights. This continues until the calculation of the last n rows and m columns of the variable matrix begins. After each alternation cycle, the weight cache unit updates the reused weights from the RRAM storage system. After completing the multiplication and accumulation calculation of the column weights and each row of the variable matrix, part of the calculation result is output, and the corresponding weight matrix column in the weight cache SRAM is replaced.

[0043] In step S202, feature map data or variable matrix data of the target inference task is obtained, and the feature map data or variable matrix data is transferred to the target shift register so as to divide the feature map data or variable matrix data through the target shift register to obtain multi-row feature map data or multi-row variable matrix data.

[0044] Furthermore, embodiments of this application can obtain relevant variable data for the target inference task, such as feature map data for the inference task of a convolutional neural network or variable matrix data for the inference task of a large language model neural network, and transfer the feature map data or variable matrix data to the target shift register, thereby using the shift register to divide the feature map data or variable matrix data to obtain multiple rows of feature map data or multiple rows of variable matrix data.

[0045] Optionally, in one embodiment of this application, obtaining feature map data or variable matrix data of the target inference task and transferring the feature map data or variable matrix data to a target shift register to divide the feature map data or variable matrix data through the target shift register to obtain multi-row feature map data or multi-row variable matrix data includes: caching the feature map data or variable matrix data through the target shift register so that the target shift register splits the feature map data or variable matrix data according to the row data volume of the target variable cache SRAM to obtain multi-row feature map data or multi-row variable matrix data.

[0046] It should be noted that, in the embodiments of this application, feature map or variable matrix data input from the outside can first be cached by a shift register. The shift register splits the data according to the number of rows of SRAM (i.e., target variable cache SRAM) to obtain multiple rows of feature map data or multiple rows of variable matrix data, thereby providing reliable data support and basis for the subsequent transmission and calculation of feature map data or variable matrix data.

[0047] In step S203, the corresponding multi-row feature map data or multi-row variable matrix data in the target shift register is obtained through the target variable cache SRAM, and the multi-row feature map data or multi-row variable matrix data is sent to multiple target multiply-accumulate units.

[0048] In step S204, multiple inverse quantization weight matrices in multiple target multiply-accumulate units are used to divide the data and feature map data blocks or variable matrix data blocks for multiply-accumulate calculations to obtain the matrix multiplication result corresponding to the target inference task.

[0049] Furthermore, embodiments of this application also require obtaining the corresponding multi-row feature map data or variable matrix data in the shift register through the target variable cache SRAM, and sending it to multiple target multiply-accumulate units, thereby providing data support for subsequent matrix operations.

[0050] Subsequently, embodiments of this application can utilize multiple inverse quantization weight matrices in multiple target multiply-accumulate units to divide data and feature map data blocks or variable matrix data blocks for multiply-accumulate calculations, thereby obtaining the matrix multiplication result corresponding to the target inference task.

[0051] Therefore, the flow control matrix operation module architecture of this application supports various neural network inference requirements, increases the utilization of computing power, reduces the consumption of data movement, and improves the efficiency of matrix calculation.

[0052] Optionally, in one embodiment of this application, obtaining the corresponding multi-row feature map data or multi-row variable matrix data in the target shift register through the target variable cache SRAM, and sending the multi-row feature map data or multi-row variable matrix data to multiple target multiply-accumulate units includes: obtaining the convolution kernel size corresponding to the feature map data or the number of rows of the variable matrix corresponding to the variable matrix data, and dividing the target variable cache SRAM according to the convolution kernel size or the number of rows of the variable matrix to obtain multiple target variable cache blocks corresponding to the target variable cache SRAM; sequentially writing each row of feature map data or each row of variable matrix data into the corresponding target variable cache block of the target variable cache SRAM through the target shift register, and determining the target shift. If the remaining feature map data or variable matrix data in the register meets the row data volume requirement of the target variable cache SRAM, then each row of feature map data or each row of variable matrix data is continuously written into the target variable cache SRAM. If the remaining feature map data or variable matrix data in the target shift register does not meet the row data volume requirement, then new feature map data or new variable matrix data corresponding to the target inference task is received again, and the new feature map data or new variable matrix data is concatenated with the remaining feature map data or variable matrix data to send the concatenated data to the target variable cache SRAM.

[0053] It should be noted that, in the embodiments of this application, the feature map or variable matrix data input from the outside is first buffered by a shift register. The shift register splits the data according to the SRAM row data size, writing one row of data (i.e., one row of feature map data or one row of variable matrix data) into the SRAM each time. When the remaining data is insufficient to fill a row, data reception will restart, and the insufficient part will continue to be temporarily stored until subsequent data arrives and is then concatenated.

[0054] Furthermore, for data at the end of a row, even if it is less than the amount of data in the row, this embodiment will write it into SRAM and record its address and the amount of valid data.

[0055] Optionally, in one embodiment of this application, the method of obtaining the corresponding multi-row feature map data or multi-row variable matrix data in the target shift register through the target variable cache SRAM and sending the multi-row feature map data or multi-row variable matrix data to multiple target multiply-accumulate units further includes: recording the amount of cached data corresponding to each target variable cache block in the target variable cache SRAM; determining whether the target variable cache SRAM meets the preset full storage requirement based on the amount of cached data, wherein if the target variable cache SRAM meets the preset full storage requirement, the target shift register is controlled to stop data read and write operations, and the multi-row feature map data or multi-row variable matrix data is output to multiple target multiply-accumulate units.

[0056] In the embodiments of this application, the SRAM is divided into blocks according to the rows of the variable matrix, with each row of data corresponding to a starting SRAM address, and the amount of data already cached in each row is recorded. When the amount of cached data is full (i.e., the target variable cache SRAM meets the preset full storage requirement), the shift register is controlled to stop writing and reading data. When there is enough data, data is output to the multiply-accumulate unit, and the calculation unit outputs the next data item after feeding back a calculation completion signal. When the shift register or the corresponding block of SRAM has no valid space, data is no longer received, and the data is pushed back into the previous level module.

[0057] As one possible approach, for inference tasks in convolutional neural networks, the input feature map data in this embodiment is transmitted row by row, with the convolution kernel as the granularity. The weight parameters are transmitted according to the output channel, transmitting the parameters required for one convolution kernel calculation to the SRAM corresponding to each multiply-accumulate unit. Figure 5 As shown.

[0058] The SRAM for caching feature map data (i.e., the target variable cache SRAM) is divided into corresponding blocks according to the size of the convolution kernel. The read and write addresses of the SRAM are calculated using the block pointer, the write pointer of each block, and the number of valid data rows in each block. Feature map data, corresponding to one row of the convolution kernel for each batch of data, is sent sequentially to the shift register from the first row to the last row. The shift register splits and concatenates the data and writes it to the address pointed to by the write pointer of the corresponding block in the SRAM, increasing the number of valid data rows in each block. When reaching the end of a row, any remaining data insufficient for a full SRAM row is padded with zeros before being written to the SRAM. Since the data size is the same for each row of the feature map, only the SRAM address where the last row's data is written is recorded, along with its valid data size. The SRAM read address is calculated using the write pointer and the valid data size. Based on the sliding window step size, the valid data size is reduced when reading the first few columns of data. When the valid data size in each SRAM block satisfies the requirements of a convolution kernel, an identifier is sent to the multiply-accumulate unit to begin calculation.

[0059] Those skilled in the art should understand that in current heterogeneous computing architectures for inference tasks, meeting the computational requirements of neural networks with various parameter scales has made storage density, data migration, and computational efficiency crucial for accelerating inference. Facing the future development of neural network inference, there is an urgent need for a matrix computing architecture with high storage density that reduces data migration and improves computational efficiency. The matrix computing architecture based on an RRAM in-memory computing system in this application fully utilizes the high storage density advantage of RRAM, distributing some data preprocessing computations to the RRAM in-memory computing system, temporarily storing reused parameters in matrix operations in the computing unit, and directly controlling computation through real-time monitoring of variable matrix data and data flow, thereby reducing data migration, improving computational efficiency, and effectively solving the bottleneck problem of future neural network inference.

[0060] The flow-controlled in-memory computing architecture design method for neural network inference proposed in this application utilizes the high storage density of RRAM arrays to store more weight parameters and allocates fewer in-memory computing resources for data preprocessing such as weight quantization and dequantization, which do not require very high precision. The weight parameters are moved from the in-memory computing system to the computing unit, where the computing unit temporarily stores the reused weights and receives feature map data or variable matrix data for multiplication and accumulation operations. The operations are directly controlled by the data flow, eliminating the need for complex control instructions and ensuring data matching and integrity. This application supports various neural network inference requirements, improves computing power utilization and matrix computation efficiency, and reduces the overhead of data movement.

[0061] Secondly, a flow control matrix operation on-chip system for neural network inference proposed according to an embodiment of this application will be described with reference to the accompanying drawings.

[0062] Figure 6 This is a block diagram of a flow control matrix operation on-chip system for neural network inference according to an embodiment of this application.

[0063] like Figure 6 As shown, the flow control matrix operation on-chip system 10 for neural network inference includes: a main control processor 100, a transceiver module 200, an RRAM storage system 300, a memory module 400, a matrix calculation module 500, and a special function calculation module 600.

[0064] The main control processor 100 is used to receive basic information corresponding to the target inference task, configure each functional module of the flow control matrix operation on-chip system according to the basic information, and update the internal cache capacity of each functional module in real time to determine the thread number of the data stream.

[0065] The transceiver module 200 is used to receive the weight parameters corresponding to the target inference task and write the weight parameters into the preset RRAM storage system 300.

[0066] The RRAM storage system 300 is used to quantize weight parameters to generate and store quantized weight parameters, and to perform dequantization operations on the quantized weight parameters to generate and output dequantized weight parameters.

[0067] The memory module 400 is used to receive the initial feature map data corresponding to the target inference task, cache the results of each intermediate layer of the neural network corresponding to the target inference task, and send the initial feature map data and the results of each intermediate layer of the neural network to the preset matrix calculation module 500 when the amount of received data meets the preset data sufficiency requirements.

[0068] The matrix calculation module 500 is used to receive the initial feature map data and the results of each intermediate layer of the neural network according to the thread number of the data stream, and to determine whether the matrix calculation module meets the preset data backpressure requirements. If the matrix calculation module 500 meets the preset data backpressure requirements, it backpressures the initial feature map data and the results of each intermediate layer of the neural network into the memory module 400. Otherwise, it performs matrix calculation by quantizing the weight parameters, the initial feature map data and the results of each intermediate layer of the neural network to obtain the matrix calculation result.

[0069] The special function calculation module 600 is used to determine whether the special function calculation module meets the data backpressure requirements. If the special function calculation module meets the data backpressure requirements, the matrix calculation result is backpressured into the memory module 400. Otherwise, the special function calculation is performed on the matrix calculation result using the preset special function to obtain the neural network inference result corresponding to the target inference task, and the neural network inference result is stored in the memory module 400.

[0070] Specifically, based on the flow control matrix operation module, embodiments of this application can construct a SoC (System on Chip) architecture oriented towards neural network inference, such as... Figure 6 As shown, the main control processor is responsible for receiving basic information about the neural network inference task and configuring other modules based on this information. In addition, the main control processor also manages the memory module's address and monitors the amount of valid data; weight parameter data is pre-written into the RRAM storage system via the transceiver module; the memory module receives the initial feature map data and caches the results of each intermediate layer of the neural network; and the special function calculation module is responsible for handling special calculations such as activation functions.

[0071] During neural network inference, in this embodiment, the main control processor first controls the transceiver module to receive pre-configured data and writes the received weight parameter data into the RRAM storage system for quantization and storage. Then, the memory module begins receiving and caching the data to be processed. The main control processor allocates corresponding threads to the data according to the inference stage and configures the matrix calculation module. After receiving sufficient data, the memory module begins sending data. The matrix calculation module and the special function calculation module receive and calculate according to the thread number of the data stream. If the calculation is incomplete or its internal cache is full, the data stream is pushed back into the memory module. Data calculated by the matrix calculation module can be directly sent to the special function calculation module for calculation or sent back to the memory module for caching. The special function calculation module can also send the result directly to the matrix calculation module or send it back to the memory module. The module determines whether to receive data based on the thread number attached to the data stream. The main control processor updates the internal cache capacity of each module in real time and schedules the data stream threads accordingly.

[0072] It should be noted that the foregoing explanation of the embodiment of the flow-controlled in-memory computing architecture design method for neural network inference also applies to the flow-controlled matrix operation on-chip system for neural network inference in this embodiment, and will not be repeated here.

[0073] The flow control matrix operation on-chip system proposed in this application for neural network inference includes a main control processor for receiving basic information corresponding to the target inference task, configuring each functional module of the flow control matrix operation on-chip system according to the basic information, and updating the internal cache capacity of each functional module in real time to determine the thread number of the data stream; a transceiver module for receiving weight parameters corresponding to the target inference task and writing the weight parameters into a preset RRAM storage system; the RRAM storage system for quantizing the weight parameters to obtain and store the quantized weight parameters, and performing dequantization on the quantized weight parameters to generate and output the dequantized weight parameters; a memory module for receiving the initial feature map data corresponding to the target inference task, caching the results of each intermediate layer of the neural network corresponding to the target inference task, and sending the initial feature map data and the results of each intermediate layer of the neural network to a preset matrix calculation module when the received data amount meets the preset data sufficiency requirement; and a matrix calculation module. The first module receives the initial feature map data and the results of each intermediate layer of the neural network according to the thread number of the data stream, and determines whether the matrix calculation module meets the preset data backpressure requirements. If the matrix calculation module meets the preset data backpressure requirements, it backpressures the initial feature map data and the results of each intermediate layer of the neural network into the memory module. Otherwise, it performs matrix calculations using quantized weight parameters, initial feature map data, and the results of each intermediate layer of the neural network to obtain the matrix calculation result. The second module, a special function calculation module, determines whether the special function calculation module meets the data backpressure requirements. If the special function calculation module meets the data backpressure requirements, it backpressures the matrix calculation result into the memory module. Otherwise, it uses preset special functions to perform special function calculations on the matrix calculation result to obtain the neural network inference result corresponding to the target inference task, and stores the neural network inference result into the memory module. This supports various neural network inference needs, increases the utilization of computing power, reduces the consumption of data movement, and improves the efficiency of matrix calculation.

[0074] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0075] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0076] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

Claims

1. A flow-controlled in-memory computing architecture design method for neural network inference, characterized in that, Includes the following steps: The task type of the target inference task is obtained, and a corresponding weight matrix partitioning operation is performed on the preset weight matrix according to the task type to obtain multiple weight matrix partitioning data. The multiple weight matrix partitioning data is then dequantized to obtain multiple dequantized weight matrix partitioning data. The multiple dequantized weight matrix partitioning data is cached in the target weight cache SRAM, and the multiple dequantized weight matrix partitioning data is sent to the multiple target multiply-accumulate units using the target weight cache SRAM. The feature map data or variable matrix data of the target inference task are obtained, and the feature map data or variable matrix data is transferred to the target shift register so as to divide the feature map data or variable matrix data through the target shift register to obtain multi-row feature map data or multi-row variable matrix data. The target variable cache SRAM is used to obtain the corresponding multi-row feature map data or multi-row variable matrix data in the target shift register, and the multi-row feature map data or multi-row variable matrix data is sent to multiple target multiply-accumulate units; The data is divided into multiple inverse quantization weight matrices in the multiple target multiplication and accumulation units and multiplied and accumulated with the feature map data block or the variable matrix data block to obtain the matrix multiplication result corresponding to the target inference task.

2. The method according to claim 1, characterized in that, The process of obtaining the task type of the target inference task, performing corresponding weight matrix partitioning operations on a preset weight matrix according to the task type to obtain multiple weight matrix partitioning data, performing dequantization operations on the multiple weight matrix partitioning data to obtain multiple dequantized weight matrix partitioning data, caching the multiple dequantized weight matrix partitioning data in the target weight cache SRAM, and sending the multiple dequantized weight matrix partitioning data to the multiple target multiply-accumulate units using the target weight cache SRAM includes: Obtain the weight parameters of a single output channel of the target convolutional neural network kernel size or the single column weight parameters of the weight matrix in the target large language model, and construct the weight matrix based on the single column weight parameters or the weight parameters of the single output channel; The task type of the target inference task is determined, and the weight matrix is ​​partitioned based on the output channel or by column according to the task type to obtain the multiple weight matrix partitioning data. Based on the preset quantization coefficients in the target RRAM computing array, the data divided by the multiple weight matrices is quantized and compressed to obtain quantized and compressed data; The quantized compressed data is stored in the target RRAM storage array, and when reading the quantized compressed data in the target RRAM storage array, the quantized compressed data is multiplied and added with the preset inverse quantization coefficients in the target RRAM computing array to obtain the multiple inverse quantization weight matrix partitioning data; The number of units in the plurality of target multiply-accumulate units is determined, and the target weight cache SRAM is divided into blocks according to the number of units to obtain a plurality of target weight cache blocks corresponding to the target weight cache SRAM; Each dequantized weight matrix partition data in the plurality of dequantized weight matrix partition data is cached to the corresponding target weight cache block.

3. The method according to claim 1, characterized in that, The step of acquiring feature map data or variable matrix data for the target inference task and transferring the feature map data or variable matrix data to a target shift register to divide the feature map data or variable matrix data through the target shift register to obtain multi-row feature map data or multi-row variable matrix data includes: The feature map data or the variable matrix data is cached by the target shift register, so that the target shift register splits the feature map data or the variable matrix data according to the row data volume of the target variable cache SRAM to obtain the multi-row feature map data or the multi-row variable matrix data.

4. The method according to claim 3, characterized in that, The step of obtaining the corresponding multi-row feature map data or multi-row variable matrix data from the target shift register through the target variable cache SRAM, and sending the multi-row feature map data or multi-row variable matrix data to multiple target multiply-accumulate units includes: Obtain the convolution kernel size corresponding to the feature map data or the number of rows of the variable matrix corresponding to the variable matrix data, and divide the target variable cache SRAM according to the convolution kernel size or the number of rows of the variable matrix to obtain multiple target variable cache blocks corresponding to the target variable cache SRAM; The target shift register sequentially writes each row of feature map data or each row of variable matrix data into the corresponding target variable cache block of the target variable cache SRAM, and determines whether the remaining feature map data or variable matrix data in the target shift register meets the row data volume requirement of the target variable cache SRAM. If the remaining feature map data or variable matrix data in the target shift register meets the row data volume requirement, then each row of feature map data or each row of variable matrix data is continuously written into the target variable cache SRAM; If the remaining feature map data or variable matrix data in the target shift register does not meet the row data volume requirement, then new feature map data or new variable matrix data corresponding to the target inference task is received again, and the new feature map data or new variable matrix data is concatenated with the remaining feature map data or variable matrix data to send the concatenated data to the target variable cache SRAM.

5. The method according to claim 4, characterized in that, The step of obtaining the corresponding multi-row feature map data or multi-row variable matrix data from the target shift register through the target variable cache SRAM, and sending the multi-row feature map data or multi-row variable matrix data to multiple target multiply-accumulate units, further includes: Record the amount of cached data corresponding to each target variable cache block in the target variable cache SRAM; Based on the amount of cached data, it is determined whether the target variable cache SRAM meets the preset full storage requirement. If the target variable cache SRAM meets the preset full storage requirement, the target shift register is controlled to stop data read and write operations, and the multi-row feature map data or the multi-row variable matrix data is output to the multiple target multiply-accumulate units.

6. The method according to claim 1, characterized in that, After using the target weight cache SRAM to divide the data of the multiple inverse quantized weight matrices and send it to the multiple target multiply-accumulate units, the method further includes: The real-time inference requirements of the target inference task are obtained, and the multiple dequantized weight matrix partitioning data cached in the target weight cache SRAM are updated according to the real-time inference requirements, so as to reuse the multiple dequantized weight matrix partitioning data for corresponding matrix multiplication operations.

7. A flow control matrix operation on-chip system for neural network inference, characterized in that, include: The main control processor is used to receive basic information corresponding to the target inference task, configure each functional module of the flow control matrix operation on-chip system according to the basic information, and update the internal cache capacity of each functional module in real time to determine the thread number of the data stream. The transceiver module is used to receive the weight parameters corresponding to the target inference task and write the weight parameters into a preset RRAM storage system; The RRAM storage system is used to quantize the weight parameters to generate and store quantized weight parameters, and to perform dequantization on the quantized weight parameters to generate and output dequantized weight parameters. The memory module is used to receive the initial feature map data corresponding to the target inference task, cache the results of each intermediate layer of the neural network corresponding to the target inference task, and send the initial feature map data and the results of each intermediate layer of the neural network to the preset matrix calculation module when the amount of received data meets the preset data sufficiency requirement. The matrix calculation module is used to receive the initial feature map data and the results of each intermediate layer of the neural network according to the thread number of the data stream, and to determine whether the matrix calculation module meets the preset data backpressure requirements. If the matrix calculation module meets the preset data backpressure requirements, the initial feature map data and the results of each intermediate layer of the neural network are backpressured into the memory module; otherwise, matrix calculation is performed using the quantized weight parameters, the initial feature map data and the results of each intermediate layer of the neural network to obtain the matrix calculation result. A special function calculation module is used to determine whether the special function calculation module meets the data backpressure requirement. If the special function calculation module meets the data backpressure requirement, the matrix calculation result is backpressured into the memory module. Otherwise, a preset special function is used to perform special function calculation on the matrix calculation result to obtain the neural network inference result corresponding to the target inference task, and the neural network inference result is stored in the memory module.

Images