Data processing method of neural network, neural network and chip

Abstract

The invention relates to a data processing method of a neural network, the neural network and a chip, the data processing method of the neural network comprises the following steps: a weight data storage step: loading weight data into a PIM; a weight data distribution step: distributing the weight data to the corresponding Bank; and a calculation step: performing Mac calculation based on the data of each Bank. The method comprises the following steps of: loading weight data to a process In Memory (PIM) unit, distributing the weight data to different Banks, and carrying out multiply-add (MAC) operation on the basis of data in each Bank. According to the method, data processing and storage are integrated, so that the transmission bottleneck of data from a memory to a processing unit is reduced, the calculation delay and power consumption are reduced, and particularly, the efficiency can be remarkably improved in large-scale neural network calculation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a neural network data processing method, a neural network and a chip. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, deep learning models, especially neural network models represented by Transformer, have made significant progress in many fields such as natural language processing (NLP) and computer vision (CV). However, the computational complexity of deep learning models and their high demand for hardware resources pose challenges to traditional computing architectures.

[0003] Most of the current mainstream computing architectures are based on the von Neumann architecture, in which data processing and storage units are separated. This separation results in the need for data to be frequently transferred between memory and processors during the computing process, thus forming a "memory wall" bottleneck, greatly limiting computing speed and increasing energy consumption. This problem is particularly prominent in the training and reasoning of artificial intelligence models, especially large-scale neural network models.

[0004] However, most traditional program development and optimization methods are designed based on the von Neumann system. Therefore, specific optimization methods for these new architectures have gradually become the key to improving computing efficiency. Especially in neural network computing, how to layout and store weight data more efficiently has become the focus of the industry.

[0005] Taking the Transformer model as an example, its calculation process is highly dependent on a large number of parallel computing operations such as matrix multiplication and vector operations. In the traditional von Neumann system, these computing operations cannot fully realize their potential due to data transmission and bandwidth limitations.

[0006] Therefore, how to optimize the weight layout and storage of neural network models such as Transformer has become an important issue in improving AI computing performance, and it is necessary to improve the data processing methods of existing neural networks. Summary of the invention

[0007] In view of the above problems, the present invention provides a data processing method of a neural network, comprising the steps of:

[0008] A weight data storage step, loading the weight data into the PIM;

[0009] A weight data allocation step, allocating the weight data to different banks; and

[0010] Calculation steps: Perform Mac calculation based on the data of each Bank.

[0011] In the present invention, weight data is loaded into the Processing In Memory (PIM) unit and distributed to different banks, and then multiplication and addition (MAC) operations are performed based on the data in each bank. This method reduces the bottleneck of data transmission from memory to processing unit by integrating data processing and storage, reduces computing delay and power consumption, and can significantly improve efficiency in large-scale neural network computing.

[0012] Optionally, the neural network is a Transformer network.

[0013] Since the Transformer network has a large number of matrix multiplication operations and a complex layer structure, it has high requirements for computing resources and bandwidth. Applying this technology to the Transformer network can optimize the calculation according to its characteristics and make full use of the parallel computing capabilities of the PIM architecture, thereby reducing the computing overhead, especially when calculating the weight matrix in the multi-head attention mechanism and feedforward network, which can greatly improve the computing speed and energy efficiency.

[0014] Optionally, the “distributing the weight data to different banks” includes:

[0015] The weight matrix in the weight data to be calculated is decomposed and allocated to different banks by columns or rows.

[0016] By decomposing the weight matrix by row or column and assigning it to different banks, this method can improve the parallelism of the calculation and make full use of the multi-bank design of the PIM architecture. The decomposed sub-matrices or sub-vectors correspond to different computing units and can be processed in parallel under the distributed computing framework, further improving the calculation speed and effectively reducing the pressure on memory bandwidth.

[0017] Optionally, the “decomposing the weight matrix in the weight data to be calculated” includes: dividing the matrix A into p sub-matrices: A=[A 1 ,A 2 ,…,A p ], where A_k is a sub-matrix of matrix A, dividing vector b into p sub-vectors: b = [b 1 ,b 2 ,…,b p ], where b k are sub-vectors of vector b, corresponding to the data in A_k respectively.

[0018] By decomposing the matrix A into p sub-matrices A k And decompose vector b into p subvectors b k, so that each sub-matrix and sub-vector can be calculated in parallel in different banks of PIM. This decomposition method effectively reduces the load of a single computing unit, optimizes the memory usage of large-scale matrix operations, maximizes parallel computing capabilities, and reduces data transmission time.

[0019] Optionally, the calculation step further includes: after completing the decomposition calculation of the matrix and the vector, completing the overall calculation of the Mac related to the bias.

[0020] In this application, after the matrix and vector decomposition calculations are performed, the Mac calculation of the bias term is added to ensure the calculation accuracy of the network. This step reduces the number of times the memory or processor needs to be accessed again by directly accumulating the bias term after the block calculation, further improving the consistency and efficiency of the calculation, while ensuring the integrity of the calculation results.

[0021] Optionally, the data processing method of the neural network includes:

[0022] Compute matrix subarray: c k =A k b k ;as well as

[0023] Calculate the matrix calculation result: accumulate the results of all calculation units:

[0024] In order to achieve the above-mentioned object of the invention, the present invention provides a neural network, and applies the data processing method of the neural network described above.

[0025] In order to achieve the above-mentioned object of the invention, the present invention provides a chip, which applies the data processing method of the neural network described above.

[0026] In summary, the data processing method of the neural network provided in this application improves computing efficiency: by storing weight data in PIM and combining the block strategy of matrices and vectors, the calculation is assigned to different banks for parallel execution, which greatly reduces the calculation time and improves the reasoning speed of the neural network.

[0027] Optimize memory bandwidth usage: By allocating and calculating weight data in PIM, data transmission between storage units and processing units is reduced, memory bandwidth bottlenecks are alleviated, and latency during data transmission is reduced.

[0028] Reduced power consumption: Memory-computing integration reduces the need for frequent data transfers and reduces the overall power consumption of the chip, making this method effective in resource-limited edge computing devices or mobile devices.

[0029] Enhanced parallel computing capability: The decomposed sub-matrices and sub-vectors are assigned to different banks for parallel computing, making full use of the multi-bank parallel computing advantages of the PIM architecture, greatly improving the processing speed of computing tasks, and is especially suitable for the computing needs of large-scale deep learning models.

[0030] Adapting to complex neural network models: This method is particularly suitable for computationally intensive neural networks, such as Transformer networks, and can significantly improve the computational performance in such complex networks, especially in multi-head attention and large-scale matrix multiplication operations.

[0031] Accuracy guarantee: During the calculation process, bias-related calculations are added to ensure the accuracy of the model calculation. At the same time, by accumulating the results of all block calculation units, the integrity and consistency of the final calculation output are guaranteed.

[0032] Hardware implementation optimization: Integrate this data processing method into chip design, fully utilize the hardware architecture advantages of PIM, and enable the chip to show higher computing efficiency, power efficiency and computing density when performing neural network tasks, which is suitable for application in special chips such as AI accelerators. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the steps of the data processing method of the neural network provided by an embodiment of the present invention.

[0034] Figure 2 It is a schematic diagram of the steps of another neural network data processing method provided by an embodiment of the present invention.

[0035] Figure 3 It is a schematic diagram of the steps of another neural network data processing method provided by an embodiment of the present invention.

[0036] Figure 4 It is a schematic diagram of the steps of another neural network data processing method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 As shown, this embodiment provides a data processing method of a neural network, comprising the steps of:

[0039] Weight data storage step, loading weight data into PIM;

[0040] A weight data allocation step, allocating the weight data to the corresponding Bank; and a calculation step, performing Mac calculation based on the data of each Bank.

[0041] Among them, PIM refers to Processing-In-Memory. PIM aims to integrate computing processing units (such as multiplication, addition and other operations) directly into memory chips, thereby reducing the bottleneck of data transmission between processors and memory. By performing calculations directly in memory, PIM can accelerate the training and reasoning process of neural networks, especially when processing large amounts of data, greatly improving performance and efficiency.

[0042] The weights of a neural network refer to the parameters that connect each neuron (or node) in the network. These parameters determine how the input signal affects the output signal. Each connection has a weight. The larger the weight, the greater the impact of the input signal on the output; the smaller the weight, the smaller the impact of the input signal on the output. It can be simply understood that the weight determines how the input data is transmitted through the network layer by layer and finally obtains the result.

[0043] In a neural network, the goal of training is to repeatedly adjust these weights so that the network can correctly learn specific patterns or rules from the input data and make accurate predictions on new data. This adjustment process is usually implemented through the backpropagation algorithm, which continuously updates the weights to reduce the error between the predicted output and the actual result.

[0044] In this implementation, the weight data is loaded into the Processing In Memory (PIM) unit and distributed to different banks, and then a multiplication-addition (MAC) operation is performed based on the data in each bank. This method reduces the bottleneck of data transmission from the memory to the processing unit by integrating data processing and storage, reduces computing latency and power consumption, and can significantly improve efficiency, especially in large-scale neural network computing.

[0045] A bank is an independent storage unit or block in DRAM. Each memory bank can be read and written independently. A DRAM chip is usually divided into multiple banks, which can be processed in parallel, allowing for more efficient memory access. Each bank consists of multiple storage cells arranged in rows and columns, with data stored at the intersections of these cells.

[0046] In PIM, each bank can be designed with processing capabilities, so that calculations can occur directly where the data is stored, reducing the delay of fetching data from the memory to the processor. This parallelization feature allows the PIM system to process data in multiple banks simultaneously, thereby improving system performance.

[0047] Preferably, the neural network mentioned in this embodiment is preferably a Transformer network. Since the Transformer network has a large number of matrix multiplication operations and a complex layer structure, it has high requirements for computing resources and bandwidth. Applying this technology to the Transformer network can optimize the calculation according to its characteristics and make full use of the parallel computing power of the PIM architecture, thereby reducing the computing overhead, especially in the calculation of weight matrices in multi-head attention mechanisms and feedforward networks, which can greatly improve the computing speed and energy efficiency.

[0048] Considering the characteristics of the current Transformer model, there are a lot of matrix calculations in an Attention or FFN processing stage. In this implementation, the calculation is optimized by completing the vector-matrix multiplication in parallel. Specifically, considering that the DDR data is provided as a basic IO read and write unit in the form of a bank, the matrix data must first be dispersed by column or row to different DDR banks to complete the data dispersion work, so as to complete the future parallel computing work. The basic data dispersion maximizes the p value according to the following formula, and p is preferably equal to the number of banks or the number of digits of the bank.

[0049] Alternatively, if Figure 2 As shown in the figure, “distributing weight data to different banks” includes:

[0050] Decompose the weight matrix in the weight data to be calculated and assign it to different banks by columns or rows.

[0051] By decomposing the weight matrix by row or column and assigning it to different banks, this method can improve the parallelism of the calculation and make full use of the multi-bank design of the PIM architecture. The decomposed sub-matrices or sub-vectors correspond to different computing units and can be processed in parallel under the distributed computing framework, further improving the calculation speed and effectively reducing the pressure on memory bandwidth.

[0052] Alternatively, if Figure 3 As shown, in this embodiment, "decomposing the weight matrix in the weight data to be calculated" includes: dividing the matrix A into p sub-matrices:

[0053] A=[A 1 ,A 2 ,…,Ap ]

[0054] Where A_k is an m×n / p submatrix.

[0055] Decompose vector: Divide vector b into p subvectors:

[0056] b=[b 1 ,b 2 ,…,b p ]

[0057] where b k is a subvector of n / p.

[0058] Alternatively, if Figure 4 As shown, in this embodiment, the calculation step also includes: after completing the decomposition calculation of the matrix and the vector, completing the overall calculation of the MAC related to the bias, calculating the sub-array: each calculation unit k calculates A k b k :

[0059] c k =A k b k

[0060] Calculate the matrix calculation result: accumulate the results of all calculation units:

[0061]

[0062] Based on the above steps, efficient Transformer calculation based on PIM·PNM can be achieved.

[0063] By decomposing the matrix A into p sub-matrices A k And decompose vector b into p subvectors b k , so that each sub-matrix and sub-vector can be calculated in parallel in different banks of PIM. This decomposition method effectively reduces the load of a single computing unit, optimizes the memory usage of large-scale matrix operations, maximizes parallel computing capabilities, and reduces data transmission time.

[0064] After the weights and data are assigned to the corresponding banks, PIM starts to perform MAC (Multiply-Accumulate) operations.

[0065] MAC is a core computational task in neural networks, especially in matrix operations such as vector-matrix multiplication (Attention and FFN processing stages in Transformer).

[0066] By decomposing matrices and vectors into multiple sub-matrices and sub-vectors and executing them in parallel on each Bank, PIM can significantly increase the computing speed to achieve improved efficiency and reduced energy consumption.

[0067] By loading weight data into PIM and distributing and parallelizing banks, the reasoning and training process of neural networks (especially Transformer models) can be significantly accelerated. This method can reduce the bottleneck of data transmission and improve overall energy efficiency, and is particularly suitable for processing large-scale matrix calculation tasks.

[0068] In this implementation, after performing the decomposition calculation of the matrix and vector, the Mac calculation of the bias term is added to ensure the calculation accuracy of the network. This step reduces the number of times the memory or processor needs to be accessed again by directly accumulating the bias term after the block calculation, further improving the consistency and efficiency of the calculation, while ensuring the integrity of the calculation results.

[0069] Optionally, this embodiment provides a neural network, applying the data processing method of the neural network described above.

[0070] PIM calculation and accumulation techniques applied to neural networks can significantly optimize the computing performance of neural networks. Especially in large-scale deep learning tasks, it can effectively reduce memory bandwidth usage, reduce data transmission time, and improve energy efficiency. It is suitable for deep network models that require a large number of matrix operations, such as Transformer.

[0071] Optionally, this embodiment provides a chip that applies the neural network data processing method described above.

[0072] Applying the above neural network data processing method to chip design can fully utilize the architectural advantages of PIM and improve the performance of the chip when performing neural network calculations. By implementing matrix decomposition and parallel computing at the hardware level, the chip's computing power can be greatly optimized, power consumption can be reduced, and computing efficiency can be improved, making it suitable for neural network inference chips or accelerator design.

[0073] So far, the technical solution of the present invention has been described in conjunction with the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to the above-mentioned specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A data processing method for a neural network, characterized in that: Includes steps: A weight data storage step, loading the weight data into the PIM; A weight data allocation step, allocating the weight data to different banks; as well as Calculation steps: Perform Mac calculation based on the data of each Bank.

2. The data processing method of a neural network according to claim 1, characterized in that: The neural network is a Transformer network.

3. The data processing method of a neural network according to claim 2, characterized in that: The “distributing the weight data to different banks” includes: The weight matrix in the weight data to be calculated is decomposed and allocated to different banks by columns or rows.

4. The neural network data processing method according to claim 3, characterized in that: The "decomposing the weight matrix in the weight data to be calculated" includes: dividing the matrix A into p sub-matrices: A = [A1, A2, ..., A p ], where A_k is a sub-matrix of matrix A, dividing vector b into p sub-vectors: b = [b1, b2, ..., b p ], where b k are sub-vectors of vector b, corresponding to the data in A_k respectively.

5. The neural network data processing method according to claim 4, characterized in that: The calculation step also includes: after completing the decomposition calculation of the matrix and the vector, completing the overall calculation of the Mac related to the bias.

6. The method for processing data of a neural network according to claim 5, characterized in that: include: Compute matrix subarray: c k =A k b k ; as well as Calculate the matrix calculation result: accumulate the results of all calculation units:

7. A neural network, characterized in that A data processing method using a neural network as described in any one of claims 1 to 6.

8. A chip, characterized in that: A data processing method using a neural network as described in any one of claims 1 to 6.

Images