Data processing method of neural network, neural network and chip

Abstract

The invention relates to a data processing method of a neural network, the neural network and a chip. The data processing method of the neural network comprises the following steps: analyzing a neural network model to an internal graph representation; identifying neural network calculation which needs to be optimized to PIM.PNM; optimizing an external interaction instruction of the PIM.PNM; and generating a calculation instruction of PIM * PNM. The application of PIM and PNM technologies in the neural network has the effects of reducing the data transmission bottleneck, improving the calculation efficiency, optimizing the storage and calculation parallelism, reducing the energy consumption, improving the memory bandwidth utilization rate and better supporting a large-scale model and real-time application.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a neural network data processing method, a neural network and a chip. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, deep learning models, especially neural network models represented by Transformer, have made significant progress in many fields such as natural language processing (NLP) and computer vision (CV). However, the computational complexity of deep learning models and their high demand for hardware resources pose challenges to traditional computing architectures.

[0003] Most of the current mainstream computing architectures are based on the von Neumann architecture, in which data processing and storage units are separated. This separation results in the need for data to be frequently transferred between memory and processors during the computing process, thus forming a "memory wall" bottleneck, greatly limiting computing speed and increasing energy consumption. This problem is particularly prominent in the training and reasoning of artificial intelligence models, especially large-scale neural network models.

[0004] However, most traditional program development and optimization methods are designed based on the von Neumann system. Therefore, specific optimization methods for these new architectures have gradually become the key to improving computing efficiency. Especially in neural network computing, how to layout and store weight data more efficiently has become the focus of the industry.

[0005] Taking the Transformer model as an example, its calculation process is highly dependent on a large number of parallel computing operations such as matrix multiplication and vector operations. In the traditional von Neumann system, these computing operations cannot fully realize their potential due to data transmission and bandwidth limitations.

[0006] Therefore, how to optimize the weight layout and storage of neural network models such as Transformer has become an important issue in improving AI computing performance, and it is necessary to improve the data processing methods of existing neural networks. Summary of the invention

[0007] In view of the above problems, the present invention provides a data processing method of a neural network, comprising the steps of:

[0008] Parse the neural network model into its internal graph representation;

[0009] Identify the neural network calculations that need to be optimized to PIM·PNM;

[0010] Optimize the external interaction instructions of PIM·PNM; and

[0011] Generates PIM·PNM calculation instructions.

[0012] The technical effects of applying PIM (Processing-In-Memory) and PNM (Processing-Near-Memory) technologies in neural networks are mainly reflected in the following aspects:

[0013] In the traditional von Neumann architecture, data needs to be frequently moved between the processor and memory, especially the weights and input features in large neural networks. This frequent data transfer will increase latency and consume a lot of energy. PIM and PNM avoid frequent data transfer between storage and computing units by performing calculations in or near memory, thereby reducing latency and power consumption.

[0014] The main computational task in neural networks is matrix multiplication (such as the multi-head attention mechanism of Transformer and the weight matrix calculation in feedforward neural networks). Under the PIM and PNM architectures, these matrix multiplication calculations can be processed directly in memory, thereby reducing the number of data movements. Through optimization strategies such as block matrix multiplication or column accumulation, the computation cache can be fully utilized to further improve computational efficiency.

[0015] The PIM and PNM architectures support parallel computing capabilities, transmitting input data in parallel to multiple computing units through multi-threaded or multi-channel processors. For the hierarchical structure of neural networks (such as the multi-layer stacking of Transformer), PIM / PNM can process the self-attention calculations of multiple levels or multiple heads in parallel at the same time, thereby accelerating the reasoning and training process of the entire model.

[0016] The PIM and PNM architectures are optimized for data movement, processing data directly in the storage unit, reducing the high energy consumption of data transmission in the von Neumann architecture. For large-scale neural network models, such as BERT or GPT, the model parameters are huge and the data movement is frequent. PIM and PNM significantly reduce computing energy consumption by reducing the frequency of memory access.

[0017] PIM and PNM make full use of memory bandwidth by placing calculations near or inside the memory. In traditional architectures, memory bandwidth often becomes a performance bottleneck, but under the PIM / PNM architecture, memory bandwidth can be used more efficiently, especially when performing matrix operations in neural networks, the bandwidth requirements are significantly optimized.

[0018] As the scale of deep learning models increases, especially the multi-layer structure and multi-head attention mechanism of the Transformer model, the PIM / PNM architecture can better handle the computational requirements of large-scale models. By keeping the weights and input features in memory as much as possible, the PIM / PNM architecture can cope with larger and larger models, especially when dealing with large amounts of data and long sequence inputs, its advantages are more significant.

[0019] In some real-time application scenarios, such as autonomous driving and intelligent monitoring, the reasoning speed of neural networks is crucial. PIM and PNM reduce data transmission and speed up calculations, making neural networks respond faster in real-time applications and meeting the requirements of low latency and high throughput.

[0020] In summary, the effects of applying PIM and PNM technologies in neural networks include reducing data transmission bottlenecks, improving computing efficiency, optimizing the parallelism of storage and computing, reducing energy consumption, improving memory bandwidth utilization, and better supporting large-scale models and real-time applications.

[0021] Optionally, the neural network is a Transformer network.

[0022] Optionally, the “external interaction instruction for optimizing PIM·PNM” includes:

[0023] Determine the matrix block size;

[0024] Dividing the matrix into sub-blocks; and

[0025] Block matrix computation.

[0026] Optionally, the determining of the matrix block size includes: processing matrices A and B using a block strategy, the block size B×B is set according to the memory capacity S, ensuring that the block data does not exceed the memory capacity, and the following conditions are met: 2B 2 ≤S.

[0027] Optionally, the block matrix calculation includes: performing calculation by block matrix multiplication, dividing the matrix A and the matrix B into sub-matrices, and calculating the result matrix C block by block according to the following formula, each sub-block of the matrix C: ij =∑ k A ik ×B kj , where A ik and B kj ) are sub-matrices of matrix A and matrix B respectively.

[0028] Optionally, the "computation instruction for generating PIM·PNM" includes: performing calculation by block matrix multiplication, dividing matrix A and matrix B into sub-matrices, and calculating block by block according to the following formula to obtain a result matrix C, each sub-block of matrix C: ij =∑ k A ik ×B kj , where A ik and B kj ) are sub-matrices of matrix A and matrix B respectively.

[0029] The blocking strategy can make better use of the cache by dividing a large matrix into multiple small matrices (sub-matrices) for operation. Since the cache capacity of modern processors is limited, and operations such as large-scale matrix multiplication often require a lot of memory bandwidth, directly processing the entire matrix may cause cache misses. The blocking strategy divides the matrix into blocks suitable for the cache size, so that each operation is concentrated in a smaller memory area, thereby reducing the cache miss rate and improving processing efficiency.

[0030] The block strategy can reduce the reliance on memory bandwidth during matrix operations. Matrix operations require frequent memory access, especially for large-scale matrix operations. By processing in blocks, each block can complete the calculation in a smaller memory area, thus reducing the need for frequent memory access. This can alleviate the memory bandwidth bottleneck and improve the throughput of matrix calculations.

[0031] The block matrix strategy makes operations such as matrix multiplication easier to parallelize. In a parallel computing environment, multiple processor cores can process different matrix blocks separately, thereby achieving efficient parallel computing. For modern deep learning models, especially large neural network models (such as Transformer), block processing can not only improve the performance of a single machine, but also give full play to the parallel capabilities of computing resources in multi-core, GPU or distributed systems.

[0032] Optionally, the "identifying neural network calculations that need to be optimized to PIM·PNM" includes: based on the size of the weight data, storing weight data smaller than a preset size in a calculation cache to avoid frequent data movement, and keeping weight data larger than a preset size in storage to optimize computing performance by reducing the number of transmissions.

[0033] Optionally, a column accumulation method is used to optimize matrix multiplication, and the column vectors of matrix A and matrix B are multiplied column by column and accumulated into a result matrix C to reduce the amount of data transmission and improve calculation efficiency.

[0034] In order to achieve the above-mentioned object of the invention, the present invention provides a neural network, and applies the data processing method of the neural network described above.

[0035] In order to achieve the above-mentioned object of the invention, the present invention provides a chip, which applies the data processing method of the neural network described above.

[0036] In summary, the data processing method of the neural network provided by the present invention, through the application of PIM and PNM technologies in the neural network, has the following effects: reducing data transmission bottlenecks, improving computing efficiency, optimizing the parallelism of storage and computing, reducing energy consumption, improving memory bandwidth utilization, and better supporting large-scale models and real-time applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the steps of the data processing method of the neural network provided by an embodiment of the present invention.

[0038] Figure 2 It is a schematic diagram of the steps of another neural network data processing method provided by an embodiment of the present invention.

[0039] Figure 3 It is a schematic diagram of the steps of the matrix calculation method provided by an embodiment of the present invention.

[0040] Figure 4 It is a schematic diagram of the steps of another matrix calculation method provided by an embodiment of the present invention.

[0041] Figure 5 It is a schematic diagram of the steps of another matrix calculation method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, this embodiment provides a data processing method of a neural network, comprising the steps of:

[0044] Parse the neural network model into its internal graph representation;

[0045] Identify the neural network calculations that need to be optimized to PIM·PNM;

[0046] Optimize the external interaction instructions of PIM·PNM; and

[0047] Generates PIM·PNM calculation instructions.

[0048] Among them, "Analysis of Neural Network Model to Internal Graph Representation", taking the Transformer model as an example, the internal graph representation of the Transformer model can be divided into multiple levels to show its core components and data flow. The following is a detailed graph representation of the Transformer model, including key parts such as attention mechanism (Attention), feed-forward neural network (Feed-Forward Network, FFN), residual connection (Residual Connections) and layer normalization (LayerNormalization).

[0049] 1. Input Layer

[0050] Input Embedding: The input of the Transformer model is usually serialized text data or other discrete data. The input data first passes through the embedding layer to convert discrete words or symbols into continuous vector representations. Each input vector can be represented as a node in the graph.

[0051] Positional Encoding: Since the Transformer model does not have a recursive structure, positional encoding is required to introduce the position information of the sequence. Positional encoding is usually added to the embedding vector, and the introduction of positional encoding can be represented by nodes in the graph.

[0052] 2. Multi-Head Self-Attention Mechanism

[0053] Self-Attention Mechanism: In Transformer, the self-attention mechanism calculates the relevance of each word in the sequence to other words through query, key and value.

[0054] Query, Key, Value Representation: In the graph, the input embedding is mapped into three different vectors: Q, K, and V, corresponding to the query, key, and value of each input word. They can be represented as three nodes in the graph, each of which is connected to the subsequent matrix operation.

[0055] Scaled Dot-Product Attention: In the self-attention mechanism, the dot product of the query and the key is first calculated to obtain the relevance matrix. This dot product process can be shown as a matrix multiplication node in the figure.

[0056] Softmax normalization: The dot product result is normalized by the Softmax function to obtain the attention weight. This operation can be represented as a Softmax node.

[0057] Weighted summation: The attention weights are multiplied by the value vector to obtain a weighted output. This process can be represented as a weighted sum node in the graph.

[0058] Multi-Head Attention: In the graph representation, each attention head can be represented as a parallel computation unit, and each head performs self-attention calculation independently. Finally, the outputs of all heads are concatenated and mapped back to the original dimension through linear transformation, which is represented as concatenation and mapping nodes in the graph.

[0059] 3. Residual Connections and Layer Normalization

[0060] Residual connection: In Transformer, the output of each layer is directly added to the input through a residual connection, forming a skip connection. This connection can be represented in the graph by a short-distance edge, showing the direct addition process of the input and output.

[0061] Layer Normalization: The result after residual connection is normalized by layer to stabilize the training. In the figure, it can be represented as a separate normalization node, connecting the residual and the input of the next layer.

[0062] 4. Feed-Forward Network (FFN)

[0063] Two-layer feedforward network: In each Transformer layer, in addition to the attention mechanism, there is a feedforward neural network consisting of two layers of linear transformations. Its calculation process can be represented in the figure as two layers of fully connected nodes, which usually includes the following steps:

[0064] Linear Transformation 1: The first layer of linear transformation maps the input to a higher dimension, usually using ReLU or GELU activation functions. It can be represented as a linear transformation node and an activation function node.

[0065] Linear Transformation 2: The second layer maps the high-dimensional representation back to the original dimension for connection to the next layer. This step can also be represented as a linear transformation node.

[0066] Residual connection and normalization: Similarly, the output of FFN also undergoes residual connection and layer normalization, and the process of these connections can be shown in the figure.

[0067] 5. Stacked Transformer Layers

[0068] Stacked attention and feedforward layers: In the graph representation, each Transformer layer contains a multi-head attention mechanism and a feedforward neural network, which are connected in series. These layers are stacked multiple times, usually 6 to 12 layers (such as BERT or GPT models). The input and output of each layer are connected through residuals, showing the dependency between layers.

[0069] Transfer between layers: The diagram can be stacked in multiple layers to show how data is transferred from one layer to the next.

[0070] 6. Output Layer

[0071] Classification or regression output: At the output of the Transformer model, there is usually an output layer that generates the final prediction results of the model (such as classification, generating text). The graph representation of the output layer can be a linear transformation node that maps the final embedding to a specific output dimension (such as vocabulary size).

[0072] Softmax output (for classification problems): For classification tasks, the output layer generally uses the Softmax function to normalize the result to obtain a probability distribution. It can be represented as a Softmax node in the graph.

[0073] 7. Decoder (applicable only to encoder-decoder model)

[0074] In an encoder-decoder Transformer (such as the original Transformer model for machine translation), the decoder is also represented in the diagram:

[0075] Decoder structure: Similar to the encoder, the decoder also has a self-attention mechanism and a feedforward network, but the decoder also contains an additional attention mechanism on the encoder output (encoder-decoder attention), which can be represented by additional connections in the diagram to represent the flow of information between the encoder and decoder.

[0076] 8. Positional Encoding Layer

[0077] Positional encoding representation: The Transformer model provides position information through positional encoding. Positional encoding can be used as a separate layer in the graph, usually by adding its result to the input embedding, and this operation can be represented by a node.

[0078] 9. Data Flow and Parallelization

[0079] Parallel data flow: The attention mechanism and feedforward neural network of the Transformer model can process input data in parallel, which can be represented by parallel nodes in the figure. Each input word can pass through multiple attention heads and feedforward neural networks at the same time, unlike RNN which depends on the previous and next time steps.

[0080] Data dependencies: Graph representations can show the dependencies between data in different layers of the Transformer and how inputs are processed in parallel through the attention mechanism of multiple heads.

[0081] By parsing the neural network model into an internal computational graph representation, computationally intensive, memory bandwidth-constrained, and difficult-to-parallelize computing nodes can be identified. These bottleneck operations are suitable for optimization through PIM (in-memory computing) or PNM (near-memory computing) technology, thereby improving the overall performance of the neural network.

[0082] Considering the computational characteristics of the Transformer model, the focus of its computation is mainly on the processing of weight matrices, including Q_weight (query weight), K_weight (key weight), V_weight (value weight) and the weight of the intermediate FFN (feedforward neural network) part. Optimizing the computational performance of the Transformer mainly lies in how to effectively process and accelerate the calculation of these weight matrices. Under the traditional von Neumann architecture, weights and input feature data need to be frequently transferred from memory to registers for calculation, which will result in a large amount of data transfer and repeated operations, thus creating a performance bottleneck.

[0083] In contrast, the computing methods of PIM (Processing in Memory) and PNM (Processing Near Memory) are completely different from traditional architectures. The core feature of PIM and PNM architectures is that they do not move data, but perform calculations directly where the data is stored. Therefore, for the compilation optimization of PIM and PNM architectures, it is necessary to focus on how to minimize data movement and reduce unnecessary data transmission as much as possible. At the same time, for data that needs to be accessed frequently, it should be stored in a local computing cache as much as possible to improve computing efficiency.

[0084] The basic idea of ​​the optimization method is to determine the storage and calculation method by comparing the size of the input features and weights. For smaller weight data, it can be stored in the local cache first to avoid frequent movement. For larger weight data, try to reduce the number of their movements and keep them in the memory for direct processing. The size of the weight data can be set based on specific needs to define the specific processing method of the weight data.

[0085] In a neural network, weights are the parameters that connect individual neurons and determine how input signals affect output results. The larger the weight, the greater the impact of the input signal on the output; the smaller the weight, the smaller the impact of the input signal on the output. The process of training a neural network is to repeatedly adjust these weights so that the network can learn specific patterns from the input data and make accurate predictions on new data. This process is usually achieved through the back-propagation algorithm, which continuously updates the weights to reduce the error between the predicted output and the actual result.

[0086] The PIM and PNM compilation optimization strategies provided in this embodiment mainly focus on the two aspects of data movement and calculation in the optimization of external interaction instructions. Figure 2 As shown, the data processing method of the neural network also includes, in the step of optimizing the external interaction instruction of PIM·PNM, performing matrix processing by data block method; and performing multiplication calculation on the block matrix. The specific optimization algorithm is as follows Figure 3 As shown, the steps include:

[0087] Determine the size of the matrix block: Assuming that the size of the calculation cache (such as SRAM) inside the PIM or PNM is S, the number of rows and columns of the external calculation matrix is ​​N, then the first step to optimize the data movement method is to determine the block size. Let the block size be B×B, where B is the dimension of the block. To ensure that the block can be completely stored in the SRAM, the amount of data in the block should not exceed the capacity of the SRAM. Assume that when performing matrix multiplication, two matrix blocks need to be stored at the same time, so B should meet the following conditions:

[0088] 2B 2 ≤S

[0089] The solution is:

[0090]

[0091] The appropriate block size can be determined by this formula.

[0092] Divide the matrix into sub-blocks: Use block matrix multiplication to perform calculations. Assuming the size of matrix (A) is (N×K) and the size of matrix (B) is (K×M), we divide them into small blocks of size (B×B) for calculation.

[0093] For matrix multiplication (C = B × B), we can divide (C) into

[0094]

[0095] sub-block.

[0096] Block matrix calculation: multiply each pair of corresponding sub-blocks and then accumulate them into the sub-blocks at the corresponding position of the result matrix.

[0097]

[0098] Among them (C ij ) is a block of (C), (A ik ) and (B kj ) are the blocks of (A) and (B) respectively. Optimization calculation formula:

[0099]

[0100] Each (C ij ) calculation requires the corresponding (A ik ) and (B kj ) are loaded into SRAM in blocks.

[0101] By dividing the matrix into blocks, the frequency of memory data transfers can be reduced and more calculations can be performed in SRAM.

[0102] In this implementation, there are two main matrix multiplication strategies:

[0103] like Figure 4 As shown, row-by-row calculation and accumulation: This method is a standard matrix multiplication method, that is, multiplying the row vector of matrix A by the column vector of matrix B. For each element Cij of the result matrix C, it is calculated by the dot product of the row of A and the column of B. Each time the calculation is performed, the corresponding rows and columns are loaded into SRAM in blocks, the operation is performed, and the results are stored.

[0104] like Figure 5 As shown, calculate and accumulate by column: Another optimization method is to perform matrix multiplication by column. This method first fixes one column of matrix B, calculates the product of each column in matrix A and the corresponding column of B in turn, and then accumulates these products to the corresponding position of the result matrix C. The specific steps are:

[0105] Extract the jth column of matrix B, denoted as vector b j ;

[0106] Operate on each column of A and extract the kth column a of A k , calculate a k and b j The product of is added to the j-th column of the result matrix C.

[0107] So far, the technical solution of the present invention has been described in conjunction with the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to the above-mentioned specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A data processing method for a neural network, characterized in that: Includes steps: Parse the neural network model into its internal graph representation; Identify the neural network calculations that need to be optimized to PIM·PNM; Optimize the external interaction instructions of PIM·PNM; and Generates PIM·PNM calculation instructions.

2. The data processing method of a neural network according to claim 1, characterized in that: The neural network is a Transformer network.

3. The data processing method of a neural network according to claim 2, characterized in that: The "external interaction instructions for optimizing PIM·PNM" include: Determine the matrix block size; Dividing the matrix into sub-blocks; and Block matrix computation.

4. The neural network data processing method according to claim 3, characterized in that: The determination of the matrix block size includes: using a block strategy to process matrices A and B, the block size B×B is set according to the memory capacity S, ensuring that the block data does not exceed the memory capacity and meets the following conditions: 2B 2 ≤S.

5. The method for processing data of a neural network according to claim 3, characterized in that: The block matrix calculation includes: performing calculation by block matrix multiplication, dividing matrix A and matrix B into sub-matrices, and calculating each sub-block of matrix C according to the following formula: ij =∑ k A ik ×B kj , where A ik and B kj ) are sub-matrices of matrix A and matrix B respectively.

6. The neural network data processing method according to claim 1, characterized in that: The "identifying the neural network calculations that need to be optimized to PIM·PNM" includes: based on the size of the weight data, storing the weight data smaller than a preset size in the calculation cache to avoid frequent data movement, and keeping the weight data larger than the preset size in the storage to optimize the calculation performance by reducing the number of transmissions.

7. The neural network data processing method according to claim 4, characterized in that: The column accumulation method is used to optimize matrix multiplication. The column vectors of matrix A and matrix B are multiplied column by column and accumulated into the result matrix C to reduce the amount of data transmission and improve the calculation efficiency.

8. A neural network, characterized in that A data processing method using a neural network as described in any one of claims 1 to 7.

9. A chip, characterized in that: A data processing method using a neural network as described in any one of claims 1 to 7.

Images