Neural network acceleration system and multi-dimensional data access method

Abstract

The invention discloses a neural network acceleration system, which comprises a direct memory access controller for receiving weight parameters from an external memory and inputting a feature map; the memory pool comprises a matrix storage device and a matrix transpose storage device, and the matrix storage device receives the weight parameters and inputs the feature map; the multi-dimensional weight caching module is used for caching weight data, the weight data are extracted from the weight parameters, and the weight data in different formats are provided according to calculation requirements; the matrix operation array is used for receiving the weight data and the input feature map and completing matrix operation according to the weight data and the input feature map to obtain a matrix output feature map; and the vector operation array is used for executing vector operations of an activation function, Softmax and LayerNorm, and is used for receiving the matrix output feature map and generating a vector output feature map. The neural network acceleration system disclosed by the invention can efficiently support neural networks such as a CNN and a Transform on a hardware level.

Description

Technical Field

[0001] This invention belongs to the field of neural network hardware accelerators, specifically relating to a neural network acceleration system and a multidimensional data access method. Background Technology

[0002] Deep learning technology has been widely applied in fields such as computer vision and natural language processing. As the complexity of deep learning models continues to increase, the demand for computing resources is also growing, making the development of dedicated hardware accelerators particularly important. The following technical challenges exist in the field of deep learning hardware acceleration: First, convolution calculations in CNN (Convolutional Neural Network) models can be converted into matrix multiplication. Transformer models contain numerous instances of normal matrix multiplication and transpose matrix multiplication. Transpose matrix multiplication can be calculated in forms such as Q·(K^T), where Q is the query matrix, K is the key matrix, and ^T represents the matrix transpose operation. Traditional deep learning accelerators often require explicitly transposing the weight matrix completely before computation, necessitating additional storage space and increasing data preprocessing time. Furthermore, achieving efficient resource reuse when normal and transpose matrix multiplications coexist is a significant challenge.

[0003] Second, there are significant differences in data access patterns among different algorithms. For example, matrix operations typically process data in parallel along the output channel dimension, while vector operations such as Softmax and LayerNorm require parallel processing along the width dimension. Existing multi-port storage arrays usually only support single-dimensional data access. When data is written in parallel along the row direction and then read along the column direction, since a column of data is located in the same storage unit, only a single data item can be obtained each time. Multiple accesses to the storage unit are required to complete the reading of a column of data, making it difficult to simultaneously meet the multi-dimensional parallel data access requirements in both the row and column directions.

[0004] The aforementioned problems severely limit the performance and efficiency of deep learning hardware accelerators. Therefore, there is an urgent need for a neural network acceleration system that can efficiently support ordinary / transpose matrix computation and multi-dimensional parallel data access at the hardware level. Summary of the Invention

[0005] To address the shortcomings and improvement needs of existing technologies, this invention provides a neural network acceleration system and a multidimensional data access method. The purpose is to develop a neural network acceleration system that efficiently supports ordinary / transpose matrix calculations and multidimensional parallel data access.

[0006] To achieve the above objectives, according to one aspect of the present invention, a neural network acceleration system is provided, comprising: Direct Memory Access Controller: Receives weight parameters and input feature maps from external memory. The input feature maps are the feature maps of the data to be processed. Memory pool: includes matrix storage device and matrix transpose storage device. The matrix storage device receives weight parameters and input feature maps. Multi-dimensional weight caching module: Connected to the memory pool, it is used to cache weight data, which is extracted from the weight parameters and provides weight data in different formats according to calculation needs; Matrix operation array: Connected to the multi-dimensional weight cache module and matrix storage device, it is used to receive weight data and input feature map, and perform matrix operations based on the weight data and input feature map to obtain matrix output feature map; Vector operation array: Connected to matrix operation array and matrix transpose storage device, the vector operation array is used to perform vector operations of activation function, Softmax, LayerNorm, and to receive matrix output feature map and generate vector output feature map. The matrix transpose storage device stores intermediate calculation results and output feature map. The output feature map includes matrix output feature map and vector output feature map.

[0007] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: The multi-dimensional weight caching module changes the direction of the weight vector covering the two-dimensional register array through address mapping algorithm and data loading mechanism. This not only enables the normal writing of weight data in the case of normal matrix multiplication, but also cleverly realizes the implicit transposition of weight data in the case of transposed matrix multiplication. No additional data preprocessing and storage space are required, which significantly improves the efficiency and resource utilization of the system when processing ordinary / transposed matrix multiplication.

[0008] Furthermore, through innovative circuit architecture design, storage units and arithmetic units are realized. As described, the matrix transpose storage device achieves dynamic switching between row-mode parallel reading and writing and column-mode parallel reading and writing of matrix data through a special cyclic matrix arrangement mechanism and flexible data routing strategy. It does not require additional data rearrangement cache and directly supports in-situ transpose access operations in random access memory, solving the efficiency problem of multi-dimensional data access for different algorithms.

[0009] Furthermore, through innovative circuit architecture design, storage units and arithmetic units are realized. For example, the neural network acceleration system of this invention can efficiently process various deep learning models, including convolutional neural networks and Transformers, through the collaborative work of the above-mentioned innovative components. At the same time, it supports the efficient execution of algorithms such as matrix multiplication, transpose matrix multiplication, convolution, depthwise convolution, as well as Softmax, LayerNorm, etc., which significantly improves the overall performance and resource utilization of the system. Attached Figure Description

[0010] Figure 1 The above is a schematic diagram of a neural network acceleration system provided according to an embodiment of the present invention; Figure 2 The above is a schematic diagram of the storage methods for 16-bit and 8-bit format data provided according to an embodiment of the present invention; Figure 3 The above is a structural schematic diagram of a multi-dimensional weighted cache module provided according to an embodiment of the present invention; Figure 4 The above is a schematic diagram of the data arrangement and loading order of a 16-bit / 8-bit precision weighted random access memory in normal mode according to an embodiment of the present invention; Figure 5 The above is a schematic diagram of a 16-bit / 8-bit precision two-dimensional register array write format in normal mode according to an embodiment of the present invention; Figure 6 The above is a schematic diagram of the data arrangement and loading order of a 16-bit precision weighted random access memory in transpose mode according to an embodiment of the present invention. Figure 7 The above is a schematic diagram of a 16-bit / 8-bit precision two-dimensional register array write format in transpose mode according to an embodiment of the present invention; Figure 8 The above is a schematic diagram of the data arrangement and loading order of an 8-bit precision weighted random access memory in transpose mode according to an embodiment of the present invention. Figure 9 The above is a schematic diagram of a 16-bit / 8-bit precision two-dimensional register array write format in deep convolution mode according to an embodiment of the present invention; Figure 10 The above is a schematic diagram illustrating the parallel dimensions and order of data access between matrix operations and Softmax / Norm operations according to an embodiment of the present invention; Figure 11 The above is a schematic diagram illustrating the data access dimensions between matrix operations and Softmax / Norm operations according to an embodiment of the present invention; Figure 12 The above is a schematic diagram of a special cyclic matrix arrangement implementation method provided according to an embodiment of the present invention; Figure 13 The above is a schematic diagram illustrating the specific data access scenarios for different rows and columns provided according to an embodiment of the present invention; Figure 14 The above is a schematic diagram of the structure of a matrix transpose storage device provided according to an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0012] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0013] Example 1: This invention discloses a neural network acceleration system, such as Figure 1 As shown, it includes: Direct Memory Access Controller: Receives weight parameters and input feature maps from external memory. The input feature maps are the feature maps of the data to be processed. Memory pool: includes matrix storage device and matrix transpose storage device. The matrix storage device receives weight parameters and input feature maps. Multi-dimensional weight caching module: Connected to the memory pool, it is used to cache weight data, which is extracted from the weight parameters and provides weight data in different formats according to calculation needs; Matrix operation array: Connected to the multi-dimensional weight cache module and matrix storage device, it is used to receive weight data and input feature map, and perform matrix operations based on the weight data and input feature map to obtain matrix output feature map; Vector operation array: Connected to the matrix operation array and the matrix transpose storage device, the vector operation array is used to perform vector operations such as activation function, Softmax, and LayerNorm, as well as to receive matrix output feature maps and generate vector output feature maps. The matrix transpose storage device stores intermediate calculation results and output feature maps, including matrix output feature maps and vector output feature maps. In embodiments of the present invention, the vector operation array supports basic operations such as addition, subtraction, multiplication, division, square root, exponentiation, logarithm, maximum value, accumulation, and mean, enabling efficient vector computation.

[0014] Furthermore, the neural network acceleration system also includes system control logic, which is used to control the operation of the neural network acceleration system, including data flow, instruction distribution, and providing timing logic.

[0015] Figure 2The above is a schematic diagram illustrating the storage methods for 16-bit and 8-bit format data according to an embodiment of the present invention. The input feature map, weight parameters, and output feature map are all stored using the NC1HWC0 format, where N represents the batch size, C represents the number of channels, H represents the height, W represents the width, C1 represents the number of channel blocks, and C0 represents the channel block size. For 16-bit data, C0 equals PE_SIZE, where PE_SIZE is the processing unit dimension, i.e., PE_SIZE = PE_ROW or PE_SIZE = PE_COL; for 8-bit data, C0 equals PE_SIZE × 2. For 16-bit format data, each storage unit stores one 16-bit data. The data is organized according to the NC1HWC0 format, where the channel dimension is divided into blocks of size C0 = PE_SIZE, and the data within each block is stored in parallel in PE_SIZE storage units. For 8-bit format data, each storage unit can store two 8-bit data. The data is also organized in the NC1HWC0 format, but the channel block size is C0=PE_SIZE×2. The data in each block is stored in parallel in PE_SIZE storage units, and each storage unit stores two consecutive 8-bit data.

[0016] In one embodiment of the present invention, the matrix storage device employs an array structure consisting of PE_SIZE random access memories, each 16 bits wide. For a width of... Height is The number of channels is For the three-dimensional tensor data, the coordinates of each vector data of size C0 are (cindex, windex, hindex), and the formula for calculating the corresponding storage address addr is:

[0017] in, Indicates the width of the block. Indicates the height of the block. The storage block coordinate index representing the channel direction. Indicates the coordinate index in the width direction. Indicates the coordinate index in the height direction.

[0018] Figure 3 The above is a schematic diagram of the structure of a multi-dimensional weighted cache module provided according to an embodiment of the present invention; specifically, the multi-dimensional weighted cache module includes a first two-dimensional register array, a second two-dimensional register array, a mode control module, an input demultiplexer, and an output multiplexer. The first two-dimensional register array and the second two-dimensional register array have identical structures, forming two independent buffers for parallel computation and data loading.

[0019] Both the first and second two-dimensional register arrays contain PE_ROW rows × PE_COL columns of register cells, where PE_ROW represents the parallelism in the row direction and PE_COL represents the parallelism in the column direction, and they satisfy the constraint that PE_ROW = 2 × PE_COL.

[0020] The input demultiplexer and output multiplexer are used to switch between data loading and output of the multi-dimensional weight caching module.

[0021] The mode control module supports three working modes: normal mode (M1), transpose mode (M2), and depthwise convolution mode (M3). The way weight data is loaded differs in different modes to adapt to different types of computing needs.

[0022] The mode control module contains four counters related to the loading of weight data in the normal matrix storage device. The first weight loading counter, denoted as cnt_ic_sel, serves as the basic weight loading counter, indicating the index of the currently loaded input channel. The second weight loading counter, denoted as cnt_rs_wei, indicates the index of the currently loaded weight window block. The third weight loading counter, denoted as cnt_ic_wei, indicates the index of the currently loaded input channel sub-block. The fourth weight loading counter, denoted as cnt_oc_wei, indicates the index of the currently loaded output channel sub-block.

[0023] Specifically, Figure 4 The diagram above illustrates the data arrangement and loading order of a 16-bit / 8-bit precision weighted random access memory in normal mode, according to an embodiment of the present invention. In normal mode, the three-dimensional weight tensor stored in the normal matrix storage device has the following dimensional characteristics: the width direction is the weight input channel, with a size of block_ic_wei; the height direction is the weight window, with a size of block_rs_wei; and the channel direction is the weight output channel, with a size of block_oc_wei. The formula for the size of the weight tensor in normal mode is as follows:

[0024] Since the parallelism PE_ROW and PE_COL are smaller than the block size of the 3D weight tensor, the width and channel directions of the 3D weight tensor can be further divided into multiple sub-blocks. The size of the sub-block in the input channel direction is PE_ROW, and the size in the output channel direction is PE_COL.

[0025] Figure 5The above is a schematic diagram of the 16-bit / 8-bit precision two-dimensional register array write format in normal mode according to an embodiment of the present invention; in normal mode, PE_COL is read from the normal matrix storage device in each cycle. 16-bit or PE_COL 2 The 8-bit weight vector is the data from multiple output channels of the three-dimensional weight tensor. Each cycle of the weight vector covers one row of the two-dimensional register array, and the entire two-dimensional register array is written over in PE_ROW cycles.

[0026] like Figure 4 As shown, the first weight loading counter cnt_ic_sel counts within the sub-block in the width direction, the second weight loading counter cnt_rs_wei counts along the height direction, the third weight loading counter cnt_ic_wei counts along the sub-block in the width direction, and the fourth weight loading counter cnt_oc_wei counts along the sub-block in the output channel direction. Therefore, the calculation formula for the weight vector coordinates (cindex, windex, hindex) in normal mode is as follows:

[0027] Figure 6 The diagram illustrates the data arrangement and loading order of a 16-bit precision weighted random access memory in transposed mode according to an embodiment of the present invention. In transposed mode, the dimensional characteristics of the three-dimensional weight tensor stored in the weighted random access memory change: the width direction is the weight output channel with a size of block_oc_wei, the height direction is the weight window with a size of block_rs_wei, and the channel direction is the weight input channel with a size of block_ic_wei. Compared with the normal mode, the width and channel are swapped in transposed mode, and the channel direction changes from an output channel to an input channel. In transposed mode, the block size formula is as follows:

[0028] like Figure 6 As shown, in the 16-bit precision transpose mode, the weight vector has a size of PE_COL in the input channel direction. Since PE_ROW = PE_COL × 2, the sub-block has two storage blocks in the input channel direction.

[0029] Figure 7The diagram illustrates the 16-bit / 8-bit precision two-dimensional register array write format in transpose mode according to an embodiment of the present invention. Unlike the normal mode, the weight vector loaded in transpose mode is data from multiple input channels, not output channels. The loaded weight vector remains PE_COL×16 bits, but the way it covers the two-dimensional register array has fundamentally changed: the weight vector first sequentially covers each column of the upper half of the two-dimensional register array, then covers each column of the lower half, completing the write of the entire array after PE_COL×2 cycles. This column-first loading method ensures that data from multiple input channels covers different rows of the array, and data from multiple output channels covers different columns of the array, thus cleverly achieving the transpose effect.

[0030] like Figure 6 As shown, the first weight loading counter, cnt_ic_sel, first counts within the first storage block of the sub-block in the width direction, and continues counting in the second storage block after reaching PE_COL. The second weight loading counter, cnt_rs_wei, counts along the height direction. The third weight loading counter, cnt_ic_wei, counts along twice the number of sub-blocks in the channel direction. The fourth weight loading counter, cnt_oc_wei, counts along the width direction of the sub-blocks. In 16-bit precision transpose mode, the formula for calculating the weight vector coordinates is as follows:

[0031] like Figure 7 As shown, in 8-bit precision transpose mode, the block size calculation formula is the same as in 16-bit precision, but there is a key difference: in 8-bit precision, the weighted random access memory (RRAM) has a block size of PE_COL × 2 in the input channel direction, while PE_ROW = PE_COL × 2. Therefore, the sub-block has only one block in the input channel direction, instead of two as in 16-bit precision. In 8-bit transpose mode, the loaded weight vector is PE_ROW × 8 bits in size. These vectors sequentially cover each column of the two-dimensional register array, in PE_COL... The entire two-dimensional register array is written in two cycles, achieving the same transpose effect. The formula for calculating the weight vector coordinates in 8-bit precision transpose mode is as follows:

[0032] Figure 9The diagram illustrates the write format of a 16-bit / 8-bit precision two-dimensional register array in deep convolution mode according to an embodiment of the present invention. In deep convolution mode, since each input channel only performs convolution operations with its corresponding output channel and is not connected to all output channels, a special data organization method is required. In 16-bit precision deep convolution mode, 16-bit data from PE_COL channels are written to the upper half of the first column of the register array; in 8-bit precision deep convolution mode, 8-bit data from PE_ROW channels are written to the first column of the register array. Other positions in the register array are marked as invalid in deep convolution mode. The multiplication results of deep convolution are not accumulated in the matrix operation array, but are directly output as multiplication results.

[0033] Figure 10 The above is a schematic diagram illustrating the parallel dimensions and sequence of data access between matrix operations and Softmax / Norm operations according to an embodiment of the present invention. In neural network processing, there are significant differences in data access patterns between different algorithms. For the output feature map random access memory, parallel computation of convolution and matrix multiplication requires parallel access to multiple output channel dimensions of the matrix, with a parallel dimension of PE_COL output channels. The time dimension is expanded sequentially along the width direction (cnt_ow), height direction (cnt_oh), and output channel block direction (cnt_oc). In the parallel computation of Softmax and LayerNorm in the Transformer model, operations such as finding the maximum value, accumulation, and mean need to be performed along the output channel dimensions of the output feature map, with a parallel dimension of PE_COL width dimensions. The time dimension is expanded sequentially along the output channel direction (cnt_oc), width block direction (cnt_ow), and height direction (cnt_oh).

[0034] Figure 11 The above is a schematic diagram illustrating the data access dimensions between matrix operations and Softmax / Norm operations according to an embodiment of the present invention. When Softmax / LayerNorm calculation is performed after a matrix operation, the result of the matrix operation is written to the output feature map random access memory in parallel according to the output channel. The Softmax / LayerNorm calculation requires parallel reading and writing to the same memory in the width direction. For a PE_COL×PE_COL matrix, its row dimension corresponds to the width direction, and its column direction corresponds to the output channel. That is, matrix operations require parallel access to a row of data in the matrix, while Softmax / LayerNorm vector operations require parallel access to a column of data in the matrix.

[0035] It should be noted that traditional multi-port storage array solutions only support single-dimensional data access. When data is written in parallel along the row direction and then read along the column direction, since a column of data resides in the same random access memory, only a single data item can be retrieved per access, resulting in low memory access bandwidth utilization. To solve this problem, this invention designs a matrix transpose storage device.

[0036] Figure 12 The diagram illustrates a special circular matrix arrangement method according to an embodiment of the present invention. To enable simultaneous reading and writing of both row and column data of the matrix, the present invention employs a special circular matrix arrangement method. For an n×n matrix, its arrangement pattern is as follows: Line 1: From arrive

[0037] Line 2: Put it at the very beginning, then... arrive

[0038] Line 3: and Put it at the very beginning, then... arrive

[0039] And so on, the elements in the i-th row are arranged as follows:

[0040] Figure 13 The above is a schematic diagram illustrating the specific data access scenarios for different rows and columns according to an embodiment of the present invention; when it is necessary to access the data in the i-th row of the matrix, the data in the i-th row can be read directly; when it is necessary to access the data in the i-th column of the matrix, it is necessary to start from the i-th element in the 0th row. Start by visiting the (ni-1)th row along the diagonal. Then, starting from the 0th element of the ni-th row, continue along the diagonal to the (n-1)-th row. This arrangement ensures that the elements of each row and each column of the matrix are distributed across different columns. Therefore, storing a column of data in the same random access memory unit allows for parallel access to rows or columns.

[0041] Figure 14 The above is a schematic diagram of the structure of a matrix transposed storage device provided according to an embodiment of the present invention. The matrix transposed storage device of the present invention includes: L random access memory units with independent read / write ports, a read / write address generation module, a read / write selection signal generation module, and a data routing network. Wherein L is a preset read / write parallelism, which is a power of 2 greater than 2.

[0042] The data routing network comprises six routing networks: write enable selection network, write address selection network, write data selection network, read enable selection network, read address selection network, and read data selection network. These networks control the routing of write enable, write address, write data, read enable, read address, and read data, respectively. Each routing network directs input data to a designated location based on its corresponding selection signal, enabling flexible data rearrangement.

[0043] The read / write address generation module generates L sets of address signals gen_addr based on the source address signal src_addr and the row / column pattern signal. This module first decomposes the source address signal src_addr, extracting the lower log2(L) bits as the offset address offset. The higher bits of src_addr are then padded with log2(L) zeros in the lower bits to form the base address base_addr. The specific calculation formula is as follows:

[0044] Where src_addr represents the source address signal, ADDR_WIDTH represents the address bit width, offset represents the offset address, base_addr represents the base address, L represents the read / write parallelism, [] represents the bit selection operation, {} represents the bit concatenation operation, and log2 represents the logarithm to the base 2.

[0045] The formula for calculating the address differs between row mode and column mode: Line mode:

[0046] Column mode:

[0047] Where gen_addr represents the generated L group of address signals; The read / write selection signal generation module is responsible for generating signals that control data routing, including write selection signals and read selection signals. The write selection signals include selection signals for the write data selection network, write address selection network, and write enable selection network; the read selection signals include selection signals for the read data selection network, read address selection network, and read enable selection network. The write selection signal generation algorithm extracts the lower log2(L) bits of the source write address as the offset address offset_w for the write operation, inverts it, adds 1 to obtain the base value base_w for the write operation, and then generates L consecutive selection signals:

[0048] Where offset_w is the offset address for the write operation, and base_w is the base value for the write operation. It is the selection signal for the write data selection network. It is the network selection signal for writing addresses. It is the selection signal that enables the network.

[0049] The read select signal generation algorithm extracts the lower log2(L) bits of the source read address as the offset address offset_r for the read operation, inverts it and adds 1 to obtain the base value base_r for the read operation, and then generates L consecutive select signals, distinguishing between address select signals and data select signals:

[0050] Where offset_r is the offset address for the read operation, and base_r is the base value for the read operation. It is the selection signal for the data selection network. It is the selection signal for reading the address to select the network. It is the selection signal for the read enable selection network.

[0051] Since the selection signal has a bit width of log2(L), these selection signals actually take the lower log2(L) bits, which are always in the range of 0 to (L-1).

[0052] Example 2: This invention discloses a method for multi-dimensional data access based on the neural network acceleration system in Embodiment 1 above, comprising the following steps: S1: Configure the working mode of the multi-dimensional weight cache module as normal mode, transpose mode or depth convolution mode according to the calculation requirements; S2: Based on the configured working mode, load weight data from the matrix storage device into the two-dimensional register array of the multi-dimensional weight cache module; S3: Configure the access mode of the matrix transpose storage device as row mode or column mode according to the computing requirements; S4: Based on the configured access mode, perform data read and write operations on the matrix transpose storage device to achieve multi-dimensional parallel access to matrix data.

[0053] Specifically, based on the above embodiment 2, the transpose matrix operation (Q) is performed during the Tranformer attention process. The workflow when a K^T task is followed by a Softmax operation task or a normal matrix operation is followed by a LayerNorm operation task can be broken down into the following steps: (1) The system control logic parses the task parameters, determines that the current task is matrix operation, and configures the multi-dimensional weight cache module to transpose or normal mode; (2) The direct memory access controller loads the data of the input feature matrix and weight matrix from external memory into the memory pool; (3) The multi-dimensional weight cache module reads weight matrix data from the normal matrix storage device and fills the two-dimensional register array; (4) The matrix operation array reads the input feature map data from the normal matrix storage device and performs multiplication and addition operations with the weight data; (5) The matrix transpose storage device is switched to row parallel access mode. The calculation results are written to the matrix transpose storage device after the activation function or other vector calculations are performed by the vector operation array. (6) The matrix transpose storage device switches to column parallel access mode, and the vector operation array reads the result data to perform Softmax or LayerNorm operation; (7) Write the result of Softmax or LayerNorm operation back to the matrix transpose storage device for subsequent operations.

[0054] (8) The direct memory access controller transfers the computation results from the matrix transpose storage device to the external memory.

[0055] In summary, this invention, through a multi-dimensional weight caching module using address mapping algorithms and data loading mechanisms, changes the direction in which the weight vector covers the two-dimensional register array. This not only enables the normal writing of weight data in normal matrix multiplication but also cleverly achieves implicit transposition of weight data in transposed matrix multiplication, eliminating the need for additional data preprocessing and storage space. This significantly improves the system's efficiency and resource utilization when processing ordinary / transposed matrix multiplication. Furthermore, through innovative circuit architecture design, the storage unit and arithmetic unit are implemented. As described, the matrix transposed storage device, through a special cyclic matrix arrangement mechanism and flexible data routing strategy, enables dynamic switching between row-mode parallel read / write and column-mode parallel read / write of matrix data. This eliminates the need for additional data rearrangement caching and directly supports in-situ transpose access operations within random access memory, solving the efficiency problem of multi-dimensional data access for different algorithms. Finally, through innovative circuit architecture design, the storage unit and arithmetic unit are realized. For example, the neural network acceleration system of this invention can efficiently process various deep learning models, including convolutional neural networks and Transformers, through the collaborative work of the above-mentioned innovative components. At the same time, it supports the efficient execution of algorithms such as matrix multiplication, transpose matrix multiplication, convolution, depthwise convolution, as well as Softmax, LayerNorm, etc., which significantly improves the overall system performance and resource utilization.

[0056] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A neural network acceleration system, characterized in that, include: Direct Memory Access Controller: Receives weight parameters and input feature maps from external memory. The input feature maps are the feature maps of the data to be processed. Memory pool: It includes a matrix storage device and a matrix transpose storage device. The matrix storage device receives weight parameters and input feature maps. Multi-dimensional weight caching module: Connected to the memory pool, it is used to cache weight data, which is extracted from the weight parameters and provides weight data in different formats according to the calculation requirements; Matrix operation array: Connected to the multi-dimensional weight cache module and matrix storage device, it is used to receive weight data and input feature map, and perform matrix operations based on the weight data and input feature map to obtain matrix output feature map; Vector operation array: Connected to matrix operation array and matrix transpose storage device, the vector operation array is used to perform vector operations of activation function, softmax and LayerNorm in convolution operation, as well as to receive matrix output feature map and generate vector output feature map. The matrix transpose storage device stores intermediate calculation results and output feature map. The output feature map includes matrix output feature map and vector output feature map.

2. The neural network acceleration system according to claim 1, characterized in that, It also includes system control logic, which controls the operation of the neural network acceleration system, including data flow, instruction distribution, and providing timing logic.

3. The neural network acceleration system according to claim 1, characterized in that, The multi-dimensional weight caching module also includes: The first two-dimensional register array and the second two-dimensional register array: contain PE_ROW rows × PE_COL columns of register cells, where PE_ROW represents the parallelism in the row direction and PE_COL represents the parallelism in the column direction, satisfying the constraint that PE_ROW = 2 × PE_COL. Mode control module: Used to configure the working mode of the multi-dimensional weight caching module, including normal mode, transpose mode and depthwise convolution mode; Input demultiplexer and output multiplexer: used to switch between data loading and output of the multi-dimensional weight caching module.

4. The neural network acceleration system according to claim 3, characterized in that, The mode control module includes four counters related to the loading of weight data in the matrix storage device: The first weight loading counter, which serves as the base weight loading counter, indicates the index of the currently loaded input channel. The second weight loading counter indicates the block index of the currently loaded weight window; The third weighted load counter indicates the currently loaded input channel subblock index; The fourth weighted load counter indicates the currently loaded output channel subblock index.

5. The neural network acceleration system according to claim 3, characterized in that, The normal mode includes: The weight vector read from the matrix storage device is data from multiple output channels. The weight vector covers one row of the two-dimensional register array, and the entire two-dimensional register array is written over in PE_ROW cycles. The formula for the size of the weight tensor in the matrix storage device is: The formula for calculating the weight vector coordinates (cindex, windex, hindex) in the normal mode is as follows: cindex = cnt_oc_wei windex = cnt_ic_wei × PE_ROW + cnt_ic_sel hindex = cnt_rs_wei The formula for calculating the storage address addr of the matrix storage device corresponding to the weight vector in the normal mode is: in, This represents the width of the weight tensor block. This represents the height of the weight tensor block. The values ​​represent the channel size of the weight tensor block: `block_ic_wei` is the size of the weight input channel in the width direction, `block_rs_wei` is the size of the weight window in the height direction, and `block_oc_wei` is the size of the weight output channel in the channel direction; `cindex` represents the storage block coordinate index in the channel direction, `windex` represents the coordinate index in the width direction, `hindex` represents the coordinate index in the height direction, `cnt_oc_wei` represents the value of the fourth weight load counter, indicating the index of the currently loaded output channel sub-block, `cnt_ic_wei` represents the value of the third weight load counter, indicating the index of the currently loaded input channel sub-block, `cnt_ic_sel` represents the value of the first weight load counter, indicating the index of the currently loaded input channel, and `cnt_rs_wei` represents the value of the second weight load counter, indicating the index of the currently loaded weight window block.

6. The neural network acceleration system according to claim 3, characterized in that, The transpose mode includes: The weight vector read from the matrix storage device includes data from multiple input channels; At 16-bit precision, the weight vector read from the matrix storage device is data from multiple input channels, with a size of PE_COL×16 bits. The weight vector sequentially covers each column of the upper half of the two-dimensional register array and each column of the lower half. After PE_COL×2 cycles, the entire array is written to cover it. At 8-bit precision, the weight vector read from the matrix storage device is data from multiple input channels, with a size of PE_ROW×8 bits. The weight vector sequentially covers each column of the two-dimensional register array, and the entire two-dimensional register array is written over in PE_COL×2 cycles. In the normal mode, the formula for the size of the weight tensor in the matrix storage device is: The formula for calculating the weight vector coordinates (cindex, windex, hindex) in the 16-bit precision transpose mode is as follows: cindex = cnt_ic_wei × (PE_ROW / PE_COL) + cnt_ic_sel / PE_COL windex = cnt_oc_wei × PE_COL + cnt_ic_sel % PE_COL hindex = cnt_rs_wei The formula for calculating the weight vector coordinates (cindex, windex, hindex) in the 8-bit precision transpose mode is as follows: cindex = cnt_ic_wei windex = cnt_oc_wei × PE_ROW + cnt_ic_sel hindex = cnt_rs_wei Wherein, cindex represents the storage block coordinate index in the channel direction, windex represents the coordinate index in the width direction, hindex represents the coordinate index in the height direction, cnt_oc_wei represents the value of the fourth weight loading counter, indicating the index of the currently loaded output channel sub-block, cnt_ic_wei represents the value of the third weight loading counter, indicating the index of the currently loaded input channel sub-block, cnt_ic_sel represents the value of the first weight loading counter, indicating the index of the currently loaded input channel, cnt_rs_wei represents the value of the second weight loading counter, indicating the index of the currently loaded weight window block, "%" represents modulo operation, and " / " represents integer division.

7. The neural network acceleration system according to claim 3, characterized in that, The deep convolution mode includes: writing 16-bit data from PE_COL channels to the upper half of the first column of the register array at 16-bit precision; writing 8-bit data from PE_ROW channels to the first column of the register array at 8-bit precision; and marking other positions in the register array as invalid in the deep convolution mode.

8. The neural network acceleration system according to claim 1, characterized in that, The matrix transpose storage device includes: L random access memory units with independent read / write ports, where L is the preset read / write parallelism, which is a power of 2 greater than 2; Read / write address generation module: Generates L sets of address signals gen_addr based on the source address signal src_addr and the row and column pattern signals; Read / Write Select Signal Generation Module: Used to generate signals that control data routing, including write select signals and read select signals; Data routing network: includes write enable selection network, write address selection network, write data selection network, read enable selection network, read address selection network, and read data selection network.

9. The neural network acceleration system according to claim 8, characterized in that, The read / write address generation module generates the address signal through the following steps: Decompose the source address src_addr, extract the lower log2(L) bits of src_addr as the offset address offset, and pad the higher bits of src_addr with log2(L) zeros in the lower bits to form the base address base_addr: offset = src_addr[log2(L)-1:0] base_addr = {src_addr[ADDR_WIDTH-1:log2(L)], {log2(L){1'b0}}} Where src_addr represents the source address signal, ADDR_WIDTH represents the address bit width, offset represents the offset address, base_addr represents the base address, L represents the read / write parallelism, [] represents the bit selection operation, {} represents the bit concatenation operation, and log2 represents the logarithm to the base 2. Different address signals are generated based on row and column patterns: Row pattern: gen_addr = base_addr + {offset, offset, ..., offset} Columnar pattern: gen_addr = base_addr + {0, 1, ..., L-1} Where gen_addr represents the generated L group of address signals; and The selection signal generation module generates a selection signal for controlling data routing through the following steps: Write selection signal generation: Extract the lower log2(L) bits of the source write address as the offset address offset_w, invert it and add 1 to obtain the base value base_w, and then generate L consecutive selection signals: base_w = ~offset_w + 1 wdata_sel = {base_w, base_w+1, ..., base_w+L-1} waddr_sel = wdata_sel wen_sel = wdata_sel Read selection signal generation: Extract the lower log2(L) bits of the source read address as the offset address offset_r, invert it and add 1 to obtain the base value base_r, and then generate the address selection signal and data selection signal respectively: base_r = ~offset_r + 1 raddr_sel = {base_r, base_r+1, ..., base_r+L-1} ren_sel = raddr_sel rdata_sel = {offset_r, offset_r+1, ..., offset_r+L-1} In this context, offset_w and offset_r represent the offset addresses for write and read operations, respectively; base_w and base_r represent the base values ​​for write and read operations, respectively; wdata_sel represents the write data selection signal; waddr_sel represents the write address selection signal; wen_sel represents the write enable selection signal; raddr_sel represents the read address selection signal; ren_sel represents the read enable selection signal; rdata_sel represents the read data selection signal; and ~ represents the bitwise inversion operation.

10. A method for implementing multi-dimensional data access using a neural network acceleration system according to any one of claims 1-9, characterized in that, Includes the following steps: S1: Configure the working mode of the multi-dimensional weight cache module as normal mode, transpose mode or depth convolution mode according to the calculation requirements; S2: Based on the configured working mode, load weight data from the matrix storage device into the two-dimensional register array of the multi-dimensional weight cache module; S3: Configure the access mode of the matrix transpose storage device as row mode or column mode according to the computing requirements; S4: Based on the configured access mode, perform data read and write operations on the matrix transpose storage device to achieve multi-dimensional parallel access to matrix data.

Images