A multi-NPU cascade structure to accelerate neural network computing

Abstract

This disclosure discloses a multi-NPU cascade structure for accelerating neural network computations. The structure comprises multiple neural network computing units (NPUs), multiple block memories, a direct memory access (DMA), and a main memory (DDR). This cascade structure enables the multiple NPUs to perform computations in parallel, with only the input and output of neural network building blocks requiring access to the main memory. The disclosed cascade structure significantly improves the utilization of neural network MACs and allows for efficient access to main memory, significantly addressing the performance bottleneck of low memory access efficiency in neural network computations.

Description

Technical Field

[0001] The present disclosure belongs to the field of deep learning technology, and in particular relates to a multi-NPU cascade structure for accelerating neural network computing. Background Art

[0002] Memory access cost (MAC) is a key metric when measuring the computational speed of neural networks. Neural network operations (such as convolution and pooling) require extensive accesses to main memory, with the data often stored as vectors or tensors. This results in a significant amount of time spent accessing main memory when processing these operations in current mainstream computing architectures (such as existing CPUs, GPUs, NPUs, or any other processor architecture). However, in neural network computations, data between layers is closely linked (the output of the upper layer serves as the input for the lower layer). Accessing main memory once for just one layer is inherently inefficient. Existing computing architectures lack the flexible and efficient data access control capabilities required for these computations, resulting in inefficient memory usage and severe performance bottlenecks.

[0003] The Bottleneck Residual Block neural network building block introduced by ResNet has a simple network structure and a small number of parameters, yet possesses powerful representation capabilities. Therefore, it has been widely used in many lightweight network structures. Accelerating the computation of this module has significant application value. Summary of the Invention

[0004] In view of this, the present disclosure provides a multi-NPU cascade structure for accelerating neural network computing, including multiple neural network computing units NPU, multiple block memories, a direct memory access DMA and a main memory DDR. The cascade structure enables the multiple neural network computing units NPU to calculate in parallel, and only the input and output of the neural network building module need to access the main memory.

[0005] Preferably,

[0006] This cascade structure is used to accelerate the Bottleneck Residual Block neural network building module.

[0007] Preferably,

[0008] The Bottleneck Residual Block structure includes two branches, one side is a three-layer convolution calculation, and the other side is a direct shortcut. The three-layer convolution calculation includes a 1*1 Conv layer, a 3*3 DWConv layer, and a 1*1 Conv layer.

[0009] Preferably,

[0010] The cascade structure includes three neural network computing units NPU, three block memories, a direct memory access DMA and a main memory DDR.

[0011] Preferably,

[0012] The three neural network computing units NPU are NPU0, NPU1 and NPU2, wherein NPU0 is used to calculate the first layer Conv on the right, NPU1 is used to calculate the second layer DWConv on the right, and NPU2 is used to calculate the third layer Conv on the right and add the calculation result of the third layer on the right to the input.

[0013] Preferably,

[0014] The three block memories are fm0 for storing the input feature map of the Bottleneck Residual Block neural network building module, fm1 for storing the result of the first layer Conv on the right and the result of the second layer DWConv on the right, and fm2 for storing the result of adding the third layer Conv on the right to the input feature map.

[0015] Preferably,

[0016] The DMA transfers data between the block memory and the main memory.

[0017] Preferably,

[0018] The DDR stores parameters required by the neural network.

[0019] Preferably,

[0020] In this cascade structure: the multiple neural network computing units NPU are used for parallel computing. Except for the input and output of the neural network building module, which need to access the main memory, no other modules or structures in the cascade structure need to access the main memory.

[0021] In addition, the present disclosure also discloses a method for accelerating computing using a multi-NPU cascade structure, which includes the following steps:

[0022] S100: The processor loads the feature map input by the Bottleneck Residual Block neural network building module from the main memory DDR into fm0 via DMA;

[0023] S200: NPU0 reads the input feature map from fm0 and calculates the first layer Conv on the right, and stores the calculation result in fm1;

[0024] S300: NPU1 reads the calculation result of the first layer on the right side from fm1, calculates the second layer DWConv on the right side, and stores the calculation result in fm1;

[0025] S400: NPU2 reads the calculation result of the second layer on the right from fm1 and calculates the third layer Conv on the right. After the calculation is completed, it reads the feature map input by the Bottleneck Residual Block neural network building module from fm0 and adds it to the calculation result of the third layer on the right. The result of the addition is stored in fm2 as the module calculation result.

[0026] S500: The module calculation result in fm2 is stored in the main memory DDR via DMA.

[0027] Through the above-mentioned technical solution, the disclosed cascade structure can significantly improve the utilization rate of the neural network MAC, efficiently accessing main memory, which is of great significance for solving the performance bottleneck of low memory access efficiency in neural network computing. This is because: in this cascade structure, although the multiple neural network computing units (NPUs) are used for parallel computing, except for the input and output of the neural network building module, no other modules or structures in the cascade structure need to access main memory, so the use of main memory is greatly reduced compared to the existing technology.

[0028] At the same time, due to the widespread use of Bottleneck Residual Block structure in neural networks, applying this cascade structure to Bottleneck Residual Block has very important application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a diagram of a multi-NPU cascade structure for accelerating neural network computing in one embodiment of the present disclosure;

[0030] Figure 2 is a diagram of the Bottleneck Residual Block structure in one embodiment of the present disclosure;

[0031] Figure 3 This is a schematic diagram of the inter-layer cascade of a multi-NPU structure applied to a Bottleneck Residual Block structure in one embodiment of the present disclosure;

[0032] Figure 4This is a structural diagram of a basic convolutional neural network module in one embodiment of the present disclosure;

[0033] Figure 5 This is a schematic diagram of the inter-layer cascade of a multi-NPU structure applied to a basic convolutional neural network module structure in one embodiment of the present disclosure;

[0034] Figure 6 This is a schematic diagram of the inter-layer cascade of a multi-NPU structure applied to an extended cluster structure in one embodiment of the present disclosure;

[0035] Figure 7 This is a diagram of the ShuffleNet v2 network structure in one embodiment of the present disclosure;

[0036] Figure 8 This is a schematic diagram of the inter-layer cascade of a multi-NPU structure applied to the ShuffleNet v2 network structure in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] In one embodiment, Figure 1 As shown, it discloses a multi-NPU cascade structure for accelerating neural network computing, including multiple neural network computing units NPU, multiple block memories, a direct memory access DMA and a main memory DDR. The cascade structure enables the multiple neural network computing units NPU to calculate in parallel, and only the input and output of the neural network building module need to access the main memory.

[0038] In this embodiment, the multi-NPU cascade design enables parallel NPU computation, effectively increasing the computational speed of the neural network architecture. Furthermore, because only the input and output of this module require access to main memory, this design significantly improves the utilization of the neural network's MACs, enabling efficient access to main memory.

[0039] In another embodiment, the cascade structure is used to accelerate the Bottleneck Residual Block neural network building module.

[0040] In this embodiment, a multi-NPU cascade design to accelerate the Bottleneck Residual Block effectively utilizes multiple NPUs within the processor and partitions block memory, enabling parallel computation and storage of data between Bottleneck Residual Block layers. This allows the Bottleneck Residual Block to be processed within the processor using multiple NPUs, simply by reading module input data from main memory.

[0041] The multi-NPU cascade design can be used to accelerate the Bottleneck Residual Block neural network building module. This can significantly increase the module's computational speed, improve neural network MAC utilization, and efficiently access main memory.

[0042] In another embodiment, Figure 2 As shown, the Bottleneck Residual Block structure includes two branches, one side is a three-layer convolution calculation, and the other side is a direct shortcut. The three-layer convolution calculation includes a 1*1 Conv layer, a 3*3 DWConv layer, and a 1*1 Conv layer.

[0043] In this embodiment, the architecture has two branches: a three-layer convolutional computation on one side and a shortcut on the other. The three layers of computation in the right branch are referred to as RA, RB, and RC, respectively. The result of the RA computation is referred to as RAFM, ​​the result of the RB computation is referred to as RBFM, and the result of the RC computation is referred to as RCFM. The input feature map is referred to as IFM, and the output is referred to as OFM.

[0044] In another embodiment, it includes three neural network computing units NPU, three block memories blockmemory, a direct memory access DMA and a main memory DDR.

[0045] In this embodiment, three NPUs (NPU0 to NPU2) are used for parallel processing to improve the module calculation speed. The architecture of this design is as follows Figure 3 As shown, it contains three NPU modules, three block memories, and a DMA. The three NPU modules perform neural network calculations in parallel. The three block memories store input data, intermediate layer calculation results, and output results. The DMA is used for communication with the main memory. The three NPUs and three block memories are fully connected.

[0046] The functional description of each area is shown in Table 1.

[0047]

[0048] Table 1

[0049] This design uses three NPUs as a cluster. To meet the needs of different application scenarios, the number of clusters can be flexibly expanded based on the requirements of on-chip resources, thereby achieving the effect of smoothly expanding computing power.

[0050] In another embodiment, the three neural network computing units NPU are NPU0, NPU1 and NPU2, wherein NPU0 is used to calculate the first layer Conv on the right, NPU1 is used to calculate the second layer DWConv on the right, and NPU2 is used to calculate the third layer Conv on the right and add the calculation result of the third layer on the right to the input.

[0051] In another embodiment, the three block memories are fm0 for storing the input feature map feature map of the Bottleneck Residual Block neural network building module, fm1 for storing the result of the first layer Conv on the right and the result of the second layer DWConv on the right, and fm2 for storing the result of adding the third layer Conv on the right and the input feature map feature map.

[0052] In another embodiment, the DMA transfers data between block memory and main memory.

[0053] In another embodiment, the DDR stores parameters required by the neural network.

[0054] In another embodiment, a method for accelerating computing using the multi-NPU cascade structure includes the following steps:

[0055] S100: The processor loads the feature map input by the Bottleneck Residual Block neural network building module from the main memory DDR into fm0 via DMA;

[0056] S200: NPU0 reads the input feature map from fm0 and calculates the first layer Conv on the right, and stores the calculation result in fm1;

[0057] S300: NPU1 reads the calculation result of the first layer on the right side from fm1, calculates the second layer DWConv on the right side, and stores the calculation result in fm1;

[0058] S400: NPU2 reads the calculation result of the second layer on the right from fm1 and calculates the third layer Conv on the right. After the calculation is completed, it reads the feature map input by the Bottleneck Residual Block neural network building module from fm0 and adds it to the calculation result of the third layer on the right. The result of the addition is stored in fm2 as the module calculation result.

[0059] S500: The module calculation result in fm2 is stored in the main memory DDR via DMA.

[0060] In this embodiment, from the perspective of on-chip resource usage, NPU0 and NPU2 only retain processing units for calculating the Conv operator, and NPU1 only retains processing units for calculating the DWConv operator, which can effectively save on-chip resources while accelerating calculations.

[0061] In another embodiment, a cascade structure is used to calculate a simple neural network module.

[0062] Figure 4 This example shows a basic convolutional neural network module consisting of a Conv layer and a Pooling layer. In this example, only NPU0 and NPU1 are required to calculate this module.

[0063] When the NPU calculates this module, Figure 5 As shown in the figure, the main memory is first accessed to obtain the input feature map and stored in fm0. NPU0 is responsible for calculating the Conv layer, using the data from fm0 as input and then storing the result in fm1. NPU1 is responsible for calculating the Pooling layer, using the Conv layer result from fm1 as input and then storing the result in fm2. Finally, the module calculation results in fm2 are stored in the main memory via DMA.

[0064] In another embodiment, the cluster is expanded to expand computing power.

[0065] like Figure 6 As shown, three NPUs form a cluster, which can be expanded to n clusters. The input feature map is cut into n equal-length parts along the height direction, and then each of the n clusters performs parallel computations, achieving smooth scaling of computing power. The figure shows the structure of an expanded four-cluster architecture.

[0066] In another embodiment, a multi-NPU cascade structure is used to calculate the shufflenet unit.

[0067] Figure 7 This figure illustrates a building block in the ShuffleNet v2 network architecture. This module has two branches. The right branch performs three neural network operations, referred to as ra, rb, and rc for convenience. The left branch performs two neural network operations, also referred to as lb and lc. The results of these two branches are concatenated and then subjected to a Channel Shuffle, becoming the module's final output.

[0068] Since the shufflenet unit is more complex than the bottleneck residual block, more NPUs are required for parallel computing. Figure 8 As shown, in this embodiment, five NPUs are instantiated for computing this module. NPU0 to NPU5 are used to compute ra, rb, rc, lb, and lc, respectively.

[0069] When the calculation starts, the processor loads the module input feature map from the main memory into fm0 via DMA.

[0070] NPU0, NPU1, and NPU2 calculate the right branch. NPU0 reads the module input feature map from fm0 and calculates ra, storing the result in fm1. NPU1 reads the first-layer calculation result from fm1 and calculates rb, storing the result in fm2. NPU2 reads the rb calculation result from fm2 and calculates rc. After the calculation is complete, the rc result is shuffled and stored in fm4.

[0071] NPU3 and NPU4 calculate the left branch. NPU3 reads the module input feature map from fm0 and calculates lb, storing the result in fm3. NPU4 reads the lb calculation result from fm3 and calculates lc. After the calculation is completed, the lc result is shuffled and stored in fm4.

[0072] Finally, the module calculation results in fm4 are stored in the main memory through DMA.

[0073] From the perspective of on-chip resource usage, NPU0, NPU2, and NPU4 only retain processing units for calculating the Conv operator, and NPU1 and NPU3 only retain processing units for calculating the DWConv operator, which can effectively save on-chip resources while accelerating calculations.

[0074] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. A method for accelerated computing based on a multi-NPU cascade structure, characterized by: The multi-NPU cascade structure includes: Three neural network computing units NPU, Three block memories: fm0, fm1 and fm2. One direct memory access DMA and one main memory DDR; This cascade structure enables multiple neural network computing units (NPUs) to perform parallel computations, with only the input and output of the neural network building blocks needing to access the main memory. The cascade structure is used to accelerate the Bottleneck Residual Block neural network building module. The Bottleneck Residual Block structure includes two branches, one for three-layer convolution calculations and the other for a direct shortcut. The three-layer convolution calculations include a 1*1 Conv layer, a 3*3 DWConv layer, and a 1*1 Conv layer. The method for accelerating computing comprises the following steps: S100: The processor loads the feature map input by the Bottleneck Residual Block neural network building module from the main memory DDR to fm0 via DMA; S200: NPU0 reads the input feature map from fm0 and calculates the first layer Conv on the right, and stores the calculation result in fm1; S300: NPU1 reads the calculation result of the first layer on the right side from fm1, calculates the second layer DWConv on the right side, and stores the calculation result in fm1; S400: NPU2 reads the calculation result of the second layer on the right from fm1 and calculates the third layer Conv on the right. After the calculation is completed, it reads the feature map input by the Bottleneck Residual Block neural network building module from fm0 and adds it to the calculation result of the third layer on the right. The result of the addition is stored in fm2 as the module calculation result. S500: The module calculation result in fm2 is stored in the main memory DDR via DMA.

2. The method for accelerated computing based on a multi-NPU cascade structure according to claim 1, characterized in that: The three neural network computing units NPU are NPU0, NPU1 and NPU2, wherein NPU0 is used to calculate the first layer Conv on the right, NPU1 is used to calculate the second layer DWConv on the right, and NPU2 is used to calculate the third layer Conv on the right and add the calculation result of the third layer on the right to the input.

3. The method for accelerating computing based on a multi-NPU cascade structure according to claim 1, characterized in that: The three block memories are fm0 for storing the input feature map of the Bottleneck Residual Block neural network building module, fm1 for storing the result of the first layer Conv on the right and the result of the second layer DWConv on the right, and fm2 for storing the result of adding the third layer Conv on the right to the input feature map.

4. The method for accelerating computing based on a multi-NPU cascade structure according to claim 1, characterized in that: The DMA transfers data between the block memory and the main memory.

5. The method for accelerating computing based on a multi-NPU cascade structure according to claim 1, characterized in that: The DDR stores parameters required by the neural network.

6. The method for accelerated computing based on a multi-NPU cascade structure according to claim 1, characterized in that: In this cascade structure: multiple neural network computing units NPU are used for parallel computing. Except for the input and output of the neural network building module, which need to access the main memory, no other modules or structures in the cascade structure need to access the main memory.

Images