A neural network processor

Abstract

The application discloses a neural network processor, and belongs to the technical field of high-performance computing.The neural network processor comprises a control core, a tensor / vector / scalar processing array, a memory unit, an on-chip network unit and an I / O interface unit.The control core is used for controlling operation and processing on each functional unit; the tensor / vector / scalar processing array is integrated with n*m tensor / vector / scalar processing units and is used for performing tensor / vector / scalar calculation; the memory unit comprises shared memory and a plurality of private memories; the on-chip network unit is used for realizing data exchange between each functional unit on the chip; the set operation acceleration unit comprises an online calculation component and an online compression / decompression component and is used for realizing online calculation and online decompression of the set operation on the chip; and the I / O interface unit is used for connecting with other neural network processors or a host processor and realizing data exchange between chips.The application can effectively improve the computing power of neural network model training and reasoning and increase the flexibility of data processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of high-performance computing, and in particular to a neural network processor. BACKGROUND

[0002] At present, the scale of neural network models, training / inference datasets are getting larger and larger, the required computing power is increasing sharply, and the computing complexity is also increasing. Neural network processors not only need to continuously improve the training and inference computing power, but also need to provide more flexible controllability. On the other hand, the scientific computing field also has various types of calculations, such as Einstein notation. Einstein notation contains a variety of scalar, vector, tensor operation types, and supports a wide range of scientific computing representations.

[0003] In the prior art, the GPU of Nvidia (including V100, A100, etc.) has the following defects: the same type of computing power cannot be run in parallel (for example, BF16 type operations must be completed before FP32 type operations can be performed), which limits the further development of performance; due to commercial considerations, only the CUDA programming interface is provided, and a more detailed control interface is not provided for programmers, which limits the algorithm design of programmers.

[0004] The TPU of Google (including TPUv1, TPUv2, TPUv3, etc.) has the following defects: the tensor, vector, and scalar calculation units are implemented as large-scale coarse-grained components, which are inefficient for certain types of calculations; TPU processors, especially training TPU, are not on the market, and a more detailed control interface is not provided for programmers, which limits the algorithm design of programmers.

[0005] In the prior art, for example, the Chinese patent application with the application number 202010822840.1 discloses a neural network processor and a data processing method. The neural network processor includes: a first cache unit for storing first input data; a second cache unit for storing weight data; an operation unit connected with the first cache unit and the second cache unit, for performing matrix multiplication operation on the first input data and the weight data to obtain intermediate calculation results; a compression unit for processing the intermediate calculation results to obtain processed data; the processed data includes compressed data and uncompressed data; a storage unit access controller connected with the compression unit, for storing the processed data into a storage device outside the neural network processor.

[0006] The neural network processor structure described above is relatively traditional, and the connection relationship and data flow between the various functional modules are relatively fixed, that is, each functional module needs to process data in sequence, thereby limiting the computing power of the entire processor. SUMMARY

[0007] The application aims to solve the problems in the prior art and provide a neural network processor which can effectively improve the computing power of neural network model training and reasoning and increase the flexibility of data processing.

[0008] The application is achieved by the following technical scheme.

[0009] A neural network processor comprises:

[0010] A control core is configured to control operations and processing on each functional unit, including starting, running, pausing, resuming, stopping, restarting and loading.

[0011] A tensor / vector / scalar processing array is integrated with n*m tensor / vector / scalar processing units and is configured to perform tensor / vector / scalar calculation.

[0012] A memory unit comprises shared memory and a plurality of private memories, and each private memory corresponds to one of the tensor / vector / scalar processing arrays.

[0013] An on-chip network unit is configured to realize data exchange between each functional unit on the chip.

[0014] A set operation acceleration unit comprises an online calculation component and an online compression / decompression component and is configured to realize online calculation and online decompression of the set operation on the chip.

[0015] An I / O interface unit is configured to be connected with other neural network processors or host processors and realize data exchange between the chip and other chips.

[0016] The on-chip network unit enables free data exchange between each functional unit on the chip, including data reuse between the plurality of tensor / vector / scalar processing arrays, and the n*m tensor / vector / scalar processing units in each tensor / vector / scalar processing array can also flexibly exchange data, thereby effectively improving the computing power and flexibility of the entire neural network processor.

[0017] As a preferred embodiment of the application, the n*m tensor / vector / scalar processing units in the tensor / vector / scalar processing array are interconnected through a MESH topology network, and the tensor / vector / scalar processing units on the same row or column can directly exchange data, and the tensor / vector / scalar processing units on the diagonal line can exchange data through the tensor / vector / scalar processing units at the intersection of the row and column as a relay.

[0018] As a preferred embodiment of the application, the tensor / vector / scalar processing units exchange data through access registers, tensor caches and multi-port local data storage units.

[0019] As a preferred embodiment of the present application, in the tensor / vector / scalar processing array, data reuse is achieved for data entered through DMA by using broadcast and multicast of DMA.

[0020] As a preferred embodiment of the present application, in the tensor / vector / scalar processing array, data reuse is achieved for data that has entered the registers, tensor cache and multi-port local data storage unit of the tensor / vector / scalar processing unit by using online remote storage access unit.

[0021] As a preferred embodiment of the present application, a plurality of tensor / vector / scalar processing arrays are included, which are interconnected through the on-chip network unit, each of the tensor / vector / scalar processing arrays has an independent memory controller for managing local memory.

[0022] As a preferred embodiment of the present application, the memory of all tensor / vector / scalar processing arrays are uniformly addressed.

[0023] As a preferred embodiment of the present application, the memory controller of each tensor / vector / scalar processing array divides the high space and low space of the local memory into private memory and shared memory respectively.

[0024] As a preferred embodiment of the present application, the memory unit logically divides the memory into private memory and shared memory through mapping of different addresses.

[0025] The present application has the advantages of effectively improving the computing power of single-chip for completing neural network model training and inference, including reducing the delay of neural network processing on a single processor, improving the throughput of neural network processing on a single processor, and reducing the network overhead when parallel processing a neural network; and significantly increasing the flexibility of implementing complex neural network model training and inference, providing programmers with the possibility of completely controlling the neural network processor, and supporting programmers to more effectively explore the potential of neural network computing power and implement efficient algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 Fig. 1 is a schematic diagram of a neural network processor according to the present application;

[0027] Figure 2 Fig. 2 is a schematic diagram of a tensor / vector / scalar processing array according to the present application;

[0028] Figure 3 Fig. 3 is a schematic diagram of a memory unit according to the present application;

[0029] Figure 4 Fig. 4 is a schematic diagram of a set operation acceleration unit according to the present application. DETAILED DESCRIPTION

[0030] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0031] For better understanding of the application, the terms used in this paper will be briefly described below:

[0032] NNP: Neural Network Processor, neural network processor.

[0033] TVSU: Tensor, Vector, Scalar processing Unit, tensor / vector / scalar processing unit.

[0034] TVSA: Tensor, Vector, Scalar processing Array, tensor / vector / scalar processing array.

[0035] mmpLDM: marionette multi-port Local Data Memory, marionette multi-port local data memory.

[0036] onRMA: on-line Remote Memory Access, on-line remote memory access.

[0037] iMC: intelligent Memory Controller, intelligent memory controller.

[0038] Ccore: Control Core, control core (main core).

[0039] aDMA: affiliate Direct Memory Access, affiliate direct memory access unit.

[0040] As shown in Figure 1 , the application provides a neural network processor, comprising:

[0041] a control core, as the main core of the whole neural network processor, for controlling the operation and processing on each functional unit, including starting, running, pausing, resuming, stopping, restarting, loading;

[0042] a memory unit, comprising a shared memory and a plurality of private memories corresponding to the tensor / vector / scalar processing arrays, wherein the shared memory is used to store full-chip shared data, and the private memories are used to store data used by individual TVSAs;

[0043] The plurality of tensor / vector / scalar processing arrays with local memories are integrated with n x m tensor / vector / scalar processing units (TVSUs), which are connected with local memories through an iMC memory controller to accelerate local memory operations, and are connected with on-chip network units through an aDMA to cooperate with mDMAs in the tensor / vector / scalar processing units to accelerate on-chip other tensor / vector / scalar processing array memory operations; wherein the TVSUs are slave cores of a master core, and the most basic computing units (multiply-add components) in the tensor processing units in the TVSUs are micro-cores of slave cores. That is, the entire neural network processor is logically structured as master, slave and micro through the control core, the TVSUs and the multiply-add components, so that the processing and calculation of data are more orderly and efficient.

[0044] The on-chip network unit is used to connect various functional units on the chip to complete data exchange between the units.

[0045] The set operation acceleration unit includes an online calculation component and an online compression / decompression component, and is used to realize online calculation and online decompression of the set operation on the chip.

[0046] The I / O interface unit is used to connect with the neural network processor or the host processor, and is also used to realize data exchange between the chip and other chips to further improve the flexibility of overall data processing.

[0047] As described above, the n x m TVSUs are interconnected through a MESH topology network, each TVSU can directly access various registers, tensor caches and mmpLDMs of the TVSUs in the same row or column, and two TVSUs at the diagonal position can realize data interaction through the TVSU at the intersection of the row and column. When data enters the TVSA through the DMA, the broadcast and multicast of the DMA are used to realize data reuse; the data that has entered various registers, tensor caches and mmpLDMs of the TVSU, or the intermediate results of the calculation process, realize data reuse through the onRMA.

[0048] In addition, each TVSA has an independent memory controller iMC for managing the local memory. The memories of all TVSAs are uniformly addressed, and all the memories are logically divided into private memories of the TVSAs and shared memories of all the TVSAs through mapping of different addresses. The memory controller iMC of each TVSA allocates the high-bit space and the low-bit space of the local memory to the private memory and the shared memory, respectively.

[0049] Specifically, as shown in the tensor / vector / scalar processing array (TVSA) shown in Figure 2

[0050] ​Tensor / vector / scalar processing unit (TVSU) is the basic computing unit of TVSA. TVSU is a core supporting RISC type instruction set, with independent instruction pipeline, executing independent instruction stream, supporting complete scalar calculation, vector calculation and tensor calculation for neural network calculation, and programmers can write different codes for each TVSU.

[0051] n×m (for example, 4×8) TVSUs are interconnected through a MESH topology network. TVSUs can directly access various registers, tensor caches and mmpLDMs of TVSUs in the same row or column, and two TVSUs in the diagonal position can realize data interaction through TVSUs located at the intersection of rows and columns.

[0052] There are two levels of data reuse mechanisms in TVSA: when data enters TVSA through DMA, broadcast and multicast of DMA are used to realize data reuse; data has entered various registers, tensor caches and mmpLDMs of TVSUs, or the intermediate results of the calculation process, which can be reused through onRMA.

[0053] Programmers can write the same or different codes for each TVSU in TVSA to realize different running modes: all TVSUs can participate in the same task, all TVSUs can participate in completely different tasks, and n×m TVSUs can also be combined to execute different tasks.

[0054] Each TVSA has an independent memory controller iMC to manage local memory (such as DDR, HBM, etc.); a single neural network processor can integrate one or more TVSAs, which are interconnected through an on-chip network (such as an on-chip ring network, a crossbar switch, etc.); TVSA accesses the memory of other TVSAs through aDMA.

[0055] As shown in Figure 3 : Each TVSA has local memory (such as DDR, HBM, etc.) accessed through an independent memory controller iMC; all TVSA memories are uniformly addressed, and all memories are logically divided into private memory (such as 0xA00000000000) of TVSA and shared memory (such as 0x0x400000000000) of all TVSAs through mapping of different addresses; the memory controller iMC of TVSA allocates the high space and low space of local memory to the private memory and shared memory respectively; during the process of neural network model training or inference, the full-chip shared data (such as model weight) can be stored in the shared memory for use by all TVSAs; the data used by a single TVSA (such as training set) is stored in the private memory.

[0056] The collection operation acceleration unit is shown in Figure 4 :

[0057] Mainly consists of two parts of online computing components and online compression / decompression components, runs in a non-blocking manner - by setting the address of local data in memory, the destination of message arrival, the collection operation acceleration unit can complete the whole process autonomously, without intermediate control.

[0058] Online computing components include multiplication and addition, transcendental function, etc. The online computing component obtains data from one or more memory addresses, uses the computing component to perform addition, multiplication, square root, logarithm, etc. Finally, the result is sent to the online compression / decompression component. The online computing component supports standard MPI protocol collection operations (such as Reduce, AllReduce, Reduce_Scatter), and also supports neural network parallel training and inference related communications.

[0059] For standard MPI protocol collection operations, the online computing component completes the specified protocol calculation, such as obtaining data to be sent from memory, and then completing summation, maximum / minimum value, logical operation, etc. by the multiplication and addition unit component.

[0060] For specific types of communication related to neural network parallel training and inference, for example, when multiple neural network processors use large-scale training data sets to participate in training the same neural network model, each neural network processor calculates the local update value ΔW of the model parameters, which needs to be globally averaged, and each neural network processor contains multiple TVSAs, each TVSA calculates its own ΔW. If all participate in the global average operation, it will occupy a lot of bandwidth. The online computing component can complete the global averaging of the local ΔW of the neural network processor - only send the ΔW after averaging, which significantly reduces the bandwidth overhead and improves communication performance.

[0061] The online compression / decompression component uses online compression / decompression algorithms (such as: Frequent Pattern Compression (FPC), Lempel-Ziv (LZ), C-Pack, Base-Delta-Immediate (BDI), Bit Plane Compression (BPC), etc.) to significantly reduce data size and improve communication performance.

[0062] The above is only the preferred specific embodiment of the present application, which is based on one implementation of the overall concept of the present application, and the protection scope of the present application is not limited thereto. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A neural network processor, comprising: include: The control core is used to control the operations and processing of various functional units, including startup, running, pausing, resuming, stopping, restarting, and loading. Tensor / vector / scalar processing array, integrating n*m tensor / vector / scalar processing units, used for tensor / vector / scalar computation; The memory unit includes shared memory and several private memory units, each of which corresponds to one of the tensor / vector / scalar processing arrays; The on-chip network unit is used to enable data exchange between various functional units on the chip. The set operation acceleration unit includes an online computing component and an online compression / decompression component, which are used to implement on-chip online computing and online decompression that support set operations; The I / O interface unit is used to connect with other neural network processors or the main processor, and to enable data exchange between chips.

2. The neural network processor of claim 1, wherein, The n*m ​​tensor / vector / scalar processing units in the tensor / vector / scalar processing array are interconnected through a MESH topology. Tensor / vector / scalar processing units in the same row or column can directly interact with each other; tensor / vector / scalar processing units on the diagonal can interact with each other through tensor / vector / scalar processing units at the intersection of rows and columns as intermediaries.

3. The neural network processor of claim 2, wherein, The tensor / vector / scalar processing units interact with each other by accessing registers, tensor caches, and multi-port local data storage units.

4. The neural network processor of claim 1, wherein, In the tensor / vector / scalar processing array, data reuse is achieved by utilizing DMA's broadcast and multicast capabilities for data entering via DMA.

5. The neural network processor of claim 1, wherein, In the tensor / vector / scalar processing array, data that has already entered the registers, tensor caches, and multi-port local data storage units of the tensor / vector / scalar processing unit is reused through the online remote storage access unit.

6. The neural network processor of claim 1, wherein, It includes multiple tensor / vector / scalar processing arrays interconnected via the on-chip network unit, each of the tensor / vector / scalar processing arrays having an independent memory controller for managing local memory.

7. The neural network processor of claim 6, wherein, All tensor / vector / scalar processing arrays use a unified memory addressing system.

8. The neural network processor of claim 6, wherein, Each of the tensor / vector / scalar processing arrays' memory controllers allocates the high-order and low-order spaces of the local memory to private memory and shared memory, respectively.

9. The neural network processor of claim 1, wherein, The memory units logically divide the memory into private memory and shared memory through the mapping of different addresses.

Images