A multi-core neural network tensor processor with scalable computing power

Abstract

The present invention discloses a multi-core neural network tensor processor with scalable computing power, comprising a PCIE controller, M MTC cores and M SDRAM controllers; each MTC core includes S STC cores, and each STC core includes L LTC cores; wherein the LTC core is the minimum computing module, and all LTC cores in the same multi-core neural network tensor processor are configured as the same functional module; the PCIE controller is used to implement access control between the multi-core neural network tensor processor and external devices; the SDRAM controller is used to correspond to the MTC core's access to an off-chip SDRAM memory. The multi-core neural network tensor processor of the present invention adopts a modular reuse design scheme, and forms a neural network tensor processor with a certain computing power specification by repeatedly calling the minimum computing module and combining these minimum computing modules together; this structure can greatly reduce the complexity of design and verification.

Description

Technical Field

[0001] The present invention relates to the field of neural network tensor processors, and in particular to a multi-core neural network tensor processor with scalable computing power. Background Art

[0002] Existing neural network tensor processors usually have a fixed internal structure and provide fixed computing performance (referred to as computing power). For example, in the previous patent 1 (the invention name is: A neural network multi-core tensor processor, application number is: 202011423696.0) and the previous patent 2 (the invention name is: A neural network tensor processor, application number is: 202011421828.6), the computing power of the tensor processor is determined by the number of computing resources, and in the neural network tensor processor, the computing resources need to match resources such as storage capacity and bus bandwidth. Once the architecture of the tensor processor is determined, the maximum computing power that the architecture can support is also basically determined and difficult to change.

[0003] On the other hand, for neural network tensor processors with ultra-high computing power (for example, 4096TOPS), the chip area may reach several hundred square millimeters. For chips with an area of ​​several hundred square millimeters, the front-end design, verification, and back-end layout design will become very time-consuming and complicated, requiring a lot of human and material resources. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a multi-core neural network tensor processor with scalable computing power. The tensor processor adopts the design concept of modular reuse. By designing the minimum computing unit, reusing the minimum computing unit and forming a structured computing module, a multi-core neural network tensor processor structure with scalable computing power is realized. The multi-core neural network tensor processor adopts a modular reuse method, which can greatly reduce the complexity of design and verification. Through a simple module combination method, various neural network tensor processor configuration schemes with different computing powers can be realized, thereby greatly reducing the design complexity of high-computing power neural network tensor processors, reducing the design and verification cycle, and ultimately achieving the design requirements of various computing power neural network tensor processors with minimal design and verification costs.

[0005] To achieve the above-mentioned objectives, the present invention provides a multi-core neural network tensor processor with scalable computing power, comprising a PCIE controller, M MTC cores and M SDRAM controllers; each MTC core includes S STC cores, and each STC core includes L LTC cores; wherein the LTC core is the smallest computing module, and all LTC cores in the same multi-core neural network tensor processor are configured as the same functional module; the PCIE controller is used to implement access control between the multi-core neural network tensor processor and external devices; the SDRAM controller is used to correspond to the MTC core's access to the off-chip SDRAM memory.

[0006] More preferably, S is a multiple of 2, and L is a multiple of 2.

[0007] Furthermore, the LTC core includes a 4D tensor core, a 1D tensor core, an instruction control unit, a local cache unit, a memory read unit, a memory write unit and a special unit; the 4D tensor core is used to implement basic operations of 4D tensor data; the 1D tensor core is used to implement basic operations of 1D data; the instruction control unit is used to obtain configuration parameters and instructions of the 4D tensor core and the 1D tensor core from an external memory; the local cache unit is used to store input data required for the 4D tensor core; the memory read unit is used to provide direct external memory read and write capabilities for each module within the minimum computing module; the special unit is used to implement computing functions related to coordinate transformation.

[0008] Further preferably, the LTC core includes one 4D tensor core and two 1D tensor cores.

[0009] Furthermore, the 4D tensor core contains P1 FP16MACs and 2*P1 INT8MACs.

[0010] Further preferably, P1 is 2 to the power of M1, and M1 is not less than 8.

[0011] Furthermore, the 1D tensor core contains P2 FP16MACs.

[0012] Further preferably, P2 is 2 to the power of M2, and M2 is 4 or 5 or 6.

[0013] Furthermore, the basic operations of the 4D tensor data include multiplication, addition, multiplication accumulation and maximum value; the basic operations of the 1D data include multiplication, addition, linear activation operation and nonlinear activation calculation.

[0014] Furthermore, a cascade path is provided between the 4D tensor core and the 1D tensor core, for directly inputting the output data of the 4D tensor core into the 1D tensor core.

[0015] Furthermore, the memory read unit is configured as follows: the 4D tensor core has two independent memory read units, which are used to respectively implement the read operations of data and parameters required for 4D calculations; the 1D tensor core has two independent memory read units, which are used to respectively implement the read operations of data and parameters required for 1D calculations; the instruction control unit and the special unit share a memory read unit; the 1D tensor core has an independent memory write unit, which is used to implement the write operation of the 4D / 1D tensor core calculation results; the special unit has an independent memory write unit, which is used to implement the write operation of the calculation results.

[0016] Furthermore, the configuration parameters are used to configure the 4D tensor core into a specific computing function and the 1D tensor core into a specific computing function; and the instructions are used to implement control of the computing process.

[0017] Further preferably, the capacity of the local cache unit is P3*32KB, where P3 is an integer not less than 8.

[0018] Furthermore, the STC core of the neural network multi-core tensor processor also includes a shared memory and a data control unit; the shared memory is used to cache input data and intermediate data commonly required by each LTC core inside the STC, or commonly required parameters; the data control unit is used to pre-fetch shared data or shared parameters existing in the off-chip SDRAM into the shared memory in advance.

[0019] Furthermore, the interconnection relationship between the MTC core, STC core and LTC core is: the LTC core can only access its own local cache unit, and cannot access the local cache units of other LTC cores; the LTC core can access the shared memory of the STC core to which it belongs, and can also access the shared memory of other STC cores; the LTC core can access the off-chip SDRAM memory of the MTC core to which it belongs, and can also access the off-chip SDRAM memory of other MTC cores.

[0020] Furthermore, the capacity of the shared memory is greater than or equal to the local storage cache.

[0021] The present invention achieves the following technical effects:

[0022] (1) The multi-core neural network tensor processor of the present invention adopts a modular reuse design scheme, by repeatedly calling the minimum computing modules and combining these minimum computing modules together, thereby forming a neural network tensor processor with a certain computing power specification.

[0023] (2) The multi-core neural network tensor processor of the present invention can form tensor processors with various computing powers by flexibly setting the number of MTC cores, STC cores and LTC cores and combining different LTC cores, STC cores and MTC cores, thereby realizing a tensor processor architecture with scalable computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a block diagram of the internal structure of the minimum computing module (LTC core) of the present invention;

[0025] Figure 2 It is a block diagram of the internal structure of the STC core of the present invention;

[0026] Figure 3 It is a block diagram of the internal structure of the MTC core of the present invention;

[0027] Figure 4 It is a block diagram of the internal structure of the neural network multi-core tensor processor of the present invention. DETAILED DESCRIPTION

[0028] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0029] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0030] The present invention proposes a multi-core neural network tensor processor with scalable computing power. The tensor processor adopts a modular reuse design scheme, repeatedly calling the minimum computing modules and combining these minimum computing modules together to form a neural network tensor processor with a certain computing power specification.

[0031] We assume that the computing power of the minimum computing module is A. The computing power is scalable, which means that the computing power specification of the multi-core neural network tensor processor is determined according to the number N of the minimum computing modules specifically called, and the specific value of its computing power is equal to A*N. The computing power is scalable, which means that the value range of N can be any positive integer.

[0032] Although the computing power can be expanded, the functionality of neural network computing is unchanged for the multi-core neural network tensor processors with different computing powers. The functionality of the multi-core neural network tensor processor is determined by the functionality of the smallest computing module, and the functionality of the neural network tensor processor with N being 1 is exactly the same as that of the neural network tensor processor with N being any other positive integer.

[0033] The minimum computing module (LTC core) is composed of a 4D Tensor Core, a 1D Tensor Core, an instruction control unit, a local cache unit, a memory read unit (LD), a memory write unit (ST) and a special unit (Recut). Its internal structure is as follows: Figure 1 shown.

[0034] In this embodiment, the minimum computing module includes one 4D Tensor Core and two 1D Tensor Cores (in different implementations, the number of 1D Tensor Cores may be other numbers, such as 1). The 4D Tensor Core contains 1024 FP16MACs and 2048 INT8MACs (in different implementations, the number of FP16MACs and INT8MACs may be other numbers, usually set to: the 4D Tensor Core contains P1 FP16MACs and 2*P1 INT8MACs, where P1 is 2 to the power of M1, and M1 is not less than 8). The 1D Tensor Core contains 16 FP16MACs (in different implementations, the number of FP16MACs may be other numbers, usually set to: the 1D Tensor Core contains P2 FP16MACs, where P2 is 2 to the power of M2, and M2 is 4 or 5 or 6).

[0035] Most of the computing functions of the minimum computing module are implemented by 4D tensor cores and 1D tensor cores, including convolution, full connection, pooling, etc. The main function of 4D tensor cores is to implement basic operations on 4D tensor data (data with dimensions (n,c,h,w)), including multiplication, addition, multiplication accumulation, maximum value, etc. The main function of 1D tensor cores is to implement basic operations on 1D data (such as data with dimension (w)), including multiplication, addition, various linear activation operations (such as Relu), various nonlinear activation operations (such as Sigmod), etc.

[0036] There is a cascade path between the 4D tensor core and the 1D tensor core, that is, the output data of the 4D tensor core can be directly input to the 1D tensor core, so as to realize the function of completing the 1D computing task immediately after the 4D computing task. Therefore, a calculation of the smallest computing module can load multiple operators at the same time, such as convolution + Relu, to achieve higher computing efficiency.

[0037] The instruction control unit is mainly used to obtain the configuration parameters and instructions of the 4D tensor core and the 1D tensor core from the external memory (the external memory can be an off-chip SDRAM or other shared memory in the tensor processor chip). The configuration parameters are used to configure the 4D tensor core into a specific computing function (such as a convolution computing function) and the 1D tensor core into a specific computing function (such as a Relu computing function). The instructions are used to control the computing process, such as starting, pausing, and ending.

[0038] The local cache unit capacity is set to a multiple of 32KB, which is usually set to no less than 8. The typical capacity of the local cache unit is 320KB, and its main function is to store the input data required by the 4D tensor core, and at the same time rearrange the input data in the order required by the 4D tensor core, thereby reducing the complexity of subsequent calculation circuits. The 1D tensor core does not require an independent cache unit, and its input can come from the output of the 4D tensor core (it can also be considered that the 1D tensor core indirectly uses the local cache of the 4D tensor core). The input of the 1D tensor core can also come directly from external memory. The input data does not need to be cached, and is directly calculated and then output.

[0039] The memory read unit LD and the memory write unit ST are mainly used to provide external memory direct read and write capabilities for each module within the minimum computing module. Among them, the 4D tensor core has two independent memory read units LD, which are used to implement the read operations of data and parameters required for 4D calculations; the 1D tensor core has two independent memory read units LD, which are used to implement the read operations of data and parameters required for 1D calculations; the instruction control unit and the special unit share one memory read unit LD; the 1D tensor core has an independent memory write unit ST, which is used to implement the write operation of the 4D / 1D tensor core calculation results (the calculation results of the 4D tensor core must be written out through the 1D tensor core); the special unit has an independent memory write unit ST, which is used to implement the write operation of the calculation results.

[0040] Special units are mainly used to implement calculation functions related to coordinate transformation. They have no calculation resources such as multiplication or addition. Their main function is to read input data and output data according to the new data arrangement. Their typical calculation functions include Reshape and Concat.

[0041] By effectively organizing multiple LTC cores together, a neural network multi-core tensor processor with high computing power can be formed. STC core is the first-level organization of the neural network multi-core tensor processor, such as Figure 2 shown.

[0042] Each STC core contains L LTC cores, preferably, L is a multiple of 2, and a typical value of L is 4 or 8. These LTC cores are completely identical in design, and multi-core design can be achieved by repeatedly calling the same LTC core. In addition, the STC core also contains a shared memory and a data control unit.

[0043] The capacity of the shared memory is usually the same as or slightly larger than the capacity of the local cache unit of the LTC core (for example, the typical capacity of the local cache unit of the LTC core is 320KB, and the typical capacity of the shared memory is 352KB). The role of the shared memory is to cache the input data and intermediate data or parameters commonly needed by the LTC cores within the STC.

[0044] Depending on the different calculation methods of the multi-core neural network tensor processor, each LTC core can use different parameters to calculate the same data, or use the same parameters to calculate different data. If these shared data or parameters exist in the off-chip SDRAM, the data control unit will pre-fetch these shared data or parameters into the shared memory in advance. After that, the LTC core will directly obtain the required data or parameters from the shared memory during the calculation process, thereby saving off-chip storage bandwidth. If the capacity of the shared memory is insufficient to store the shared data or parameters, the LTC core will directly read the required data or parameters from the off-chip SDRAM during the calculation process.

[0045] The MTC core is the second level organization of the neural network multi-core tensor processor, such as Figure 3 shown.

[0046] Each MTC core includes S STC cores, preferably, S is a multiple of 2, and a typical value of S is 4 or 8. These STC cores are completely identical in design, and a complex design with more cores can be implemented by repeatedly calling the same STC core.

[0047] All STC cores in the MTC core access the same off-chip SDRAM memory through the same SDRAM controller. Therefore, the computing unit in the MTC core uses a three-level cache structure of LTC local cache unit -> STC shared memory -> off-chip SDRAM memory. Among them, the off-chip SDRAM memory stores all data required by all computing units, the STC shared memory stores data required by multiple LTC cores, and the LTC local cache unit stores the data required for LTC internal unit calculations.

[0048] The MTC core contains multiple STC cores, and the STC core contains multiple LTC cores. The interconnection relationship between them is:

[0049] An LTC core can only access its own local cache unit, and cannot access the local cache units of other LTC cores;

[0050] The LTC core can access the shared memory of the STC core to which it belongs, and can also access the shared memory of other STC cores;

[0051] The LTC core can access the off-chip SDRAM memory of the MTC core to which it belongs, and can also access the off-chip SDRAM memory of other MTC cores;

[0052] In the MTC core, only the LTC local cache unit is exclusive to each LTC core and cannot be accessed by other LTC cores. The STC shared memory and the off-chip SDRAM memory are shared and can be accessed by any STC core in the MTC core. In the initial stage of operation of the multi-core neural network tensor processor, all data required for calculation is stored in the off-chip SDRAM memory. During operation, as needed, if some data will be accessed by multiple STC cores of the MTC core, the data will be prefetched into the STC shared memory. Furthermore, during operation, as needed, some data will also be prefetched into the LTC local cache unit.

[0053] The highest level organization of the neural network multi-core tensor processor is as follows Figure 4 shown.

[0054] The neural network multi-core tensor processor includes a PCIE controller (PCI Express Controller), multiple MTC cores and multiple SDRAM controllers (SDRAM Controller), wherein the number of MTC cores is the same as the number of SDRAM controllers.

[0055] The neural network multi-core tensor processor includes M MTC cores, each MTC core includes S STC cores, and each STC core includes L LTC cores. By flexibly setting the number of MTC cores, STC cores, and LTC cores, and combining different LTC cores, STC cores, and MTC cores, a tensor processor with multiple computing powers can be formed, thereby realizing a tensor processor architecture with scalable computing power.

[0056] Assuming that the computing power of LTC is A, the computing power of the neural network multi-core tensor processor is M*S*L*A, where the value range of M, S, and L is 1 to any positive integer. Therefore, the neural network multi-core tensor processor has flexible computing power scalability. By configuring different values ​​for M, S, and L, neural network multi-core tensor processors with different computing powers can be implemented. For example, if the computing power A of the LTC core is 4TOPS, and the configuration of the neural network multi-core tensor processor adopts M=8, S=8, and L=16, the total computing power of the tensor processor is 8*8*16*4=4096TOPS.

[0057] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A multi-core neural network tensor processor with scalable computing power, characterized in that: It includes a PCIE controller, M MTC cores and M SDRAM controllers; each MTC core includes S STC cores, and each STC core includes L LTC cores; the LTC core is the smallest computing module, and all LTC cores in the same multi-core neural network tensor processor are configured as the same functional module; The PCIE controller is used to implement access control between the multi-core neural network tensor processor and external devices; The SDRAM controller is used to correspond to the MTC core's access to the off-chip SDRAM memory; The LTC core includes a 4D tensor core, a 1D tensor core, an instruction control unit, a local cache unit, a memory read unit, a memory write unit and a special unit; the 4D tensor core is used to implement basic operations of 4D tensor data; the 1D tensor core is used to implement basic operations of 1D data; the instruction control unit is used to obtain configuration parameters and instructions of the 4D tensor core and the 1D tensor core from an external memory; the local cache unit is used to store input data required by the 4D tensor core; the memory read unit is used to provide direct external memory read and write capabilities for each module within the minimum computing module; the special unit is used to implement computing functions related to coordinate transformation.

2. The multi-core neural network tensor processor with scalable computing power as claimed in claim 1, characterized in that: S is a multiple of 2, and L is a multiple of 2.

3. The multi-core neural network tensor processor with scalable computing power as claimed in claim 1, characterized in that: The LTC core includes one 4D tensor core and two 1D tensor cores.

4. The multi-core neural network tensor processor with scalable computing power as claimed in claim 2, characterized in that: The 4D tensor core internally includes P1 FP16 MACs and 2*P1 INT8 MACs.

5. The multi-core neural network tensor processor with scalable computing power as claimed in claim 4, characterized in that: P1 is 2 to the power of M1, and M1 is not less than 8.

6. The multi-core neural network tensor processor with scalable computing power as claimed in claim 1, characterized in that: The 1D Tensor Core contains P2 FP16 MACs internally.

7. The multi-core neural network tensor processor with scalable computing power as claimed in claim 6, characterized in that: P2 is 2 to the power of M2, and M2 is 4 or 5 or 6.

8. The multi-core neural network tensor processor with scalable computing power as claimed in claim 1, characterized in that: The basic operations of the 4D tensor data include multiplication, addition, multiplication accumulation and maximum value; the basic operations of the 1D data include multiplication, addition, linear activation operation and nonlinear activation calculation.

9. The multi-core neural network tensor processor with scalable computing power as claimed in claim 1, characterized in that: A cascade path is provided between the 4D tensor core and the 1D tensor core, for directly inputting the output data of the 4D tensor core into the 1D tensor core.

10. The multi-core neural network tensor processor with scalable computing power as claimed in claim 9, characterized in that: The memory read unit is configured as follows: the 4D tensor core has two independent memory read units, which are used to respectively implement the read operations of data and parameters required for 4D calculations; the 1D tensor core has two independent memory read units, which are used to respectively implement the read operations of data and parameters required for 1D calculations; the instruction control unit and the special unit share a memory read unit; the 1D tensor core has an independent memory write unit, which is used to implement the write operation of the 4D / 1D tensor core calculation results; the special unit has an independent memory write unit, which is used to implement the write operation of the calculation results.

11. The multi-core neural network tensor processor with scalable computing power as claimed in claim 1, characterized in that: The configuration parameters are used to configure the 4D tensor core into a specific computing function and the 1D tensor core into a specific computing function; the instructions are used to implement the control of the computing process.

12. The multi-core neural network tensor processor with scalable computing power as claimed in claim 1, characterized in that: The capacity of the local cache unit is P3*32KB, where P3 is an integer not less than 8.

13. The multi-core neural network tensor processor with scalable computing power as claimed in claim 1, characterized in that: The STC core of the multi-core neural network tensor processor also includes a shared memory and a data control unit; The shared memory is used to cache input data and intermediate data commonly needed by each LTC core in the STC, or commonly needed parameters; The data control unit is used to pre-fetch the shared data or shared parameters existing in the off-chip SDRAM into the shared memory in advance.

14. The multi-core neural network tensor processor with scalable computing power as claimed in claim 13, characterized in that: The interconnection relationship between the MTC core, STC core and LTC core is: the LTC core can only access its own local cache unit, and cannot access the local cache units of other LTC cores; the LTC core can access the shared memory of the STC core to which it belongs, and can also access the shared memory of other STC cores; the LTC core can access the off-chip SDRAM memory of the MTC core to which it belongs, and can also access the off-chip SDRAM memory of other MTC cores.

15. The multi-core neural network tensor processor with scalable computing power as claimed in claim 13, characterized in that: The capacity of the shared memory is greater than or equal to the local storage cache.

Images