A tensor, vector, scalar computation acceleration and data scheduling system

Abstract

A tensor, vector, scalar calculation acceleration and data scheduling system belongs to the technical field of high-performance computing. The present application comprises: a scalar processing unit: for fetching and decoding instructions, and sending instructions; a vector processing unit: for receiving vector instructions from the scalar processing unit, completing the scheduling and calculation of vector type data; a tensor processing unit: for receiving tensor instructions from the scalar processing unit, completing the scheduling and calculation of tensor type data; a multi-port local data storage unit: for storing data processed by different processing units, and supporting simultaneous read and write of different processing units to their own processing data; a multi-functional direct memory access unit: for realizing data communication between memory and each functional unit; an online remote storage access unit: for realizing direct interaction of data between different processing units. The present application can effectively improve the computing power of tensor calculation, vector calculation and scalar calculation in complex algorithms and the flexibility of data scheduling.

Description

Technical Field

[0001] The present invention relates to the field of high-performance computing technology, and in particular to a tensor, vector, and scalar computing acceleration and data scheduling system. Background Art

[0002] Currently, the scale of neural network models and training / inference datasets is increasing, requiring a dramatic increase in computing power and increasing computational complexity. Neural network processors must continuously improve the computing power for training and inference while also providing greater flexibility and controllability. Meanwhile, the field of scientific computing also encompasses various types of computation, such as Einstein notation. Einstein notation includes a rich set of scalar, vector, and tensor operations, supporting a wide range of scientific computation representations.

[0003] In existing technologies, NVIDIA's GPUs (including V100, A100, etc.) have the following defects: the same type of computing power cannot run in parallel (for example, BF16 type operations must be completed before FP32 type operations can be performed), which limits further performance development; due to commercial considerations, only the CUDA programming interface is provided, and programmers are not provided with a lower-level and more detailed control interface, which restricts programmers' algorithm design.

[0004] Google's TPUs (including TPUv1, TPUv2, TPUv3, etc.) have the following defects: tensor, vector, and scalar computing units are implemented as large-scale coarse-grained components, which are inefficient for certain types of calculations; TPU processors, especially training TPUs, are not yet available on the market, and do not provide programmers with a lower-level and more detailed control interface, which limits programmers' algorithm design.

[0005] In summary, existing technologies have poor computing power and data scheduling flexibility for tensor calculations, vector calculations, and scalar calculations in complex algorithms, and are unable to bring out the optimal performance of neural network processors. Summary of the Invention

[0006] The purpose of the present invention is to solve the problems existing in the above-mentioned prior art and to provide a tensor, vector, and scalar calculation acceleration and data scheduling system, which can effectively improve the computing power and data scheduling flexibility of tensor calculation, vector calculation, and scalar calculation in complex algorithms.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A tensor, vector, and scalar computing acceleration and data scheduling system, comprising:

[0009] Scalar processing unit: used for instruction fetching and decoding, as well as sending different types of control instructions to corresponding functional units; also completes the scheduling and calculation of scalar type data;

[0010] Vector processing unit: used to receive vector instructions from the scalar processing unit and complete the scheduling and calculation of vector type data;

[0011] Tensor processing unit: used to receive tensor instructions from the scalar processing unit and complete the scheduling and calculation of tensor type data;

[0012] Multi-port local data storage unit: contains multiple read / write ports and multiple sub-storage units. The read / write ports, sub-storage units, and read / write ports and sub-storage units are independent of each other. It is used to store data processed by different processing units and supports simultaneous reading and writing of data processed by different processing units.

[0013] Multifunctional direct memory access unit: used to implement data communication between memory and various functional units, and also used to support online data transposition;

[0014] Online remote storage access unit: used to achieve direct data interaction between different tensor / vector / scalar processing units on-chip or between chips.

[0015] The present invention significantly improves computing power by unifying instruction fetching and decoding, and sending different types of control instructions to corresponding functional units, making each unit completely asynchronous. Furthermore, the system incorporates a multi-port local data storage unit and multiple access units, enabling flexible data scheduling between processing units on-chip or between chips.

[0016] As a preferred embodiment of the present invention, the scalar processing unit includes:

[0017] The scalar computing unit obtains instructions through the instruction cache, directly accesses the memory through read and write instructions, and completes the calculation of scalar type data;

[0018] Scalar registers, accessing vector registers and multi-port local data storage units of other tensor / vector / scalar processing units on or between chips through the online remote storage access unit; accessing local multi-port local storage units through the scalar data local storage access unit;

[0019] The scalar data local storage access unit realizes data interaction between the scalar register and the multi-port local storage unit, and completes the conversion of various data types.

[0020] As a preferred embodiment of the present invention, the vector processing unit includes:

[0021] A vector computing unit is used to receive vector instructions from the scalar processing unit and complete the calculation of vector type data;

[0022] Vector registers, accessing vector registers and multi-port local data storage units of other tensor / vector / scalar processing units on or between chips through the online remote storage access unit; accessing local multi-port local storage units through the vector data local storage access unit;

[0023] The vector data local storage access unit realizes data interaction between the vector register and the multi-port local storage unit and completes the conversion of various data types.

[0024] As a preferred embodiment of the present invention, the tensor processing unit includes:

[0025] The tensor computing unit is used to receive tensor instructions from the scalar processing unit and complete the calculation of scalar type data;

[0026] Tensor cache, accessing the tensor cache and multi-port local data storage unit of other tensor / vector / scalar processing units on or between chips through the online remote storage access unit; accessing the local multi-port local storage unit through the tensor data local storage access unit;

[0027] The tensor data local storage access unit realizes data interaction between the tensor cache and the multi-port local storage unit and completes the conversion of various data types.

[0028] As a preferred embodiment of the present invention, the tensor computing unit includes multiple micro-cores, which are the most basic computing units of the tensor computing unit and perform indivisible calculations related to the application algorithm.

[0029] As a preferred embodiment of the present invention, each of the sub-storage units has an independent read and write port, and an independent global addressing.

[0030] As a preferred embodiment of the present invention, the addresses of the sub-storage units are continuous.

[0031] The advantages of the present invention are that it can effectively improve the computing power of tensor calculations, vector calculations, and scalar calculations in complex algorithms and the flexibility of data scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a tensor, vector, and scalar computing acceleration and data scheduling system according to the present invention;

[0033] Figure 2 is a schematic diagram of a scalar processing unit in the present invention;

[0034] Figure 3 is a schematic diagram of a vector processing unit in the present invention;

[0035] Figure 4 is a schematic diagram of a scalar / vector processing unit in the present invention;

[0036] Figure 5 is a schematic diagram of a tensor processing unit in the present invention;

[0037] Figure 6 Schematic diagram of a tensor computing unit in the present invention;

[0038] Figure 7 Schematic diagram of a multi-port local data storage unit in the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] To facilitate a better understanding of the present invention, the following is a brief description of the terms used herein:

[0041] TVSU: Tensor, Vector, Scalar processing Unit, tensor / vector / scalar processing unit.

[0042] SU: Scalar processing Unit, scalar processing unit.

[0043] VU: Vector processing Unit, vector processing unit.

[0044] TU: Tensor processing Unit, tensor processing unit.

[0045] mDMA: multi-function Direct Memory Access, multi-function direct memory access.

[0046] mmpLDM: marionette multi-port Local Data Memory, marionette multi-port local data storage.

[0047] onRMA: online Remote Memory Access, online remote storage access.

[0048] sLMA: scalar Local Memory Access, scalar data local storage access.

[0049] vLMA: vector Local Memory Access, vector data local storage access.

[0050] tLMA: tensor Local Memory Access, tensor data local storage access.

[0051] ICache: Instruction Cache, instruction cache.

[0052] The present invention provides a tensor, vector, scalar calculation acceleration and data scheduling system. The entire system is composed of multiple TVSUs distributed on or between chips. The structure of each TVSU is as follows: Figure 1 Shown, including:

[0053] Scalar processing unit: It has a simple multi-fetch, multi-decode, and multi-issue instruction pipeline; it supports RISC-type instruction sets, which are used to implement instruction fetch and decoding of tensor / vector / scalar processing units, and send different types of control instructions to other functional units (such as vector processing units, tensor processing units, mmpLDM, mDMA, onRMA, etc.). It is also used to interact with memory, local data storage units, and scalar registers and local data storage units of other TVSUs.

[0054] Vector processing unit: used to receive vector instructions from the scalar processing unit, running in a completely non-blocking manner. Vector instructions are included in the instruction set; it is also used to interact with the local data storage unit and the vector registers and local data storage units of other TVSUs.

[0055] Tensor Processing Unit: Receives tensor instructions from the scalar processing unit and operates in a completely non-blocking manner. Tensor instructions are included in the instruction set. It also interacts with the local data storage unit and the tensor caches and local data storage units of other tensor / vector / scalar processing units. Certain application algorithms can also utilize the tensor processing unit computing mode to perform more complex calculations, such as the HASH algorithm and the Montgomery modular multiplication algorithm.

[0056] Multi-port local data storage unit mmpLDM: Figure 7 As shown, it includes multiple read-write ports and multiple internal sub-storage units. The read-write ports, the sub-storage units, and the read-write ports and sub-storage units are independent of each other. Each port can only correspond to a single sub-storage unit inside the storage unit at the same time; other units can operate a sub-storage unit through a certain port; the sub-storage units of the storage unit are managed in a unified manner and can be scheduled both independently and uniformly.

[0057] The entire unit is used to implement various optimization algorithms for neural network model training and inference, Einstein notation, and other computations. For example, different sub-storage units can implement double / multi-buffering to support software / algorithm pipelining. Different sub-storage units store different types of data, such as LDM1 for model parameters, LDM2 for training / inference data, and LDM3 for training / inference results. It also supports simultaneous reading and writing of data by different processing units. Each functional unit that reads and writes mmpLDM (such as the scalar processing unit, vector processing unit, tensor processing unit, multi-function direct memory access unit, and online remote storage access unit) has an mmpLDM port. The mmpLDM can also be accessed by other TVSUs via onRMA.

[0058] At the same time, each sub-storage unit has an independent read / write port and an independent global addressing function, and access to different sub-storage units is distinguished by the data address. In addition, the addresses of the sub-storage units are continuous. For example, mmpLDM has two sub-storage units, LDM1 and LDM2. LDM1 has a capacity of 128KB and LDM2 has a capacity of 128KB. The starting address of LDM1 is 0xB00000000000, and the starting address of LDM2 is 0xB00000020000. Therefore, all sub-storage units can be used as continuous control. Of course, the capacity of each sub-storage unit can be different.

[0059] Online Remote Memory Access Unit (onRMA): It can directly read and write various registers, caches, and mmpLDMs in other scalar processing units, vector processing units, or tensor processing units, as well as directly read and write various local registers, caches, and mmpLDMs, enabling direct interaction of data between different scalar processing units, vector processing units, and tensor processing units.

[0060] Multifunctional Direct Memory Access Unit (mDMA): Used for data transposition, broadcast, and multicast functions, it supports coarse-grained data reuse. It supports data communication between memory and TVSUs, including broadcast, multicast, unicast, and data transposition. Data access modes work in conjunction with algorithms between TVSUs to leverage different levels of data reuse. For example, broadcast enables data reuse across all TVSUs, while multicast allows data reuse across certain TVSUs. It also supports online data transposition, reducing on-chip data transposition overhead and improving data access performance.

[0061] Specifically, such as Figure 2 As shown, the scalar processing unit SU includes:

[0062] The scalar computing unit obtains instructions through the ICache. The instruction set contains control instructions for all components. It uses Load / Store instructions to directly access memory through the gLS unit and complete calculations on scalar data. Its computing performance matches that of the vector processing unit and tensor processing unit.

[0063] Scalar registers, accessing vector registers and mmpLDMs of other TVSUs on-chip or between chips through the online remote storage access unit; accessing local mmpLDMs through the scalar data local storage access unit;

[0064] The scalar data local storage access unit sLMA implements data interaction between the scalar register and mmpLDM, and completes the conversion of various data types, such as conversion between single-precision floating point and half-precision floating point, conversion between single-precision floating point and half-word integer, etc.

[0065] The start, pause, and stop controls of the entire scalar processing unit SU are completed by the control core of the neural network processor. There are three ways to access data: the first is to use the Load / Store instruction to directly access the memory through the gLS unit; the second is to access the mmpLDM through the sLMA; the third is to access the scalar registers and mmpLDM of other TVSUs through onRMA to cooperate with the entire system to achieve flexible data scheduling.

[0066] like Figure 3 As shown, the vector processing unit VU specifically includes:

[0067] The vector computing unit is used to receive vector instructions from the scalar processing unit and complete the calculation of vector type data. Its computing performance matches that of the tensor processing unit. For example, the computing performance of the vector processing unit is 1 / 32 or 1 / 16 of that of the tensor processing unit. It supports neural network model training and reasoning, Einstein notation operations, etc.

[0068] Vector registers, accessing vector registers and mmpLDMs of other TVSUs on-chip or between chips through the online remote storage access unit; accessing local mmpLDMs through the scalar data local storage access unit; accessing local mmpLDMs through the vector data local storage access unit;

[0069] The vector data local memory access unit vLMA is used to implement the interaction between vector registers and mmpLDM, and complete the conversion of various data types, such as the conversion between single-precision floating point and half-precision floating point, and the conversion between single-precision floating point and half-word integer. It is also used to complete data rearrangement, such as relocating 16 consecutive half-words in mmpLDM to the 16 components of the vector register.

[0070] The control of starting, pausing and stopping the entire vector processing unit VU is completed by the control core of the neural network processor. There are two ways to access data: the first is to access mmpLDM through vLMA; the second is to access the vector registers and mmpLDM of other TVSUs through onRMA to achieve flexible data scheduling in conjunction with the entire system. In addition, in actual application, the vector processing unit VU can be coupled to the scalar processing unit SU to form an SVU. The specific structure is as follows: Figure 4 shown.

[0071] like Figure 5 As shown, the vector processing unit TU specifically includes:

[0072] The tensor computing unit is used to receive tensor instructions from the scalar processing unit and complete tensor calculations or tensor-based calculations (such as the HASH algorithm, the Montgomery modular multiplication algorithm, etc.), making full use of the spatial locality and temporal locality of data and calculations; it supports neural network model training and reasoning, Einstein notation operations, etc., and its computing performance matches that of the vector processing unit. For example, the computing performance of the tensor processing unit is 32 times or 16 times that of the vector processing unit; the most basic computing unit in the tensor computing unit is the microkernel, such as Figure 6 As shown, it is used to complete indivisible calculations related to the application algorithm, such as the multiplication and addition components in the tensor processing unit, or the round function and modular multiplication of the HASH algorithm.

[0073] Tensor cache, accessing the tensor cache and mmpLDM of other TVSUs on-chip or between chips through the online remote storage access unit; accessing the local mmpLDM through the tensor data local storage access unit;

[0074] The tensor data local storage access unit realizes data interaction between the tensor cache and the multi-port local storage unit and completes the conversion of various data types.

[0075] The control of starting, pausing, stopping, etc. of the entire tensor processing unit TU is completed by the control core of the neural network processor; there are two ways to access data: the first is to access mmpLDM through tLMA; the second is to access the tensor buffer and mmpLDM of other TVSUs through onRMA to cooperate with the entire system to achieve flexible data scheduling.

[0076] Finally, the present invention can be used as a basic unit for larger computing components to build higher performance computing components and neural network processors.

[0077] The above description is merely a preferred embodiment of the present invention, which is one implementation method based on the overall concept of the present invention. The scope of protection of the present invention is not limited to this embodiment. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A tensor, vector, and scalar computing acceleration and data scheduling system, characterized in that: include: Scalar processing unit: used for instruction fetching and decoding, as well as sending different types of control instructions to corresponding functional units; also completes the scheduling and calculation of scalar type data; Vector processing unit: used to receive vector instructions from the scalar processing unit and complete the scheduling and calculation of vector type data; Tensor processing unit: used to receive tensor instructions from the scalar processing unit and complete the scheduling and calculation of tensor type data; Multi-port local data storage unit: contains multiple read / write ports and multiple sub-storage units. The read / write ports, sub-storage units, and read / write ports and sub-storage units are independent of each other. It is used to store data processed by different processing units and supports simultaneous reading and writing of data processed by different processing units. Multifunctional direct memory access unit: used to implement data communication between memory and various functional units, and also used to support online data transposition; Online remote storage access unit: used to realize direct data interaction between different tensor / vector / scalar processing units on or between chips; The multi-port local data storage unit is connected to the scalar processing unit, the vector processing unit, the tensor processing unit, the multi-function direct memory access unit and the online remote storage access unit; A multifunctional direct memory access unit is connected to a scalar processing unit; The online remote storage access unit is connected to the scalar processing unit, the vector processing unit and the tensor processing unit; the scalar processing unit is connected to the vector processing unit and the tensor processing unit.

2. A tensor, vector, and scalar calculation acceleration and data scheduling system according to claim 1, characterized in that: The scalar processing unit comprises: The scalar computing unit obtains instructions through the instruction cache, directly accesses the memory through read and write instructions, and completes the calculation of scalar type data; Scalar registers, accessing vector registers and multi-port local data storage units of other tensor / vector / scalar processing units on or between chips through the online remote storage access unit; accessing local multi-port local storage units through the scalar data local storage access unit; The scalar data local storage access unit realizes data interaction between the scalar register and the multi-port local storage unit, and completes the conversion of various data types.

3. A tensor, vector, and scalar calculation acceleration and data scheduling system according to claim 1, characterized in that: The vector processing unit comprises: A vector computing unit is used to receive vector instructions from the scalar processing unit and complete the calculation of vector type data; Vector registers, accessing vector registers and multi-port local data storage units of other tensor / vector / scalar processing units on or between chips through the online remote storage access unit; accessing local multi-port local storage units through the vector data local storage access unit; The vector data local storage access unit realizes data interaction between the vector register and the multi-port local storage unit and completes the conversion of various data types.

4. A tensor, vector, and scalar calculation acceleration and data scheduling system according to claim 1, characterized in that: The tensor processing unit includes: The tensor computing unit is used to receive tensor instructions from the scalar processing unit and complete the calculation of tensor type data; Tensor cache, accessing the tensor cache and multi-port local data storage unit of other tensor / vector / scalar processing units on or between chips through the online remote storage access unit; accessing the local multi-port local storage unit through the tensor data local storage access unit; The tensor data local storage access unit realizes data interaction between the tensor cache and the multi-port local storage unit and completes the conversion of various data types.

5. A tensor, vector, scalar calculation acceleration and data scheduling system according to claim 4, characterized in that: The tensor computing unit includes multiple micro-cores, which are the most basic computing units of the tensor computing unit and complete indivisible calculations related to the application algorithm.

6. A tensor, vector, and scalar calculation acceleration and data scheduling system according to claim 1, characterized in that: Each of the sub-storage units has an independent read and write port, and an independent global address.

7. A tensor, vector, scalar calculation acceleration and data scheduling system according to claim 6, characterized in that: The addresses of the sub-storage units are continuous.

Images