A unified computational approach for general-purpose neural network tensor processors

Abstract

The present invention discloses a unified computing method for a general neural network tensor processor, which realizes instruction computing and operator computing simultaneously under a unified environment and method, and expresses all the contents of the neural network algorithm and application in the form of operators, wherein the calculations related to the neural network algorithm are expressed in the form of neural network operators, and the calculations related to the neural network application are expressed in the form of custom instruction operators. The unified computing method of the present invention can execute operators and instructions simultaneously in a general neural network tensor processor, so that the system has both the high performance and low energy consumption characteristics of a data processor and the general flexibility of an instruction processor.

Description

Technical Field

[0001] The present invention relates to the field of neural network tensor processors, and in particular to a unified calculation method for general neural network tensor processors. Background Art

[0002] Existing processors and their computing methods can be roughly divided into two categories. One is based on instructions and uses a stored program computing model. Traditional CPUs, GPUs, or DSPs belong to this type of instruction processors; the other is based on data and uses a computing graph model. Previous patent 1 (the name of the invention is: A neural network multi-core tensor processor, application number: 202011423696.0) and previous patent 2 (the name of the invention is: A neural network tensor processor, application number: 202011421828.6) belong to this type of data processors.

[0003] The calculation of the instruction processor is centered around instructions, and its minimum calculation granularity is a single instruction. The calculation of the data processor is centered around data, and its minimum calculation granularity is usually an operator. The operator here refers to a functional operation whose calculation granularity and operand granularity are much larger than that of the instruction, such as convolution.

[0004] Typically, an instruction performs a single computational function (such as an addition operation) on an operand (such as a 32-bit number or a 64-bit number). An operator performs a complex computational function (such as a convolution operation, which consists of multiple multiplications and additions) on a series of operands (such as tensor data with length, width, and height being H, W, and C, respectively). Therefore, compared with processors that natively support instruction computations and processors that natively support operator computations, the internal structures of processors that natively support instruction computations differ greatly, and the corresponding computational methods are also different.

[0005] For the two completely different types of processors mentioned above, the calculation methods are completely different in the calculation of neural network algorithms. For instruction processors, operators can be simulated and run by combining instructions, which has high versatility, but their computing performance is usually low or their computing energy consumption is high (for example, neural network algorithms can be executed by CPU, but the speed is usually very slow, and neural network algorithms can be executed by GPU, but the energy consumption is usually very high). For data processors, the calculation of operators can be natively supported, with high performance and low energy consumption. However, it cannot support the calculation of instructions, so it is often limited in the versatility of calculation. Summary of the invention

[0006] In view of the shortcomings of the above existing solutions, the present invention proposes a unified calculation method for a general neural network tensor processor, which can execute operators and instructions simultaneously in the general neural network tensor processor, so that the system has both the high performance and low energy consumption characteristics of a data processor and the general flexibility of an instruction processor. The general neural network tensor processor combines the characteristics of an instruction processor and a data processor, and contains a data flow kernel and an instruction flow kernel, wherein the data flow kernel can execute the above operators, and the instruction flow kernel can execute the above instructions.

[0007] The specific technical solutions proposed by the present invention are as follows:

[0008] A unified computing method for a general-purpose neural network tensor processor simultaneously implements instruction computing and operator computing in a unified environment and method, and expresses the contents of neural network algorithms and applications in the form of operators, wherein the calculations related to the neural network algorithm are expressed in the form of neural network operators, and the calculations related to the neural network application are expressed in the form of custom instruction operators.

[0009] Furthermore, the neural network operator refers to the coarse-grained, modular functional operation used in the neural network algorithm. The neural network operator cannot be disassembled and is executed in the data flow kernel of the general neural network tensor processor. The neural network operator is ultimately mapped to direct calculation operations of certain circuits in the data flow kernel.

[0010] Furthermore, the custom instruction operator refers to a functional operation used in a neural network application, which is executed in the instruction stream kernel of a general neural network tensor processor. The custom instruction operator is ultimately converted into an equivalent instruction combination, and each instruction is mapped to a direct calculation operation of certain circuits in the instruction stream kernel.

[0011] Furthermore, the operator is used to obtain input data or intermediate data, and calculate output data or intermediate data according to a calculation method determined by the operator type. According to the type of different operators, some operators also have parameters, which exist as another implicit input of the operator.

[0012] Furthermore, the calculations related to the neural network algorithm are also expressed in the form of composite operators. When deployed, the composite operator establishes a mapping relationship with the hardware in the following manner: the composite operator is mapped to an equivalent operator combination consisting of a neural network operator and a custom instruction operator, and the conversion rules between the composite operator and the neural network operator and the custom instruction operator are written into the compilation software.

[0013] Furthermore, the composite operator includes a neural network operator, a custom instruction operator, and optional items: composite parameters and composite data; wherein the composite parameter is a parameter obtained by training the composite operator, and the parameter will be decomposed into multiple parts according to the needs of the internal operator of the specific composite operator, and respectively allocated to the corresponding internal operators for use as parameters; the composite data is an additional data path between the internal operator of the composite operator and the external operator.

[0014] Furthermore, the composite operator has a parallel mechanism, and the composite operator includes multiple neural network operators with the same functions, which are used to realize parallel computing.

[0015] Furthermore, the composite operator has an iterative mechanism for realizing the nesting of composite operators. In the composite operator mapping process completed by the compiler software, all nested composite operators will be expanded and finally merged into a new composite operator consisting only of neural network operators and custom instruction operators.

[0016] Furthermore, the general neural network tensor processor is abstracted as a processor including a data flow kernel and an instruction flow kernel, wherein the data flow kernel is used to execute the neural network operator, and the instruction flow kernel is used to execute the custom instruction operator.

[0017] The present invention achieves the following technical effects:

[0018] The unified computing method for a general-purpose neural network tensor processor proposed in the present invention expresses all contents of neural network algorithms and applications in the form of operators, so that operators and instructions can be executed simultaneously in a general-purpose neural network tensor processor, so that the system has both the high performance and low energy consumption characteristics of a data processor and the general flexibility of an instruction processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The neural network algorithm of the present invention is composed of a neural network operator and a custom instruction operator;

[0020] Figure 2 It is a neural network algorithm with composite operators;

[0021] Figure 3 is an example of a transformation that is a composite operator;

[0022] Figure 4 is another example of a transformation using a composite operator;

[0023] Figure 5 This is an example of nested composite operators. DETAILED DESCRIPTION

[0024] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0025] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0026] The present invention proposes a unified computing method for a general neural network tensor processor, which can realize instruction computing and operator computing in a unified environment and method. The core idea of ​​the unified computing method is to express all the contents of the neural network algorithm and application in the form of operators, wherein the calculations related to the neural network algorithm are expressed in the form of neural network operators, and the calculations related to the neural network application are expressed in the form of custom instruction operators.

[0027] The neural network operator refers to the coarse-grained functional operation used in the neural network algorithm. Neural network operators usually have a certain degree of versatility, that is, different neural network algorithms will use the same type of operators, such as typical convolution, pooling, activation, etc.

[0028] The custom instruction operator refers to the functional operation used in the neural network application. Custom instruction operators are usually not universal, that is, custom instruction operators are usually custom developed by developers according to application requirements. Different neural network applications use different custom instruction operators, such as data input, data output, data screening, etc. Custom instruction operators are defined as operators custom developed by developers according to application requirements.

[0029] A neural network algorithm consisting of a neural network operator and a custom instruction operator is as follows: Figure 1 As shown. Among them, the input data is the input of the neural network algorithm, and the output data is the output of the neural network algorithm. The main function of each operator (including custom instruction operators and neural network operators) is to obtain input data or intermediate data, and calculate the output data or intermediate data according to the calculation method determined by the operator type (such as convolution). Among them, according to the type of different operators, some operators may also have parameters, which exist as another implicit input of the operator.

[0030] The above neural network algorithm composed of a neural network operator and a custom instruction operator can be used for a new type of general neural network tensor processor. The general neural network tensor processor can be abstracted as a processor including a data stream kernel and an instruction stream kernel, which can execute neural network operators and custom instruction operators. Among them, the neural network operator is executed in the data stream kernel. The neural network operator cannot be disassembled, and it is ultimately mapped to a direct operation of certain circuits in the data stream kernel; the custom instruction operator is executed by the instruction stream kernel. The custom instruction operator is ultimately converted into an equivalent instruction combination, and each instruction is mapped to a direct operation of certain circuits in the instruction stream kernel.

[0031] Furthermore, the neural network algorithm can also be composed of composite operators. The composite operator refers to a new operator formed by combining the original neural network operator with the custom instruction operator. The composite operator can provide new computing functions, and the new computing function does not need to increase or modify the native hardware architecture of the general neural network tensor processor, but by using different parts of the native architecture of the general neural network tensor processor (that is, the operation of different operators) respectively, and by combining multiple neural network operators and custom instruction operators to achieve more complex calculations.

[0032] From the perspective of neural network structure and algorithm training, the neural network algorithm containing composite operators is no different from the ordinary neural network algorithm. Figure 2 As shown in the figure. A composite operator is defined as a new operator with independent software implementation, and its parameters can be obtained by training algorithm training tools (such as Pytorch). Therefore, from the perspective of neural network algorithm training, there is no difference between composite operators and neural network operators and custom instruction operators.

[0033] However, in the neural network algorithm reasoning deployment phase based on the general neural network tensor processor, since there is no direct mapping relationship between the composite operator and the hardware, the composite operator is mapped to a combination of equivalent operators consisting of the neural network operator and the custom instruction operator. The conversion rules between the composite operator and the neural network operator and the custom instruction operator are determined by the designer during the composite operator definition phase and written into the compiler software responsible for the conversion task.

[0034] Figure 3 This is an example of a composite operator conversion. From a basic structure, a composite operator consists of a neural network operator / custom instruction operator, composite data, and composite parameters. The composite data and composite parameters are not necessary. Depending on the requirements of a specific composite operator, a composite operator can consist of only a neural network operator / custom instruction operator.

[0035] The specific function of the composite operator is determined by the combination of the neural network operator and the custom instruction operator. The composite parameter is the parameter obtained by the composite operator training. The parameter will be split into multiple parts according to the needs of the specific internal operators of the composite operator and allocated to the corresponding internal operators as parameters for use.

[0036] In addition, in neural network algorithms that do not contain composite operators, there are only data paths of input data / intermediate data and output data / intermediate data between operators. However, in the operation process of some composite operators, there are often additional hidden data paths (typically such as hidden and cell data of LSTM). Therefore, in neural network algorithms that contain composite operators, composite data is used to represent these possible hidden data paths. The composite data is used or updated by internal operators and can also be used or updated by external operators. Composite data is an additional data path between the internal operators of the composite operator and the external operators (in addition to the traditional input data, output data and intermediate data paths).

[0037] Figure 3 The composite operator shown is composed of multiple neural network operators and custom instruction operators with different functions. Its main purpose is to realize new operators with more complex functions. Figure 4 is another example of composite operator transformation, which consists of multiple neural network operators with the same function. Its main purpose is to achieve parallel acceleration by splitting data and parameters into small parts and assigning them to multiple identical neural network operators for execution (in multi-core processors, Figure 4 The multiple neural network operators A within the composite operator shown can be simultaneously assigned to multiple processor cores for parallel computing).

[0038] Composite operators have a nesting mechanism, which can realize neural network algorithms that contain composite operators. An example of composite operator nesting is as follows: Figure 5 As shown in the figure, the nesting level of composite operators is infinite. In the composite operator mapping process completed by the compiler software, all nested composite operators will be expanded and eventually merged into a new composite operator consisting of only neural network operators and custom instruction operators.

[0039] Based on the above composite operator mechanism, by combining neural network operators / custom instruction operators, composite parameters and composite data, all complex operator calculations of neural networks can be realized, such as LSTM, GRU, Transformer, etc. At the same time, many traditional non-neural network operators can also be realized, such as mean filter operator, Gaussian filter operator, Sobel operator, etc., which greatly enriches the flexibility of system functions and improves the scalability of system functions.

[0040] The present invention relates to a general neural network tensor processor, which is abstracted as a processor including a data stream kernel and an instruction stream kernel, wherein the data stream kernel can execute the above-mentioned operator, and the instruction stream kernel can execute the above-mentioned instruction. The unified calculation method proposed in the present invention can execute operators and instructions simultaneously in the general neural network tensor processor, so that the system has both the high performance and low energy consumption characteristics of a data processor and the general flexibility of an instruction processor.

[0041] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A unified computing method for a general neural network tensor processor, characterized in that: In the general neural network tensor processor, instruction calculation and operator calculation are implemented simultaneously in a unified environment and method, and the contents of the neural network algorithm and application are all expressed in the form of operators, wherein the calculation related to the neural network algorithm is expressed in the form of neural network operators, and the calculation related to the neural network application is expressed in the form of custom instruction operators; The neural network operator refers to a coarse-grained, modular functional operation used in a neural network algorithm. The neural network operator cannot be disassembled and is executed in the data flow kernel of a general neural network tensor processor. The neural network operator is ultimately mapped to a direct operation of certain circuits in the data flow kernel; The custom instruction operator refers to a functional operation used in a neural network application, which is executed in the instruction stream kernel of a general neural network tensor processor. The custom instruction operator is ultimately converted into an equivalent instruction combination, and each instruction is mapped to a direct operation of certain circuits in the instruction stream kernel; The operator is used to obtain input data or intermediate data, and calculate output data or intermediate data according to a calculation method determined by the operator type. According to different operator types, some operators also have parameters, which exist as another implicit input of the operator.

2. The unified calculation method according to claim 1, characterized in that: The calculations related to the neural network algorithm are also expressed in the form of composite operators. When deployed, the composite operator establishes a mapping relationship with the hardware in the following way: the composite operator is mapped to an equivalent operator combination consisting of a neural network operator and a custom instruction operator, and the conversion rules between the composite operator and the neural network operator and the custom instruction operator are written into the compilation software.

3. The unified calculation method according to claim 2, characterized in that: The composite operator includes a neural network operator, a custom instruction operator, and optional items: composite parameters and composite data; wherein the composite parameter is a parameter obtained by training the composite operator, and the parameter will be decomposed into multiple parts according to the needs of the internal operator of the specific composite operator, and respectively allocated to the corresponding internal operators for use as parameters; the composite data is an additional data path between the internal operator of the composite operator and the external operator.

4. The unified calculation method according to claim 3, characterized in that: The composite operator has a parallel mechanism, and the composite operator includes a plurality of neural network operators with the same functions, which are used to realize parallel computing.

5. The unified calculation method according to claim 2, characterized in that: The composite operator has an iterative mechanism for realizing the nesting of composite operators. In the composite operator mapping process completed by the compiler software, all nested composite operators will be expanded and finally merged into a new composite operator consisting only of a neural network operator and a custom instruction operator.

6. The unified calculation method according to claim 1, characterized in that: The general neural network tensor processor is abstracted as a processor including a data flow kernel and an instruction flow kernel, wherein the data flow kernel is used to execute the neural network operator, and the instruction flow kernel is used to execute the custom instruction operator.

Images