A complex instruction set microarchitecture for neural network processors

Abstract

The present disclosure discloses a neural network processor complex instruction set micro-architecture, including: a general-purpose processor, an instruction dispatch unit (DISP), a computation instruction and format conversion instruction queue (IQ) (ALU), a memory access instruction LDMR instruction queue (IQ) (LDMR), a memory access instruction SVMR instruction queue (IQ) (SVMR), a write-back module (Retire), a scoreboard, a parameter storage module (OP_param_talbe), a data storage module, a memory access LDMR execution unit, a memory access SVMR execution unit, a CONV scheduling unit, a DWCL scheduling unit, an MV scheduling unit, an FTRANS scheduling unit, a matrix processing unit (MPU), and a vector processing unit (VPU). This micro-architecture enables concurrent execution of multiple operator calculations in a neural network, reducing off-chip memory bandwidth consumption.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence neural network technology, and in particular relates to a complex instruction set micro-architecture of a neural network processor. Background Art

[0002] In recent years, the demand for edge-side computing of neural networks has been growing. Due to the extremely high power consumption requirements of edge-side computing, GPUs with high computing power consumption are difficult to apply. However, dedicated computing chips ASICs have long development cycles and high costs, making it difficult to quickly apply them to meet the growing demand for neural network computing. FPGAs, due to their programmability, have become an important device for edge-side neural network computing.

[0003] Currently, there are many specific applications for edge-side neural network computing on FPGAs. However, most applications have the disadvantage of being highly customized for specific networks. Replacing a network requires re-modifying the RTL code, making simulation verification and deployment difficult and time-consuming. Summary of the Invention

[0004] In view of this, the present disclosure provides a neural network processor complex instruction set micro-architecture, including: a general-purpose processor, an instruction dispatch unit DISP, a computing instruction and format conversion instruction queue IQ (ALU), a memory access instruction LDMR instruction queue IQ (LDMR), a memory access instruction SVMR instruction queue IQ (SVMR), a write-back module Retire, a scoreboard, a parameter storage module OP_param_talbe, a data storage module, a memory access LDMR execution unit, a memory access SVMR execution unit, a CONV scheduling unit, a DWCL scheduling unit, an MV scheduling unit, an FTRANS scheduling unit, a matrix processing unit MPU and a vector processing unit VPU;

[0005] The IQ (LDMR) stores all LDMR instructions; the IQ (SVMR) stores all SVMR instructions; the IQ (ALU) stores Conv instructions, DWCL instructions, MV instructions, FTRANS instructions and SETPARAM instructions.

[0006] Preferably,

[0007] The LDMR instruction is responsible for moving data from the DDR to the x-th matrix operation register MR[x]. The instruction format is as follows:

[0008] LDMR Len,MR d ,DDR_addr

[0009] Among them, MR dIndicates the target start address of the data to be moved to MR[x], in units of 64B; DDR_addr indicates the start address from DDR, in units of 1B; Len indicates the length of the data to be moved, in units of 64B;

[0010] The encoding format of the LDMR instruction is as follows:

[0011]

[0012] The op_code is encoded as 6′b000101, and op_size is the length of the instruction, indicating how many words the instruction occupies.

[0013] Preferably,

[0014] The SVMR instruction is responsible for moving data from the x-th matrix operation register MR[x] to the DDR. The instruction format is as follows:

[0015] SVMR Len, MR s ,DDR_addr

[0016] Among them, MR s Indicates the starting address of data to be moved from MR[x], in units of 64B; DDR_addr indicates the target starting address of data to be moved to DDR, in units of 1B; Len indicates the length of the data to be moved, in units of 64B;

[0017] The encoding format of the SVMR instruction is as follows:

[0018]

[0019] Among them, op_code is encoded as 6′b000110, and op_size is the length of the instruction, indicating how many words the instruction occupies.

[0020] Preferably,

[0021] The CONV instruction is responsible for performing a standard convolution operation on the data. The instruction format is as follows: CONV op_param, wt_Size, bs_Size, dSize, sLen, MR d , MR wt , MR bs , MR s1 , {MR S[n]}

[0022] Among them, op_param is a pointer to the operand parameter table OP_param_table, indicating which set of parameters should be used for this convolution operation; wt_Size and bs_Size respectively represent the number of indexes occupied by the weight Weight and bias Bias data of the current convolution operation, counting from 0; dSize represents the number of indexes occupied by the structure data of the current convolution operation on the x-th matrix operation register MR[x], counting from 0; sLen is the number of indexes occupied by each row of source data of the current convolution operation, counting from 0; MR d , MR s[n] MR is the starting address of the result data and source data; bs , MR wt is the starting address of offset and weight;

[0023] The CONV instruction encoding format is as follows:

[0024]

[0025] Among them, op_code is encoded as 6′b001000, MR sn and MR sn' respectively represent the starting addresses of the neural network processor Core0 and the neural network processor Core1, and op_size is the length of the instruction, indicating how many words the instruction occupies.

[0026] Preferably,

[0027] The DWCL instruction is responsible for performing channel-independent operations on the data, including depth-wise convolution and pooling operations. The instruction format is as follows:

[0028] DWCL op_param, wt_Size, bs_Size, dSize, sLen, MR d , MR wt , MR bs , MR s1 , {MR s[n]}

[0029] Among them, op_param is a pointer to the operand parameter table OP_param_table, indicating which set of parameters needs to be used for this operation; wt_Size and bs_Size respectively represent the number of index lndex occupied by the weight Weight and bias Bias data of the current convolution operation; dSize represents the number of index lndex occupied by the result data of this operation; sLen is the number of index lndex occupied by each row of source data in the current operation; MRd , MR s[n] MR is the starting address of the result data and source data; bs , MR wt is the starting address of offset and weight;

[0030] The DWCL instruction encoding format is as follows:

[0031]

[0032] Among them, op_code is encoded as 6′b001001, MR sn and MR sn' respectively represent the starting addresses of the neural network processor Core0 and the neural network processor Core1, and op_size is the length of the instruction, indicating how many words the instruction occupies.

[0033] Preferably,

[0034] The MV instruction is responsible for moving the data in the x-th matrix operation register MR[x], and converting or rearranging the data according to the parameters during the movement. The instruction format is as follows:

[0035] MV op_param, dSize, sLen, MR d , MR s

[0036] Among them, op_param is a pointer to the operand parameter table OP_param_table, which indicates the parameters of the data to be moved this time; dSize is the number of indexes occupied by the result data; sLen is the number of indexes occupied by the source data; MRd and MRs are the starting addresses of the result data and source data;

[0037] The MV instruction encoding format is as follows:

[0038]

[0039] Among them, op_code is encoded as 6′b001010, and op_size is the length of the instruction, indicating how many words the instruction occupies.

[0040] Preferably,

[0041] The FTRANS instruction is responsible for data format conversion between ch64 format and ch8 format. The instruction format is as follows:

[0042] FTRANS op_param,dSize,sLen,MR d , MR s

[0043] Among them, op_param is a pointer to the operand parameter table OP_param_table, which indicates the parameters when performing Format conversion; dSize is the number of index lndex occupied by the result data; sLen is the number of index lndex occupied by the source data; MR d , MR s The starting address of the result data and source data;

[0044] The FTRANS instruction encoding format is as follows:

[0045]

[0046] Among them, op_code is encoded as 6′b001011, and op_size is the length of the instruction, indicating how many words the instruction occupies.

[0047] Preferably,

[0048] The SETPARAM instruction is responsible for configuring the parameters of each operator. This instruction can be used to write to any index of the operand parameter table OP_param_table. The instruction format is as follows:

[0049] SETPARAM op_param, Pd0, {Pd[n]}

[0050] Wherein, op_param is the index address of the operand parameter table OP_param_table to be written; Pdn is the data to be written;

[0051] The SETPARAM instruction encoding format is as follows:

[0052]

[0053] Among them, op_code is encoded as 6′b001100, and op_size is the length of the instruction, indicating how many words the instruction occupies.

[0054] Preferably,

[0055] The x-th matrix operation register MR[x] has three types, namely input feature map matrix register MR fm , weight matrix register MR wt and bias matrix register MR bs .

[0056] Preferably,

[0057] Memory MR fmContains 32 indexes, denoted as MR0 to MR31, each index is a section of SRAM with a width of 64B and a depth of 128; the memory MR wt Contains 32 indexes lndex, and MR fm Unified addressing, recorded as MR32~MR63; memory MR bs Contains 16 indexes, uniformly addressed as MR64 to MR79.

[0058] Through the above technical solution, the instruction set and microarchitecture are deployed on the FPGA, and the entire network calculation can be completed through software programming on the RISC-V processor. At the same time, through software optimization, multiple operators of the neural network can be calculated concurrently, reducing the consumption of off-chip memory bandwidth. In addition, since the software is programmable, after the hardware design is fixed, the software can be modified to meet the certain computational adaptation of some new neural networks introduced by subsequent algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a structural diagram of a complex instruction set micro-architecture of a neural network processor provided in one embodiment of the present disclosure;

[0060] Figure 2 This is a diagram of the Scoreboard structure in one embodiment of the present disclosure;

[0061] Figure 3 It is an MPU that performs convolution calculation graph (matrix multiplication by matrix) in one embodiment of the present disclosure;

[0062] Figure 4 The MPU performs a convolution operation (vector multiplication by matrix) in one embodiment of the present disclosure;

[0063] Figure 5 is a structural diagram of MRA, MRB and MRC in one embodiment of the present disclosure;

[0064] Figure 6 This is a structural diagram of the CONV calculation internal control state machine in one embodiment of the present disclosure;

[0065] Figure 7 This is a structural diagram of the DWCL calculation internal control state machine in one embodiment of the present disclosure;

[0066] Figure 8 is a flowchart of MV control scheduling in one embodiment of the present disclosure;

[0067] Figure 9 is a diagram of the composition format of each instruction in one embodiment of the present disclosure;

[0068] Figure 10is a storage structure diagram of a CISC instruction set in one embodiment of the present disclosure;

[0069] Figure 11 is a diagram of the calculation process of a convolutional neural network in one embodiment of the present disclosure;

[0070] Figure 12 This is a diagram of resource occupancy of a dual-core deployment of K7-410T in one embodiment of the present disclosure;

[0071] Figure 13 This is a diagram of resource usage in a quad-core deployment of a V7690T in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0072] In one embodiment, Figure 1 As shown, it discloses a neural network processor complex instruction set microarchitecture, including: a general processor, an instruction dispatch unit DISP, a computing instruction and format conversion instruction queue IQ (ALU), a memory access instruction LDMR instruction queue IQ (LDMR), a memory access instruction SVMR instruction queue IQ (SVMR), a write-back module Retire, a scoreboard, a parameter storage module OP_param_talbe, a data storage module, a memory access LDMR execution unit, a memory access SVMR execution unit, a CONV scheduling unit, a DWCL scheduling unit, an MV scheduling unit, an FTRANS scheduling unit, a matrix processing unit MPU and a vector processing unit VPU;

[0073] The IQ (LDMR) stores all LDMR instructions; the IQ (SVMR) stores all SVMR instructions; the IQ (ALU) stores Conv instructions, DWCL instructions, MV instructions, FTRANS instructions and SETPARAM instructions.

[0074] In this embodiment, the instruction set in this microarchitecture design is complex and dynamically generated by the processor. This allows for updates to the instruction set and software to meet the needs of future applications or computing network development. While the current instruction design uses 6 bits for op_code, this design only uses the encoding space for 7 of these instructions, providing the hardware foundation for instruction expansion. The instruction encoding in this microarchitecture design takes multi-core computing into account, reserving instruction encoding space for multi-core computing, thus effectively supporting multi-core computing expansion. Dual-core design has been considered in this design. Similar approaches can be used for designs with more cores. The instruction-based computing process in this microarchitecture design also implements neural network computing through software programming, thus supporting a wider range of neural networks. The storage model in this microarchitecture design can be increased or decreased based on the selected FPGA resources to meet the computing requirements of images with different input resolutions. This microarchitecture design does not describe the front-end and back-end computations of neural networks. However, by designing complex instructions and adding functionality, some front-end and back-end computations can be extended to instruction execution.

[0075] This microarchitecture design and its instruction set are suitable for FPGA neural network computing deployment, but the described instruction set microarchitecture can also be applied to AISC design. It is mainly used for accelerated neural network computing.

[0076] The CISC instruction hardware structure design includes a general-purpose processor, an instruction dispatch unit Dispatch (DISP for short), a calculation instruction and format conversion instruction queue (Issue Queue (Algorithm logic unit, IQ (ALU) for short), a memory access instruction LDMR instruction queue (Issue Queue (LDMR, IQ (LDMR) for short), a memory access instruction SVMR instruction queue (IssueQueue (SVMR, IQ (SVMR) for short), a write-back module (Retire), a scoreboard Socore board, a parameter storage module OP_param_talbe[31:0], a data storage module (including MRA, MRB and BR), a memory access LDMR execution unit, a memory access SVMR execution unit, a CONV scheduling unit, a DWCL scheduling unit, a MV scheduling unit and a FTRANS scheduling unit, a matrix processing unit (MPU for short), and a vector processing unit (VPU for short).

[0077] The DISP stores a string of CISC instructions (which can be dynamically generated by the processor). The general-purpose processor decides whether to issue a CISC instruction based on the available space in the DISP. When an instruction is present in the DISP, the DTSP identifies its type and dispatches it to the Issue Queue (IQ) of the ALU, LDMR, and SVMR based on the instruction type. Simultaneously, the instruction is sent to the Retire module.

[0078] These three IQs store different types of instructions. The LDMR queue stores all LDMR instructions; the SVMR module stores all SVMR instructions; and the ALU module stores all other instructions, including CONV, DWCL, MV, FTRANS, and SETPARAM. All three IQs are in-order execution queues, and can execute out of order. Whether instructions in the three IQs can be sent to subsequent functional units or function scheduling units depends primarily on whether the Matrix Registers in the instructions have dependencies. This means that the instructions in the three IQs must determine whether their operands have read-after-write (RAW) dependencies, write-after-read (WAR) dependencies, or write-after-write (WAW) dependencies. If these dependencies are absent, the instructions in the IQ queues can be sent to the subsequent functional execution unit or function scheduling unit for pipeline processing. In this dependency judgment, there is no need to judge the dependency of the processor's general-purpose architecture registers, because the dependency of the general-purpose architecture processor has been checked and released within the processor. In other words, when the general-purpose processor writes instructions into the DISP module, there is no dependency between these instructions on the general-purpose registers.

[0079] The Scoreboard mainly records the read and write properties of the Matrix Register of all instructions in the IQ queue. Its structure is as follows: Figure 2As shown. The first row in the table is the Matrix Register in the instruction, with a total of 32 records, from 001 to 011, each row represents an instruction. The vertical intersection box of each row and MRx (x fetches instructions 0 to 31) records the read and write properties of the Matrix Register in the current instruction row. To determine whether an instruction can be sent from the IQ to the subsequent functional unit or function scheduling unit and to release the dependency of the instruction, it mainly depends on whether the MR register of the current instruction row and the MR register of the instruction before the instruction row (the so-called before refers to the row with a smaller instruction number than the current row) have correlations such as WAR, RAW, and WAW. If there are no such correlations, the instruction can continue to be sent to the subsequent functional execution unit or function scheduling unit. For instruction number 005, the MR0 register has the W (write) attribute, while the MR0 columns of instructions 004, 003, 002, and 001 have neither the W attribute nor the R (read) attribute record. Therefore, the MR0 register in instruction 005 has no WAR, RAW, or WAW dependencies on the instructions before it, so instruction 005 can be directly sent to the subsequent function execution unit or function scheduling unit. Instruction 007 has R attributes on both MR0 and MR1 registers, while instruction 006 has W attributes on MR1 and W attributes on MR0 of instruction 005. This indicates that instruction 007 needs to wait for instruction 005 to be executed before the W attribute of MR0 is released, and needs to wait for instruction 006 to be executed before the W attribute of MR1 is released. Therefore, instruction 007 can only be sent to the subsequent functional unit or function scheduling unit for execution after instructions 005 and 006 are all executed. The final implementation of Scoreboad is a fifo (First In First Out) structure, where Head identifies the head of the fifo (indicating the instruction that entered the fifo earliest but has not been executed or has been executed but the space has not been released), and Tail identifies the tail of the fifo (indicating the instruction that entered the fifo latest but has not been executed).

[0080] The OP_param_table stores all operator parameters. The op_parameter_table divides its parameter storage into 32 indices, designated P0 through P31. Each index has a data bit width of 4 bytes and a depth of 5. The Retire module accepts pending instructions from the DISP and receives signals from the three IQs indicating instruction completion, thereby completing the instruction retirement operation. When all instructions have been retired, a DISP empty signal is sent to the general processor. Based on this full signal and the remaining space in the DISP, the general processor decides whether to continue sending CTSC instructions to the DISP module.

[0081] The MPU execution unit mainly performs a Figure 3 The convolution calculation process for two 8-row, 8-column matrices is shown. Matrices A, B, and C are all 8-row, 8-column matrices. Taking A as an example, each of the 8 horizontal rectangles in A contains 8 elements, and each element contains 8 bytes of data, making matrix A an 8*8 matrix with a total of 64 elements. Matrices B and C also have the same structure as matrix A.

[0082] The MPU execution unit can also execute an Figure 4 The convolution calculation process of a matrix with 1 row and 8 columns and a matrix with 8 rows and 8 columns is shown. Matrix E is a matrix with 1 row and 8 columns, matrix F is a matrix with 8 rows and 8 columns, and matrix G is a matrix with 1 row and 8 columns. Each element of matrices E, F, and G is 8 bytes of data.

[0083] The VPU mainly performs operations such as quantization, pooling, and bias addition on channel results.

[0084] MRA, MRB, and MRC are all on-chip storage structures. Their specific structures are as follows: Figure 5 As shown. MRx (x refers to A, B, C) consists of n banks, each bank has the same structure, and is composed of four parts a / b / c / d. For n banks of MRx, we consecutively number the structure a / b / c / d of each bank, which constitutes MR0, MR1, MR2, ..., MR31, ..., MR 4n-1; consecutively numbering a / b / c / d of MRA, MRB, and MRC constitutes a numbered storage structure from MR0 to MR79. In other words, MRfm, MRwt, and MRbs are Figure 5 The result of consecutive numbering according to the above Bank organization form.

[0085] like Figure 6As shown in the figure, the CONV module performs calculations according to the input feature map rows covered by a convolution kernel granularity, which is implemented through a state machine. There are three states in total: IDEL state, calculation state (calculation process in the irregular dotted box), and calculation completion state (feedback calculation completion). When the instruction in IQ is sent to the CONV control scheduling module and the start_calc calculation starts, CONV switches from the IDLE state to the "calculation state". In the calculation state, the most basic calculation granularity of the CONV module is a parallel calculation of "8×8×8". The channel direction calculation of the input feature map is performed first. The channel direction calculation of the feature map is actually a channel / 8 loop calculation process. If the calculation of the direction of the input feature map is completed, the convolution kernal width direction is calculated, and when the kernal width direction is switched, the granularity calculation of "8×8×8" is entered again, and the channel direction calculation of the feature map is performed again, and then a kernal width direction loop calculation is entered. Each kernal width direction loop calculation is actually a feature map input channel / 8 loop calculation. After completing the kernal width direction calculation, the kernal hight direction calculation is performed again, and when the kernal hight direction is switched, the granularity calculation of "8×8×8" is entered again, and then a new kernal width calculation loop and the input feature map channel direction calculation loop are entered. After the hight direction calculation, 8 groups of outputs in the output feature map channel / 8 group are obtained. Among these 8 groups of outputs, each group contains 8 output channels; further perform the channel / 8 group calculation of the out feature map. If the channel / 8 group calculation of ofm is not completed, continue the cyclic calculation including the aforementioned several cyclic calculation processes until the channel / 8 group calculation of ofm is completed; after the channel / 8 calculation of ofm is completed, enter the width direction calculation of the input feature map. The width calculation of the input feature map is still a width / 8 group calculation. The calculation granularity in this group is a cycle of all the aforementioned cyclic calculations. When the width / 8 group calculation of the input feature map can be completed, the input feature map row calculation covered by a convolution kernel granularity is completed.

[0086] For Depthwise convolution, the control of its calculation process is similar to the CONV calculation scheduling process mentioned above. The difference is that the control state machine of DWC calculation is as follows Figure 7 As shown, it can be obtained from Figure 7 It can be seen that the calculation process of DWC does not require the channel / 8 group calculation loop process of the output feature map. Therefore, the number of output feature map channels of depth-wise convolution is equal to the number of input feature map channels, and there is no need to accumulate the convolution calculation results between input channels. Other than this, the actual calculation process of DWC is the same as the calculation process of CONV mentioned above.

[0087] like Figure 8 As shown in the figure, the MV control scheduling module mainly reads the source data of the data length indicated by sLen from the register address indicated by MRs according to the source register in the instruction. Then, it reads the parameter parameter from the param_table according to the op_param pointer and performs operations on the source data according to the parameter parameter. These operations are defined by the software. After the operation is completed, the data is written to the MRd register.

[0088] The execution process of the FTRANS control scheduling module is similar to that of the MV control scheduling module. The difference is that the param_table of the FTRANS instruction is different from the param_table of the MV instruction, that is, the operations defined by these two sets of parameters can be different. Different operations (such as transposition operations and data compensation operations in MV instructions, and channel format conversion operations in FTRANS instructions) can be supported separately by hardware, thereby accelerating them by hardware execution.

[0089] In another embodiment, queue IQ(LDMR) stores all LDMR instructions; queue IQ(SVMR) stores all SVMR instructions; and queue IQ(ALU) stores all other instructions.

[0090] In this embodiment, the complex instruction set can be generated with the assistance of processor software code, and the complex instruction set CISC code is written into the specific implementation of the microarchitecture. The microarchitecture manages the disorder and dependencies of the instructions and concurrently executes the reading, writing, calculation and other processes. The instruction format of the CISC instruction set used is summarized as follows:

[0091]

[0092]

[0093]

[0094] SETPARAM op-param{Pd[n]}

[0095] The encoding of CISC instructions is a string of code words in word (4B) units, using the little endian format. The format of each instruction is as follows Figure 9 As shown. The format of the first word, INST 0, is the same, and the subsequent words vary according to the definition of op_code. INST0 mainly includes the dSize field (5 bits), bs_Size field (5 bits), wt_Size field (5 bits), op_para field (5 bits), op_size field (6 bits), and op_code field (6 bits), a total of 6 fields. The dSize field identifies the number of destination MR registers. This field has a total of 5 bits and can indicate up to 32 MR registers. The bs_Size field identifies the number of MR registers occupied by the bias used in the current calculation. This field has a total of 5 bits and can indicate up to 32 MR registers. The wt_Size field identifies the number of MR registers occupied by the weight used in the current calculation. This field has a total of 5 bits and can indicate up to 32 MR registers. Op_param is a pointer to the param table. It has 5 bits and can indicate 32 locations. op_size indicates the total number of words in the instruction. This field has a total of 6 bits and can indicate up to 64 words. op_code indicates the type of instruction unique identification code.

[0096] The SET PARAM instruction can have n operands. In addition to the INST0 instruction code, it also has n-1 words that make up its instruction code. These n-1 words represent the parameter values. The number of n is determined by the op_size in the instruction code.

[0097] For CONV, DWCL, MV, and FTRANS instructions, there can also be n operands. In addition to the INST0 instruction code, these instructions also have n-1 words that constitute their instruction code. However, the INST1 of all these instructions has the same instruction format, that is, INST1 is as follows: Figure 9 (for CONV / DWCL / MV / FTRANS part): contains two 16-bit fields, one 16-bit field represents the MRd field; the other 16-bit field represents the sLen field. The other n-2 words, their composition also contains two 16-bit fields: one 16-bit field represents the MRd field; the other 16-bit field represents the sLen field. bs Or MRs domain; a 16-bit represents MRwt Or MRs field. The number of n is determined by the op_size of the instruction encoding.

[0098] For the LDMR and SVMR instructions, there are three operands. In addition to the INST0 instruction code, there are two words that constitute the instruction code. The instruction format of INST1 of these two instructions is as follows: Figure 9 (for LDMR / SVMR part): contains two 16-bit fields: one 16-bit field represents MRd; one field represents Len field. The format of INST2 instruction for these two instructions is as follows Figure 9 As shown in (for LDMR / SVMR part), they constitute a 32-bit DDR_addr field.

[0099] Note: All Size parameters are counted starting from 0. For example, when op_size is 0, it means the current instruction occupies 1 word, and so on. All Len and sLen parameters are counted starting from 1. The instruction length is the Len value.

[0100] The content within the {} in the above command format is optional. The following describes the combination of the above complex commands.

[0101] 1)

[0102] In this group of instructions, LDMR and SVMR are the instruction names. LDMR is the abbreviation of load matrix register, and SVMR is the abbreviation of save matrix register. Len is the first operand, which is the abbreviation of Length and refers to the data length; MR fm It is the second operand, which is the abbreviation of Matrix Register. fm refers to the feature map. Here, fm is used to identify MR to store the feature map. The second operand can also be MR wt , wt refers to the weight file weight, where wt is used to identify MR to store the weight file; the second operand can also be MR bs , bs refers to the bias file bias, where bs is used to identify MR used to store the bias file; the third operand is DDR_addr, which is the abbreviation of double data rate address and is used to store the address of the data in DDR. op size Refers to operation size, that is, the length of the operation code; wt sizeRefers to weight size, that is, the size of the weight data. These two parameters will be encoded into the instruction code. According to the instruction name and the type of the second operand, this group of instructions can be subdivided into the following six types: Among them, the LDMR instruction can be divided into three types according to the type of the second operand, specifically the LDMR instruction for the feature image feature map (abbreviated as fm), the LDMR instruction for the weight data Weight (abbreviated as wt), and the LDMR instruction for the bias data Bias (abbreviated as bs); the SVMR instruction can be divided into three types according to the type of the second operand, specifically the SVMR instruction for the feature map fm, the SVMR instruction for the weight data wt, and the SVMR instruction for the compilation data bs.

[0103] 2)

[0104] In this group of instructions, CONV and DWCL are the instruction names. CONV is the abbreviation of convolution, and DWCL is the abbreviation of Depthwise convolution. This group of instructions is a variable-length instruction, and the number of operands can be n. op_param is the first operand, which is the abbreviation of operation parameter; wt_Size is the second operand, which is the abbreviation of weight size; bs_Size is the third operand, which is the abbreviation of bias size; dSize is the fourth operand, which is the abbreviation of data Size; sLen is the fifth operand, which is the abbreviation of source Length; MRd fm It is the sixth operand, which is the abbreviation of Matrix Register destination. fm value is feature map. Here fm is used to identify MRd to store feature map. The sixth operand can also be MRd wt , wt refers to the weight file weight, where wt is used to identify MRd to store the weight file; MR wt It is the seventh operand. MR is the abbreviation of Matrix Register. wt refers to the weight file weight. Here wt is used to identify MR to store the weight file. MR bsIt is the eighth operand, MR is the abbreviation of Matrix Register, and bs is the bias file. Here, bs is used to identify MR as the storage of the bias file. MRs[n] is the nth operand. MRs refers to the abbreviation of Matrix Register source, and [n] represents the nth operand. Based on the instruction name and the type of the sixth operand, this group of instructions can be divided into four types: CONV instructions can be divided into two types based on the type of the sixth operand, specifically CONV instructions with the destination matrix register being the storage fm and CONV instructions with the destination matrix register being the storage wt. DWCL instructions can be divided into two types based on the type of the sixth operand, specifically DWCL instructions with the destination matrix register being the storage fm and DWCL instructions with the destination matrix register being the storage wt.

[0105] 3)

[0106] In this group of instructions, MV and FTRANS are the instruction names, MV is the abbreviation of move, and FTRANS is the abbreviation of format transform. op_param is the first operand; dSize is the second operand; sLen is the third operand; MRd fm The fourth operand can also be MRd wt MRs is the fifth operand. The abbreviations and meanings of these operands are the same as those described in the CONV instruction and the DWCL instruction. According to the instruction name and the type of the fourth operand, this group of instructions can be divided into four types: MV instructions can be divided into two types according to the type of the fourth operand, specifically MV instructions with the destination matrix register as the storage fm and MV instructions with the destination matrix register as the storage wt; FTRANS instructions can be divided into two types according to the type of the fourth operand, specifically FTRANS instructions with the destination matrix register as the storage fm and FTRANS instructions with the destination matrix register as the storage wt.

[0107] 4)SETPARAM op_param{Pd[n]}

[0108] In this instruction, SETPARAM is the instruction name, short for set parameter. op_param is the first operand, short for operation parameter. Pd[n] is the n+2th operation, where n starts at 0, and Pd is short for parameter destination.

[0109] In another embodiment, the LDMR instruction is responsible for moving data from DDR to MR[x], and the instruction format is as follows:

[0110] LDMR Len,MR d ,DDR_addr

[0111] Among them, MR d Indicates the target start address of the data to be moved to MR[x], in units of 64B; DDR_addr indicates the start address from DDR, in units of 1B; Len indicates the length of the data to be moved, in units of 64B;

[0112] The encoding format of the LDMR instruction is as follows:

[0113]

[0114] The op_code is encoded as 6′b000101. op_size is the length of the instruction, indicating how many words the instruction occupies. 0[19:0] means 20 zeros.

[0115] In this embodiment, when there are two neural network processors Core, MR d The highest bit, MR

[15] , is used to indicate which core is enabled. The SVMR instruction also uses this design.

[0116] In another embodiment, the SVMR instruction is responsible for moving data from MR[x] to DDR, and the instruction format is as follows:

[0117] SVMR Len,MRs,DDR_addr

[0118] Among them, MRs represents the starting address of the data to be moved from MR[x], in units of 64B; DDR_addr represents the target starting address of the data to be moved to DDR, in units of 1B; Len represents the length of the data to be moved, in units of 64B;

[0119] The encoding format of the SVMR instruction is as follows:

[0120]

[0121] Among them, op_code is encoded as 6′b000110, op_size is the length of the instruction, indicating how many words the instruction occupies, and 0[19:0] represents 20 zeros.

[0122] In another embodiment, the CONV instruction is responsible for performing a standard convolution operation on the data, and the instruction format is as follows:

[0123] CONV op_param, wt_Size, bs_Size, dSize, sLen, MRd, MRwt, MRbs, MRs1, {MRs[n]}

[0124] Among them, op_param is a pointer to the operand parameter table OP_param_table, indicating which set of parameters needs to be used for this convolution operation; wt_Size and bs_Size respectively represent the number of indexes occupied by the weight Weight and bias Bias data of the current convolution operation, counting from 0; dSize represents the number of indexes occupied by the structure data of the current convolution operation on the x-th matrix operation register MR[x], counting from 0; sLen is the number of indexes occupied by each row of source data of the current convolution operation, counting from 0; MRd and MRs[n] are the starting addresses of the result data and source data; MRbs and MRwt are the starting addresses of the offset and weight;

[0125] The CONV instruction encoding format is as follows:

[0126]

[0127] Among them, op_code is encoded as 6′b001000, MRsn and MRsn' represent the starting addresses of the neural network processor Core0 and the neural network processor Core1 respectively, and op_size is the length of the instruction, indicating how many words the instruction occupies.

[0128] In this embodiment, the target addresses of Core0 and Core1 are the same. Note that a neural network processor can contain multiple cores, so that the computing power can be smoothly expanded. The MR[x] in these cores is independently addressed, so each identical CISC instruction can be executed independently in these cores. At present, when performing calculations greater than 1x1, the input data of multiple cores may be different, so CONV and DWCL support specifying different input MRs for multiple cores. In addition, multiple cores use MR

[15] in LDMR and SVMR to distinguish which core performs the data transfer operation between MR and DDR.

[0129] MRsn and MRsn' represent the starting addresses of the neural network processor Core0 and the neural network processor Core1 respectively, where n can be 1, 2, 3, 4, ..., n-1, n.

[0130] In another embodiment, the DWCL instruction is responsible for performing channel-independent operations on data, including depth-wise convolution and pooling operations. The instruction format is as follows:

[0131] DWCL op_param, wt_Size, bs_Size, dSize, sLen, MRd, MRwt, MRbs, MRs1, {MRs[n]}

[0132] Among them, op_param is a pointer to the operand parameter table OP_param_table, indicating which set of parameters needs to be used for this operation; wt_Size and bs_Size respectively represent the number of index lndex occupied by the weight Weight and bias Bias data of the current convolution operation; dSize represents the number of index lndex occupied by the result data of this operation; sLen is the number of index lndex occupied by each row of source data in the current operation; MRd and MRs[n] are the starting addresses of the result data and source data; MRbs and MRwt are the starting addresses of the offset and weight;

[0133] The DWCL instruction encoding format is as follows:

[0134]

[0135] Among them, op_code is encoded as 6′b001001, MRsn and MRsn' represent the starting addresses of the neural network processor Core0 and the neural network processor Core1 respectively, and op_size is the length of the instruction, indicating how many words the instruction occupies.

[0136] MRsn and MRsn' represent the starting addresses of the neural network processor Core0 and the neural network processor Core1 respectively, where n can be 1, 2, 3, 4, ..., n-1, n.

[0137] In another embodiment, the MV instruction is responsible for moving the data in the x-th matrix operation register MR[x], and converting or rearranging the data according to the parameters during the movement. The instruction format is as follows:

[0138] MV op_param, dSize, sLen, MRd, MRs

[0139] Among them, op_param is a pointer to the operand parameter table OP_param_table, which indicates the parameters of the data to be moved this time; dSize is the number of index lndex occupied by the result data; sLen is the number of index Index occupied by the source data; MRd and MRs are the starting addresses of the result data and source data;

[0140] The MV instruction encoding format is as follows:

[0141]

[0142] Among them, op_code is encoded as 6′b001010, op_size is the length of the instruction, indicating how many words the instruction occupies, 0[9:0] means 10 zeros, and 0[15:0] means 16 zeros.

[0143] In another embodiment, the FTRANS instruction is responsible for data format conversion between ch64 format and ch8 format, and the instruction format is as follows:

[0144] FTRANS op_param, dSize, sLen, MRd, MRs

[0145] Among them, op_param is a pointer to the operand parameter table OP_param_table, which indicates the parameters when performing Format conversion; dSize is the number of indexes occupied by the result data; sLen is the number of indexes occupied by the source data; MRd and MRs are the starting addresses of the result data and source data;

[0146] The FTRANS instruction encoding format is as follows:

[0147]

[0148] Among them, op_code is encoded as 6′b001011, op_size is the length of the instruction, indicating how many words the instruction occupies, 0[9:0] means 10 zeros, and 0[15:0] means 16 zeros.

[0149] In another embodiment, the SETPARAM instruction is responsible for configuring the parameters of each operator. This instruction can be used to write to any index of the operand parameter table OP_param_table. The instruction format is as follows:

[0150] SETPARAM op_param, Pd0, {Pd[n]}

[0151] Wherein, op_param is the index address of the operand parameter table OP_param_table to be written; Pdn is the data to be written;

[0152] The SETPARAM instruction encoding format is as follows:

[0153]

[0154] Among them, op_code is encoded as 6′b001100, op_size is the length of the instruction, indicating how many words the instruction occupies, 0[9:0] means 10 zeros, and 0[15:0] means 16 zeros.

[0155] In this embodiment, Pdn is the data to be written, which is no more than 5 and exists in the instruction in the form of an immediate value. The n in Pdn can be 0, 1, 2, 3...n-1, n.

[0156] In another embodiment, the three queues, namely, queue IQ (ALU), queue IQ (LDMR), and queue IQ (SVMR), are all ln-order, and the three queues can be executed out of order.

[0157] In another embodiment, the x-th matrix operation register MR[x] has three types, namely, input feature map matrix register MR fm , weight matrix register MR wt and bias matrix register MR bs .

[0158] In this embodiment, the storage structure of the CTSC instruction set is as follows: Figure 10 As shown. CTSC instruction storage mainly consists of five parts: the memory for storing feature maps, called Martrix Register feature map, abbreviated as MR fm ; The memory for storing weight weights is called Matrix Register feature map, abbreviated as MR wt ; The memory for storing bias offset is called Matrix Register feature map, also known as MR bs The memory that stores the parameter is called Parameter table, abbreviated as Param table; the memory that stores constant 0 is called Matrix Register 127, abbreviated as MR127. fm , MR wt , MR bsThe abbreviation is MR[x], where [] indicates that the value of x is variable and can be fm, wt, or bs. In a specific implementation, the above memory can be SRAM. For convenience of description, SRAM is used in the following description.

[0159] MR[x] is a general term for three types of data: SRAM that stores feature maps, MR fm ; SRAM for storing weight, MR wt ; SRAM that stores bias offset, MR bs MR fm Can be Figure 10 The CISC processor Operator (including CONV, DWCL, etc.) can also be operated by Figure 10 The processor in reads and writes at a 4B granularity; while MR wt and MR bs It is generally not rewritten in calculations and is only used by Figure 10 The CISC processor Operator in the read operation. The data in MR[x] is read by DMA ( Figure 10 The CISC processor's Operator uses the LDMR or SVMR function to perform read and write operations with the DDR.

[0160] In the storage model design, MR fm , MR wt , MR bs , param_table all belong to on-chip memory. Through this storage design, the feature map and weight data, bias data, and parameter data in the calculation process are all stored in the on-chip memory. The calculation data results can also be stored in these on-chip memories, thereby reducing the intermediate data from being moved back and forth between DDR and the computing unit during the calculation process, effectively reducing the use of DDR bandwidth.

[0161] In another embodiment, wherein the memory MR fm It can be operated by the operator of the neural network processor (specifically including LDMR (read matrix register) and SVMR (write matrix register)), or read and write by the processor at a 4B granularity; and the memory MR wt and memory MR bs The read operation is performed by the Operator of the neural network processor (specifically including LDMR (read matrix register) and SVMR (write matrix register)), and the data in MR[x] is read and written to DDR through DMA.

[0162] In another embodiment, the memory MR fm Contains 32 indexes, denoted as MR0 to MR31, each lndex is a section of SRAM with a width of 64B and a depth of 128; the memory MR wt Contains 32 indexes lndex, and MR fm Unified addressing, recorded as MR32~MR63; memory MR bs Contains 16 indexes lndex, uniformly addressed as MR64 to MR79.

[0163] For this embodiment, MR wt Each lndex and MR fm Same. MR bs Since the amount of data to be stored is relatively small, 16 indexes are reserved and uniformly addressed as MR64 to MR79. fm The same. Since constant 0 is unavoidable in calculations, our design defines MR127 as a constant 0. This means that any operation accessing MR127 will result in an all-0 result. Since MRn in instructions must represent the starting address of the operand on the MR, the following convention is adopted: MR is a 16-bit data structure, where MR[15:7] represents the index address, and MR[6:0] represents the address within each index.

[0164] In another embodiment, the Param table includes 32 indexes, which are denoted as P0 to P31. The data bit width of each index is 4 bytes and the depth is 5.

[0165] In this embodiment, the Param table is used to store the parameters of each operator, which can store the parameters of CONV, DWCL, MV, and FTRANS respectively. The parameter format of each operator is different, and the same space is reused for storage. The parameter length of each operator currently does not exceed 5 words, so it is currently designed based on 5 words. The data bit width of each index in the Paramtable is 4B and the depth is 5. Its physical meaning is determined by the called operator.

[0166] In another embodiment, the following is an example code for performing inter-layer cascade calculations of a 3x3 convolution and a 2x2 max Pooling:

[0167]

[0168] In line 001, we first configure the parameters for the CONV instruction calculation. Here, the parameter pointer value is set to conv_param0. P0, P1, P2, and P3 are four parameter values, each 4 bytes. P0, P1, P2, and P3 contain padding information, which is used when the CONV instruction is executed in line 007.

[0169] In line 002, the parameters for the DWCL instruction calculation are configured. Here, the parameter pointer is set to dwc_param0. P0, P1, and P2 are three 4-byte parameter values. P0, P1, and P2 contain max pooling information, which is used when the DWCL instruction is executed in line 010.

[0170] In line 003, the LDMR instruction reads the weight data length indicated by dlen_wt from the DDR address ddr_wt0 and then stores this data into the matrix register rwt0. rwt0 is just an alias for the matrix register; the actual register is determined by the value of the variable rwt0.

[0171] On line 004, the LDMR instruction reads the offset data length indicated by dlen_bs from the DDR memory at address ddr_bs0, and then stores this data into matrix register rwt0. rbs0 is just another name for the matrix register; the actual register is determined by the value of the variable rbs0.

[0172] In line 005, the LDMR instruction reads the input feature map data length indicated by dlen_ifm from address ddr_fm0 and stores this data in matrix register r0. r0 is just an alias for the matrix register; the actual register is determined by the value of the variable r0.

[0173] On line 006, the LDMR instruction reads the input feature map data length indicated by dlen_ifm from address ddr_fm1 and stores this data in matrix register r1. r1 is just an alias for a matrix register; the actual register is determined by the value of r1.

[0174] At this point, we input the weight and bias rows into the matrix registers that have already read the feature map. Since this is a 3×3 convolution calculation, the padding of the first row is all zeros. This all-zero padding information is already described in the parameter information of row 001. Now we have the conditions for calculating the 3×3 convolution, so we can use the CONV instruction to perform the calculation.

[0175] At line 007, the 3×3 convolution calculation process begins. The CONV instruction reads weight data of length size_wt from the matrix register indicated by wt (the actual value of the wt variable is equal to rwt0, so they represent the same matrix register); reads bias data of length size_bs from the matrix register indicated by bs (the actual value of the bs variable is equal to rbs0, so they represent the same matrix register); reads all zero values ​​from the matrix register indicated by zero, with a data length of the length indicated by size_ifm; reads the value read from DDR and written to r0 by LDMR on line 004 from the r0 register, with a data length of the length indicated by size_ifm; reads the value written to r1 on line 005 from the r1 register, with a data length of the length indicated by size_ifm. The result of the CONV instruction execution is written to the r8 register, and the length of the written data is the length specified by size_ofm.

[0176] On line 008, the LDMR instruction reads the input feature map data length indicated by dlen_ifm from address ddr_fm2 and stores this data in matrix register r2. r2 is just an alias for the matrix register; the actual register is determined by the value of the variable r2.

[0177] At this point, we put the two rows of input feature maps, weitht and bias, into the matrix register that has been read. Since it is a 3×3 convolution calculation, the sliding step size is 1. In this way, after the LDMR execution of line 008 is completed, the three rows r0, r1, and r2 can be convolved again.

[0178] Line 009 begins the 3×3 convolution calculation process with a sliding stride of 1. The CONV instruction reads weight data of length size_wt from the matrix register indicated by wt (the actual value of the wt variable is based on the original rwt0, with the weight length required for the actual calculation of the r0 and r1 input feature maps increased); reads bias data of length size_bs from the matrix register indicated by bs (the actual value of the bs variable is based on rbs0, with the bias length required for the actual calculation of the r0 and r1 input feature maps increased); reads the value read from DDR and written to r0 by LDMR on line 004 from the r0 register, with the data length indicated by size_ifm; reads the value written to r1 on line 005 from the r1 register, with the data length indicated by size_ifm; and reads the value written to r2 on line 008 from the r2 register, with the data length indicated by size_ifm. The result of the CONV instruction is written to the r9 register, and the length of the written data is the length specified by size_ofm.

[0179] Now we have two rows of data calculated by CONV, stored in two matrix registers r8 and r9 respectively. Now we can perform max pooling calculation on these two rows of data.

[0180] In line 010, the DWCL instruction reads two rows of data calculated by CONV from registers r8 and r9 according to the parameters in dwc_param0. Finally, the result is written to r15 through a 2×2 sliding window and a max pooling operation with a sliding step of 2.

[0181] In line 011, the result calculated in 010 is stored in register r15. The SVMR instruction writes the result in r15 to ddr_fm8 of DDR, and the write length is specified by dlen_ifm.

[0182] From this point on, the 3x3 convolution and 2x2 max Pooling in the above example are cascaded between layers to complete the calculation. From this code example, this complex instruction set can complete the convolution and pooling calculations through software programming methods combined with hardware circuit design. The calculation process of a general neural network is as follows Figure 11The computational graph can be abstracted as shown in the dashed box A in the figure below: The input image is pre-processed. After pre-processing, the input image enters the first layer of the network for computation. The output of the first layer serves as the input for the second layer, which then computes the image in the second layer, then in the n-1 layer, and finally in the nth layer. The output of the nth layer undergoes post-processing, completing the entire neural network computation. The computation of each layer can be further broken down into the computational process shown in the dashed box B: The input feature map (with dimensions of width Xi, height Yi, and Ci channels) is computed in N groups. These computations are typically convolutions, with kernels of size M×N and Ki channels. The final output feature map is of width Xo, height Yo, and Co channels. The computation of each group is further structured using the example code above, the core of which is based on the computations of this complex instruction set: LDMR, SVMR, CONV, DWCL, MV, FTRANS, and SETPARAM.

[0183] In another embodiment, the instruction set described in this micro-architecture is implemented on a dual-core deployment on Xilinx's K7-410T and a quad-core deployment on Xilinx's K7-690T. Figure 12 As shown in the figure. When deploying dual cores on Xilinx K7-410T, the actual LUT resource usage is about 80K, DSP occupies 448, BLOCKRAM occupies 1.7MB, and the peak computing power can reach 400Gops at a clock frequency of 166MHz. The resource usage of quad cores deployed on V7690T is shown in the figure. Figure 13 As shown, when deployed on a Xilinx V7-690T with four cores, the actual LUT resource usage is approximately 160KB, the DSP uses 896 memory blocks, and the block RAM uses 3.4MB. At a clock frequency of 166MHz, the peak computing power can reach 800 Gops. This neural network accelerator can smoothly scale computing power across multiple cores on different Xilinx devices.

[0184] In another embodiment, the instruction set encoding and compilation method is described in detail as follows. The detailed steps of the complex instruction encoding method and compilation method are as follows:

[0185] (1) Define the machine code bit width:

[0186] The machine encoding bit width of this instruction set is variable, but it is always an integer multiple of 4 bytes.

[0187] (2) Define the instruction type:

[0188] This complex instruction set defines four instruction types: memory access instructions, calculation instructions, format conversion instructions, and parameter configuration instructions. These four categories contain a total of seven instructions.

[0189] (3) Define addressing mode

[0190] The addressing mode of all instructions is immediate addressing. In other words, the data address passed into each instruction is an immediate value.

[0191] (4) Divide the instruction code field

[0192] INST0 for all instructions is divided into six fields: dSize (5 bits), bs_Size (5 bits), wt_Size (5 bits), op_para (5 bits), op_size (6 bits), and op_code (6 bits). The remaining words of variable-length instructions have different meanings depending on the instruction type.

[0193] (5) Coding:

[0194] The seven instructions are encoded separately, and their encoding formats and instruction functions are as follows:

[0195] a. Memory access instruction LDMR

[0196] Instruction function: Move the data in the DDR memory to the Matrix Register register.

[0197] Instruction mnemonics: LDMR Len, MRd, DDR_addr

[0198] Move Len 64B of data in DDR_addr to the MRd register. The MRd register only indicates the initial register of the destination register.

[0199] Instruction format: The instruction consists of 3 words.

[0200] The format of the first word is:

[0201]

[0202] The 5-bit dSize value indicates how many MRd registers are occupied. This encoding value is dynamically encoded according to the program writing requirements; the 5-bit bs_Size, 5-bit wt_Size and 5-bit op_param fields are all 0, and the corresponding encodings are 5'b00000, 5'b00000, and 5'b00000 respectively; the 6-bit op_size field is encoded as 6'b000010, and the 6-bit op_code field is encoded as 6'b000101.

[0203] The second word format is:

[0204]

[0205] The 16-bit MRd can be further divided into

[0206]

[0207] The specific meaning of the internal division of 16 bits is as follows: bits 0 to 6 represent the offset within each MR; bits 7 to 13 represent the number of the MR register, with an instruction range of 0 to 127; bit 14 is reserved and unused; bit 15 indicates which core is in dual-core mode, 0 indicates core 0 operation, and 1 indicates core 1 operation.

[0208] The 16-bit Len indicates how many 64 bytes are read from the DDR.

[0209] The format of the third word is:

[0210]

[0211] The 32-bit DDR_addr represents the byte address of DDR.

[0212] b. Memory access instruction SVMR

[0213] Instruction function: Move the data in the Matrix Register register to the DDR memory.

[0214] Instruction mnemonics: SVMR Len, MRs, DDR_addr

[0215] Move Len 64B of data in MRs to DDR, and the DDR destination start address is DDR_addr.

[0216] Instruction format: The instruction consists of 3 words.

[0217] The format of the first word is:

[0218]

[0219] The 5-bit dSize value indicates how many MRs registers are occupied. This encoding value is dynamically encoded according to the program writing requirements; the 5-bit bs_Size, 5-bit wt_Size and 5-bit op_param fields are all 0, and the corresponding encodings are 5'b00000, 5'b00000, and 5'b00000 respectively; the 6-bit op_size field is encoded as 6'b000010, and the 6-bit op_code field is encoded as 6'b000110.

[0220] The second word format is:

[0221]

[0222] The 16-bit MRs can be further divided into

[0223]

[0224] The specific meaning of the internal division of 16 bits is as follows: bits 0 to 6 represent the offset within each MR; bits 7 to 13 represent the number of the MR register, with an instruction range of 0 to 127; bit 14 is reserved and unused; bit 15 indicates which core is in dual-core mode, 0 indicates core 0 operation, and 1 indicates core 1 operation.

[0225] The 16-bit Len indicates how many 64B data are read from the MR.

[0226] The format of the third word is:

[0227]

[0228] The 32-bit DDR_addr represents the byte address of DDR.

[0229] c. Calculation instruction CONV

[0230] Instruction function: Read data from the Matrix Register, perform convolution calculation, and write the calculation result back to the Matrix Register.

[0231] Instruction mnemonics:

[0232] CONV op_param, wt_Size, bs_Size, sLen, MRd, MRwt, MRbs, MRs1, {MRs[n]}

[0233] According to the parameters in op_param, weight data is read from MRwt (MRwt describes which MR, wr_Size describes the amount of data to be read), bias data is read from MRbs (MRbs describes which MR, bs_Size describes the amount of data to be read), and feature map data is read from MRs1 and MRs[n] (MRs1 and MRs[n] describe which MR, sLen describes the amount of data to be read). Then, according to the convolution calculation type described in op_param, such as 3×3 convolution calculation with stride 1, calculation is performed. After the calculation is completed, the calculation result is written to the destination Matrix Register register described by MRd.

[0234] Instruction format: The instruction consists of 4 words (only for 1×1 window convolution).

[0235] The format of the first word is:

[0236]

[0237] The 5-bit dSize value indicates how many MRd registers are occupied in total, and this encoding value is dynamically encoded according to the program writing requirements; the 5-bit bs_Size value indicates that the bias data read starts from the MRbs register, but how many indices are read; the 5-bit wt_Size indicates that the weitht data read starts from the MRwt register, but how many indices are read; the 5-bit op_param value indicates the address of the parameter data read, and the range of instructions is 0-31. The corresponding encoding is dynamically encoded according to the program writing requirements; the 6-bit op_size field encoding is determined according to the length of the entire instruction. For example, when calculating a 1×1 convolution, the op_size encoding is 6'b000011; the 6-bit op_code field encoding is 6'b001000.

[0238] The second word format is:

[0239]

[0240] The 16-bit MRd value indicates the MRd register.

[0241] The 16-bit sLen value indicates the number of Matrix Registers indicated by MRs1 and MRs1' in the subsequent 4th word.

[0242] The format of the third word is:

[0243]

[0244] The 16-bit MRbs value indicates the location where the bias is stored in the Matrix Register; the 16-bit MRwt value indicates the location where the weight is stored in the Matrix Register.

[0245] The format of the fourth word is:

[0246]

[0247] The 16-bit MRs1' identifies the Matrix Register register of core1; the 16-bit MRs1 identifies the Matrix Register register of core0.

[0248] For 1×1 convolution, the CONV instruction length is only INST0, INST1, INST2, INST3. For 3×3 convolution, the CONV instruction length is only INST0, INST1, INST2, INST3, INST4, INST5. The encoding of INST4 and INST5 is the same as that of INST3, but the encoding value may change according to the programming intention.

[0249] d. Calculation instruction DWCL

[0250] Instruction function: Read data from the Matrix Register register, perform depth-wise separable convolution calculation or pooling operation, and write the calculation result back to the Matrix Register register.

[0251] Instruction mnemonics:

[0252] DWCL op_param, wt_Size, bs_Size, sLen, MRd, MRwt, MRbs, MRs1, {MRs[n]}

[0253] According to the parameters in op_param, weight data is read from MRwt (MRwt describes which MR, wr_Size describes the amount of data to be read), bias data is read from MRbs (MRbs describes which MR, bs_Size describes the amount of data to be read), and feature map data is read from MRs1 and MRs[n] (MRs1 and MRs[n] describe which MR, sLen describes the amount of data to be read). Then, according to the convolution calculation type described in op_param, such as 3X3 convolution calculation with stride 1, calculation is performed. After the calculation is completed, the calculation result is written to the destination Matrix Register register described by MRd.

[0254] Instruction format: The instruction consists of 4 words (only for 1×1 window).

[0255] The format of the first word is the same as the INST0 format of the CONV instruction. The only difference is that the 6-bit op_code field is encoded as 6'b001001.

[0256] The format of the second word is the same as the INST1 instruction format of the CONV instruction.

[0257] The format of the third word is the same as the INST2 instruction format of the CONV instruction.

[0258] The format of the fourth word is the same as the INST3 instruction format of the CONV instruction.

[0259] For 1×1 depthwise separable convolution, the DWCL instruction lengths are INST0, INST1, INST2, and INST3. For 3×3 depthwise separable convolution, the DWCL instruction lengths are INST0, INST1, INST2, INST3, INST4, and INST5. The encodings of INST4 and INST5 are the same as those of INST3, but the encoding values ​​may change depending on the programming intent.

[0260] e.Format conversion instruction MV

[0261] Instruction function: Read data from the Matrix Register, perform format conversion such as transposition and compensation, and write the converted result back to the Matrix Register.

[0262] Instruction mnemonics: MV op_param, dSize, sLen, MRd, MRs

[0263] Read data from MRs according to the parameters in op_param (MRd describes which MR, and sLen describes the amount of data to read). Then, perform format conversion according to the conversion method described in op_param and write the converted result to the destination Matrix Register described by MRd (MRd describes which MR, and dSize describes the amount of data to write).

[0264] Instruction format: There are 3 words in total.

[0265] The format of the first word is:

[0266]

[0267] The 5-bit dSize value indicates how many MRd registers are occupied in total, and this encoding value is dynamically encoded according to the program writing requirements; the 5-bit bs_Size and 5-bit wt_Size are 0; the 5-bit op_param value indicates the address of the parameter data to be read, and the instruction range is 0-31. The corresponding encoding is dynamically encoded according to the program writing requirements; the 6-bit op_size field encoding is determined by the length of the entire instruction, and the op_size encoding is 6'b000010; the 6-bit op_code field encoding is 6'b001010.

[0268] The second word format is:

[0269]

[0270] The 16-bit MRd value indicates the MRd register.

[0271] The 16-bit sLen value indicates the number of Matrix Registers indicated by MRs in the subsequent 4th word.

[0272] The format of the third word is:

[0273]

[0274] The 16-bit MRs' identifies the Matrix Register register of core1; the 16-bit MRs identifies the Matrix Register register of core0.

[0275] f. Format conversion instruction FTRANS

[0276] Instruction function: Read data from the Matrix Register register, convert the feature map channel format, and write the converted result back to the Matrix Register register.

[0277] Instruction mnemonics: FTRANS op_param, dSize, sLen, MRd, MRs

[0278] Read data from MRs according to the parameters in op_param (MRd describes which MR, and sLen describes the amount of data to read). Then, perform format conversion according to the conversion method described in op_param and write the converted result to the destination Matrix Register described by MRd (MRd describes which MR, and dSize describes the amount of data to write).

[0279] Instruction format: There are 3 words in total.

[0280] The INST0 encoding of the FTRANS instruction is the same as the INST0 encoding of the MV instruction, the only difference is that the 6-bit op_code field is encoded as 6'b001011.

[0281] The format of the second word is the same as the INST1 instruction format of the MV instruction.

[0282] The format of the third word is the same as the INST2 instruction format of the MV instruction.

[0283] g. Parameter configuration instruction SETPARAM

[0284] Command function: configure the parameters required for running each command into the Parameter table.

[0285] Instruction mnemonic: SETPARAM op_param, Pd0[31:0], {Pdn[31:0]}

[0286] According to the parameter pointer in op_param, Pd0[31:0], Pd1[31:0], ...Pdn[31:0] are configured in the parameter table. n ranges from 0 to 4. Instruction format: Depending on the value of n, the instruction has n+2 words.

[0287] The format of the first word is:

[0288]

[0289] Among them, the 5-bit dSize, 5-bit bs_Size and 5-bit wt_Size are 0; the value of the 5-bit op_param indicates the address of the parameter data to be written, and the instruction range is 0-31. The corresponding encoding is dynamically encoded according to the program writing requirements; the 6-bit op_size field encoding is determined by the length of the entire instruction, and op_size is encoded as 6'b000010 (when n=2); the 6-bit op_code field encoding is 6'b001100.

[0290] (5) Programming and compilation methods

[0291] During actual programming, we define each instruction using C language structures and then compile it using gcc to generate a binary file. For the RTSC-V processor, this binary file is the machine code for the RISC-V processor's native instruction set. In other words, we leverage gcc and the generic RISC-V processor instruction set machine code to implement programming and compilation support for this complex instruction set.

[0292] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. A neural network processor complex instruction set microsystem, comprising: General-purpose processor, instruction dispatch unit DISP, computing instruction and format conversion instruction queue IQ (ALU), memory access instruction LDMR instruction queue IQ (LDMR), memory access instruction SVMR instruction queue IQ (SVMR), write back module Retire, scoreboard, parameter storage module OP_param_talbe, data storage module, memory access LDMR execution unit, memory access SVMR execution unit, CONV scheduling unit, DWCL scheduling unit, MV scheduling unit, FTRANS scheduling unit, matrix processing unit MPU and vector processing unit VPU; The IQ (LDMR) stores all LDMR instructions, which are responsible for moving data from the DDR to the x-th matrix operation register MR[x]. The IQ (SVMR) stores all SVMR instructions, which are responsible for moving data from the x-th matrix operation register MR[x] to the DDR. The IQ (ALU) stores CONV instructions, DWCL instructions, MV instructions, FTRANS instructions, and SETPARAM instructions, wherein the CONV instruction is responsible for performing standard convolution operations on the data, the DWCL instruction is responsible for performing channel-independent operations on the data, including depth-wise convolution and pooling operations, the MV instruction is responsible for moving the data in the x-th matrix operation register MR[x], and converting or rearranging the data according to the parameters while moving. The FTRANS instruction is responsible for converting the data format between ch64 format and ch8 format. The SETPARAM instruction is responsible for configuring the parameters of each operator. This instruction can be used to write to any index of the operand parameter table OP_param_table. The instruction format of the CONV instruction is as follows: CONV op_param, wt_Size, bs_Size, dSize, sLen, MR d , MR wt , MR bs , MR s1 , {MR s[n] } Among them, op_param is a pointer to the operand parameter table OP_param_table, indicating which set of parameters should be used for this convolution operation; wt_Size and bs_Size respectively represent the number of indexes occupied by the weight Weight and bias Bias data of the current convolution operation, counting from 0; dSize represents the number of indexes occupied by the structure data of the current convolution operation on the x-th matrix operation register MR[x], counting from 0; sLen is the number of indexes occupied by each row of source data of the current convolution operation, counting from 0; MR d , MR s[n] MR is the starting address of the result data and source data; bs , MR wt is the starting address of offset and weight; The complex instruction set completes the convolution and pooling calculations through software programming combined with hardware circuit design; the calculation process of the neural network is as follows: the pre-processing process of the input image, after the pre-processing, the input image enters the first layer of the network for calculation, the calculation output of the first layer is used as the calculation input of the second layer, then passes through the calculation of the second layer, then passes through the calculation of the n-1 layer, and finally passes through the calculation of the n layer. After the calculation output of the n layer is post-processed, the calculation of the entire neural network is completed; the calculation of each layer is further decomposed into the following calculation process: for the input feature map, N Groups are calculated, the feature map size is width Xi, height Yi, and there are Ci channels in total. The Group calculation is a convolution calculation, and its corresponding convolution kernel size is M×N, with a total of Ki channels. The final output feature map size is width Xo, height Yo, and there are Co channels in total; wherein, the core of each Group calculation is the calculation based on the complex instructions, namely the LDMR instruction, SVMR instruction, CONV instruction, DWCL instruction, MV instruction, FTRANS instruction and SETPARAM instruction.

2. The microsystem according to claim 1, wherein: The instruction format of the LDMR instruction is as follows: LDMR Len, MR d , DDR_addr Among them, MR d Indicates the target start address of the data to be moved to MR[x], in units of 64B; DDR_addr indicates the start address from DDR, in units of 1B; Len indicates the length of the data to be moved, in units of 64B; The encoding format of the LDMR instruction is as follows: ； The op_code is encoded as 6'b000101, and op_size is the length of the instruction, indicating how many words the instruction occupies.

3. The microsystem according to claim 1, wherein: The instruction format of the SVMR instruction is as follows: SVMR Len, MR s , DDR_addr Among them, MR s Indicates the starting address of data to be moved from MR[x], in units of 64B; DDR_addr indicates the target starting address of data to be moved to DDR, in units of 1B; Len indicates the length of the data to be moved, in units of 64B; The encoding format of the SVMR instruction is as follows: ； Among them, op_code is encoded as 6'b000110, and op_size is the length of the instruction, indicating how many words the instruction occupies.

4. According to the microsystem of claim 1, the CONV instruction encoding format is as follows: ； in, op_code is encoded as 6'b001000, MR sn and MR sn' respectively represent the starting addresses of the neural network processor Core0 and the neural network processor Core1, and op_size is the length of the instruction, indicating how many words the instruction occupies.

5. The microsystem according to claim 1, wherein: The instruction format of the DWCL instruction is as follows: DWCL op_param, wt_Size, bs_Size, dSize, sLen, MR d , MR wt , MR bs , MR s1 , {MR s[n] } Among them, op_param is a pointer to the operand parameter table OP_param_table, indicating which set of parameters needs to be used for this operation; wt_Size and bs_Size respectively represent the number of indexes occupied by the weight Weight and bias Bias data of the current convolution operation; dSize represents the number of indexes occupied by the result data of this operation; sLen is the number of indexes occupied by each row of source data in the current operation; MR d , MR s[n] MR is the starting address of the result data and source data; bs , MR wt is the starting address of offset and weight; The DWCL instruction encoding format is as follows: ； Among them, op_code is encoded as 6'b001001, MR sn and MR sn' respectively represent the starting addresses of the neural network processor Core0 and the neural network processor Core1, and op_size is the length of the instruction, indicating how many words the instruction occupies.

6. The microsystem according to claim 1, wherein: The instruction format of the MV instruction is as follows: MV op_param, dSize, sLen, MR d , MR s Among them, op_param is a pointer to the operand parameter table OP_param_table, which indicates the parameters of the data to be moved this time; dSize is the number of indexes occupied by the result data; sLen is the number of indexes occupied by the source data; MRd and MRs are the starting addresses of the result data and source data; The MV instruction encoding format is as follows: ； Among them, op_code is encoded as 6'b001010, and op_size is the length of the instruction, indicating how many words the instruction occupies.

7. The microsystem according to claim 1, wherein: The instruction format of the FTRANS instruction is as follows: FTRANS op_param, dSize, sLen, MR d , MR s Among them, op_param is a pointer to the operand parameter table OP_param_table, which indicates the parameters when performing Format conversion; dSize is the number of indexes occupied by the result data; sLen is the number of indexes occupied by the source data; MR d , MR s The starting address of the result data and source data; The FTRANS instruction encoding format is as follows: ； Among them, op_code is encoded as 6'b001011, and op_size is the length of the instruction, indicating how many words the instruction occupies.

8. The microsystem according to claim 1, wherein: The instruction format of the SETPARAM instruction is as follows: SETPARAM op_param, Pd0, {Pd[n]} Wherein, op_param is the index address of the operand parameter table OP_param_table to be written; Pdn is the data to be written; The SETPARAM instruction encoding format is as follows: ； Among them, op_code is encoded as 6'b001100, and op_size is the length of the instruction, indicating how many words the instruction occupies.

9. The microsystem according to claim 2, wherein: The x-th matrix operation register MR[x] has three types, namely input feature map matrix register MR fm , weight matrix register MR wt and bias matrix register MR bs .

10. The microsystem according to claim 9, wherein: Memory MR fm Contains 32 indexes, denoted as MR0 to MR31, each index is a section of SRAM with a width of 64B and a depth of 128; the memory MR wt Contains 32 indexes, and MR fm Unified addressing, recorded as MR32~MR63; memory MR bs Contains 16 indexes, uniformly addressed as MR64 to MR79.

Images