RISC-V-based general-purpose neural network processor microarchitecture

Abstract

This disclosure reveals a general-purpose neural network AI processor microarchitecture based on RISC-V and dedicated extended instruction sets, including a processor front-end unit, an instruction decoding and dispatch unit, a scalar execution unit, a vector-matrix execution unit, and a multi-level data storage unit. This microarchitecture employs a Turing-complete fine-grained instruction set to implement arbitrary algorithms and utilizes dedicated vector and matrix instructions for efficient computation of neural network operators, thus balancing computational power and flexibility for neural network inference. The microarchitecture adopts a superscalar out-of-order issue architecture in its hardware architecture, enabling concurrent execution of scalar, vector, and matrix instructions, optimizing the microarchitecture for deep neural network inference to ensure accelerator execution efficiency.

Description

Technical Field

[0001] This disclosure belongs to the field of artificial intelligence neural network technology, and specifically relates to a general neural network processor microarchitecture based on RISC-V extensions. Background Technology

[0002] With the increasingly widespread application of convolutional deep neural networks in real-world scenarios, supporting a wide variety of neural networks with a single hardware setup has become a research hotspot. Existing technologies fall into two main categories. The first uses a coprocessor model, performing computation and scheduling at a fixed-scale operator granularity, achieving a high performance-to-power ratio. However, due to the inherently large scheduling granularity, it cannot provide sufficient flexibility. The other category uses an instruction set model. Instruction set accelerators, which are close to GPU-style, suffer from low energy efficiency when executing convolutional operators due to a lack of support for large-scale matrix operations. Instruction set accelerators optimized for neural network operators each have their own advantages and disadvantages. Based on the axis of programming flexibility and convolutional performance, instruction set-style neural network accelerators represent two extremes. Dedicated neural network accelerators, exemplified by Huawei's Da Vinci series, prioritize neural network performance. To achieve high execution efficiency, their scalar, vector, and matrix instructions employ early GPU command queue scheduling, leading to complex programming and debugging difficulties. In contrast, Intel's AVX instruction extensions emphasize flexibility and versatility. While extending vector and matrix instructions, they suffer from limitations such as low cache bandwidth, low data parallelism, and limited effective execution time due to their fully programmable nature. Despite the numerous design challenges of instruction set-style accelerators, they remain the future direction given the rapid iteration of neural network algorithms and the continuous evolution of operator types. Summary of the Invention

[0003] In view of this, this disclosure provides a general-purpose neural network processor microarchitecture based on RISC-V extensions, including a processor front-end unit, an instruction decoding and dispatch unit, a scalar execution unit, a vector matrix execution unit, and a multi-level data storage unit; wherein,

[0004] The processor front-end unit consists of an instruction fetch module, a fast decoding module, an instruction prediction module, and an instruction buffer module, and supports parallel instruction fetching through four paths.

[0005] The instruction decoding and distribution unit consists of an instruction decoding module, a register renaming module, and an instruction distribution module, and supports parallel decoding and distribution of two instructions.

[0006] The scalar execution unit consists of a scalar ALU module, a scalar MDU module, a scalar LSU module, and a scalar register module, and employs completely out-of-order execution.

[0007] The vector matrix execution unit consists of a vector matrix queue module, a matrix MAC module, a vector ALU module, a vector LSU module, and a vector register module, and is executed in a fixed order.

[0008] The multi-level data storage unit consists of an L1 instruction buffer module, an L1 data buffer module, an L2 shared buffer module, an on-chip data memory (DTCM) module, and an on-chip SIMD local storage module.

[0009] Preferred,

[0010] The vector matrix execution unit supports 8-bit integer multiplication and addition operations, supports 512 multiplications and 512 additions per cycle, and can complete an independent matrix multiplication of [8, 8], or a vector matrix multiplication of [1, 8] and [64, 8].

[0011] Preferred,

[0012] The vector matrix execution unit supports 8-bit, 16-bit, and 32-bit integer operations, performs operations on 64 integer data per cycle, and supports aggregate instructions that execute in multiple cycles with a single instruction.

[0013] Preferred,

[0014] The vector matrix execution unit employs a double-buffered renaming mechanism for the specified registers.

[0015] Preferred,

[0016] The vector matrix execution unit adopts a delayed execution strategy, and only begins sequential execution after ensuring that there are no branch jump instructions before the vector matrix instructions.

[0017] Preferred,

[0018] The multi-level data storage unit uses cache blocks of different granularities to achieve data management and balance between control code and computational data.

[0019] Preferred,

[0020] The microarchitecture's pipeline includes an 8-stage scalar basic pipeline, a scalar multiplication and division pipeline, a scalar memory access pipeline, and a vector matrix pipeline.

[0021] Preferred,

[0022] The scalar part of the pipeline backend mainly includes ALU / MDU / AGU instruction wake-up, ALU / MDU / AGU register read, ALU arithmetic and logical operations, ALU write-back, MDU multiplication and division operations, and MDU multiplication and division write-back.

[0023] Preferred,

[0024] The vector matrix section at the back end of the pipeline mainly includes vector matrix instruction wake-up, vector matrix instruction register read, vector matrix instruction execution, and vector matrix instruction write-back.

[0025] Preferred,

[0026] The common components of the pipeline backend mainly include instruction decoding register renaming and allocation, and instruction dispatch register renaming and updating.

[0027] Based on the above technical solutions, given that the current neural network inference process can be abstracted into a series of combinations of scalar, vector, and matrix operations, this microarchitecture adopts a RISC-style instruction set to ensure implementation flexibility, and uses scalar, vector, and matrix instructions to be executed concurrently in the hardware architecture to ensure execution efficiency.

[0028] This microarchitecture employs a Turing-complete, fine-grained instruction set to implement arbitrary algorithms and novel vector and matrix instructions for efficient computation of neural network operators, thus balancing computational power and flexibility for neural network inference. In terms of hardware architecture, this microarchitecture utilizes a superscalar out-of-order issue structure to achieve concurrent execution of scalar, vector, and matrix instructions, optimizing the microarchitecture for deep neural network inference to ensure the accelerator's execution efficiency. Attached Figure Description

[0029] Figure 1 This is a core framework diagram of a general-purpose neural network AI processor microarchitecture based on RISC-V extensions provided in one embodiment of this disclosure;

[0030] Figure 2 This is an overall pipeline structure diagram of one embodiment of the present disclosure;

[0031] Figure 3 This is a diagram illustrating the instruction rollback method for the vector matrix portion in one embodiment of this disclosure;

[0032] Figure 4 This is a 2D-MESH multicore design architecture diagram of one embodiment of this disclosure;

[0033] Figure 5 This is a chip back-end layout and routing diagram in one embodiment of this disclosure. Detailed Implementation

[0034] In one embodiment, such as Figure 1 As shown, it discloses a general-purpose neural network AI processor microarchitecture based on RISC-V extension, including a processor front-end unit, an instruction decoding and dispatch unit, a scalar execution unit, a vector matrix execution unit, and a multi-level data storage unit; wherein, the processor front-end unit is composed of an instruction fetch module, a fast decoding module, an instruction prediction module, and an instruction buffer module, and supports parallel instruction fetching through four paths;

[0035] The instruction decoding and distribution unit consists of an instruction decoding module, a register renaming module, and an instruction distribution module, and supports parallel decoding and distribution of two instructions.

[0036] The scalar execution unit consists of a scalar ALU module, a scalar MDU module, a scalar LSU module, and a scalar register module, and employs completely out-of-order execution.

[0037] The vector matrix execution unit consists of a vector matrix queue module, a matrix MAC module, a vector ALU module, a vector LSU module, and a vector register module, and is executed in a fixed order.

[0038] The multi-level data storage unit consists of an L1 instruction buffer module, an L1 data buffer module, an L2 shared buffer module, an on-chip data memory (DTCM) module, and an on-chip SIMD local storage module.

[0039] In this embodiment, the instruction fetch module reads multiple instructions from the instruction cache in 16-word alignment; the read instructions are then quickly decoded by the fast decoding module to determine the instruction type. The instruction prediction module employs coarse-grained prediction based on the current PC and fine-grained prediction based on local history and instruction content.

[0040] The instruction decoding module supports various standard RISC-V basic instructions, atomic operation instructions, fixed-point multiplication and division instructions, control status register operation instructions, and specially extended vector matrix instructions, and distributes these instructions to the corresponding execution units. The register renaming module is implemented using a standard unified physical register approach. It first maps logical registers to 64 physical registers, and generates snapshots of the register mapping table and free list when a jump instruction is encountered.

[0041] The ALU module corresponds to common arithmetic and logic instructions, and is implemented using a standard full adder and comparison circuits; the scalar MDU module uses standard fixed-point multiplication, division, and remainder circuits to implement fixed-point multiplication and division instructions; the scalar LSU module corresponds to read and store instructions, and uses read / write memory circuits that conform to the weak consistency specification.

[0042] The MAC module is implemented using a three-dimensional pulsating array.

[0043] The RISC-V instruction set is organized in a modular way, with each module represented by a single English letter.

[0044] The most basic and only mandatory instruction set component of RISC-V is a subset of basic integer instructions represented by the letter 'I'. Using this subset of integer instructions, a complete software compiler can be implemented. In one embodiment, the superscalar out-of-order processor implemented therein, to meet general neural network AI computing tasks, specifically extends RISC-V's vector and matrix instructions, and implements a microarchitecture based on RISC-V 32IMA and extended instructions. This microarchitecture can be programmed using the general-purpose C language, and neural network programming can be implemented with the help of the GCC compiler.

[0045] This general-purpose neural network AI processor microarchitecture based on RISC-V and its dedicated extended instruction set is suitable for general-purpose processor computing and also for general-purpose neural network programming computing tasks. This general-purpose neural network AI processor microarchitecture based on RISC-V and its dedicated extended instruction set has the following dedicated features: (1) Single-core matrix multiplication and addition computing power expansion: In a single core, the matrix multiplication and addition computing power can be improved by adding a VMU; (2) Error recovery capability expansion: In a single core, the branch prediction failure penalty can be reduced by adding a checkpoint, thereby improving the branch prediction error recovery capability; (3) Multi-core smooth expansion: The single-core design reserves a multi-core smooth expansion interface to facilitate multi-core smooth expansion.

[0046] When applied to neural networks, the architecture is first streamlined: convolution operations are converted into batch matrix multiplication and accumulation instructions (MAC); operations related to multilayer perceptrons and RNNs are converted into vector and matrix multiplication and accumulation instructions; activation functions and element operations are summarized into a series of SIMD-structured vector instructions; and data format conversions for software, matrices, and vectors are abstracted into matrix transformations and transposes. Furthermore, this microarchitecture adopts a superscalar out-of-order processor design style, enabling parallel execution and scheduling of various instructions, thereby maintaining higher neural network computational efficiency while preserving programming flexibility and simplicity.

[0047] In another embodiment, the vector matrix execution unit supports 8-bit integer multiplication and addition operations, supports 512 multiplications and 512 additions per cycle, and can complete an independent matrix multiplication of [8, 8], or a vector matrix multiplication of [1, 8] and [64, 8].

[0048] In another embodiment, the vector matrix execution unit supports 8-bit, 16-bit, and 32-bit integer operations, performs operations on 64 integer data per cycle, and supports aggregate instructions that execute in multiple cycles with a single instruction.

[0049] In this embodiment, the processor microarchitecture integrates vector and matrix instructions into a scalar pipeline architecture.

[0050] In another embodiment, the vector matrix execution unit employs a double-buffered renaming mechanism for the specified register.

[0051] In this embodiment, the pipeline employs different register renaming strategies for scalar instructions and vector-matrix instructions. Scalar instructions use a traditional renaming strategy, while the vector-matrix execution unit uses a dedicated renaming mechanism. The purpose of this mechanism is not to eliminate pseudo-dependencies between instructions, but to improve the execution unit's efficiency while maintaining software code simplicity.

[0052] Specifically, although vector matrix instructions employ a sequential execution strategy, a renaming strategy is still used for the dedicated registers in this part to improve instruction execution efficiency in order to resolve the interdependence between vector and pointer instructions. Register VR8 is designated as the destination register for matrix instructions, and it uses a dual physical register design. Whenever this register appears as the destination register, it is allocated to one of the registers; the next time it is specified, it is allocated to the other register; this process is repeated cyclically, thereby resolving pipeline delays caused by read / write conflicts of this register.

[0053] In another embodiment, the vector matrix execution unit employs a delayed execution strategy, ensuring that there are no branch jump instructions before the vector matrix instructions before starting sequential execution.

[0054] In this embodiment, the extra power consumption overhead caused by meaningless instruction rollback is avoided.

[0055] In another embodiment, the multi-level data storage unit employs cache blocks of different granularities to achieve data management and balance between control code and computational data.

[0056] In this embodiment, the microarchitecture employs a multi-level caching structure, leveraging the spatial and temporal locality of the program to reduce the latency of average memory reference operations.

[0057] To address the speed mismatch between the processor and memory, the microarchitecture disclosed herein is, for example, named HiPU200 (equivalent to a HIPU). It employs a multi-level cache structure, leveraging the spatial and temporal locality of memory to reduce the latency of average memory reference operations. Specifically, the first-level cache includes separate data and instruction caches tailored to the different characteristics of data and instructions. The data cache utilizes a victim cache for performance optimization, while the instruction cache employs a stream buffer hardware prefetch strategy for further performance optimization.

[0058] In another embodiment, the microarchitecture employs a composite branch prediction strategy, where fast coarse-grained branch prediction is provided by the BTB and RAS structures, while accurate fine-grained branch prediction is provided by the BHT structure.

[0059] Branch instructions appear frequently in applications. To ensure that the processor operates on the correct program path as much as possible, the HiPU200 adopts a composite branch prediction strategy. Fast coarse-grained branch prediction is provided by the BTB and RAS structures, while accurate fine-grained branch prediction is provided by the BHT structure, thus achieving a trade-off between branch prediction speed and accuracy.

[0060] HiPU200 supports a checkpoint mechanism, which allows switching back to a saved historical processor state after a branch prediction failure, reducing penalties for misjudgments.

[0061] In another embodiment, the microarchitecture employs an ordered submission strategy, utilizing a 64-item ROB structure to achieve ordered submission of instructions.

[0062] To support precise exception handling, the HiPU200 adopts a sequential commit strategy, utilizing a 64-item ROB structure to achieve sequential commit of instructions.

[0063] The acceleration core supports any activation function that can be quantized into a look-up table format, and uses a 9-bit look-up table to implement any activation function with a slope no greater than 2.

[0064] In another embodiment, the microarchitecture's pipeline includes an 8-stage scalar fundamental pipeline, a scalar multiplication and division pipeline, a scalar memory access pipeline, and a vector matrix pipeline. The overall pipeline structure is as follows: Figure 2 As shown.

[0065] In another embodiment, the front end of the pipeline mainly includes three stages: PC generation, obtaining I-Cache status, and obtaining I-Cache instructions, which are used to obtain the correct instructions from the I-cache and send the instructions to the subsequent pipeline.

[0066] In this embodiment, the front end of the pipeline mainly includes three stages: PC generation, I-Cache status acquisition, and I-Cache instruction acquisition. Its primary responsibility is to retrieve the correct instructions from the I-Cache and send them to subsequent stages in the pipeline. Its performance metrics are mainly reflected in:

[0067] 1) The delay when sending instructions to subsequent pipelines is short enough;

[0068] 2) The prediction accuracy is high enough when the instruction encounters a jump instruction.

[0069] Since the I-Cache cannot guarantee data retrieval at every clock cycle, an instruction buffer is inserted between the front-end and back-end ID stages of the pipeline to alleviate latency caused by slow instruction fetching. This buffer is called the front-end instruction buffer. The instruction buffer is designed with a structure of 4 instructions for writing and 2 instructions for reading.

[0070] Stage 0: Instruction fetch stage, initiating a read instruction cache command.

[0071] This stage gathers instruction fetch addresses and branch prediction results from various sources, arbitrates them according to priority, and starts or stops access to the instruction cache and branch prediction unit based on the current pipeline status. Instruction fetch addresses come from the PC (Program Counter), CGPR (Coarse-grained Prediction Unit), FGPR (Fine-grained Prediction Unit), interrupt / exception handling module, instruction rollback module, and other locations.

[0072] After the instruction address is determined, a read operation on the cache will be initiated in this stage. Since this design is a 4-fetch design, it can complete 4 instruction read requests per clock cycle.

[0073] Stage 1: Instruction fetch stage, obtaining instruction cache status.

[0074] This stage can obtain the cache hit result of the read operation initiated in the previous stage. If the cache hits, the next stage is notified that the obtained instruction is valid; otherwise, the next stage is notified that a valid instruction could not be obtained.

[0075] This stage generates a coarse-grained instruction fetch address prediction based on the current PC value. Since no specific instruction has been fetched at this stage, prediction can only be made based on the PC value. This stage uses the BTB (Branch Target Buffer) method for prediction. The BTB records the PC values ​​encountered previously with jumps, as well as the values ​​after the jumps. By querying the BTB, a rough prediction of whether a jump is needed for the current PC can be obtained.

[0076] BTB updates when the jump instruction is submitted to ROB.

[0077] Stage 2: Instruction fetch stage, obtaining instruction cache data.

[0078] This stage can obtain the instruction data for the cache read operation initiated in the first stage. If the cache has been hit by the instruction feedback from the previous stage, this stage can obtain up to 4 instructions and insert them into the front-end instruction buffer.

[0079] This stage performs fine-grained instruction prediction based on the specific instructions obtained, correcting the coarse-grained prediction from the previous stage. Because this stage obtains the specific instructions, rapid decoding facilitates more accurate predictions: for direct jump instructions (relative to the program counter), the jump address can be directly determined; for conditional jump instructions, the 4-stage jump state machine can be used to more accurately predict whether a jump will occur. For function call or return instructions, the RAS (Return Address Stack) mechanism can be used to accurately predict the jump and return addresses.

[0080] By using the results of fine-grained instruction prediction to correct the instruction address in the first stage, more accurate instruction prediction results can be obtained.

[0081] In another embodiment, the general components of the pipeline backend mainly include instruction decoding register renaming allocation and instruction dispatch register renaming update.

[0082] In this embodiment, the back-end of the out-of-order multiple issue processor is divided into a variety of different FUs (Function Units). After each FU completes its own instruction calculation, it submits the result to the ROB (ReorderBuffer).

[0083] Before instructions enter various Function Units (FUs), there are some common pipeline stages, such as instruction decoding and instruction dispatch. In parallel with these, instructions also need to have their WAR and WAW dependencies removed by the register renaming module. When handling instruction dependencies in subsequent FUs, only the RAW dependency needs to be considered.

[0084] Stage 3: Instruction decoding, register renaming and allocation

[0085] This stage reads instructions from the front-end instruction buffer and simultaneously decodes the read instructions, mapping the logical register labels of the read instructions to physical register labels.

[0086] Each cycle reads a maximum of two instructions from the front-end instruction buffer. After decoding, the instructions are divided into the following four categories according to the FU (Full Execution Unit):

[0087] ALU: Performs scalar arithmetic and logical operations, and jumps to determine instructions;

[0088] MDU: Performs scalar multiplication and triggers instructions;

[0089] LSU (including AGU): Executes scalar memory access instructions;

[0090] VMU: Executes vector and matrix instructions.

[0091] At the same time, the parameters required by the FU corresponding to the instruction are obtained by decoding and parsing.

[0092] The register renaming module enables the conversion between logical registers and physical registers for instructions. Since superscalar out-of-order processors need to support instruction rollback, this stage retains a snapshot of eight jump instructions for use during instruction rollback.

[0093] Stage 4: Instruction dispatch, register renaming and updating

[0094] This stage is responsible for distributing the decoded instructions to different function units (FUs) and updating the records in the register renaming module.

[0095] Each cycle can decode a maximum of 2 instructions. If both instructions can be found at the same time and there is an available function (FU), then the two instructions are assigned to the corresponding FU.

[0096] When each Function Unit (FU) receives instructions during the instruction distribution phase, it stores the instructions in its respective FU instruction queue. The implementation methods for instruction queues vary, and different designs are chosen based on the specific circumstances.

[0097] In another embodiment, the scalar portion of the pipeline backend mainly includes ALU / MDU / AGU instruction wake-up, ALU / MDU / AGU register read, ALU arithmetic and logical operations, ALU write-back, MDU multiplication and division operations, and MDU multiplication and division write-back.

[0098] In this embodiment, a scalar section at the back end of the production line

[0099] The execution of each Function Unit (FU) is completely parallel and independent of each other. Scalar-dependent FUs mainly consist of two Address Logic Units (ALUs), one Address Management Unit (MDU), and one Memory Utility Unit (LSU). The LSU can be divided into AGUs (Address Generation Units) that handle register dependencies and LSUs (Memory Access Units) that handle memory dependencies. These FUs can be further subdivided into multi-stage pipelines for design.

[0100] Stage 5: ALU / MDU / AGU instruction wake-up

[0101] The instruction wake-up phase of the ALU / MDU / AGU is responsible for finding instructions that are completely decoupled from instructions in their respective transmit queues, marking them as valid, and sending them to the subsequent pipeline.

[0102] The instruction queue implementations of the three FUs differ. The ALU's instruction queue uses a compressed FIFO storage method. The advantage of this storage method is that it can store more valid instructions, meaning that the instruction queue space is not occupied after the instruction is issued. The AGU and MDU, on the other hand, use a non-compressed FIFO storage method. This simplifies the design logic without affecting instruction execution efficiency.

[0103] The command issuance queue depth is 8 for all three types of FUs.

[0104] The instructions sent during the launch phase include whether all source registers are ready at that time, and then all states will be updated based on the broadcast data in the wake-up network.

[0105] Stage 6: ALU / MDU / AGU Register Reading

[0106] The ALU / MDU / AGU register read phase is responsible for retrieving valid source operands from registers. If an instruction from the previous phase is invoked, a register read request will be issued, and this phase can obtain the data returned by the register. If the corresponding register is being updated, it will be directly replaced with the updated data, avoiding simultaneous read and write operations on the same address.

[0107] Because the superscalar out-of-order processor uses register renaming technology, the FU actually uses physical registers for computation. The HiPU200 acceleration core contains 64 physical registers. The logical register AR0 is always mapped to the physical register PR0, and the remaining physical registers are mapped by the RAT (Rename List).

[0108] Stage 7: ALU Arithmetic and Logical Operations

[0109] The ALU (Arithmetic Logic Unit) stage is responsible for the specific execution of instructions. All operations handled by the ALU module can be completed within one cycle.

[0110] The source operands of the ALU module can be registers or immediate values. The executable instructions are categorized as: arithmetic, logical, comparison, and jump operations. See Table 1 for the various types of operations.

[0111] Instruction categories Detailed calculations arithmetic +，- logic &，|，～，&&，||，！ Compare ＞＝，＜ Jump Relative jump, absolute jump, conditional jump

[0112] Table 1

[0113] Stage: ALU writes back

[0114] The ALU write-back phase is used to write data back to the register bank and update the corresponding information in the ROB. If the instruction in this phase is valid and the current instruction has a destination register, the calculation result will be written to the register bank.

[0115] During this phase, instruction execution information will be synchronized to the ROB. If it is a normal arithmetic instruction, the ROB will be notified that the current instruction has been completed; if it is a jump judgment instruction, the ROB will be notified whether the current jump is correct, as well as the correct jump address and other information.

[0116] In addition, a bypass circuit is designed into the pipeline to ensure that write-back data is sent to the next issue instruction in advance. The write-back data is fed into the bypass network in advance, so that even if the data has not yet been written back to the register set, subsequent instructions can still obtain the correct data through the bypass network.

[0117] During the write-back phase, the instruction's destination register label also needs to be broadcast. If any instruction currently awaiting execution has a dependency on this destination register, the dependency is removed. Due to the presence of a bypass network in the design, this broadcast instruction can be sent in advance at a fixed number of clock cycles.

[0118] Stage 7-10: MDU Multiplication and Division Operations

[0119] The multiplication and division phase of the MDU is responsible for performing fixed-point multiplication and division operations on scalar data. Since multiplication and division operations are relatively complex and cannot be completed in one cycle, these two operations are divided into multiple cycles.

[0120] Multiplication operations are performed using a 3-stage pipelined multiplier, with data written back in the 3rd stage. Each cycle can compute a new set of multiplication operations.

[0121] Since the result of a 32-bit multiplication is 64 bits, if two consecutive multiplication instructions are encountered, which calculate the high and low bits of the product result respectively, the two operations will be combined.

[0122] The division operation uses a blocking 8-cycle delay divider. Since the computation result delay is fixed, the pipeline wake-up mechanism is relatively simple.

[0123] Stage 8-11: MDU Multiplication and Division Write Back

[0124] Similar to the write-back phase of the ALU, the multiplication and division write-back phase of the MDU is responsible for writing the multiplication and division results back to the corresponding physical registers, and at the same time notifying the ROB that the current instruction calculation is complete.

[0125] Since the divider may result in a division by zero, when notifying the ROB of the current instruction calculation status, it can also report the division by zero exception to the main control logic.

[0126] In another embodiment, the vector matrix section at the pipeline backend mainly includes vector matrix instruction wake-up, vector matrix instruction register read, vector matrix instruction execution, and vector matrix instruction write-back.

[0127] In this embodiment, the vector matrix receives instructions from the instruction dispatch module and places them into the instruction queue of the vector matrix. It is worth noting that, as... Figure 3 As shown, since each instruction in a vector matrix instruction operates on a large amount of data, allowing out-of-order execution would result in significant overhead during instruction rollback. Therefore, vector matrix instructions are executed in a fixed order; only when a vector instruction enters or exits the retired state in the ROB (Retire Instruction Block) is it actually sent to the subsequent pipeline for execution.

[0128] It's worth noting that matrix instructions have a relatively long execution time. To prevent them from blocking the pipeline, matrix instructions can be marked as completed in the ROB before execution. The status of matrix operations can be queried using scalar instructions.

[0129] Stage 5: Vector Matrix Command Wake-up

[0130] The vector matrix instruction is responsible for determining whether the oldest instruction can be woken up during the instruction wake-up phase. If the instruction's dependencies are completely removed, the current instruction is set as valid and sent to subsequent parts of the pipeline.

[0131] The vector matrix instruction queue has a depth of 16 and uses a standard FIFO structure for storage.

[0132] Vector matrix instructions use a scoreboard approach to arbitrate instruction dependencies.

[0133] Stage 6: Read register from vector matrix instruction

[0134] The vector matrix instruction is responsible for reading data from the vector register set during the register read phase. This data may be rewritten by the bypass network.

[0135] Because the vector matrix instruction is designed using a single-issue mode, only one instruction accesses the register set at a time. The vector register set is designed with two read interfaces and one write interface.

[0136] The vector register set contains 16 registers (VR0-VR15). VR0 is always 0; reading from VR0 always returns 0, and writing to VR0 has no effect. VR1-VR15 are general-purpose readable and writable registers.

[0137] Stage 7-N: Vector Matrix Instruction Execution

[0138] Vector-matrix instructions are responsible for the specific execution of vector and matrix operations during the execution phase. Vector operations, as shown in Table 2, include arithmetic, logical operations, comparisons, parameter configuration, predicate operations, memory access operations, and aggregation operations. The clock cycles consumed range from 1 to 5 cycles. To avoid excessive port usage during instruction write-back, the instruction write-back times are grouped into three types.

[0139] Matrix operations mainly include matrix-matrix, matrix-vector, and multiplication-addition and multiplication-accumulation operations at corresponding positions within a matrix. The time consumed is configured through parameters. If the configured number of loops is large, the instruction execution time will be very long. To prevent matrix instructions from blocking the pipeline, they are marked as completed in advance by the scoreboard.

[0140] Instruction categories Detailed calculations arithmetic +，-，* logic &，|，～，&&，||，！ Compare ＞＝，＜ Parameter configuration CSRW Predicate Operations swap, &, |, ~ Memory access operations vld, vst Aggregation operations fusion+ / - / *

[0141] Table 2

[0142] Stage[N+1]: Vector matrix instruction write-back

[0143] During the write-back phase, the vector matrix instruction is primarily responsible for writing the calculation results of the instruction back to the vector register set, while simultaneously notifying the score board to update the status of each register.

[0144] Similar to scalar processes, the computation data for vector matrix instructions is fed back to the preceding pipeline via a bypass network, thereby reducing instruction wait time.

[0145] In another embodiment, the pipeline back-end - memory access section

[0146] Pipeline memory access instructions are divided into two stages: one is the address generation part that solves register dependencies, which is processed in the AGU; the other is the memory access part that solves memory dependencies.

[0147] Weak consistency storage method and Fence instruction

[0148] The HiPU200 acceleration core uses a weak consistency model for its storage, which is based on two core principles:

[0149] For read and write operations that access the same or overlapping addresses, the order of operations must strictly follow the order of instructions.

[0150] For read and write operations that access different or non-overlapping addresses, the order can be arbitrarily changed.

[0151] Note that for two read operations accessing the same address, their order must strictly match the instruction order. This avoids some atomicity issues.

[0152] If there are specific requirements for the order of memory access reads and writes, the order of memory access operations can be explicitly specified using the Fence instruction.

[0153] Read / write command arbitration mechanism:

[0154] Because the HiPU200 acceleration core uses a cache mechanism, memory access operations may encounter cache misses, causing the operation to wait for a long time. However, subsequent instructions have a high probability of not encountering cache misses and can be accessed quickly. Therefore, a read / write caching mechanism is needed to cache instructions that cannot be accessed successfully at the moment, prioritizing subsequent read / write accesses. After a delay, the previously unsuccessful commands are then processed.

[0155] To decouple read and write instructions for memory access operations, the instruction cache queue is divided into two. Each cycle can issue one read operation and one write operation simultaneously.

[0156] Atomic operations:

[0157] In addition to regular read and write operations, the HiPU200 acceleration core also supports hardware atomic operations to solve the consistency problem of multiple acceleration cores accessing the same data.

[0158] The accelerated core supports standard AMO instructions, but does not support LR and SD instructions.

[0159] Because the memory access unit separates the Load queue and the Store queue, when processing atomic operations, it is necessary to establish a dependency relationship between the instructions in these two queues and select them out.

[0160] In another embodiment, a pipeline rollback mechanism is used.

[0161] Out-of-order processors execute instructions after the currently displayed program counter (PC) ahead of time. If a jump or conditional branch prediction error occurs, the prematurely executed instructions must be discarded, and registers contaminated by the premature execution must be restored. The HiPU200 acceleration core supports two rollback mechanisms: Flush rollback and Recovery rollback.

[0162] Flush rollback mechanism

[0163] The Flush rollback mechanism is the most reliable rollback mechanism for accelerating the core. The processor only recognizes the current instruction as completed when it retires from the ROB (Logical Register to Physical Register) table. At this point, the processor maintains a complete set of RAT (Logical Register to Physical Register Mapping Table), PRF (Physical Register Set), and PRM (Physical Register Status Table). If, during the retirement process, the processor finds that the next instruction is inconsistent with the predicted instruction, it initiates the Flush rollback mechanism, overwriting the existing entries in these three tables, clearing the pipeline, and restarting instruction fetching from the new instruction address.

[0164] The drawback of the flush rollback mechanism is its relatively slow response time. Because flushing requires clearing the pipeline, it introduces a delay of at least 8 cycles.

[0165] Recovery rollback mechanism

[0166] To compensate for the shortcomings of flush rollback, the accelerator also features a fast rollback mechanism: Recovery rollback. This mechanism can partially flush the instructions in the pipeline as soon as conditional statements or jump statements have finished executing, thus not affecting the final retire behavior of instructions in the ROB.

[0167] The Recovery mechanism also requires maintaining a backup of the RAT and PRM. Each backup corresponds to a rollback point. The HiPU200 accelerated core is designed with 8 backups, meaning that the processor can use the Recovery mechanism to roll back 8 conditional statements or jump statements that were executed prematurely. This greatly reduces the performance loss caused by instruction prediction errors.

[0168] In another embodiment, the processor microarchitecture design has been completed using TSMC's 28nm process. The fabricated chip employs an on-chip network (NOC) to form a 2D-MESH multi-core design, and its multi-core structure is as follows: Figure 4 As shown. The chip back-end layout and routing diagram is as follows. Figure 5 As shown. The chip's features are as follows:

[0169] (1) Die Size: 11.7x8.2x0.78mm

[0170] (2)Package: 40x40mm(FC BGA2300)

[0171] (3)28nm@1GHz, 1TOPs@INT8 / Core, 13 Core / Chip

[0172] (4) 2-way DDR4 2400MHz@64bits, 8xPCIe 3.0

[0173] (5) DNN acceleration kernel based on RISC-V self-developed extended instruction set

[0174] (6) Scalar instructions are executed out of order using multiple issue (SuperScalar OOO) to improve instruction parallelism.

[0175] (7) Customize vector and matrix instructions for AI inference operators, and improve data parallelism through in-order execution. MAC utilization is ≥90% during convolution operations.

[0176] (8) Realize inter-core data communication and smooth expansion of the number of cores based on the on-chip interconnect network NoC.

[0177] In this embodiment, we implemented a 4×4 2D-MESH structure with 13 computing cores. Using our reserved NOC interface, it can be expanded into any m×n 2D-MESH structure, thereby enabling the construction of computing chips with more cores.

[0178] Although the embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this disclosure is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of this disclosure, and all of these are within the scope of protection of this disclosure.

Claims

1. A general-purpose neural network AI processor microarchitecture based on RISC-V extensions, comprising: The processor comprises a front-end unit, an instruction decoding and dispatch unit, a scalar execution unit, a vector matrix execution unit, and a multi-level data storage unit; among which, The processor front-end unit consists of an instruction fetch module, a fast decoding module, an instruction prediction module, and an instruction buffer module, and supports parallel instruction fetching through four paths. The instruction decoding and distribution unit consists of an instruction decoding module, a register renaming module, and an instruction distribution module, and supports parallel decoding and distribution of two instructions. The scalar execution unit consists of a scalar ALU module, a scalar MDU module, a scalar LSU module, and a scalar register module, and employs completely out-of-order execution. The vector matrix execution unit consists of a vector matrix queue module, a matrix MAC module, a vector ALU module, a vector LSU module, and a vector register module, and is executed in a fixed order. The multi-level data storage unit consists of an L1 instruction buffer module, an L1 data buffer module, an L2 shared buffer module, an on-chip data memory (DTCM) module, and an on-chip SIMD local storage module.

2. The microarchitecture according to claim 1, wherein, Preferably, the vector matrix execution unit supports 8-bit integer multiplication and addition operations, supports 512 multiplications and 512 additions per cycle, and can complete an independent matrix multiplication of [8, 8], or a vector matrix multiplication of [1, 8] and [64, 8].

3. The microarchitecture according to claim 1, wherein, The vector matrix execution unit supports 8-bit, 16-bit, and 32-bit integer operations, performs operations on 64 integer data per cycle, and supports aggregate instructions that execute in multiple cycles with a single instruction.

4. The microarchitecture according to claim 1, wherein, The vector matrix execution unit employs a double-buffered renaming mechanism for the specified registers.

5. The microarchitecture according to claim 1, wherein, The vector matrix execution unit adopts a delayed execution strategy, and only begins sequential execution after ensuring that there are no branch jump instructions before the vector matrix instructions.

6. The microarchitecture according to claim 1, wherein, The multi-level data storage unit uses cache blocks of different granularities to achieve data management and balance between control code and computational data.

7. The microarchitecture according to claim 1, wherein, The microarchitecture's pipeline includes an 8-stage scalar basic pipeline, a scalar multiplication and division pipeline, a scalar memory access pipeline, and a vector matrix pipeline.

8. The microarchitecture according to claim 7, wherein, The scalar part of the pipeline backend mainly includes ALU / MDU / AGU instruction wake-up, ALU / MDU / AGU register read, ALU arithmetic and logical operations, ALU write-back, MDU multiplication and division operations, and MDU multiplication and division write-back.

9. The microarchitecture according to claim 7, wherein, The vector matrix section at the back end of the pipeline mainly includes vector matrix instruction wake-up, vector matrix instruction register read, vector matrix instruction execution, and vector matrix instruction write-back.

10. The microarchitecture according to claim 7, wherein, The common components of the pipeline backend mainly include instruction decoding register renaming and allocation, and instruction dispatch register renaming and updating.

Images