A TPU based on a RISC-V custom tensor extension instruction set

Abstract

The application discloses a TPU based on a RISC-V self-defined tensor expansion instruction set, which comprises a multi-core system composed of four TPU cores, a multi-stage storage unit, an on-chip SRAM, an MCI control module and a PCIe interface. Each TPU core comprises a RISC-V scalar instruction processing unit, a scalar calculation unit, an L1 i-cache, an exclusive L1T cache, a core LSU, a tensor / vector register stack and a tensor / vector calculation unit. The TPU separates the instruction path from the data path, supports parallel or cooperative operation of the scalar, vector and tensor calculation units, and supports tensor configuration loading, data transfer and calculation operation through the self-defined tensor expansion instruction. While maintaining the RISC-V ecological compatibility, the application improves the tensor data transfer efficiency and parallel calculation efficiency, and reduces the energy consumption and scheduling overhead in the deep learning calculation scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence processors, RISC-V instruction set extensions, and deep learning acceleration hardware technology, and particularly to a TPU based on a custom tensor extension instruction set of RISC-V. Background Technology

[0002] With the rapid development of artificial intelligence technologies, represented by deep learning, applications such as image recognition, speech processing, natural language processing, and multimodal models are placing higher demands on matrix operation capabilities, on-chip storage bandwidth, and data reuse efficiency. Traditional CPUs have advantages in general control, branch processing, and system software ecosystem, but they are easily limited by computational density and storage bandwidth in terms of large-scale tensor parallel computing and energy efficiency.

[0003] GPUs improve deep learning throughput through massively parallel execution units and dedicated matrix / tensor computation units, but their general-purpose graphics and parallel computing architectures may still suffer from high energy consumption and large data transfer overhead in some inference scenarios. Dedicated TPUs or domain-specific AI chips can perform hardware optimization for core operators such as convolution and matrix multiplication and addition, but if they adopt closed or highly customized instruction sets, they may incur costs in terms of software migration, debugging, and ecosystem compatibility.

[0004] The RISC-V instruction set architecture is characterized by its openness, simplicity, modularity, and scalability, and provides extension space for user-defined instructions, resulting in a well-developed software and hardware ecosystem. However, the RISC-V basic instruction set and general vector extensions do not directly solve problems in deep learning tensor operators such as data handling, tensor shape configuration, matrix block computation, and multi-level cache coordination.

[0005] Therefore, it is necessary to propose a custom tensor-extended TPU architecture based on RISC-V. While maintaining the compatibility of the RISC-V instruction ecosystem, this architecture improves tensor computation efficiency, data reuse efficiency, and system programmability through dedicated tensor instructions, clearly separated instruction and data paths, and the combination of multi-core shared and core-exclusive memory hierarchies. Summary of the Invention

[0006] The purpose of this invention is to provide a TPU based on a custom tensor extension instruction set of RISC-V, in order to solve the problems of insufficient throughput of existing general-purpose processors in tensor computing scenarios, insufficient compatibility of dedicated AI chip software ecosystem, and complex scheduling caused by the mixing of instruction stream and tensor data stream.

[0007] This invention provides a TPU based on a RISC-V custom tensor extension instruction set, comprising a multi-core system consisting of four TPUCores, multi-level memory units, on-chip SRAM, an MCI control module, and a PCIe interface. Each TPUCore includes a RISC-V scalar instruction processing unit, a scalar computation unit, an L1 i-Cache, an L1T Cache, an LSU, a tensor register file and / or a vector register file, a tensor computation unit, and a vector computation unit. The RISC-V scalar instruction processing unit includes a PC counter, an instruction fetch module, a decoding module, and a dispatch and control module. The TPU has separate instruction and data paths. The instruction path is used to fetch and dispatch scalar instructions, vector instructions, and custom tensor extension instructions. The data path is used to transport tensor data and / or vector data and send them to the corresponding computation units. The RISC-V scalar instruction processing unit is responsible for PC maintenance, instruction fetching, decoding, dispatching and control; the scalar computation unit is responsible for performing scalar operations and address calculations; and the tensor computation unit and vector computation unit are responsible for performing highly parallel tensor / vector operations.

[0008] As a further improvement of the present invention, the MCI control module is used for consistency management and scheduling control among multiple TPU Cores, scheduling multiple TPU Cores, sharing storage resources and computing units according to system status information, and managing shared access among the TPU Cores; the MCI control module is connected to the host machine via PCIe, and the host machine writes program instructions or control descriptors to the on-chip SRAM through the PCIe and the MCI control module.

[0009] As a further improvement of the present invention, the instruction path sequentially includes the PCIe and / or the MCI control module, the on-chip SRAM, the L1 i-Cache, the PC counter in the RISC-V scalar instruction processing unit, the instruction fetch module, the decoding module, and the dispatch and control module; the dispatch and control module sends the scalar instruction to the scalar computation unit, and sends the vector instruction and the custom tensor extension instruction to the vector computation unit and the tensor computation unit respectively through the vector control path and the tensor control path.

[0010] As a further improvement of the present invention, the data path sequentially includes HBM, shared L2 Cache, L1T Cache, LSU, tensor register file and / or vector register file, and tensor computation unit and / or vector computation unit; wherein input data, weight data and intermediate results are written to HBM by the host machine or DMA, and enter the corresponding register file through the data path, and the computation results of each computation unit are written back in reverse along the data path.

[0011] As a further improvement of the present invention, the decoding module decodes RISC-V 64-bit general scalar instructions, RISC-V 64-bit vector extension instructions, and Tensor extension instructions implemented based on the RISC-V custom encoding space; wherein, the Tensor extension instructions are different from the naming convention of the standard RISC-V single-letter extensions.

[0012] As a further improvement of the present invention, the scalar computation unit includes a scalar register and an ALU for performing RISC-V scalar operations and address calculations; the LSU is used for data alignment, loading, storage, and rearrangement operations between the L1T Cache and the tensor register file and / or vector register file.

[0013] As a further improvement of the present invention, the tensor computation unit includes multiple tensor execution units (EUs), which are organized in a two-dimensional array or a grouped array for performing parallel computations on tensor data. The parallel computations include one or more of matrix multiplication and addition, convolution, and accumulation operations. The vector computation unit includes multiple vector EUs, which are used to perform parallel computations on vector data. The parallel computations include one or more of vector arithmetic operations, logical operations, and reduction operations.

[0014] As a further improvement of the present invention, the custom Tensor expansion instructions include configuration loading instructions, data transfer instructions, and data computation instructions; the configuration loading instructions are used to load the tensor shape, step size, offset, operation mode, and data type; the data transfer instructions are used to transfer data between the HBM, the shared L2 Cache, the L1T Cache, the LSU, and the tensor register file and / or vector register file; the data computation instructions are used to trigger the tensor computation unit to perform tensor operations.

[0015] As a further improvement of the present invention, the tensor calculation unit supports at least one of the following data formats: INT8, W4A16, and FP16; wherein INT8 represents an 8-bit integer, W4A16 represents a quantization format of 4-bit integer weights and 16-bit activation values, and FP16 represents a 16-bit floating-point format.

[0016] As a further improvement of the present invention, the tensor calculation unit, vector calculation unit, and scalar calculation unit constitute three independent calculation pipelines, which can execute tensor, vector, and scalar calculation instructions in parallel or in concert.

[0017] As a further improvement of the present invention, the scalar register includes 32 64-bit general-purpose registers; the tensor register file and / or the vector register file includes 32 registers and is allocated 4096 bytes of storage space, forming a tensor register file and a vector register file that are logically isolated from each other but can be accessed uniformly by the control logic.

[0018] As a further improvement of the present invention, the shared L2 Cache and the L1T Cache adopt a channel-based cache organization scheme. Tensor data is stored in the format [N,C,W,H], where N represents the batch, C represents the number of channels, W represents the feature map width, and H represents the feature map height. The data of each channel is stored independently, and the same channel of different tensors is mapped to the same storage block. When the number of channels exceeds the available storage blocks, a round-robin method is used for allocation.

[0019] The beneficial effects of this invention are as follows: First, it implements tensor instructions based on a custom extension of RISC-V, maintaining compatibility and migration foundation with the RISC-V ecosystem and compiler / runtime; Second, the tensor computation unit, vector computation unit, and scalar computation unit each undertake computation tasks of different granularities, and can execute different computation instructions in parallel when there are no data dependencies or resource conflicts, thereby improving system throughput and operating efficiency; Third, the instruction path and data path are separated, avoiding mutual interference between the instruction cache and the tensor data cache, which is conducive to the parallel overlap of instruction control, data handling, and computation execution; Fourth, the L1T Cache works in conjunction with the shared L2 Cache, which is beneficial for multi-core data sharing, consistency management, and tensor data reuse. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is an exemplary embodiment of the present invention, illustrating the overall architecture of a TPU based on a custom tensor extension instruction set of RISC-V. It shows the RISC-V scalar instruction processing unit, scalar computation unit, L1 i-Cache, tensor / vector register file, tensor computation unit, vector computation unit, LSU, L1T Cache within a single TPU Core, and a multi-core system consisting of four TPU Cores, a shared L2 Cache, HBM, SRAM, MCI, and PCIe.

[0022] Figure 2 This is an instruction path diagram of a TPU based on a custom tensor extension instruction set of RISC-V, as described in an exemplary embodiment of the present invention. It shows the process by which the host writes instructions to the on-chip SRAM via PCIe / MCI, fetches instructions using L1 i-Cache, and the RISC-V scalar instruction processing unit completes PC, fetch, decode, dispatch and control, and sends scalar instructions or tensor / vector control information to the corresponding computing unit.

[0023] Figure 3 This is a data path diagram of a TPU based on a custom tensor extension instruction set of RISC-V, as described in an exemplary embodiment of the present invention. It shows the process by which tensor / vector data is written from the host machine or DMA to the HBM, and then enters the tensor computation unit or vector computation unit through the shared L2 cache, the core-exclusive L1T cache, the core LSU, and the tensor / vector register file. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0026] Furthermore, the terminology used in the description of this invention is for illustrative purposes only and is not intended to limit the scope of the invention. The terms "comprising" and / or "including" are used to specify the presence of said elements, steps, operations, and / or components, but do not exclude the presence or addition of one or more other elements, steps, operations, and / or components. The terms "first," "second," etc., may be used to describe various elements, do not represent an order, and do not limit these elements. Moreover, in the description of this invention, unless otherwise stated, "a plurality of" means two or more. These terms are used only to distinguish one element from another. These and / or other aspects become apparent in conjunction with the following drawings, and those skilled in the art will more readily understand the description of the embodiments of the invention. The drawings are used for illustrative purposes only to depict the embodiments of the invention. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown in the invention can be employed without departing from the principles of the invention.

[0027] like Figure 1 As shown in the embodiment of the present invention, a TPU based on a RISC-V custom tensor extension instruction set includes a multi-core system consisting of four TPU Cores, a multi-level storage unit, on-chip SRAM, an MCI control module, and a PCIe interface; the four TPU Cores include TPU Core0, TPU Core1, TPU Core2, and TPU Core3; the multiple TPU Cores are connected to HBM, on-chip SRAM, MCI, and PCIe interface through a shared L2 cache.

[0028] Each TPU Core internally includes a RISC-V scalar instruction processing unit and a scalar computation unit. The RISC-V scalar instruction processing unit includes a PC counter, an L1 i-Cache access interface, an instruction fetch module, a decoding module, and a dispatch and control module, which are mainly responsible for program control, tensor / vector instruction decoding, and computation unit startup control. The scalar computation unit includes a scalar register and an ALU, which are mainly responsible for scalar operations and address calculations.

[0029] Each TPU Core further includes a tensor / vector register file, a tensor computation unit, a vector computation unit, an LSU, and an L1T cache. The tensor computation unit consists of multiple EUs, which can be organized in a two-dimensional array or a grouped array, and is used to perform tensor operations. The vector computation unit consists of multiple vector EUs and is used to perform vector operations.

[0030] TPU stands for Tensor Processing Unit, which is used to accelerate core deep learning computations such as tensor data handling, matrix multiplication and addition, and convolution.

[0031] MCI: Memory Control Interface, is a multi-core control and interconnect module used to connect PCIe, storage tiers, and multiple TPU Cores, and to manage task scheduling, shared access, and consistency-related controls.

[0032] LSU: Load / Store Unit, which is used to perform data loading, storage, alignment, and rearrangement between cache, register file, and compute unit.

[0033] L1 i-Cache and L1T Cache: L1 i-Cache is a level 1 instruction cache used for instruction fetching; L1T Cache is a core-exclusive level 1 tensor data cache used for tensor / vector data access.

[0034] Tensor / Vector Register File: A high-speed register resource located between the LSU and the tensor / vector computation unit, used to temporarily store tensor blocks, vector data, and computation results to be computed.

[0035] EU: Execution Unit. Both tensor computation units and vector computation units can be composed of multiple EUs to improve parallel computing capabilities.

[0036] On-chip SRAM: A high-speed static memory integrated inside the chip, used to temporarily store instructions, data, tensor blocks, and intermediate calculation results.

[0037] HBM: High Bandwidth Memory, which typically uses multi-layer DRAM stacking and wide bit-width interfaces to provide high-throughput data read and write capabilities for TPU Cores, tensor computing units, or other highly parallel computing modules.

[0038] PCIe: Peripheral Component Interconnect Express, is a high-speed serial computer expansion bus interface used to connect host processors, accelerators, storage devices, or other peripherals.

[0039] In one embodiment, the MCI control module is used for consistency management and scheduling control among multiple TPU Cores. It schedules multiple TPU Cores, shares storage resources, and manages shared access among the TPU Cores based on system status information. The MCI control module is connected to the host machine via PCIe, and the host machine writes program instructions or control descriptors to the on-chip SRAM through the PCIe and the MCI control module. The MCI control module receives task and control information from the host machine and performs scheduling based on multi-core running status, cache access status, and computing resource occupancy status.

[0040] In one implementation, such as Figure 2 As shown, the instruction path enters from the host side and sequentially passes through the PCIe and / or the MCI control module, the on-chip SRAM, the L1 i-Cache, the PC counter in the RISC-V scalar core, the instruction fetch module, the decoding module, and the dispatch and control module.

[0041] In the instruction path, the host machine writes the program to be executed, task descriptor, or microcode fragment into the on-chip SRAM via PCIe / MCI. The L1 i-Cache fetches instructions from the on-chip SRAM and provides a stable instruction stream to the RISC-V scalar instruction processing unit. The PC counter inside the RISC-V scalar instruction processing unit maintains the instruction address. The instruction fetch module fetches instructions from the L1 i-Cache according to the PC counter. The decoding module identifies the instruction type and then sends the instruction to the corresponding computation unit for execution via the dispatch and control module. For RISC-V scalar instructions, the dispatch and control module sends them to the scalar computation unit for execution; for RISC-V vector extension instructions and custom Tensor extension instructions, the dispatch and control module generates corresponding control information or microinstructions and sends them to the tensor computation unit or vector computation unit via the tensor / vector control path.

[0042] The instruction path transmits instructions and control information, and does not carry large-scale tensor data. The L1 i-Cache is used for instruction access, and the core-exclusive L1T Cache is used for tensor / vector data access. The two are separated in structure and function, thereby reducing the conflict between instruction fetching and tensor data handling.

[0043] In one implementation, such as Figure 3As shown, the data path enters from the host side and sequentially passes through the HBM, the shared L2 Cache, the L1T Cache, the LSU, the tensor / vector register file, and the tensor calculation unit and / or the vector calculation unit; wherein the input data, weight data, and intermediate results are written to the HBM by the host or DMA and enter the corresponding register file through the data path, and the calculation results of each calculation unit are written back in reverse along the data path.

[0044] In this data path, HBM is responsible for providing large-capacity and high-bandwidth data storage, the shared L2 cache is responsible for cross-core data reuse and sharing, the core-dedicated L1T cache is responsible for reducing single-core access latency, LSU is responsible for alignment, loading, storage, masking, and rearrangement, and the tensor / vector register file is responsible for high-speed temporary storage before and after computation. This path is independent of the instruction path shown in Figure 2, allowing data prefetching, DMA transfer, configuration loading, and computation execution to overlap in time.

[0045] In one embodiment, the decoding module decodes RISC-V 64-bit general scalar instructions (RV64G), RISC-V 64-bit vector extension instructions (RV64V), and Tensor extension instructions implemented based on a custom RISC-V encoding space.

[0046] Among them, RV64G: RISC-V 64-bit general-purpose combined instruction set, which typically includes basic and commonly used standard extensions such as I, M, A, F, and D, used to implement general-purpose scalar program control and scalar operations.

[0047] RV64V: RISC-V 64-bit Vector Extension Instruction Set Configuration, which introduces V vector extensions on the RV64 architecture to support vector parallel operations in single-instruction multiple-data form.

[0048] Custom Tensor Extension Instruction Set: This set includes instructions for tensor configuration, tensor data transfer, and tensor computation based on a custom RISC-V encoding space. To avoid confusion with other standard or reserved single-letter extensions in RISC-V, this manual does not use the standard single-letter T extension naming convention, but instead refers to it simply as "Tensor Extension" or "Tensor Extension".

[0049] In one embodiment, the scalar computation unit includes a scalar register and an ALU for performing RISC-V scalar operations and address calculations; the tensor computation unit includes multiple tensor execution units (EUs), organized in a two-dimensional array or a grouped array, for performing parallel computations on tensor data, the parallel computations including one or more of matrix multiplication-addition, convolution, and accumulation operations; the vector computation unit includes multiple vector EUs, which are used to perform parallel operations on vector data, the parallel computations including one or more of vector arithmetic operations, logical operations, and reduction operations. The vector EUs can also perform data preprocessing and some post-processing operations. The LSU is used for data alignment, loading, storage, and rearrangement operations between the L1T Cache and the tensor / vector register file.

[0050] In one embodiment, the custom tensor expansion instructions include configuration loading instructions, data transfer instructions, and data computation instructions, as shown in Table 1 below. The configuration loading instructions, after decoding, can update the core's internal configuration registers or control buffers to load the tensor shape, step size, offset, operation mode, and data type. The data transfer instructions are used to transfer data between the HBM, the shared L2 Cache, the L1T Cache, the LSU, and the tensor register file and / or vector register file, as shown in Table 2 below. The data computation instructions are used to trigger the tensor computation unit to perform tensor operations, as shown in Table 3 below.

[0051] Table 1 Examples of Tensor Expansion Instruction Categories

[0052] Data transfer instructions can be selected in different forms depending on the tensor layout and computational requirements. The mnemonics in the table below are used to illustrate one embodiment of the invention; the specific encoding can be adjusted according to the RISC-V custom instruction encoding rules.

[0053] Table 2 Examples of Tensor Data Transfer Instructions

[0054] Tensor data computation instructions use matrix blocks or tensor blocks as their basic operation objects. The tensor computation unit can further convert the decoded tensor computation instructions into microinstructions or control sequences for multiple EUs, enabling multiple EUs to execute in parallel in a pipelined or array manner.

[0055] Table 3 Examples of Tensor Data Calculation Commands

[0056] In one embodiment, the tensor computation unit supports at least one of multiple data formats, including INT8, W4A16, and FP16; wherein INT8 represents an 8-bit integer, W4A16 represents a quantization format of 4-bit integer weights and 16-bit activation values, and FP16 represents a 16-bit floating-point format.

[0057] In one implementation, the tensor computation unit, vector computation unit, and scalar computation unit constitute three independent computation pipelines. When there are no data dependencies or resource conflicts between scalar instructions, vector instructions, and tensor extension instructions, the distribution and control module can enable scalar computation, vector computation, and tensor computation to run in parallel. When dependencies exist, the control logic ensures the correctness of the results through methods such as pausing, forwarding, renaming, or sequential submission.

[0058] In one embodiment, the scalar register includes 32 64-bit general-purpose registers for storing data, addresses, and control variables processed by RISC-V scalar instructions. The tensor register file and / or the vector register file include 32 registers and are allocated 4096 bytes of storage space, forming a tensor register file and a vector register file that are logically isolated from each other but can be accessed uniformly by the control logic.

[0059] In one implementation, the shared L2 Cache and the L1T Cache employ a channel-based cache organization scheme. Tensor data is stored in the format [N,C,W,H], where N represents the batch size, C represents the number of channels, W represents the feature map width, and H represents the feature map height. Data for each channel is stored independently to improve the efficiency of continuous access and reuse within the same channel. The same channels of different tensors are mapped to the same storage block to quickly locate data in the same channel during convolution, matrix block computation, or channel-by-channel post-processing. When the number of channels exceeds the available storage blocks, a round-robin allocation method is used. Within a single cache, all channel data of the previous tensor can be stored first, followed by the data of the next tensor, with a sequential priority storage method used within each channel.

[0060] This cache organization method, in conjunction with the data path shown in Figure 3, ensures that data movement between HBM, shared L2 Cache, L1T Cache, LSU, and register file has a clear data source and destination. This helps reduce invalid movement, improve on-chip data reuse, and support coordinated access to shared data by multiple TPU Cores.

[0061] In one embodiment, the execution flow of the TPU of the present invention is exemplified as follows: The first step involves the host machine issuing task descriptors, program instructions, and configuration blocks via PCIe / MCI. The MCI writes the program instructions to the on-chip SRAM and can write input data, weight data, or DMA descriptors to the HBM or relevant control registers.

[0062] In the second step, the RISC-V scalar instruction processing unit fetches instructions from the on-chip SRAM through the L1 i-Cache. The decoding module identifies scalar instructions, vector instructions, and custom tensor extension instructions, and the dispatch and control module sends them to the corresponding execution units.

[0063] The third step involves configuring the loading instruction to write the tensor shape, step size, offset, data type, and operation mode into the configuration register group inside the TPUCore. This configuration register group is connected to the distribution and control module and provides control parameters to the tensor computation unit, vector computation unit, and LSU. The data transfer instruction controls the LSU to load tensor blocks from the L2 Cache or L1T Cache into the tensor / vector register file according to the configuration.

[0064] In the fourth step, the tensor computation instruction starts the EU array in the tensor computation unit to perform convolution or matrix multiplication and addition, the vector computation instruction starts the vector computation unit to perform vector preprocessing, postprocessing or auxiliary computation, and the scalar computation unit continues to be responsible for loops, branches and address updates.

[0065] The fifth step involves writing the calculation results back to the tensor / vector register file, and then writing them back to HBM via the LSU, L1T Cache, and shared L2 Cache, or providing them to other TPU Cores for further processing via the shared L2 Cache.

[0066] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0067] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention and form different embodiments.

[0068] Those skilled in the art will understand that although the invention has been described with reference to exemplary embodiments, various changes may be made and its elements may be substituted with equivalents without departing from the scope of the invention. Furthermore, many modifications may be made to adapt particular situations or materials to the teachings of the invention without departing from the essential scope of the invention.

Claims

1. A TPU based on a RISC-V custom tensor extension instruction set, characterized in that, The system comprises a multi-core system consisting of four TPU Cores, multi-level storage units, on-chip SRAM, an MCI control module, and a PCIe interface. The four TPU Cores are TPU Core0, TPU Core1, TPU Core2, and TPU Core3. Each TPU Core includes a RISC-V scalar instruction processing unit, a scalar computation unit, an L1 i-Cache, an L1T Cache, an LSU, a tensor register file and / or a vector register file, a tensor computation unit, and a vector computation unit. The RISC-V scalar instruction processing unit includes a PC counter, an instruction fetch module, a decoding module, and a dispatch and control module. Each TPU has separate instruction and data paths. The instruction path is used to fetch and dispatch scalar instructions, vector instructions, and custom tensor extension instructions. The data path is used to transport tensor data and / or vector data and send it to the corresponding computation unit.

2. The TPU as described in claim 1, characterized in that, The MCI control module is used for consistency management and scheduling control among multiple TPU Cores. It schedules multiple TPU Cores, shares storage resources and computing units according to system status information, and manages shared access among the TPU Cores. The MCI control module is connected to the host machine via PCIe. The host machine writes program instructions or control descriptors to the on-chip SRAM through the PCIe and the MCI control module.

3. The TPU as described in claim 1, characterized in that, The instruction path sequentially includes the PCIe and / or the MCI control module, the on-chip SRAM, the L1 i-Cache, the PC counter in the RISC-V scalar instruction processing unit, the instruction fetch module, the decoding module, and the dispatch and control module; the dispatch and control module sends the scalar instruction to the scalar computation unit, and sends the vector instruction and the custom tensor extension instruction to the vector computation unit and the tensor computation unit respectively through the vector control path and the tensor control path.

4. The TPU as described in claim 1, characterized in that, The data path sequentially includes HBM, shared L2Cache, L1T Cache, LSU, tensor register file and / or vector register file, and tensor computation unit and / or vector computation unit; wherein input data, weight data and intermediate results are written to HBM by the host or DMA, and enter the corresponding register file through the data path, and the computation results of each computation unit are written back in reverse along the data path.

5. The TPU as described in claim 1, characterized in that, The decoding module decodes RISC-V 64-bit general scalar instructions, RISC-V 64-bit vector extension instructions, and Tensor extension instructions implemented based on a custom RISC-V encoding space; wherein, the Tensor extension instructions differ from the naming convention of the standard RISC-V single-letter extensions.

6. The TPU as described in claim 1, characterized in that, The scalar computation unit includes a scalar register and an ALU, used to perform RISC-V scalar operations and address calculations; the LSU is used to perform alignment, loading, storage, and rearrangement operations between the L1T Cache and the tensor register file and / or vector register file.

7. The TPU as described in claim 1, characterized in that, The tensor computation unit includes multiple tensor execution units (EUs), which are organized in a two-dimensional array or a grouped array to perform parallel computations on tensor data. The parallel computations include one or more of matrix multiplication and addition, convolution, and accumulation operations. The vector computation unit includes multiple vector EUs, which are used to perform parallel computations on vector data. The parallel computations include one or more of vector arithmetic operations, logical operations, and reduction operations.

8. The TPU as described in claim 4, characterized in that, The custom tensor expansion instructions include configuration loading instructions, data transfer instructions, and data calculation instructions; the configuration loading instructions are used to load the tensor shape, step size, offset, operation mode, and data type; the data transfer instructions are used to transfer data between the HBM, the shared L2 Cache, the L1T Cache, the LSU, and the tensor register file and / or vector register file; the data calculation instructions are used to trigger the tensor calculation unit to perform tensor operations.

9. The TPU as described in claim 1, characterized in that, The tensor computation unit supports at least one of the following data formats: INT8, W4A16, and FP16. INT8 represents an 8-bit integer, W4A16 represents a quantization format of 4-bit integer weights and 16-bit activation values, and FP16 represents a 16-bit floating-point format.

10. The TPU as claimed in claim 1, characterized in that, The tensor computation unit, vector computation unit, and scalar computation unit constitute three independent computation pipelines, which can execute tensor, vector, and scalar computation instructions in parallel or collaboratively.

11. The TPU as claimed in claim 7, characterized in that, The scalar registers include 32 64-bit general-purpose registers; the tensor register file and / or the vector register file include 32 registers and are allocated 4096 bytes of storage space, forming a tensor register file and a vector register file that are logically isolated from each other but can be accessed uniformly by the control logic.

12. The TPU as claimed in claim 4, characterized in that, The shared L2 Cache and the L1T Cache adopt a channel-based cache organization scheme. Tensor data is stored in the format [N,C,W,H], where N represents the batch size, C represents the number of channels, W represents the feature map width, and H represents the feature map height. The data of each channel is stored independently, and the same channel of different tensors is mapped to the same storage block. When the number of channels exceeds the available storage blocks, a round-robin allocation method is used.

Images