Artificial intelligence chip and operating method thereof

Abstract

The present application provides an artificial intelligence chip and an operating method thereof. The artificial intelligence chip comprises a tensor core, a tensor emission unit, a general-purpose computing core, a general-purpose emission unit, and a thread block splitting unit. In response to the thread block splitting unit operating in a first operating mode, the thread block splitting unit dispatches a thread bundle to the general-purpose emission unit, and the general-purpose emission unit emits instructions to the tensor core and the general-purpose computing core. In response to the thread block splitting unit operating in a second operating mode, the thread block splitting unit dispatches a tensor thread bundle related to tensor computation to the tensor emission unit, the tensor emission unit emits tensor computation instructions to the tensor core, the thread block splitting unit dispatches a non-tensor thread bundle not related to tensor computation to the general-purpose emission unit, and the general-purpose emission unit emits non-tensor computation instructions to the general-purpose computing core.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of integrated circuits, and in particular to an Artificial Intelligence (AI) chip and an operation method thereof. BACKGROUND

[0002] A warp is a basic execution unit in a Graphics Processing Unit (GPU), a General-Purpose GPU (GPGPU) chip or an Artificial Intelligence chip. Generally, each warp contains 32 threads. When the number of threads in a thread group or a thread block defined by a user is not an integer multiple of 32, the thread group or the thread block will be allocated to N+1 warps, where N represents an integer. A warp is a hardware-level thread set, and a thread group is a logical-level thread set. The threads in each warp are executed in a Single Instruction Multiple Thread (SIMT) manner, that is, the threads belonging to the same warp will execute the same instruction.

[0003] The general computing core and the tensor core share these warps. Because the same warp can only execute the same instruction stream (assembler), the instruction emission unit will emit instructions to the general computing core and the tensor core in the order of the instruction stream. In other words, the instruction emission unit cannot send instructions to the general computing core and the tensor core at the same time, but can only serially emit general computing instructions, tensor computing instructions and other instructions. The general computing core and the tensor core are physically independent. Under the prior art, the general computing core and the tensor core share the same warp scheduling and instruction emission unit, which causes their respective instruction execution to be not completely asynchronous. The general computing core and the tensor core can block each other, thereby reducing the execution efficiency. For example, if the warp calculated by the general computing core cannot exit in time, the tensor core cannot start the next warp, causing waste of computing power, i.e., long tail effect. How to improve the execution efficiency is one of the many technical issues in the field of Artificial Intelligence chips. SUMMARY

[0004] The present application provides an Artificial Intelligence chip and an operation method thereof to improve the execution efficiency.

[0005] In an embodiment of the present invention, the artificial intelligence chip includes multiple tensor cores, multiple tensor thread bundle scheduling and instruction issuance units (also referred to as tensor issuance units), multiple general-purpose computing cores, multiple general-purpose thread bundle scheduling and instruction issuance units (also referred to as general-purpose issuance units), and a thread block partitioning unit. A first tensor thread bundle scheduling and instruction issuance unit among the multiple tensor thread bundle scheduling and instruction issuance units is coupled to a first tensor core among the multiple tensor cores. A first general-purpose thread bundle scheduling and instruction issuance unit among the multiple general-purpose thread bundle scheduling and instruction issuance units is coupled to a first tensor core and a first general-purpose computing core among the multiple general-purpose computing cores. The thread block partitioning unit is coupled to the multiple tensor thread bundle scheduling and instruction issuance units and the multiple general-purpose thread bundle scheduling and instruction issuance units. The thread block partitioning unit partitions the current thread block into multiple thread bundles. In response to the thread block splitting unit operating in a first operating mode, the thread block splitting unit dispatches each of the plurality of thread bundles to one of the plurality of general thread bundle scheduling and instruction issuing units, and the first general thread bundle scheduling and instruction issuing unit issues each instruction of the current thread bundle to one of the first tensor core and the first general computing core. In response to the thread block splitting unit operating in a second operating mode, the thread block splitting unit dispatches each of at least one tensor thread bundle involving tensor computation to one of the plurality of tensor thread bundle scheduling and instruction issuing units, the first tensor thread bundle scheduling and instruction issuing unit issues the tensor computation instructions of the current tensor thread bundle to the first tensor core, the thread block splitting unit dispatches each of at least one non-tensor thread bundle not involving tensor computation to one of the plurality of general thread bundle scheduling and instruction issuing units, and the first general thread bundle scheduling and instruction issuing unit issues the non-tensor computation instructions of the current non-tensor thread bundle to the first general computing core.

[0006] In an embodiment of the present invention, the operation method includes: dividing a current thread block into multiple thread bundles by a thread block segmentation unit of an artificial intelligence chip; in response to the thread block segmentation unit operating in a first operation mode, dispatching each of the multiple thread bundles to one of a plurality of general-purpose thread bundle scheduling and instruction issuing units, and issuing each instruction of the current thread bundle to one of a first tensor core and a first general-purpose computing core by the first general-purpose thread bundle scheduling and instruction issuing unit; and in response to the thread block segmentation unit operating in a second operation mode, the thread block segmentation unit dispatching each of the multiple thread bundles into multiple thread bundles to one of a plurality of general-purpose thread bundle scheduling and instruction issuing units; and issuing each instruction of the current thread bundle to one of a first tensor core and a first general-purpose computing core by the first general-purpose thread bundle scheduling and instruction issuing unit; and in response to the thread block segmentation unit operating in a second operation mode, dispatching each of the multiple thread bundles into multiple thread bundles to one of a plurality of general-purpose thread bundle scheduling and instruction issuing units. Each of at least one tensor thread bundle involving tensor computation is dispatched to one of a plurality of tensor thread bundle scheduling and instruction issuing units. The first tensor thread bundle scheduling and instruction issuing unit issues the tensor computation instructions of the current tensor thread bundle to the first tensor core. The thread block splitting unit dispatches each of at least one non-tensor thread bundle that does not involve tensor computation to one of a plurality of general thread bundle scheduling and instruction issuing units. The first general thread bundle scheduling and instruction issuing unit issues the non-tensor computation instructions of the current non-tensor thread bundle to the first general computing core.

[0007] Based on the above, the thread block segmentation unit can dispatch tensor thread bundles involving tensor computation to the tensor thread bundle scheduling and instruction issuing unit, and dispatch non-tensor thread bundles not involving tensor computation to the general thread bundle scheduling and instruction issuing unit. Therefore, when the tensor thread bundle scheduling and instruction issuing unit issues tensor computation instructions for tensor thread bundles to the tensor core, the general thread bundle scheduling and instruction issuing unit can simultaneously issue non-tensor computation instructions for non-tensor thread bundles to the general computing core. In other words, the tensor core and the general computing core can execute their respective computational operations simultaneously. Therefore, in the second operating mode, the execution efficiency of the AI ​​chip can be effectively improved.

[0008] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of a circuit block of an artificial intelligence chip according to one embodiment;

[0010] Figure 2 This is a schematic diagram of a circuit module of an artificial intelligence chip according to an embodiment of the present invention;

[0011] Figure 3 This is a flowchart illustrating an operation method of an artificial intelligence chip according to an embodiment of the present invention;

[0012] Figure 4This is a schematic diagram of a circuit module of a tensor thread bundle scheduling and instruction issuing unit according to an embodiment of the present invention.

[0013] Explanation of icon numbers

[0014] 100: The first artificial intelligence chip;

[0015] 200: The second artificial intelligence chip;

[0016] 110: First thread block segmentation unit;

[0017] 210: Second thread block segmentation unit;

[0018] 120_1: First thread beam scheduling and instruction issuing unit;

[0019] 120_N: Nth thread beam scheduling and instruction issuing unit;

[0020] 130: First thread beam synchronization unit;

[0021] 240: Second thread beam synchronization unit;

[0022] 140_1: Tensor kernel number 1_1;

[0023] 140_N: The kernel of the 1_Nth tensor;

[0024] 250: Tensor kernel;

[0025] 250_1: Tensor kernel number 2_1;

[0026] 250_N: The kernel of the 2_Nth tensor;

[0027] 150_1: The first general-purpose computing core;

[0028] 150_N: The 1st general-purpose computing core;

[0029] 260_1: The 2nd general-purpose computing core;

[0030] 260_N: The 2nd general-purpose computing core;

[0031] 160: First shared memory area;

[0032] 270: Second shared memory area;

[0033] 211: Independent scheduling switch register;

[0034] 220: Tensor Thread Bundle Scheduling and Instruction Issuance Unit;

[0035] 220_1: Tensor Thread Bundle Scheduling and Command Issuance Unit;

[0036] 220_N: The Nth tensor thread bundle scheduling and instruction issuing unit;

[0037] 221: Buffer;

[0038] 222: Thread beam scheduling unit;

[0039] 223: Instruction decoding unit;

[0040] 224: Operand read unit;

[0041] 225: Scalar instruction execution unit;

[0042] 226: Scalar register set;

[0043] 227: Tensor kernel instruction forwarding unit;

[0044] 230_1: First general-purpose thread beam scheduling and instruction issuing unit;

[0045] 230_N: Nth general-purpose thread bundle scheduling and instruction issuing unit. Detailed Implementation

[0046] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same component symbols are used in the drawings and description to denote the same or similar parts.

[0047] The term "coupled (or connected)" as used throughout this specification (including the claims) may refer to any direct or indirect means of connection. For example, if the text describes a first device coupled (or connected) to a second device, it should be interpreted as the first device being directly connected to the second device, or the first device being indirectly connected to the second device through other devices or some means of connection. The terms "first," "second," etc., used throughout this specification (including the claims) are used to name components or distinguish different embodiments or scopes, and are not intended to limit the upper or lower limit of the number of components, nor to limit the order of components. Furthermore, wherever possible, components, parts, and steps using the same reference numerals in the drawings and embodiments represent the same or similar parts. Components, parts, and steps using the same reference numerals or the same terms in different embodiments may be referred to mutually in the relevant descriptions. It should be understood that the features of the following embodiments can be combined with each other. For example, the features of the second embodiment can be combined with the features of the first embodiment. Those skilled in the art can select appropriate combinations of features according to actual design requirements.

[0048] Computing devices such as artificial intelligence (AI) chips can provide enormous computing power. This immense computing power stems from the large number of internal hardware cores. An AI chip typically contains multiple programmable processors, such as a Stream Processor Cluster (SPC). Each programmable processor typically contains multiple Compute Units (CUs), and each Compute Unit typically contains multiple Execution Units (EUs), such as Tensor Cores (Tcores) and General-Purpose Compute Cores. General-Purpose Compute Cores typically include at least one of Integer (INT) cores, Floating Point (FP) cores, and Vector Cores (Vcores). By organizing these various types of Compute Units through programming, AI chips can support general-purpose computing, scientific computing, and neural network computing.

[0049] Figure 1 This is a schematic diagram of a circuit module of an artificial intelligence chip according to one embodiment. Figure 1 The first artificial intelligence chip 100 includes a first thread block segmentation unit 110, multiple thread bundle scheduling and instruction issuing units (e.g. Figure 1 The first thread beam scheduling and instruction issuing unit 120_1, ..., the Nth thread beam scheduling and instruction issuing unit 120_N), the first thread beam synchronization unit 130, and multiple tensor kernels (e.g. Figure 1 Tensor kernel 1_1, ..., tensor kernel 1_N, and multiple general-purpose computational kernels (e.g., tensor kernel 140_1, ..., tensor kernel 1_N, tensor kernel 140_N). Figure 1 The system comprises a first general-purpose computing core 150_1, ..., a first general-purpose computing core 150_N, and a first shared memory area 160. Based on actual design and application, the first general-purpose computing cores 150_1 to 150_N include at least one of integer cores, floating-point cores, and vector cores. The first thread bundle scheduling and instruction issuing unit 120_1 is coupled to the first tensor core 140_1 and the first general-purpose computing core 150_1, and so on, with the Nth thread bundle scheduling and instruction issuing unit 120_N coupled to the first tensor core 140_N and the first general-purpose computing core 150_N. The number N of the first thread bundle scheduling and instruction issuing units 120_1 to 120_N can be determined according to actual design and application.

[0050] The first thread block segmentation unit 110 is coupled to the first thread bundle scheduling and instruction issuing units 120_1 to the Nth thread bundle scheduling and instruction issuing units 120_N. The thread block segmentation unit (first thread block segmentation unit 110) segments the current thread block into multiple thread bundles. According to user-specified rules, the first thread block segmentation unit 110 dispatches all thread bundles to the first thread bundle scheduling and instruction issuing units 120_1 to the Nth thread bundle scheduling and instruction issuing units 120_N. For example, the first thread block segmentation unit 110 allocates multiple thread bundles of a thread block to the first thread bundle scheduling and instruction issuing units 120_1 to the Nth thread bundle scheduling and instruction issuing units 120_N in a polling or fixed order. The first thread block segmentation unit 110 dispatches each thread bundle to one of the first thread bundle scheduling and instruction issuing units 120_1 to the Nth thread bundle scheduling and instruction issuing units 120_N. The first thread bundle scheduling and instruction issuing unit 120_1 issues each instruction of the local thread bundle to one of the first_1 tensor core 140_1 and the first_1 general-purpose computing core 150_1. For example, the first thread bundle scheduling and instruction issuing unit 120_1 issues tensor computation instructions involving tensor computation to the first_1 tensor core 140_1, and issues non-tensor computation instructions not involving tensor computation to the first_1 general-purpose computing core 150_1. Similarly, the Nth thread bundle scheduling and instruction issuing unit 120_N issues tensor computation instructions of the local thread bundle to the first_N tensor core 140_N, and issues non-tensor computation instructions of the local thread bundle to the first_N general-purpose computing core 150_N.

[0051] The first thread bundle synchronization unit 130 is coupled to the first thread bundle scheduling and instruction issuing units 120_1 to the Nth thread bundle scheduling and instruction issuing units 120_N to receive user-written synchronization instructions in the thread bundle. The first thread bundle synchronization unit 130 selectively controls the instruction issuing of one or more of the first thread bundle scheduling and instruction issuing units 120_1 to the Nth thread bundle scheduling and instruction issuing units 120_N based on the synchronization instructions. The first shared memory area 160 is coupled to the first_1 tensor cores 140_1 to the first_N tensor cores 140_N and the first_1 general-purpose computing cores 150_1 to the first_N general-purpose computing cores 150_N. The first_1 tensor cores 140_1 to the first_N tensor cores 140_N and the first_1 general-purpose computing cores 150_1 to the first_N general-purpose computing cores 150_N can exchange data in the first shared memory area 160.

[0052] The first thread block segmentation unit 110 allocates multiple thread bundles of a thread block to the first thread bundle scheduling and instruction issuing units 120_1 to the Nth thread bundle scheduling and instruction issuing units 120_N according to a polling or fixed order. The first general-purpose computing cores 150_1 to 1_N and the first tensor cores 140_1 to 1_N share these thread bundles. Each thread bundle can only execute the same instruction stream (assembler), so the first thread bundle scheduling and instruction issuing units 120_1 to 1_N will issue instructions sequentially to the first general-purpose computing cores 150_1 to 1_N and the first tensor cores 140_1 to 1_N according to the order of the instruction stream. In other words, the instruction issuing device (each of the first thread beam scheduling and instruction issuing units 120_1 to the Nth thread beam scheduling and instruction issuing units 120_N) cannot send instructions to the general-purpose computing core and the tensor core simultaneously (for example, the first thread beam scheduling and instruction issuing unit 120_1 cannot send instructions to the first general-purpose computing core 150_1 and the first tensor core 140_1 simultaneously), but can only issue non-tensor computing instructions (general-purpose computing instructions), tensor computing instructions and other instructions serially.

[0053] The first general-purpose computing core 150_1 and the first tensor core 140_1 are physically independent, but share the first thread bundle scheduling and instruction issuing unit 120_1. If the thread bundle computed by the first general-purpose computing core 150_1 cannot exit in time, the first tensor core 140_1 cannot start the next thread bundle, resulting in wasted computing power and reduced execution efficiency. The same applies to the Nth thread bundle scheduling and instruction issuing unit 120_N, the Nth tensor core 140_N, and the Nth general-purpose computing core 150_N.

[0054] The following embodiments decouple the tensor core from the general-purpose computing core by introducing a dedicated thread bundle scheduling and instruction issuing unit for the tensor core. Logically, the thread bundles of the tensor core and the general-purpose computing core still belong to the same thread block, and they still execute the same program, only jumping to their respective program segments according to the thread bundle sequence number.

[0055] Figure 2 This is a schematic diagram of a circuit module of an artificial intelligence chip according to an embodiment of the present invention. Figure 2 In this embodiment, the second artificial intelligence chip 200 includes a second thread block segmentation unit 210, multiple tensor thread bundle scheduling and instruction issuing units (which may be referred to as tensor issuing units, for example...). Figure 2The first tensor thread bundle scheduling and instruction issuing unit 220_1, ..., the Nth tensor thread bundle scheduling and instruction issuing unit 220_N), and multiple general thread bundle scheduling and instruction issuing units (which can be called general issuing units, for example) Figure 2 The first general-purpose thread beam scheduling and instruction issuing unit 230_1, ..., the Nth general-purpose thread beam scheduling and instruction issuing unit 230_N), the second thread beam synchronization unit 240, and multiple tensor kernels (e.g., Figure 2 Tensor kernel 2_1, ..., tensor kernel 2_N, and multiple general-purpose computational kernels (e.g., tensor kernel 250_1, ..., tensor kernel 2_N, tensor kernel 250_N). Figure 2 The second general-purpose computing cores 260_1, ..., 260_N and the second shared memory area 270. The number N of the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing units 220_N can be determined according to the actual design and application.

[0056] In this context, any one of the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing unit 220_N can be equivalent to the first tensor thread bundle scheduling and instruction issuing unit, and any other one of the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing unit 220_N can be equivalent to the second tensor thread bundle scheduling and instruction issuing unit. For example, the first tensor thread bundle scheduling and instruction issuing unit 220_1 can be equivalent to the first tensor thread bundle scheduling and instruction issuing unit, and the Nth tensor thread bundle scheduling and instruction issuing unit 220_N can be equivalent to the second tensor thread bundle scheduling and instruction issuing unit.

[0057] Any one of the first general thread bundle scheduling and instruction issuing units 230_1 to the Nth general thread bundle scheduling and instruction issuing unit 230_N can be equivalent to the first general thread bundle scheduling and instruction issuing unit, and any other one of the first general thread bundle scheduling and instruction issuing units 230_1 to the Nth general thread bundle scheduling and instruction issuing unit 230_N can be equivalent to the second general thread bundle scheduling and instruction issuing unit. For example, the first general thread bundle scheduling and instruction issuing unit 230_1 can be equivalent to the first general thread bundle scheduling and instruction issuing unit, and the Nth general thread bundle scheduling and instruction issuing unit 230_N can be equivalent to the second general thread bundle scheduling and instruction issuing unit.

[0058] In this context, any one of the 2_1st tensor kernel 250_1 to the 2_Nth tensor kernel 250_N can be equivalent to the first tensor kernel, and any other one of the 2_1st tensor kernel 250_1 to the 2_Nth tensor kernel 250_N can be equivalent to the second tensor kernel. For example, the 2_1st tensor kernel 250_1 can be equivalent to the first tensor kernel, and the 2_Nth tensor kernel 250_N can be equivalent to the second tensor kernel.

[0059] In this context, any one of the second-1 general-purpose computing cores 260_1 to the second-N general-purpose computing cores 260_N can be equivalent to the first general-purpose computing core, and any other one of the second-1 general-purpose computing cores 260_1 to the second-N general-purpose computing cores 260_N can be equivalent to the second general-purpose computing core. For example, the second-1 general-purpose computing core 260_1 can be equivalent to the first general-purpose computing core, and the second-N general-purpose computing core 260_N can be equivalent to the second general-purpose computing core.

[0060] Depending on the design, in some embodiments, at least one of the above-mentioned second thread block segmentation unit 210, first tensor thread bundle scheduling and instruction issuing units 220_1 to Nth tensor thread bundle scheduling and instruction issuing units 220_N, first general thread bundle scheduling and instruction issuing units 230_1 to Nth general thread bundle scheduling and instruction issuing units 230_N, second thread bundle synchronization unit 240, second_1 tensor core 250_1 to second_N tensor core 250_N, and second_1 general computing core 260_1 to second_N general computing core 260_N can be implemented as hardware circuitry. In other embodiments, at least one of the following can be implemented as a combination of multiple hardware, firmware, and software (i.e., programs): the second thread block segmentation unit 210, the first tensor thread bundle scheduling and instruction issuing unit 220_1 to the Nth tensor thread bundle scheduling and instruction issuing unit 220_N, the first general thread bundle scheduling and instruction issuing unit 230_1 to the Nth general thread bundle scheduling and instruction issuing unit 230_N, the second thread bundle synchronization unit 240, the second_1 tensor core 250_1 to the second_N tensor core 250_N, and the second_1 general computing core 260_1 to the second_N general computing core 260_N.

[0061] In terms of hardware, at least one of the above-mentioned second thread block segmentation unit 210, first tensor thread bundle scheduling and instruction issuing units 220_1 to Nth tensor thread bundle scheduling and instruction issuing units 220_N, first general thread bundle scheduling and instruction issuing units 230_1 to Nth general thread bundle scheduling and instruction issuing units 230_N, second thread bundle synchronization unit 240, second_1 tensor core 250_1 to second_N tensor core 250_N, and second_1 general computing core 260_1 to second_N general computing core 260_N can be implemented as a logic circuit on an integrated circuit. For example, the related functions of at least one of the following components can be implemented in one or more hardware controllers, microcontrollers, hardware processors, microprocessors, application-specific integrated circuits (ASICs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), and central processing units (CPUs). Various logical blocks, modules, and circuits in a CPU or other processing unit. The related functions of at least one of the following: 2nd thread block segmentation unit 210, 1st tensor thread bundle scheduling and instruction issuance units 220_1 to Nth tensor thread bundle scheduling and instruction issuance units 220_N, 1st general-purpose thread bundle scheduling and instruction issuance units 230_1 to Nth general-purpose thread bundle scheduling and instruction issuance units 230_N, 2nd thread bundle synchronization unit 240, 2_1st tensor cores 250_1 to 2_Nth tensor cores 250_N, and 2_1st general-purpose computing cores 260_1 to 2_Nth general-purpose computing cores 260_N can be implemented as hardware circuits, such as various logical blocks, modules, and circuits in an integrated circuit, using hardware description languages ​​(e.g., Verilog HDL or VHDL) or other suitable programming languages.

[0062] In software or firmware form, the relevant functions of at least one of the following: the second thread block segmentation unit 210, the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing units 220_N, the first general thread bundle scheduling and instruction issuing units 230_1 to the Nth general thread bundle scheduling and instruction issuing units 230_N, the second thread bundle synchronization unit 240, the second_1 tensor core 250_1 to the second_N tensor core 250_N, and the second_1 general computing core 260_1 to the second_N general computing core 260_N can be implemented as programming codes. For example, at least one of the following can be implemented using general programming languages ​​(such as C, C++, or assembly language) or other suitable programming languages: the second thread block segmentation unit 210, the first tensor thread bundle scheduling and instruction issuance units 220_1 to the Nth tensor thread bundle scheduling and instruction issuance units 220_N, the first general-purpose thread bundle scheduling and instruction issuance units 230_1 to the Nth general-purpose thread bundle scheduling and instruction issuance units 230_N, the second thread bundle synchronization unit 240, the second_1 tensor cores 250_1 to the second_N tensor cores 250_N, and the second_1 general-purpose computing cores 260_1 to the second_N general-purpose computing cores 260_N. The programming code can be recorded and stored in a non-transitory machine-readable storage medium. In some embodiments, the non-transitory machine-readable storage medium includes, for example, semiconductor memory and / or a storage device. Electronic devices (e.g., CPUs, hardware controllers, microcontrollers, hardware processors, or microprocessors) can read and execute programming code from a non-transitory machine-readable storage medium to implement the relevant functions of at least one of the following: second thread block partitioning unit 210, first tensor thread bundle scheduling and instruction issuing units 220_1 to Nth tensor thread bundle scheduling and instruction issuing units 220_N, first general thread bundle scheduling and instruction issuing units 230_1 to Nth general thread bundle scheduling and instruction issuing units 230_N, second thread bundle synchronization unit 240, second_1 tensor cores 250_1 to 2_N tensor cores 250_N, and second_1 general computing cores 260_1 to 2_N general computing cores 260_N.

[0063] The second thread block segmentation unit 210 includes an independent scheduling switch register 211. The state of the independent scheduling switch register 211 depends on the kernel descriptor. The kernel descriptor is generated by the driver. Generally, the kernel descriptor is a structure that packages the address of the instruction binary file of the currently executable program, user configuration information, required hardware resources, etc., into a fixed format and serves as the unique identifier of the currently executable program. In response to the independent scheduling switch register 211 indicating that the switch is off, the second thread block segmentation unit 210 operates in the first operating mode. In response to the independent scheduling switch register 211 indicating that the switch is on, the second thread block segmentation unit 210 operates in the second operating mode.

[0064] Figure 3 This is a flowchart illustrating an operation method of an artificial intelligence chip according to an embodiment of the present invention. Please refer to... Figure 2 and Figure 3 In step S310, the second thread block segmentation unit 210 segments the current thread block into multiple thread bundles. The second thread block segmentation unit 210 is coupled to the first tensor thread bundle scheduling and instruction issuance units 220_1 to the Nth tensor thread bundle scheduling and instruction issuance units 220_N and the first general thread bundle scheduling and instruction issuance units 230_1 to the Nth general thread bundle scheduling and instruction issuance units 230_N. In response to the second thread block segmentation unit 210 operating in the first operating mode, the second thread block segmentation unit 210 dispatches each thread bundle to one of the first general thread bundle scheduling and instruction issuance units 230_1 to the Nth general thread bundle scheduling and instruction issuance units 230_N (step S320). The second thread block segmentation unit 210 operating in the first operating mode can be referred to... Figure 1 The relevant descriptions of the first thread block segmentation unit 110 shown are used and can be extrapolated from there, so they will not be repeated here.

[0065] In response to the second thread block splitting unit 210 operating in the second operating mode, the second thread block splitting unit 210 dispatches each tensor thread bundle involving tensor computation to one of the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing units 220_N, and the second thread block splitting unit 210 dispatches each non-tensor thread bundle not involving tensor computation to one of the first general thread bundle scheduling and instruction issuing units 230_1 to the Nth general thread bundle scheduling and instruction issuing units 230_N (step S340). In some application examples, the program descriptor can define and indicate tensor thread bundles involving tensor computations among multiple thread bundles, and the second thread block splitting unit 210 dispatches each tensor thread bundle to one of the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing units 220_N based on the program descriptor, and dispatches each non-tensor thread bundle to one of the first general thread bundle scheduling and instruction issuing units 230_1 to the Nth general thread bundle scheduling and instruction issuing units 230_N.

[0066] The second thread bundle synchronization unit 240 is coupled to the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing units 220_N and the first general thread bundle scheduling and instruction issuing units 230_1 to the Nth general thread bundle scheduling and instruction issuing units 230_N to receive synchronization instructions. The second thread bundle synchronization unit 240 selectively controls the instruction issuing of one or more of the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing units 220_N and the first general thread bundle scheduling and instruction issuing units 230_1 to the Nth general thread bundle scheduling and instruction issuing units 230_N based on the synchronization instructions. The second thread bundle synchronization unit 240 can be referred to... Figure 1 The relevant descriptions of the first thread bundle synchronization unit 130 shown are used and can be extrapolated from there, so they will not be repeated here.

[0067] The first tensor thread bundle scheduling and instruction issuing unit 220_1 is coupled to the second_1 tensor core 250_1, and the first general-purpose thread bundle scheduling and instruction issuing unit 230_1 is coupled to the second_1 tensor core 250_1 and the second_1 general-purpose computing core 260_1. Similarly, the tensor thread bundle scheduling and instruction issuing unit 220_N is coupled to the second_N tensor core 250_N, and the Nth general-purpose thread bundle scheduling and instruction issuing unit 230_N is coupled to the second_N tensor core 250_N and the second_N general-purpose computing core 260_N. Based on practical design and application, the second_1 general-purpose computing core 260_1 to the second_N general-purpose computing core 260_N include at least one of integer cores, floating-point cores, and vector cores. Tensor kernels 2_1 to 2_N (250_1 to 2_N) and general-purpose computation kernels 2_1 to 2_N (260_1 to 2_N) can be referenced. Figure 1 The relevant explanations of the 1_1 tensor kernel 140_1 to the 1_N tensor kernel 140_N and the 1_1 general computing kernel 150_1 to the 1_N general computing kernel 150_N shown are provided and can be deduced by analogy, so they will not be repeated here.

[0068] Here, the first tensor thread bundle scheduling and instruction issuing unit 220_1, the first general-purpose thread bundle scheduling and instruction issuing unit 230_1, the second-first tensor core 250_1, and the second-first general-purpose computing core 260_1 are used as illustrative examples. The remaining tensor thread bundle scheduling and instruction issuing units (e.g., the Nth tensor thread bundle scheduling and instruction issuing unit 220_N), the remaining general-purpose thread bundle scheduling and instruction issuing units (e.g., the Nth general-purpose thread bundle scheduling and instruction issuing unit 230_N), the remaining tensor cores (e.g., the second-nth tensor core 250_N), and the remaining general-purpose computing cores (e.g., the second-nth general-purpose computing core 260_N) can be deduced by referring to the relevant descriptions of the first tensor thread bundle scheduling and instruction issuing unit 220_1, the first general-purpose thread bundle scheduling and instruction issuing unit 230_1, the second-first tensor core 250_1, and the second-first general-purpose computing core 260_1, and therefore will not be elaborated further. In response to the second thread block segmentation unit 210 operating in the first operation mode, the second thread block segmentation unit 210 dispatches a thread bundle to the first general thread bundle scheduling and instruction issuing unit 230_1 (step S320), and so on, the second thread block segmentation unit 210 dispatches a thread bundle to the Nth general thread bundle scheduling and instruction issuing unit 230_N (step S320). The first general thread bundle scheduling and instruction issuing unit 230_1 issues each instruction of the current thread bundle to one of the second_1 tensor core 250_1 and the second_1 general computing core 260_1 (step S330), and so on, the Nth general thread bundle scheduling and instruction issuing unit 230_N issues each instruction of the current thread bundle to one of the second_N tensor core 250_N and the second_N general computing core 260_N (step S330). In the first operating mode, the first tensor thread bundle scheduling and instruction issuance unit 220_1 to the Nth tensor thread bundle scheduling and instruction issuance unit 220_N are idle. The second artificial intelligence chip 200 operating in the first operating mode can be referenced... Figure 1 The following is a description of the first artificial intelligence chip 100.

[0069] In response to the second thread block segmentation unit 210 operating in the second operating mode, the second thread block segmentation unit 210 dispatches tensor thread bundles involving tensor computation from among multiple thread bundles to the first tensor thread bundle scheduling and instruction issuing unit 220_1, and the first tensor thread bundle scheduling and instruction issuing unit 220_1 issues the tensor computation instruction of the current tensor thread bundle to the second_1 tensor core 250_1 (step S340). In response to the second thread block segmentation unit 210 operating in the second operating mode, the second thread block segmentation unit 210 dispatches non-tensor thread bundles not involving tensor computation from among multiple thread bundles to the first general thread bundle scheduling and instruction issuing unit 230_1, and the first general thread bundle scheduling and instruction issuing unit 230_1 issues the non-tensor computation instruction of the current non-tensor thread bundle to the second_1 general computing core 260_1 (step S350). This embodiment introduces a first tensor thread bundle scheduling and instruction issuing unit 220_1 dedicated to the second_1 tensor core 250_1, decoupling the second_1 tensor core 250_1 from the second_1 general-purpose computing core 260_1 in the second operating mode. Similarly, the second thread block splitting unit 210 dispatches non-tensor thread bundles to the Nth general-purpose thread bundle scheduling and instruction issuing unit 230_N, and the Nth general-purpose thread bundle scheduling and instruction issuing unit 230_N issues non-tensor computation instructions for the non-tensor thread bundles to the second_Nth general-purpose computing core 260_N (step S350).

[0070] The second shared memory area 270 is coupled to the second_1 tensor cores 250_1 to the second_N tensor cores 250_N and the second_1 general-purpose computing cores 260_1 to the second_N general-purpose computing cores 260_N. The second_1 tensor cores 250_1 to the second_N tensor cores 250_N and the second_1 general-purpose computing cores 260_1 to the second_N general-purpose computing cores 260_N can exchange data in the second shared memory area 270. The second shared memory area 270 can be referenced... Figure 1 The relevant descriptions of the first shared storage area 160 shown are as follows, and by analogy, they will not be repeated here.

[0071] In summary, the second thread block splitting unit 210 can dispatch tensor thread bundles involving tensor computation to the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing units 220_N, and dispatch non-tensor thread bundles that do not involve tensor computation to the first general thread bundle scheduling and instruction issuing units 230_1 to the Nth general thread bundle scheduling and instruction issuing units 230_N. Therefore, when the first tensor thread bundle scheduling and instruction issuing unit 220_1 to the Nth tensor thread bundle scheduling and instruction issuing unit 220_N issues tensor computation instructions of tensor thread bundles to the second_1 tensor cores 250_1 to the second_N tensor cores 250_N, the first general thread bundle scheduling and instruction issuing unit 230_1 to the Nth general thread bundle scheduling and instruction issuing unit 230_N can simultaneously issue non-tensor computation instructions of non-tensor thread bundles to the second_1 general computing cores 260_1 to the second_N general computing cores 260_N. That is, the second_1 tensor cores 250_1 to the second_N tensor cores 250_N and the second_1 general computing cores 260_1 to the second_N general computing cores 260_N can simultaneously execute their respective computation operations. Therefore, in the second operating mode, the execution efficiency of the second artificial intelligence chip 200 can be effectively improved.

[0072] Figure 4 This is a schematic diagram of a circuit module of a tensor thread bundle scheduling and instruction issuing unit according to an embodiment of the present invention. Figure 4 The tensor thread bundle scheduling and instruction issuing unit 220 shown can be used as... Figure 2 This is one of many implementation examples of each of the first tensor thread bundle scheduling and instruction issuing units 220_1 to the Nth tensor thread bundle scheduling and instruction issuing units 220_N shown. Figure 4 The second thread block segmentation unit 210, tensor thread bundle scheduling and instruction issuing unit 220, second thread bundle synchronization unit 240, and tensor core 250 shown can be referenced. Figure 1 The descriptions of the second thread block segmentation unit 210, the first tensor thread bundle scheduling and instruction issuance unit 220_1 to the Nth tensor thread bundle scheduling and instruction issuance unit 220_N, the second thread bundle synchronization unit 240, and the first_1 tensor core 140_1 to the first_N tensor core 140_N shown are omitted here.

[0073] exist Figure 4In this embodiment, the tensor thread bundle scheduling and instruction issuing unit 220 includes a buffer 221, a thread bundle scheduling unit 222, an instruction decoding unit 223, an operand reading unit 224, a scalar instruction execution unit 225, a scalar register file 226, and a tensor kernel instruction forwarding unit 227. The buffer 221 is coupled to the second thread block splitting unit 210. The second thread block splitting unit 210 dispatches the tensor thread bundle corresponding to the tensor thread bundle scheduling and instruction issuing unit 220 to the buffer 221. The tensor thread bundle includes functions such as reading instructions and checking whether thread bundle synchronization is complete, which will not be described in detail here.

[0074] Thread bundle scheduling unit 222 is coupled to buffer 221. Thread bundle scheduling unit 222 selects and issues an instruction for a thread bundle in buffer 221 to instruction decoding unit 223. For example, thread bundle scheduling unit 222 selects a thread bundle from the thread bundles in buffer 221 using a polling or fixed priority method and issues an instruction for the selected thread bundle to instruction decoding unit 223.

[0075] Instruction decoding unit 223 is coupled to thread beam scheduling unit 222 to receive instructions. Instruction decoding unit 223 decodes the received instructions to generate a decoding result for operand reading unit 224. For example, instruction decoding unit 223 parses the instruction field and passes the parsed instruction opcode, operand address, and other information to operand reading unit 224. Operand reading unit 224 is coupled to instruction decoding unit 223, scalar register set 226, scalar instruction execution unit 225, and tensor kernel instruction forwarding unit 227.

[0076] The instruction segment belonging to the tensor thread bundle in the program contains not only tensor kernel instructions but also some scalar computation instructions. These scalar computation instructions are used to calculate information such as the starting address and matrix size of subsequent tensor kernel instructions. Therefore, a scalar register set 226 and a scalar instruction execution unit 225 are required. In response to the decoding result of the instruction decoding unit 223 representing the scalar computation instruction, the operand read unit 224 sends the scalar computation instruction to the scalar instruction execution unit 225. The scalar instruction execution unit 225 writes the execution result of the scalar computation instruction (such as the starting address, matrix size, etc.) to the scalar register set 226.

[0077] In response to the decoding result of the instruction decoding unit 223 representing the tensor kernel instruction, the operand read unit 224 reads the corresponding information (such as start address, matrix size, etc.) of the tensor kernel instruction from the scalar register group 226 based on the decoding result. The operand read unit 224 sends the tensor kernel instruction and corresponding information to the tensor kernel instruction forwarding unit 227. The tensor kernel instruction forwarding unit 227 sends the tensor kernel instruction and corresponding information to the corresponding tensor kernel 250.

[0078] The advantages of this embodiment include:

[0079] 1. The general-purpose computing core and the tensor core have their own thread beam scheduling and instruction issuing mechanisms. The instruction execution of the general-purpose computing core and the tensor core can be completely asynchronous and will not block each other.

[0080] 2. If the tensor kernel finishes all computations first, the thread bundle can exit in time, and the tensor kernel can start the computation of the next thread bundle, thus eliminating the long tail effect and improving the utilization rate of tensor kernel computing power.

[0081] 3. Logically, the thread bundles of the tensor core and the general-purpose computing core still belong to the same thread block. Synchronization between the tensor core and the general-purpose computing core is still completed within a single thread block, which makes it easy to perform synchronization and data exchange within the thread block.

[0082] 4. Backward compatibility. If the switch for independent thread bundle scheduling and instruction issuance of the tensor core is not turned on (independent scheduling switch register 211), that is, if the second thread block splitting unit 210 operates in the first operating mode, the second artificial intelligence chip 200 can be fully compatible with the scheme of sharing thread bundles between the tensor core and the general computing core.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An artificial intelligence chip, characterized in that, The artificial intelligence chip includes: Multiple tensor kernels; Multiple tensor thread bundle scheduling and instruction issuing units, wherein a first tensor thread bundle scheduling and instruction issuing unit among the multiple tensor thread bundle scheduling and instruction issuing units is coupled to a first tensor core among the multiple tensor cores; Multiple general-purpose computing cores; Multiple general-purpose thread bundle scheduling and instruction issuance units, wherein a first general-purpose thread bundle scheduling and instruction issuance unit is coupled to a first tensor core and a first general-purpose computing core among the multiple general-purpose computing cores; and a thread block splitting unit, coupled to the multiple tensor thread bundle scheduling and instruction issuance units and the multiple general-purpose thread bundle scheduling and instruction issuance units, wherein the thread block splitting unit splits the current thread block into multiple thread bundles. In response to the thread block splitting unit operating in a first operating mode, the thread block splitting unit dispatches each of the plurality of thread bundles to one of the plurality of general thread bundle scheduling and instruction issuing units, and the first general thread bundle scheduling and instruction issuing unit issues each instruction of the current thread bundle to one of the first tensor core and the first general computation core, wherein in the first operating mode, the plurality of tensor thread bundle scheduling and instruction issuing units are idle; and in response to the thread block splitting unit operating in a second operating mode, the thread block splitting unit dispatches at least one tensor thread bundle involving tensor computation from the plurality of thread bundles. Each thread bundle is assigned to one of the plurality of tensor thread bundle scheduling and instruction issuing units. The first tensor thread bundle scheduling and instruction issuing unit issues tensor computation instructions for the current tensor thread bundle to the first tensor core. The thread block splitting unit assigns each of at least one non-tensor thread bundle among the plurality of thread bundles that does not involve tensor computation to one of the plurality of general thread bundle scheduling and instruction issuing units. The first general thread bundle scheduling and instruction issuing unit issues non-tensor computation instructions for the current non-tensor thread bundle to the first general computing core, thereby decoupling the tensor core from the general computing core in the second operation mode.

2. The artificial intelligence chip according to claim 1, characterized in that, The second tensor thread beam scheduling and instruction issuance unit among the plurality of tensor thread beam scheduling and instruction issuance units is coupled to the second tensor core among the plurality of tensor cores. The second general-purpose thread beam scheduling and instruction issuance unit among the plurality of general-purpose thread beam scheduling and instruction issuance units is coupled to the second tensor core and the second general-purpose computing core among the plurality of general-purpose computing cores. In response to the thread block splitting unit operating in the first operating mode, the second general thread bundle scheduling and instruction issuing unit issues each instruction of the local current thread bundle to one of the second tensor core and the second general computing core; In response to the thread block splitting unit operating in the second operating mode, the second tensor thread bundle scheduling and instruction issuing unit issues tensor computation instructions of the local current tensor thread bundle to the second tensor core, and the second general thread bundle scheduling and instruction issuing unit issues non-tensor computation instructions of the local current non-tensor thread bundle to the second general computing core.

3. The artificial intelligence chip according to claim 1, characterized in that, The thread block segmentation unit includes: Independent scheduling switch register, wherein the state of the independent scheduling switch register depends on the program descriptor. In response to the independent scheduling switch register indicating that the switch is off, the thread block splitting unit operates in the first operating mode; and in response to the independent scheduling switch register indicating that the switch is on, the thread block splitting unit operates in the second operating mode.

4. The artificial intelligence chip according to claim 1, characterized in that, The thread block splitting unit, based on the program descriptor, dispatches each of the at least one tensor thread bundle to one of the plurality of tensor thread bundle scheduling and instruction issuing units, and dispatches each of the at least one non-tensor thread bundle to one of the plurality of general thread bundle scheduling and instruction issuing units.

5. The artificial intelligence chip according to claim 1, characterized in that, The artificial intelligence chip also includes: A thread bundle synchronization unit is coupled to the plurality of tensor thread bundle scheduling and instruction issuing units and the plurality of general thread bundle scheduling and instruction issuing units to receive synchronization instructions, wherein the thread bundle synchronization unit selectively controls the instruction issuing of one or more of the plurality of tensor thread bundle scheduling and instruction issuing units and the plurality of general thread bundle scheduling and instruction issuing units based on the synchronization instructions.

6. The artificial intelligence chip according to claim 1, characterized in that, The artificial intelligence chip also includes: A shared storage area is coupled to the plurality of tensor cores and the plurality of general-purpose computing cores, wherein the plurality of tensor cores and the plurality of general-purpose computing cores exchange data in the shared storage area.

7. The artificial intelligence chip according to claim 1, characterized in that, Each of the plurality of tensor thread bundle scheduling and instruction issuing units includes: A buffer, coupled to the thread block splitting unit, wherein the thread block splitting unit dispatches the corresponding of the at least one tensor thread bundle to the buffer; A thread beam scheduling unit, coupled to the buffer, wherein the thread beam scheduling unit selects and issues an instruction for a thread beam in the buffer; An instruction decoding unit is coupled to the thread bundle scheduling unit to receive the instruction, wherein the instruction decoding unit decodes the instruction to generate a decoding result. Scalar register set; Scalar instruction execution unit; A tensor core instruction forwarding unit; and an operand reading unit, coupled to the instruction decoding unit, the scalar register group, the scalar instruction execution unit, and the tensor core instruction forwarding unit, wherein... In response to the decoding result representing a scalar computation instruction, the operand read unit sends the scalar computation instruction to the scalar instruction execution unit, and the scalar instruction execution unit writes the execution result of the scalar computation instruction to the scalar register set; and in response to the decoding result representing a tensor core instruction, the operand read unit reads the corresponding information of the tensor core instruction from the scalar register set based on the decoding result, the operand read unit sends the tensor core instruction and the corresponding information to the tensor core instruction forwarding unit, and the tensor core instruction forwarding unit sends the tensor core instruction and the corresponding information to the corresponding tensor core among the plurality of tensor cores.

8. A method for operating an artificial intelligence chip, characterized in that, The operation method includes: The current thread block is divided into multiple thread bundles by the thread block segmentation unit of the artificial intelligence chip. The thread block segmentation unit is coupled to multiple tensor thread bundle scheduling and instruction issuing units and multiple general thread bundle scheduling and instruction issuing units of the artificial intelligence chip. The first tensor thread bundle scheduling and instruction issuing unit of the multiple tensor thread bundle scheduling and instruction issuing units is coupled to the first tensor core of the multiple tensor cores of the artificial intelligence chip. The first general thread bundle scheduling and instruction issuing unit of the multiple general thread bundle scheduling and instruction issuing units is coupled to the first tensor core and the first general computing core of the multiple general computing cores of the artificial intelligence chip. In response to the thread block splitting unit operating in a first operating mode, the thread block splitting unit dispatches each of the plurality of thread bundles to one of the plurality of general thread bundle scheduling and instruction issuing units, and the first general thread bundle scheduling and instruction issuing unit issues each instruction of the current thread bundle to one of the first tensor core and the first general computing core. In the first operating mode, the plurality of tensor thread bundle scheduling and instruction issuing units are idle. And in response to the thread block splitting unit operating in a second operating mode, the thread block splitting unit dispatches at least one tensor thread bundle from the plurality of thread bundles that involves tensor computation. Each thread bundle is assigned to one of the plurality of tensor thread bundle scheduling and instruction issuing units. The first tensor thread bundle scheduling and instruction issuing unit issues the tensor computation instruction of the current tensor thread bundle to the first tensor core. The thread block splitting unit assigns each of at least one non-tensor thread bundle among the plurality of thread bundles that does not involve the tensor computation to one of the plurality of general thread bundle scheduling and instruction issuing units. The first general thread bundle scheduling and instruction issuing unit issues the non-tensor computation instruction of the current non-tensor thread bundle to the first general computing core, thereby decoupling the tensor core from the general computing core in the second operation mode.

9. The operating method according to claim 8, characterized in that, The second tensor thread beam scheduling and instruction issuance unit in the plurality of tensor thread beam scheduling and instruction issuance units is coupled to the second tensor core in the plurality of tensor cores, the second general-purpose thread beam scheduling and instruction issuance unit in the plurality of general-purpose thread beam scheduling and instruction issuance units is coupled to the second tensor core and the second general-purpose computing core in the plurality of general-purpose computing cores, and the operation method further includes: In response to the thread block splitting unit operating in the first operating mode, the second general-purpose thread bundle scheduling and instruction issuing unit issues each instruction of the local current thread bundle to one of the second tensor core and the second general-purpose computing core; and in response to the thread block splitting unit operating in the second operating mode, the second tensor thread bundle scheduling and instruction issuing unit issues tensor computation instructions of the local current tensor thread bundle to the second tensor core, and the second general-purpose thread bundle scheduling and instruction issuing unit issues non-tensor computation instructions of the local current non-tensor thread bundle to the second general-purpose computing core.

10. The operating method according to claim 8, characterized in that, The operation method further includes: In response to the independent scheduling switch register of the thread block splitting unit indicating that the switch is off, the thread block splitting unit operates in the first operating mode, wherein the state of the independent scheduling switch register depends on the program descriptor; and in response to the independent scheduling switch register indicating that the switch is on, the thread block splitting unit operates in the second operating mode.

11. The operating method according to claim 8, characterized in that, The operation method further includes: The thread block splitting unit dispatches each of the at least one tensor thread bundle to one of the plurality of tensor thread bundle scheduling and instruction issuing units based on the program descriptor, and dispatches each of the at least one non-tensor thread bundle to one of the plurality of general thread bundle scheduling and instruction issuing units.

12. The operating method according to claim 8, characterized in that, The operation method further includes: The thread bundle synchronization unit of the artificial intelligence chip selectively controls the instruction issuance of one or more of the plurality of tensor thread bundle scheduling and instruction issuance units and the plurality of general thread bundle scheduling and instruction issuance units based on synchronization instructions, wherein the thread bundle synchronization unit is coupled to the plurality of tensor thread bundle scheduling and instruction issuance units and the plurality of general thread bundle scheduling and instruction issuance units to receive the synchronization instructions.

13. The operating method according to claim 8, characterized in that, The operation method further includes: The plurality of tensor cores and the plurality of general-purpose computing cores exchange data in the shared memory area of ​​the artificial intelligence chip, wherein the shared memory area is coupled to the plurality of tensor cores and the plurality of general-purpose computing cores.

14. The operating method according to claim 8, characterized in that, Each of the plurality of tensor thread beam scheduling and instruction issuing units includes a buffer, a thread beam scheduling unit, an instruction decoding unit, an operand reading unit, a scalar register set, a scalar instruction execution unit, and a tensor kernel instruction forwarding unit. The operation method further includes: The thread block splitting unit dispatches the corresponding entity from the at least one tensor thread bundle to the buffer, wherein the buffer is coupled to the thread block splitting unit, and the thread bundle scheduling unit is coupled to the buffer; An instruction for a thread bundle in the buffer is selected and transmitted by the thread bundle scheduling unit, wherein the instruction decoding unit is coupled to the thread bundle scheduling unit to receive the instruction; The instruction decoding unit decodes the instruction to generate a decoding result, wherein the operand reading unit is coupled to the instruction decoding unit, the scalar register group, the scalar instruction execution unit, and the tensor kernel instruction forwarding unit; In response to the decoding result representing a scalar computation instruction, the operand reading unit sends the scalar computation instruction to the scalar instruction execution unit, and the scalar instruction execution unit writes the execution result of the scalar computation instruction to the scalar register set; and in response to the decoding result representing a tensor kernel instruction, the operand reading unit reads the corresponding information of the tensor kernel instruction from the scalar register set based on the decoding result, the operand reading unit sends the tensor kernel instruction and the corresponding information to the tensor kernel instruction forwarding unit, and the tensor kernel instruction forwarding unit sends the tensor kernel instruction and the corresponding information to the corresponding tensor kernel among the plurality of tensor kernels.

Images