Vector core module of artificial intelligence chip and its operation method

Abstract

The present invention provides a vector core module of an artificial intelligence chip and an operating method thereof, for improving the efficiency of the vector core module in an application scenario where "the vector core is the master and the tensor core module is the slave." The vector core module includes a vector core instruction execution pipeline, a tensor core instruction processing pipeline, and an instruction scheduling unit. The instruction scheduling unit performs instruction classification to distinguish vector core instructions and tensor core instructions from thread bundles. In response to the thread bundle including a vector core instruction, the instruction scheduling unit sends the vector core instruction to the vector core instruction execution pipeline. The vector core instruction execution pipeline executes the vector core instruction and stores the execution result in a memory module. In response to the thread bundle including a tensor core instruction, the instruction scheduling unit sends the tensor core instruction to the tensor core instruction processing pipeline. The tensor core instruction processing pipeline processes the tensor core instruction and sends the processing result to the tensor core module.

Description

Technical Field

[0001] The present invention relates to an artificial intelligence (AI) chip, and in particular to a vector core module and an operating method thereof. Background Art

[0002] Computing devices such as artificial intelligence (AI) chips can provide enormous computing power. This immense computing power stems from their numerous internal hardware cores. An AI chip typically contains multiple programmable multiprocessor clusters, such as a stream processor cluster (SPC). Each programmable multiprocessor cluster typically contains multiple programmable multiprocessors, such as multiple compute units (CUs, or cores), and each CU typically contains multiple execution units (EUs, or cores), such as at least one of an integer (INT) core module, a floating-point (FP) core module, a tensor core (Tcore) module, and a vector core (Vcore) module. By programming and organizing various types of compute units, programmable multiprocessors can support general-purpose computing, scientific computing, and neural network computing.

[0003] The Tensor Core module is a domain-specific architecture (DSA) for executing AI operations. It handles specialized AI operators, including but not limited to tensor data manipulation, matrix multiply and add (MMA), and convolution. In addition to operation-related control information, the Tensor Core module typically uses configuration registers to obtain additional tensor data tags (including but not limited to the tensor's address coordinates, tensor size, and boundary zero padding). The contents of these registers are not statically configured, but rather dynamically configured by instructions. However, they can be generated using only fixed-point operations or simple data shifts, typically scalar operations.

[0004] Single Instruction Multiple Threads (SIMT) instructions are a common programming instruction for graphics processing units (GPUs). SIMT instructions have been widely used for large-scale parallel computing tasks, such as graphics rendering and AI operations. GPU hardware executes SIMT instructions through vector core modules to perform the corresponding vector operations. Vector operations include but are not limited to floating-point operations, fixed-point operations, and logical operations. The pipeline architecture of the vector core modules supports large-scale parallel computing tasks. This pipeline architecture includes multiple processes, such as instruction fetch, instruction scheduling, decoding, operand fetching, operation execution, and result writeback. Vector core modules are also often used as master modules for memory modules and tensor core modules. Based on SIMT instructions, the vector core module (master module) generates operation control information and tensor core configuration information for slave modules (such as tensor core modules).

[0005] Using conventional vector core modules to send tensor core instruction configuration information presents the following problems: Due to the reusability requirements of SIMT computing resources, the latency of even fixed-point instructions is relatively long, affected by floating-point instructions. This results in a relatively slow speed for conventional vector core modules to send tensor core configuration information. Furthermore, tensor core configuration information only requires fixed-point operations or data movement, and only works in a single thread. Therefore, the resource utilization of conventional vector core modules running tensor core instructions is very low. In addition, vector core instructions and tensor core instructions share the same launch unit and subsequent pipeline, which hinders the efficiency of vector core instructions with higher performance requirements, resulting in a gap between actual computing power and expected computing power. In the application scenario of "vector core module as host and tensor core module as slave", how to improve the efficiency of vector core modules is one of the many technical issues in the field of AI chips. Summary of the Invention

[0006] The present invention provides a vector core module and an operation method thereof, for improving the efficiency of the vector core module in an application scenario where "the vector core module is a host and the tensor core module is a slave".

[0007] In an embodiment according to the present invention, the vector core module includes a vector core instruction execution pipeline, a tensor core instruction processing pipeline, and an instruction scheduling unit. The vector core instruction execution pipeline executes vector core instructions and stores the execution results in a memory module. The tensor core instruction processing pipeline processes tensor core instructions and sends the processing results to the tensor core module. The instruction scheduling unit is coupled to the vector core instruction execution pipeline and the tensor core instruction processing pipeline. The instruction scheduling unit classifies instructions for at least one thread warp to distinguish between vector core instructions and tensor core instructions from the at least one thread warp. In response to the at least one thread warp including a vector core instruction, the instruction scheduling unit sends the vector core instruction to the vector core instruction execution pipeline. In response to the at least one thread warp including a tensor core instruction, the instruction scheduling unit sends the tensor core instruction to the tensor core instruction processing pipeline.

[0008] In an embodiment according to the present invention, the operating method includes: the instruction scheduling unit of the vector core module classifies instructions on at least one thread bundle to distinguish vector core instructions and tensor core instructions from the at least one thread bundle; in response to the at least one thread bundle including a vector core instruction, the instruction scheduling unit sends the vector core instruction to the vector core instruction execution pipeline of the vector core module; the vector core instruction execution pipeline executes the vector core instruction and stores the execution result in the memory module; in response to the at least one thread bundle including a tensor core instruction, the instruction scheduling unit sends the tensor core instruction to the tensor core instruction processing pipeline of the vector core module; and the tensor core instruction processing pipeline processes the tensor core instruction and sends the processing result to the tensor core module.

[0009] Based on the above, the vector core module adds a tensor core instruction processing pipeline (an independent pipeline for generating and emitting tensor core configuration information). The tensor core instruction processing pipeline only runs tensor core instructions, while vector core instructions are executed by the vector core instruction execution pipeline. In some embodiments, the newly added tensor core instruction processing pipeline processes only scalar data and does not perform vector-to-scalar conversion operations. Therefore, the latency of the tensor core instruction processing pipeline is much lower than that of the vector core instruction execution pipeline, and transmission to the tensor core module is relatively fast. In some embodiments, the newly added tensor core instruction processing pipeline only supports fixed-point addition, fixed-point multiplication, fixed-point multiply-add, and simple data transfer. Therefore, the latency of the tensor core instruction processing pipeline is much lower than that of the vector core instruction execution pipeline. Compared to the vector core instruction execution pipeline, the tensor core instruction processing pipeline transmits tensor core configuration information to the tensor core module relatively quickly. In the application scenario where the vector core module is the master and the tensor core module is the slave, the vector core module of the present invention can improve efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1is a schematic diagram of a circuit block of a vector core module according to one embodiment;

[0011] Figure 2 is a circuit module diagram of a vector core module according to an embodiment of the present invention;

[0012] Figure 3 is a flowchart of an operating method of a vector core module according to an embodiment of the present invention;

[0013] Figure 4 is a schematic diagram of a circuit module of an instruction scheduling unit according to an embodiment of the present invention;

[0014] Figure 5 is a schematic diagram of a circuit module of an instruction scheduling unit according to another embodiment of the present invention;

[0015] Figure 6 FIG. 2 is a schematic diagram of a circuit module of a tensor core instruction operation unit according to an embodiment of the present invention.

[0016] Explanation of Figure Numbers

[0017] 11. Memory module 1,

[0018] 21. Second memory module,

[0019] 12. The first tensor core module,

[0020] 22. The second tensor core module,

[0021] 100. The first vector core module,

[0022] 200, the second vector core module,

[0023] 110, 1st instruction cache,

[0024] 210, 2nd instruction cache,

[0025] 120. First instruction acquisition unit,

[0026] 220, a second instruction acquisition unit,

[0027] 130. The first instruction transmitting unit,

[0028] 241, the second instruction transmission unit,

[0029] 251, the third instruction transmission unit,

[0030] 140, the first instruction decoding unit,

[0031] 242, the second instruction decoding unit,

[0032] 252, the third instruction decoding unit,

[0033] 150, 1st vector register group,

[0034] 260, 2nd vector register group,

[0035] 160, 1st scalar register group,

[0036] 270, 2nd scalar register group,

[0037] 170, the first vector core operation unit,

[0038] 243, the second vector core operation unit,

[0039] 180. The first effective thread detection unit,

[0040] 280, the second effective thread detection unit,

[0041] 230, instruction dispatch unit,

[0042] 231. Instruction Classifier,

[0043] 232_1, the first warp instruction classifier,

[0044] 232_2, the second warp instruction classifier,

[0045] 232_n, nth warp instruction classifier,

[0046] 233, vector core instruction polling arbiter,

[0047] 234, Tensor Core instruction polling arbiter,

[0048] 240, vector core instruction execution pipeline,

[0049] 250, Tensor Core instruction processing pipeline,

[0050] 253, tensor core instruction operation unit,

[0051] 610, fixed-point multiplier,

[0052] 620, the first gate,

[0053] 630, the second gate,

[0054] 650, the third gate,

[0055] 640, fixed-point adder,

[0056] a6, the first operand,

[0057] b6, the second operand,

[0058] c6, the third operand. DETAILED DESCRIPTION

[0059] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0060] The term "coupled (or connected)" as used throughout the specification of this case (including the claims) may refer to any direct or indirect means of connection. For example, if a first device is described as being coupled (or connected) to a second device, it should be interpreted as meaning that the first device can be directly connected to the second device, or that the first device can be indirectly connected to the second device through other devices or some other means of connection. The terms "first," "second," etc. mentioned throughout the specification of this case (including the claims) are used to name components or to distinguish different embodiments or scopes, and are not used to limit the upper or lower limit of the number of components, nor to limit the order of components.

[0061] Single Instruction Multiple Threads (SIMT) instructions are a common programming instruction for graphics processors. SIMT instructions are widely used for large-scale parallel computing tasks such as graphics rendering and artificial intelligence (AI) operations. SIMT instructions perform parallel operations using warps as the basic unit. A warp contains multiple threads. For example, a SIMT32 instruction represents an instruction that includes 32 threads. A single SIMT32 instruction enables threads in a warp of 32 threads to execute the same operation in parallel. This allows GPUs or AI chips to process multiple data points or computing tasks simultaneously, thereby improving computing efficiency.

[0062] Figure 1 1 is a circuit module diagram of a first vector core module 100 according to one embodiment. Based on the actual operation scenario, the first vector core module 100 can act as a host, and the first tensor core module 12 can act as a slave. The first tensor core module 12 is used to process some special operators of AI operations, including but not limited to tensor data handling, matrix multiplication and addition, and convolution operations. In addition to the control information related to the operation, the first tensor core module 12 usually also uses the method of configuring registers to obtain more tensor data labels, including but not limited to the address coordinate information, size information and boundary zero padding information of the tensor. The contents of these registers are not statically configured, but dynamically configured by instructions, but only require fixed-point operations or simple data movement to generate, and are usually scalar operations.

[0063] The first vector core module 100 executes SIMT instructions to perform corresponding vector operations. Vector operations include, but are not limited to, floating-point operations, fixed-point operations, and logical operations. The pipeline architecture of the first vector core module 100 supports large-scale parallel computing tasks. The pipeline architecture includes multiple processes, such as instruction fetch, instruction scheduling, decoding, operand fetching, operation execution, and result writeback. The first vector core module 100 writes the operation results back to the first memory module 11. Operations on the first memory module 11 include, but are not limited to, load, store, and atomic operations.

[0064] In an application scenario where the first vector core module 100 serves as the master module for the first memory module 11 and the first tensor core module 12, the first vector core module 100 generates operation control information for the slave modules based on SIMT instructions, as well as data or operands required by the first memory module 11 and tensor core configuration information required by the first tensor core module 12. The first vector core module 100 utilizes the SIMT pipeline architecture and arithmetic logic units (ALUs) to send tag information (including but not limited to tensor address coordinate information, tensor size information, boundary zero padding information, and other information) of the phase tensor data to the first tensor core module 12, which serves as a slave.

[0065] In detail, Figure 1 The first vector core module 100 shown includes a first instruction cache 110, a first instruction fetch unit 120, a first instruction issue unit 130, a first instruction decode unit 140, a first vector register file 150, a first scalar register file 160, a first vector core operation unit 170, and a first active thread detection unit 180. The first instruction fetch unit 120 reads a sufficient number of instructions from the first instruction cache 110 and passes them one by one to the first instruction issue unit 130 for issuance and execution. The first vector core module 100 does not distinguish between instruction types. Whether instructions that only process vector data, scalar data, or memory data (collectively referred to herein as "vector core instructions"), or instructions that only configure tensor tag registers or control the first tensor core module 12 (collectively referred to herein as "tensor core instructions"), they are all passed from the first instruction fetch unit 120 to the first instruction issue unit 130 and issued to the first instruction decode unit 140 in program order.

[0066] After any instruction is issued from the first instruction issue unit 130, it is decoded by the first instruction decode unit 140 to obtain the vector core operand type and address information. The first instruction decode unit 140 reads data from the first vector register file 150 and passes it to the first vector core operation unit 170. The operand may come from the first vector register file 150 (independent data for each thread) or from the first scalar register file 160 (shared data for each thread). In addition, the first instruction decode unit 140 also obtains vector core operation control information and passes it to the first vector core operation unit 170 along with the operand.

[0067] After receiving the operand and operation type, the first vector core operation unit 170 performs the calculation. The first vector core operation unit 170 includes multiple thread operation units to support SIMT instructions that operate multiple threads. For example, each of N independent thread operation units executes a corresponding thread. Each thread operation unit includes logical computing resources, floating-point computing resources, fixed-point computing resources, etc. All thread operation units can execute corresponding operations simultaneously and generate execution results after a fixed delay. After the first vector core operation unit 170 completes executing the vector core instruction, the first vector core operation unit 170 can directly write the execution results of each thread operation unit back to the first vector register group 150. Based on the vector core instruction of "the first vector core module 100 is the master and the first memory module 11 is the slave", the first vector core module 100 directly sends the execution results of each thread to the first memory module 11, and at the same time sends control information for the memory operation.

[0068] After the first vector core operation unit 170 completes processing a tensor core instruction, the first active thread detection unit 180 selects the lowest-numbered active thread. For example, it selects one processing result from the multiple thread operation units of the first vector core operation unit 170 and writes it back to the first scalar register bank 160. Based on the tensor core instruction with the first vector core module 100 as the master and the first tensor core module 12 as the slave, the first active thread detection unit 180 selects the lowest-numbered active thread, selects one processing result from the multiple thread operation units of the first vector core operation unit 170, and sends it to the first tensor core module 12, along with control information for the tensor core operation. This active thread is typically numbered 1. When the first vector core operation unit 170 processes a tensor core instruction, it uses only one thread operation unit to process the tensor core instruction, while the remaining thread operation units of the first vector core operation unit 170 remain idle.

[0069] Figure 1The first vector core module 100 shown sends the tensor core instruction configuration information, which has the following problems. First, due to the reusability requirements of SIMT computing resources, affected by floating-point instructions, even the delay of fixed-point instructions will be relatively long, resulting in the entire pipeline device sending tensor core configuration information relatively slowly. Furthermore, the tensor core configuration information only requires fixed-point operations or data movement, and only works in a single thread operation unit of the first vector core operation unit 170. Therefore, the resource utilization of the pipeline running tensor core instructions of the first vector core module 100 is very low. In addition, the vector core instructions and the tensor core instructions share the first instruction emission unit 130 and the subsequent pipeline, thereby hindering the efficiency of the vector core instructions with higher performance requirements, resulting in a gap between the actual computing power and the expected computing power.

[0070] Figure 2 FIG2 is a circuit block diagram of a second vector core module 200 according to an embodiment of the present invention. Based on actual operation scenarios, the second vector core module 200 can act as a master, and the second tensor core module 22 can act as a slave. Figure 2 The second vector core module 200, the second memory module 21 and the second tensor core module 22 can refer to Figure 1 The related descriptions of the first vector core module 100, the first memory module 11 and the first tensor core module 12 are shown and can be deduced by analogy. Figure 2 In the illustrated embodiment, the second vector core module 200 includes a second instruction cache 210, a second instruction fetch unit 220, an instruction dispatch unit 230, a vector core instruction execution pipeline 240, a tensor core instruction processing pipeline 250, a second vector register file 260, a second scalar register file 270, and a second active thread detection unit 280. The vector core instruction execution pipeline 240 executes vector core instructions and stores the execution results in the second memory module 21. The second vector register file 260 and the second scalar register file 270 are coupled to the vector core instruction execution pipeline 240. The second scalar register file 270 is also coupled to the tensor core instruction processing pipeline 250.

[0071] After executing a vector core instruction, the vector core instruction execution pipeline 240 writes the vector data corresponding to the vector core instruction back to the second vector register group 260. The second vector register group 260 provides the vector data to the vector core instruction execution pipeline 240. A second active thread detection unit 280 is coupled to the vector core instruction execution pipeline 240. The second active thread detection unit 280 detects the execution result of the vector core instruction execution pipeline 240 to determine the active thread. The second active thread detection unit 280 writes the scalar data corresponding to the active thread back to the second scalar register group 270. The vector core instruction execution pipeline 240 and the tensor core instruction processing pipeline 250 can access the scalar data in the second scalar register group 270. Figure 2 The second vector register group 260, the second scalar register group 270 and the second active thread detection unit 280 can refer to Figure 1 The related descriptions of the first vector register group 150 , the first scalar register group 160 and the first active thread detection unit 180 are similar and will not be repeated here.

[0072] exist Figure 2 In the illustrated embodiment, the vector core instruction execution pipeline 240 includes a second instruction issuing unit 241, a second instruction decoding unit 242, and a second vector core operation unit 243. The second instruction issuing unit 241 is coupled to the instruction scheduling unit 230 to receive vector core instructions. The second instruction issuing unit 241 issues vector core instructions. The second instruction decoding unit 242 is coupled to the second instruction issuing unit 241 to receive vector core instructions. The second instruction decoding unit 242 decodes the vector core instructions to generate operands and operation types. The second vector core operation unit 243 is coupled to the second instruction decoding unit 242 to receive the operands and operation types. The second vector core operation unit 243 operates on the operands based on the operation types to generate an execution result for the second memory module 21. The second vector core operation unit 243 includes multiple thread operation units, each of which performs operations for a corresponding thread. The multiple thread operation units generate execution results for the second memory module 21. Figure 2 The second instruction issuing unit 241, the second instruction decoding unit 242 and the second vector core operation unit 243 can refer to Figure 1 The related descriptions of the first instruction issuing unit 130 , the first instruction decoding unit 140 and the first vector core operation unit 170 are similar and will not be repeated here.

[0073] The second instruction fetch unit 220 is coupled to the second instruction cache 210 to fetch at least one warp. Figure 2 The second instruction cache 210 and the second instruction fetch unit 220 can refer to Figure 1 The related description of the first instruction cache 110 and the first instruction fetch unit 120 is similar to that of the first instruction cache 110 and the first instruction fetch unit 120, so it will not be repeated here. Figure 1 The first vector core module 100 shown, Figure 2The second vector core module 200 shown adds an instruction scheduling unit 230 and a tensor core instruction processing pipeline 250. The second instruction acquisition unit 220 is coupled to the instruction scheduling unit 230 to provide thread bundles. The newly added instruction scheduling unit 230 distinguishes and distributes tensor core instructions and vector core instructions to corresponding pipelines. In an embodiment that supports multiple thread bundles, the instruction scheduling unit 230 can simultaneously distribute one tensor core instruction and one vector core instruction from different thread bundles to the corresponding pipelines at the same time. The added tensor core instruction processing pipeline 250 will only run tensor core instructions. The tensor core instruction processing pipeline 250 only processes scalar data, and there are no redundant operations for vector and scalar conversion. The processing of tensor core instructions by the tensor core instruction processing pipeline 250 only includes one or more of fixed-point addition, fixed-point multiplication, fixed-point multiplication and addition, and data transfer. Therefore, the delay of the tensor core instruction processing pipeline 250 is much smaller than the delay of the vector core instruction execution pipeline 240. Compared with Figure 1 The first vector core module 100 shown, Figure 2 The second vector core module 200 is shown to send the tensor core configuration information to the second tensor core module 22 relatively quickly.

[0074] Figure 3 This is a flowchart of an operation method of a vector core module according to an embodiment of the present invention. Figure 2 and Figure 3 In step S310, the instruction scheduling unit 230 classifies instructions for at least one warp to distinguish between vector core instructions and tensor core instructions from the at least one warp. The instruction scheduling unit 230 is coupled to the vector core instruction execution pipeline 240 to provide vector core instructions. The instruction scheduling unit 230 is coupled to the tensor core instruction processing pipeline 250 to provide tensor core instructions.

[0075] In response to the warp including a vector core instruction, the instruction scheduling unit 230 sends the vector core instruction to the vector core instruction execution pipeline 240 (step S320). The vector core instruction execution pipeline 240 executes the vector core instruction and stores the execution result in the second memory module 21 (step S330). In response to the warp including a tensor core instruction, the instruction scheduling unit 230 sends the tensor core instruction to the tensor core instruction processing pipeline 250 (step S340). The tensor core instruction processing pipeline 250 processes the tensor core instruction and sends the processing result (e.g., tensor core configuration information) to the second tensor core module 22 (step S350).

[0076] exist Figure 2In the illustrated embodiment, the tensor core instruction processing pipeline 250 includes a third instruction issuing unit 251, a third instruction decoding unit 252, and a tensor core instruction operation unit 253. The third instruction issuing unit 251 is coupled to the instruction scheduling unit 230 to receive tensor core instructions. The third instruction issuing unit 251 sequentially issues tensor core instructions in program order. The third instruction decoding unit 252 is coupled to the third instruction issuing unit 251 and the second scalar register file 270. The third instruction decoding unit 252 decodes the tensor core instruction to generate operands and operation types. For example, based on the decoding result, the third instruction decoding unit 252 reads the operands from the second scalar register file 270 and passes them to the tensor core instruction operation unit 253. The tensor core instruction operation unit 253 is coupled to the third instruction decoding unit 252 to receive the operands and operation types. The tensor core instruction operation unit 253 processes the operands based on the operation type, generates a processing result (such as tensor core configuration information) to the second tensor core module 22, and simultaneously sends control information of the tensor core operation.

[0077] In summary, the second vector core module 200 adds a tensor core instruction processing pipeline 250 (an independent pipeline for generating and issuing tensor core configuration information). The tensor core instruction processing pipeline 250 only executes tensor core instructions, while vector core instructions are handed off to the vector core instruction execution pipeline 240 for execution. In some embodiments, the third instruction decode unit 252 and the tensor core instruction operation unit 253 only process scalar data and do not perform vector-to-scalar conversion operations. Therefore, the latency of the tensor core instruction processing pipeline 250 is much lower than that of the vector core instruction execution pipeline 240, and instructions are sent to the second tensor core module 22 relatively quickly. In some embodiments, the tensor core instruction operation unit 253 only supports fixed-point addition, fixed-point multiplication, fixed-point multiply-add, and simple data transfer. Therefore, the latency of the tensor core instruction processing pipeline 250 is much lower than that of the vector core instruction execution pipeline 240. Compared to the vector core instruction execution pipeline 240, the tensor core instruction processing pipeline 250 sends tensor core configuration information to the second tensor core module 22 relatively quickly. In the application scenario where the second vector core module 200 is the master and the second tensor core module 22 is the slave, the second vector core module 200 can improve efficiency.

[0078] Figure 4 FIG. 2 is a schematic diagram of a circuit module of the instruction scheduling unit 230 according to an embodiment of the present invention. Figure 4 The instruction dispatch unit 230 can be used as Figure 2 One of many implementation examples of the instruction dispatch unit 230 is shown. Figure 4 The second instruction acquisition unit 220, the instruction scheduling unit 230, the vector core instruction execution pipeline 240 and the tensor core instruction processing pipeline 250 can refer to Figure 2 The relevant instructions are not repeated here. Figure 4 In the illustrated embodiment, the thread warp provided by the second instruction fetch unit 220 is a single thread warp, and the instruction scheduling unit 230 includes an instruction classifier 231. An input of the instruction classifier 231 is coupled to the second instruction fetch unit 220 to receive the single thread warp. The instruction classifier 231 is coupled to the vector core instruction execution pipeline 240 and the tensor core instruction processing pipeline 250. The instruction classifier 231 classifies instructions for the single thread warp. In response to the single thread warp including a vector core instruction, the instruction classifier 231 sends the vector core instruction to the vector core instruction execution pipeline 240. In response to the single thread warp including a tensor core instruction, the instruction classifier 231 sends the tensor core instruction to the tensor core instruction processing pipeline 250.

[0079] Figure 5 FIG. 2 is a schematic diagram of a circuit module of the instruction scheduling unit 230 according to another embodiment of the present invention. Figure 5 The instruction dispatch unit 230 can be used as Figure 2 One of many implementation examples of the instruction dispatch unit 230 is shown. Figure 5 The second instruction acquisition unit 220, the instruction scheduling unit 230, the vector core instruction execution pipeline 240 and the tensor core instruction processing pipeline 250 can refer to Figure 2 The relevant instructions are not repeated here. Figure 5 In the illustrated embodiment, the thread warps provided by the second instruction fetch unit 220 are multi-thread warps, and the instruction scheduling unit 230 includes a plurality of thread warp instruction classifiers (eg Figure 5 The first warp instruction classifier 232_1, the second warp instruction classifier 232_2, ..., the nth warp instruction classifier 232_n), the vector core instruction polling arbiter 233, and the tensor core instruction polling arbiter 234 are shown. Each of the first warp instruction classifier 232_1 through the nth warp instruction classifier 232_n has an input terminal coupled to the second instruction fetch unit 220 to receive a corresponding warp from the multiple warps. Each of the first warp instruction classifier 232_1 through the nth warp instruction classifier 232_n performs instruction classification on the corresponding warp to distinguish between vector core instructions and tensor core instructions from the corresponding warp.

[0080] The vector core instruction polling arbiter 233 is coupled to the first through nth warp instruction classifiers 232_1, 232_n. The vector core instruction polling arbiter 233 polls the first through nth warp instruction classifiers 232_1, 232_n to obtain vector core instructions. The output of the vector core instruction polling arbiter 233 is coupled to the vector core instruction execution pipeline 240 to provide vector core instructions. The tensor core instruction polling arbiter 234 is coupled to the first through nth warp instruction classifiers 232_1, 232_n. The tensor core instruction polling arbiter 234 polls the first through nth warp instruction classifiers 232_1, 232_n to obtain tensor core instructions. The output of the tensor core instruction polling arbiter 234 is coupled to the tensor core instruction processing pipeline 250 to provide tensor core instructions.

[0081] Figure 6 FIG. 2 is a schematic diagram of a circuit module of a tensor core instruction operation unit 253 according to an embodiment of the present invention. Figure 6 The tensor core instruction operation unit 253 shown can be used as Figure 2 One of many implementation examples of the tensor core instruction operation unit 253 is shown. Figure 6 The third instruction decoding unit 252, the tensor core instruction operation unit 253 and the second tensor core module 22 can refer to Figure 2 The relevant instructions are not repeated here. Figure 6 In the illustrated embodiment, the tensor core instruction operation unit 253 includes a fixed-point multiplier 610, a first gate 620, a second gate 630, a fixed-point adder 640, and a third gate 650. A first input of the third gate 650 is coupled to the third instruction decode unit 252 to receive a first operand a6. An output of the third gate 650 is coupled to the second tensor core module 22 to provide a processing result of the tensor core instruction operation unit 253. A first input of the fixed-point multiplier 610 is coupled to the third instruction decode unit 252 to receive a first operand a6. A second input of the fixed-point multiplier 610 is coupled to the third instruction decode unit 252 to receive a second operand b6. An output of the fixed-point multiplier 610 is coupled to a second input of the third gate 650.

[0082] A first input of the first gate 620 is coupled to the third instruction decode unit 252 to receive the first operand a6. A second input of the first gate 620 is coupled to the output of the fixed-point multiplier 610. Based on control of the third instruction decode unit 252, the first gate 620 selects one of the first operand a6 and the output of the fixed-point multiplier 610 to be transmitted to the fixed-point adder 640. A first input of the second gate 630 is coupled to the third instruction decode unit 252 to receive the second operand b6. A second input of the second gate 630 is coupled to the third instruction decode unit 252 to receive the third operand c6. Based on control of the third instruction decode unit 252, the second gate 630 selects one of the second operand b6 and the third operand c6 to be transmitted to the fixed-point adder 640. A first input of the fixed-point adder 640 is coupled to the output of the first gate 620. A second input terminal of the fixed-point adder 640 is coupled to the output terminal of the second gate 630 . An output terminal of the fixed-point adder 640 is coupled to a third input terminal of the third gate 650 .

[0083] Based on the control of the third instruction decode unit 252, the third gate 650 selects one of the first operand a6, the output of the fixed-point multiplier 610, and the output of the fixed-point adder 640 to be transmitted to the second tensor core module 22. For data movement requirements, the third gate 650 directly transmits the first operand a6 as the processing result (e.g., tensor core configuration information) to the second tensor core module 22. For multiplication requirements, the third gate 650 selects the output "a6·b6" of the fixed-point multiplier 610 as the processing result (e.g., tensor core configuration information) to be transmitted to the second tensor core module 22. For addition requests, the first gate 620 selects the first operand a6 to be transmitted to the fixed-point adder 640, the second gate 630 selects the second operand b6 to be transmitted to the fixed-point adder 640, and the third gate 650 selects the output "a6+b6" of the fixed-point adder 640 to be transmitted as the processing result (e.g., tensor core configuration information) to the second tensor core module 22. For multiplication and addition requests, the first gate 620 selects the output of the fixed-point multiplier 610 to be transmitted to the fixed-point adder 640, the second gate 630 selects the third operand c6 to be transmitted to the fixed-point adder 640, and the third gate 650 selects the output "a6·b6+c6" of the fixed-point adder 640 to be transmitted as the processing result (e.g., tensor core configuration information) to the second tensor core module 22.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vector kernel module, characterized in that: The vector core module includes: A vector core instruction execution pipeline executes vector core instructions and stores the execution results in a memory module; A tensor core instruction processing pipeline processes tensor core instructions and sends processing results to the tensor core module, wherein the processing results are tensor core configuration information, the vector core module is a master, and the tensor core module is a slave; and an instruction scheduling unit coupled to the vector core instruction execution pipeline and the tensor core instruction processing pipeline, wherein the instruction scheduling unit classifies instructions for at least one thread warp to distinguish the vector core instructions from the tensor core instructions from the at least one thread warp; In response to the at least one warp including the vector core instruction, the instruction scheduling unit sends the vector core instruction to the vector core instruction execution pipeline; and In response to the at least one warp including the tensor core instruction, the instruction scheduling unit sends the tensor core instruction to the tensor core instruction processing pipeline.

2. The vector core module according to claim 1, characterized in that: The vector core instruction execution pipeline includes: a first instruction issuing unit, coupled to the instruction scheduling unit to receive the vector core instruction, wherein the first instruction issuing unit issues the vector core instruction; a first instruction decoding unit coupled to the first instruction issuing unit to receive the vector core instruction, wherein the first instruction decoding unit decodes the vector core instruction to generate an operand and an operation type; and The vector core operation unit is coupled to the first instruction decoding unit to receive the operand and the operation type, wherein the vector core operation unit operates the operand based on the operation type to generate the execution result to the memory module.

3. The vector core module according to claim 2, characterized in that: The vector core operation unit includes: A plurality of thread operation units, wherein each of the plurality of thread operation units executes an operation of a corresponding thread, and the plurality of thread operation units generate the execution results to the memory module.

4. The vector core module according to claim 1, characterized in that: The Tensor Core instruction processing pipeline processes only scalar data.

5. The vector core module according to claim 1, characterized in that: The processing of the tensor core instruction by the tensor core instruction processing pipeline includes only one or more of data movement, fixed-point addition, fixed-point multiplication, and fixed-point multiplication-addition.

6. The vector core module according to claim 1, characterized in that: The tensor core instruction processing pipeline includes: a second instruction issuing unit, coupled to the instruction scheduling unit to receive the tensor core instruction, wherein the second instruction issuing unit issues the tensor core instruction; a second instruction decoding unit coupled to the second instruction issuing unit, wherein the second instruction decoding unit decodes the tensor core instruction to generate an operand and an operation type; and A tensor core instruction operation unit is coupled to the second instruction decoding unit to receive the operand and the operation type, wherein the tensor core instruction operation unit processes the operand based on the operation type to generate the processing result to the tensor core module.

7. The vector core module according to claim 6, characterized in that: The tensor core instruction operation unit includes: a first gate, wherein a first input terminal of the first gate is coupled to the second instruction decoding unit to receive a first operand of the operands, and an output terminal of the first gate is coupled to the tensor core module to provide the processing result; a fixed-point multiplier, wherein a first input terminal of the fixed-point multiplier is coupled to the second instruction decoding unit to receive the first operand, a second input terminal of the fixed-point multiplier is coupled to the second instruction decoding unit to receive a second operand among the operands, and an output terminal of the fixed-point multiplier is coupled to the second input terminal of the first gate; a second gate, wherein a first input terminal of the second gate is coupled to the second instruction decoding unit to receive the first operand, and a second input terminal of the second gate is coupled to an output terminal of the fixed-point multiplier; a third gate, wherein a first input terminal of the third gate is coupled to the second instruction decoding unit to receive the second operand, and a second input terminal of the third gate is coupled to the second instruction decoding unit to receive a third operand among the operands; and A fixed-point adder, wherein a first input of the fixed-point adder is coupled to the output of the second gate, a second input of the fixed-point adder is coupled to the output of the third gate, and an output of the fixed-point adder is coupled to the third input of the first gate.

8. The vector core module according to claim 1, characterized in that: The vector core module also includes: instruction cache; and An instruction fetch unit is coupled to the instruction cache to fetch the at least one warp, wherein the instruction fetch unit is coupled to the instruction scheduling unit to provide the at least one warp.

9. The vector core module according to claim 8, characterized in that: The at least one thread warp is a single thread warp, and the instruction scheduling unit includes: An instruction classifier, wherein an input end of the instruction classifier is coupled to the instruction fetch unit to receive the single warp, the instruction classifier is coupled to the vector core instruction execution pipeline and the tensor core instruction processing pipeline, and the instruction classifier performs instruction classification on the single warp. In response to the single warp including the vector core instruction, the instruction classifier sends the vector core instruction to the vector core instruction execution pipeline; and In response to the single warp including the Tensor Core instruction, the instruction classifier sends the Tensor Core instruction to the Tensor Core instruction processing pipeline.

10. The vector core module according to claim 8, characterized in that: The at least one thread warp is a multi-thread warp, and the instruction scheduling unit includes: a plurality of warp instruction classifiers, wherein an input terminal of each of the plurality of warp instruction classifiers is coupled to the instruction fetch unit to receive a corresponding warp from the multiple warps, and each of the plurality of warp instruction classifiers performs instruction classification on the corresponding warp to distinguish the vector core instructions from the tensor core instructions from the corresponding warp. a vector core instruction polling arbiter coupled to the plurality of warp instruction classifiers, wherein the vector core instruction polling arbiter polls the plurality of warp instruction classifiers to obtain the vector core instruction, and an output end of the vector core instruction polling arbiter is coupled to the vector core instruction execution pipeline to provide the vector core instruction; and A tensor core instruction polling arbiter is coupled to the plurality of thread warp instruction classifiers, wherein the tensor core instruction polling arbiter polls the plurality of thread warp instruction classifiers to obtain the tensor core instructions, and an output end of the tensor core instruction polling arbiter is coupled to the tensor core instruction processing pipeline to provide the tensor core instructions.

11. The vector core module according to claim 1, characterized in that: The vector core module also includes: a vector register group coupled to the vector core instruction execution pipeline to provide vector data, wherein the vector core instruction execution pipeline writes the vector data corresponding to the vector core instruction back to the vector register group after the vector core instruction is executed; a scalar register set coupled to the vector core instruction execution pipeline and the tensor core instruction processing pipeline to provide scalar data; and An active thread detection unit is coupled to the vector core instruction execution pipeline, wherein the active thread detection unit detects the execution result to determine an active thread, and the active thread detection unit writes the scalar data corresponding to the active thread back to the scalar register group.

12. A method for operating a vector core module, characterized in that: The operation method includes: performing instruction classification on at least one warp by an instruction scheduling unit of the vector core module to distinguish vector core instructions from tensor core instructions from the at least one warp, wherein the instruction scheduling unit is coupled to the vector core instruction execution pipeline and the tensor core instruction processing pipeline; In response to the at least one thread warp including the vector core instruction, the instruction scheduling unit sends the vector core instruction to the vector core instruction execution pipeline of the vector core module; The vector core instruction execution pipeline executes the vector core instruction and stores the execution result in the memory module; In response to the at least one thread warp including the tensor core instruction, the instruction scheduling unit sends the tensor core instruction to the tensor core instruction processing pipeline of the vector core module; and The tensor core instruction processing pipeline processes the tensor core instruction and sends the processing result to the tensor core module, wherein the processing result is tensor core configuration information, the vector core module is the host, and the tensor core module is the slave.

13. The operating method according to claim 12, characterized in that: The operation method further includes: issuing the vector core instruction by a first instruction issuing unit of the vector core instruction execution pipeline, wherein the first instruction issuing unit is coupled to the instruction scheduling unit to receive the vector core instruction; decoding the vector core instruction by a first instruction decode unit of the vector core instruction execution pipeline to generate operands and an operation type, wherein the first instruction decode unit is coupled to the first instruction issue unit to receive the vector core instruction; and A vector core operation unit of the vector core instruction execution pipeline operates the operand based on the operation type to generate the execution result to the memory module, wherein the vector core operation unit is coupled to the first instruction decoding unit to receive the operand and the operation type.

14. The operating method according to claim 13, characterized in that: The operation method further includes: Each of the multiple thread operation units of the vector core operation unit executes an operation of a corresponding thread, wherein the multiple thread operation units generate the execution results to the memory module.

15. The operating method according to claim 12, characterized in that: The Tensor Core instruction processing pipeline processes only scalar data.

16. The operating method according to claim 12, characterized in that: The processing of the tensor core instruction by the tensor core instruction processing pipeline includes only one or more of data movement, fixed-point addition, fixed-point multiplication, and fixed-point multiplication-addition.

17. The operating method according to claim 12, characterized in that: The operation method further includes: issuing the tensor core instruction by a second instruction issue unit of the tensor core instruction processing pipeline, wherein the second instruction issue unit is coupled to the instruction dispatch unit to receive the tensor core instruction; decoding the tensor core instruction by a second instruction decode unit of the tensor core instruction processing pipeline to generate operands and an operation type, wherein the second instruction decode unit is coupled to the second instruction issue unit; and A tensor core instruction operation unit of the tensor core instruction processing pipeline processes the operand based on the operation type to generate the processing result to the tensor core module, wherein the tensor core instruction operation unit is coupled to the second instruction decoding unit to receive the operand and the operation type.

18. The operating method according to claim 12, characterized in that: The at least one warp is a single warp, and the operating method further includes: fetching, by an instruction fetch unit of the vector core module, the at least one thread warp from an instruction cache of the vector core module, wherein the instruction fetch unit is coupled to the instruction scheduling unit to provide the at least one thread warp; An instruction classifier of the instruction scheduling unit performs instruction classification on the single warp, wherein an input end of the instruction classifier is coupled to the instruction fetch unit to receive the single warp, and the instruction classifier is coupled to the vector core instruction execution pipeline and the tensor core instruction processing pipeline; In response to the single warp including the vector core instruction, sending the vector core instruction to the vector core instruction execution pipeline by the instruction classifier; and In response to the single warp including the tensor core instruction, the instruction classifier sends the tensor core instruction to the tensor core instruction processing pipeline.

19. The operating method according to claim 12, characterized in that: The at least one thread warp is a multi-thread warp, and the operation method further includes: fetching, by an instruction fetch unit of the vector core module, the at least one thread warp from an instruction cache of the vector core module, wherein the instruction fetch unit is coupled to the instruction scheduling unit to provide the at least one thread warp; receiving, by each of a plurality of warp instruction classifiers of the instruction scheduling unit, a corresponding warp among the plurality of warps; performing instruction classification on the corresponding warp by each of the plurality of warp instruction classifiers to distinguish the vector core instructions and the tensor core instructions from the corresponding warp; polling the plurality of warp instruction classifiers by a vector core instruction polling arbiter of the instruction scheduling unit to obtain the vector core instruction, wherein an output terminal of the vector core instruction polling arbiter is coupled to the vector core instruction execution pipeline to provide the vector core instruction; and A tensor core instruction polling arbiter of the instruction scheduling unit polls the plurality of warp instruction classifiers to obtain the tensor core instructions, wherein an output terminal of the tensor core instruction polling arbiter is coupled to the tensor core instruction processing pipeline to provide the tensor core instructions.

20. The operating method according to claim 12, characterized in that: The operation method further includes: The vector core instruction execution pipeline writes the vector data corresponding to the vector core instruction back to the vector register group of the vector core module after the execution of the vector core instruction is completed; Providing the vector data to the vector core instruction execution pipeline by the vector register group; The valid thread detection unit of the vector core module detects the execution result to determine the valid thread; The active thread detection unit writes the scalar data corresponding to the active thread back to the scalar register group of the vector core module; and The scalar register group provides the scalar data to the vector core instruction execution pipeline and the tensor core instruction processing pipeline.

Images