Execution unit and chip

Abstract

This application relates to an execution unit and a chip. The execution unit includes a shared memory control unit and shared memory; the shared memory control unit includes a shuffling conversion unit, a write unit, and a read unit; after receiving a shuffling instruction, the shuffling conversion unit allocates a target cache line for it in the shuffling cache space of the shared memory. The shuffling instruction includes the address of the source data register, the address of the source channel index register, the address of the target register, and a mask; the source channel index register stores thread index data; the shuffling conversion unit converts the shuffling instruction into write and read instructions, calls the write unit to execute the write instruction to write the source data in the source data register to the target cache line, calls the read unit to execute the read instruction to read the source data from the target cache line, uses the multiplexer in the read unit to select the data to be written to the corresponding thread from the source data using the thread index data, and stores it in the target register according to the mask. This application can support shuffling instructions in hardware.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to an execution unit and chip. Background Technology

[0002] In related technologies, graphics processing units (GPUs) contain a large number of parallel processing units that can be programmed to perform single-instruction multiple-data (SIMD) parallel computations. This allows GPUs to be used not only for hardware acceleration of graphics rendering but also for general-purpose parallel computing. With the rapid development of deep neural networks (DNNs), GPUs can be used to parallel accelerate the operators of deep neural networks (DNNs), achieving high acceleration performance.

[0003] In related technologies, DNN operators can be implemented using general-purpose computing platforms, such as the Open Computing Language (OpenCL), Direct3D Compute Shaders (CS), Compute Unified Device Architecture (CUDA), and Radeon Open Compute (ROCm). These platforms divide the computation space into a series of work groups. Each work group is scheduled to an execution unit (EU) on the chip, and then further subdivided into multiple wave bundles. Each wave bundle typically contains waveSize threads (lanes), executing in a single-instruction-multiple-data manner. Each thread has a lane index, with values ​​ranging from 0 to waveSize-1. waveSize is often set to 32 or 64, referred to as simd32 or simd64 wave bundles.

[0004] Most operator programs allow each thread to perform calculations using its own data. However, certain special operators, such as bitonic sort, merge addition, and prefix sort, require different threads within a thread bundle to access each other's data. Using multiple instructions to exchange data between threads can impact performance. Therefore, companies like NVIDIA have proposed the Thread Bundle Shuffle instruction (SHFL) to quickly solve the data exchange problem between different threads within the same thread bundle at the hardware level, thereby improving operator optimization performance.

[0005] SHFL is an instruction for an Arithmetic Logic Unit (ALU). However, if SHFL is implemented directly in the ALU, a large number of multiplexers (MUX) are required as selection components, resulting in significant hardware overhead.

[0006] Since the Direct3D rendering engine generally does not require data exchange between different threads within a thread bundle, it does not need to support SHFL. However, with the increasing prevalence of AI inference, there is a need to accelerate large AI models in GPU products, naturally leading to operator computations such as merge addition, which in turn creates a need for data shuffling within the thread bundle.

[0007] Depending on the shuffling requirements, there are several methods in related technologies to simulate the SHFL instruction: (1) If it is a broadcast shuffling operation, such as copying an element from a certain thread to all threads, the scalar register method can be used. First, move the element to a certain scalar register, and then move it to the data of other threads. Generally, it requires three or four instructions, which has low execution efficiency and will slightly affect performance.

[0008] (2) If it is a case of up / down translation of data, it will be much more complicated. If it is a multiple of 4 translations, multiple selection instructions (DEALI) can be used to complete it. However, if it is an odd number of translations, it will be much more complicated and difficult to simulate with a few instructions. It may even require line-by-line judgment, which may require a dozen selection instructions and mask setting instructions, resulting in low execution efficiency.

[0009] (3) If it is a shuffling situation of XOR operation, multiple selection instructions (DEALI) and XOR instructions (XOR) are required together, which also requires about 2 to 8 instructions, resulting in low execution efficiency.

[0010] (4) If it is a shuffle instruction that supports indexing, since the index position is dynamically calculated, it is difficult to implement using a limited number of instructions.

[0011] As can be seen from the above, if the hardware does not support the SHFL instruction, simulating SHFL requires several instructions, which will affect the execution efficiency of related operators; in some cases, it is difficult to achieve, which will have an adverse effect on AI inference computation. Summary of the Invention

[0012] The technical problem solved by this application is to provide an execution unit and chip that can support shuffle instructions in hardware, realize data exchange between different threads within a thread bundle, and improve execution efficiency.

[0013] According to a first aspect of the embodiments of this application, an execution unit is provided, comprising: a shared memory control unit and shared memory; the shared memory control unit and the shared memory are communicatively connected; the shared memory includes a shuffle cache space. The shared memory control unit includes a shuffling and conversion unit, a write unit, and a read unit; the read unit includes a multiplexer. The shuffling conversion unit is used to allocate a target cache line for the shuffling instruction in an empty cache line in the shuffling cache space after receiving the shuffling instruction. The shuffling instruction is used to realize data exchange between different threads within the target thread bundle. The shuffling instruction includes a first register address of the source data register, a second register address of the source channel index register, a third register address of the target register, and a mask. The source data register is used to store source data, the source channel index register is used to store thread index data, the target register is used to store the shuffled data, and the value of each bit of the mask is used to indicate whether the corresponding thread writes the shuffled data. The shuffle conversion unit is further configured to convert the shuffle instruction into a write instruction and a read instruction, wherein the starting address of the target cache line is the base address of the write instruction; the write instruction further includes the first register address; the read instruction includes the mask, the second register address, the third register address, and the starting address; The shuffling conversion unit is also used to call the writing unit to execute the write instruction and write the source data stored in the source data register to the target cache line; The shuffling conversion unit is also used to call the read unit to execute the read instruction, read the source data from the target cache line, and then use the multiplexer to select the data to be written to the corresponding thread from the source data using the thread index data, and store it in the target register according to the mask.

[0014] Compared with the prior art, the beneficial effects of this application are as follows: the execution unit includes a shared memory control unit and shared memory, the shared memory control unit and shared memory are communicatively connected, the shared memory includes a shuffling cache space, the shared memory control unit includes a shuffling conversion unit, a write unit and a read unit, the read unit includes a multiplexer, after receiving a shuffling instruction, the shuffling conversion unit allocates a target cache line for the shuffling instruction in a free cache line in the shuffling cache space, the shuffling instruction includes the first register address of the source data register, the second register address of the source channel index register, the third register address of the target register and a mask, the source data register is used to store source data, the source channel index register is used to store thread index data, and the value of each bit of the mask is used to indicate whether the corresponding thread is... The shuffled data is written to the target register, which stores the shuffled data. The shuffle conversion unit also converts the shuffle instructions into write and read instructions. The starting address of the target cache line is the base address of the write instruction, which also includes the address of the first register. The read instruction includes a mask, the address of the second register, the address of the third register, and the starting address. The shuffle conversion unit calls the write unit to execute the write instruction, writing the source data stored in the source data register to the target cache line in shared memory. Then, the shuffle conversion unit also calls the read unit to execute the read instruction, reading the source data from the target cache line. Then, using the multiplexer in the read unit, the thread index data is used to select the data to be written to the corresponding thread from the source data, and the data is stored in the target register according to the mask. The data in the target register is the shuffled data. In summary, the technical solution of this application can support shuffle instructions in hardware, realize data exchange between different threads within a thread bundle, and thus improve execution efficiency.

[0015] In one embodiment, the shuffling conversion unit may include a control register and a shuffling buffer capacity register; The data stored in the control register is used to indicate whether the operator of the target thread bundle uses the shuffle instruction; The data stored in the shuffle cache capacity register is used to indicate the size of the shuffle cache space; the size of the shuffle cache space is measured in cache lines. The data stored in the control register and the data stored in the shuffle buffer capacity register are configured by the driver program according to the operator program. When the data stored in the control register indicates that the operator uses the shuffle instruction, the shared memory control unit enables data exchange between different threads within the target thread bundle based on the shuffle instruction. When the operator does not use the shuffle instruction, the shared memory control unit disables the data exchange between different threads within the target thread bundle based on the shuffle instruction and releases the shuffle cache space. In this way, the released shuffle cache space can be used as shared memory without needing to initialize the shared memory space of the workgroup, thus improving the execution efficiency of shared memory.

[0016] In one implementation, the size of the shuffle buffer space can be greater than or equal to 1. When the size of the shuffle buffer space is greater than 1, the shuffle conversion unit can support the execution of more than one shuffle instruction simultaneously.

[0017] In one implementation, the size of the shuffle buffer space can be greater than the number of clock cycles required for the total delay of executing the write instruction and the read instruction. This allows multiple shuffle instructions to be executed simultaneously, avoiding instruction bubbles and improving execution efficiency.

[0018] In one implementation, the shuffle cache space may be located at the top of the shared memory; When the shared memory includes M cache lines, the base address of the shuffle cache space is M - SHFLBufSize, where M is greater than SHFLBufSize, SHFLBufSize is the size of the shuffle cache space, and the address range of the shuffle cache space is (M - BufSize, M - 1).

[0019] In one implementation, after the shared memory control unit receives the shuffle instruction, the shuffle conversion unit can determine whether there is a free cache line in the shuffle cache space. If not, the shuffle instruction is suspended; if it is, a target cache line is allocated for the shuffle instruction from the free cache lines in the shuffle cache space.

[0020] In one implementation, if the line number of the target cache line is bufidx, then the line number address SHFLWrIdx of the target cache line in the shared memory is SHFLWrIdx = M - SHFLBufSize + bufidx.

[0021] In one embodiment, the execution unit may further include a thread scheduling unit, an instruction cache, and a thread bundle register; The thread scheduling unit is configured to select the target thread bundle from multiple thread bundles in each clock cycle and control the target thread bundle to retrieve the corresponding operator program from the instruction cache, the operator program including the shuffle instruction; The thread bundle register is used to temporarily store the source data and shuffled data of shuffling instructions for multiple thread bundles; the thread bundle includes multiple threads.

[0022] According to a second aspect of the embodiments of this application, a chip is provided, including the execution unit as described above.

[0023] Compared with the prior art, the beneficial effects of this application are as follows: the execution unit in the chip includes a shared memory control unit and shared memory, the shared memory control unit and shared memory are communicatively connected, the shared memory includes a shuffling cache space, the shared memory control unit includes a shuffling conversion unit, a write unit and a read unit, the read unit includes a multiplexer, and after receiving a shuffling instruction, the shuffling conversion unit allocates a target cache line for the shuffling instruction in a free cache line in the shuffling cache space. The shuffling instruction includes the first register address of the source data register, the second register address of the source channel index register, the third register address of the target register and a mask. The source data register is used to store source data, the source channel index register is used to store thread index data, and the value of each bit of the mask is used to indicate the corresponding thread. Whether to write shuffled data, the target register is used to store the shuffled data. The shuffle conversion unit also converts the shuffle instruction into write and read instructions. The starting address of the target cache line is the base address of the write instruction, which also includes the address of the first register. The read instruction includes a mask, the address of the second register, the address of the third register, and the starting address. The shuffle conversion unit calls the write unit to execute the write instruction, writing the source data stored in the source data register into the target cache line in shared memory. Then, the shuffle conversion unit also calls the read unit to execute the read instruction, reading the source data from the target cache line. Then, using the multiplexer in the read unit, the thread index data is used to select the data to be written to the corresponding thread from the source data, and the data is stored in the target register according to the mask. The data in the target register is the shuffled data. In summary, the technical solution of this application can support shuffle instructions in hardware, realize data exchange between different threads within a thread bundle, and thus improve execution efficiency.

[0024] In one implementation, the chip may be a graphics processor or an artificial intelligence chip. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating a shuffling operation according to an exemplary embodiment.

[0026] Figure 2 This is a schematic diagram illustrating a shuffling operation according to another exemplary embodiment.

[0027] Figure 3 This is a schematic diagram illustrating a shuffling operation according to another exemplary embodiment.

[0028] Figure 4 This is a schematic diagram of the structure of an execution unit according to an exemplary embodiment.

[0029] Figure 5 This is a schematic diagram illustrating a write operation according to an exemplary embodiment.

[0030] Figure 6 This is a schematic diagram illustrating a read operation according to an exemplary embodiment. Detailed Implementation

[0031] Unless otherwise defined, the technical or scientific terms used in this specification and claims shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. Specific embodiments of this application will be described below in conjunction with the accompanying drawings. It should be noted that, in order to provide a concise description, this specification cannot exhaustively describe all features of the actual embodiments. Without departing from the spirit and scope of this application, those skilled in the art can modify and substitute the embodiments of this application, and the resulting embodiments are also within the protection scope of this application.

[0032] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0033] Before introducing the execution unit in this application, let's first introduce the shuffling instruction.

[0034] The thread bundle shuffling operation selects the corresponding thread data from the source data based on the input thread index, and uses this data as the target thread's data. The shuffling instruction is shown below: (Pn)SHFLRd,Rs1,Rs0 Equation (1) In this instruction, SHFL is the index-mode shuffle instruction, Rs0 is the first register address of the source data register (used to store source data), Rs1 is the second register address of the source channel index register (used to store thread index data), Rd is the third register address of the destination register (used to store the shuffled data), and Pn is the mask, a string of binary numbers. Each bit of the mask indicates whether the corresponding thread should write the shuffled data. The number of bits in the mask is equal to the number of threads in the thread bundle. For example, a simd32 thread bundle includes 32 threads, and the mask for that thread bundle is 32 bits. For instance, a bit of 1 indicates that the corresponding thread in the destination register will be written with the new shuffled data, while a bit of 0 indicates that the corresponding thread in the destination register will not write new data.

[0035] This shuffle instruction requires the addresses of two source registers: Rs0 and Rs1. For example, the following formula (2) is a specific shuffle instruction.

[0036] (P1)SHFLR2, R4, R6 (2) Assume the thread bundle contains 32 threads. In equation (2), the first register address R6, the source data register, is used to store the source data; the second register address R4, the source channel index register, is used to store the thread index data; and the third register address R2, the destination register, is used to store the new data after shuffling, which can also be called the destination data. The shuffling instruction uses the thread index data to select the data to be written to the corresponding thread from the source data and stores it in the destination register according to the mask. By providing different thread index data, we can achieve different shuffling results.

[0037] For example, such as Figure 1 As shown, the source data stored in the source data register at the first register address R6 is divided into two thread segments. The thread index WavIdx of the source data is 0, 1, 2, ..., 30, 31. The shuffling result is that each thread segment contains 16 threads. In each thread segment, each thread copies the data of its 7th thread. Then, the thread index data stored in the source channel index register at the second register address R4 is 6, 6, ..., 6, 6; 22, 22, ..., 22, 22. And P1 is set to 0xFFFFFFFF. The shuffling result will be written to all threads in the target register at the third register address R2.

[0038] like Figure 2As shown, if we want to divide the source data stored in the source data register at address R6 into 8 thread segments Q0~Q7, each with 4 threads, and the original order of the 8 thread segments is Q0, Q1, Q2, Q3, Q4, Q5, Q6, Q7, and want to reorder them as Q0, Q2, Q4, Q6, Q1, Q3, Q5, Q7, then we can set the thread index data to 0, 1, 2, 3, 8, 9, 10, 11, 16, 17, 18, 19, 24, 25, 26, 27, 4, 5, 6, 7, 12, 13, 14, 15, 20, 21, 22, 23, 28, 29, 30, 31, where P1 = 0xFFFFFFFF, the shuffle result will be written to all threads in the target register at address R2.

[0039] like Figure 3 As shown, if you want to move all the source data stored in the source data register at the first register address R6 from left to right by 5 positions and copy it to the target register at the third register address R2, while keeping the data of the first 5 threads in the target register at the third register address R2 unchanged, you can set the thread index data to 0, 0, 0, 0, 0, 0, 1, 2, 3, ..., 24, 25, 26, and set the mask P1 = 0xFFFFFF20 to obtain the data in the target register at the third register address R2.

[0040] from Figure 3 As can be seen, each thread retrieves its value from the source data register at the first register address R6 based on the thread index data stored in the source channel index register at the second register address R4. Then, based on the bit mask of P1, it decides whether to write the value to the destination register at the third register address R2. Since the lower 5 bits of P1 are 0, indicating that the lower 5 threads do not participate in the shuffling operation, their original values ​​are preserved.

[0041] The shuffling instructions have been described above. The execution unit in this application will be described below.

[0042] One embodiment of this application provides an execution unit. This execution unit can be applied to general computing tasks on a GPU, or to an artificial intelligence chip. The artificial intelligence chip can be a general-purpose graphics processing unit (GPGPU) or a neural processing unit (NPU), but is not limited to these. See also... Figure 4The execution unit may include: a thread scheduling unit 41, an instruction cache 42, a thread bundle register 43, an arithmetic logic unit 44, a shared memory control unit 45, and shared memory 46.

[0043] The following is a brief introduction to the execution unit and the functions of each hardware component. Then, we will explain how the execution unit supports shuffle instructions.

[0044] The Thread Scheduling Unit (WVC) 41 is a control unit used to manage and schedule each thread bundle, performing steps such as scheduling thread bundles, fetching instructions, decoding, and issuing. In each clock cycle, a suitable thread bundle needs to be selected from multiple thread bundles according to priority, instructions are read from the instruction cache 42, and executed.

[0045] The arithmetic logic unit 44 is the core component of the execution unit, used to complete the calculation function of the thread bundle. It can process the calculation of multiple threads simultaneously in a SIMD manner, including arithmetic operations, logical operations, bitwise operations, jump judgments, and special functions.

[0046] The instruction cache (IC) 42 is used to read instructions for each thread bundle based on the requested instruction offset address. The instruction cache 42 can be a type of cache, which may contain cache space, matching control logic, tag units, etc. The cache space of the instruction cache 42 can be organized and managed in the form of multiple cache lines.

[0047] The Wave Register File (WRF) 43 is used to temporarily store the source and destination data of instructions for multiple wave bundles. Each register contains multiple threads (this application uses a wave bundle containing 32 threads as an example) and can be read, written and computed in Single Instruction Stream Multiple Data (SIMD).

[0048] Shared memory 46 is a readable and writable static random-access memory (SRAM). Each workgroup can be allocated one block of shared memory 46 for data exchange between multiple thread bundles within the workgroup. Figure 4 In the example shown, the total size of the shared memory 46 is M cache lines, and the M cache lines are numbered from 0 to M-1.

[0049] like Figure 4As shown, the shared memory control unit 45 may include a shared memory controller 451, a shuffling conversion unit 452, a writing unit 453, a reading unit 454, and an atomic operation unit 455. The shared memory controller 451 is communicatively connected to the thread scheduling unit 41, and also communicatively connected to the writing unit 453, the reading unit 454, the atomic operation unit 455, and the shuffling conversion unit 452. The shuffling conversion unit 452 is also communicatively connected to the writing unit 453 and the reading unit 454, respectively. The shared memory control unit 45 controls the shuffling conversion unit 452, the writing unit 453, the reading unit 454, and the atomic operation unit 455 through the shared memory controller 451.

[0050] The thread bundle can access the shared memory 46 by reading and writing through the shared memory control (SMC) 45. The access instructions for the shared memory 46 can be divided into three types of instructions: read instructions, write instructions and atomic operation instructions.

[0051] Read instruction: The SMC reads data from the SM and returns it to the corresponding thread bundle.

[0052] Write command: SMC writes the data of the thread bundle into SM.

[0053] Atomic operation instructions: These are atomic operations for shared memory 46. They read data from SM and return it to the corresponding thread bundle, and perform operations such as addition, subtraction, maximum value (max), minimum value (min), exch, and comparison and swap (cmp_exch), and write the results back to shared memory 46.

[0054] The shuffle conversion unit 452 is used to convert shuffle instructions into write and read instructions for shared memory 46, and write the results into the thread bundle register 43. How the shuffle conversion unit 452 implements the shuffle instructions will be described in detail below.

[0055] The thread scheduling unit 41 schedules the execution of multiple thread bundles. Each time, a thread bundle is selected, and instructions are retrieved from the instruction cache 42. If it is a normal ALU instruction, it is sent to the arithmetic logic unit 44 for execution. If it is an SM instruction (including three types: read instructions, write instructions, and atomic operation instructions) or an SHFL instruction, it is sent to the shared memory control unit 45 for processing.

[0056] The above provides a brief overview of the execution unit and the functions of each hardware component. The following section details how the execution unit supports shuffle instructions.

[0057] like Figures 4 to 6 As shown, the shared memory 46 includes a shuffle cache space 461, and the read unit 454 includes a multiplexer 4541.

[0058] The shuffling conversion unit 452, upon receiving a shuffling instruction, allocates a target cache line for the shuffling instruction from an empty cache line in the shuffling cache space 461. As described above, the shuffling instruction is used to implement data exchange between different threads within the target thread bundle. The shuffling instruction includes the first register address of the source data register, the second register address of the source channel index register, the third register address of the target register, and a mask. The source data register stores the source data, the source channel index register stores the thread index data, the target register stores the shuffled data, and each bit of the mask indicates whether the corresponding thread has written the shuffled data.

[0059] Then, the shuffle conversion unit 452 converts the shuffle instruction into write and read instructions. The starting address of the target cache line is the base address of the write instruction, which also includes the address of the first register. The read instruction includes a mask, the address of the second register, the address of the third register, and the starting address of the target cache line. Next, the shuffle conversion unit 452 calls the write unit 453 to execute the write instruction, writing the source data stored in the source data register into the target cache line. Then, the shuffle conversion unit 452 calls the read unit 454 to execute the read instruction, reading the source data from the target cache line. Then, using the multiplexer 4541, the data to be written to the corresponding thread is selected from the source data using the thread index data, and stored in the target register according to the mask. In this way, the execution unit can support shuffle instructions in hardware, enabling data exchange between different threads within a thread bundle, thereby improving execution efficiency.

[0060] In one embodiment, the shuffling conversion unit 452 includes a control register, a shuffling cache capacity register, and a cache resource allocation unit. Data stored in the control register is used to indicate whether the operator of the target thread bundle uses a shuffling instruction. Data stored in the shuffling cache capacity register is used to indicate the size of the shuffling cache space 461, the size of which is measured in cache lines. The cache resource allocation unit is used to set the size of the shuffling cache space 461 according to the data in the shuffling cache capacity register.

[0061] The data stored in the control register and the data stored in the shuffle buffer capacity register are configured by the driver program according to the operator program. Before the operator program executes, the driver program can configure the data in the control register and the data in the shuffle buffer capacity register according to the operator program.

[0062] When the data stored in the control register indicates that the operator program uses the shuffle instruction, the shared memory control unit 45 enables the function of data exchange between different threads within the target thread bundle based on the shuffle instruction. When the operator program does not use the shuffle instruction, the shared memory control unit 45 disables the function of data exchange between different threads within the target thread bundle based on the shuffle instruction and releases the shuffle cache space 461.

[0063] The data in the control register can be 1 bit. For example, when the operator uses the shuffle instruction, the data in the control register can be configured to 1, so that the shared memory control unit 45 enables the function of data exchange between different threads within the target thread bundle based on the shuffle instruction. The Compute Shared Thread Controller (CSTC) notifies the shared memory control unit 45 to initialize the shared memory 46 space for each workgroup. When the operator does not use the shuffle instruction, the data in the control register can be configured to 0, so that the shared memory control unit 45 disables the function of data exchange between different threads within the target thread bundle based on the shuffle instruction and releases the shuffle cache space 461 for use as shared memory 46. This eliminates the need to initialize the shared memory 46 space for the workgroup, thus improving the execution efficiency of shared memory 46.

[0064] In 3D applications or other operator programs, shuffle instructions are generally not needed to exchange data between different threads in a thread bundle. Therefore, by setting the data in the control register, the SMC can be made to disable the function of exchanging data between different threads in the target thread bundle based on shuffle instructions, so as to release the shuffle cache space 461 and use it as shared memory 46. The shared memory 46 space of the workgroup is not initialized, which can improve the execution efficiency of shared memory 46.

[0065] When the operator does not use the shuffle instruction, it is not necessary to set the data in the shuffle buffer capacity register.

[0066] In one embodiment, when the thread bundle includes 32 threads, the data in the shuffle cache capacity register can be 5 bits. The size of the shuffle cache space 461 can be between 0 and 31.

[0067] In one embodiment, when the operator uses a shuffle instruction, the size of the shuffle buffer space 461 can be equal to 1.

[0068] In another embodiment, when the operator uses a shuffle instruction, the size of the shuffle buffer space 461 can be greater than 1, so that the shuffle conversion unit 452 can support the execution of more than one shuffle instruction at the same time.

[0069] In one embodiment, the size of the shuffle buffer space 461 can be greater than the number of clock cycles required for the total delay of executing write and read instructions. This allows multiple shuffle instructions to be executed simultaneously, avoiding instruction gaps and improving execution efficiency.

[0070] Since the shuffle conversion unit 452 needs to split a single shuffle instruction into two SM access requests—the first to write to the SM and the second to read from it—and because the path for reading and writing to the SM is relatively long and cannot return results immediately, it requires multiple clock cycles to complete. If the shuffle conversion unit 452 receives multiple shuffle instructions, but the size of the shuffle buffer space 461 is only set to 1, then other shuffle instructions must be paused and wait for the current shuffle instruction to complete before execution. This can easily create instruction bubbles and reduce execution efficiency. Therefore, it is best to set the size of the shuffle buffer space 461 to be greater than the number of clock cycles for the total delay of reading and writing to the SM. For example, setting it to 10 means that a maximum of 10 SHFLs will not be paused. Since the shuffle buffer space 461 is shared with the SM, an excessively large shuffle buffer space 461 will affect the allocated size of the SM, and thus affect the execution efficiency of the SM. The specific value of the shuffle buffer space 461 should be determined based on actual performance testing.

[0071] In one embodiment, such as Figure 4 As shown, the shuffle conversion unit 452 includes a register array 4521, in which a control register and a shuffle buffer capacity register may be located.

[0072] In one embodiment, such as Figure 4 and Figure 5 As shown, the shuffle cache space 461 is located at the top of shared memory 46. When shared memory 46 contains M cache lines, the base address of the shuffle cache space 461 is M - SHFLBufSize, where M is greater than SHFLBufSize, and SHFLBufSize is the size of the shuffle cache space 461. The address range of the shuffle cache space 461 is (M - BufSize, M - 1). Each cache line can store the data of all threads in a thread bundle. For example, if M = 1024 and SHFLBufSize = 10, then the SM space with cache line numbers (1014 ~ 1023) in SM will be used as the shuffle cache space 461.

[0073] After the shared memory control unit 45 receives the shuffle instruction, the shuffle conversion unit 452 determines whether there is a free cache line in the shuffle cache space 461. If there is no free cache line, the shuffle instruction is suspended. If there is a free cache line, a target cache line is allocated in the free cache line in the shuffle cache space 461 for the shuffle instruction as the address of the write instruction.

[0074] If the line number of the target cache line is bufidx, then the line number address SHFLWrIdx of the target cache line in shared memory 46 is SHFLWrIdx = M - SHFLBufSize + bufidx. The value of bufidx ranges from (0, SHFLBufSize - 1).

[0075] like Figure 5 As shown, the shuffle conversion unit 452 calls the write unit 453 to execute the write instruction, writing the source data stored in the source data register to the target cache line. Figure 5 Rs0 in the instruction represents the address of the first register of the source data register, indicating the source data stored at that address. The write instruction can be written in the following form: SMWR(SHFLWrIdx,SMLineOffset)Rs0 formula (3) SMWR is the write instruction, and SMLineOffset is the offset address of each thread. Since the entire data of Rs0 needs to be written in sequence, when the thread bundle includes 32 threads and is executed according to simd32, SMLineOffset = {0, 1,2, 3, …, 31}, which means that 32 threads write to the shuffle buffer space 461 consecutively.

[0076] After the write operation is completed, the shuffling and conversion unit 452 calls the read unit 454 to execute a read instruction, reading the source data from the target cache line. The read instruction here can take the following form: (Pn)SMRDRd, (SHFLWrIdx, Rs1) Formula (4) SMRD is a read instruction, Rd is the address of the third register of the destination register, SHFLWrIdx is the line number address of the destination cache line in shared memory 46, Rs1 is the address of the second register of the source channel index register, and the internal offset of the SM cache line comes from the address of the second register of the source channel index register of SHFL, Rs1.

[0077] SMC will read the entire line of data at position SHFLWrIdx from SM, and then, as follows Figure 6 As shown, the multiplexer 4541 in the read unit 454 uses the thread index data in Rs1 to select the data to be written to the corresponding thread from the source data, and stores it in the destination register according to the mask. When the thread bundle includes 32 threads, since the read unit 454 has a multiplexer 4541 with 32 data channels, it can select the appropriate channel result according to Rs1 in the shuffle instruction, use it as the shuffled data, and store it in the destination register according to the mask. This process utilizes the existing register write logic of SMC and does not require additional hardware logic.

[0078] In existing technologies, AI operators frequently use shuffle instructions to exchange data between different wires on a given thread bundle, but this requires a relatively expensive multiplexer (e.g., 4541) to support. This application translates the shuffle instructions into special SM access methods, which offers the following advantages compared to implementing them directly in the ALU module: (1) The shuffle instruction is implemented by using the existing SM read and write logic, which has relatively low hardware overhead; (2) By adding a shuffle conversion unit 452 in the SMC, the shuffle instructions are converted into write and read operations on the shared memory 46, which has little impact on the overall hardware. (3) By adding a control register to control the SHFL function switch, the use of shuffle instructions can be flexibly controlled; when shuffle instructions are not needed, there is no need to allocate SM cache space for shuffle instructions, thus saving SM space and improving the execution efficiency of SM.

[0079] Another embodiment of this application provides a chip. The chip includes the execution unit of any of the above embodiments.

[0080] In one implementation, the chip described above may be a graphics processor or an artificial intelligence chip.

[0081] In one implementation, the aforementioned artificial intelligence chip may be a general-purpose graphics processor or a neural network processor, but is not limited thereto.

[0082] Since the chip includes the aforementioned execution unit, it can also support shuffle instructions in hardware, enabling data exchange between different threads within a thread bundle, thereby improving execution efficiency.

[0083] The various embodiments of this application can complement each other.

[0084] In this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0085] The above description of the embodiments is intended to enable those skilled in the art to understand and apply this application. It will be apparent to those skilled in the art that various modifications can be easily made to these embodiments, and the general principles described herein can be applied to other embodiments without creative effort. Therefore, this application is not limited to the embodiments described herein, and any improvements and modifications made by those skilled in the art based on the disclosure of this application without departing from the scope and spirit of this application are within the scope of this application.

Claims

1. An execution unit, characterized in that, include: A shared memory control unit and shared memory; the shared memory control unit is communicatively connected to the shared memory; the shared memory includes a shuffle cache space; The shared memory control unit includes a shuffling and conversion unit, a write unit, and a read unit; the read unit includes a multiplexer. The shuffling conversion unit is used to allocate a target cache line for the shuffling instruction in an empty cache line in the shuffling cache space after receiving the shuffling instruction. The shuffling instruction is used to realize data exchange between different threads within the target thread bundle. The shuffling instruction includes a first register address of the source data register, a second register address of the source channel index register, a third register address of the target register, and a mask. The source data register is used to store source data, the source channel index register is used to store thread index data, the target register is used to store the shuffled data, and the value of each bit of the mask is used to indicate whether the corresponding thread writes the shuffled data. The shuffle conversion unit is further configured to convert the shuffle instruction into a write instruction and a read instruction, wherein the starting address of the target cache line is the base address of the write instruction; the write instruction further includes the first register address; the read instruction includes the mask, the second register address, the third register address, and the starting address; The shuffling conversion unit is also used to call the writing unit to execute the write instruction and write the source data stored in the source data register to the target cache line; The shuffling conversion unit is also used to call the read unit to execute the read instruction, read the source data from the target cache line, and then use the multiplexer to select the data to be written to the corresponding thread from the source data using the thread index data, and store it in the target register according to the mask.

2. The execution unit as described in claim 1, characterized in that, The shuffling conversion unit includes a control register and a shuffling buffer capacity register; The data stored in the control register is used to indicate whether the operator of the target thread bundle uses the shuffle instruction; The data stored in the shuffle cache capacity register is used to indicate the size of the shuffle cache space; the size of the shuffle cache space is measured in cache lines. The data stored in the control register and the data stored in the shuffle buffer capacity register are configured by the driver program according to the operator program. When the data stored in the control register indicates that the operator program uses the shuffle instruction, the shared memory control unit enables the data exchange function between different threads within the target thread bundle based on the shuffle instruction. When the operator program does not use the shuffle instruction, the shared memory control unit disables the data exchange function between different threads within the target thread bundle based on the shuffle instruction and releases the shuffle cache space.

3. The execution unit as described in claim 2, characterized in that, The size of the shuffle cache space is greater than or equal to 1.

4. The execution unit as described in claim 3, characterized in that, The size of the shuffle cache space is greater than the number of clock cycles required for the total delay of executing the write instruction and the read instruction.

5. The execution unit as described in claim 2, characterized in that, The shuffling cache space is located at the top of the shared memory; When the shared memory includes M cache lines, the base address of the shuffle cache space is M - SHFLBufSize, where M is greater than SHFLBufSize, SHFLBufSize is the size of the shuffle cache space, and the address range of the shuffle cache space is (M - BufSize, M - 1).

6. The execution unit as described in claim 5, characterized in that, After the shared memory control unit receives the shuffle instruction, the shuffle conversion unit determines whether there is a free cache line in the shuffle cache space. If not, the shuffle instruction is suspended. If it is, a target cache line is allocated for the shuffle instruction from the free cache lines in the shuffle cache space.

7. The execution unit as described in claim 5, characterized in that, If the line number of the target cache line is bufidx, then the line number address SHFLWrIdx of the target cache line in the shared memory is SHFLWrIdx = M - SHFLBufSize + bufidx.

8. The execution unit as described in claim 1, characterized in that, It also includes a thread scheduling unit, instruction cache, and thread bundle register; The thread scheduling unit is configured to select the target thread bundle from multiple thread bundles in each clock cycle and control the target thread bundle to retrieve the corresponding operator program from the instruction cache, the operator program including the shuffle instruction; The thread bundle register is used to temporarily store the source data and shuffled data of shuffling instructions for multiple thread bundles; the thread bundle includes multiple threads.

9. A chip, characterized in that, Includes the execution unit as described in any one of claims 1-8.

10. The chip as described in claim 9, characterized in that, The chip is a graphics processor or an artificial intelligence chip.

Images