Universal computing unit and computing device

Abstract

The invention provides a general purpose computing unit and computing equipment. The general-purpose computing unit comprises a plurality of execution units, wherein each execution unit is used for executing an operation instruction by taking a thread bundle as a unit; and a thread local register for each execution unit, including a plurality of thread beam registers allocated for a plurality of thread beams run by the execution unit, the plurality of thread beam registers being non-overlapping, where the thread local register further includes a shared register shared by the plurality of thread beams.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of processors, and more specifically, to a general computing unit and a computing device. BACKGROUND

[0002] Currently, in the programming mode of a computing device such as a General Graphic Processing Unit (GPGPU), there exist concepts of thread grid, thread block, warp and thread, which correspond to different levels of entities in the computing device, respectively. For example, a thread block is fixedly scheduled to be executed on a general computing unit of the computing device, while a warp is scheduled to be executed on an execution unit in the general computing unit.

[0003] In a general programming scenario, there can be a scenario where data is shared or exchanged between warps. Currently, for data sharing or exchanging between different warps within the same execution unit or different execution units, a shared memory shared by all execution units of the general computing unit is usually used. However, when multiple warps within one execution unit need to share or exchange data, since the shared memory is located outside the execution unit, the delay of data exchange through the shared memory is longer, and the bandwidth between the execution unit and the shared memory is smaller, so that the efficiency of data exchange between these warps is lower, and can affect the operation of the shared memory by other execution units. SUMMARY

[0004] To solve the above problems, the present disclosure provides a general computing unit and a computing device containing the same, which allocates a shared register dedicated to data exchange between multiple warps of an execution unit in a thread local register for each execution unit, so that data exchange between warps of the execution unit is more efficient.

[0005] According to one aspect of the present disclosure, a general computing unit is provided. The general computing unit comprises: a plurality of execution units, wherein each execution unit is configured to execute an operation instruction in a unit of warp; and a thread local register for each execution unit, comprising a plurality of warp registers allocated for a plurality of warps run by the execution unit, the plurality of warp registers being mutually exclusive, wherein the thread local register further comprises a shared register shared by the plurality of warps.

[0006] In some implementations, the shared register is part of one of the plurality of warp registers.

[0007] In some implementations, the shared register is a reserved register in the thread local registers.

[0008] In some implementations, the plurality of thread bundle registers includes a first thread bundle register for a first thread bundle and a second thread bundle register for a second thread bundle, wherein when the first thread bundle is a data producer and the second thread bundle is a data consumer, the produced data is written to the shared register by the first thread bundle, and a synchronization instruction is executed to notify the shared register that the data is ready; and after the second thread bundle is dispatched to the execution unit, a synchronization instruction is executed by the second thread bundle to wait until the shared register data is ready and data is read from the shared register to the second thread bundle register.

[0009] In some implementations, the plurality of thread bundle registers includes a first thread bundle register for a first thread bundle and a second thread bundle register for a second thread bundle, wherein when the first thread bundle is a data consumer and the second thread bundle is a data producer, a synchronization instruction is executed by the first thread bundle to wait until the shared register data is ready and data is read from the shared register to the first thread bundle register; and after the second thread bundle is dispatched to the execution unit, the produced data is written to the shared register by the second thread bundle, and a synchronization instruction is executed to notify the shared register that the data is ready.

[0010] In some implementations, the general purpose computing unit further includes a shared memory for the plurality of execution units, wherein when thread bundles of two different execution units in the plurality of execution units are a data producer and a data consumer respectively, data exchange between the thread bundles of the two different execution units is performed through the shared memory.

[0011] According to another aspect of the disclosure, a computing device is provided. The computing device includes a plurality of general purpose computing units as described above; and a global memory for data exchange between the plurality of general purpose computing units. BRIEF DESCRIPTION OF DRAWINGS

[0012] The disclosure will be better understood and other objects, details, features, and advantages of the disclosure will become apparent from the following, detailed description, taken in conjunction with the accompanying drawings.

[0013] Figure 1 An exemplary structural schematic diagram of a computing device is shown.

[0014] Figure 2 An exemplary structural schematic diagram of a stream processor cluster 200 is shown.

[0015] Figure 3 An exemplary structural diagram of a general purpose computing unit 300 is shown.

[0016] Figure 4 A thread bundle scheduling diagram of multiple execution units in a general purpose computing unit is shown.

[0017] Figure 5A An exemplary diagram of thread local registers for an execution unit according to some embodiments of the present application is shown.

[0018] Figure 5B An exemplary diagram of thread local registers for an execution unit according to some other embodiments of the present application is shown.

[0019] Figure 5C An exemplary diagram of thread local registers for an execution unit according to some further embodiments of the present application is shown. DETAILED DESCRIPTION

[0020] Preferred embodiments of the present disclosure will be described herein below with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art.

[0021] The term "comprising" and variations thereof as used herein are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Unless otherwise specified, the term "or" as used herein is intended to mean "and / or". The term "based on" means "based, at least in part, on". The terms "one embodiment" and "some embodiments" as used herein mean "at least one example embodiment". The term "another embodiment" as used herein means "at least one additional embodiment". The terms "first", "second", and the like as used herein can refer to different or same objects.

[0022] Figure 1 An exemplary structural diagram of a computing device 100 is shown. As shown in Figure 1 The computing device 100 can include multiple Stream Processing Clusters (SPCs) 110 (e.g., SPC 110a, SPC 110b, and SPC 110c) and a general purpose computing unit 300 (e.g., a general purpose computing unit 300a, a general purpose computing unit 300b, and a general purpose computing unit 300c) as shown in Figure 1The diagram exemplarily illustrates 16 stream processor clusters 110. The computing device 100 assigns tasks to each stream processor cluster 110 in units of a thread grid. Multiple stream processor clusters 110 communicate with each other, for example, via a network-on-chip (NoC) 120. Furthermore, each stream processor cluster 110 can also communicate with off-chip memory 130 located outside the computing device 100 to read data from or write data to off-chip memory 130. Off-chip memory 130, for example, can communicate with each stream processor cluster 110 of the computing device 100 via protocols such as PCIe, and has a large storage capacity but low bandwidth for transmission with the computing device 100.

[0023] Here, the computing device 100 can be a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), or other devices with similar structures, such as AI chips or AI accelerators.

[0024] Figure 2 An exemplary structural schematic diagram of a stream processor cluster 200 is shown. Figure 2 The stream processor cluster 200 shown can, for example, be used as... Figure 1 The computing device 100 includes a stream processor cluster 110. For example... Figure 2 As described above, the stream processor cluster 200 may include multiple general-purpose computing units 210 ( Figure 2 The example shows four general-purpose computing units 210 and one or more tensor computing units 220. Figure 2 An example is shown of a tensor computation unit 220. The stream processor cluster 200 distributes tasks to each general-purpose processing unit 210 in units of thread blocks. The tensor computation unit 220 performs matrix operations such as matrix multiplication and addition, and convolution, while the general-purpose computation unit 210 performs other general-purpose computations besides matrix operations, such as vector operations and scalar operations. Note that... Figure 2 In this paper, the general computing unit 210 and the tensor computing unit 220 are presented as independent hardware only from a functional perspective. In some embodiments, the general computing unit 210 and the tensor computing unit 220 can be integrated, for example, the tensor computing unit 220 can be implemented in the general computing unit 210.

[0025] In addition, the stream processor cluster 200 may also include a global memory 230, which is used to implement data exchange between general computing units 210 or between general computing units 210 and tensor computing units 220.

[0026] Figure 3 An exemplary structural schematic diagram of a general-purpose computing unit 300 is shown. Figure 3 The general-purpose computing unit 300 shown can, for example, be used as... Figure 2 The general-purpose computing unit 210 of the stream processor cluster 200. For example... Figure 3 As described above, the general-purpose computing unit 300 may include multiple execution units 310 ( Figure 3 The diagram exemplarily illustrates four execution units 310 and shared memory 320. The general-purpose computing unit 300 distributes tasks to each execution unit 310 in units of warps. Shared memory 320 is used for data exchange between the multiple execution units 310.

[0027] like Figure 3 As shown, each execution unit 310 may include a vector operation unit 312, a scalar operation unit 314, a thread local register (TLR) 316, and a warp scalar register (WSR) 318. The vector operation unit 312 can execute issued instructions (e.g., Single Instruction Multiple Data (SIMD) instructions) on a thread-by-thread basis to perform vector operations, such as 8-bit or 16-bit vector multiplication. The scalar operation unit 314 can be used to perform scalar operations, such as 16-bit or 32-bit scalar addition. The thread local register 316 is a register group consisting of multiple registers, used to store the input data and / or output data required by each thread of the thread bundle running on each vector operation unit 312. The thread local register 316 can be a register space (e.g., 256kB) consisting of multiple registers, each register capable of storing one element of a matrix (e.g., a 16-bit floating-point or 8-bit integer). The thread bundle scalar register 318 is used to store the input data required by the scalar arithmetic unit 314 and / or the output data generated therefrom.

[0028] Typically, each execution unit 310 can sequentially invoke multiple operation instructions, each operation instruction corresponding to a thread bundle. For example, Figure 4 A schematic diagram illustrating thread bundle scheduling of multiple execution units 310 within the general-purpose computing unit 300 is shown. For example... Figure 4As shown in FIG. 4, it is assumed that four execution units 310 are scheduled with thread bundle WO, thread bundle Wl, thread bundle W2, and thread bundle W3 in one instruction scheduling period, and scheduled with thread bundle W4, thread bundle W5, thread bundle W6, and thread bundle W7 in the next instruction scheduling period. For the same execution unit 310 (such as the first execution unit 310), the thread bundles (such as thread bundle WO and W4) scheduled in succession for the same execution unit 310 have a time difference in scheduling, but they run in a pipelined manner, and the running time of the two thread bundles can overlap. That is, thread bundle W4 can start running before thread bundle WO is completely run.

[0029] At present, data exchange between different thread bundles (such as thread bundle WO and thread bundle W4) in the same execution unit 310 (such as the first execution unit 310), and data exchange between different thread bundles (such as thread bundle WO and thread bundle Wl) of different execution units 310 (such as the first execution unit 310 and the second execution unit 310) need to be performed through the shared memory 320. At this time, as shown in FIG. 5, the data exchange between different thread bundles (such as thread bundle WO and thread bundle W4) in the same execution unit 310 (such as the first execution unit 310) is performed through the shared memory 320. Figure 3 As shown in FIG. 6, each kernel function has a large demand for the computing and storage resources of the general computing unit 300, and usually occupies the entire shared memory 330 of the general computing unit 300 to mask the latency overhead of accessing the global memory 230.

[0030] In addition, in the traditional general computing scenario, each thread block only processes a limited amount of data, and different amounts of data are processed by specifying different sizes of thread grids when starting the kernel function, and each thread block usually occupies fewer resources to fully utilize the concurrent parallel characteristics of multiple thread blocks in the general computing unit 300, so that there is a scenario in which multiple thread blocks run on one general computing unit 300 at the same time.

[0031] In the modern AI computing scenario, in order to further mask the latency overhead of accessing the global memory 230, the user often needs to query the number of general computing units 300 on the computing device 100 to start the corresponding thread grid. Each thread block occupies all the resources of one general computing unit 300 for a long time, and processes the overall workload by dynamically querying the global index (tile scheduler).

[0032] However, in this case, one thread block will occupy all the resources of one general computing unit 300 for a long time, and at this time, one thread block actually occupies all the register resources on the general computing unit 300. In this case, there is a possibility of using the characteristic that "one thread block actually occupies all the register resources on the general computing unit 300" to exchange data between thread bundles.

[0033] However, due to the small bandwidth between execution unit 310 and shared memory 320, and the fact that shared memory 320 is located outside execution unit 310, the latency of data exchange between thread bundles via shared memory 320 is long, which may affect the workload of other execution units 310, especially when the thread bundles exchanging data are thread bundles assigned to the same execution unit 310.

[0034] To address the aforementioned problems, this disclosure provides a scheme for performing data exchange between thread bundles of execution unit 310 by utilizing the thread local register 316 of execution unit 310. More specifically, a shared register dedicated to data exchange between multiple thread bundles of execution unit 310 can be allocated from the thread local register 316 of execution unit 310, making the data exchange between thread bundles of execution unit 310 more efficient.

[0035] Figures 5A to 5C A schematic diagram of a thread-local register 316 for an execution unit 310 according to an embodiment of the present invention is shown. For example, as Figures 5A to 5C As shown, thread-local register 316 can be a 256kB register space with physical addresses from 0 to 255. The physical address of thread-local register 316 supports wrapback; that is, access to physical address 256 can wrap back to physical address 0.

[0036] The thread-local register 316 may include multiple thread bundle registers allocated for multiple thread bundles executed by the execution unit 310, and these thread bundle registers do not overlap. The multiple thread bundle registers for the multiple thread bundles are allocated in the thread-local register 316 one thread bundle at a time in the order of the thread bundles, and the register space of the thread bundle register of each thread bundle is contiguous and does not overlap with the register space of the thread bundle registers of other thread bundles.

[0037] Furthermore, according to an embodiment of the present invention, the thread local register 316 also includes a shared register shared by a plurality of thread bundles executed by the execution unit 310.

[0038] For example, such as Figures 5A to 5C As shown, taking the first execution unit 310 as an example, it is assumed that the thread bundle run by the execution unit 310 includes the first thread bundle (for example, as described above). Figure 4 The aforementioned thread bundle W0) and the second thread bundle (e.g., as combined above) Figure 4The thread local register 316 of the execution unit 310 may include a first thread bundle register 3162 for the first thread bundle W0 and a second thread bundle register 3164 for the second thread bundle W4. The first thread bundle register 3162 may correspond to a contiguous register space, and the second thread bundle register 3164 may correspond to another contiguous register space, and these two register spaces do not overlap. For example, as... Figures 5A to 5C As shown, assume that the first thread bundle register 3162 allocated to the first thread bundle W0 and the second thread bundle register 3164 allocated to the second thread bundle W4 are both 64kB register groups. The physical addresses of the first thread bundle register 3162 are X~X+63, and the physical addresses of the second thread bundle register 3164 are X+64~X+127, where X is 0 or a positive integer. Furthermore, the register spaces with physical addresses 0~X and X+128~255 are unallocated register spaces, also known as reserved registers.

[0039] In addition, the thread-local register 316 of the execution unit 310 also includes a shared register 3166 shared by the first thread bundle W0 and the second thread bundle W4.

[0040] In some embodiments, shared register 3166 may be part of a first thread bundle register 3162 or a second thread bundle register 3164. For example, as Figure 5A As shown, shared register 3166 can be part of the first thread bundle register 3162, such as a 1kB register space at physical address X+63. In this case, the first thread bundle W0 can write data to this register space, and the second thread bundle W4 can read data from this register space. Or, as... Figure 5B As shown, shared register 3166 can also be part of the second thread bundle register 3164, such as a 1kB register space with physical address X+127. In this case, the second thread bundle W4 can write data to this register space, and the first thread bundle W0 can read data from this register space. Although X+63 and X+127 are used as examples to illustrate the case where shared register 3166 is located in the first thread bundle register 3162 and the second thread bundle register 3164 respectively, those skilled in the art will understand that shared register 3166 can be located at any location in the first thread bundle register 3162 or the second thread bundle register 3164, and is not limited to the specific locations mentioned above.

[0041] In other embodiments, shared register 3166 may be a reserved register in thread-local register 316, i.e., a register space located in thread-local register 316 that has not been allocated, such as... Figure 5CThe register space shown is 1kB within the register space with physical addresses 0~X or X+128~255. Figure 5C The diagram illustrates shared register 3166 as a register space with physical address X+128. Similarly, shared register 3166 can be located anywhere in the unallocated register space of thread-local register 316.

[0042] Those skilled in the art will understand that the above description of shared register 3166 is based on 1kB as an example only. The size of shared register 3166 is not limited to 1kB, but can be any value smaller than the corresponding physical register space (first thread bundle register 3162, second thread bundle register 3164 or reserved register).

[0043] Because multiple thread bundles within the same execution unit 310 have sequential relationships, access to the shared register 3166 can be implemented in different ways depending on whether these thread bundles act as data producers or data consumers. Here, a data producer refers to a thread bundle that generates data, and a data consumer refers to a thread bundle that uses the data generated by the data producer. Therefore, there is a dependency relationship between data consumers and data producers; data consumers depend on data producers.

[0044] In some embodiments, such as Figure 5A As shown, shared register 3166 is part of the first thread bundle register 3162. Assuming the first thread bundle W0 is the data producer and the second thread bundle W4 is the data consumer, in this case, the first thread bundle W0 can write the generated data to shared register 3166 and execute a synchronization instruction to notify shared register 3166 that the data is ready. After the second thread bundle W4 is sent to execution unit 310, it can execute a synchronization instruction to wait until it is confirmed that shared register 3166 is ready and then read the data from shared register 3166 into the second thread bundle register 3164. Here, the write instruction, synchronization instruction, and read instruction are all register hardware instructions. In other embodiments, such as Figure 5B As shown, shared register 3166 is part of the second thread bundle register 3164. Assuming the second thread bundle W4 is the data producer and the first thread bundle W0 is the data consumer, in this case, a synchronization instruction can be executed through the first thread bundle W0 to wait until data in shared register 3166 is ready and then read data from shared register 3166 into the first thread bundle register 3162. After the second thread bundle W4 is issued to the execution unit 310, it writes the generated data into shared register 3166 and executes a synchronization instruction to notify shared register 3166 that data is ready. Similarly, write instructions, synchronization instructions, and read instructions are all register hardware instructions.

[0045] In some other embodiments, such as Figure 5C As shown, shared register 3166 is a reserved register in thread-local register 316. Assuming the first thread bundle W0 is a data producer and the second thread bundle W4 is a data consumer, in this case, with... Figure 5A Similarly, the first thread bundle W0 can write the generated data to shared register 3166 and execute a synchronization instruction to notify shared register 3166 that the data is ready. After the second thread bundle W4 is issued to the execution unit 310, a synchronization instruction can be executed to wait until shared register 3166 is ready and then read the data from shared register 3166 into the second thread bundle register 3164. On the other hand, assuming the second thread bundle W4 is the data producer and the first thread bundle W0 is the data consumer, in this case, the first thread bundle W0 can execute a synchronization instruction to wait until shared register 3166 is ready and then read the data from shared register 3166 into the first thread bundle register 3162. After the second thread bundle W4 is issued to the execution unit 310, it writes the generated data to shared register 3166 and executes a synchronization instruction to notify shared register 3166 that the data is ready. Similarly, the write instruction, synchronization instruction, and read instruction are all register hardware instructions.

[0046] Furthermore, when thread bundles in different execution units act as data producers and data consumers respectively, their data exchange can still be performed through shared memory 320, which will not be elaborated here.

[0047] By utilizing shared registers within the execution unit (such as shared register 3166) to achieve data exchange between different thread bundles within that execution unit, faster transfers can be achieved compared to using shared memory (such as shared memory 330). For example, the bandwidth of shared memory is typically 1 or 2 kilobytes per second, while the bandwidth of registers is typically much larger (e.g., 1 kilobyte per second). (Number of execution units within each general-purpose computing unit). Furthermore, compared to using shared memory (such as shared memory 330), data latency can be significantly reduced. For example, register latency is typically within 10 cycles, while shared memory latency is typically around 30 cycles. Therefore, since completing a data exchange involves one read and one write, the time overhead for data exchange between different thread bundles within the same execution unit can be reduced from approximately 60 cycles to approximately 20 cycles.

[0048] Further, data exchange between different thread bundles within an execution unit is implemented using shared registers (e.g., shared registers 3166) within the execution unit, and multiple execution units can perform their respective internal data exchange in parallel without affecting the workloads of other execution units.

[0049] Further, data exchange between different thread bundles within an execution unit is implemented using shared registers (e.g., shared registers 3166) within the execution unit, and there is no need to allocate dedicated space in shared memory (e.g., shared memory 330) for data exchange between multiple thread bundles within each execution unit, thereby saving the usage of shared memory.

[0050] The general-purpose computing unit and computing device according to the present disclosure are described above in conjunction with the accompanying drawings. However, those skilled in the art can understand that the general-purpose computing unit and computing device described above do not necessarily include all the components shown in the drawings, and can only include some or more components necessary for performing the functions described in the present disclosure, and the connection manner of the components is not limited to the form shown in the drawings.

[0051] The present disclosure can be implemented as a method, a computing device, a system, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein for executing various aspects of the present disclosure. The computing device can include at least one processor and at least one memory coupled to the at least one processor, and the memory can store instructions for execution by the at least one processor. The instructions, when executed by the at least one processor, can cause the computing device to perform the method described above.

[0052] In one or more example designs, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include computer-readable storage media. A computer-readable storage medium can be any available medium or

[0053] Various units of the apparatus disclosed herein can be implemented using discrete hardware components, or can be integrated in one hardware component, such as a processor. For example, the various exemplary logical blocks, modules, and circuits described in conjunction with the present disclosure can be implemented or executed with a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of the above for performing the functions described herein.

[0054] Those of ordinary skill in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both.

[0055] The foregoing description of the present disclosure has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the disclosure be limited not with this detailed description, but rather by the claims appended hereto.

Claims

1. A general-purpose computing unit, characterized in that, include: Multiple execution units, each of which executes arithmetic instructions in units of thread bundles; as well as The thread-local registers for each execution unit include multiple thread bundle registers allocated for multiple thread bundles running by the execution unit, the multiple thread bundle registers not overlapping. The thread-local registers mentioned therein also include shared registers shared by the plurality of thread bundles.

2. The general-purpose computing unit as described in claim 1, characterized in that, The shared register is part of one of the plurality of thread bundle registers.

3. The general-purpose computing unit as described in claim 1, characterized in that, The shared register is a reserved register in the thread-local register.

4. The general-purpose computing unit as described in claim 1, characterized in that, The plurality of thread bundle registers includes a first thread bundle register for a first thread bundle and a second thread bundle register for a second thread bundle. When the first thread bundle acts as a data producer and the second thread bundle acts as a data consumer, the first thread bundle writes the generated data into the shared register and executes a synchronization instruction to notify the shared register that the data is ready. as well as After the second thread bundle is sent to the execution unit, a synchronization instruction is executed through the second thread bundle to wait until the shared register data is ready and the data is read from the shared register into the second thread bundle register.

5. The general-purpose computing unit as described in claim 1, characterized in that, The plurality of thread bundle registers includes a first thread bundle register for a first thread bundle and a second thread bundle register for a second thread bundle. When the first thread bundle acts as a data consumer and the second thread bundle acts as a data producer, the first thread bundle executes a synchronization instruction to wait until the shared register data is ready and read the data from the shared register into the first thread bundle register. as well as After the second thread bundle is sent to the execution unit, the generated data is written to the shared register through the second thread bundle, and a synchronization instruction is executed to notify the shared register that the data is ready.

6. The general-purpose computing unit as described in claim 1, characterized in that, Also includes: Shared memory is used for the plurality of execution units, wherein when the thread bundles of two different execution units among the plurality of execution units act as data producers and data consumers respectively, data exchange between the thread bundles of the two different execution units is performed through the shared memory.

7. A computing device, characterized in that, include: Multiple general-purpose computing units as described in any one of claims 1 to 6; as well as Global memory, which is used for data exchange between multiple general-purpose computing units.

Images