Gpu and related methods

Abstract

A GPU and related methods are disclosed. The GPU includes a plurality of warp processor groups, an available hardware resource table, a resource comparator, a warp scheduler, and a global scheduler. Each warp processor group is configured to execute at least one thread block. The available hardware resource table records available hardware resources of the plurality of warp processor groups. The resource comparator selects, from a plurality of first-in-line kernel codes in a plurality of kernel flows, at least one schedulable kernel code whose required hardware resources are less than the available hardware resources of the plurality of warp processor groups based on the available hardware resource table. The warp scheduler selects a selected kernel code from the at least one schedulable kernel code. The global scheduler assigns the selected kernel code to the plurality of warp processor groups for execution and updates the available hardware resource table based on hardware resource usage states of the plurality of warp processor groups.

Description

Technical Field

[0001] This application relates to a GPU, and more particularly to a GPU capable of processing multiple kernel codes in parallel. Background Technology

[0002] Because of their parallel computing capabilities, graphics processing units (GPUs) are not only used to render 3D images but also frequently used to accelerate artificial intelligence models or big data analysis that require massive parallel computation. Generally, a GPU can comprise multiple streaming multiple processors (SMs), and each SM can contain multiple streaming processors (SPs). When a central processing unit (CPU) wants to utilize the GPU to perform computations, it packages the required operations into multiple kernel codes. Each kernel code can contain multiple thread blocks, and each thread block can contain multiple thread bundles. In this scenario, the GPU can sequentially assign the multiple thread blocks within each kernel code to the multiple streaming multiple processors within it for execution. The multiple streaming processors within each streaming multiple processor group can then execute the multiple threads of the corresponding thread bundles within each thread block in parallel.

[0003] In some cases, to improve GPU computing performance, the central processing unit (CPU) can place non-dependent kernel code into different kernel streams. When the GPU receives multiple kernel streams, it can execute the kernel code in different streams in parallel, thereby improving computing performance. However, since the multiple kernel codes in each kernel stream still have a fixed execution order, and some different kernel codes may compete for the same type of hardware resources, the GPU still faces considerable limitations in scheduling kernel code, making it difficult to effectively improve the GPU's hardware utilization. In other words, how to schedule kernel code in multiple kernel streams to improve the overall computing efficiency of the GPU remains a problem that urgently needs to be solved in this field. Summary of the Invention

[0004] One of the purposes of this application is to disclose a GPU and related methods to solve the above-mentioned problems.

[0005] One embodiment of this application provides a GPU for executing multiple kernel streams, each kernel stream containing multiple kernel codes to be executed sequentially, and each kernel code including multiple thread blocks. The GPU includes multiple stream processor groups, an available hardware resource table, a resource comparator, a stream scheduler, and a global scheduler. Each stream processor group is used to execute at least one thread block. The available hardware resource table records the available hardware resources of the multiple stream processor groups. The resource comparator, based on the available hardware resource table, selects at least one first schedulable kernel code from multiple first-order kernel codes in the multiple kernel streams whose required hardware resources are less than the available hardware resources of the multiple stream processor groups. The stream scheduler selects a kernel code from the at least one first schedulable kernel code as the selected kernel code. The global scheduler allocates the multiple thread blocks in the selected kernel code to the multiple stream processor groups for execution and updates the available hardware resource table based on the hardware resource usage status of the multiple stream processor groups.

[0006] Another embodiment of this application provides a GPU method for executing multiple kernel streams, wherein each kernel stream contains multiple kernel codes to be executed sequentially, and each kernel code includes multiple thread blocks. The method includes recording available hardware resources of multiple stream processor groups in an available hardware resource table; selecting, based on the available hardware resource table, at least one first schedulable kernel code from multiple first-order kernel codes in the multiple kernel streams whose required hardware resources are less than the available hardware resources of the multiple stream processor groups; selecting a kernel code from the at least one first schedulable kernel code as the selected kernel code; allocating multiple thread blocks from the selected kernel code to the multiple stream processor groups for execution; and updating the available hardware resource table based on the hardware resource usage status of the multiple stream processor groups.

[0007] The GPU and related methods provided in the embodiments of this application can select appropriate kernel code from the first-order kernel code in each kernel stream when multiple kernel streams are received, and allocate it preferentially, thereby improving the hardware utilization and overall computing performance of the GPU. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of an embodiment of the GPU of this application.

[0009] Figure 2 It is a timing diagram that uses a polling method to schedule kernel code in the kernel stream.

[0010] Figure 3 This is a flowchart of an embodiment of the GPU method of this application.

[0011] Figure 4 It is based on Figure 3 The method is a timing diagram for scheduling kernel code in the kernel stream.

[0012] Figure 5 yes Figure 3 Flowchart of the sub-steps of the method. Detailed Implementation

[0013] The following disclosure provides various implementations or examples that can be used to achieve different features of this disclosure. Specific examples of components and configurations described below are for simplification purposes. It is understood that these descriptions are illustrative only and are not intended to limit the scope of this disclosure. For example, in the following description, forming a first feature on or over a second feature may include, in some embodiments, the first and second features being in direct contact with each other; and may also include, in some embodiments, additional components being formed between the first and second features, such that the first and second features may not be in direct contact. Furthermore, component symbols and / or reference numerals may be reused in multiple embodiments of this disclosure. Such reuse is for the purpose of brevity and clarity and does not in itself represent a relationship between the different embodiments and / or configurations discussed.

[0014] While the numerical ranges and parameters used to define the broader scope of this application are approximate values, the relevant values ​​in the specific embodiments have been presented as precisely as possible. However, any numerical value inevitably contains standard deviations due to individual test methods. Here, "approximately" generally means that the actual value is within plus or minus 10%, 5%, 1%, or 0.5% of a particular value or range. Alternatively, the term "approximately" may mean that the actual value falls within the acceptable standard error of the average, as determined by those skilled in the art to which this application pertains. It is understood that, except for experimental examples, or unless explicitly stated otherwise, all ranges, quantities, values, and percentages used herein (e.g., to describe material usage, duration, temperature, operating conditions, quantity ratios, and the like) are modified with "approximately". Therefore, unless otherwise stated, the numerical parameters disclosed in this specification and the accompanying claims are approximate values ​​and are subject to change as needed. At a minimum, these numerical parameters should be understood as the indicated significant digits and values ​​obtained by applying general rounding. In this context, a range of values ​​is expressed as a distance from one endpoint to the other or between the two endpoints; unless otherwise stated, all ranges of values ​​herein include the endpoints.

[0015] Figure 1This is a schematic diagram of an embodiment of the Graphics Processing Unit (GPU) 100 of this application. The GPU 100 includes multiple stream processor groups 110, an available hardware resource table 120, a resource comparator 130, a stream scheduler 140, and a global scheduler 150. In this embodiment, the GPU 100 can be applied in computer systems, such as data centers, cloud servers, smartphones, laptops, desktop computers, etc., and can be used to perform computations desired by the computer system. For example, when the Central Processing Unit (CPU) of a computer system wants to use the GPU to perform computations, it wraps the required operations into kernel code and can place independent kernel code into different kernel streams so that the GPU can execute multiple kernel codes in parallel. The GPU 100 can then schedule and schedule the kernel code in multiple kernel streams.

[0016] For example, such as Figure 1 As shown, GPU 100 can receive three kernel streams ST1, ST2 and ST3, and each kernel stream ST1 to ST3 contains multiple kernel codes. Kernel stream ST1 contains kernel codes KA1, KA2, KA3 and KA4, kernel stream ST2 may contain kernel codes KB1 and KB2, and kernel stream ST3 may contain kernel codes KC1 and KC2.

[0017] In this embodiment, the kernel code in each kernel stream needs to be executed sequentially. For example, kernel code KA2 in kernel stream ST1 needs to wait for kernel code KA1 to be scheduled before it can be scheduled, and kernel code KA3 needs to wait for kernel code KA2 to be scheduled before it can be scheduled, and so on.

[0018] Furthermore, in this embodiment, different kernel codes may require different hardware resources depending on the differences in their operational content. For example, kernel codes KA1 and KA2 in kernel stream ST1, kernel code KB1 in kernel stream ST2, and kernel code KC1 in kernel stream ST3 may belong to the first type of kernel code (in... Figure 1 (represented by white squares), and the required hardware resources can be, for example, 70% of the available hardware resources of multiple stream processors 110 as first-type hardware resources and 30% as second-type hardware resources. Conversely, the kernel codes KA3 and KA4 in kernel stream ST1, KB2 in kernel stream ST2, and KC2 in kernel stream ST3 can be second-type kernel codes (in...). Figure 1(represented by shading squares in the text), and the required hardware resources may be, for example, 20% of the available hardware resources of the multiple stream processors 1101 of a first type and 60% of the second type of hardware resources. In some embodiments, the first type of hardware resources may be, for example, a register file shared by the multiple stream processors 110, and the second type of hardware resources may be, for example, memory shared by the multiple stream processors 110.

[0019] In this scenario, since the first type of kernel code and the second type of kernel code require different hardware resources, and their resource requirements are complementary (i.e., one type of kernel code has a higher demand for the first type of hardware resources, while the other has a lower demand, and the combined demand for the first type of hardware resources does not exceed the total amount of hardware resources available to the GPU), ideally, the GPU should be able to execute both types of kernel code simultaneously. However, when simply using a round-robin method to schedule the kernel code in kernel streams ST1 to ST3, situations may frequently arise where the GPU's hardware resources are insufficient to execute the assigned kernel code, forcing it to wait, resulting in low overall GPU hardware utilization.

[0020] Figure 2 This is a timing diagram showing the scheduling of kernel code in kernel streams ST1 to ST3 using a round-robin method. For example... Figure 2 As shown, the GPU scheduler can sequentially visit kernel streams ST1, ST2, and ST3 to execute the kernel code in each stream. For example, the scheduler will first visit kernel stream ST1. At this point, since no kernel code has been scheduled, the GPU's hardware resources are sufficient to execute the first kernel code KA1 in kernel stream ST1. Therefore, the scheduler can allocate kernel code KA1 to one of its stream processor groups for execution. Next, according to the round-robin rule, the scheduler will visit kernel stream ST2. Since kernel code KA1 has already occupied 70% of the GPU's first-type hardware resources and 30% of its second-type hardware resources, there are not enough hardware resources to execute the first kernel code KB1 in kernel stream ST2. In this case, the GPU will wait for kernel code KA1 to finish executing before it can continue executing kernel code KB1.

[0021] Similarly, after scheduling kernel code KB1 in kernel stream ST2, the scheduler will visit kernel stream ST3. Since kernel code KB1 has already occupied 70% of the GPU's Type I hardware resources and 30% of its Type II hardware resources, there are not enough hardware resources to execute the first kernel code KC1 in kernel stream ST3. In this situation, the GPU will wait for kernel code KB1 to finish executing before continuing to execute kernel code KC1.

[0022] like Figure 2 As shown, although the GPU has enough hardware resources to process two different types of kernel code at the same time, when scheduling kernel streams in a polling manner, the GPU can only execute kernel code KA2 of kernel stream ST1 and kernel code KB2 of kernel stream ST2 simultaneously in time period P1. In other time periods, the GPU can only execute a single kernel code, resulting in low hardware utilization of the GPU and poor overall computing performance.

[0023] To improve the hardware utilization and computing performance of the GPU, the GPU 100 of this application can schedule multiple kernel codes in kernel streams ST1 to ST3 in a more flexible manner through the available hardware resource table 120, resource comparator 130, stream scheduler 140 and global scheduler 150.

[0024] Figure 3 This is a flowchart of an embodiment of the GPU method 200 of this application. In this embodiment, the GPU 100 can execute method 200, and method 200 may include steps S210 to S250. Figure 4 It is a timing diagram for scheduling kernel code in kernel streams ST1 to ST3 according to method 200.

[0025] In step S210, the GPU 100 can record the available hardware resources of the multiple stream processor groups 110 in the available hardware resource table 120. For example, the available hardware resources of each stream processor group 110 may include multiple stream processors, a register file, and at least one memory. In this embodiment, the register file and memory can be used to store data or instructions required when the stream processor executes a thread, or to temporarily store data generated during computation. However, this application is not limited thereto; in some other embodiments, the available hardware resources of the stream processor group 110 may also include other types of hardware, such as cache.

[0026] In step S220, the resource comparator 130 can select, based on the available hardware resource table 120, at least one schedulable kernel code whose required hardware resources are less than the available hardware resources of the multiple stream processor groups 110 from multiple first-order kernel codes KA1, KB1, and KC1 in multiple kernel streams ST1 to ST3. In this embodiment, since in Figure 4 Before time period P1, the GPU has not executed any kernel code. Therefore, in step S220, the available hardware resources of the stream processor group 110 should be sufficient to execute any one of the kernel codes KA1, KB1, and KC1. In this case, the resource comparator 130 can select kernel codes KA1, KB1, and KC1 as schedulable kernel codes. That is, according to method 200, the GPU 100 can simultaneously consider scheduling kernel codes KA1, KB1, and KC1 in kernel streams ST1, ST2, and ST3, without having to schedule the kernel codes in kernel streams ST1, ST2, and ST3 in turn according to the polling order.

[0027] Next, in step S230, the stream scheduler 140 can select a kernel code from the schedulable kernel codes selected in step S220 as the selected kernel code, and in step S240, the global scheduler 150 can assign the selected kernel code selected in step S230 to multiple stream processor groups 110 for execution.

[0028] In this embodiment, in order to optimize the hardware utilization and computing performance of GPU 100, GPU 100 can select the kernel code that should be executed first as the selected kernel code based on the waiting time of currently schedulable kernel codes KA1, KB1 and KC1, the required hardware resources and / or the user's preset priority level for each kernel stream.

[0029] Figure 5 This is a flowchart of the sub-steps of step S230 in this application. For example... Figure 5 As shown, step S230 may include sub-steps S231 to S237. In sub-step S231, the flow scheduler 140 may determine whether the waiting time of the schedulable kernel code with the longest waiting time among the multiple schedulable kernel codes, such as KA1, KB1, and KC1, has exceeded a predetermined threshold. If the waiting time of the schedulable kernel code with the longest waiting time has exceeded the predetermined threshold, then to avoid excessive delay of subsequent kernel codes due to the excessively long waiting time of the schedulable kernel code, the flow scheduler 140 may correspondingly execute sub-step S232 to preferentially select the schedulable kernel code with the longest waiting time as the selected kernel code.

[0030] However, if in sub-step S231 the waiting time of the schedulable kernel code with the longest waiting time is also less than the predetermined threshold, it may indicate that there is no kernel code that needs to be executed immediately. In this case, the flow scheduler 140 can further select the most appropriate kernel code as the selected kernel code based on the hardware resources required by each schedulable kernel code and the priority level preset by the user.

[0031] For example, the resource comparator 130 can also select, based on the available hardware resource table 120, the kernel code whose required hardware resources best match the available hardware resources of the multiple stream processor groups 110 from among the multiple schedulable kernel codes KA1, KB1, and KC1 as a candidate kernel code. In this embodiment, the kernel code whose required hardware resources best match the available hardware resources of the stream processor group 110 refers to the kernel code that can make the most complete use of the remaining hardware resources. For example, if the available hardware resources of the multiple stream processor groups 110 are currently 30% of the first type of hardware resources and 60% of the second type of hardware resources, the resource comparator 130 can preferentially select the schedulable kernel code with higher requirements for the first type of hardware resources and lower requirements for the second type of hardware resources as the selected kernel code. However, this application is not limited to this. In some other embodiments, the resource comparator 130 can also select the kernel code with the fewest required hardware resources from among the multiple first-priority kernel codes as a candidate kernel code based on the available hardware resource table 120. In this way, even when most kernel code requires relatively low hardware resources, the probability of GPU 100 executing multiple kernel codes simultaneously can be effectively increased, thereby improving the hardware utilization of GPU 100.

[0032] If the resource comparator 130 selects candidate kernel code, the flow scheduler 140 will preferentially select the candidate kernel code suggested by the resource comparator 130 as the selected kernel code in step S234. However, in some cases, the resource comparator 130 may not be able to select candidate kernel code based on hardware resource requirements. For example, if the GPU has not yet been used to execute any kernel code, or if all schedulable kernel codes require the same amount of hardware resources, the resource comparator 130 may have difficulty selecting the most suitable candidate kernel code. In this case, the flow scheduler 140 may further execute sub-step S235 to make a decision based on the priority level and waiting time of the kernel flow in which the kernel code belongs.

[0033] In sub-step S235, the flow scheduler 140 can determine whether there is schedulable kernel code belonging to a kernel flow with a preset high priority level. In this embodiment, the GPU 100 allows the user to set the priority level of each kernel flow according to the criticality of each kernel flow when generating kernel flows using the central processing unit, so that the GPU can prioritize processing kernel flows with higher priority levels. Therefore, when the flow scheduler 140 determines that there is schedulable kernel code belonging to a kernel flow with a preset high priority level, it can execute sub-step S236 to select the schedulable kernel code belonging to the kernel flow with the preset high priority level and the longest waiting time as the selected kernel code. For example, if kernel flow ST1 is preset to have a high priority level, while kernel flows ST2 and ST3 are not set to have a high priority level, then the flow scheduler 140 will select the schedulable kernel code KA1 belonging to kernel flow ST1 as the selected kernel code in step S236.

[0034] However, if no schedulable kernel code belongs to a kernel stream with a high priority level, for example, if kernel streams ST1, ST2 and ST3 are not set to a high priority level, the stream scheduler 140 will execute step S237 to select the schedulable kernel code with the longest waiting time as the selected kernel code.

[0035] In this embodiment, the stream scheduler 140 may select kernel code KA1 as the selected kernel code in step S230, and the global scheduler 150 may allocate kernel code KA1 to each stream processor group 110 in step S240. Generally, each kernel code may include multiple thread blocks, so when the global scheduler 150 performs scheduling, it can allocate multiple thread blocks to each stream processor group 110 in units of thread blocks, and each stream processor group 110 can execute at least one thread block.

[0036] After kernel code KA1 is scheduled to each stream processor group 110, each stream processor 110 will occupy the originally available hardware resources during the execution of the corresponding thread block. Therefore, in step S250, the global scheduler 150 can update the available hardware resource table 120 based on the available hardware resources of each stream processor 110 after it is scheduled to execute kernel code KA1, so as to serve as the basis for the next scheduling of kernel code.

[0037] In some embodiments, when the central processing unit generates kernel streams, it can establish a hardware resource table for each kernel code to record the hardware resources required by each thread block in each kernel code. In this way, the global scheduler 150 can predict the hardware resources that each kernel code will occupy when scheduling each kernel code, and thus can update the available hardware resource table 120 accordingly. Furthermore, in step S220, the resource comparator 130 can also determine, based on the available hardware resource table 120 and the hardware resource tables in each first-order kernel code KA1, KB1, and KC1, whether the hardware resources required by the first-order kernel codes KA1, KB1, and KC1 are less than the available hardware resources of the stream processor group 110.

[0038] After kernel code KA1 is assigned to each stream processor group 110 for execution, GPU 100 will continue to execute step S260 to determine whether there are any unassigned schedulable codes to continue scheduling. In this embodiment, although there are still unassigned kernel codes in kernel streams ST1, ST2, and ST3, while kernel code KA1 has not finished executing, the required hardware resources of the first-priority kernel codes KA2, KB1, and KC1 in kernel streams ST1, ST2, and ST3 are greater than the available hardware resources of the current stream processor group 110. Therefore, GPU 100 needs to wait for kernel code KA1 to finish executing and release the hardware resources before executing step S230 again to select the next assigned kernel code from the schedulable kernel codes KA2, KB1, and KC1 in kernel streams ST1, ST2, and ST3.

[0039] like Figure 4 As shown, the stream scheduler 140 can select kernel code KB1 as the selected kernel code in time period P2 and allocate kernel code KB1 to multiple stream processor groups 110. In this case, 70% of the available hardware resources of the multiple stream processor groups 110 will be occupied by the first type of hardware resources and 30% by the second type of hardware resources, so the available hardware resources are 30% of the first type of hardware resources and 70% by the second type of hardware resources. In this case, although the required hardware resources of kernel codes KA2 and KC1, which are currently in the first order in kernel streams ST1 and ST3, will be greater than the available resources of stream processor group 110, kernel code KB2, which was originally following kernel code KB1 in kernel stream ST2, will have the first order in kernel stream ST2, and its required hardware resources are: 20% of the first type of hardware resources and 60% of the second type of hardware resources. That is, while multiple stream processor groups 110 are executing kernel code KB1, multiple stream processor groups 110 still have enough hardware resources to execute kernel code KB2. Therefore, in Figure 4In this process, the stream scheduler 140 may also select kernel code KB2 as a selected kernel code during the execution period P2 of kernel code KB1 and assign it to multiple stream processor groups 110 for execution.

[0040] In other words, after the global scheduler 150 assigns the selected kernel code KB1 to multiple stream processor groups 110 for execution, the resource comparator 130, in addition to determining whether the required hardware resources of the kernel codes KA2 and KC1, which were originally in the first priority and had not yet been assigned, are less than the available hardware resources of the stream processor group 110, will also determine whether the required hardware resources of the kernel code KB2, which is in the same kernel stream ST2 as kernel code KB1 and has replaced kernel code KB1 as the first priority, are less than the available hardware resources of the stream processor group 110, thereby selecting the schedulable kernel code.

[0041] like Figure 4 As shown, after kernel code KB2 is allocated, the stream scheduler 140 needs to wait for kernel code KB1 to finish executing before it has sufficient hardware resources to execute kernel code KC1 or KA2. During period P3, after kernel code KB1 finishes execution, since the waiting time for kernel code KC1 is longer than that for kernel code KA2, the stream scheduler 140 can preferentially select kernel code KC1 as the selected kernel code and allocate it to multiple stream processor groups 110 via the global scheduler 150. Then, after kernel code KB2 finishes execution, since the available hardware resources of stream processor group 110 will be sufficient to execute kernel code KC2, the stream scheduler 140 will select kernel code KC2 as the selected kernel code and allocate it to multiple stream processor groups 110 via the global scheduler 150. In other words, during period P3, the GPU 100 will have a portion of the time available to execute both kernel code KC1 and kernel code KC2 simultaneously.

[0042] Because method 200 selects appropriate kernel code from the first-order kernel code in kernel streams ST1, ST2, and ST3 for priority allocation, GPU 100 has a higher probability of executing multiple kernel codes simultaneously, such as... Figure 4 As shown. For example, the stream scheduler 140 can select the kernel code that requires the fewest hardware resources, best matches the required hardware resources with the available hardware resources, or has the longest waiting time as the selected kernel code. In this way, the hardware utilization and overall computing performance of the GPU 100 can be improved.

[0043] Furthermore, this application does not limit the multiple stream processor groups 110 in GPU 100 to executing only two thread blocks in kernel code at a time. In some embodiments, if some kernel code requires fewer hardware resources, the multiple stream processor groups 110 in GPU 100 may execute more than two kernel codes at a time.

[0044] In summary, the GPU and related methods provided in the embodiments of this application can select appropriate kernel code from the first-order kernel code in each kernel stream when multiple kernel streams are received, and allocate it first. For example, the kernel code that requires the least hardware resources, the most matching between required hardware resources and available hardware resources, or the longest waiting time can be selected as the selected kernel code. Therefore, the hardware utilization rate and overall computing performance of the GPU can be improved.

[0045] The foregoing description briefly outlines the features of certain embodiments of this application, enabling those skilled in the art to more fully understand the various forms of this disclosure. Those skilled in the art will readily recognize that this disclosure serves as a basis for designing or modifying other processes and structures to achieve the same objectives and / or advantages as the embodiments described herein. Those skilled in the art should understand that these equivalent embodiments remain within the spirit and scope of this disclosure, and various changes, substitutions, and modifications can be made without departing from the spirit and scope of this disclosure.

Claims

1. A GPU for executing multiple kernel streams, each kernel stream containing multiple kernel code segments to be executed sequentially, each kernel code segment comprising multiple thread blocks, characterized in that, The GPU includes: Multiple stream processor groups, each stream processor group is used to execute at least one thread block; A hardware resource table is used to record the available hardware resources of the multiple stream processor groups; A resource comparator is used to select, based on the available hardware resource table, at least one first schedulable kernel code from a plurality of first-order kernel codes in the plurality of kernel streams whose required hardware resources are less than the available hardware resources of the plurality of stream processor groups; A stream scheduler is configured to select a kernel code as the selected kernel code from the at least one first schedulable kernel code; and A global scheduler is used to allocate multiple thread blocks in the selected kernel code to the multiple stream processor groups for execution, and to update the available hardware resource table based on the hardware resource usage status of the multiple stream processor groups.

2. The GPU as described in claim 1, characterized in that: Each kernel code also includes a hardware resource requirement table, which records the hardware resources required for each thread block within that kernel code; and The resource comparator determines whether the required hardware resources of the multiple first-order kernel codes are less than the available hardware resources of the multiple stream processor groups based on the available hardware resource table and the multiple required hardware resource tables of the multiple first-order kernel codes.

3. The GPU as described in claim 1, characterized in that... When the waiting time of the first schedulable kernel code with the longest waiting time among the at least one first schedulable kernel code is greater than a predetermined threshold, the flow scheduler preferentially selects the first schedulable kernel code with the longest waiting time as the selected kernel code.

4. The GPU as described in claim 1, characterized in that... The resource comparator is further used to select, based on the available hardware resource table, the kernel code from the at least one first schedulable kernel code that best matches the required hardware resources with the available hardware resources of the plurality of stream processor groups as a candidate kernel code.

5. The GPU as described in claim 1, characterized in that... The resource comparator is further used to select, based on the available hardware resource table, the kernel code that requires the least amount of hardware resources from the at least one first schedulable kernel code as a candidate kernel code.

6. The GPU as described in claim 4 or 5, characterized in that... When the waiting time of the longest schedulable kernel code among the at least one first schedulable kernel code is less than a predetermined threshold, the flow scheduler preferentially selects the candidate kernel code as the selected kernel code.

7. The GPU as described in claim 1, characterized in that... When the waiting time of the longest schedulable kernel code among the at least one first schedulable kernel code is less than a predetermined threshold, the flow scheduler preferentially selects the first schedulable kernel code that belongs to the kernel flow with a preset high priority level and has the longest waiting time as the selected kernel code.

8. The GPU as described in claim 1, characterized in that, During at least a period of time, the plurality of stream processor groups are multiple thread blocks that process at the same time from at least two kernel code.

9. The GPU as described in claim 1, characterized in that, After the global scheduler assigns the selected kernel code to the plurality of stream processor groups for execution, the resource comparator, based on the updated available hardware resource table, selects at least one second schedulable kernel code from the plurality of kernel codes that have not yet been assigned among the plurality of first-priority kernel codes and from the kernel stream to which the selected kernel code originally belonged, and from the kernel codes following the selected kernel code, the second schedulable kernel code whose required hardware resources are less than the available hardware resources of the plurality of stream processor groups.

10. The GPU as claimed in claim 1, characterized in that, The hardware resources of each stream processor group include multiple stream processors, a register file, and at least one memory, wherein the register file and at least one memory are used to store the data or instructions required when the multiple stream processors execute threads.

11. A GPU method for executing multiple kernel streams, each kernel stream containing multiple kernel code segments to be executed sequentially, each kernel code segment comprising multiple thread blocks, characterized in that, The method includes: Record the available hardware resources of multiple stream processor groups in the available hardware resource table; Based on the available hardware resource table, at least one first schedulable kernel code is selected from the multiple first-order kernel codes in the multiple kernel streams whose required hardware resources are less than the available hardware resources of the multiple stream processor groups; A kernel code is selected from the at least one first schedulable kernel code as the selected kernel code; Assigning multiple thread blocks in the selected kernel code to the multiple stream processor groups for execution; and The available hardware resource table is updated based on the hardware resource usage status of the multiple stream processor groups.

12. The method as described in claim 11, characterized in that, Also includes: When generating the multiple kernel streams, the required hardware resources of each thread block in each kernel code are recorded in the required hardware resource table of each kernel code. and Based on the available hardware resource table and the multiple required hardware resource tables of the multiple first-priority kernel codes, it is determined whether the required hardware resources of the multiple first-priority kernel codes are less than the available hardware resources of the multiple stream processor groups.

13. The method as described in claim 11, characterized in that... The step of selecting a kernel code from the at least one first schedulable kernel code as the selected kernel code includes: When the waiting time of the first schedulable kernel code with the longest waiting time among the at least one first schedulable kernel code is greater than a predetermined threshold, the first schedulable kernel code with the longest waiting time is preferentially selected as the selected kernel code.

14. The method as described in claim 11, characterized in that, Also includes: Based on the available hardware resource table, the kernel code that best matches the required hardware resources of the multiple first-priority kernel codes is selected as the candidate kernel code.

15. The method as described in claim 11, characterized in that, Also includes: Based on the available hardware resource table, the kernel code that requires the least hardware resources is selected from the plurality of first-priority kernel codes as the candidate kernel code.

16. The method as described in claim 14 or 15, characterized in that... The step of selecting a kernel code from the at least one first schedulable kernel code as the selected kernel code includes: When the waiting time of the first schedulable kernel code with the longest waiting time among the at least one first schedulable kernel code is less than a predetermined threshold, the candidate kernel code is preferentially selected as the selected kernel code.

17. The method as described in claim 11, characterized in that... The step of selecting a kernel code from the at least one first schedulable kernel code as the selected kernel code includes: When the waiting time of the longest schedulable kernel code among the at least one first schedulable kernel code is less than a predetermined threshold, the first schedulable kernel code that belongs to the kernel stream with a preset high priority level and has the longest waiting time is preferentially selected as the selected kernel code.

18. The method as described in claim 11, characterized in that, During at least one period of time, the plurality of stream processor groups process at least two thread blocks simultaneously, and the at least two thread blocks are from at least two different kernel codes.

19. The method as described in claim 11, characterized in that, Also includes: After the selected kernel code is assigned to the plurality of stream processor groups for execution, based on the updated available hardware resource table, at least one second schedulable kernel code is selected from the plurality of kernel codes that have not yet been assigned among the plurality of first-priority kernel codes and from the kernel stream to which the selected kernel code originally belonged, and from the kernel codes following the selected kernel code, the required hardware resources are less than the available hardware resources of the plurality of stream processor groups.

20. The method as described in claim 11, characterized in that, The hardware resources of each stream processor group include multiple stream processors, a register file, and at least one memory, wherein the register file and at least one memory are used to store the data or instructions required when the multiple stream processors execute threads.

Images