GPU scheduling optimization method and device for small kernel

Abstract

The invention relates to the technical field of graphics processing units, and provides a GPU scheduling optimization method and device for a small kernel. The method comprises the following steps: for a plurality of kernel groups, inserting a restart command among the kernel groups without a dependency relationship to obtain an execution sequence of the plurality of kernel groups; according to the execution sequence, serializing the plurality of kernel groups into a command stream executable by hardware, and writing the command stream into a computing cache; and analyzing the command stream in the computing cache, and when the restart command is detected, directly reading the kernel group behind the restart command so as to execute the read kernel group, thereby solving the problem of serious GPU performance loss caused by overlarge proportion of delay of a scheduler and time consumed by kernel start in total time required for processing an application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of graphics processors, and particularly relates to a GPU scheduling optimization method and device for small kernels. BACKGROUND

[0002] A kernel in a graphics processor (GPU) and general-purpose computing on graphics processing units (GPGPU) refers to a core for calculation, which is a set of codes; generally, one kernel is used to process a large amount of data, and the data is scheduled to each calculation core of the GPU or GPGPU for processing. Hereinafter, the execution of a kernel by a GPU is taken as an example for description, and the execution of a kernel by a GPGPU is the same, which will not be described hereinafter.

[0003] At present, the scale of a high-performance GPU is large, and the capacity of a double data rate synchronous dynamic random-access memory (DDR) or high bandwidth memory (HBM) required by the GPU is also very large, resulting in a very large delay of the GPU in accessing the DDR or HBM. Hereinafter, the access to the DDR is taken as an example for description, and the access to the HBM is the same, which will not be described hereinafter.

[0004] A kernel is often written into the DDR of the GPU by a host device (for example, a central processing unit); when the GPU performs calculation, the corresponding kernel is read from the DDR. After the execution of each kernel is completed, the scheduler is informed, and then the scheduler is processed. After the processing is completed, the scheduler informs the GPU to execute the next kernel.

[0005] However, in some current applications (for example, artificial intelligence large model calculation), a large number of very small kernels need to be executed; and in previous applications (for example, image rendering using a GPU), a very small number of large kernels usually need to be executed. The running time of each small kernel is very short relative to the running time of a large kernel; and the number of times of scheduling by the scheduler increases dramatically when a very large number of small kernels are executed relative to the execution of a very small number of large kernels. Since the scheduler also needs to access the DDR to obtain relevant information during processing, the delay of the scheduler when scheduling the next kernel is very large. In this way, the GPU has a large amount of time for waiting for scheduling by the scheduler when executing tasks, and each kernel also needs to read the corresponding kernel from the DDR when starting, during which time the GPU cannot perform calculations. Therefore, the delay of the scheduler and the time consumed by the kernel to start account for a large proportion of the total time required to process the application, resulting in a long period of idle state of the GPU calculation resources, serious loss of GPU performance, and poor practicability.

[0006] Therefore, it is urgent to overcome the defects of the prior art in the technical field. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a GPU scheduling optimization method and device for small kernels, which aims to use a restart command flag to directly execute a kernel group, and when a restart command is detected, directly read and execute the kernel group located after the restart command, so that the next kernel group can be directly read after the execution of a kernel group is completed, without waiting for the scheduler to schedule the next kernel again or starting the next kernel, thereby solving the problem that the delay of the scheduler and the time consumed by the kernel to start account for a large proportion of the total time required to process the application, resulting in a serious loss of GPU performance.

[0008] The present application adopts the following technical solutions: In a first aspect, the present application provides a GPU scheduling optimization method for small kernels, comprising: For a plurality of kernel groups, a restart command is inserted between kernel groups without a dependency relationship to obtain an execution order of the plurality of kernel groups; According to the execution order, the plurality of kernel groups are serialized into a hardware executable command stream, and the command stream is written into a calculation cache; resolving the command stream in the computing cache, when the restart command is detected, directly reading a kernel group located after the restart command to execute the read kernel group.

[0009] Further, the inserting a restart command between kernel groups without a dependency relationship to obtain the execution sequence of the plurality of kernel groups comprises: obtaining at least one barrier between the plurality of kernel groups; obtaining two kernel groups located before and after the barrier, and taking a dependency relationship between the two kernel groups located before and after the barrier as a to-be-judged dependency relationship; performing necessity detection on the to-be-judged dependency relationship to generate the execution sequence.

[0010] Further, the necessity detection on the to-be-judged dependency relationship to generate the execution sequence comprises: selectively inserting a restart command or an end command between the corresponding kernel groups according to whether the to-be-judged dependency relationship passes the necessity detection.

[0011] Further, each of the kernel groups comprises at least one kernel function. The selectively inserting a restart command or an end command between the corresponding kernel groups according to whether the to-be-judged dependency relationship passes the necessity detection comprises: when the to-be-judged dependency relationship passes the necessity detection, obtaining at least two kernel groups of the to-be-judged dependency relationship, taking a kernel group before the dependency relationship as a first kernel group and a kernel group after the dependency relationship as a second kernel group, and inserting an end command between a last kernel function of the first kernel group and a first kernel function of the second kernel group; when the to-be-judged dependency relationship does not pass the necessity detection, obtaining at least two kernel groups of the to-be-judged dependency relationship, taking a first obtained kernel group as a third kernel group and a second obtained kernel group as a fourth kernel group, and inserting a restart command between a last kernel function of the third kernel group and a first kernel function of the fourth kernel group.

[0012] Further, the inserting a restart command between kernel groups without a dependency relationship to obtain the execution sequence of the plurality of kernel groups further comprises: For the plurality of kernel groups, a restart command is inserted between kernel groups without a barrier.

[0013] Further, the parsing the command stream in the compute cache, when the restart command is detected, directly reading a kernel group after the restart command to execute the read kernel group includes: starting to parse the command stream, determining a first first pending command of a first kernel group according to a scheduler, to execute the first kernel group from the first first pending command; continuing to parse the command stream, when a restart command is detected, sequentially reading a next address of the restart command, obtaining a first second pending command of a next kernel group according to the next address, to directly read a kernel group after the restart command; starting to execute the next kernel group from the first second pending command, to execute all second pending commands in the next kernel group to execute the read kernel group.

[0014] Further, the starting to parse the command stream, determining a first first pending command of a first kernel group according to a scheduler includes: the scheduler transmitting a base address of the first kernel group and a read pointer position of the compute cache to a command processor; the command processor starting to parse the command stream according to the base address and the read pointer position to obtain an address of the first kernel group; the command processor loading the first first pending command from the address of the first kernel group to an instruction cache according to the address of the first kernel group; the command processor configuring a compute resource and a thread bundle for the first first pending command, using the compute resource and the thread bundle to execute the first first pending command in the instruction cache, to execute all first pending commands in the first kernel group from the first first pending command;

[0015] Further, after the starting to parse the command stream, determining a first first pending command of a first kernel group according to a scheduler, further includes: when the command processor detects an end command, sending a kernel end signal to the scheduler; the scheduler receiving the end signal, accessing a memory unit to obtain a current read pointer position; The command processor acquires a first first pending command of the next kernel group according to the address of the end signal and the current read pointer position, and loads the first first pending command from the video memory to the instruction cache; The command processor configures a computing resource and a thread bundle for the first first pending command, and executes the first first pending command in the instruction cache using the computing resource and the thread bundle, so as to execute all the first pending commands in the next kernel group starting from the first first pending command.

[0016] In a second aspect, the present application provides a GPU scheduling optimization device for small kernels, comprising: At least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the GPU scheduling optimization method for small kernels in the first aspect.

[0017] In a third aspect, the present application provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the GPU scheduling optimization method for small kernels in the first aspect.

[0018] In a fourth aspect, a computer program product containing instructions is provided, which makes the computer or processor execute the GPU scheduling optimization method for small kernels in the first aspect when the instructions are run on the computer or processor.

[0019] In a fifth aspect, the present application provides a GPU scheduling optimization system for small kernels, which comprises the GPU scheduling optimization device for small kernels in the second aspect and uses the GPU scheduling optimization method for small kernels in the first aspect to complete the interaction of the GPU scheduling optimization device for small kernels in the second aspect.

[0020] Compared with the prior art, the present application has at least the following beneficial effects: The application inserts a restart command between kernel groups without dependency, uses the restart command to mark the kernel group that can be executed directly, obtains the execution order of multiple kernel groups, serializes the multiple kernel groups into a hardware executable command stream according to the execution order, and writes the command stream into a computing cache. The command stream is parsed, and when a restart command is detected, the kernel group after the restart command is directly read and processed. Compared with the prior art, after one kernel group is processed, the next kernel group is directly read, without waiting for the scheduler to schedule the next kernel again, without starting the next kernel, without accessing the DDR, and avoiding the delay caused by scheduling and starting. Therefore, during the process of parsing the command stream, one kernel group is processed and then the next kernel group is processed. In the calculation scene of a large number of small kernels, the time from the end of processing of one small kernel group to the start of processing of the next small kernel is reduced from thousands of clock cycles to 0. Since there is no dependency between most small kernels, the restart command can be used to mark in most cases, that is, the delay caused by scheduling and starting can be skipped, the performance of the GPU can be greatly improved, and the practicability is high. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments of the application. Obviously, the drawings described below are only some of the embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0022] Figure 1 is a specific example diagram of a data structure of a computing cache in the prior art provided by the embodiments of the application; Figure 2 is a flowchart of a GPU scheduling optimization method for small kernels provided by the embodiments of the application; Figure 3 is a flowchart of step 10 provided by the embodiments of the application; Figure 4 is a flowchart of step 102 provided by the embodiments of the application; Figure 5 is a specific example diagram of a data structure of a computing cache of the embodiments of the application provided by the embodiments of the application; Figure 6 is a specific example diagram of a scheduling system provided by the embodiments of the application; Figure 7 is a flowchart of step 30 provided by the embodiments of the application; Figure 8 This is a flowchart illustrating step 301 provided in an embodiment of the present invention; Figure 9 This is a flowchart illustrating a process for handling an end command, provided by an embodiment of the present invention. Figure 10 This is a schematic diagram illustrating a specific example of the computation time of a prior art provided by an embodiment of the present invention; Figure 11 This is a schematic diagram illustrating a specific example of the computation time of an embodiment of the present invention. Figure 12 This is a schematic diagram of the architecture of a GPU scheduling optimization device for small kernels provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0025] In the description of this invention, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this disclosure.

[0026] In the description of the present application, the terms "first", "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined as "first", "second" can be explicitly or implicitly included one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise stated, the meaning of "a plurality of" is two or more. In addition, for example, in the description, the same type of nouns can also be described as two independent individuals by adding "A", "B" at the end, in which case the features defined as "A", "B" are only used for the purpose of distinguishing the same type of individual description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated.

[0027] In describing some embodiments, "coupled", "coupled", and "connected" and their derivatives can be used. For example, the term "connected" can be used to describe some embodiments to indicate that two or more components have direct physical or electrical contact with each other. For another example, the term "coupled" can be used to describe some embodiments to indicate that two or more components have direct physical or electrical contact. However, the term "connected" or "coupled" can also refer to two or more components that do not have direct contact with each other, but still cooperate or interact with each other, such as "optical coupling", "wireless connection", etc. The embodiments disclosed herein are not necessarily limited to the content of the present application.

[0028] In the description of the present application, the expression "A and / or B" (where A and B are used to represent specific feature content) includes the following three combinations: only A, only B, and a combination of A and B.

[0029] In the present application, "about", "approximately" or "approximately" includes the value stated and the average value within the acceptable deviation range of the specific value, wherein the acceptable deviation range is determined by the person skilled in the art considering the measurement being discussed and the error related to the measurement of the specific quantity (i.e. the limitation of the measurement system).

[0030] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0031] In order to further illustrate the reason for the serious loss of GPU performance, first explain the concept of kernel: A kernel is a logical concept of a block of code that executes in parallel on a GPU, while a kernel function is a function that specifically implements this block of code; that is, a kernel function defines the operations for each thread or thread block, and the kernel function is the concrete implementation of the kernel. A kernel group refers to a logically related set of kernel functions that execute in parallel on the GPU; a kernel function contains multiple commands to be processed.

[0032] The following is a further explanation of the reasons for the severe performance loss in GPUs: Compared to the past when GPUs were typically used to process a very small number of large kernels, as each kernel becomes smaller and the number of kernels increases, processing a large number of small kernels requires the GPU to access DDR and read the corresponding small kernels from DDR before starting them. After each small kernel is processed, the scheduler needs to access DDR again to schedule the next kernel. Because the latency of GPU accessing DDR is relatively large, the time spent starting small kernels and the scheduler latency are both significant. Consequently, the time spent starting a large number of small kernels and the scheduler latency account for an increasingly large proportion of the total time required to process the application. Since the GPU cannot perform computations during kernel startup and scheduler latency, its computing resources remain idle for extended periods.

[0033] Specifically, such as Figure 1 The image shows a specific example of a circular buffer data structure. An application consists of multiple kernels, which are stored in a circular buffer located in DDR.

[0034] When processing this application, when the command processor starts launching a kernel group, the scheduler first accesses the DDR to obtain the base address of the kernel group being launched in DDR, as well as the read pointer of the compute cache. It then informs the GPU's command processor (CP) of the obtained base address and read pointer. The command processor uses the base address and read pointer to retrieve the first pending command for the newly launched kernel group. The command processor then continuously parses the pending commands in the compute cache until it encounters an end command, at which point it informs the scheduler that processing of the current kernel group has ended. The scheduler then launches the next kernel group, and this process repeats.

[0035] In this flow, there are two latencies: one is the latency of kernel start, that is, the command processor needs to first obtain the read pointer through the scheduler, and the scheduler needs to access the DDR to obtain the read pointer; the other is the latency of the scheduler, that is, after processing a kernel group, the scheduler needs to re-access the DDR to obtain the current read pointer, and then re-schedule the next kernel group.

[0036] The access latency of the DDR is usually between 100 and 200 clock cycles; and with the continuous development of demand and technology, the scale of high-performance GPUs is getting larger and larger, and when data is transmitted through the Network on Chip (NoC), a latency of 100 to 200 clock cycles is also generated. As a result, the total latency of the two latencies is 300 to 400 clock cycles.

[0037] In previous applications (for example, using a GPU to complete a traditional image rendering task), a large kernel processing usually takes tens of thousands of clock cycles, so the latency of 300 to 400 clock cycles has little effect on the performance of the GPU. In the calculation scene of a large number of small kernels, the processing of a small kernel usually only takes 100 to 200 clock cycles, or even only takes within 100 clock cycles; and the total latency of the pre-step of dispatching and starting the small kernel is 300 to 400 clock cycles, so the time consumption of dispatching and starting accounts for a large proportion of the total time required by the processing application, resulting in serious loss of GPU performance and poor practicability.

[0038] To solve the above problems, as shown in Figure 2 The embodiment of the present application provides a GPU scheduling optimization method for small kernels, which comprises the following steps: Step 10: For a plurality of kernel groups, a restart command is inserted between kernel groups without a dependency relationship, to obtain an execution sequence of the plurality of kernel groups.

[0039] Each of the kernel groups comprises at least one kernel function; the kernel functions in each kernel group, and the to-be-processed commands in the kernel functions, can have a mutual dependency relationship, that is, after some to-be-processed commands are executed, subsequent to-be-processed commands can be executed on the basis. The parameters carried by the restart command are selected by a person skilled in the art according to a specific use scenario, and are not limited herein.

[0040] The embodiment of the present application inserts a restart command between kernel groups without a dependency relationship, to identify a kernel group that can be executed directly without depending on any kernel group.

[0041] Step 20: according to the execution sequence, serialize the plurality of kernel groups into a hardware executable command stream, and write the command stream into a computing cache.

[0042] The specific manner of serializing the plurality of kernel groups into a hardware executable command stream and writing the command stream into a computing cache is selected by a person skilled in the art according to a specific use scenario, and is not limited herein.

[0043] Step 30: parse the command stream in the computing cache, and when the restart command is detected, directly read the kernel group located after the restart command to execute the read kernel group.

[0044] For the kernel group that can be directly executed without any kernel group, since it is marked by the restart command in step 10, when the restart command is detected in the process of parsing the command stream, it indicates that the next kernel group is located after the restart command; therefore, the next address of the restart command in the computing cache is directly obtained, without scheduling and starting the next kernel, the first to-be-processed command in the next kernel group is read according to the next address, and direct execution is performed.

[0045] The present application inserts a restart command between kernel groups without a dependency relationship, uses the restart command to mark a kernel group that can be directly executed, obtains an execution sequence of the plurality of kernel groups, serializes the plurality of kernel groups into a hardware executable command stream according to the execution sequence, and writes the command stream into a computing cache. The command stream is parsed, when the restart command is detected, the kernel group located after the restart command is directly read and processed. Compared with the prior art, after a kernel group is processed, the next kernel group is directly read, without waiting for the scheduler to schedule the next kernel again, without starting the next kernel, without accessing the DDR, and avoiding the delay caused by scheduling and starting; therefore, in the process of parsing the command stream, after a kernel group is processed, the next kernel group is processed, in the computing scenario of a large number of small kernel groups, the time from the end of processing of a small kernel group to the start of processing of the next small kernel group is reduced from thousands of clock cycles to 0; and since there is no dependency relationship between most small kernel groups, the restart command can be used in most cases, that is, the delay caused by scheduling and starting can be skipped, the performance of the GPU can be greatly improved, and the practicability is high.

[0046] In order to illustrate the process of generating the execution sequence of the plurality of kernel groups, Figure 3 As shown, the step 10 comprises: Step 101: Obtain at least one fence between the plurality of kernel groups.

[0047] The determination of the fence between the plurality of kernel groups is selected by a person skilled in the art according to a specific use scenario, which is not limited here.

[0048] The dependency relationship between each kernel is displayed when the user uses it. In the prior art, all the dependency relationships are handled by the scheduler.

[0049] Each kernel function in the plurality of kernel groups is originally executed in a certain logical order and has no obvious dependency relationship between each other. In order to improve the processing efficiency, the GPU needs to process multiple kernel groups in parallel; and in order to ensure that each kernel function can be successfully and efficiently executed, the designer often writes the data structure of the calculation cache according to the execution order between each kernel function, so as to execute the plurality of kernel groups as much as possible in disorder, and thus achieve the optimal performance of the GPU. In an embodiment, when designing the data structure of the calculation cache, for the plurality of kernel groups, if the kernel function B2 needs to start processing after the kernel function Al is calculated, the designer will insert a fence between the kernel group 1 to which the kernel function Al belongs and the kernel group 2 to which the kernel function B2 belongs.

[0050] When the compiler reads the data structure of the calculation cache written by the designer, it will detect the fence written by the designer.

[0051] The data structure of the calculation cache includes a plurality of kernel groups arranged in a certain order. Since there are often a plurality of kernel groups that can be directly executed in parallel, i.e. without waiting for other kernel groups to be processed before processing, in the embodiment of the present application, the compiler directly inserts a restart command between the kernel groups without a fence, i.e. the step 10 further includes: for the plurality of kernel groups, inserting a restart command between the kernel groups without a fence.

[0052] Step 102: Obtain two kernel groups located before and after the fence, and take the dependency relationship between the two kernel groups located before and after the fence as a to-be-judged dependency relationship.

[0053] Step 103: Perform necessity detection on the to-be-judged dependency relationship to generate the execution order.

[0054] The specific manner of necessity detection is selected by a person skilled in the art according to a specific use scenario, which is not limited here.

[0055] The designer only marks the kernel groups that may have a dependency relationship, and the compiler actually performs the necessity detection to determine whether there is a dependency relationship between the corresponding kernel groups; the necessity detection is performed when there is a dependency relationship between the kernel groups; and the necessity detection is not performed when there is no dependency relationship between the kernel groups.

[0056] In an embodiment, the compiler selectively inserts a restart command or an end command between the corresponding kernel groups according to whether the dependency relationship to be judged passes the necessity detection. Since the setting of the restart command needs the cooperation of the software, the software needs to determine that there is no dependency relationship between a kernel group and the next kernel group located after the kernel group, and does not need to wait until the previous kernel group completes all calculations before starting the next kernel group, so that the restart command can be directly inserted between the two kernel groups. Otherwise, the end command needs to be used at the end of the previous kernel group to inform the scheduler that the current kernel group has been processed, and then the scheduler starts the next kernel group.

[0057] Specifically, as shown in Figure 4 the selectively inserting a restart command or an end command between the corresponding kernel groups according to whether the dependency relationship to be judged passes the necessity detection includes: Step 1021: When the dependency relationship to be judged passes the necessity detection, at least two kernel groups of the dependency relationship to be judged are obtained, a kernel group before the dependency relationship is taken as a first kernel group, and a kernel group after the dependency relationship is taken as a second kernel group; an end command is inserted between the last kernel function of the first kernel group and the first kernel function of the second kernel group.

[0058] When the kernel group 2 needs to wait for the kernel group 1 to be processed before execution, the kernel group before the dependency relationship is the kernel group 1, and the kernel group after the dependency relationship is the kernel group 2.

[0059] For example, the kernel group 1 and the kernel group 2 of the dependency relationship to be judged are obtained, the kernel group 1 is taken as the first kernel group, and the kernel group 2 is taken as the second kernel group; an end command is inserted between the last kernel function of the first kernel group and the first kernel function of the second kernel group.

[0060] Step 1022: When the to-be-judged dependency relationship fails to pass the necessity detection, at least two kernel groups of the to-be-judged dependency relationship are acquired, a kernel group acquired first is taken as a third kernel group, and a kernel group acquired later is taken as a fourth kernel group; and a restart command is inserted between the last kernel function of the third kernel group and the first kernel function of the fourth kernel group.

[0061] The acquisition sequence refers to a kernel group arranged in front in sequence in a data structure of a calculation cache written by a designer and read by a compiler. When the compiler acquires the kernel group 3 first and then acquires the kernel group 4, the kernel group in front in the acquisition sequence is the kernel group 3, and the kernel group in back in the acquisition sequence is the kernel group 4.

[0062] For example, the kernel group 3 and the kernel group 4 of the to-be-judged dependency relationship are acquired, the kernel group 3 is taken as the third kernel group, and the kernel group 4 is taken as the fourth kernel group; and a restart command is inserted between the last kernel function of the third kernel group and the first kernel function of the fourth kernel group.

[0063] As shown in FIG. 6, the kernel group 3 and the kernel group 4 of the to-be-judged dependency relationship are acquired, the kernel group 3 is taken as the third kernel group, and the kernel group 4 is taken as the fourth kernel group; and a restart command is inserted between the last kernel function of the third kernel group and the first kernel function of the fourth kernel group. Figure 5 FIG. 7 shows a specific example of a data structure of a calculation cache according to an embodiment of the present application.

[0064] After the restart command or the end command is specified between the kernel groups according to the dependency relationship between the kernel groups, the user mode drive (UMD for short) writes the multiple kernel groups required by the user to process in the form of a command stream to the DDR through step 20 on the software level. The software of the user mode drive directly interacts with an application programming interface (API for short), and a specific example of a scheduling system is as shown in FIG. 8. Figure 6As shown; wherein, API is, for example, Compute Unified Device Architecture (CUDA). First, the user calls the user mode driver through the API, and the user mode driver writes the plurality of kernel groups into the DDR through the command stream, and updates the write pointer of the computing cache; in some application scenarios (for example, artificial intelligence model computing), a large number of small kernels are placed in the same computing cache, and these small kernels are divided into a plurality of kernel groups, and each kernel group is started by one kick; wherein, the specific starting mode is selected by a person skilled in the art according to the specific use scenario, which is not limited here.

[0065] And at the hardware level, the next kernel group can be directly read when the restart command is identified. Specifically, as shown in Figure 7 The step 30 comprises: Step 301: start to parse the command stream, determine the first first pending command of the first kernel group according to the scheduler, and execute the first kernel group from the first pending command.

[0066] The command stream is parsed by the command processor, the address of the first first pending command of the first kernel group required by the user is determined by the scheduler, the first first pending command is read, and then the pending commands in the first kernel group are executed from the first first pending command, and all the pending commands in the first kernel group are executed.

[0067] Step 302: continue to parse the command stream, and when the restart command is detected, sequentially read the next address of the restart command, obtain the first second pending command of the next kernel group according to the next address, and directly read the kernel group after the restart command.

[0068] After all the pending commands in the first kernel group are executed, the command processor continues to parse the command stream, and if a restart command is detected, the next address of the restart command is sequentially read, and the first second pending command of the next kernel group is obtained according to the next address, so as to directly read the kernel group after the restart command. Since the first second pending command of the next kernel group can be obtained by sequentially reading the next address of the restart command, the corresponding address does not need to be obtained through the scheduler, and the kernel group after the restart command can be directly read; the next kernel does not need to be scheduled again through the scheduler, and the next kernel does not need to be started, that is, the DDR does not need to be accessed, so that the delay caused by the scheduling and starting can be avoided.

[0069] Step 303: All the second pending commands in the next kernel group are executed from the first second pending command, so as to execute the read kernel group.

[0070] After the command processor directly reads the first second pending command of the kernel group after the restart command, the pending commands in the kernel group after the restart command are executed from the second pending command, and all the pending commands in the kernel group after the restart command are executed.

[0071] In an embodiment, the computing cache is a ring cache, and when reading the kernel group from the computing cache, the read pointer of the computing cache also needs to be updated, and when writing the kernel group into the computing cache, the write pointer of the computing cache also needs to be updated. For example, when the ring cache defines an address region, when the addresses from 1 to 100 are used to store the to-be-processed commands to be executed, in the initial state, the addresses from 1 to 50 can be used to store the to-be-processed commands, and at this time, the write pointer points to 50; after that, the software can go to handle other transactions, and when obtaining the kernel group that can be executed from the computing cache, only the kernel group at the address of 50 can be obtained. When obtaining the kernel group at the address of 40 from the computing cache, the read pointer of the computing cache points to the kernel group at the address of 40. In an optional embodiment, the position of the read pointer is updated every first preset time, and the position of the read pointer is stored to the DDR, and the read pointer is subsequently obtained from the DDR by the scheduler, wherein the first preset time is determined by a person skilled in the art according to a specific use scenario, which is not limited herein. When the software reads the address of 40, it can be known that the to-be-processed commands stored in the addresses from 0 to 40 have been taken away, and the addresses from 0 to 40 in the computing cache can be overwritten, that is, the addresses from 0 to 40 can be used to store new to-be-processed commands; every time when storing new to-be-processed commands into the computing cache, when sequentially storing to the address of 100, the new to-be-processed commands can be written from the address of 0 according to the position of the read pointer; for example, when the read pointer points to the address of 40, at this time, the new to-be-processed commands can be written to the addresses from 0 to 40, and so on, so that the addresses from 0 to 100 in the computing cache are used as storage space to store the to-be-processed commands. In an optional embodiment, the position of the write pointer is updated every second preset time, and the position of the write pointer is stored to the DDR, and the write pointer is subsequently obtained from the DDR by the scheduler, wherein the second preset time is determined by a person skilled in the art according to a specific use scenario, and the second interval preset time can be set to: updating the position of the write pointer every 10 addresses.

[0072] For a plurality of kernel groups required to be processed by a user, the first time to start depends on the scheduling of the scheduler, as shown in the following table: Figure 8 The step 301 includes: Step 3011: The scheduler transmits the base address of the first kernel group and the position of the read pointer of the computing cache to the command processor.

[0073] The position of the read pointer is the position of the read pointer.

[0074] The first kernel group is started again by the scheduler, and when the command processor starts to start a kernel group, the scheduler first accesses the DDR to obtain the read pointer and the write pointer of the calculation cache and transmits them to the command processor.

[0075] Step 3012: The command processor starts to parse the command stream according to the base address and the read pointer position to obtain the address of the first kernel group.

[0076] Step 3013: The command processor loads the first first to-be-processed command into the instruction cache from the video memory according to the address of the first kernel group.

[0077] After the command processor obtains the address of the first kernel group, the first first to-be-processed command is loaded from the video memory into the instruction cache to execute the first first to-be-processed command.

[0078] Step 3014: The command processor configures calculation resources and thread bundles for the first first to-be-processed command, uses the calculation resources and thread bundles to execute the first first to-be-processed command in the instruction cache, and executes all first to-be-processed commands in the first kernel group from the first first to-be-processed command.

[0079] The command processor allocates resources for the execution of the first first to-be-processed command and executes it, and for all first to-be-processed commands in the first kernel group, the next address of the first to-be-processed command is loaded into the instruction cache, resources are allocated and executed.

[0080] After the first kernel group is processed, the command processor continues to parse the command stream, and when the end command is detected, it is indicated that the start of the first to-be-processed command after the end command must depend on the scheduling of the scheduler, as shown in the following step 301. Figure 9 The step 301 further includes the following steps: Step 401: When the command processor detects the end command, a kernel end signal is sent to the scheduler.

[0081] The kernel end signal is used for the command processor to inform the scheduler that the current kernel group has been executed, and the specific composition format of the kernel end signal is selected by a person skilled in the art according to a specific use scenario, which is not limited here.

[0082] Step 402: The scheduler receives the end signal, accesses the memory unit to obtain the current read pointer position.

[0083] The memory unit includes DDR or HBM.

[0084] When the scheduler knows that the current kernel group has finished execution, it checks the unprocessed kernel group that depends on the kernel group that has finished execution, selects the next kernel group from the unprocessed kernel group, and starts.

[0085] Step 403: The command processor acquires the first first pending command of the next kernel group according to the address of the end signal and the current read pointer position, and loads the first first pending command from the video memory to the instruction cache.

[0086] Step 404: The command processor configures computing resources and thread bundles for the first first pending command, and executes the first first pending command in the instruction cache using the computing resources and thread bundles, so as to execute all first pending commands in the next kernel group starting from the first first pending command.

[0087] The starting process after the end command is the same as the process of the first start, and depends on the scheduling of the scheduler.

[0088] In an embodiment, the operation time of the prior art start is as shown in Figure 10 No matter the first start or each start from the second time, there is a delay caused by the scheduling of the scheduler, the acquisition of the read pointer, and the acquisition of the pending command. The operation time of the start using the small-kernel GPU scheduling optimization method of the embodiment of the present application is as shown in Figure 11 The delay caused by the scheduling of the scheduler, the acquisition of the read pointer, and the acquisition of the pending command in the first start is inevitable. From the second time, once the restart command is detected, the process of acquiring the pending command can be started in advance, and the time consumption of the start trigger is greatly shortened. This is a very big optimization for a large number of small-kernel computing scenarios that require multiple starts of small kernels.

[0089] As shown in Figure 12 is a schematic diagram of an architecture of the small-kernel GPU scheduling optimization device of the embodiment of the present application. The small-kernel GPU scheduling optimization device of the embodiment of the present application includes one or more processors 21 and a memory 22. Among them, Figure 12 In an embodiment, the processor 21 is taken as an example.

[0090] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 12 In an embodiment, the connection through the bus is taken as an example.

[0091] The memory 22, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs and non-volatile computer executable programs, such as the GPU scheduling optimization method for small kernels in the embodiment. The processor 21 executes the GPU scheduling optimization method for small kernels by running the non-volatile software programs and instructions stored in the memory 22.

[0092] The memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 can optionally include a memory remotely arranged with respect to the processor 21, which can be connected to the processor 21 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0093] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the GPU scheduling optimization method for small kernels in the above-mentioned embodiments is executed, for example, each step of the GPU scheduling optimization method for small kernels of the embodiments of the application described above is executed.

[0094] The embodiments of the application also provide a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are executed by one or more processors, for example Figure 12 a processor 21, so that the above-mentioned one or more processors can execute the GPU scheduling optimization method for small kernels in the embodiments of the application, for example, each step of the GPU scheduling optimization method for small kernels of the embodiments of the application described above is executed; and the above-mentioned various modules, units can also be realized Figure 12 the GPU scheduling optimization method for small kernels in the embodiments of the application, for example, each step of the GPU scheduling optimization method for small kernels of the embodiments of the application described above is executed; and the above-mentioned various modules, units can also be realized Figure 12 the GPU scheduling optimization method for small kernels in the embodiments of the application, for example, each step of the GPU scheduling optimization method for small kernels of the embodiments of the application described above is executed.

[0095] It is worth noting that the information interaction, execution process and the like between the modules and units in the above-mentioned apparatus and system are based on the same concept as the processing method embodiments of the application, and the specific content can be referred to the description in the method embodiments of the application, which will not be described here.

[0096] Those skilled in the art can understand that all or part of the steps in the various methods of the embodiments can be completed by instructing the related hardware with a program, and the program can be stored in a computer readable storage medium, which can include Read Only Memory (ROM), Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0097] The above description is merely the preferred embodiments of the present application, and is not used to limit the present application. Any modification, equivalent replacement and improvement made in the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A GPU scheduling optimization method for small kernels, characterized in that, The method comprises the following steps: inserting a restart command between kernel groups without dependency relationship to obtain an execution sequence of the kernel groups; serializing the kernel groups into a hardware executable command stream according to the execution sequence, and writing the command stream into a computing cache; parsing the command stream in the computing cache, and directly reading kernel groups after the restart command to execute the read kernel groups when the restart command is detected.

2. The method for GPU scheduling optimization for small kernels according to claim 1, wherein, The step of inserting a restart command between kernel groups without dependency relationship to obtain an execution sequence of the kernel groups comprises the following steps: obtaining at least one barrier between the kernel groups; obtaining two kernel groups before and after the barrier, and taking dependency relationship between the two kernel groups as a to-be-judged dependency relationship; performing necessity detection on the to-be-judged dependency relationship to generate the execution sequence. 3.The method for GPU scheduling optimization for small kernels according to claim 2, wherein, The step of performing necessity detection on the to-be-judged dependency relationship to generate the execution sequence comprises the following steps: selectively inserting a restart command or an end command between corresponding kernel groups according to whether the to-be-judged dependency relationship passes the necessity detection.

4. The method for GPU scheduling optimization for small kernels according to claim 3, wherein, Each kernel group comprises at least one kernel function. The step of selectively inserting a restart command or an end command between corresponding kernel groups according to whether the to-be-judged dependency relationship passes the necessity detection comprises the following steps: when the to-be-judged dependency relationship passes the necessity detection, obtaining at least two kernel groups of the to-be-judged dependency relationship, taking a kernel group before the dependency relationship as a first kernel group, and taking a kernel group after the dependency relationship as a second kernel group; inserting an end command between a last kernel function of the first kernel group and a first kernel function of the second kernel group; when the to-be-judged dependency relationship does not pass the necessity detection, obtaining at least two kernel groups of the to-be-judged dependency relationship, taking a first obtained kernel group as a third kernel group, and taking a second obtained kernel group as a fourth kernel group; inserting a restart command between a last kernel function of the third kernel group and a first kernel function of the fourth kernel group.

5. The method for GPU scheduling optimization for small kernel according to claim 2, wherein, The step of inserting a restart command between kernel groups without dependency relationship to obtain an execution sequence of the kernel groups further comprises the following steps: inserting a restart command between kernel groups without barrier.

6. The method for GPU scheduling optimization for small kernels according to claim 1, wherein, The step of parsing the command stream in the computing cache, and directly reading kernel groups after the restart command to execute the read kernel groups when the restart command is detected comprises the following steps: starting to parse the command stream, and determining, according to a scheduler, a first first pending command of a first kernel group to start execution of the first kernel group from the first first pending command; continuing to parse the command stream, and when a restart command is detected, sequentially reading a next address of the restart command, and obtaining a first second pending command of a next kernel group according to the next address to directly read the kernel group after the restart command; starting to execute all second pending commands in the next kernel group from the first second pending command to execute the read kernel group.

7. The method for GPU scheduling optimization for small kernels according to claim 6, wherein, The starting to parse the command stream and determining, according to a scheduler, a first first pending command of a first kernel group includes: the scheduler transmits a base address of the first kernel group and a read pointer position of the calculation cache to a command processor; the command processor starts to parse the command stream according to the base address and the read pointer position to obtain an address of the first kernel group; the command processor loads the first first pending command from the first kernel group to an instruction cache from a display memory according to the address of the first kernel group; the command processor configures calculation resources and a thread bundle for the first first pending command, and uses the calculation resources and the thread bundle to execute the first first pending command in the instruction cache to start execution of all first pending commands in the first kernel group from the first first pending command.

8. The method for GPU scheduling optimization for small kernels according to claim 7, wherein, After the starting to parse the command stream and determining, according to a scheduler, a first first pending command of a first kernel group, the method further includes: when the command processor detects an end command, sending a kernel end signal to the scheduler; the scheduler receives the end signal, accesses a memory unit, and obtains a current read pointer position; the command processor obtains a first first pending command of a next kernel group according to an address of the end signal and the current read pointer position, and loads the first first pending command from the next kernel group to an instruction cache; the command processor configures calculation resources and a thread bundle for the first first pending command, and uses the calculation resources and the thread bundle to execute the first first pending command in the instruction cache to start execution of all first pending commands in the next kernel group from the first first pending command.

9. A GPU scheduling optimization apparatus for small kernels, comprising: The GPU scheduling optimization device for small kernels includes at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to implement the GPU scheduling optimization method for small kernels in any one of claims 1 to 8 after being executed by the processor.

10. A non-transitory computer storage medium, comprising, The computer storage medium stores computer executable instructions executed by one or more processors to complete the GPU scheduling optimization method for the small kernel in any one of claims 1 to 8.

Images