GPU computing power sharing control methods, devices, equipment, media, and software products

By dividing the operator's output matrix into tiles and establishing a mapping relationship between CTA and tiles in the GPU computing power sharing control method, and dynamically allocating CTA to different GPUs for execution, the problem of uneven GPU load is solved, and computing efficiency and resource utilization are improved.

CN123114772APending Publication Date: 2026-11-17TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611499570.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-23
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In multi-tasking deployments and uneven GPU loads, the computing power of idle GPUs is difficult to be used to share the computation of busy GPUs during operation, resulting in wasted computing resources and low efficiency.

Method used

By dividing the output matrix of the operator into multiple tiles and determining a CTA for each tile, a mapping relationship between CTA and tiles is established, and CTAs are dynamically allocated to different GPUs for execution. The number of CTAs is determined based on the real-time resource status of the GPUs, thus achieving cross-GPU computational load balancing.

Benefits of technology

It enables dynamic allocation of computing tasks without changing the upper-layer application code, improving computing efficiency and resource utilization, and shortening operator call time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114772A_ABST
    Figure CN123114772A_ABST
Patent Text Reader

Abstract

This specification provides a GPU computing power sharing control method, apparatus, device, medium, and program product. The method includes: in response to an operator invocation request, determining a borrowing GPU to execute the operator; dividing the output matrix of the operator into multiple tiles; determining a CTA for each tile; and establishing a mapping relationship between CTAs and tiles; determining at least one borrowing GPU from multiple GPUs whose real-time resource status meets the borrowing conditions; determining the number of CTAs allocated to the borrowing GPU and each borrowing GPU based on the real-time resource status of the borrowing GPU and the determined borrowing GPU; allocating each CTA between the borrowing GPU and each borrowing GPU based on the number of CTAs; and each borrowing GPU and each borrowing GPU executing its allocated CTA and writing its calculation results to the corresponding positions in the output matrix according to the mapping relationship.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to the field of parallel computing technology for graphics processing units (GPUs), and in particular to GPU computing power sharing control methods, devices, equipment, media, and program products. Background Technology

[0002] GPUs, with their massively parallel computing units, are widely used in deep learning training and inference, scientific computing, and graphics rendering. In deep learning frameworks, neural network computations are expressed as a series of operator calls, such as matrix multiplication, convolution, and attention operators, each corresponding to an independent operation. When an operator is called, the corresponding kernel program is launched to the GPU. This kernel program creates a large number of Cooperative Thread Arrays (CTAs), which are then allocated by the GPU's hardware scheduler to the GPU's internal Streaming Multiprocessors (SMs) for execution.

[0003] In related technologies, once the kernel program of an operator is started, all its CTAs can only be scheduled by the hardware scheduler of that GPU within the SM (Software Scheduler) of that GPU. Upper-layer software cannot schedule specific computations within the operator to be executed on other GPUs. As a result, in multi-tasking deployments with uneven GPU loads, the computing power of idle GPUs is difficult to be used to share the computations of busy GPUs during operation. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this specification provides GPU computing power sharing control methods, devices, equipment, media, and program products.

[0005] According to a first aspect of the embodiments of this specification, a GPU computing power sharing control method is provided, applied to a computer system comprising multiple GPUs, the method comprising: In response to an operator call request, the borrowed GPU for executing the operator is determined, the output matrix of the operator is divided into multiple tiles, a CTA is determined for each tile, and a mapping relationship between the CTA and the tile is established; wherein, the input data required to execute each CTA is the data area of ​​the tile mapped by the CTA in the input matrix of the operator.

[0006] From the plurality of GPUs, at least one borrowing GPU whose real-time resource status meets the borrowing conditions is determined. Based on the real-time resource status of the borrowed GPU and the determined borrowing GPU, the number of CTAs allocated to the borrowed GPU and each borrowing GPU is determined. Based on the number of CTAs, each CTA is allocated between the borrowed GPU and each borrowing GPU, wherein the number of CTAs allocated to a GPU is positively correlated with the size of the GPU's remaining free resources.

[0007] The borrowing GPU and each lending GPU execute their assigned CTA and write their calculation results into the corresponding positions in the output matrix according to the mapping relationship.

[0008] According to a second aspect of the embodiments of this specification, a GPU computing power sharing control device is provided, applied to a computer system including multiple GPUs, the device comprising: The mapping relationship establishment module is used to respond to the operator call request, determine the borrowed GPU to execute the operator, divide the output matrix of the operator into multiple tiles, determine a CTA for each tile, and establish a mapping relationship between the CTA and the tile; wherein, the input data required to execute each CTA is the data area of ​​the tile mapped by the CTA in the input matrix of the operator.

[0009] The CTA allocation module is used to determine at least one borrowed GPU whose real-time resource status meets the borrowing conditions from the plurality of GPUs, determine the number of CTAs to be allocated to the borrowed GPU and each borrowed GPU based on the real-time resource status of the borrowed GPU and the determined borrowed GPU, and allocate each CTA among the borrowed GPU and each borrowed GPU based on the number of CTAs, wherein the number of CTAs allocated to a GPU is positively correlated with the size of the remaining free resources of the GPU.

[0010] The CTA execution module is used for the borrowing GPU and each borrowing GPU to execute their assigned CTAs and write their respective calculation results into the corresponding positions in the output matrix according to the mapping relationship.

[0011] According to a third aspect of the embodiments of this specification, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described in the first aspect.

[0012] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in the first aspect.

[0013] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in the first aspect.

[0014] The technical solutions provided in the embodiments of this specification may include the following beneficial effects: In the embodiments described in this specification, when an operator is invoked, the output matrix of the operator is divided into multiple tiles, and a CTA is determined for each tile, establishing a one-to-one mapping relationship. Through this mapping relationship, the GPUs executing each part of the operator can determine the output position and required input data corresponding to each CTA, making the calculations between each CTA independent of each other, thereby providing a basis for allocating each CTA to different GPUs for execution.

[0015] Based on this, each CTA is allocated between borrowing and lending GPUs. This segmentation extends to the tile level within the operator, allowing each segmented computation to be executed independently by different GPUs without migrating the entire operator.

[0016] The number of CTAs allocated to each GPU is determined based on the real-time resource status of the borrowed GPU and each borrowed GPU. This allows the computational scope of each GPU to be dynamically determined according to the currently available computing power of each GPU when the operator is called, without the need to fix the computational division between GPUs before the task starts.

[0017] The borrowing GPU and each borrowing GPU execute their assigned CTA and write their calculation results into the corresponding tile position in the output matrix according to the output position determined by the mapping relationship. Since each CTA corresponds one-to-one with each tile in the mapping relationship, the positions written by each GPU do not overlap, and the output matrix is ​​complete without merging.

[0018] The Tile partitioning, CTA allocation, and CTA execution described above are all completed within the operator call, without changing the upper-level application code. The operator call is still initiated in the original way, and the output matrix that has been written to the corresponding position is directly obtained when the call ends.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0021] Figure 1 This is a flowchart illustrating a GPU computing power sharing control method according to an exemplary embodiment of this specification.

[0022] Figure 2 This is a schematic diagram illustrating, according to an exemplary embodiment, the division of multiple tiles into CTA groups.

[0023] Figure 3 This is a schematic diagram illustrating input data transmission based on a K-packet model according to an exemplary embodiment of this specification.

[0024] Figure 4 This is a flowchart illustrating a cross-process execution CTA according to an exemplary embodiment of this specification.

[0025] Figure 5 This is a schematic diagram of a GPU computing power sharing system illustrated in this specification according to an exemplary embodiment.

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of this specification.

[0027] Figure 7 This is a block diagram illustrating a GPU computing power sharing control device according to an exemplary embodiment of this specification. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0029] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0031] Before describing the various embodiments in detail, some of the terms used in this specification will be explained.

[0032] CTA refers to the execution unit created when the kernel program starts, also known as a thread block.

[0033] SM refers to the physical computing unit inside the GPU. A GPU consists of multiple SMs. The GPU's hardware scheduler allocates each CTA to an SM for execution based on the resource status of the SM.

[0034] A tile is a sub-matrix block obtained by dividing the output matrix of an operator; each sub-matrix block is a tile. Dividing the output matrix into tiles is an existing practice in related technologies, and the improvement of this application does not lie in the tile division itself.

[0035] The borrowed GPU refers to the GPU corresponding to the current operator call request. The borrowed GPU refers to the GPU that is borrowed to share some of the computation with the borrowed GPU. The borrowed GPU is determined from GPUs other than the borrowed GPU based on the real-time resource status. The terms "borrowed GPU" and "borrowed GPU" are only valid in the context of a single operator call request and are independent of the physical location of the data; the same GPU can be used as both a borrowed GPU and a borrowed GPU in different operator calls.

[0036] This solution can be applied to computer systems containing multiple GPUs. Specifically, the multiple GPUs in a computer system can have the following relationship during runtime: ① Multiple GPUs on the same server, where the borrowing GPU and the lending GPU belong to the same process.

[0037] ② Multiple GPUs on the same server, where borrowed GPUs and lent GPUs are used by different processes.

[0038] ③ This solution is also applicable to multiple GPUs in a server cluster that are interconnected via high-speed interconnection, such as GPUs located on different servers that are interconnected via point-to-point high-speed interconnection in a supernode, provided that the interconnection bandwidth meets the requirements.

[0039] The embodiments described in this specification will now be described in detail.

[0040] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a GPU computing power sharing control method according to an exemplary embodiment, including steps 101-103: Step 101: In response to the operator call request, determine the borrowed GPU to execute the operator, divide the output matrix of the operator into multiple tiles, determine a CTA for each tile, and establish a mapping relationship between the CTA and the tile; wherein, the input data required to execute each CTA is the data area of ​​the tile mapped by the CTA in the input matrix of the operator.

[0041] In this embodiment, the upper-layer application calls the operator in the original way. For example, when a deep learning framework performs neural network calculations, it calls the operator according to the original calling interface and calling process. This solution does not modify the calculation code of the upper-layer application.

[0042] In response to an operator call request from an upper-layer application, the GPU executing the operator is identified as the borrowed GPU. During the call, the operator's output matrix is ​​divided into multiple tiles, and a CTA is assigned to each tile. One tile is computed by one CTA, and there is a one-to-one mapping between CTAs and tiles.

[0043] In related technologies, after the operator's kernel program starts, the execution of its CTAs is determined by the GPU's hardware scheduler, and the upper-layer software cannot control the execution order of each CTA. This solution modifies the implementation of the original operator, redefining the mapping relationship between tiles and CTAs, so that each CTA is arranged according to the position of its mapped tile in the output matrix. Therefore, the execution of each CTA can be controlled according to the expected order and range, providing a foundation for subsequently allocating each CTA to different GPUs for execution.

[0044] After establishing the mapping relationship, the output position and required input data corresponding to each CTA are determined. Specifically, the output position of each CTA is determined based on the position of the tile mapped to that CTA in the output matrix; the required input data for each CTA is the data region corresponding to the tile mapped to that CTA in the operator's input matrix, and this data region is also determined based on the tile's position.

[0045] Taking matrix multiplication D=A×B as an example, the tile in the i-th row and j-th column of the output matrix D is calculated from the corresponding i-th row block in the input matrix A and the corresponding j-th column block in the input matrix B. That is, the row and column positions of this tile in the output matrix uniquely determine the row and column blocks required to calculate this tile in the input matrix. Correspondingly, the CTA responsible for calculating this tile outputs the tile's position in the output matrix, and the required input data is the data from that row and column block. Since the tiles do not overlap, and the tiles mapped by each CTA are different, there is no data dependency between the CTAs, and their calculations are independent.

[0046] Step 102: Determine at least one borrowing GPU from the plurality of GPUs whose real-time resource status meets the borrowing conditions. Based on the real-time resource status of the borrowed GPU and the determined borrowing GPU, determine the number of CTAs allocated to the borrowed GPU and each borrowing GPU. Based on the number of CTAs, allocate each CTA between the borrowed GPU and each borrowing GPU, wherein the number of CTAs allocated to a GPU is positively correlated with the size of the GPU's remaining free resources.

[0047] In one implementation, real-time resource status can be represented by the number of idle SMs (Smart Streams) on the GPU, i.e., the number of SMs currently not occupied by the GPU. The real-time resource status of each GPU can be obtained by directly querying the total number of currently idle SMs for each GPU before starting the kernel program.

[0048] The lending condition is used to determine whether a GPU is currently capable of handling the computational demands of this operator call. For example, the lending condition can include the following situations: the GPU is completely idle; or, the GPU is partially occupied by other tasks, but the number of idle task managers (SMs) reaches a preset number. Correspondingly, if the GPU is close to saturation or the number of idle SMs is below the preset number, the lending condition is not met.

[0049] In one implementation, the process hosting the borrowing GPU queries the real-time resource status of each GPU (excluding the borrowing GPU) one by one, and identifies the GPUs that meet the lending criteria as the lending GPUs. In another implementation, a global scheduling process records the real-time resource status of each GPU and recommends GPUs that meet the lending criteria when operators are called. Alternatively, the user can specify the lending GPU.

[0050] If none of the multiple GPUs meet the lending conditions, all CTAs are allocated to the borrowing GPU, and the operator is executed only on the borrowing GPU, which degenerates into the single-GPU execution method in related technologies.

[0051] After determining the borrowed GPU, the number of CTAs allocated to the borrowed GPU and each borrowed GPU can be further determined based on the real-time resource status of the borrowed GPU and the determined borrowed GPU. In one implementation, the number of idle SMs of the borrowed GPU and each borrowed GPU can be obtained; based on the number of idle SMs, the number of CTAs allocated to each borrowed GPU and each borrowed GPU can be determined, so that the number of CTAs allocated to each GPU matches its number of idle SMs, thereby achieving time balance for each GPU to complete its allocated CTAs.

[0052] For example, the number of CTAs allocated to each idle SM can be kept approximately the same. Assuming a GPU has 108 SMs, and this operator call generates 486 CTAs, with 108 idle SMs borrowed from the GPU and 54 from the borrowed GPU, then the borrowing GPU is allocated 324 CTAs, and the borrowing GPU is allocated 162 CTAs. The time it takes for both GPUs to complete their assigned CTAs is approximately the same. Therefore, the completion time of this operator call will not be prolonged due to one side assuming an excessive number of CTAs, which helps to shorten the total execution time of this operator call. If both GPUs are completely idle, the CTAs are approximately evenly distributed.

[0053] Besides using the number of idle SMs to represent real-time resource status, other metrics can also be used, such as memory utilization, the depth of the kernel program queue to be executed, and the amount of computation currently being performed by the GPU. Accordingly, when determining the number of CTAs allocated to each GPU, the number of CTAs allocated to each GPU should be positively correlated with the amount of idle resources reflected by the corresponding metrics.

[0054] Step 103: The borrowing GPU and each lending GPU execute their assigned CTA and write their calculation results into the corresponding positions in the output matrix according to the mapping relationship.

[0055] On the borrowed GPU, the kernel program of the operator is launched by the process running the borrowed GPU. The input matrix is ​​stored in the memory of the borrowed GPU, and the kernel program retrieves the input data required for each assigned CTA from the memory of the borrowed GPU itself, and executes each assigned CTA.

[0056] For each loaned GPU, the input data required for each CTA allocated to that loaned GPU is transferred to the loaned GPU's memory. The kernel program of the operator launched by the process of the loaned GPU then executes the allocated CTA based on the acquired input data. The range of CTAs specified within the kernel programs launched in different loaned GPUs constitutes the CTAs allocated to that loaned GPU.

[0057] The borrowed and lent GPUs each launch the same kernel program, differing only in their launch parameters. These parameters limit the set of CTAs executed on each GPU, effectively distributing the CTAs of a single operator call across multiple GPUs. The lent GPU launches the same operator's kernel program, eliminating the need to port or rewrite the operator's computation program separately for the lent GPU.

[0058] After each GPU completes its computation, it writes its results to the corresponding positions in the output matrix according to the mapping relationship. Since the tiles mapped by each CTA are different, the positions where each GPU writes to the output matrix do not overlap, and the output matrix is ​​complete without merging. The upper layer obtains the output matrix written to the corresponding positions in the original way, without needing to be aware of the GPUs involved in the computation, nor needing to modify the upper layer's computation code.

[0059] There are several ways to borrow the GPU to write computation results to the output matrix. In one implementation, the borrowed GPU directly writes the computation results to the corresponding positions in the output matrix.

[0060] In another implementation, the borrowed GPU first writes the computation result into the local cache of the borrowed GPU, and then asynchronously sends the computation result in the local cache back to write it into the corresponding position in the output matrix.

[0061] Since the remote write instruction latency can be high when the borrowed GPU directly writes data to the output matrix of the borrowed GPU, it can block the computation flow on the borrowed GPU. This embodiment first writes to the local cache and then asynchronously sends the data back, enabling the computation side of the borrowed GPU to continue executing subsequent CTA calculations more quickly.

[0062] Next, we will continue to describe an example of transferring the input data required by the CTA assigned to each loaned GPU to the corresponding loaned GPU.

[0063] Alternatively, when transferring the input data required by each CTA allocated to the loaned GPU to the memory of the loaned GPU, the data can be transferred tile by tile.

[0064] The tiles mapped by each CTA assigned to the same lending GPU can be adjacent or not adjacent in the output matrix.

[0065] In one embodiment, based on the fact that the tiles mapped by each CTA assigned to the same lending GPU are adjacent in the output matrix, each pair of adjacent tiles in the output matrix can also be divided into a CTA group, and the CTA group corresponding to each CTA can be determined according to the mapping relationship. For each lending GPU, the input data of each CTA group to which the lending GPU belongs is transmitted to the lending GPU together, using the CTA group as the transmission unit.

[0066] One way to divide CTA groups is to divide multiple adjacent tiles in the output matrix in both the row and column directions into a CTA group, so that the input data required by each CTA in the same CTA group can be transmitted together.

[0067] like Figure 2As shown, taking matrix multiplication D=A×B as an example, the input matrix A has a dimension of M×K, the input matrix B has a dimension of K×N, and the output matrix D has a dimension of M×N. Assume that the tiles in the output matrix D are linearly numbered from 0 to 15 in positional order. The input data required for multiple tiles in the same row direction includes the same row block of the input matrix A, and the input data required for multiple tiles in the same column direction includes the same column block of the input matrix B. For example, the CTA group consisting of tiles numbered 0, 1, 4, and 5 corresponds to the input data of row A and column B of the input matrix A.

[0068] In this embodiment, row blocks and column blocks shared by each CTA can be repeatedly used by CTAs within the group after a single transmission, reducing redundant transmissions.

[0069] It should be noted that the number R of CTAs allocated to the loaned GPU does not need to be aligned with the boundaries of the CTA group; that is, R does not need to be an integer multiple of the CTA group size. For example, Figure 2 CTAs numbered 0, 1, 4, and 5 are assigned to the lending GPU, while the remaining CTAs are assigned to the borrowing GPU. When the allocation boundary falls within a certain CTA group (for example, when the number of CTAs allocated to the lending GPU is 3, the allocation boundary falls within a CTA group consisting of tiles numbered 0, 1, 4, and 5), the input data for that CTA group is still transmitted to the lending GPU as a whole. The lending GPU only executes the assigned CTAs and does not repeatedly execute unassigned CTAs from the received CTA group. In this way, the lending GPU's communication side receives the complete CTA group, while the computation side only executes the portion that is actually allocated.

[0070] When determining the size of a CTA group (i.e., the number of CTAs within the group), the following criteria can be used: First, the number of CTAs within the group should be an integer multiple of the number of idle SMs on the borrowed GPU, to cover the number of CTAs that the GPU can execute simultaneously at one time, thus reducing SM idle time. Second, the smallest integer multiple should be used, provided that the amount of input data required by the CTA group does not exceed the local temporary storage capacity of the borrowed GPU.

[0071] When transmitting input data, only the input data required by each CTA allocated to the lending GPU can be transmitted to the lending GPU, without transmitting all the input data of the operators, so that the communication overhead is proportional to the amount of computation shared by the lending GPU.

[0072] Since the interconnect bandwidth between GPUs is typically much lower than the bandwidth of the GPU's local memory, if each CTA on the borrowed GPU were to remotely read the required data one by one during computation, it would consume a significant amount of interconnect bandwidth. Therefore, the input data required by the borrowed GPU is transferred in batches to the borrowed GPU's local memory for temporary storage, and each CTA reads from the locally stored data.

[0073] Based on CTA groups as the transmission unit, all input data of a CTA group can be transmitted at once. Alternatively, in one embodiment, the input data of each CTA group can be divided into several groups along the summation dimension K of the matrix multiplication, and each group can be transmitted sequentially to the corresponding lending GPU. The principle is that matrix multiplication is decomposable along the summation dimension K: the partial multiplication and addition results of each group are accumulated along the summation dimension K to obtain the complete calculation result, which provides the basis for group transmission and group calculation.

[0074] The lending GPU sequentially stores the received packets into multiple buffers. A buffer is a region in the lending GPU's local memory that temporarily stores packet data; there can be two or more buffers. Packets are rotated between buffers according to the following rules: packets are alternately transferred to different buffers in sequence; buffers where packets have been computed continue to store subsequently transmitted packets; and buffers containing packets are not used to store other packets until any packet has been computed.

[0075] Taking two buffers as an example: packet K0 is transmitted to buffer 1, and packet K1 is transmitted to buffer 2. While the GPU's computational flow consumes K0 in buffer 1, the GPU's communication flow continues to transmit K1 to buffer 2. After K0 is computed, buffer 1 continues to store the subsequent packet K2, and so on.

[0076] Taking three buffers as an example: packets K0, K1, and K2 are transmitted sequentially to buffer 1, buffer 2, and buffer 3; after K0 is calculated and buffer 1 is released, packet K3 is transmitted to buffer 1. The more buffers there are, the greater the margin for the communication flow to lead the calculation flow, which is more conducive to reducing the waiting time of the calculation flow.

[0077] During the transmission of each packet, the borrowed GPU performs CTA calculations based on the packets that have been transmitted, and accumulates the calculation results for each packet along the summation dimension K to obtain the complete calculation result.

[0078] The following is a specific example to illustrate this, such as... Figure 2 As shown, row A is divided into K0, K1, K2, and K3 along dimension K, and column B is divided into K0, K1, K2, and K3 along dimension K. Figure 3 As shown, the communication stream borrowing the GPU sequentially copies packets K0, K1, K2, and K3, while the computation stream consumes the arriving packets sequentially. Specifically, while the communication stream copies packet K1, the computation stream consumes packet K0, and so on. Transmission and computation overlap in time, allowing the computation stream to begin working without waiting for the complete input data transmission to finish.

[0079] Since this solution can be applied to both scenarios where the borrowing and lending GPUs are in the same process or different processes, the following section explains how the borrowed GPU executes the allocated CTA from a process perspective: ① Borrowing a GPU and lending out a GPU belong to the same process.

[0080] At this point, the process itself supports launching tasks on multiple GPUs. The sharing of addresses and dependency synchronization of input data and output matrices between GPUs can be achieved through ordinary stream dependencies within the process, without the need to establish a dedicated cross-process mechanism. When calling an operator, the process allocates each CTA between the borrowing GPU and the lending GPU, transmits the input data required for the allocated CTA to the lending GPU, and launches the same kernel program on each of the two GPUs, specifying the range of CTAs to be executed.

[0081] ② Borrowing and lending out the GPU are used by different processes.

[0082] At this point, a shared memory region and notification mechanism are established between the process hosting the borrowed GPU and the processes hosting each borrowed GPU. Each process hosting a borrowed GPU accesses the input and output matrices of the operator through the shared memory region. In this scenario, the borrowed and borrowed GPUs can reside on the same server or on different servers within a server cluster. When located on the same server, the shared memory region and notification mechanism can be implemented via GPU interconnect within the server. When located on different servers within a server cluster, the shared memory region and notification mechanism are still applicable in scenarios where interconnect bandwidth is sufficient (e.g., GPUs connected via high-speed point-to-point interconnect in a supernode).

[0083] The shared memory region can be constructed as follows: the process that borrows the GPU exports the memory region allocated for operators in the borrowed GPU's memory as a handle that can be passed across processes through the Virtual Memory Management (VMM) mechanism; each process that borrows the GPU imports the handle and maps the memory region to the same virtual address to access the shared memory region.

[0084] Thus, the process that borrows the GPU can read the input matrix of the operator according to its permissions and write the calculation results to the non-overlapping output region of the output matrix.

[0085] The notification mechanism can be implemented by sharing events through inter-process communication (IPC). For example, the process running the GPU can borrow the event handle by calling the cudaIpcGetEventHandle function, and the process running the GPU can import the corresponding event by calling the cudaIpcOpenEventHandle function.

[0086] Based on this, once the input data required for each CTA allocated to each loaned GPU is ready, the process hosting the loaned GPU sends a ready notification instruction to the processes hosting each loaned GPU via a notification mechanism. This ready notification instruction instructs the processes hosting the loaned GPUs to retrieve the input data required for the allocated CTA from the shared memory region. The data location indicated by the ready notification instruction and kernel program information can be carried in the request. In response to the ready notification instruction, each process hosting the loaned GPU retrieves the corresponding input data from the shared memory region and starts the kernel program to execute the allocated CTA based on the retrieved input data. When any loaned GPU completes the allocated CTA, the process hosting the loaned GPU sends a completion notification instruction to the process hosting the loaned GPU via the notification mechanism, enabling the process hosting the loaned GPU to read the output matrix.

[0087] In one implementation, a resident worker thread can be set in the process where the GPU is borrowed. This resident worker thread waits for a ready notification instruction on the local stream. Upon receiving the instruction, it performs the acquisition of input data and the startup of the kernel program, and records the completion event after completion, thus avoiding reliance on global synchronization on the host side.

[0088] Next, we will illustrate this with a concrete implementation example. Let's assume the process containing the borrowed GPU A is called the source process, and the process containing the borrowed GPU B is called the target process. (Combined...) Figure 4 This paper introduces the process of implementing CTA based on shared memory regions and IPC mechanisms.

[0089] like Figure 4 As shown, a shared control area is established between the source and target processes. This shared control area is located in inter-process shared memory and stores VMM handles / VA proposals, an IPC event handle pool, and a request circular queue. A VA proposal refers to the virtual address that the source process suggests the target process to map. The request circular queue consists of several request slots, each transitioning between the states kEmpty, kWriting, kReady, kRunning, and kDone. kEmpty indicates idle, kWriting indicates being filled, kReady indicates published and awaiting collection, kRunning indicates execution, and kDone indicates completion.

[0090] The source process establishes a unified virtual address, allocates physical memory through CUDA VMM, and exports a handle (step ①).

[0091] The target process imports the VMM handle, maps it to the same virtual address, and sets access permissions (step ①').

[0092] The source process creates and exports IPC events (step ②, for example, calling the cudaIpcGetEventHandle function to export handles to input ready and completion events).

[0093] The target process imports IPC events (step ②', for example, by calling the cudaIpcOpenEventHandle function).

[0094] Each time the operator is called: the source process fills the request slot and publishes it (step ③). The request slot carries a unified address and the allocated CTA range [cta_begin, cta_count] (i.e., a half-open interval consisting of the starting CTA number and the number of CTAs), and the state is set to kReady.

[0095] The resident worker thread of the target process polls the circular queue of requests (step ④), and the request slots that are in the kReady state are set to kRunning.

[0096] Each stream waits for an input ready event through the cudaStreamWaitEvent function, and then starts the kernel program (step ⑤). The communication stream executes the remote data copy kernel (K-group pipeline), and the computation stream executes the Tile operator to compute the kernel (specifying the CTA range).

[0097] Once the target process has completed its computation, a completion event is recorded (step ⑥', for example, calling the cudaEventRecord function to record end_event), and the status is set to kDone.

[0098] The source process waits for the completion event, reclaims the request slot, and returns the execution result (step ⑥).

[0099] The output matrix can be written in either the borrowed GPU directly back to the unified address of the borrowed GPU, or it can be written to the local cache first and then asynchronously returned according to the completed CTA groups.

[0100] This solution can also be extended to multiple lending GPUs. The interaction logic between each lending GPU and the lending GPU is similar to the implementation logic between lending GPU A and lending GPU B described above, and will not be repeated here.

[0101] The above embodiments are described using matrix multiplication operators as a complete example. This method can also be applied to other operators where the output can be divided into blocks, and the input data required for each block can be determined based on the position of the block, such as batch matrix multiplication operators, convolution operators, and attention projection operators. Furthermore, this solution is not limited to specific matrix sizes, data types, tile sizes, or GPU interconnect forms.

[0102] Finally combined Figure 5 This manual provides a complete overview of the solution. Figure 5 As shown, the computer system includes a source process (the process where GPU A is borrowed) and a target process (the process where GPU B is borrowed), and the two GPUs are connected via a GPU peer-to-peer high-speed interconnect.

[0103] On the source process side: The input matrices A and B and the output matrix D of the operators are stored in the GPU's memory. The Tile sharding module determines the CTA range executed by each GPU based on the real-time load (i.e., real-time resource status).

[0104] The inter-process event module is responsible for sending and receiving ready and completed input events.

[0105] The unified address module enables unified address access across processes through VMM memory mapping.

[0106] On the target process side: resident worker threads are responsible for starting the kernel program.

[0107] The communication stream (SM) copies the far ends of the sub-regions of input matrices A and B to the local HBM temporary cache (HBM, High Bandwidth Memory) via the GPU peer-to-peer high-speed interconnect.

[0108] The Tile operator calculation module performs calculations within the specified CTA range and asynchronously writes the results to the corresponding positions in the output matrix D.

[0109] During each operator call, the process where the borrowing GPU resides queries the number of idle SMs for each GPU, determines the number of borrowed GPUs and the number of CTAs allocated to each GPU based on the real-time resource status, and allocates each CTA to each GPU based on the mapping relationship. The Tile sharding module then sends an unload request to the target process.

[0110] In cross-process scenarios, the inter-process event module sends an input ready event. The resident worker thread of the process where the borrowed GPU resides waits for this event, obtains the input data required by the allocated CTA from the shared memory area, and starts the kernel program on the borrowed GPU.

[0111] The communication stream borrowed from the GPU is prepared into the cache group by group along the summation dimension according to the CTA group (corresponding to Figure 4 The process involves the SM (Smart Communicator) writing sub-regions A and B into the local HBM temporary cache, with the computation stream consuming data in packets. Simultaneously, the GPU is borrowed to execute the remaining CTAs.

[0112] Each GPU writes its computation result to the corresponding position in the output matrix D; after borrowing a GPU, a completion notification command is sent; once the process of the borrowing GPU receives the completion notification and confirms that the result is available, it returns the call result to the upper layer. From the upper layer's perspective, the GPUs involved in this operator call and their respective computational scopes are invisible, and the upper layer does not need to make any modifications.

[0113] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of the apparatus and the terminal to which it is applied.

[0114] Figure 6 This is a schematic diagram illustrating the structure of an electronic device according to an exemplary embodiment. Figure 6 As shown, at the hardware level, the electronic device 600 includes a processor 602, an internal bus 604, a network interface 606, memory 608, and non-volatile memory 610, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, for example, the processor 602 reads the corresponding computer program from the non-volatile memory 610 into memory 608 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0115] Figure 7 This is a block diagram illustrating a GPU computing power sharing control device according to an exemplary embodiment of this specification. Figure 7 As shown, this device can be applied to a computer system containing multiple GPUs to implement the technical solution of this specification. The device includes: The mapping relationship establishment module 702 is used to respond to the operator call request, determine the borrowed GPU to execute the operator, divide the output matrix of the operator into multiple tiles, determine a CTA for each tile, and establish a mapping relationship between the CTA and the tile; wherein, the input data required to execute each CTA is the data area corresponding to the tile mapped by the CTA in the input matrix of the operator.

[0116] CTA allocation module 704 is used to determine at least one borrowed GPU whose real-time resource status meets the borrowing conditions from the plurality of GPUs, determine the number of CTAs to be allocated to the borrowed GPU and each borrowed GPU based on the real-time resource status of the borrowed GPU and the determined borrowed GPU, and allocate each CTA between the borrowed GPU and each borrowed GPU based on the number of CTAs, wherein the number of CTAs allocated to a GPU is positively correlated with the size of the remaining free resources of the GPU.

[0117] CTA execution module 706 is used for the borrowing GPU and each borrowing GPU to execute the assigned CTA, and to write their respective calculation results into the corresponding positions in the output matrix according to the mapping relationship.

[0118] Optionally, the CTA execution module 706 is specifically used to transfer the input data required for the CTA assigned to each lending GPU to the corresponding lending GPU, and start the kernel program of the operator in each lending GPU so that each lending GPU executes the assigned CTA based on its own acquired input data; wherein, the range of CTAs specified in the kernel program started on different lending GPUs is the CTA assigned to each of them.

[0119] Optionally, the tiles mapped by each CTA assigned to the same lending GPU are adjacent in the output matrix. The CTA execution module 706 is specifically used to divide each pair of adjacent tiles in the output matrix into a CTA group, and determine the CTA group corresponding to each CTA according to the mapping relationship. For each lending GPU, the input data of each CTA group to which the assigned CTA belongs is transmitted to the corresponding lending GPU together, using the CTA group as the transmission unit.

[0120] Optionally, the CTA execution module 706 is specifically configured to, for each CTA group of input data, divide the input data into several groups along the summation dimension of matrix multiplication, and transmit each group sequentially to the corresponding borrowed GPU; store the received groups sequentially into multiple caches; wherein, the cache where the group has been calculated continues to be used to store the subsequently transmitted groups, and the cache where the group is located is not used to store other groups before any group has been calculated; during the transmission of each group, CTA calculation is performed based on the groups that have been transmitted, and the calculation results for each group are accumulated along the summation dimension to obtain a complete calculation result.

[0121] Optionally, the CTA execution module 706 is specifically configured to establish a shared memory region and a notification mechanism between the process where the borrowing GPU resides and the processes where each lending GPU resides. Each lending GPU process accesses the input and output matrices of the operator through the shared memory region. When the input data required for each CTA allocated to each lending GPU is ready, the process where the borrowing GPU resides sends a ready notification instruction to each lending GPU process through the notification mechanism. The ready notification instruction instructs each lending GPU process to obtain the input data required for the allocated CTA from the shared memory region. In response to the ready notification instruction, each lending GPU process obtains the corresponding input data from the shared memory region and starts the kernel program to execute the allocated CTA based on the obtained input data. When any lending GPU completes the allocated CTA, the process where any lending GPU resides sends a completion notification instruction to the process where the borrowing GPU resides through the notification mechanism.

[0122] Optionally, the shared memory region is constructed as follows: the process where the borrowed GPU resides uses a virtual memory management mechanism to export the memory region allocated for the operator in the memory of the borrowed GPU as a handle that can be passed across processes; each process where the borrowing GPU resides imports the handle and maps the memory region to the same virtual address to access the shared memory region.

[0123] Optionally, the notification mechanism is an IPC mechanism.

[0124] Optionally, the CTA allocation module 704 is specifically used to obtain the number of idle SMs of the borrowed GPU and the number of idle SMs of each borrowed GPU; and to determine the number of CTAs allocated to each borrowed GPU and each borrowed GPU based on the number of idle SMs, so that the number of CTAs allocated to each GPU matches the number of idle SMs of the GPU, thereby satisfying the time balance for each GPU to complete its allocated CTA.

[0125] Optionally, the CTA execution module 706 is specifically used to first write the calculation result obtained by the borrowed GPU into the local cache of the borrowed GPU, and then asynchronously send back the calculation result in the local cache to write it into the corresponding position in the output matrix.

[0126] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0127] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0128] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the aforementioned GPU computing power sharing control methods provided in this application.

[0129] Specifically, computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0130] This specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of any of the aforementioned GPU computing power sharing control methods.

Claims

1. A GPU computing power sharing control method, characterized in that, Applied to a computer system containing multiple GPUs, the method includes: In response to an operator invocation request, the borrowed GPU for executing the operator is determined, the output matrix of the operator is divided into multiple tiles, a CTA is determined for each tile, and a mapping relationship between the CTA and the tile is established; wherein, the input data required to execute each CTA is the data region corresponding to the tile mapped by the CTA in the input matrix of the operator; From the plurality of GPUs, at least one borrowing GPU whose real-time resource status meets the borrowing conditions is determined. Based on the real-time resource status of the borrowed GPU and the determined borrowing GPU, the number of CTAs allocated to the borrowed GPU and each borrowing GPU is determined. Based on the number of CTAs, each CTA is allocated between the borrowed GPU and each borrowing GPU, wherein the number of CTAs allocated to a GPU is positively correlated with the size of the GPU's remaining free resources. The borrowing GPU and each lending GPU execute their assigned CTA and write their calculation results into the corresponding positions in the output matrix according to the mapping relationship.

2. The method according to claim 1, characterized in that, Each lent GPU executes its assigned CTA, including: The input data required for the CTA assigned to each lending GPU is transferred to the corresponding lending GPU. The kernel program of the operator is started in each lending GPU so that each lending GPU executes the assigned CTA based on its own acquired input data. The range of CTAs specified in the kernel program started on different lending GPUs is the CTA assigned to each of them.

3. The method according to claim 2, characterized in that, The tiles mapped to each CTA assigned to the same lending GPU are adjacent in the output matrix. The step of transferring the input data required by each lending GPU's assigned CTA to the corresponding lending GPU includes: The output matrix is ​​divided into a CTA group for each adjacent multiple tiles, and the CTA group corresponding to each CTA is determined according to the mapping relationship; For each loaned GPU, the input data of each CTA group to which the assigned CTA belongs is transferred to the corresponding loaned GPU, using CTA groups as the transfer unit.

4. The method according to claim 3, characterized in that, The method of transmitting input data of each assigned CTA to its respective CTA group to the corresponding lending GPU, using CTA groups as the transmission unit, includes: For each CTA group of input data, the input data is divided into several groups along the summation dimension of matrix multiplication, and each group is transmitted sequentially to the corresponding lending GPU. The received packets are stored sequentially into multiple buffers; the buffer where the packets have been calculated continues to be used to store packets to be transmitted subsequently, and the buffer where any packet is located is not used to store other packets until the calculation of any packet is completed. Any lending GPU executes the assigned CTA based on the acquired input data, including: During the transmission of each packet, CTA calculation is performed based on the packets that have been transmitted, and the calculation results for each packet are accumulated along the summation dimension to obtain the complete calculation result.

5. The method according to claim 2, characterized in that, The borrowing GPU and the lending GPU are currently used by different processes. The step of transferring the input data required for the CTA allocated to each lending GPU to the corresponding lending GPU, and starting the kernel program of the operator in each lending GPU so that each lending GPU executes the allocated CTA based on its acquired input data, includes: A memory sharing region and notification mechanism are established between the process where the borrowing GPU is located and the processes where each borrowing GPU is located. Each process where the borrowing GPU is located accesses the input matrix and output matrix of the operator through the memory sharing region. When the input data required for each CTA allocated to each borrowed GPU is ready, the process of the borrowed GPU sends a ready notification instruction to the process of each borrowed GPU through the notification mechanism; wherein, the ready notification instruction instructs the process of each borrowed GPU to obtain the input data required for the allocated CTA from the shared memory region. Each process that lends out a GPU responds to the ready notification instruction by obtaining the corresponding input data from the shared memory region and starting the kernel program to execute the allocated CTA based on the obtained input data; When any borrowed GPU completes the allocation of a CTA, the process of the borrowed GPU sends a completion notification instruction to the process of the borrowed GPU through the notification mechanism.

6. The method according to claim 5, characterized in that, The shared memory region is constructed in the following manner: The process where the borrowed GPU resides uses a virtual memory management mechanism to export the memory region allocated for the operator in the memory of the borrowed GPU as a handle that can be passed across processes. Each process that borrows the GPU imports the handle and maps the memory region to the same virtual address to access the shared memory region.

7. The method according to claim 5, characterized in that, The notification mechanism is an IPC mechanism.

8. The method according to claim 1, characterized in that, The step of determining the number of CTAs allocated to the borrowed GPU and each borrowed GPU based on the real-time resource status of the borrowed GPU and the determined loaned GPU includes: Obtain the number of free SMs of the borrowed GPU and the number of free SMs of each borrowed GPU; Based on the number of idle SMs, the number of CTAs allocated to each borrowed GPU and each borrowed GPU is determined, so that the number of CTAs allocated to each GPU matches the number of idle SMs of the GPU, thereby satisfying the time balance for each GPU to complete its allocated CTA.

9. The method according to claim 1, characterized in that, Each lending GPU writes the computation result into the corresponding position in the output matrix according to the mapping relationship, including: The lent GPU first writes the calculation result it has calculated into the local cache of the lent GPU, and then asynchronously sends the calculation result in the local cache back to write it into the corresponding position in the output matrix.

10. A GPU computing power sharing control device, characterized in that, The device, applicable to a computer system containing multiple GPUs, comprises: The mapping relationship establishment module is used to respond to the operator call request, determine the borrowed GPU to execute the operator, divide the output matrix of the operator into multiple tiles, determine a CTA for each tile, and establish a mapping relationship between the CTA and the tile; wherein, the input data required to execute each CTA is the data area corresponding to the tile mapped by the CTA in the input matrix of the operator; The CTA allocation module is used to determine at least one borrowed GPU whose real-time resource status meets the borrowing conditions from the plurality of GPUs, determine the number of CTAs to be allocated to the borrowed GPU and each borrowed GPU based on the real-time resource status of the borrowed GPU and the determined borrowed GPU, and allocate each CTA among the borrowed GPU and each borrowed GPU based on the number of CTAs, wherein the number of CTAs allocated to a GPU is positively correlated with the size of the remaining free resources of the GPU; The CTA execution module is used for the borrowing GPU and each borrowing GPU to execute their assigned CTAs and write their respective calculation results into the corresponding positions in the output matrix according to the mapping relationship.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1-9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1-9.