GPU resource allocation method, system, electronic device and storage medium

Abstract

The present application relates to a GPU resource allocation method, system, electronic device and storage medium, wherein the method comprises: when the AI ​​application container is running, monitoring the initial process's call operation on the GPU interface, generating a GPU resource request, and the scheduler generates the first GPU resource group configuration information based on this request, which includes the GPU configuration information of each process node and the number of process nodes, and starts the initial process as multiple job processes, the number of which matches the process nodes, and determines the mapping relationship. Based on the mapping relationship, a remote call link is established, and the request is transmitted to the GPU device for execution. When the task is completed, the scheduler releases the GPU resources occupied by the container. Through this application, the problems of low utilization and complex multi-node resource management caused by static binding and release delays in traditional GPU resource management are solved, the utilization rate is significantly improved, and the programming complexity of multi-machine and multi-card scenarios is simplified.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a GPU resource allocation method, system, electronic device, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI), especially the exponential growth in the scale of deep learning models (such as large language models and multimodal models), the demand for GPU computing power is exploding. With the convergence of cloud and edge computing, GPU resources have become the core infrastructure supporting AI training, inference, real-time rendering, and other scenarios. However, traditional GPU resource management models are gradually exposing systemic bottlenecks in scenarios such as surging elastic computing demands, high concurrency for short tasks, and distributed training across multiple machines and multiple GPUs.

[0003] Existing technologies mainly include GPU resource management based on static pre-allocation (such as Kubernetes DevicePlugin), strong container binding (such as Pod lifecycle-associated resources), virtual segmentation of video memory and computing resources (such as video memory sharding and time slice rotation), and master node scheduling (such as cluster resource aggregation and distribution). However, the following problems still exist: static pre-allocation leads to idle and wasted resources; container binding relies on passive release, and the recovery delay is significant; virtual segmentation introduces performance loss and complex management; master node scheduling cannot respond to dynamic demands in real time, and lacks a multi-process cross-node collaboration mechanism, resulting in low resource utilization and high complexity in multi-machine and multi-GPU task development.

[0004] Currently, no effective solution has been proposed to address the problems of low utilization and complex multi-node resource management caused by static binding and release delays in traditional GPU resource management in related technologies. Summary of the Invention

[0005] The embodiments of the present application provide a GPU resource allocation method, system, electronic device and storage medium to at least solve the problems of low utilization and complex multi-node resource management caused by static binding and release delays in traditional GPU resource management in related technologies.

[0006] In a first aspect, an embodiment of the present application provides a GPU resource allocation method, comprising:

[0007] When the AI ​​application container is running, monitoring the call operation of the initial process in the AI ​​application container to the GPU interface, and generating a GPU resource request in response to the monitored call operation;

[0008] Generate, via a scheduler, first GPU resource group configuration information for the initial process according to the GPU resource request; wherein the first GPU resource group configuration information includes GPU resource configuration information corresponding to each process node and the number of the process nodes;

[0009] According to the first GPU resource group configuration information, the initial process is started into multiple job processes, the number of the job processes matches the number of the process nodes, and a mapping relationship between the job processes and the process nodes is determined;

[0010] Based on the mapping relationship between the job process and the process node, a remote call link is established between the job process and the allocated GPU resource, and based on the remote call link, the GPU resource request is transmitted to the GPU device in the GPU resource to execute the computing task;

[0011] When it is monitored that the computing tasks of all the job processes are completed, the GPU resources occupied by the AI ​​application container are released through the scheduler.

[0012] In some embodiments, generating, by a scheduler, first GPU resource group configuration information for the initial process based on the GPU resource request includes:

[0013] The GPU resource request includes a scheduling strategy; the scheduling strategy includes at least one of a node number strategy, an affinity strategy, and a priority strategy;

[0014] Generate first GPU resource group configuration information for the initial process via the scheduler according to the scheduling policy.

[0015] In some embodiments, determining the mapping relationship between the job process and the process node includes:

[0016] monitoring a call operation on a GPU interface by each of the job processes in the AI ​​application container, and generating the GPU resource request corresponding to the initial process when the call operation is detected;

[0017] generating, via the scheduler, second GPU resource group configuration information for the job process according to the GPU resource request; wherein the second GPU resource group configuration information is the same as the first GPU resource group configuration information;

[0018] Based on the second GPU resource group configuration information, a mapping relationship between the job process and the process node is determined.

[0019] In some embodiments, establishing a remote call link between the job process and the allocated GPU resource based on the mapping relationship between the job process and the process node includes:

[0020] A communication protocol type is selected via the process node based on the acquired network environment information, and a remote call link is established between the job process and the allocated GPU resources based on the communication protocol type and a mapping relationship between the job process and the process node.

[0021] In some embodiments, establishing a remote call link between the job process and the allocated GPU resource based on the mapping relationship between the job process and the process node, and transmitting the GPU resource request to a GPU device in the GPU resource to execute a computing task based on the remote call link includes:

[0022] Based on the node information in the first GPU resource group configuration information, generating a unique corresponding GPU backend processing process via a node agent process on the process node corresponding to each job process;

[0023] A remote call link is established between the job process and the GPU back-end processing process. Based on the remote call link, the GPU resource request is transmitted to the process node corresponding to the job process. Then, the GPU resource request is transmitted to the GPU device in the GPU resource via the GPU back-end processing process on the process node.

[0024] In some embodiments, determining the mapping relationship between the job process and the process node includes:

[0025] A unique process node is allocated to each job process through a shared memory and semaphore mechanism, and a mapping relationship between the job process and the process node is determined.

[0026] In some embodiments, the method further comprises:

[0027] When it is detected that the AI ​​application container exits abnormally, the computing task of the job process is forcibly terminated and the GPU resources occupied by the AI ​​application container are released.

[0028] In a second aspect, an embodiment of the present application provides a GPU resource allocation system, including: a remote call hijacking library, a scheduler, and a process node;

[0029] A remote call hijacking library, configured to monitor, when an AI application container is running, calls to a GPU interface by an initial process in the AI ​​application container, and generate GPU resource requests in response to the monitored calls;

[0030] A scheduler, configured to generate first GPU resource group configuration information for the initial process according to the GPU resource request; wherein the first GPU resource group configuration information includes GPU resource configuration information corresponding to each process node and the number of the process nodes;

[0031] The scheduler is further configured to release the GPU resources occupied by the AI ​​application container via the scheduler when detecting that the computing tasks of all the job processes have ended;

[0032] The process node is used to establish a remote call link between the job process and the allocated GPU resource based on the mapping relationship between the job process and the process node, and based on the remote call link, transmit the GPU resource request to the GPU device in the GPU resource to perform the computing task.

[0033] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the GPU resource allocation method as described in the first aspect above is implemented.

[0034] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the GPU resource allocation method described in the first aspect above is implemented.

[0035] Compared with related technologies, the GPU resource allocation method provided in the embodiment of the present application solves the problems of low utilization and complex multi-node resource management caused by static binding and release delays in traditional GPU resource management through dynamic on-demand allocation, instant release mechanism and virtualization aggregation of multi-node GPU resources, significantly improves utilization and simplifies the programming complexity of multi-machine and multi-card scenarios.

[0036] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 1 is a hardware structure block diagram of a terminal of a GPU resource allocation method according to an embodiment of the present invention;

[0039] Figure 2is a flowchart of a GPU resource allocation method according to an embodiment of the present application;

[0040] Figure 3 This is a schematic diagram of the process of initiating the first GPU call during the initial process in an AI application according to an embodiment of the present application;

[0041] Figure 4 This is a schematic diagram of a process in an AI application calling a GPU according to an embodiment of the present application;

[0042] Figure 5 This is a schematic diagram of using multiple nodes and multiple cards in multiple processes in an AI application according to an embodiment of the present application;

[0043] Figure 6 This is a schematic diagram of the process of starting multiple processes by an AI application according to an embodiment of the present application;

[0044] Figure 7 This is a schematic diagram of the process of launching a GPU backend processing process according to an embodiment of the present application;

[0045] Figure 8 is a flowchart of a GPU backend processing process according to an embodiment of the present application;

[0046] Figure 9 2 is a schematic diagram of the mapping relationship between the GPU subscript in the AI ​​application process and the GPU subscript in the GPU backend processing process according to an embodiment of the present application;

[0047] Figure 10 This is a flow chart of a scheduler reclaiming GPU resources for an AI application according to an embodiment of the present application;

[0048] Figure 11 This is a schematic diagram of the deployment form of the GPU resource allocation method according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means and should not be understood as the contents disclosed in the present application being insufficient.

[0050] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0051] Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "an," "the," and similar expressions used herein do not denote limitations on quantity and may refer to either the singular or the plural. The terms "comprise," "include," "have," and any variations thereof, used herein, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules (units) is not limited to the listed steps or units but may also include steps or units not listed, or may include other steps or units inherent to the process, method, product, or apparatus. The terms "connected," "connected," "coupled," and similar expressions used herein are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used herein, "plurality" means greater than or equal to two. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" may mean: A exists alone; A and B exist simultaneously; or B exists alone. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0052] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of a GPU resource allocation method according to an embodiment of the present invention. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1Different configurations shown.

[0053] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the GPU resource allocation method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0054] Transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0055] This embodiment provides a GPU resource allocation method. Figure 2 is a flowchart of a GPU resource allocation method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0056] Step S201: When the AI ​​application container is running, monitor the call operation of the initial process in the AI ​​application container to the GPU interface, and generate a GPU resource request in response to the monitored call operation;

[0057] A GPU hijacking library (such as the CUDA hijacking library) is embedded in the AI ​​application container. The front-end library hijacks the standard GPU API through a dynamic link library, monitoring the job process's calls to the GPU interface in real time. Upon detecting the first GPU API call (such as memory allocation or kernel launch), the hijacking library instantiates a client singleton, generates a GPU resource request, and submits it to the scheduler (Rmanager). The client singleton encapsulates container information (such as the Pod namespace and scheduling policy) into a GPU resource request and sends it to the scheduler (Rmanager) to initiate the dynamic allocation process. The GPU resource request includes the AI ​​application's GPU type, quantity, memory requirements, scheduling policy, and priority. This resource request is triggered only when the AI ​​application actually requires GPU computing (rather than pre-binding it at container launch). This avoids idleness caused by static resource reservations. By synchronizing resource requests with task execution, the latency associated with traditional "startup wait-to-resource allocation" approaches is reduced. Supporting interception of mainstream GPU APIs such as CUDA and OpenCL, the solution is compatible with various deep learning frameworks, enhancing its flexibility.

[0058] Step S202: Generate, via the scheduler, first GPU resource group configuration information for the initial process according to the GPU resource request; wherein the first GPU resource group configuration information includes GPU resource configuration information corresponding to each process node and the number of process nodes;

[0059] The scheduler receives GPU resource requests from the hijacked library Client singleton. Based on the real-time resource pool status, the scheduler screens qualified nodes and GPUs and generates the first GPU resource group configuration information. This information includes the assigned physical node IP / hostname and corresponding GPU resources (i.e., the GPU resource configuration information corresponding to each process node), the number of process nodes, and the correspondence between the AI ​​application logical GPU index and the actual physical GPU UUID. The scheduler then marks the assigned GPU as "occupied" to prevent duplicate requests from other tasks. Resource requests from all processes within the same container return the same resource group configuration, ensuring consistency in multi-process collaboration. This step transforms complex multi-node resource coordination into a policy matching problem through flexible scheduling strategies (such as the number of nodes and affinity), without the need for manual intervention. The optimal GPU combination is allocated based on real-time resource status, avoiding resource fragmentation caused by static allocation (such as idle high-end GPUs). The scheduler dynamically aggregates multi-node GPU resources into a virtual resource pool based on requests triggered by the hijacking library, so that users do not need to be aware of the physical GPU location. AI applications can operate remote multi-node GPUs like using local single-node GPUs, and all processes in the same AI container obtain the same resource group, namely the first GPU resource group configuration information. Developers do not need to deal with multi-node heterogeneous environments, and the code can degenerate into a single-machine multi-card mode. For example, distributed training tasks only need to start processes with the same number of nodes, and each process is automatically bound to the GPU on a unique node.

[0060] Step S203: According to the first GPU resource group configuration information, the initial process is started into multiple job processes, the number of job processes matches the number of process nodes, and a mapping relationship between the job processes and the process nodes is determined;

[0061] The initial process parses the first GPU resource group configuration information returned by the scheduler, extracts the number of allocated process nodes (for example, if the resource group contains three nodes), and dynamically generates the same number of job processes based on the number of process nodes. For example, if the resource group is distributed across two nodes, two processes are launched, one for each node. The initial process (the first process launched) writes the node number to a shared file (such as / etc / client_node_num) for subsequent processes to read. Other processes, through polling or event monitoring, start after detecting that the configuration file is ready. The number of job processes launched is equal to the number of process nodes minus one. The node list is stored in shared memory, and semaphores are used to implement mutual exclusive access between processes, ensuring that each process is assigned a unique process node. For example, if the resource group contains nodes A, B, and C, process 1 is bound to node A, process 2 to node B, and process 3 to node C. Each process sets an environment variable at startup to identify the node and GPU resources it is bound to. The environment variable indicates the number of GPUs on the node (for example, if node A has two GPUs, it is set to 2). Each process is restricted to accessing only the GPU resources of the current node through environment variables or API parameters to avoid cross-node competition. The logical GPU index is mapped to the UUID of the physical GPU, and transparent conversion is achieved through the hijacking library. This step does not require the developer to manually specify the number of processes. The system automatically generates corresponding processes based on the number of nodes in the resource group, reducing programming complexity. Each process only occupies the GPU resources of the node to which it is bound, avoiding idle resources on other nodes. The resources between job processes do not interfere with each other. A single node failure only affects the corresponding process. The task can continue to be executed through other nodes, and all processes in the same container obtain the same resource view, ensuring the consistency of multi-process collaboration. Developers do not need to write cross-node communication code. The system automatically handles multi-node communication through process binding and remote calls.

[0062] Step S204: establishing a remote call link between the job process and the allocated GPU resource based on the mapping relationship between the job process and the process node, and transmitting the GPU resource request to the GPU device in the GPU resource to execute the computing task based on the remote call link;

[0063] Among them, the physical node and GPU device UUID corresponding to each job process are determined based on the first GPU resource group configuration information returned by the scheduler. The binding relationship between the process and the node is allocated through shared memory or environment variables (such as process 1 → GPU0 of node A, process 2 → GPU1 of node B). Each job process requests to launch the backend processing process (Process) through the Proxy proxy process on the node to which it is bound. The Proxy launches the Process process on the target node, which is responsible for receiving remote GPU API call requests and forwarding them to the local GPU hardware driver. The Client object establishes a remote call link with the Process based on the obtained Processendpoint information, which is usually implemented through a network communication protocol (such as TCP, RDMA or UDS). All GPU API calls (such as video memory allocation, kernel function startup) and associated data (such as input tensors) are transmitted to the Process through the remote call link, and the Process calls the local GPU driver to execute the computing task. In this step, GPU resources are dynamically allocated and remote call links are established according to the actual needs of the job process, achieving efficient utilization of GPU resources and avoiding resource waste and under-allocation caused by static allocation. The establishment of remote call links and the rapid transmission of GPU resource requests improve the response speed and execution efficiency of tasks, providing users with a better user experience and promoting the widespread promotion and application of AI applications.

[0064] In step S205, when it is detected that the computing tasks of all job processes are completed, the GPU resources occupied by the AI ​​application container are released through the scheduler.

[0065] The scheduler monitors all job processes in the AI ​​application container in real time, confirming whether the processes are still running through heartbeat monitoring and process status queries. When the scheduler detects that all job processes for an AI application have exited or completed their computational tasks, it determines that the AI ​​application's computational tasks have ended. Once this determination is made, the scheduler immediately triggers the GPU resource release process. The scheduler notifies the proxy processes on the relevant nodes that the GPU resources occupied by the AI ​​application container need to be released. Upon receiving the release notification, the node proxy processes are responsible for terminating or clearing the backend processing processes associated with the AI ​​application, releasing the occupied GPU resources. Specifically, the scheduler marks the released GPU resources as unused and adds them to the available resource pool. The resource collection module updates the system's GPU resource status in real time, ensuring that the scheduler can make subsequent resource allocations based on the latest resource status. The scheduler also implements a garbage collection mechanism. When it detects the exit of an AI application container, it notifies the node proxy processes to kill the corresponding backend processing processes and release the associated resources. This ensures that even if an AI application container exits abnormally, the occupied GPU resources are promptly reclaimed, preventing resource leaks. This step ensures dynamic management and efficient utilization of system resources by immediately releasing the GPU resources occupied by the AI ​​application container, avoiding idle resources and waste, and improving overall resource utilization. The immediate release of resources enables the system to support more concurrent AI application tasks; when a new AI application requires GPU resources, the system can quickly allocate previously released resources to meet the needs of the new task; the dynamic allocation and immediate release of resources reduce the need for manual intervention and reduce operation and maintenance costs. The system can automatically manage the allocation and release of GPU resources, reducing the burden on operation and maintenance personnel; supplementary measures such as the garbage collection mechanism enhance the stability of the system. Even if the AI ​​application container exits abnormally, the system can ensure that the relevant resources are recovered in time, avoiding impact on other parts of the system.

[0066] Through the above steps, the present application solves the problem through dynamic allocation and immediate release strategies. In step S201, the system monitors the call operation of the initial process to the GPU interface in the AI ​​application container in real time and generates a GPU resource request. This dynamic monitoring mechanism avoids static binding of resources. In step S202, the scheduler generates the first GPU resource group configuration information for the initial process according to the request. This information specifies the GPU resource configuration and quantity corresponding to each process node in detail, thereby realizing refined allocation of resources. Step S203 splits the initial process into multiple job processes according to this configuration information, ensuring that the number of job processes matches the number of process nodes, thereby simplifying multi-node resource management. Step S204 realizes remote execution of computing tasks by establishing a remote call link between the job process and the allocated GPU resources, thereby improving resource utilization efficiency. Finally, in step S205, the system monitors the completion of all job process tasks and immediately releases the GPU resources occupied by the AI ​​application container, thereby avoiding waste caused by delayed resource release. Compared with traditional methods, this application significantly improves the utilization of GPU resources and simplifies multi-node resource management through dynamic allocation, multi-process management and instant release strategy, effectively solving the problems of low utilization and complex management in traditional GPU resource management.

[0067] In some embodiments, generating, via a scheduler, first GPU resource group configuration information for an initial process based on a GPU resource request includes:

[0068] The GPU resource request includes a scheduling strategy; the scheduling strategy includes at least one of a node number strategy, an affinity strategy, and a priority strategy;

[0069] The scheduler generates first GPU resource group configuration information for the initial process according to the scheduling policy.

[0070] The scheduler supports flexible combinations of three types of policies: node count policy, affinity policy, and priority policy. Scheduling policies include, but are not limited to, combinations of these three policies. Additional scheduling policies can be expanded in the future to flexibly combine with these three policies. Node count policies include: LEAST_NODES, which prioritizes GPU resources to the fewest nodes (e.g., concentrating four GPUs on two nodes) to reduce cross-node communication overhead; MOST_NODES, which distributes GPU resources to the most nodes (e.g., allocating four GPUs to four nodes) to improve task fault tolerance; and SPECIFY_NODES, which allocates resources based on a specified number of nodes (e.g., forcing allocation to three nodes). Affinity policies include: CLIENT_AFFINITY, which prioritizes GPU resources on the same node as the AI ​​application container to reduce data transmission latency; and CLUSTER_AFFINITY, which allocates resources only within a specified cluster (e.g., only within a "High-Performance Computing Cluster"). Priority policies dynamically adjust the order of resource allocation based on task priority to ensure that high-priority tasks receive resources first. Different types of policies can be used in combination (for example, LEAST_NODES + CLUSTER_AFFINITY), and policies of the same type are mutually exclusive (for example, LEAST_NODES and MOST_NODES cannot be selected at the same time).

[0071] Based on the determined scheduling policy, the scheduler queries the currently available GPU resource information in the system, including the GPU type, quantity, memory size, and node location. Based on the scheduling policy and available resource information, the scheduler generates the first GPU resource group configuration information for the initial process. This configuration information specifies the GPU resource configuration information for each process node, including the GPU card UUID, node location, and the number of process nodes. The scheduler returns this generated first GPU resource group configuration information to the AI ​​application container, which then launches multiple job processes and allocates GPU resources.

[0072] By supporting multiple scheduling strategies, this embodiment enables the scheduler to allocate GPU resources more flexibly and make dynamic adjustments based on actual needs, thereby improving resource utilization; the node number strategy and affinity strategy in the scheduling strategy enable the scheduler to consider resource allocation issues in a multi-node environment, simplifying the complexity of multi-node resource management; the priority strategy allows users to assign different priorities to different AI applications based on the importance and urgency of the applications, ensuring that critical applications can obtain GPU resources first; the scheduler can dynamically generate GPU resource group configuration information based on the user-specified scheduling strategy, enabling the system to adapt to resource allocation requirements in different scenarios, enhancing the system's flexibility and scalability; by providing flexible scheduling strategies, users can more finely control the allocation of GPU resources, thereby optimizing the execution efficiency and performance of AI applications and improving user experience.

[0073] In some embodiments, determining a mapping relationship between a job process and a process node includes:

[0074] Monitors the calls to the GPU interface by each job process in the AI ​​application container. When a call is detected, it generates a GPU resource request corresponding to the initial process.

[0075] generating, via the scheduler, second GPU resource group configuration information for the job process according to the GPU resource request; the second GPU resource group configuration information being the same as the first GPU resource group configuration information;

[0076] Based on the second GPU resource group configuration information, a mapping relationship between the job process and the process node is determined.

[0077] During the operation of the AI ​​application container, each job process continuously monitors its GPU interface calls. This monitoring mechanism ensures that the system can detect the job process's demand for GPU resources in real time. When a job process detects a GPU interface call, the system generates a GPU resource request for the job process, corresponding to the initial process. This request includes information such as the type and quantity of GPU resources required by the job process. After receiving the GPU resource request from the job process, the scheduler generates a second GPU resource group configuration for the job process based on the request and the currently available GPU resources in the system. This second GPU resource group configuration is identical to the first GPU resource group configuration generated for the initial process. Therefore, the GPU resource allocation results are consistent for both the initial process and subsequently launched job processes, ensuring fair and consistent resource allocation. Based on this second GPU resource group configuration, the system determines which process node each job process should be assigned to and establishes a mapping between the job process and the process node. This mapping serves as the basis for subsequent remote GPU resource calls and computational task execution. This embodiment generates the same second GPU resource group configuration information as the initial process, ensuring that the GPU resource allocation results obtained by both the initial process and the subsequently started job processes are consistent. This avoids unfair resource allocation and improves the overall performance of the system. Since all job processes follow the same resource allocation strategy, the system can more simply manage and schedule GPU resources, reducing system complexity and improving management efficiency. By monitoring the job process's call operations on the GPU interface in real time and dynamically allocating GPU resources, the system can more effectively utilize GPU resources and avoid idle and wasted resources. Through a resource management method based on dynamic allocation and immediate release, the system can more flexibly respond to changes in resource requirements in different scenarios. At the same time, it also supports system scalability, allowing the system to easily add new job processes and GPU resources.

[0078] In some embodiments, establishing a remote call link between a job process and an allocated GPU resource based on a mapping relationship between the job process and the process node includes:

[0079] Through the process node, based on the acquired network environment information, the communication protocol type is selected, and based on the mapping relationship between the communication protocol type and the job process and the process node, a remote call link is established between the job process and the allocated GPU resources.

[0080] At each process node, the system obtains current network environment information, including but not limited to network bandwidth, latency, and support for specific network protocols (such as UDS, RDMA, TCP, UDP, etc.). Based on this acquired network environment information, the system adaptively selects the communication protocol most appropriate for the current network conditions. For example, if RDMA is supported between nodes and the network conditions are good, the system may select the RDMA protocol to reduce communication latency; if network conditions are poor, the system may select the more stable TCP protocol. Based on the selected communication protocol and the previously determined mapping between the job process and the process node, the system establishes a remote call link between the job process and the allocated GPU resources. This remote call link is implemented via a network communication protocol and allows the job process to transmit GPU API call requests and related data to a remote GPU device for execution. By adaptively selecting the communication protocol, this embodiment allows the system to dynamically adjust communication methods based on changes in the network environment, thereby optimizing communication efficiency and reducing communication latency. The system's ability to select the most appropriate communication protocol based on changes in the network environment helps enhance the system's stability and reliability under varying network conditions.

[0081] In some embodiments, based on the mapping relationship between the job process and the process node, a remote call link is established between the job process and the allocated GPU resource, and based on the remote call link, a GPU resource request is transmitted to a GPU device in the GPU resource to execute a computing task, including:

[0082] Based on the node information in the first GPU resource group configuration information, a unique corresponding GPU backend processing process is generated via a node agent process on a process node corresponding to each job process;

[0083] A remote call link is established between the job process and the GPU backend processing process. Based on the remote call link, the GPU resource request is transmitted to the process node corresponding to the job process. Then, through the GPU backend processing process on the process node, the GPU resource request is transmitted to the GPU device in the GPU resource.

[0084] Among them, based on the node information in the first GPU resource group configuration information, the system will identify which process node each job process should be assigned to. On each corresponding process node, through the node proxy process (Proxy), the system will dynamically generate a unique corresponding GPU backend processing process (Process). The GPU backend processing process is responsible for processing the GPU resource request from the job process and forwarding it to the actual GPU device. Between the job process and the generated GPU backend processing process, the system will establish a remote call link. This link is based on the network communication protocol and allows the job process to transmit the GPU resource request and related data to the GPU backend processing process. Once the remote call link is established, the job process can transmit the GPU resource request to the GPU backend processing process on its corresponding node through this link. After receiving the request, the GPU backend processing process will further forward the request to the actual GPU device to perform the computing task.

[0085] By dynamically generating GPU backend processing processes, this embodiment enables the system to more effectively utilize GPU resources, ensuring that each job process can obtain the required computing power, thereby improving overall resource utilization; the system can dynamically generate and manage GPU backend processing processes according to the needs of the job process, which enables the system to flexibly respond to changes in resource requirements in different scenarios; the node agent process is responsible for managing and scheduling the GPU backend processing processes, simplifying the system's management and maintenance of GPU resources; at the same time, the job process does not need to worry about the specific allocation and management details of the underlying GPU resources, but only needs to interact with the GPU backend processing process through remote call links.

[0086] In some embodiments, determining a mapping relationship between a job process and a process node includes:

[0087] A unique process node is allocated to each job process through shared memory and semaphore mechanisms, and the mapping relationship between the job process and the process node is determined.

[0088] During system startup, a shared memory area is pre-allocated to store information related to process node allocation, such as process node status and assigned job process IDs. The shared memory access permissions are set to allow read and write operations by processes involved in job process allocation and management (such as the scheduler and node agent). Multiple semaphores are created to control concurrent access to the shared memory. The allocation process is as follows: A shared memory area is created within the AI ​​application container to store a node allocation status table. The node allocation table is initialized to [0, 1, ..., n-1] (n is the number of nodes in the resource group), with each node marked as "unallocated." Semaphores are initialized as mutex locks to ensure atomic access to the shared memory. This means that only one process can access and modify the index array in the shared memory at any given time, preventing conflicts when multiple processes attempt to allocate node indexes simultaneously. Upon startup, each process attempts to acquire the semaphore lock for exclusive access to the shared memory. After the first process acquires the semaphore, it traverses the node allocation table in shared memory, finds the first "unallocated" node (e.g., subscript 0), marks it as "allocated," and records the corresponding node information (such as IP address and GPU UUID). Subsequent processes repeat this process, assigning the remaining node indices (e.g., subscript 1, 2, and so on) until all nodes are assigned. The semaphore ensures that only one process modifies the allocation table at a time, preventing duplicate node assignments. Each process sets environment variables (e.g., NODE_INDEX = 0, GPU_UUIDS = "GPU-xxx, GPU-yyy") based on the assigned node indices to identify the node and GPU resources it is bound to. The logical GPU index is mapped to the physical GPU UUID via a hijacking library, ensuring that the process only accesses the GPU resources of the bound node. This ensures that the GPU resources of the corresponding node in the resource group returned by the scheduler are correctly used. If a process exits abnormally, the proxy notifies the scheduler to release the occupied node and resets the node's status to "unallocated" in shared memory. As the number of nodes in the resource group changes, the shared memory area can be dynamically expanded to accommodate the new node allocation requirements.

[0089] This embodiment uses shared memory and semaphore mechanisms to ensure that each job process can be assigned a unique node index, avoiding conflicts caused by multiple processes trying to use the same node resources; since each process can be accurately allocated node resources and the resources can be released immediately after use, the overall utilization of GPU resources is improved; the shared memory and semaphore mechanism provides a simple and effective way to manage the mapping relationship between job processes and process nodes, reducing the complexity of resource management; using semaphores as mutex locks ensures the atomicity of shared memory access, avoids data inconsistency problems caused by concurrent access, and thus enhances the stability of the system.

[0090] In some embodiments, the method further comprises:

[0091] When an abnormal exit of the AI ​​application container is detected, the computing tasks of the job process are forcibly terminated and the GPU resources occupied by the AI ​​application container are released.

[0092] The system monitors the status of AI application containers, for example, using APIs provided by Docker or Kubernetes, and regularly checks the container's running status. When a container's status changes to "Exited" or "Abnormally Terminated," subsequent processing is triggered. In addition to container-level monitoring, the system also monitors the status of key job processes within the container. If these processes terminate unexpectedly or remain unresponsive for an extended period, this is considered an abnormal container exit. The system analyzes the log files of the container and job processes for signs of abnormal exits or errors. If the logs contain critical errors or exceptions, the system determines that the container has abnormally exited. Once an AI application container exits abnormally, the system immediately sends a termination signal to all remaining job processes within the container. If a process does not respond within a specified time, a forced termination signal (such as SIGKILL) is sent. If the system integrates with job management tools (such as Kubernetes), the APIs provided by these tools can be used to terminate the job processes associated with the abnormal container. The system notifies the GPU resource scheduler (such as Rmanager) of the abnormal exit of the AI ​​application container, which then marks the associated GPU resources as unused. The scheduler starts the garbage collection mechanism, cleans up the resource allocation records related to the abnormal container, and releases these GPU resources for use by other containers. After receiving notification from the scheduler, the proxy process on the node is responsible for killing any backend processing process related to the abnormal container and reporting the task termination information to the scheduler. This embodiment releases the GPU resources occupied by the abnormal container in a timely manner, allowing the system to utilize these resources more efficiently and avoid resource waste; forcibly terminating the abnormal job process can prevent system instability or crashes caused by the abnormal process, ensuring the overall stability of the system; the automated exception handling and resource release mechanism simplifies the complexity of resource management and reduces the need for manual intervention; users do not need to worry about the long-term resource occupation caused by abnormal containers. The system can automatically handle these abnormal situations and improve the user experience. This mechanism, combined with the dynamic resource allocation strategy, enables the system to flexibly adjust resource allocation according to actual needs, further improving resource utilization and system performance.

[0093] This embodiment also provides a GPU resource allocation system, including: a remote call hijacking library, a scheduler, and a process node;

[0094] A remote call hijacking library, used to monitor the initial process in the AI ​​application container's call operations on the GPU interface when the AI ​​application container is running, and generate GPU resource requests in response to the monitored call operations;

[0095] The scheduler is configured to generate first GPU resource group configuration information for the initial process according to the GPU resource request; wherein the first GPU resource group configuration information includes GPU resource configuration information corresponding to each process node and the number of process nodes;

[0096] The scheduler is also used to release the GPU resources occupied by the AI ​​application container when the computing tasks of all job processes are completed.

[0097] The process node is used to establish a remote call link between the job process and the allocated GPU resources based on the mapping relationship between the job process and the process node, and transmit the GPU resource request to the GPU device in the GPU resource to execute the computing task based on the remote call link.

[0098] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0099] In a preferred embodiment of this application, the AI ​​application container or job pod (collectively referred to as a container) is not pre-bound to GPU hardware resources. When the job process in the container actually calls the GPU API, it indicates that it has begun to use GPU resources. At this moment, the GPU resources are dynamically allocated through the scheduler, and the GPU computing of the API application is executed through the GPU remote call. When the AI ​​application execution ends, the GPU resources are immediately released. The preferred embodiment of this application includes the following steps:

[0100] Step S301, Dynamic GPU Allocation: AI Application Container Resource Requirements: Before or after launching an AI application container, users or developers can declare the container's GPU resource requirements through the container configuration file or a specific API. GPU resource requirements include, but are not limited to, the required GPU type, quantity, GPU memory size, and specific scheduling strategies. For example, for some deep learning training tasks, users can explicitly specify in the configuration file that they require four NVIDIA A100 devices, and require that all four cards be on the same node.

[0101] Step S302: API call interception: While the AI ​​application is running in the container, the container's job processes are continuously monitored, and a pre-set monitoring mechanism is used to capture any GPU API calls made by the job processes. This monitoring mechanism, based on system call interception technology, intercepts GPU-related API call requests from the job processes at the API level. Once such a call is detected, the interception library instantiates a client singleton object upon the first such call in the process, which immediately triggers the subsequent GPU resource request process.

[0102] Step S303: GPU resource application: Figure 3 FIG. 1 is a schematic diagram of the process of initiating the first call to the GPU in an AI application according to an embodiment of the present application. Figure 3 As shown, when the hijacking library detects that the AI ​​application process calls the GPU API, the client object in the hijacking library initiates a resource request to the scheduler. The GPU resource request contains information about the AI ​​application, including but not limited to the pod namespace, name, scheduling policy, and priority. The scheduler pre-allocates GPU resources that meet the conditions based on the relevant information of the available GPU resources in the entire system. Each process in the AI ​​application will have the above-mentioned GPU resource application process, and the scheduling results obtained are the same. For example, if a container needs to use two cards and the number of nodes is specified as 2, two processes will be started in the container, and each process will obtain the scheduling results of the two cards when applying for resources. Obviously, the two cards are distributed on two nodes.

[0103] Figure 4 FIG. 1 is a flow chart of a process calling a GPU in an AI application according to an embodiment of the present application. Figure 4 As shown in the figure, after the client object requests a GPU resource group from the scheduler, the following steps are performed: determining how many processes the AI ​​application needs to launch, allocating a unique node to each process, specifying the number of GPU cards used by each process, and assigning the specific GPU UUID number. The number of processes to launch is written to a file, such as / etc / client_node_num; if the file already exists, it is ignored. After completing this step, subsequent processes can be launched. AllocateUniqueIndex allocates a unique node based on the number of nodes. One implementation method for reference is to allocate a unique node to each process using a combination of shared memory and semaphores within the AI ​​application container. The corresponding GPU array information, including the UUID, is also saved. This allows a process to use the GPU resources of a node in the resource group returned by the scheduler. The environment variable GPU_NUMS is written to indicate the number of GPUs available to this process. This number is equal to the number of GPU cards allocated on the node assigned in the previous step. Figure 5Schematic diagram of using multiple nodes and multiple cards in multiple processes in an AI application according to an embodiment of the present application. Figure 5 Specifically, the above steps use a scheduler response with the resource group {node1: 1 GPU card; node2: 2 GPU cards; node3: 3 GPU cards} as an example. In the first step, the number 3 is written to the / etc / client_node_num file, indicating that three processes need to be started. In the second step, each process is assigned a unique node and stores the GPU information in its client object for management; index is the index subscript. Third, each process writes the corresponding environment variable GPU_NUMS, which contains the number of GPUs corresponding to the node assigned in step 2.

[0104] Step S304: Scheduling Strategy Execution: The scheduler allocates appropriate GPU resources to the AI ​​job process based on the resource request initiated by the hijacking library and the scheduling strategy. The resource group returned by the scheduler includes GPU card information and corresponding node information. The scheduler marks the card as allocated.

[0105] Scheduler scheduling policies include multiple categories, each of which can have multiple policies, and new scheduling policies can be expanded. The scheduler returns the same scheduling result for resource request requests initiated by all processes in the same container. Scheduler scheduling policies include multiple categories, each of which can have multiple policies. Scheduling policies of different categories can overlap, but are mutually exclusive within a category, meaning only one policy can be specified within a category. Specifically, Category 1: Number of nodes: LEAST_NODES (minimum number of nodes possible), MOST_NODES (maximum number of nodes possible), and SPECIFY_NODES (specific number of nodes). Category 2: CLIENT_AFFINITY (affinity) prioritizes GPU resources allocated to the node where the AI ​​application resides, while CLUSTER_AFFINITY (specifically, GPU resources allocated to a specific cluster). Category 3: Priority.

[0106] Step S305: Decision on which node to use for multiple processes in the AI ​​application container: Figure 6 Schematic diagram of the process of starting multiple processes by an AI application according to an embodiment of the present application. Figure 6 As shown, the scheduler returns the number of nodes on which GPU resources are distributed. AI applications need to launch the same number of processes, each using the GPU resources of a node. The resource information returned by the scheduler includes GPU resources on multiple nodes. Each process corresponds to a node, so the number of GPUs on each node may not be the same. A unique node is assigned to each process through a combination of shared memory and semaphores within the AI ​​application container.

[0107] Step S306: Apply for backend processing: Figure 7 Schematic diagram of the process of starting a GPU backend processing process according to an embodiment of the present application. Figure 7 As shown, each process in an AI application dynamically starts a process through the corresponding node proxy process. The proxy provides an RPC server for starting and destroying processes, detecting RDMA network card devices, and other functions. It uses the TCP protocol and provides methods such as Fork, Join, and Detect. Fork supports multiple protocol parameters, including USE_TCP, USE_RDMA, and USE_UDS.

[0108] When the client object applies for specific GPU resources, it will know which specific GPU cards on which nodes to use. In the process of an AI application, the client object applies for a GPU backend processing process to the proxy process of the node based on the GPU node information obtained in the above steps. After receiving the request, the Proxy returns the endpoint information of the Process. The client object applies for the backend processing process to the Proxy process on the node through an RPC call. By detecting the network status of the node where the client is located and the node where the Proxy process is located, it adaptively confirms the type of network communication protocol, including UDS / RDMA / TCP / UDP, etc. Among them, Figure 8 : is a flowchart of the GPU backend processing process according to an embodiment of the present application, such as Figure 8 As shown in the figure, the specific steps of a backend processing process include the following: Checking whether its own IP address and the proxy are on the same node. If so, FORK uses the USE_UDS parameter. Upon receiving this parameter, the proxy creates an RPC service based on a Unix domain socket and returns the path of the Unix domain socket to ensure optimal RPC call performance on the same node. The client detects whether there is an available RDMA device on its host. If so, it obtains the IP address of the RDMA device and uses the Detect method to detect whether the proxy node has an available RDMA device. If so, the proxy node's RDMA network card IP address is returned. The client then checks whether the RDMA network is reachable. Based on the client's detection results, the fork method is executed using the USE_RDMA or USE_TCP parameter and returns the endpoint of the rpc-server.

[0109] Step S307: Establish a remote call link: Figure 9 : is a schematic diagram of the mapping relationship between the GPU subscript in the AI ​​application process and the GPU subscript in the GPU backend processing process according to an embodiment of the present application, such as Figure 9 As shown in the figure, based on the endpoint information of the backend processing process obtained in the previous step, the client establishes a connection with the process and saves the connection information. All subsequent calls will transmit the job process's API call request and related data to the remote process through the network communication protocol, and then forward it to the GPU device.

[0110] Step S308: Task reporting, or health check: The node proxy reports the launched process information and AI application process information to the scheduler. After the AI ​​application process begins running, the scheduler performs a health check. If all AI application processes have completed execution, the scheduler immediately triggers the GPU resource release step.

[0111] Step S309: GPU immediate release: Figure 10 This is a flowchart of the scheduler reclaiming GPU resources for an AI application according to an embodiment of the present application, such as Figure 10 As shown in the figure, the node proxy process monitors all backend processing processes it has launched and immediately reports any process exits to the scheduler. If the scheduler's health check detects that all backend processing processes for an AI application have exited, the GPU resources occupied by the AI ​​application are immediately marked as unused. Furthermore, the scheduler has a garbage collection mechanism. When an AI application container exits, it also notifies the node proxy process and kills the corresponding processing process.

[0112] Step S310, Resource Status Update: The resource collection module updates the status of the GPU resources in the system in real time and incorporates the released resources into the available resource pool. The updated resource information can be used by subsequent job processes for resource allocation, ensuring dynamic management and efficient utilization of system resources.

[0113] In this method, AI applications support multi-process execution, and the specific number of processes started is strongly related to the scheduling strategy used. The number of processes started is equal to the number of nodes on which the GPU resources used are distributed. For example, if three cards are distributed on two nodes, two processes are started. For example, if the scheduling strategy specifies three nodes, the application can start three processes to perform tasks according to the strategy. When the minimum or maximum number of nodes is specified for the scheduling strategy, the application cannot know in advance how many processes to start. By starting a process first, the subsequent processes wait for the first process to be written to the configuration file before making a decision. After the first process applies for GPU resources, it knows exactly how many processes to start, and writes this information into the configuration file.

[0114] The preferred embodiment of the present application includes a system for on-demand use and immediate release of GPU resources based on dynamic allocation, the system including: a remote call hijacking library, a scheduler (Rmanager), a node-level GPU resource management module Ragent, a node proxy process (Proxy), and a GPU backend processing process (Process).

[0115] The remote call hijacking library includes a front-end library, a client library, and a back-end library. Each GPU-dependent library corresponds to a front-end hijacking library and a back-end hijacking library. The front-end hijacking library also includes a client library. The client library uses a singleton design pattern. In a front-end business process, all GPU hijacking front-end libraries use the same client object. Taking CUDA as an example, the implementation logic of the hijacking library requires special handling of some device APIs: For APIs that obtain the number of GPU cards, such as cuDeviceGetCount cudaGetDeviceCount, the value of the environment variable GPU_NUMS written in the client object must be returned, indicating that the business process can use both cards. For device acquisition and setting APIs, such as CudaGetDevice CudaSetDevice CuDeviceGet, the GPU card index value in the business container is not equivalent to the index on the node where the actual card is located. When implementing these interfaces, the client sends the UUID that uniquely identifies the GPU to the back-end library, which converts it into the actual index of the GPU card on the node. When a client object is first instantiated, it requests GPU resources from Rmanager. Once the resources are obtained, information about how many nodes these GPU resources consist of is written to the container's environment variables or business container configuration file through a specific method. This information is used by the AI ​​application to determine how many processes to start. The remote call hijacking library includes front-end and back-end implementations. When a process first detects a GPU interface call request, it generates a GPU resource request and interacts with the scheduler to obtain the scheduling result. If the call is not the first time, the GPU call request is directly forwarded from the front-end to the back-end.

[0116] The scheduler (Rmanager) dynamically allocates GPU resources and monitors the task lifecycle. Rmanager dynamically allocates GPU resources at the container level. Processes within the same container receive the same resource request, and Rmanager returns the same result. This means that multiple processes within the container receive the same resource allocation when they start requesting resources. For an application, resource allocation can have the following results: Pending (cluster resources are insufficient, waiting); Scheduled (scheduling completed); Finished (run completed); and Terminated (abnormal termination).

[0117] Ragent, a node-level GPU resource management module, has a built-in Kubernetes device-plugin. Unlike the native NVIDIA device-plugin, Ragent does not implement the Allocate interface, so GPU cards are not strongly bound to specific containers, and GPU resources are independent of the pod or container lifecycle. It includes features such as resource discovery, identifying GPU cards on nodes and collecting GPU card information; registering GPU resource information with kubelete and creating resource identifiers; error handling and fault recovery, marking GPU cards as unavailable when they are unavailable. These capabilities are consistent with the native NVIDIA device-plugin. Unlike the native device-plugin, Ragent does not implement the Allocate interface, so GPU cards are not strongly bound to specific containers, and GPU resources are independent of the pod or container lifecycle.

[0118] The Proxy process, acting as a proxy on the node, receives requests such as fork, join, and detect from the client, and accordingly starts or kills processes. When starting or killing a process, it reports task information to Rmanager. This task information includes the pod name, container name, and process ID of the client and server. Rmanager understands which cards are running which tasks.

[0119] The GPU backend processing process (Process) is the backend implementation of the GPU card remote call API hijacking. The Process provides RPC services and calls the implementation of the hijacking library backend code after receiving the call request from the hijacking library frontend.

[0120] AI applications, which refer to AI applications running within this patented system, require a small amount of code adaptation. Specifically, the AI ​​application adaptation point requires running the AI ​​application using a multi-process approach, where the number of processes started depends on the scheduling results returned by the scheduler. Using TensorFlow and PyTorch as examples, the AI ​​application adaptation point involves using multiprocessing.Process to start the application, deleting the / etc / client_node_num file, starting one process first, and then starting subsequent processes based on the status of the first process during startup. A recommended approach is to write the required number of processes into the / etc / client_node_num file in the first process started. Other processes wait for the / etc / client_node_num file to be ready, then read its value, num. If it is greater than 1, they start num-1 processes. Another recommended approach for AI applications is to specify the CLIENT_NODE_NUM environment variable to indicate the number of nodes on which the application will run. Rmanager supports scheduling by specifying a specific number of nodes, and returns GPU resources for the specified number of nodes. The application can then simply create CLIENT_NODE_NUM processes for processing.

[0121] in addition, Figure 11 FIG. 1 is a schematic diagram of a deployment form of a GPU resource allocation method according to an embodiment of the present application. Figure 11 As shown below, the various components are as follows: AI applications can be deployed on any node in the cluster; ragent: Dameonset deployment, on nodes with GPU cards; proxy: Dameonset deployment, on nodes with GPU cards; Rmanager: Active / standby deployment, with only the active node providing services; RDMA network connectivity is optional between all nodes in the cluster. This is one deployment method. Of course, this is not limited to this single method; multi-cluster deployment is also supported, provided that the network between the clusters is reachable.

[0122] This preferred embodiment achieves the aggregation of multi-node GPU resources, meaning that conventional multi-machine, multi-GPU scenarios can be reduced to a single-machine, multi-GPU scenario. For example, if an AI application requires 10 GPUs, and there are exactly 10 GPUs available in the cluster, distributed across four nodes, the user's AI application does not need to specify specific nodes. Instead, it can be used as a single machine with multiple GPUs, simply launching four processes to run the task.

[0123] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0124] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0125] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0126] S1, when the AI ​​application container is running, monitoring the call operation of the initial process in the AI ​​application container to the GPU interface, and generating a GPU resource request in response to the monitored call operation;

[0127] S2, generating, via the scheduler, first GPU resource group configuration information for the initial process according to the GPU resource request; wherein the first GPU resource group configuration information includes GPU resource configuration information corresponding to each process node and the number of process nodes;

[0128] S3: Based on the first GPU resource group configuration information, the initial process is started into multiple job processes, the number of job processes matches the number of process nodes, and a mapping relationship between the job processes and the process nodes is determined;

[0129] S4, based on the mapping relationship between the job process and the process node, establish a remote call link between the job process and the allocated GPU resource, and based on the remote call link, transmit the GPU resource request to the GPU device in the GPU resource to execute the computing task;

[0130] S5: When the computing tasks of all job processes are monitored to be completed, the GPU resources occupied by the AI ​​application container are released through the scheduler.

[0131] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0132] In addition, in conjunction with the GPU resource allocation method in the above embodiments, the present application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, it implements any one of the GPU resource allocation methods in the above embodiments.

[0133] Those skilled in the art should understand that the various technical features of the above-described embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A GPU resource allocation method, characterized in that: include: When the AI ​​application container is running, monitoring the call operation of the initial process in the AI ​​application container to the GPU interface, and generating a GPU resource request in response to the monitored call operation; Generate, via a scheduler, first GPU resource group configuration information for the initial process according to the GPU resource request; wherein the first GPU resource group configuration information includes GPU resource configuration information corresponding to each process node and the number of the process nodes; According to the first GPU resource group configuration information, the initial process is started into multiple job processes, the number of the job processes matches the number of the process nodes, and a mapping relationship between the job processes and the process nodes is determined; Based on the mapping relationship between the job process and the process node, a remote call link is established between the job process and the allocated GPU resource, and based on the remote call link, the GPU resource request is transmitted to the GPU device in the GPU resource to execute the computing task; When it is monitored that the computing tasks of all the job processes are completed, the GPU resources occupied by the AI ​​application container are released through the scheduler.

2. The GPU resource allocation method according to claim 1, wherein: Generating, by the scheduler, first GPU resource group configuration information for the initial process according to the GPU resource request, includes: The GPU resource request includes a scheduling strategy; the scheduling strategy includes at least one of a node number strategy, an affinity strategy, and a priority strategy; Generate first GPU resource group configuration information for the initial process via the scheduler according to the scheduling policy.

3. The GPU resource allocation method according to claim 1, wherein: Determining the mapping relationship between the job process and the process node includes: monitoring a call operation on a GPU interface by each of the job processes in the AI ​​application container, and generating the GPU resource request corresponding to the initial process when the call operation is detected; generating, via the scheduler, second GPU resource group configuration information for the job process according to the GPU resource request; wherein the second GPU resource group configuration information is the same as the first GPU resource group configuration information; Based on the second GPU resource group configuration information, a mapping relationship between the job process and the process node is determined.

4. The GPU resource allocation method according to claim 1, wherein: The establishing of a remote call link between the job process and the allocated GPU resource based on the mapping relationship between the job process and the process node includes: A communication protocol type is selected via the process node based on the acquired network environment information, and a remote call link is established between the job process and the allocated GPU resources based on the communication protocol type and a mapping relationship between the job process and the process node.

5. The GPU resource allocation method according to claim 1, wherein: The step of establishing a remote call link between the job process and the allocated GPU resource based on the mapping relationship between the job process and the process node, and transmitting the GPU resource request to a GPU device in the GPU resource to execute a computing task based on the remote call link, includes: Based on the node information in the first GPU resource group configuration information, generating a unique corresponding GPU backend processing process via a node agent process on the process node corresponding to each job process; A remote call link is established between the job process and the GPU back-end processing process. Based on the remote call link, the GPU resource request is transmitted to the process node corresponding to the job process. Then, the GPU resource request is transmitted to the GPU device in the GPU resource via the GPU back-end processing process on the process node.

6. The GPU resource allocation method according to claim 1, wherein: Determining the mapping relationship between the job process and the process node includes: A unique process node is allocated to each job process through a shared memory and semaphore mechanism, and a mapping relationship between the job process and the process node is determined.

7. The GPU resource allocation method according to claim 1, wherein: The method further comprises: When it is detected that the AI ​​application container exits abnormally, the computing task of the job process is forcibly terminated and the GPU resources occupied by the AI ​​application container are released.

8. A GPU resource allocation system, characterized in that: include: Remote call hijacking library, scheduler, and process node; A remote call hijacking library, configured to monitor, when an AI application container is running, calls to a GPU interface by an initial process in the AI ​​application container, and generate GPU resource requests in response to the monitored calls; A scheduler, configured to generate first GPU resource group configuration information for the initial process according to the GPU resource request; wherein the first GPU resource group configuration information includes GPU resource configuration information corresponding to each process node and the number of the process nodes; The scheduler is further configured to release the GPU resources occupied by the AI ​​application container via the scheduler when the computing tasks of all job processes are monitored to be completed; The process node is used to establish a remote call link between the job process and the allocated GPU resource based on the mapping relationship between the job process and the process node, and based on the remote call link, transmit the GPU resource request to the GPU device in the GPU resource to perform the computing task.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the GPU resource allocation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the GPU resource allocation method according to any one of claims 1 to 7 when running.

Images