Node distribution method and system for GPU use process in NUMA environment

Abstract

The invention discloses a node distribution method and system for a GPU (Graphics Processing Unit) use process in an NUMA (Non Uniform Memory Access) environment, and the method comprises the following steps: in a kernel mode, capturing an event that the process executes a new program and collecting meta-information of the process through an eBPF program mounted at a system call entry tracking point, and judging whether the process is a target GPU process or not based on the collected meta-information of the process; if the process is the target GPU process, transmitting meta-information of the process to a user mode, and if the process is not the target GPU process, ending and exiting; and for the meta-information transmitted to each target GPU process of the user mode, reading the meta-information of the target GPU process through the user mode daemon process, evaluating each node in the NUMA environment based on a preset node allocation rule, selecting an optimal NUMA node, and binding the target GPU process to a CPU of the optimal NUMA node for execution. The GPU computing efficiency and the resource utilization rate can be improved.

Description

Technical Field

[0001] This invention relates to the field of GPU computing technology, and specifically to a method and system for allocating nodes for GPU processes in a NUMA environment. Background Technology

[0002] In recent years, the widespread application of GPUs in fields such as deep learning and scientific computing has led to a surge in the frequency of GPU-intensive workloads and their resource demands on servers. These workloads involve frequent, large-scale data exchanges between host memory and GPU memory, and their performance is extremely sensitive to the NUMA topology between the CPU and GPU. If the CPU running the process is far from the target GPU topology, or if the process's memory is allocated on a remote node, it will result in significant cross-node access latency, PCIe bus contention, and additional memory copy overhead, severely limiting the effective release of GPU computing power.

[0003] Current mainstream NUMA / CPU affinity management technologies (such as numactl and static cpuset allocation) have the following limitations when dealing with GPU-intensive scenarios: 1. Static strategies cannot adapt to dynamic loads: Machine learning model services and batch processing tasks often start new processes or containers frequently, while tools such as numactl need to be configured before the process starts, making it difficult to dynamically adjust newly started processes at runtime; Insufficient response timing: Scheduling schemes based on user-space polling or post-detection usually adjust after the process has performed part of the operation, missing the critical window for localization of the first memory allocation; 2. Insufficient response timing: Schemes based on user-space polling or post-detection / scheduling often adjust only after the process has performed several operations, missing the opportunity for localization of the first batch of memory allocation; 3. Limited coverage: It can only support a single scenario of containers or bare metal. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for node allocation of GPU processes in a NUMA environment, which addresses the above-mentioned problems in the prior art. The present invention aims to improve GPU computing efficiency and resource utilization by rationally scheduling GPU processes.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for allocating nodes for GPU processes in a NUMA environment includes the following steps: S101, in kernel mode, captures events of a process executing a new program and collects the process's metadata by attaching an eBPF program to the system call entry point trace point. Based on the collected process metadata, it determines whether the process is the target GPU process. If it is the target GPU process, it passes the process's metadata to user mode; otherwise, it terminates and exits. S102: For the metadata of each target GPU process passed to the user space, the user space daemon reads the metadata of the target GPU process and evaluates each node in the NUMA environment based on the preset node allocation rules, selects the optimal NUMA node, and binds the target GPU process to the CPU of the optimal NUMA node for execution.

[0006] Optionally, in step S101, the system call entry tracing point includes tracepoint / syscalls / sys_enter_execve and tracepoint / syscalls / sys_enter_execveat. When a process starts executing execve or execveat, the eBPF program mounted at the system call entry tracing point is triggered to immediately capture the process's metadata. The metadata includes part or all of the process ID, command line arguments, environment variables, process kernel data structures, and parent process ID.

[0007] Optionally, determining whether a process is a target GPU process based on the collected process metadata includes: the eBPF program reading the preset target GPU process filtering rules in BPF_MAP_TYPE_ARRAY, and determining whether the process corresponding to the collected process metadata is a target GPU process according to the target GPU process filtering rules.

[0008] Optionally, step S101 further includes determining whether the corresponding process has been allocated a NUMA node based on the metadata of the collected process. If the process has not been allocated a NUMA node and the process is the target GPU process, then the metadata of the process is passed to the user space; otherwise, the process ends and exits. Determining whether a process has been allocated NUMA nodes based on the collected process metadata includes: reading the nr_cpus_allowed field in the process kernel data structure; if the value of nr_cpus_allowed is equal to the number of CPUs in the system, then the process has not been allocated NUMA nodes; otherwise, the process has been allocated NUMA nodes.

[0009] Optionally, in step S101, if the captured process metadata determines that the corresponding process is the target GPU process, then the metadata of the process is written into BPF_MAP_TYPE_RINGBUF.

[0010] Optionally, in step S102, the user-space daemon process reads the metadata of the target GPU process and evaluates each node in the NUMA environment based on a preset node allocation rule to select the optimal NUMA node. This includes: the user-space daemon process listens to the BPF_MAP_TYPE_RINGBUF type storage; when new metadata of the target GPU process is written into the BPF_MAP_TYPE_RINGBUF, a preset callback function is triggered to read the current available memory percentage of each NUMA node and the topological distance between each NUMA node and the target GPU device; and based on the preset node allocation rule, the optimal NUMA node is selected from all NUMA nodes. The name of the target GPU device is obtained by reading the command-line parameters or environment variables of the target GPU process. The node allocation rules include: filtering all NUMA nodes with available memory greater than a first preset threshold, and selecting the node with the shortest topological distance to the target GPU device from these NUMA nodes as the optimal NUMA node.

[0011] Optionally, binding the target GPU process to the CPU of the optimal NUMA node for execution includes: generating a corresponding CPU affinity mask based on the set of CPU cores contained in the optimal NUMA node, calling sched_setaffinity before the target GPU process's first memory allocation, and applying the CPU affinity mask to the target GPU process so that the target GPU process can only run on the CPU cores of the optimal NUMA node.

[0012] Furthermore, the present invention also provides a node allocation system for GPU processes in a NUMA environment, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the above-described node allocation method for GPU processes in a NUMA environment.

[0013] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the node allocation method for GPU-based processes in a NUMA environment by a processor.

[0014] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the above-described node allocation method for GPU-based processes in a NUMA environment via a processor.

[0015] Compared with existing technologies, this invention has the following main advantages: By mounting an eBPF program in kernel mode to capture events of a process executing a new program in real time, NUMA-aware scheduling decisions can be made at the early stage of GPU process startup. This is beneficial for completing CPU binding and memory policy settings in the early stage of process execution and before the first memory allocation, ensuring memory locality and fundamentally reducing cross-node access latency and extra copying. This not only helps avoid the migration costs caused by the adjustment lag of traditional solutions, but also helps avoid performance losses caused by remote memory allocation, PCIe path contention, etc., thus improving GPU computing efficiency and resource utilization. Since system calls are the necessary unified interface for all applications to interact with the operating system, this method can cover all deployment environments, from traditional physical machines and cloud hosts to various containers, in a completely consistent and transparent manner, without requiring any special adaptation or configuration modification for different scenarios, making it widely applicable. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the execution flow of the GPU process node allocation method in a NUMA environment according to the present invention.

[0017] Figure 2 This is a schematic diagram illustrating the metadata transmission of the node allocation method for GPU processes in a NUMA environment according to the present invention. Detailed Implementation

[0018] The technical solution of the present invention will now be described in further detail with reference to the accompanying drawings.

[0019] Figure 1 and Figure 2 An embodiment of the present invention is shown, which describes a method for allocating process nodes to a GPU in a NUMA environment, including the following steps: S101, in kernel mode, captures events of a process executing a new program and collects the process's metadata by attaching an eBPF program to the system call entry point trace point. Based on the collected process metadata, it determines whether the process is the target GPU process. If it is the target GPU process, it passes the process's metadata to user mode; otherwise, it terminates and exits. S102: For the metadata of each target GPU process passed to the user space, the user space daemon reads the metadata of the target GPU process and evaluates each node in the NUMA environment based on the preset node allocation rules, selects the optimal NUMA node, and binds the target GPU process to the CPU of the optimal NUMA node for execution.

[0020] By mounting an eBPF program in kernel mode to capture events of processes executing new programs in real time, NUMA-aware scheduling decisions can be made at the early stage of GPU process startup. This is beneficial for completing CPU binding and memory policy settings in the early stages of process execution and before the first memory allocation, ensuring memory locality and fundamentally reducing cross-node access latency and additional copying. This not only avoids the migration costs caused by the adjustment lag of traditional solutions, but also avoids performance losses caused by remote memory allocation and PCIe path contention, thus improving GPU computing efficiency and resource utilization. Since system calls are the necessary unified interface for all applications to interact with the operating system, this method can cover all deployment environments, from traditional physical machines and cloud hosts to various containers, in a completely consistent and transparent manner, without requiring any special adaptation or configuration modifications for different scenarios, making it widely applicable.

[0021] As an optional implementation, in this embodiment, in step S101, the system call entry tracing point includes tracepoint / syscalls / sys_enter_execve and tracepoint / syscalls / sys_enter_execveat. When a process starts executing execve or execveat, the eBPF program mounted at the system call entry tracing point is triggered to immediately capture the process's metadata. The metadata includes part or all of the process ID, command line parameters, environment variables, process kernel data structures, and parent process ID.

[0022] As an optional implementation, determining whether a process is a target GPU process based on the collected process metadata includes: the eBPF program reads the preset target GPU process filtering rules in BPF_MAP_TYPE_ARRAY, and determines whether the process corresponding to the collected process metadata is a target GPU process according to the target GPU process filtering rules. When a new GPU application is deployed or an old application is renamed, there is no need to recompile and load the eBPF program; simply updating the rules in the mapping will take effect immediately, realizing policy updates and thus adapting to changing business requirements. In one possible embodiment, the target GPU process filtering rules may be: first, identifying processes whose process name or parent process name contains "python" or "vllm"; then, further determining whether their command-line parameters contain a specific server startup subcommand (e.g., "vllm.entrypoints.openai.api_server" or "serve"); if both conditions are met, the process is determined to be a target GPU process.

[0023] As an alternative implementation, see Figure 2The target GPU process filtering rules preset in BPF_MAP_TYPE_ARRAY are updated by reading the configuration file containing the target GPU process filtering rules through the user-space daemon.

[0024] As an optional implementation, step S101 further includes determining whether the corresponding process has already undergone NUMA node allocation based on the collected process metadata. If the process has not undergone NUMA node allocation and is the target GPU process, then the process's metadata is transmitted to user space; otherwise, the process terminates and exits. This effectively reduces unnecessary scheduling overhead and improves overall system efficiency. Furthermore, this approach allows for smooth compatibility and collaborative work with existing ecosystems and management tools. Processes may have already completed specific affinity settings through various means (such as manually using numactl, container orchestration systems like Kubernetes topology managers, or job scheduling systems). This embodiment proactively avoids overwriting or interfering with these existing, potentially more granular or specific, binding states by identifying and respecting them. Therefore, this invention is not an exclusive alternative but rather a supplementary optimization solution that can be seamlessly integrated into complex management stacks to specifically handle GPU processes not covered by existing tools, thereby achieving more complete and conflict-free performance optimization of the system's global GPU load.

[0025] As an optional implementation, determining whether a corresponding process has been allocated NUMA nodes based on the collected process metadata includes: reading the nr_cpus_allowed field in the process kernel data structure; if the value of nr_cpus_allowed is equal to the number of CPUs in the system, then it is determined that the corresponding process has not been allocated NUMA nodes; otherwise, it is determined that the corresponding process has been allocated NUMA nodes.

[0026] As an optional implementation, in step S101, if the captured process's metadata is determined to be the target GPU process, then the process's metadata is written into BPF_MAP_TYPE_RINGBUF. Using the BPF_MAP_TYPE_RINGBUF type for storage facilitates the efficient, lock-free transfer of the target process's metadata from kernel mode to user mode, thus ensuring transmission efficiency and reliability in high-concurrency scenarios.

[0027] As an optional implementation, in step S102, the user-space daemon reads the metadata of the target GPU process and evaluates each node in the NUMA environment based on a preset node allocation rule to select the optimal NUMA node. This includes: the user-space daemon listens to the BPF_MAP_TYPE_RINGBUF type storage; when new metadata of the target GPU process is written into the BPF_MAP_TYPE_RINGBUF, a preset callback function is triggered to read the current available memory percentage of each NUMA node and the topological distance between each NUMA node and the target GPU device; and based on the preset node allocation rule, the optimal NUMA node is selected from all NUMA nodes. The name of the target GPU device is obtained by reading the command-line parameters or environment variables of the target GPU process. The node allocation rules include: filtering all NUMA nodes with available memory greater than a first preset threshold, and selecting the node with the shortest topological distance to the target GPU device from these NUMA nodes as the optimal NUMA node. In one possible embodiment, the first preset threshold is 10%. If there are no NUMA nodes with available memory greater than the first preset threshold, then the node with the shortest topological distance to the target GPU device from the nodes with the most available memory is selected as the optimal node.

[0028] As an optional implementation, the topology distance between each GPU device and NUMA nodes is obtained by traversing the system's PCI device driver directory and stored in a preset data structure (such as a mapping table). The specific process is as follows: First, the ` / sys / bus / pci / drivers / ` directory is traversed, and the corresponding driver subdirectory is located based on the known GPU driver name. This directory contains symbolic links of all GPU devices currently bound to the driver, with each link named after its PCI bus address. By parsing these links, the detailed information directory of each GPU device in sysfs is accessed, and key information such as the NUMA node associated with the device and its PCI location is extracted. Finally, based on the above information, the topology distance between each GPU device and each NUMA node in the system is calculated, and the result is stored for querying by the user-mode daemon. Of course, the user-mode daemon can also calculate the corresponding topology distance by reading the metadata of the target GPU process.

[0029] As an optional implementation, binding the target GPU process to the CPU of the optimal NUMA node for execution includes: generating a corresponding CPU affinity mask based on the set of CPU cores contained in the optimal NUMA node; calling `sched_setaffinity` before the initial memory allocation of the target GPU process to apply the CPU affinity mask to the target GPU process, ensuring that the target GPU process can only run on the CPU cores of the optimal NUMA node. The CPU affinity set by `sched_setaffinity` is automatically inherited by all child threads and child processes subsequently created by the process via fork or clone. This means that for a complex multi-process / multi-threaded GPU application, such as a deep learning training task, only one precise binding is needed on the initial main process, and the resulting entire process tree will be automatically constrained within the optimal NUMA node, eliminating the huge overhead of managing each thread or child process individually in the application.

[0030] In addition, this embodiment also provides a node allocation system for GPU processes in a NUMA environment, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the above-described node allocation method for GPU processes in a NUMA environment.

[0031] In addition, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the node allocation method for GPU-based processes in a NUMA environment by a processor.

[0032] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the above-described node allocation method for GPU-based processes in a NUMA environment via a processor.

[0033] Those skilled in the art will understand that the technical solutions provided by the embodiments of this application may be in the form of a method, system, or computer program product. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create an implementation for the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0034] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for allocating nodes for GPU processes in a NUMA environment, characterized in that, Includes the following steps: S101, in kernel mode, captures events of a process executing a new program and collects the process's metadata by attaching an eBPF program to the system call entry point trace point. Based on the collected process metadata, it determines whether the process is the target GPU process. If it is the target GPU process, it passes the process's metadata to user mode; otherwise, it terminates and exits. S102: For the metadata of each target GPU process passed to the user space, the user space daemon reads the metadata of the target GPU process and evaluates each node in the NUMA environment based on the preset node allocation rules, selects the optimal NUMA node, and binds the target GPU process to the CPU of the optimal NUMA node for execution.

2. The node allocation method for GPU processes in a NUMA environment according to claim 1, characterized in that: In step S101, the system call entry tracing points include tracepoint / syscalls / sys_enter_execve and tracepoint / syscalls / sys_enter_execveat. When a process starts executing execve or execveat, the eBPF program mounted at the system call entry tracing point is triggered to immediately capture the process's metadata. The metadata includes part or all of the process ID, command line arguments, environment variables, process kernel data structures, and parent process ID.

3. The node allocation method for GPU processes in a NUMA environment according to claim 1, characterized in that: Determining whether a process is a target GPU process based on the collected process metadata includes: the eBPF program reads the preset target GPU process filtering rules in BPF_MAP_TYPE_ARRAY, and determines whether the process corresponding to the collected process metadata is a target GPU process according to the target GPU process filtering rules.

4. The node allocation method for GPU processes in a NUMA environment according to claim 1, characterized in that: In step S101, it is also included to determine whether the corresponding process has been allocated a NUMA node based on the metadata of the collected process. If the process has not been allocated a NUMA node and the process is the target GPU process, the metadata of the process is passed to the user space; otherwise, the process ends and exits. Determining whether a process has been allocated NUMA nodes based on the collected process metadata includes: reading the nr_cpus_allowed field in the process kernel data structure; if the value of nr_cpus_allowed is equal to the number of CPUs in the system, then the process has not been allocated NUMA nodes; otherwise, the process has been allocated NUMA nodes.

5. The node allocation method for GPU processes in a NUMA environment according to claim 1, characterized in that: If, in step S101, the captured process's metadata is determined to be the target GPU process, then the process's metadata is written into BPF_MAP_TYPE_RINGBUF.

6. The node allocation method for GPU processes in a NUMA environment according to claim 5, characterized in that: In step S102, the user-space daemon reads the metadata of the target GPU process and evaluates each node in the NUMA environment based on the preset node allocation rules to select the optimal NUMA node. This includes the user-space daemon listening to the BPF_MAP_TYPE_RINGBUF type storage. When new metadata of the target GPU process is written into the BPF_MAP_TYPE_RINGBUF, a preset callback function is triggered to read the current available memory percentage of each NUMA node and the topological distance between each NUMA node and the target GPU device. Based on the preset node allocation rules, the optimal NUMA node is selected from all NUMA nodes. The name of the target GPU device is obtained by reading the command line parameters or environment variables of the target GPU process. The node allocation rules include: filtering all NUMA nodes with available memory greater than a first preset threshold, and selecting the node with the shortest topological distance to the target GPU device from these NUMA nodes as the optimal NUMA node.

7. The node allocation method for GPU processes in a NUMA environment according to claim 1, characterized in that: Binding the target GPU process to the CPU of the optimal NUMA node for execution includes: generating a corresponding CPU affinity mask based on the set of CPU cores contained in the optimal NUMA node; calling sched_setaffinity before the target GPU process's first memory allocation; applying the CPU affinity mask to the target GPU process; and ensuring that the target GPU process can only run on the CPU cores of the optimal NUMA node.

8. A node allocation system for GPU processes in a NUMA environment, comprising interconnected microprocessors and memory, characterized in that, The microprocessor is programmed or configured to execute the node allocation method for GPU-used processes in a NUMA environment as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the node allocation method for GPU-used processes in a NUMA environment as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the node allocation method for GPU-used processes in a NUMA environment as described in any one of claims 1 to 7.

Images