A NUMA-aware load balancing scheduling method, system and medium
By evaluating the process migration overhead between NUMA nodes and prioritizing the migration of processes with the lowest overhead, the load balancing problem under the NUMA architecture is solved, and system performance is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- KYLIN CORP
- Filing Date
- 2022-12-28
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies lack effective load balancing scheduling methods under NUMA architecture, resulting in high cross-node migration overhead and impacting system performance.
By evaluating the cross-node migration cost of processes in each NUMA node, the process with the lowest cross-node migration cost is migrated first. A red-black tree is used to record the process migration cost, and appropriate processes are selected for migration between NUMA nodes to reduce memory migration operations.
It effectively reduces the overhead of memory migration operations between NUMA nodes, reduces the load on the interconnect bus, and improves the overall performance of the system.
Smart Images
Figure CN116126525B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of server scheduling technology for NUMA architecture, specifically to a NUMA-aware load balancing scheduling method, system, and medium. Background Technology
[0002] NUMA (Non-Uniform Memory Access) is a design for multiprocessors where memory access time depends on the processor's memory location. The NUMA architecture introduces the concept of nodes, each with its own internal CPU, bus, memory, and I / O slots. Each node's internal CPU can also access the memory and I / O slots of other nodes via the interconnect bus. Therefore, each CPU can access the entire system's memory. However, in NUMA, accessing a processor's local memory (where the CPU and memory reside on the same node) is significantly faster and less latency-intensive than accessing non-local memory (also called remote memory, where the CPU and memory are not on the same NUMA node). This is the origin of the term NUMA (Non-Uniform Memory Access). Due to this characteristic, to better utilize system performance, application development should minimize information exchange between different nodes. The interconnect bus is the physical implementation for interconnecting the nodes in a NUMA architecture CPU. A CPU in one node can access the internal resources (such as memory) of other nodes through the interconnect bus. The bandwidth and latency of the interconnect bus are important indicators for evaluating the performance of a NUMA architecture CPU.
[0003] In recent years, with the rapid development of domestic server CPU technology, domestic manufacturers have successively released a number of server CPUs based on the NUMA architecture. This has placed higher demands on software design, requiring software to make every effort to avoid cross-node access and reduce operations on the interconnect bus.
[0004] To reduce race conditions caused by process management, the Linux kernel's CFS scheduler allocates a task queue to each CPU. However, this introduces a new problem: during system operation, some CPUs may be very busy, while others are idle. In extreme cases, one CPU's ready queue may have a large number of processes, while other CPUs' ready queues may be empty. To address this issue, the Linux kernel implements load balancing between CPU ready queues. Load balancing aims to optimize the performance of the system's multi-core CPUs, preventing some CPUs from being constantly busy with a large number of processes waiting to run, while other CPUs are idle. This involves migrating processes from heavily loaded CPUs to less loaded CPUs. However, process migration comes at a cost. For example, load balancing between processes on two different physical CPUs can invalidate caches, leading to decreased efficiency. Furthermore, for NUMA systems, migration between different nodes will incur even greater losses, as not only will cache invalidation occur, but memory migration will also be performed after process migration. However, traditional process migration and load balancing techniques do not include a NUMA-aware load balancing scheduling method. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a NUMA-aware load balancing scheduling method, system and medium to address the above-mentioned problems in the prior art. The present invention perceives the resource usage of processes on NUMA nodes during load balancing between NUMA nodes and prioritizes the migration of processes with the least cross-node migration loss, thereby minimizing the memory migration operation overhead caused by load balancing between NUMA nodes, reducing the load on the interconnect bus and thus improving the overall performance of the entire system.
[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0007] A NUMA-aware load balancing scheduling method includes:
[0008] S101, evaluates cross-node migration overhead for processes in each NUMA node;
[0009] S102, when performing load balancing between NUMA nodes, select the process with the least cross-node migration loss for the processes in the NUMA node to be migrated and migrate it to other NUMA nodes.
[0010] Optionally, in step S101, when evaluating the cross-node migration loss for processes in each NUMA node, the cross-node migration loss is related to the process's remote memory consumption, CPU load, and local memory consumption. The cross-node migration loss is negatively correlated with both the process's remote memory consumption and CPU load, and positively correlated with the process's local memory consumption.
[0011] Optionally, the remote memory consumption and local memory consumption of the process are determined based on the NUMA node where the memory requested by the process is located. The portion of the memory requested by the process that is within the local NUMA node is the local memory consumption, and the portion of the memory requested by the process that is outside the local NUMA node is the remote memory consumption.
[0012] Optionally, the remote memory consumption and local memory consumption of the process are calculated based on the traversal of the process's address space. For each memory page traversed in the address space, the NUMA node information of the memory is obtained through the page_to_nid function call. The memory page of the local NUMA node is taken as the local memory consumption of the process, and the memory page outside the local NUMA node is taken as the remote memory consumption of the process.
[0013] Optionally, the CPU load of the process is obtained through the process's quantized load_avg.
[0014] Optionally, the functional expression for evaluating cross-node migration loss in step S101 is:
[0015] migrate_loss=W0*local_mm-W1*remote_mm-W2*load_avg
[0016] In the above formula, migration_loss is the cross-node migration loss, W0 to W2 are weight parameters, local_mm is the local memory consumption, remote_mm is the remote memory consumption, and load_avg is the CPU load.
[0017] Optionally, step S101 further includes using a red-black tree to record the processes entering the ready queue in the NUMA node and their cross-node migration losses: A structure variable `migrate_tasks` is added to the scheduler's run queue `cfs_rq` and initialized to `RB_ROOT` to represent the root node of the ready queue's red-black tree; a structure variable `rb_task` is added to the process structure `task_struct` to represent a node in the red-black tree, used to insert the process into the root node of the ready queue's red-black tree, and a structure variable `migrate_loss` is added to represent the process's evaluated cross-node migration loss, used as the key value for insertion into the ready queue's red-black tree; variables `local_mm` and `remote_mm` are added to the memory structure `mm_struct`, where `local_mm` represents local memory consumption and `remote_mm` represents remote memory consumption; when any current process `current` enters the ready queue, the cross-node migration loss is evaluated to obtain the cross-node migration loss `migrate_loss`, and the current process `current` is inserted into the ready queue's red-black tree using the cross-node migration loss `migrate_loss` as the red-black tree key value.
[0018] Optionally, after selecting the process with the least cross-node migration loss for the processes in the NUMA node to be migrated in step S102 and migrating it to other NUMA nodes, the step also includes deleting the process with the least cross-node migration loss from the red-black tree of the ready queue.
[0019] Furthermore, the present invention also provides a NUMA-aware load balancing scheduling system, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the NUMA-aware load balancing scheduling method.
[0020] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program for being programmed or configured by a microprocessor to execute the NUMA-aware load balancing scheduling method.
[0021] Compared with existing technologies, the present invention has the following advantages: The method of the present invention includes evaluating the cross-node migration loss for processes in each NUMA node; when performing load balancing between NUMA nodes, the process with the least cross-node migration loss in the NUMA node to be migrated is selected and migrated to other NUMA nodes. By being aware of the resource usage of processes on NUMA nodes during load balancing between NUMA nodes and prioritizing the migration of processes with the least cross-node migration loss, the present invention can minimize the memory migration operation overhead caused by load balancing between NUMA nodes, reduce the load on the interconnect bus, and thus improve the overall performance of the entire system. Attached Figure Description
[0022] Figure 1 An analytical example for the purposes of this invention.
[0023] Figure 2 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.
[0024] Figure 3 This is a flowchart illustrating the use of red-black trees in an embodiment of the present invention.
[0025] Figure 4 This is a flowchart illustrating the insertion of a red-black tree when a process enters the ready queue, according to an embodiment of the present invention. Detailed Implementation
[0026] Example 1:
[0027] When a process migrates across nodes to another NUMA node, the process's memory is also migrated, and the memory is moved from one NUMA node to another via the interconnect bus. To better illustrate the problem this invention aims to solve, as follows... Figure 1 For example, node0 and node1 are two different NUMA nodes. At a certain moment, the CPU of node1 is completely idle, while the load on node0 is very heavy, requiring some tasks to be migrated across nodes to node1. At this point: migrating high-memory, low-load process 'a' to node1 only reduces the load on node0 by 5%, but then 20% of the memory needs to be migrated to node1 via the interconnect bus. However: migrating low-memory, high-load process 'b' to node1 reduces the load on node0 by 30%, and only 3% of the memory needs to be migrated to node1 via the interconnect bus. Furthermore: migrating process 'c', which consumes remote memory across nodes, to node1 not only reduces the load on node0 but also eliminates the need for memory migration and removes cross-node memory access by process 'c'. Therefore, this invention enables the system to perceive the resource usage of processes on NUMA nodes during load balancing between NUMA nodes, and prioritizes the migration of processes with the least cross-node migration loss. This can minimize the memory migration operation overhead caused by load balancing between NUMA nodes, reduce the load on the interconnect bus, and thus improve the overall performance of the entire system.
[0028] like Figure 2 As shown, the NUMA-aware load balancing scheduling method in this embodiment includes:
[0029] S101, evaluates cross-node migration overhead for processes in each NUMA node;
[0030] S102, when performing load balancing between NUMA nodes, select the process with the least cross-node migration loss for the processes in the NUMA node to be migrated and migrate it to other NUMA nodes.
[0031] Testing revealed that cross-node migration overhead is related to a process's remote memory consumption, CPU load, and local memory consumption. Therefore, in step S101 of this embodiment, when evaluating cross-node migration overhead for processes in each NUMA node, the cross-node migration overhead is related to these three factors. Specifically, the cross-node migration overhead is negatively correlated with both remote memory consumption and CPU load, and positively correlated with local memory consumption. Therefore, processes with high remote memory consumption and low memory load across NUMA nodes can be prioritized for migration, reducing unnecessary inter-NUMA node memory migration operations caused by load balancing, thereby reducing the data load on the interconnect bus and improving the overall system performance. In particular, dividing memory consumption into remote memory consumption and local memory consumption effectively reduces the data load on the interconnect bus, thus improving the overall system performance.
[0032] To score the cross-node migration overhead of a process, this embodiment uses two dimensions: remote memory consumption and local memory consumption. As an optional implementation, in this embodiment, the remote memory consumption (remote_mm) and local memory consumption (local_mm) of a process are determined based on the NUMA node where the memory requested by the process resides. The portion of the memory requested by the process within its own NUMA node is considered local memory consumption, while the portion outside its own NUMA node is considered remote memory consumption. That is, when a process requests memory, the remote memory consumption (remote_mm) and local memory consumption (local_mm) are calculated and assigned values based on the NUMA node where the requested memory resides. These values are then recalculated during processes such as memory migration. This implementation is slightly more complex but has higher operating efficiency.
[0033] As an optional implementation, in this embodiment, the CPU load of a process is obtained through the process's quantized load (load_avg). Quantized load, introduced in the pelt algorithm implemented in the Linux 3.8 kernel, is used to measure CPU load. It is calculated by multiplying the process's runnable time by its total sampling time and then by the process's weight. When a process's quantized load is very close to its weight, it indicates that the process is constantly consuming CPU resources, meaning high CPU utilization. Conversely, a smaller quantized load indicates a smaller workload and less CPU resource consumption, meaning low CPU utilization. The total quantized load of the CPU is obtained by summing the quantized loads of all processes in the CPU ready queue. Alternatively, other methods can be used to obtain the process's CPU load as needed.
[0034] In this embodiment, the processes in the CPU ready queue are sorted, and the cross-node migration loss is evaluated by considering three aspects: remote memory consumption, local memory consumption, and CPU consumption. Specifically, the function expression for evaluating the cross-node migration loss in step S101 is as follows:
[0035] migrate_loss=W0*local_mm-W1*remote_mm-W2*load_avg
[0036] In the above formula, `migrate_loss` represents the cross-node migration loss, W0 to W2 are weight parameters, `local_mm` represents local memory consumption, `remote_mm` represents remote memory consumption, and `load_avg` represents CPU load. It should be noted that the weight parameters can be set according to actual needs. W0 to W2 are used to weight the process's local memory consumption, remote memory consumption, and CPU load, respectively. The purpose of weighting is to distinguish the contribution of these three factors to the cross-node migration loss. For example, setting W0 to 0 means that the contribution of local memory consumption to the cross-node migration loss is not considered, and only the process's remote memory and CPU load are considered. The higher the remote memory consumption and the higher the CPU load, the lower the cross-node migration loss score, and the more likely the process is to be selected for cross-node migration. In short, the lower the local memory consumption, the higher the remote memory consumption, and the higher the CPU load, the lower the cross-node migration loss score, and the more likely the process is to be selected for cross-node migration.
[0037] To achieve better overall performance and controllable time complexity in operations such as searching, inserting, and deleting, such as Figure 3As shown, step S101 in this embodiment also includes using a red-black tree to record the processes entering the ready queue in the NUMA node and their cross-node migration losses: A structure variable `migrate_tasks` is added to the scheduler's run queue `cfs_rq` and initialized to `RB_ROOT` to represent the root node of the ready queue's red-black tree; a structure variable `rb_task` is added to the process structure `task_struct` to represent a node in the red-black tree, used to insert the process into the root node of the ready queue's red-black tree, and a structure variable `migrate_loss` is added to represent the process's evaluated cross-node migration loss, used as the key value for insertion into the ready queue's red-black tree; variables `local_mm` and `remote_mm` are added to the memory structure `mm_struct`, where `local_mm` represents local memory consumption and `remote_mm` represents remote memory consumption; when any current process `current` enters the ready queue, the cross-node migration loss is evaluated to obtain the cross-node migration loss `migrate_loss`, and the current process `current` is inserted into the ready queue's red-black tree using the cross-node migration loss `migrate_loss` as the red-black tree key value. Considering that current kernel memory information statistics for processes do not differentiate between local and remote memory usage, only recording the process's total memory consumption, this embodiment introduces the variables `local_mm` and `remote_mm` into the `mm_struct` structure to differentiate between local and remote memory consumption. Furthermore, for process CPU consumption, the existing process quantization load `load_avg` can be directly used for measurement. Using `local_mm`, `remote_mm`, and `load_avg`, the process's cross-node migration overhead can be scored, and the score result is used as the key value for insertion into the red-black tree. For example... Figure 3 As shown, when the system performs load balancing among NUMA nodes, the sorted ready queue processes are traversed, and the process with the lowest cross-NUMA node migration loss score is selected for migration until the load among NUMA nodes is balanced. Step S102, after selecting the process with the lowest cross-node migration loss from the processes in the NUMA nodes to be migrated and migrating it to other NUMA nodes, also includes removing the process with the lowest cross-node migration loss from the red-black tree of the ready queue.
[0038] It should be noted that operations such as insertion, search, and deletion in red-black trees can reuse function calls such as rb_link_node, rb_insert_color, rb_first, and rb_erase from the red-black tree framework implemented in the Linux system. For example, when deleting the process with the lowest cross-node migration cost from the red-black tree in the ready queue, the rb_erase function is called to remove it from the red-black tree. Similarly, selecting the process with the lowest cross-node migration cost from the NUMA node to be migrated involves calling the rb_first function to retrieve the process with the lowest node migration cost from the red-black tree.
[0039] like Figure 4 As shown, the steps for inserting the current process (current) into the red-black tree of the ready queue, using the cross-node migration loss (migrate_loss) as the red-black tree key, include:
[0040] S201, initialize the node variables of the red-black tree, including the node variable link and the parent node patent, and initialize them to NULL; initialize the temporary task tmp_task and set the variable leftmost to true;
[0041] S202, receive the current process to be inserted and its cross-node migration loss migration_loss, and the running queue in which it is located is cfs_rq;
[0042] S203, by assigning the root node cfs_rq→migrate_tasks→rb_node to the node variable link;
[0043] S204. Determine whether the red-black tree has been traversed, i.e., whether the condition that link is empty (link == NULL) is true. If true, jump to step S205; otherwise, jump to step S206.
[0044] S205, by executing the rb_link_node function, the current process is inserted into the parent node patent of the red-black tree, the tree structure is adjusted to maintain balance, the node color is set (red or black), and then the process terminates and exits; the format of executing the rb_link_node function is:
[0045] rb_link_node(¤t→rb_task,&cfs_rq→migrate_tasks,leftmost)
[0046] Where ¤t→rb_task refers to the current node of the current process, cfs_rq→migrate_tasks refers to the root node of the red-black tree in the ready queue, and leftmost indicates whether there is a rightward movement during the traversal of the red-black tree; it is true if there is no movement, and false otherwise. The function format for setting the node color is:
[0047] rb_insert_color(¤t→rb task, &cfs rq→migrate_task, leftmost)
[0048] S206, assign the node variable link to the parent node patent so that the child node under the node variable link can be found to insert into the current process current;
[0049] S207, by executing the command: container_of(link, struct task_struct, rb_task), the corresponding task structure task_struct is retrieved from the node variable link and assigned to the temporary task tmp_task;
[0050] S208, determine whether the migration loss of the current process is less than the migration loss of the temporary task tmp_task. If it is true, move the node variable link to the left node (link = patent → rb_left) and jump to step S204; otherwise, move the node variable link to the right node (link = patent → rb_tight), assign the variable leftmost to false, and jump to step S204.
[0051] In summary, the NUMA-aware load balancing scheduling method in this embodiment addresses the characteristics of slow CPU access speed and high latency across the NUMA node interconnect bus in NUMA architecture. It proposes to optimize the selection of migration processes during load balancing between NUMA nodes by prioritizing the migration of processes with cross-node remote memory consumption and low memory but high load. This minimizes the large amount of memory migration operations caused by process migration across nodes, thereby reducing the data load on the interconnect bus and improving the overall system performance.
[0052] Furthermore, this embodiment also provides a NUMA-aware load balancing scheduling system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute a NUMA-aware load balancing scheduling method. Additionally, this embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by the microprocessor to execute a NUMA-aware load balancing scheduling method.
[0053] Example 2:
[0054] This embodiment is basically the same as Embodiment 1, with the main difference being the method of obtaining the process's remote memory consumption and local memory consumption. In this embodiment, the process's remote memory consumption and local memory consumption are calculated based on the process's address space traversal. For each memory page traversed, the NUMA node information is obtained through the page_to_nid function call. The memory page of this NUMA node is taken as the process's local memory consumption, and the memory pages outside this NUMA node are taken as the process's remote memory consumption. Compared with Embodiment 1, the method for calculating the process's remote memory consumption and local memory consumption in this embodiment is simpler but has slightly lower operating efficiency.
[0055] Furthermore, this embodiment also provides a NUMA-aware load balancing scheduling system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute a NUMA-aware load balancing scheduling method. Additionally, this embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by the microprocessor to execute a NUMA-aware load balancing scheduling method.
[0056] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0057] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A NUMA-aware load balancing scheduling method, characterized in that, include: S101, evaluate the cross-node migration overhead for processes in each NUMA node: migrate_loss=W0*local_mm - W1*remote_mm -W2*load_avg In the above formula, migration_loss is the cross-node migration loss, W0~W2 are weight parameters, local_mm is the local memory consumption, remote_mm is the remote memory consumption, and load_avg is the CPU load. A red-black tree is used to record the processes entering the ready queue in the NUMA node and their cross-node migration loss: a structure variable migration_tasks is added to the scheduler's run queue cfs_rq and initialized to RB_ROOT to represent the root node of the ready queue red-black tree; a structure variable rb_task is added to the process structure task_struct to represent the node of the red-black tree, used to insert the process into the ready queue. The root node of the black tree, and a structure variable `migrate_loss` to represent the estimated cross-node migration loss of the process, are used as the key value for insertion into the red-black tree of the ready queue; variables `local_mm` and `remote_mm` are added to the memory structure `mm_struct`, where `local_mm` is the local memory consumption and `remote_mm` is the remote memory consumption; when any current process `current` enters the ready queue, the cross-node migration loss is evaluated to obtain the cross-node migration loss `migrate_loss`, and the current process `current` is inserted into the red-black tree of the ready queue using the cross-node migration loss `migrate_loss` as the key value; S102, when performing load balancing between NUMA nodes, select the process with the least cross-node migration loss for the processes in the NUMA nodes to be migrated and migrate it to other NUMA nodes, and delete the process with the least cross-node migration loss from the red-black tree of the ready queue.
2. The NUMA-aware load balancing scheduling method according to claim 1, characterized in that, The remote memory consumption and local memory consumption of the process are determined based on the NUMA node where the memory requested by the process is located. The portion of the memory requested by the process that is within the memory of the local NUMA node is the local memory consumption, and the portion of the memory requested by the process that is outside the memory of the local NUMA node is the remote memory consumption.
3. The NUMA-aware load balancing scheduling method according to claim 1, characterized in that, The remote memory consumption and local memory consumption of the process are calculated based on the process's address space traversal. For each memory page traversed in the address space, the NUMA node information of the memory is obtained through the page_to_nid function call. The memory page of the local NUMA node is taken as the local memory consumption of the process, and the memory page outside the local NUMA node is taken as the remote memory consumption of the process.
4. A NUMA-aware load balancing scheduling system, comprising interconnected microprocessors and memory, characterized in that, The microprocessor is programmed or configured to execute the NUMA-aware load balancing scheduling method according to any one of claims 1 to 3.
5. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to execute the NUMA-aware load balancing scheduling method according to any one of claims 1 to 3.