AI computing power synergy method and system based on heterogeneous resource pooling and dynamic scheduling
By dividing physical GPU resources into virtual computing units and integrating them into an elastic memory pool, combined with a dynamic scheduler, the problem of resource waste in traditional AI computing power management is solved, achieving efficient utilization of AI computing power and smooth task operation.
Patent Information
- Application Number
- CN202512006418.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-24
AI Technical Summary
Traditional AI computing power usage models allocate and manage resources on a physical device basis, resulting in idle or wasted computing power and video memory resources. Existing virtual machine technology is difficult to precisely and flexibly divide and reorganize these resources, leading to low energy efficiency in AI computing power.
The physical GPU resources are divided into multiple virtual computing units, integrated into an elastic memory space pool, and monitored and controlled in real time by a dynamic scheduler to achieve decoupling and dynamic coordination between computing power and GPU memory, adapting to the needs of different AI computing tasks.
It improves the energy efficiency of AI computing power, enables flexible combination of computing power and memory, improves the utilization of GPU resources, and ensures the smooth operation of AI computing tasks and fault monitoring.
Smart Images
Figure CN121918935A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of GPU virtualization, and in particular to a method and system for improving AI computing power efficiency based on heterogeneous resource pooling and dynamic scheduling. Background Technology
[0002] Currently, the demand for computing power in AI computing tasks is growing exponentially. Traditional AI computing power usage models typically allocate and manage resources on a physical device basis, such as allocating computing power and video memory to AI computing tasks of different sizes using an entire GPU card. This easily leads to idle or wasted computing power and video memory resources. Moreover, it is impossible to dynamically adjust the allocation of computing power and video memory resources according to the actual load during task execution. Existing virtual machine technology also makes it difficult to perform fine-grained and flexible partitioning and reorganization of computing power and video memory. Overall, the energy efficiency of AI computing power is low, so improvements are needed. Summary of the Invention
[0003] In order to abstract and refine physical GPU resources into multiple virtual computing units, construct a flexible memory pool, and achieve dynamic coordination after decoupling computing power and memory resources to improve AI computing power efficiency, this application provides an AI computing power efficiency enhancement method and system based on heterogeneous resource pooling and dynamic scheduling.
[0004] The above-mentioned objective of this application is achieved through the following technical solution:
[0005] A method for improving AI computing power efficiency based on heterogeneous resource pooling and dynamic scheduling includes the following steps:
[0006] Divide physical GPU resources into multiple virtual computing units;
[0007] The host memory and device memory definitions of physical GPU nodes are integrated into a unified elastic memory space pool;
[0008] When an AI computing task is received, the corresponding number and specifications of virtual computing units are matched to the AI computing task based on the task requirements.
[0009] Based on the video memory requirements of AI computing tasks in the virtual computing unit, a preset proportion of elastic memory is allocated from the elastic memory space pool to each matched virtual computing unit.
[0010] The preset dynamic scheduler monitors the running virtual computing units in real time and reclaims or allocates elastic memory based on preset control strategies, as well as migrates the AI computing tasks of the virtual computing units.
[0011] By adopting the above technical solution, the physical GPU resources used for AI computing task training are abstracted into virtual computing units. Simultaneously, a portion of the host memory and device memory of each physical GPU node are selected and integrated into a flexibly allocated elastic memory pool. When an AI computing task is received, the computing power is first matched to the corresponding number and specifications of virtual computing units. Then, based on memory requirements, elastic memory that can be flexibly reclaimed and used is matched from the elastic memory pool. This achieves decoupling and matching of computing power and memory, while dynamically combining them. For AI computing tasks with different computing power and memory requirements, more flexible combined allocation of computing power and memory can be achieved, improving the energy efficiency of the physical GPU. Finally, during the operation of the virtual computing units, a preset dynamic scheduler can monitor the operating status of computing power and memory separately, reclaiming or allocating more elastic memory in real time to ensure the GPU resource utilization of the AI computing task. It also performs migration and control of the AI computing task to achieve smooth operation and physical GPU fault monitoring, realizing dynamic coordination after decoupling of computing power and memory resources, and improving the energy efficiency of AI computing power.
[0012] Optionally, the step of dividing physical GPU resources into multiple virtual computing units includes:
[0013] Deploy a lightweight agent on the physical nodes of a physical GPU cluster;
[0014] Hardware virtualization technology is used to divide the computing power into multiple virtual computing power instances that are isolated from video memory at the hardware level;
[0015] By assigning different performance specifications to virtual computing instances, the lightweight agent abstracts these different virtual computing instances into multiple virtual computing units of different specifications.
[0016] By adopting the above technical solutions, virtual computing units with different performance specifications provide more suitable running containers for AI computing tasks. By deploying lightweight agents on the physical nodes of the GPU cluster, the abstract transformation of virtual computing units is realized, and the static attributes and dynamic states of virtual computing units can be collected. The static attributes include the corresponding computing power and the matching exclusive memory, and the dynamic states include the idle state and power consumption, providing an accurate reference basis for matching subsequent AI computing tasks.
[0017] Optionally, the step of integrating the host memory and device memory definitions of physical GPU nodes into a unified elastic memory space pool includes:
[0018] Based on memory filtering rules, free memory and shareable, reclaimable memory are filtered from host memory and device video memory;
[0019] Register the address information of the selected free memory and shareable / reclaimable memory and generate corresponding query information to obtain multiple memory segments with different addresses and sizes;
[0020] A contiguous virtual address space is constructed based on the address information of memory segments, and the query information of each memory segment is associated to obtain an elastic memory space pool.
[0021] By adopting the above technical solution, memory filtering rules are used to filter out some memory in the host operating system that cannot be occupied by user space, in order to ensure the stability of system operation. Then, free, reclaimable, and shareable memory is selected from host memory and device video memory as elastic memory, including free memory, page cache, lazy-allocated memory, non-critical memory, and free parts of device video memory, which can be elastically called. Furthermore, the address information of the selected elastic memory is registered to generate query information. In the virtual address space, the physical GPU node location of the elastic memory can be determined through the address registration information and query information, thereby realizing the subsequent precise allocation and reclamation of elastic memory.
[0022] Optionally, the step of matching a corresponding number and specification of virtual computing units to the AI computing task based on the task requirements when receiving an AI computing task includes:
[0023] When an AI computing task is received, the peak computing power requirement of the AI computing task is identified, and a set of virtual computing power units that meet the peak computing power requirement is selected for the AI computing task.
[0024] Identify whether the AI computing task has topology and affinity requirements. If so, perform a second screening of the virtual computing units to obtain a combination of candidate virtual computing units.
[0025] Based on a preset optimization strategy, the cost of each virtual computing unit in the selected combination of multiple virtual computing units or the set of virtual computing units is calculated to select the target virtual computing unit.
[0026] By adopting the above technical solution, when matching virtual computing units for AI computing tasks, a distributed matching process is performed based on specific needs. First, the peak computing power requirements of the AI computing tasks need to be considered to ensure their smooth operation. Next, the affinity topology requirements of the AI computing tasks are matched, including ensuring high-bandwidth, low-latency interconnections between virtual computing units for intensive tasks. Multiple AI computing tasks belonging to the same job also need to be set up on multiple virtual computing units adjacent to the physical GPU nodes. Finally, an optimization strategy is used to assess the operating cost. This mainly involves determining the operating cost of the candidate virtual computing unit combination or individual virtual computing unit after matching, based on the current operating status of the GPU cluster. A comparison can be made using the historical operating data of the GPU cluster to approximate the cost, selecting the candidate virtual computing unit combination or individual virtual computing unit with the lowest cost for matching. This reduces the impact of matching virtual computing units, including potential risks and the need for additional adjustment steps.
[0027] Optionally, the step of matching a preset proportion of elastic memory from the elastic memory space pool for each matched virtual computing unit based on the video memory requirements of the AI computing tasks in the virtual computing unit includes:
[0028] Identify the video memory requirements of AI computing tasks and the exclusive memory size of the allocated virtual computing units, and calculate the video memory gap of the virtual computing units.
[0029] Based on the memory gap and preset slicing rules, corresponding elastic memory is allocated from the elastic memory space pool and matched to the corresponding virtual computing units to meet the memory requirements.
[0030] By adopting the above technical solution, when allocating elastic memory to virtual computing units, it is necessary to first obtain the exclusive memory size and video memory requirements of the corresponding virtual computing unit. After calculating the video memory gap, elastic memory with corresponding address registration information is selected from the elastic memory space pool and allocated to the virtual computing unit according to the preset slicing rules. At this time, the allocated elastic memory includes two cases: those from the same address registration information and those from multiple different address registration information. The purpose of the allocation is mainly to consider the load balancing of the physical GPU cluster operation. The situation that meets the video memory requirements includes the sum of elastic memory and exclusive memory just reaching the video memory requirement parameters, or the actual sum of elastic memory and exclusive memory exceeds the video memory requirement by a certain margin, so as to prevent the memory required by the actual running AI computing tasks from increasing.
[0031] Optionally, the step of allocating corresponding elastic memory from the elastic memory space pool and matching it to the corresponding virtual computing units based on the video memory gap and preset slicing rules to meet the video memory requirements includes:
[0032] Identify the vector feature information of AI computing tasks in virtual computing units, and match reference features from a preset feature database whose approximate value of the vector feature information is greater than a threshold.
[0033] Obtain the historical operating parameters corresponding to the reference features, and identify the changing characteristics of memory requirements in the historical operating parameters;
[0034] The peak memory characteristics and duration of the video memory demand variation are obtained, and the corresponding elastic memory is allocated to the virtual computing unit based on the peak memory characteristics and duration.
[0035] By adopting the above technical solution, the specific matching and judgment of the elastic memory size requires feature vectorization of the AI computing task to obtain vector characteristic information. Then, feature comparison is performed on the vector characteristic information to obtain the historical running parameters corresponding to the approximate reference characteristics. This allows for the determination of the changing characteristics of the video memory demand during the current AI computing task, including the magnitude and duration of the change. This provides a reliable reference value for the current elastic memory allocation size. For example, based on historical running data, it is known that the actual video memory demand of the reference characteristic during a similar GPU load is 10GB higher than the currently judged video memory demand for 5 minutes. Therefore, when matching elastic memory, the sum of the matched elastic memory and dedicated memory should be 10GB more than the video memory demand to ensure the smooth operation of the AI computing task.
[0036] Optionally, the preset dynamic scheduler monitors the running virtual computing units in real time, and reclaims or allocates elastic memory based on a preset control strategy, as well as migrates the AI computing tasks of the virtual computing units, including:
[0037] When the dynamic scheduler identifies that the actual video memory occupancy rate of any virtual computing unit is lower than the preset occupancy rate, the control strategy reclaims a preset proportion of elastic memory based on the current actual video memory demand of the virtual computing unit, and reallocates and divides the elastic memory to the virtual computing unit to meet the video memory demand after the occupancy rate is greater than the preset occupancy rate.
[0038] When it is determined that the physical GPU node where the virtual computing unit is located is in an abnormal operating condition, the control strategy creates virtual computing units of the same specifications on other physical GPU nodes and issues a migration command to migrate the virtual computing unit with abnormal operating condition to the new physical GPU node.
[0039] By adopting the above technical solutions, the preset winter retrieval model can flexibly adjust AI computing tasks from two dimensions: computing power and memory, during real-time monitoring of virtual computing units. If the actual memory usage of AI computing tasks decreases below a preset level during virtual computing unit operation, the dynamic scheduler reclaims some elastic memory based on the control model and reallocates it after the usage recovers. Simultaneously, it can further allocate memory when virtual computing units are insufficient, achieving flexible scheduling of elastic memory to meet the memory requirements of all virtual computing units in the entire GPU cluster. For adjusting computing power, a control strategy is used to hot-migrate virtual computing units on abnormal physical GPU nodes. For example, if a GPU node host overheats, virtual computing units of the same specifications can be created on other physical GPU hosts and allocated the same amount of elastic memory. The abnormal virtual computing units are then migrated and copied to new physical GPUs to maintain the smooth operation of AI computing tasks.
[0040] The second objective of this invention is achieved through the following technical solution:
[0041] An AI computing power efficiency enhancement system based on heterogeneous resource pooling and dynamic scheduling includes:
[0042] The unit partitioning module is used to divide physical GPU resources into multiple virtual computing units;
[0043] The memory consolidation module is used to integrate the host memory and device video memory definitions of physical GPU nodes into a unified elastic memory space pool.
[0044] The computing power matching module is used to match the corresponding number and specifications of virtual computing power units to the AI computing task based on the task requirements when an AI computing task is received.
[0045] The memory matching module is used to match a preset proportion of elastic memory from the elastic memory space pool for each matched virtual computing unit based on the video memory requirements of the AI computing tasks in the virtual computing unit.
[0046] The monitoring module is used by a preset dynamic scheduler to monitor the running virtual computing units in real time, and to reclaim or allocate elastic memory based on preset control strategies, as well as to migrate the AI computing tasks of the virtual computing units.
[0047] The above-mentioned objective three of this application is achieved through the following technical solution:
[0048] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described AI computing power enhancement method based on heterogeneous resource pooling and dynamic scheduling.
[0049] The fourth objective of this application is achieved through the following technical solution:
[0050] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling.
[0051] In summary, this application includes at least one of the following beneficial technical effects:
[0052] 1. Based on the memory requirements, flexible memory that can be flexibly reclaimed and called is matched from the elastic memory space pool. This achieves decoupling and matching of computing power and memory, while dynamically combining computing power and memory. For AI computing tasks with different computing power and memory requirements, more flexible combination allocation of computing power and memory can be achieved, improving the energy efficiency of physical GPUs. Finally, during the operation of the virtual computing unit, the preset dynamic scheduler can monitor the operation status of computing power and memory respectively, reclaim or allocate more elastic memory in real time to ensure the GPU resource utilization of AI computing tasks, and perform migration and control of AI computing tasks to achieve smooth operation of AI computing tasks and fault monitoring of physical GPUs. This achieves dynamic coordination after decoupling computing power and memory resources, improving the energy efficiency of AI computing power.
[0053] 2. Virtual computing units with different performance specifications provide more suitable running containers for AI computing tasks. By deploying lightweight agents on the physical nodes of the GPU cluster, the abstract transformation of virtual computing units can be realized, and the static attributes and dynamic states of virtual computing units can be collected. The static attributes include the corresponding computing power and the matching exclusive memory, and the dynamic states include the idle state and power consumption, providing an accurate reference basis for matching subsequent AI computing tasks.
[0054] 3. The optimization strategy for judging operating costs mainly involves determining the operating cost of the candidate virtual computing unit combination or individual virtual computing unit after matching under the current operating status of the GPU cluster. This can be done by comparing the historical operating data of the GPU cluster and selecting the candidate virtual computing unit combination or individual virtual computing unit with the lowest cost for matching. This reduces the impact of matching virtual computing units or individual virtual computing units, including potential risks and additional adjustment steps.
[0055] 4. For the specific matching and judgment of the elastic memory size, it is necessary to perform feature vectorization on the AI computing task to obtain vector characteristic information, and then perform feature comparison on the vector characteristic information to obtain the historical running parameters corresponding to the approximate reference features. In this way, it is possible to judge the changing characteristics of the video memory demand during the current AI computing task, including knowing the magnitude of the change and the duration of the change, so as to provide a reliable reference value for the current elastic memory allocation size. Attached Figure Description
[0056] Figure 1 This is a flowchart of an implementation of an AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling according to this application.
[0057] Figure 2 This is a flowchart of step S20 in an embodiment of an AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling in this application;
[0058] Figure 3 This is a flowchart of step S42 in an embodiment of an AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling in this application;
[0059] Figure 4 This is a schematic block diagram of a computer device according to this application. Detailed Implementation
[0060] The following is in conjunction with the appendix Figure 1-4 This application will be described in further detail.
[0061] In the following embodiments, such as Figure 1 As shown, this application discloses an AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling, which specifically includes the following steps:
[0062] S10: Divide physical GPU resources into multiple virtual computing units;
[0063] In this embodiment, physical GPU resources refer to physical GPU clusters, and virtual computing units refer to virtualized GPU training instances.
[0064] Specifically, step S10 includes the following steps:
[0065] S11: Deploy a lightweight agent on the physical nodes of a physical GPU cluster;
[0066] S12: Employs hardware virtualization technology to create multiple virtual computing instances that isolate computing power from video memory at the hardware level;
[0067] S13: Assign different performance specifications to virtual computing instances. The lightweight agent abstracts different virtual computing instances into multiple virtual computing units of different specifications.
[0068] The lightweight agent includes technologies such as MIG and SR-IOV for virtualizing physical GPUs. It divides the physical GPU into multiple virtual computing instances that isolate computing power and memory at the hardware level. These virtual computing instances are then assigned different performance rules, such as a 20% computing power percentage and a 25% memory bandwidth specification. The lightweight agent abstracts these different virtual computing instances into virtual computing units and collects their static attributes and dynamic states. Static attributes include the instruction set supported by the virtual computing unit (e.g., fetch instructions), as well as the theoretical computing power and dedicated memory size. Dynamic states refer to the virtual computing unit's operating status and real-time power consumption, including both idle and running states.
[0069] S20: Integrates the host memory and device memory definitions of physical GPU nodes into a unified elastic memory space pool;
[0070] In this embodiment, the memory used for consolidation in host memory and device video memory is only a portion of the total memory, which includes idle memory and shareable memory. The elastic memory pool contains a manager for managing and scheduling memory from different physical GPU nodes.
[0071] Specifically, refer to Figure 2 Step S20 includes the following steps:
[0072] S21: Based on memory filtering rules, filter out free memory and shareable / reclaimable memory from host memory and device video memory;
[0073] S22: Register the address information of the selected free memory and shareable / reclaimable memory and generate corresponding query information to obtain multiple memory segments with different addresses and sizes;
[0074] S23: Construct a contiguous virtual address space based on the address information of memory segments, and associate the query information of each memory segment to obtain an elastic memory space pool.
[0075] The memory filtering rules, when filtering, retain the memory required by the operating system itself, the memory used by system processes, and some memory that cannot be accessed by users. The host operating system itself needs to use a certain amount of memory to run; this memory cannot be directly used or reallocated by user-space applications. Many system services and daemons reside in memory, and the memory they occupy is usually considered "used" and cannot be taken away; otherwise, it would lead to system instability.
[0076] Therefore, the host's free memory, page cache, lazy-allocated memory, and non-critical process memory are acquired as callable memory. Free memory refers to physical memory pages that are not currently used by any user process or critical system service. For GPU device video memory, in addition to the guaranteed exclusive portion allocated to each instance by mechanisms such as MIG, the remaining video memory space, as well as the video memory area exposed to the host for access through Unified Memory technology, are included. Among the shareable and reclaimable video memory, page cache refers to cache pages reserved by the operating system to accelerate disk I / O. This part of memory can be reclaimed by pages when needed by the application, so it can be regarded as a shareable resource included in the over-allocation capacity calculation of the pool. Lazy-allocated memory is virtual memory that some processes have requested but have not yet actually accessed.
[0077] By calling APIs provided by operating system kernel modules, device drivers, or specific hardware management libraries, the available memory capacity for pooling on each node can be queried and registered, that is, free memory and shared, reclaimable memory can be queried and registered.
[0078] A virtual address space is a globally contiguous view of the virtual address space. When elastic memory needs to be allocated for a virtual computing unit, a segment is allocated from this virtual address space, and then this virtual address is mapped to physical memory pages allocated from different physical GPU nodes and different types of storage media, such as local physical GPU memory, memory of other physical GPU nodes, host DRAM, or even high-speed SSDs, via an API. This process is transparent to upper-layer applications.
[0079] S30: When an AI computing task is received, the corresponding number and specifications of virtual computing units are matched for the AI computing task based on the task requirements.
[0080] In this embodiment, the task requirements include peak computing power requirements and memory requirements. This step mainly focuses on matching the computing power requirements.
[0081] Specifically, step S30 includes the following steps:
[0082] S31: When an AI computing task is received, identify the peak computing power requirement of the AI computing task and select a set of virtual computing power units that meet the peak computing power requirement for the AI computing task.
[0083] S32: Identify whether the AI computing task has topology and affinity requirements. If so, perform a second screening of the virtual computing units to obtain a combination of candidate virtual computing units.
[0084] S33: Based on the preset optimization strategy, perform cost calculations on each virtual computing unit in the selected combination of multiple virtual computing units or the set of virtual computing units, and select the target virtual computing unit.
[0085] Specifically, the task requirements of the received AI computing tasks are transformed into a demand vector for one or more virtual computing units (VCUs) based on historical performance analysis, thereby determining the matching between the number of VCUs and the peak computing power. Furthermore, the topology and task affinity requirements of the AI computing tasks need to be analyzed. For communication-intensive tasks, there are requirements for topological relationships, necessitating a topological environment with high bandwidth and low latency interconnection between VCUs. For multiple AI computing tasks within the same job, such as the relationship between a parameter server and a worker, they need to be scheduled to physical GPU nodes that are geographically adjacent.
[0086] Finally, an optimization strategy is used to calculate the costs of candidate virtual computing unit combinations after screening based on peak computing power requirements, topology requirements, and affinity requirements. The optimal virtual computing unit or combination is then selected. Cost calculations include performance prediction, fragmentation costs, energy costs, and affinity interference. Performance prediction refers to predicting the expected execution time of the candidate virtual computing unit or combination, using methods such as historical performance data, performance models, or the simple weakest link principle to select the weakest unit within the group. Fragmentation costs assess the fragmentation impact on the resource pool after selecting the candidate virtual computing unit or combination. For example, selecting a large, contiguous block of physical GPU resources may leave more organized GPU idle resources than selecting several scattered small blocks, the latter being more costly. Energy costs refer to the estimated electricity or financial costs of running the selected virtual computing unit or combination. Affinity interference refers to the presence of highly interfering tasks, such as high I / O loads, already running on the physical GPU node where the candidate virtual computing unit or combination resides.
[0087] S40: Based on the video memory requirements of AI computing tasks in the virtual computing unit, a preset proportion of elastic memory is matched from the elastic memory space pool for each matched virtual computing unit.
[0088] In this embodiment, the video memory requirement is the total memory required to run the AI computing task based on the profile of the AI computing task. The preset proportion of elastic memory refers to the adjustable memory used to make up the total memory required for the AI computing task, excluding the exclusive memory bound to the virtual computing unit itself. It comes from a pre-built elastic memory space pool.
[0089] Specifically, step S40 includes the following steps:
[0090] S41: Identify the video memory requirements of AI computing tasks and the exclusive memory size of the allocated virtual computing units, and calculate the video memory gap of the virtual computing units.
[0091] S42: Based on the video memory gap and preset slicing rules, allocate corresponding elastic memory from the elastic memory space pool to the corresponding virtual computing units to meet the video memory requirements.
[0092] Among them, there are two situations: the memory gap and the elastic memory to be allocated are equal in size, or the elastic memory to be allocated is larger than the memory gap. The slicing rules need to take into account the memory changes during the actual operation of AI computing tasks, as well as the specific slicing location from the elastic memory pool. The specific slicing location is the address registration information, and the overall GPU cluster load balancing after slicing should be considered.
[0093] Furthermore, refer to Figure 3 Step S42 includes the following steps:
[0094] S421: Identify the vector feature information of AI computing tasks in the virtual computing unit, and match reference features from the preset feature database whose approximate value of the vector feature information is greater than a threshold.
[0095] S422: Obtain the historical operating parameters corresponding to the reference features, and identify the changing characteristics of memory requirements in the historical operating parameters;
[0096] S423: Obtain the peak memory characteristics and duration in the characteristics of changes in memory demand, and allocate corresponding elastic memory for virtual computing units based on the peak memory characteristics and duration.
[0097] In this embodiment, the final allocation decision for the elastic memory size needs to be analyzed from the vector feature information of the AI computing task, that is, the computing power and memory requirement profile of the AI computing task. The method is to use parameter features similar to the current AI computing task and judge the changing features during the operation from the historical running parameters of the eucalyptus features, that is, to identify the changes in memory requirements during the operation, including the peak value of the video memory and the duration of the change. Based on this, the most heavily allocated elastic memory for the AI computing task is determined. For example, if the reference feature actually runs under similar GPU load conditions, its video memory requirement is 10G higher than the currently judged video memory requirement for 5 minutes. Therefore, when matching elastic memory, the sum of the matched elastic memory and the exclusive memory should be 10G more than the video memory requirement to ensure the smooth operation of the AI computing task.
[0098] S50: The preset dynamic scheduler monitors the running virtual computing units in real time, and reclaims or allocates elastic memory based on preset control strategies, as well as migrates the AI computing tasks of the virtual computing units.
[0099] In this embodiment, a pre-defined dynamic scheduler continuously monitors and collects real-time resource utilization data for AI computing tasks during task execution, including GPU utilization, fluctuations in video memory usage, and the health status of virtual computing units and the global memory pool. Control strategies are used to regulate the actual memory usage and abnormal conditions of virtual computing units in AI computing tasks.
[0100] Specifically, step S50 includes the following steps:
[0101] S51: When the dynamic scheduler identifies that the actual video memory occupancy rate of any virtual computing unit is lower than the preset occupancy rate, the control strategy reclaims a preset proportion of elastic memory based on the current actual video memory demand of the virtual computing unit, and reallocates and divides the elastic memory to the virtual computing unit to meet the video memory demand after the occupancy rate is greater than the preset occupancy rate.
[0102] S52: When it is determined that the physical GPU node where the virtual computing power unit is located has an abnormal operating condition, the control strategy creates virtual computing power units of the same specification on other physical GPU nodes and issues a migration command to migrate the virtual computing power unit with the abnormal operating condition to the new physical GPU node.
[0103] Abnormal situations include overheating of the physical GPU node host and hardware malfunctions.
[0104] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0105] In one embodiment, an AI computing power efficiency enhancement system based on heterogeneous resource pooling and dynamic scheduling is provided. This system corresponds to the AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling described in the above embodiments. The AI computing power efficiency enhancement system based on heterogeneous resource pooling and dynamic scheduling includes:
[0106] The unit partitioning module is used to divide physical GPU resources into multiple virtual computing units;
[0107] The memory consolidation module is used to integrate the host memory and device video memory definitions of physical GPU nodes into a unified elastic memory space pool.
[0108] The computing power matching module is used to match the corresponding number and specifications of virtual computing power units to the AI computing task based on the task requirements when an AI computing task is received.
[0109] The memory matching module is used to match a preset proportion of elastic memory from the elastic memory space pool for each matched virtual computing unit based on the video memory requirements of the AI computing tasks in the virtual computing unit.
[0110] The monitoring module is used by a preset dynamic scheduler to monitor the running virtual computing units in real time, and to reclaim or allocate elastic memory based on preset control strategies, as well as to migrate the AI computing tasks of the virtual computing units.
[0111] Optionally, the cell partitioning module includes:
[0112] The agent deployment submodule is used to deploy lightweight agents on physical nodes of a physical GPU cluster.
[0113] The instance partitioning submodule is used to divide multiple virtual computing instances at the hardware level, which are isolated from the computing power and video memory, using hardware virtualization technology.
[0114] The unit abstraction submodule is used to assign different performance specifications to virtual computing instances. The lightweight agent abstracts different virtual computing instances into multiple virtual computing units with different specifications.
[0115] Optional, the memory integration module includes:
[0116] The memory filtering submodule is used to filter free memory and shareable, reclaimable memory from host memory and device video memory based on memory filtering rules;
[0117] The address registration submodule is used to register the address information of the selected free memory and shareable / reclaimable memory and generate corresponding query information to obtain multiple memory segments with different addresses and sizes.
[0118] The association submodule is used to construct a contiguous virtual address space based on the address information of memory segments, and associate the query information of each memory segment to obtain an elastic memory space pool.
[0119] Optionally, the computing power matching module includes:
[0120] The peak computing power matching submodule is used to identify the peak computing power requirements of AI computing tasks when an AI computing task is received, and to select a set of virtual computing power units that meet the peak computing power requirements for the AI computing task.
[0121] The topology association matching submodule is used to identify whether the AI computing task has topology requirements and affinity requirements. If so, the virtual computing power units are screened a second time to obtain a combination of virtual computing power units to be selected.
[0122] The cost matching submodule is used to calculate the cost of each virtual computing unit in a set of selected virtual computing units based on a preset optimization strategy, and to select the target virtual computing unit.
[0123] Optional, the memory matching module includes:
[0124] The gap calculation submodule is used to identify the video memory requirements of AI computing tasks and the exclusive memory size of the allocated virtual computing units, and to calculate the video memory gap of the virtual computing units.
[0125] The slice matching submodule is used to allocate corresponding elastic memory from the elastic memory space pool to the corresponding virtual computing units based on the video memory gap and preset slice rules, so as to meet the video memory requirements.
[0126] Optionally, the slice matching submodule includes:
[0127] The feature matching unit is used to identify the vector feature information of AI computing tasks in the virtual computing unit and to match reference features from the preset feature database that are approximately greater than a threshold to the vector feature information.
[0128] The change feature recognition unit is used to obtain the historical operating parameters corresponding to the reference features and identify the change features of memory requirements in the historical operating parameters.
[0129] The memory matching unit is used to obtain the peak characteristics and duration of video memory in the characteristics of video memory demand changes, and to allocate corresponding elastic memory for virtual computing power units based on the peak characteristics and duration of video memory.
[0130] Optional, the monitoring module includes:
[0131] The memory control submodule is used to, when the dynamic scheduler identifies that the actual video memory occupancy rate of any virtual computing unit is lower than the preset occupancy rate, reclaim a preset proportion of elastic memory based on the current actual video memory demand of the virtual computing unit, and reallocate and divide the elastic memory to the virtual computing unit to meet the video memory demand after the occupancy rate is greater than the preset occupancy rate.
[0132] The computing power migration submodule is used to create virtual computing power units of the same specifications on other physical GPU nodes when it is determined that the physical GPU node where the virtual computing power unit is located has an abnormal operating condition, and to issue a migration command to migrate the virtual computing power unit with an abnormal operating condition to the new physical GPU node.
[0133] Specific limitations regarding the AI computing power enhancement system based on heterogeneous resource pooling and dynamic scheduling can be found in the limitations of the AI computing power enhancement method based on heterogeneous resource pooling and dynamic scheduling mentioned above, and will not be repeated here. Each module in the aforementioned AI computing power enhancement system based on heterogeneous resource pooling and dynamic scheduling can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0134] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an AI computing power enhancement method based on heterogeneous resource pooling and dynamic scheduling.
[0135] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an AI computing power enhancement method based on heterogeneous resource pooling and dynamic scheduling.
[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements an AI computing power enhancement method based on heterogeneous resource pooling and dynamic scheduling.
[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0139] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for improving AI computing power efficiency based on heterogeneous resource pooling and dynamic scheduling, characterized in that: Divide physical GPU resources into multiple virtual computing units; The host memory and device memory definitions of physical GPU nodes are integrated into a unified elastic memory space pool; When an AI computing task is received, the corresponding number and specifications of virtual computing units are matched to the AI computing task based on the task requirements. Based on the video memory requirements of AI computing tasks in the virtual computing unit, a preset proportion of elastic memory is allocated from the elastic memory space pool for each matched virtual computing unit. The preset dynamic scheduler monitors the running virtual computing units in real time and reclaims or allocates elastic memory based on preset control strategies, as well as migrates the AI computing tasks of the virtual computing units.
2. The AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling according to claim 1, characterized in that, The step of dividing physical GPU resources into multiple virtual computing units includes: Deploy a lightweight agent on the physical nodes of a physical GPU cluster; Hardware virtualization technology is used to divide the computing power into multiple virtual computing power instances that are isolated from video memory at the hardware level; By assigning different performance specifications to virtual computing instances, the lightweight agent abstracts these different virtual computing instances into multiple virtual computing units of different specifications.
3. The AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling according to claim 1, characterized in that, The step of integrating the host memory and device memory definitions of physical GPU nodes into a unified elastic memory space pool includes: Based on memory filtering rules, free memory and shareable, reclaimable memory are filtered from host memory and device video memory; Register the address information of the selected free memory and shareable / reclaimable memory and generate corresponding query information to obtain multiple memory segments with different addresses and sizes; A contiguous virtual address space is constructed based on the address information of memory segments, and the query information of each memory segment is associated to obtain an elastic memory space pool.
4. The AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling according to claim 1, characterized in that, The step of matching a corresponding number and specification of virtual computing units to the AI computing task based on the task requirements when an AI computing task is received includes: When an AI computing task is received, the peak computing power requirement of the AI computing task is identified, and a set of virtual computing power units that meet the peak computing power requirement is selected for the AI computing task. Identify whether the AI computing task has topology and affinity requirements. If so, perform a second screening of the virtual computing units to obtain a combination of candidate virtual computing units. Based on a preset optimization strategy, the cost of each virtual computing unit in the selected combination of multiple virtual computing units or the set of virtual computing units is calculated to select the target virtual computing unit.
5. The AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling according to claim 1, characterized in that, The step of allocating a preset proportion of elastic memory from the elastic memory space pool to each matched virtual computing unit based on the video memory requirements of AI computing tasks in the virtual computing unit includes: Identify the video memory requirements of AI computing tasks and the exclusive memory size of the allocated virtual computing units, and calculate the video memory gap of the virtual computing units. Based on the memory gap and preset slicing rules, corresponding elastic memory is allocated from the elastic memory space pool and matched to the corresponding virtual computing units to meet the memory requirements.
6. The AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling according to claim 5, characterized in that, The step of allocating corresponding elastic memory from the elastic memory space pool and matching it to the corresponding virtual computing units based on the video memory gap and preset slicing rules to meet the video memory requirements includes: Identify the vector feature information of AI computing tasks in virtual computing units, and match reference features from a preset feature database whose approximate value of the vector feature information is greater than a threshold. Obtain the historical operating parameters corresponding to the reference features, and identify the changing characteristics of memory requirements in the historical operating parameters; The peak memory characteristics and duration of the video memory demand variation are obtained, and the corresponding elastic memory is allocated to the virtual computing unit based on the peak memory characteristics and duration.
7. The AI computing power efficiency enhancement method based on heterogeneous resource pooling and dynamic scheduling according to claim 1, characterized in that, The preset dynamic scheduler monitors the running virtual computing units in real time, and reclaims or allocates elastic memory based on a preset control strategy, as well as migrates the AI computing tasks of the virtual computing units, including: When the dynamic scheduler identifies that the actual video memory occupancy rate of any virtual computing unit is lower than the preset occupancy rate, the control strategy reclaims a preset proportion of elastic memory based on the current actual video memory demand of the virtual computing unit, and reallocates and divides the elastic memory to the virtual computing unit to meet the video memory demand after the occupancy rate is greater than the preset occupancy rate. When it is determined that the physical GPU node where the virtual computing unit is located is in an abnormal operating condition, the control strategy creates virtual computing units of the same specifications on other physical GPU nodes and issues a migration command to migrate the virtual computing unit with abnormal operating condition to the new physical GPU node.
8. An AI computing power efficiency enhancement system based on heterogeneous resource pooling and dynamic scheduling, characterized in that: The unit partitioning module is used to divide physical GPU resources into multiple virtual computing units; The memory consolidation module is used to integrate the host memory and device video memory definitions of physical GPU nodes into a unified elastic memory space pool. The computing power matching module is used to match the corresponding number and specifications of virtual computing power units to the AI computing task based on the task requirements when an AI computing task is received. The memory matching module is used to match a preset proportion of elastic memory from the elastic memory space pool for each matched virtual computing unit based on the video memory requirements of the AI computing tasks in the virtual computing unit. The monitoring module is used by a preset dynamic scheduler to monitor the running virtual computing units in real time, and to reclaim or allocate elastic memory based on preset control strategies, as well as to migrate the AI computing tasks of the virtual computing units.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the AI computing power enhancement method based on heterogeneous resource pooling and dynamic scheduling as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the AI computing power enhancement method based on heterogeneous resource pooling and dynamic scheduling as described in any one of claims 1 to 7.