Heterogeneous gpu cluster scheduling method and system based on granularity segmentation and container awareness

Abstract

Embodiments of the present disclosure disclose a heterogeneous GPU cluster scheduling method and system based on granularity segmentation and container awareness. A specific embodiment of the method includes: converging the physical memory of the heterogeneous GPUs of each computing node of the cluster into a resource pool, calculating the transient difference of the memory demand based on real-time kernel state awareness after receiving the target container memory allocation request, and carrying out two-stage quota arbitration based on the difference; reserving safe buffer memory for each heterogeneous GPU, and when the total allocation quota of the resource pool reaches the preset safe warning quota, mapping the safe buffer memory to the target container; when the target container exhausts the physical memory of the current GPU, overflowing the read-only tensor data in it to the physical memory of other GPUs on the same node to complete the heterogeneous GPU cluster scheduling. The embodiment is related to cloud computing resource management, and through the cooperative mechanism of real-time kernel state awareness and two-stage quota arbitration, the response efficiency and accuracy of heterogeneous memory resource allocation are improved, and the overall operation efficiency of the cluster is improved.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of cloud computing resource management technology, specifically to a method and system for scheduling heterogeneous GPU clusters based on granular segmentation and container awareness. Background Technology

[0002] Heterogeneous graphics processing unit (GPU) cluster scheduling is mainly used in scenarios such as large model training, AI online inference, and scientific and industrial simulation. It can be compatible with GPU hardware of different models, computing power, and video memory within the cluster, and complete the adaptation management of tasks and hardware to reduce resource waste.

[0003] Currently, in the scheduling of heterogeneous GPU clusters, the scheduling method of static resource reservation is usually adopted. This method sets the resource limit in advance during the container startup phase, allocates the entire GPU exclusively, or allocates a fixed GPU quota. Its resource specifications remain fixed throughout the entire runtime of the container and will not dynamically adjust the physical resource mapping relationship according to the memory fluctuations during the actual operation of the task.

[0004] With the deployment of large model training and high-concurrency inference tasks, the demand for GPU memory in multi-tenant tasks exhibits strong dynamic fluctuations. The above methods have the following problems in heterogeneous GPU cluster scheduling: First, when allocating video memory according to a fixed quota, if a container initiates a sudden video memory request, the scheduler cannot respond in time, which can easily lead to the exhaustion of physical video memory and trigger a hardware overflow. Second, the lack of feedforward awareness of the container kernel's allocation intention during scheduling and allocation can easily cause sudden memory overflow, directly triggering hardware crashes and causing cascading interruptions of tasks on the same node.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure propose a heterogeneous GPU cluster scheduling method and system based on granular segmentation and container awareness to solve the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a heterogeneous GPU cluster scheduling method based on granular segmentation and container awareness, the method comprising: Obtain device information of physical video memory in heterogeneous graphics processors of each computing node in the cluster; divide each physical video memory into multiple physical video memory blocks, aggregate the physical video memory blocks into a resource pool, and mark each physical video memory block; device information includes: video memory capacity, video memory bandwidth and bus topology attributes. In response to receiving a video memory allocation request from the target container, the graphics processor requirement parameters carried by the target container are parsed, and the transient difference in video memory requirement is calculated. The transient difference in video memory requirement is the degree to which the target container's video memory deviates from its steady state at an instant. If the transient difference in video memory demand is lower than the preset stability threshold, the first-order quota arbitration is triggered, and physical video memory blocks are allocated from the resource pool according to the reference granularity set by the target container; if the transient difference in video memory demand is not lower than the preset stability threshold, and the duration reaches the preset sampling window, the second-order adaptive expansion arbitration is triggered, and physical video memory blocks are allocated; the stability threshold is the limit for determining the stable state of video memory demand in the target container. The control computing node reserves a safety buffer memory for each heterogeneous graphics processor. When the total allocation quota of the resource pool reaches the preset safety warning quota, and the transient difference in the memory demand of the target container exceeds the preset stability threshold, the safety buffer memory is mapped to the target container. The safety warning quota is the total resource usage alarm threshold of the resource pool. The safety buffer memory is the memory resource reserved in the resource pool in advance to cope with the instantaneous memory burst of the task. When the target container runs out of physical video memory on the current graphics processor and there is no available physical video memory block in the compute node, the pre-marked read-only tensor data in the target container is overflowed into the physical video memory of the graphics processor on the same node, and the virtual page table is copied during the mapping write. When the target container initiates a write operation on the overflowed read-only tensor data, the instruction sending of the target container is paused, the read-only tensor data is asynchronously swapped back to the physical video memory block of the graphics processor, and then the instruction sending is resumed, thus completing the heterogeneous graphics processor cluster scheduling.

[0009] Secondly, some other embodiments of this disclosure provide a heterogeneous GPU cluster scheduling system based on granular partitioning and container awareness, the system comprising: The heterogeneous GPU acquisition module acquires the device information of physical video memory in the heterogeneous graphics processors of each computing node in the cluster; it divides each physical video memory into multiple physical video memory blocks, aggregates the physical video memory blocks into a resource pool, and marks each physical video memory block; the device information includes: video memory capacity, video memory bandwidth, and bus topology attributes. The containerization awareness module, in response to receiving a video memory allocation request sent by the target container, parses the graphics processor requirement parameters carried by the target container and calculates the transient difference in video memory requirement, which is the degree to which the target container's video memory deviates from the steady state at an instant. The scheduling decision module triggers the first-order quota arbitration when the transient difference in video memory demand is lower than the preset stability threshold, allocating physical video memory blocks from the resource pool according to the benchmark granularity set by the target container; if the transient difference in video memory demand is not lower than the preset stability threshold and the duration reaches the preset sampling window, the second-order adaptive expansion arbitration is triggered to allocate physical video memory blocks; the stability threshold is the limit for determining the stable state of video memory demand of the target container. The dynamic adjustment module controls the computing nodes to reserve a safety buffer memory for each heterogeneous graphics processor. When the total allocation quota of the resource pool reaches the preset safety warning quota, and the transient difference in the memory demand of the target container exceeds the preset stability threshold, the safety buffer memory is mapped to the target container. The safety warning quota is the total resource usage alarm threshold of the resource pool. The safety buffer memory is the memory resource reserved in the resource pool in advance to cope with the instantaneous memory burst of the task. The scheduling agent module, when the target container runs out of physical video memory on the current graphics processor and there is no available physical video memory block in the compute node, overflows the pre-marked read-only tensor data in the target container to the physical video memory of the graphics processor on the same node, and copies the virtual page table during mapping and writing; when the target container initiates a write operation on the overflowed read-only tensor data, the instruction sending of the target container is paused, the read-only tensor data is asynchronously swapped back to the physical video memory block of the graphics processor, and then instruction sending is resumed to complete the heterogeneous GPU cluster scheduling.

[0010] The various embodiments of this disclosure have the following beneficial effects: the heterogeneous GPU cluster scheduling method and system based on granular partitioning and container awareness, as described in some embodiments of this disclosure, improve the accuracy of resource allocation and cluster management efficiency for heterogeneous GPU clusters. Specifically, the reason for the above-mentioned problems in existing scheduling methods is that they lack the ability to perceive the needs of target containers and manage precise scheduling, resulting in scheduling containers failing to respond in a timely manner and memory overflows being prone to sudden bursts. Therefore, the heterogeneous GPU cluster scheduling method and system based on granular partitioning and container awareness, as described in some embodiments of this disclosure, can not only intelligently perceive container needs but also accurately allocate cluster resources, preventing memory overflows and achieving efficient scheduling and management of heterogeneous GPU cluster resources.

[0011] In some embodiments of this disclosure, physical video memory is divided into fixed-granularity physical video memory blocks and aggregated into a global heterogeneous video memory virtualization resource pool. Simultaneously, interception is set during container runtime to detect video memory allocation in real time. This method uses the transient difference in video memory demand as a driving signal to achieve adaptive resource response: when the target container demand is stable, allocation is performed according to the baseline granularity; when demand suddenly increases, physical video memory blocks are dynamically added from the local machine or heterogeneous graphics processors with topological proximity, based on the heterogeneous video memory topology distance matrix. This disclosure significantly improves the response efficiency of heterogeneous video memory resource allocation through a collaborative mechanism of kernel-mode real-time awareness and two-stage quota arbitration, achieving reasonable resource allocation based on container awareness. Furthermore, through a two-stage container allocation strategy, it effectively improves the utilization rate of heterogeneous video memory resources and the overall operating performance of the cluster. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of the heterogeneous GPU cluster scheduling method based on granular segmentation and container awareness, which is one of the embodiments of this disclosure. Figure 2 This is a flowchart illustrating some embodiments of the heterogeneous GPU cluster scheduling method based on granular segmentation and container awareness according to this disclosure; Figure 3 This is a schematic diagram of the structure of some embodiments of the heterogeneous GPU cluster scheduling system based on granular segmentation and container awareness according to the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 and Figure 2 Flowcharts 100 and 200 illustrate some embodiments of a heterogeneous GPU cluster scheduling method based on granular partitioning and container awareness according to this disclosure. This heterogeneous GPU cluster scheduling method based on granular partitioning and container awareness includes the following steps: Step 101: Obtain device information of physical video memory in heterogeneous graphics processing units (GPUs) of each computing node in the cluster; divide each physical video memory into multiple physical video memory blocks, aggregate the physical video memory blocks into a resource pool, and mark each physical video memory block; device information includes: video memory capacity, video memory bandwidth, and bus topology attributes.

[0021] In some embodiments, the execution entity (e.g., the cluster scheduler) of the heterogeneous GPU cluster scheduling method based on granular segmentation and container awareness obtains the capacity, bandwidth and bus topology attributes of the physical memory in the heterogeneous graphics processors (GPUs) of each computing node in the cluster, divides each physical memory into multiple independent addressable physical memory blocks according to a preset fixed granularity, and extracts the physical connection relationship and physical transmission latency between the heterogeneous graphics processors. The physical memory blocks after being divided are grouped according to the type of physical connection relationship. Physical memory blocks connected by high-bandwidth interconnect bus are grouped into memory blocks in the same topology group, and physical memory blocks connected by general-purpose bus are grouped into memory blocks in different topology groups. Based on memory blocks in the same topology group and memory blocks across topology groups, a memory topology distance matrix is ​​established in conjunction with physical transmission latency. The memory bandwidth of all physical memory blocks is divided into memory bandwidth levels, namely high bandwidth level and standard bandwidth level, to construct a resource pool of heterogeneous virtualized physical memory blocks. The physical memory blocks are then marked one by one according to the memory bandwidth level and bus topology distance. Memory blocks in the same topology group are memory resource blocks within the same topology domain; memory blocks across topology groups are memory resource blocks within different topology domains; the heterogeneous memory topology distance matrix is ​​a heterogeneous graphics processor topology distance matrix oriented towards physical memory blocks.

[0022] In other embodiments, when constructing the resource pool, resource hierarchical scheduling rules are defined simultaneously. First, the physical connection relationships and physical transmission latency between heterogeneous GPUs are extracted. The segmented physical memory blocks are grouped according to the interconnect bus type. Physical memory blocks connected via high-bandwidth interconnect buses are grouped into memory blocks in the same topology group, and physical memory blocks connected via general-purpose buses are grouped into memory blocks in different topology groups. A heterogeneous memory topology distance matrix is ​​constructed based on the physical transmission latency. The matrix elements are used to characterize the access latency and bandwidth attenuation coefficient between different physical memory blocks. Physical memory blocks are divided into high-bandwidth level and standard bandwidth level according to memory bandwidth. In terms of resource allocation strategy, high-bandwidth level physical memory blocks are preferentially allocated to the target container corresponding to the model training task, and standard bandwidth level physical memory blocks are allocated to the target container corresponding to the model inference task.

[0023] In some optional implementations of certain embodiments, the cluster scheduler begins scanning the hardware attributes of the graphics processors on each compute node, reading the memory capacity, memory bandwidth, and bus topology attributes of each physical graphics processor. For example, the cluster scheduler may find that the compute node has two high-performance graphics processors, GPU-A and GPU-B, with a memory capacity of 80GB and a memory bandwidth of 2000GB / s, which are directly connected via an NVLink high-bandwidth interconnect bus. There are also two general-purpose graphics processors, GPU-C and GPU-D, with a memory capacity of 32GB and a memory bandwidth of 900GB / s, which are connected to the motherboard via a PCIe general-purpose bus.

[0024] Each physical memory block is divided into an independently addressable physical memory block according to a preset fixed granularity. The preset cluster scheduler divides each physical memory block into 256MB blocks with a fixed granularity. Therefore, GPU-A and GPU-B each get 320 physical memory blocks, and GPU-C and GPU-D each get 128 physical memory blocks.

[0025] The cluster scheduler, based on the physical connection relationships between the graphics processors, groups the physical memory blocks of GPU-A and GPU-B connected via the high-bandwidth interconnect bus into the same topology group, and groups the physical memory blocks of GPU-C and GPU-D connected via the general-purpose bus into cross-topology groups.

[0026] The cluster scheduler identifies the graphics processor's bus type based on physical connections, determining whether the graphics processor uses a high-bandwidth interconnect bus or a general-purpose bus link. It then retrieves the pre-built hardware-specific baseline latency for the corresponding link and adjusts it using the link's hardware topology and correction coefficients. This results in an interconnect latency of 10μs between GPU-A and GPU-B, and a transmission latency of 25μs between GPU-A and GPU-C. Based on the transmission latency and the physical memory connections, a heterogeneous memory topology distance matrix is ​​established. Matrix elements record the access latency and bandwidth attenuation coefficient between any two physical memory blocks. The cluster scheduler then further divides the physical memory blocks into high-bandwidth and standard-bandwidth levels according to memory bandwidth. For the high-bandwidth tier, GPU-A and GPU-B have a memory bandwidth of 2000GB / s, so the cluster scheduler marks the physical memory blocks of GPU-A and GPU-B as high-bandwidth tiers. GPU-C and GPU-D have a memory bandwidth of 900GB / s, so the physical memory blocks of GPU-C and GPU-D are marked as standard-bandwidth tiers. The tiered physical memory blocks are aggregated into a global heterogeneous memory virtualization resource pool. Based on the above, GPU-A and GPU-B each allocate 320 physical memory blocks, and GPU-C and GPU-D each allocate 128 physical memory blocks, for a total of 896 physical memory blocks in the global heterogeneous memory virtualization resource pool, with a total capacity of 224GB. The physical memory blocks are marked one by one according to the memory bandwidth level and the bus topology distance. For example, the labels are generated for the hierarchical physical memory blocks based on the bus topology. The interconnection delay between GPU-A and GPU-B is marked as 10μs in the topology distance matrix, and the transmission delay between GPU-A and GPU-C is marked as 25μs.

[0027] In terms of resource allocation strategy, high-bandwidth physical memory blocks are preferentially allocated to the target containers corresponding to model training tasks, while standard-bandwidth physical memory blocks are allocated to the target containers corresponding to model inference tasks.

[0028] Step 102: In response to receiving the video memory allocation request sent by the target container, the graphics processor requirement parameters carried by the target container are parsed, and the transient difference of video memory requirement is calculated. The transient difference of video memory requirement is the degree to which the target container's video memory deviates from the steady state at an instant.

[0029] In some embodiments, in response to receiving a video memory allocation request sent by the target container, the video memory allocation function called in user mode is intercepted, and the number of locked physical video memory page tables, virtual video memory allocation page tables and the time interval of video memory allocation requests are collected in kernel mode. The video memory page table allocation rate, video memory lock rate and video memory demand transient difference are calculated respectively.

[0030] The number of virtual memory allocation page tables increases within the preset sampling window is counted, and then divided by the sampling window duration to calculate the memory page table allocation rate. Divide the number of locked video memory page tables in the current physical page table by the total number of virtual video memory page tables requested to obtain the video memory lock rate; The transient difference in video memory demand is calculated by multiplying the video memory page table allocation rate by the first weighting coefficient and subtracting the product of the video memory lock rate, the preset base rate, and the second weighting coefficient.

[0031] In some alternative implementations of certain embodiments, based on the above content, refer to Figure 3 The diagram illustrates a flow 300 of some embodiments of the heterogeneous GPU cluster scheduling method and system based on granular segmentation and container awareness according to this disclosure. A containerization awareness module 302 is deployed in the environment where the compute node containers are running. In the containerization awareness module 302, a kernel-mode awareness unit inserts intercept probes at the hook point between the container control group and the extended Berkeley packet filter. When a target container initiates a memory allocation request, the intercept probe intercepts the user-mode memory allocation function call. The intercept probe also intercepts the user-mode memory allocation function call in kernel mode. For example, when training container C1 starts, it uses high-bandwidth GPU-A memory and initiates a memory allocation request. The intercept probe intercepts this user-mode memory allocation request in kernel mode and collects the number of locked pages in the kernel-mode physical memory, the number of virtual memory allocation page tables, and the time interval between memory allocation requests.

[0032] The preset sampling window is 10ms. The number of virtual memory allocation page tables that have increased within the preset sampling window is divided by the sampling window duration to obtain the memory page table allocation rate. If the virtual memory allocation page table of device C1 in the sampling window increases to 1024 pages, with each page being 64KB, then 1024 pages divided by the 10ms sampling duration yields a memory page table allocation rate of 102,400 pages per second.

[0033] The memory lock rate is obtained by dividing the number of currently locked physical page tables by the total number of virtual page tables requested. If the number of currently locked physical page tables is 4096 and the total number of virtual page tables requested is 16384, the memory lock rate is 0.25.

[0034] The transient difference in memory demand is calculated by multiplying the memory page table allocation rate by the first weighting factor and subtracting the product of the memory lock rate, the preset base rate, and the second weighting factor. The first weighting factor is fixed at 1.2, and the second weighting factor is fixed at 0.8. The memory page table allocation rate of 102,400 pages per second is amplified to 122,880 pages per second by the first weighting factor of 1.2. The memory lock rate of 0.25 is multiplied by the preset base rate of 100,000 pages per second to obtain 25,000 pages per second, which is then scaled down to 20,000 pages per second by the second weighting factor of 0.8. The difference between these two factors results in a transient difference in memory demand of 102,880 pages per second. The setting of the first and second weighting coefficients performs bidirectional weighted correction on the memory access rate, amplifying the pressure difference between high-load and low-load tasks. Tasks with access rates higher than the baseline rate are given weight gain, while tasks with access rates lower than the baseline rate are given weight suppression. This allows the cluster scheduler to accurately identify the actual memory pressure of tasks and accurately calculate the transient difference in memory demand, preventing scheduling misjudgments. It ensures the resource response speed of high-pressure tasks while avoiding ineffective resource preemption by low-load tasks, thereby improving the resource utilization and task operation stability of heterogeneous GPU clusters.

[0035] Step 103: If the transient difference in video memory demand is lower than the preset stable threshold, the first-order quota arbitration is triggered, and physical video memory blocks are allocated from the resource pool according to the benchmark granularity set by the target container; if the transient difference in video memory demand is not lower than the preset stable threshold, and the duration reaches the preset sampling window, the second-order adaptive expansion arbitration is triggered, and physical video memory blocks are allocated. The stable threshold is the limit for determining the stable state of video memory demand in the target container.

[0036] In some embodiments, the cluster scheduler receives the transient difference in video memory demand. If the transient difference in video memory demand is lower than a preset stability threshold, a first-order quota arbitration is triggered, and physical video memory blocks are allocated from the resource pool according to the baseline granularity declared by the target container. If the transient difference in video memory demand is not lower than the preset stability threshold and the duration reaches a preset sampling window, a second-order adaptive expansion arbitration is triggered, and an elastic buffer quota is calculated based on the current video memory bandwidth level of the target container. Specifically, the total number of physical video memory blocks currently allocated to the target container is obtained, the video memory bandwidth level (high bandwidth level or standard bandwidth level) of the physical video memory blocks occupied by the target container is called, the preset quota coefficient corresponding to the video memory bandwidth level is used, and the product of the number of currently allocated physical video memory blocks and the preset quota coefficient corresponding to the video memory bandwidth level is calculated as the elastic buffer quota.

[0037] After calculating the elastic buffer quota, search for unallocated physical memory blocks in the physical memory of the graphics processor currently occupied by the target container. If the number of remaining physical memory blocks in the graphics processor is less than the elastic buffer quota, query the heterogeneous memory topology distance matrix, and search for the secondary graphics processor with the smallest topology distance to the graphics processor and the number of remaining physical memory blocks that reaches the elastic buffer quota. Lock the corresponding number of physical memory blocks in the secondary graphics processor and allocate the physical memory blocks to the target container.

[0038] Elastic buffer quota is a dynamically adjustable amount of video memory buffer configured for the target container; secondary graphics processor is another GPU device located on the same physical node as the primary graphics processor that is not currently being used by the container.

[0039] In some optional implementations of some embodiments, the pre-configured hyperparameters of the cluster scheduler are set, with a preset stability threshold of 50,000 pages per second. When container C1 is in the model loading phase, the cluster scheduler compares the transient difference in video memory demand with the preset stability threshold. If the transient difference in target video memory demand is lower than the stability threshold, the target container is in a loading stability state. The cluster scheduler performs the first-order quota arbitration, and allocates 16 physical video memory blocks from the GPU-A resource pool to the target container according to the baseline granularity of 4GB declared by the target container.

[0040] If the transient difference in video memory demand exceeds the stable threshold and remains so for several consecutive sampling windows, the target container enters a tensor surge training state, triggering the second-order adaptive expansion arbitration. When container C1 enters the large model batch training stage, during expansion, the quota coefficient of the high bandwidth level is preset to 2.0, and the elastic buffer quota allocated according to the high bandwidth level is 8GB, which means that 32 physical video memory blocks need to be allocated. If the transient difference in video memory demand is 102,880 pages per second, which is higher than the stable threshold of 50,000 pages per second, and remains so for 3 consecutive sampling windows totaling 30ms, then the scheduler switches to the second-order adaptive expansion arbitration.

[0041] During expansion, the cluster scheduler first searches for free physical memory blocks within the target container's currently occupied graphics processor (primary graphics processor). If the number of free blocks is insufficient to meet the elastic buffer quota, it checks the topology distance matrix to locate the secondary graphics processor with the closest topological distance and sufficient free blocks. It then locks the corresponding number of physical memory blocks within the secondary processor and mounts them to the target container's virtual memory space via the NVLink high-bandwidth interconnect bus's cross-graphics processor memory virtual mapping page table. In the example above, the cluster scheduler first searches for free blocks in graphics processor GPU-A. GPU-A currently has only 8 physical memory blocks totaling 2GB, less than the elastic buffer quota of 32 physical memory blocks. The cluster scheduler then checks the topology distance matrix to locate the secondary graphics processor GPU-B, which is the closest to GPU-A in topological distance, has a latency of 10μs, and sufficient free blocks. It then locks 32 physical memory blocks totaling 8GB in GPU-B and mounts them to container C1 via the NVLink high-bandwidth interconnect bus's cross-graphics processor memory virtual mapping page table.

[0042] The embodiments disclosed above divide physical video memory into fixed-granularity physical video memory blocks and aggregate them into a global heterogeneous video memory virtualization resource pool. Simultaneously, interception is set during container runtime to detect video memory allocation in real time. The transient difference in video memory demand is used as a driving signal to achieve adaptive resource response: when the target container demand is stable, allocation is performed according to the baseline granularity; when demand suddenly increases, physical video memory blocks are dynamically added from the local machine or a heterogeneous graphics processor with a topological distance matrix. This disclosure, through a collaborative mechanism of real-time kernel-mode perception and two-stage quota arbitration, can not only intelligently perceive container demand but also accurately allocate cluster resources, preventing sudden video memory overflows. This significantly improves the response efficiency of heterogeneous video memory resource allocation, achieving reasonable resource allocation based on container awareness, effectively improving the utilization rate of heterogeneous video memory resources and the overall operating performance of the cluster.

[0043] In other embodiments, triggering a second-order adaptive expansion arbitration to allocate physical memory blocks further includes: When searching for physical memory blocks and mounting the virtual memory space of the target container, the target container's kernel instruction issuance frequency is reduced, and the relinquished computing power quota is allocated to high-priority containers on the same node. The physical memory blocks are then mounted and the memory page tables are mapped. After the physical memory blocks are mounted and the memory page tables are mapped, the target container's instruction issuance frequency and computing power quota are restored, and the high-priority containers are those currently running core tasks stably.

[0044] In some optional implementations of certain embodiments, when adding physical memory blocks, a virtual mapping page table across the graphics processor memory is established via a high-bandwidth interconnect bus and mounted to the virtual memory space of the target container. The search for and cross-card mounting of the allocated physical memory blocks will occupy a sampling window. Within this window, the scheduling agent unit in the scheduling agent module 305 reduces the kernel instruction issuance frequency of the training container to the computing core and temporarily overlays the transferred computing power time slices onto high-priority containers on the same node that are in a stable operating state. Once the physical memory block mounting and page table mapping are completed, the original instruction issuance frequency and computing power quota of the training container are restored. Based on the above, if the average time taken for the NVLink cross-card memory mapping process is 20ms, then the cross-card search and mounting of container C1 will occupy a sampling window of about 20ms. Within the window, the scheduling agent unit will reduce the kernel instruction issuance frequency of container C1 to the computing core by 30%, and temporarily overlay the transferred computing power time slices to the high-priority inference container C3 running stably on the same node. After the mounting of 32 physical memory blocks and page table mapping are completed, the original instruction issuance frequency and computing power quota of container C1 will be restored.

[0045] The above embodiments of this disclosure establish a time-sharing coordination between memory expansion and computation core instruction issuance time slices, binding the addition of memory space with the allocation of computing time. During the dynamic addition of elastic buffer quotas and cross-card page table mounting, the scheduling agent module 305 reduces the instruction issuance frequency of the target container to the computation core and temporarily allocates the transferred computing quota to high-priority containers in a stable operating state on the same node. After the physical memory block mounting and page table mapping are completed, its original computing quota is restored. Through the time-sharing coordination of memory addition and computing power transfer, data bus contention and computation pipeline pauses during cross-bus memory mapping are reduced, ensuring the stability of the overall computing power throughput of the cluster.

[0046] Step 104: Control the computing node to reserve a safety buffer memory for each heterogeneous graphics processor. When the total allocation quota of the resource pool reaches the preset safety warning quota, and the transient difference in the memory demand of the target container exceeds the preset stability threshold, the safety buffer memory is mapped to the target container. The safety warning quota is the total resource usage alarm threshold of the resource pool. The safety buffer memory is the memory resource reserved in the resource pool in advance to cope with the instantaneous memory burst of the task.

[0047] In some embodiments, the access frequency of physical video memory blocks allocated to the target container in the monitoring computing node is monitored, and a safety buffer video memory is reserved for each heterogeneous graphics processor; when resource pool allocation is triggered, if the total allocation quota of the resource pool reaches the preset safety warning quota, and the transient difference of the video memory demand of the target container exceeds the preset stability threshold, and the access frequency of the target container is lower than the preset access frequency threshold, the safety buffer video memory is mapped to the target container; physical video memory blocks in the target container that are lower than the access frequency threshold are marked as swappable, and the video memory page tables of the swappable physical video memory blocks are revoked.

[0048] The memory page table mapping in the safe buffer is switched to the physical memory block where the memory page table has been revoked, and the safe buffer is released. When the target container initiates a write operation on the physical memory block in the swappable state, a memory page fault is triggered, the current memory page table mapping is locked, and the target container's bandwidth on the bus is limited until the memory page table is restored.

[0049] In some optional implementations of certain embodiments, a fixed amount of video memory resources is reserved in the resource pool as a hard preemption safety buffer video memory. This buffer is not allocated to the target container for normal computing use, but is specifically used to handle sudden bursts of video memory requests from tasks. When a sudden burst of tasks occurs in the target container, this safety buffer video memory is allocated to the current target container. After the task stabilizes, quota arbitration is triggered to reclaim the memory, and it is then returned to the resource pool.

[0050] The hardware isolation unit in the containerization awareness module 302 of the computing node monitors the access frequency of the allocated physical video memory blocks of the target container. It reserves a hardware preemption safety buffer video memory for each graphics processor, which is independent of the resource pool. When the total allocation quota of the resource pool reaches the preset safety warning quota and the transient difference of the video memory demand of the target container exceeds the preset stability threshold, the hardware isolation unit first maps the safety buffer video memory to the target container in place, triggering a burst request allocation. After the additional allocation is completed, the video memory page table of the target container is switched to the newly allocated physical video memory block and the safety buffer video memory is returned to provide buffer time for video memory allocation.

[0051] When the total allocation quota of the resource pool reaches the preset security alert quota and the access frequency of the target container is lower than the preset activity threshold, the hardware isolation unit marks the physical video memory blocks that are not frequently accessed in the target container as swappable and removes the valid bit mark of the physical video memory block in the video memory page table. When the target container initiates a write operation on the swappable physical video memory block, a video memory page fault is triggered. The hardware isolation unit captures the page fault, locks the current video memory page table mapping, limits the bandwidth occupied by the target container on the bus, and allows the swappable data to be smoothly queued until the page table is restored.

[0052] Based on the above, 5% of the GPU-A's video memory, totaling 4GB, or 16 physical memory blocks, is reserved as a hard-preempted safety buffer, independent of the resource pool. When the total resource pool utilization reaches the 90% safety alert quota, if container C1 suddenly makes an unpredictable large-scale video memory request, the hardware isolation unit will map the 16 physical memory blocks of the safety buffer to container C1 to respond to the sudden video memory request. Within the same 20ms sampling window, the cluster scheduler scans the access frequency of the low-priority container C2 on the same node. If the access frequency of container C2 falls below the access frequency threshold in the past 500ms, the hardware isolation unit marks the physical memory block occupied by container C2 as swappable, removes its page table valid bit, and reclaims it for appending to container C1. The memory page table mapping of container C1 is switched to the newly allocated and locked physical memory block. After the append allocation is completed, the safety buffer memory is returned to the reserved area of ​​physical memory. When container C2 initiates a write operation on the swappable physical memory block again, a memory page fault is triggered. The hardware isolation unit catches the fault, locks the memory page table mapping, and reduces the bus bandwidth occupied by container C2, allowing container C2 to queue smoothly until the memory page table is restored.

[0053] The embodiments disclosed above reserve a safety buffer memory in physical video memory, independent of the virtualization resource pool, and combine this with a method of monitoring the access frequency of target containers through an isolation mechanism to form a dual out-of-bounds protection, addressing the risk of sudden video memory overflow and preventing task interruption. When the resource pool allocation quota reaches the safety warning quota and the transient difference in video memory demand of the target container exceeds the stability threshold, the safety buffer memory provides a smooth transition for the target container. At the same time, low-frequency accessed physical video memory blocks are marked as swappable, allowing idle video memory to flow back to containers on the same node for utilization. When a container initiates read / write operations on this content, it triggers in-situ locking of the video memory page table and access rate limiting. Through the coordinated action of the safety buffer and page table rate limiting, the video memory overflow of a single container will not affect other containers on the same node, effectively ensuring the continuous and stable operation of multiple tasks on the same node.

[0054] Step 105: When the target container runs out of physical video memory on the graphics processor and there is no available physical video memory block in the compute node, the pre-marked read-only tensor data in the target container is overflowed into the physical video memory of the secondary graphics processor on the same node, and the video memory page table is copied during the mapping write. When the target container initiates a write operation on the overflowed read-only tensor data, the instruction sending of the target container is paused, the read-only tensor data is asynchronously swapped back to the physical video memory block of the graphics processor, and then the instruction sending is resumed to complete the heterogeneous GPU cluster scheduling.

[0055] In some embodiments, during the dynamic allocation of the target container, read-only tensor data is pre-marked. Tensors that have not been written to in the running access statistics of N consecutive sampling windows are marked as read-only tensors. These read-only tensor data are tensor data that are only read accessed and not modified during the running task of the target container. This type of data is preferentially used as the target memory overflow migration object to alleviate the memory pressure of the running graphics processor.

[0056] When the target container runs out of physical video memory on the graphics processor and there is no available physical video memory block in the compute node, the pre-marked read-only tensor data in the target container overflows into the physical video memory of the secondary graphics processor on the same node, mapping the write-on-copy virtual page table. When the target container initiates a write operation on the overflowed read-only tensor data, the scheduling agent unit suspends the target container from issuing instructions to the GPU compute core, and copies the read-only tensor data stored in the physical video memory of the secondary graphics processor back to the physical video memory block of the current graphics processor through the high-bandwidth interconnect bus. During the copy, the write-on-copy video memory page table is updated, the video memory page table mapping address is corrected, and the virtual address is re-pointed to the newly migrated physical video memory block of the current graphics processor. During the copy process, instruction sending is not resumed, and instructions do not enter the current graphics processor, preventing the use of the old overflow copy while writing data, and avoiding data corruption and memory out-of-bounds errors.

[0057] After the read-only tensor is copied from the secondary graphics processor back to the physical memory block of the current graphics processor, the copy-on-write memory page table is updated synchronously. After the data migration is completed, the instruction sending of the target container is resumed, and the relevant running tasks continue to be executed.

[0058] In some optional implementations of certain embodiments, according to the above, if container C1 completely exhausts the physical memory of GPU-A and the entire node has no more physical memory blocks to add, the cluster scheduler will overflow the pre-marked read-only tensor data in container C1 to the GPU-B memory or host memory of the same node, and establish a copy-on-write memory page table mapping for it; when container C1 initiates a write operation on the overflowed read-only tensor data, the instruction issuance of container C1 to the computing core is paused, the read-only tensor data is asynchronously swapped back to the physical memory block of GPU-A, and then the instruction issuance of container C1 is resumed.

[0059] The above embodiments of this disclosure establish a smooth overflow and copy-on-write mechanism for read-only tensors across heterogeneous video memory, providing multi-level backoff and fault tolerance capabilities for extreme conditions where physical video memory is exhausted. When the physical video memory of the target container is exhausted and the cluster has no additional space, the cluster scheduler overflows the pre-marked read-only tensors to the heterogeneous graphics processor video memory or host memory on the same node and maps them to a copy-on-write virtual page table. When the target container initiates a write operation on the overflowed tensor, the system pauses instruction issuance and resumes execution after the tensor is asynchronously swapped back. This method ensures that the process will not be directly interrupted in extreme resource-scarce scenarios, thereby enabling long-cycle tasks to run continuously.

[0060] Continue to refer to Figure 3 The diagram illustrates the structure 300 of some embodiments of a heterogeneous GPU cluster scheduling system based on granular partitioning and container awareness according to this disclosure. This heterogeneous GPU cluster scheduling system based on granular partitioning and container awareness includes: The heterogeneous GPU acquisition module 301 acquires the device information of the physical video memory in the heterogeneous graphics processors of each computing node in the cluster; it divides each physical video memory into multiple physical video memory blocks, aggregates the physical video memory blocks into a resource pool, and marks each physical video memory block; the device information includes: video memory capacity, video memory bandwidth, and bus topology attributes.

[0061] The containerization awareness module 302, in response to receiving a video memory allocation request sent by the target container, parses the graphics processor requirement parameters carried by the target container and calculates the transient difference in video memory requirement. The transient difference in video memory requirement is the degree to which the target container's video memory deviates from the steady state at an instant.

[0062] The scheduling decision module 303, if the transient difference in video memory demand is lower than the preset stable threshold, triggers the first-order quota arbitration and allocates physical video memory blocks from the resource pool according to the benchmark granularity set by the target container; if the transient difference in video memory demand is not lower than the preset stable threshold and the duration reaches the preset sampling window, it triggers the second-order adaptive expansion arbitration and allocates physical video memory blocks; the stable threshold is the limit for determining the stable state of video memory demand of the target container.

[0063] The dynamic adjustment module 304 controls the computing node to reserve a safety buffer memory for each heterogeneous graphics processor. When the total allocation quota of the resource pool reaches the preset safety warning quota, and the transient difference in the memory demand of the target container exceeds the preset stability threshold, the safety buffer memory is mapped to the target container. The safety warning quota is the total resource usage alarm threshold of the resource pool. The safety buffer memory is the memory resource reserved in the resource pool in advance to cope with the instantaneous memory burst of the task.

[0064] The scheduling agent module 305, when the target container runs out of physical video memory on the current graphics processor and there is no available physical video memory block in the computing node, overflows the pre-marked read-only tensor data in the target container to the physical video memory of the graphics processor on the same node, and copies the virtual page table during mapping and writing; when the target container initiates a write operation on the overflowed read-only tensor data, the instruction sending of the target container is paused, the read-only tensor data is asynchronously swapped back to the physical video memory block of the graphics processor, and then the instruction sending is resumed to complete the heterogeneous GPU cluster scheduling.

[0065] In summary, the heterogeneous GPU cluster scheduling method and system based on granular segmentation and container awareness disclosed in this disclosure can not only intelligently perceive container demand but also accurately allocate cluster resources, achieving efficient scheduling and management of heterogeneous GPU cluster resources. Some embodiments of this disclosure divide physical memory into fixed-granularity physical memory blocks and aggregate them into a global heterogeneous memory virtualization resource pool. Simultaneously, interception is set during container runtime to perceive memory allocation in real time. This method uses the transient difference in memory demand as a driving signal to achieve adaptive resource response: when the target container demand is stable, allocation is performed according to the baseline granularity; when demand suddenly increases, physical memory blocks are dynamically added from the local machine or topologically adjacent heterogeneous graphics processors based on the heterogeneous memory topology distance matrix. This disclosure significantly improves the response efficiency of heterogeneous memory resource allocation through a collaborative mechanism of kernel-mode real-time awareness and two-stage quota arbitration, achieving reasonable resource allocation based on container awareness. Furthermore, through a two-stage container allocation strategy, it effectively improves the utilization rate of heterogeneous memory resources and the overall operating performance of the cluster.

[0066] Furthermore, some embodiments of this disclosure establish a time-sharing coordination mechanism for memory expansion and computation core instruction issuance time slices, binding the addition of memory space with the allocation of computing time. During the dynamic addition of elastic buffer quotas and cross-card page table mounting, the scheduling agent unit reduces the instruction issuance frequency of the target container to the computation core and temporarily allocates the transferred computing quota to high-priority containers in a stable operating state on the same node. After the physical memory block mounting and page table mapping are completed, its original computing quota is restored. Through the time-sharing coordination of memory addition and computing power transfer, data bus contention and computation pipeline pauses during cross-bus memory mapping are reduced, ensuring the stability of the overall computing throughput of the cluster.

[0067] In some embodiments of this disclosure, a safety buffer memory, independent of the virtualization resource pool, is reserved in the physical video memory. This buffer, combined with monitoring the access frequency of the target container through an isolation mechanism, forms a dual-level protection against sudden overflows, addressing the risk of sudden video memory overflows. When the resource pool allocation quota reaches the safety warning quota and the transient difference in the video memory demand of the target container exceeds the stability threshold, the safety buffer memory provides a smooth transition for the target container. Simultaneously, low-frequency accessed physical video memory blocks are marked as swappable. When a container initiates read / write operations on this content, the video memory page table is locked in situ and access rate is limited. Through the coordinated action of the safety buffer and page table rate limiting, a video memory overflow in a single container will not affect other containers on the same node, effectively ensuring the continuous and stable operation of multiple tasks on the same node.

[0068] Other embodiments of this disclosure establish a smooth overflow and copy-on-write mechanism for read-only tensors across heterogeneous video memory, providing multi-level backoff and fault tolerance capabilities for extreme conditions where physical video memory is exhausted. When the physical video memory of the target container is exhausted and the cluster has no additional space, the cluster scheduler overflows the pre-marked read-only tensors to the heterogeneous graphics processor video memory or host memory on the same node and maps them to a copy-on-write virtual page table. When the target container initiates a write operation on the overflowed tensor, the system pauses instruction issuance and resumes execution after the tensor is asynchronously swapped back. This method ensures that the process will not be directly interrupted in extreme resource-scarce scenarios, thereby enabling long-cycle tasks to run continuously.

[0069] The heterogeneous GPU cluster scheduling method and system based on granular segmentation and container awareness in some embodiments of this disclosure, through container kernel-level lightweight interception and cluster scheduler coordination mechanism, do not require modification of upper-layer application code, business logic and underlying graphics processor drivers, and the system has extremely light additional control overhead, ensuring the stability of the overall computing power throughput of the cluster, and improving the utilization rate of heterogeneous memory resources and the overall operating performance of the cluster.

[0070] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A heterogeneous GPU cluster scheduling method based on granular segmentation and container awareness, characterized in that, include: Obtain device information of physical video memory in heterogeneous graphics processors of each computing node in the cluster; divide each physical video memory into multiple physical video memory blocks, aggregate the physical video memory blocks into a resource pool, and mark each physical video memory block; the device information includes: video memory capacity, video memory bandwidth, and bus topology attributes; In response to receiving a video memory allocation request sent by the target container, the graphics processor requirement parameters carried by the target container are parsed, and the transient difference in video memory requirement is calculated. The transient difference in video memory requirement is the degree to which the target container's video memory deviates from its steady state at an instant. If the transient difference in video memory demand is lower than a preset stability threshold, a first-order quota arbitration is triggered, and the physical video memory block is allocated from the resource pool according to the reference granularity set by the target container; if the transient difference in video memory demand is not lower than the preset stability threshold, and the duration reaches a preset sampling window, a second-order adaptive expansion arbitration is triggered, and the physical video memory block is allocated; the stability threshold is a limit value for determining the stable state of video memory demand in the target container; The computing node reserves a safety buffer memory for each heterogeneous graphics processor. When the total allocation quota of the resource pool reaches the preset safety warning quota, and the transient difference in memory demand of the target container exceeds the preset stability threshold, the safety buffer memory is mapped to the target container. The safety warning quota is the total resource usage alarm threshold of the resource pool. The safety buffer memory is the memory resource reserved in the resource pool in advance to cope with instantaneous memory bursts during tasks. When the target container runs out of physical video memory on the current graphics processor and there is no available physical video memory block in the computing node, the pre-marked read-only tensor data in the target container overflows into the physical video memory of the graphics processor on the same node, and the virtual page table is copied during the mapping write. When the target container initiates a write operation on the overflowed read-only tensor data, the instruction sending of the target container is paused, the read-only tensor data is asynchronously swapped back to the physical video memory block of the graphics processor, and the instruction sending is resumed to complete the heterogeneous graphics processor cluster scheduling.

2. The method according to claim 1, characterized in that, The step of dividing each physical video memory block into multiple physical video memory blocks and aggregating the physical video memory blocks into a resource pool includes: Each physical memory block is divided into multiple independently addressable physical memory blocks according to a preset fixed granularity, and the physical connection relationship and physical transmission latency between the heterogeneous graphics processors are extracted. The physical memory blocks after being divided are grouped according to the physical connection relationship type. The physical memory blocks connected by the high-bandwidth interconnect bus are grouped into memory blocks in the same topology group, and the physical memory blocks connected by the general-purpose bus are grouped into memory blocks across topology groups. Based on the same topology group memory blocks and the cross topology group memory blocks, a heterogeneous memory topology distance matrix is ​​established in conjunction with the physical transmission delay. All physical memory blocks are divided into high bandwidth level and standard bandwidth level by the memory bandwidth of the heterogeneous memory topology distance matrix, thus constructing the resource pool of heterogeneous virtual physical memory blocks. The same topology group memory block refers to a memory resource block within the same topology domain; the cross topology group memory block refers to a memory resource block within different topology domains; the heterogeneous memory topology distance matrix is ​​a heterogeneous graphics processor topology distance matrix oriented towards physical memory blocks.

3. The method according to claim 1, characterized in that, The step of responding to a received video memory allocation request from a target container, parsing the graphics processor requirement parameters carried by the target container, and calculating the transient difference in video memory requirement includes: In response to receiving a video memory allocation request from the target container, the video memory allocation function called in user mode is intercepted. In kernel mode, the number of locked video memory page tables and virtual video memory allocation page tables of the physical video memory are collected, and the time interval between video memory allocation requests is calculated. The video memory page table allocation rate, video memory lock rate, and transient difference of video memory demand are calculated respectively.

4. The method according to claim 3, characterized in that, The calculation of memory page table allocation rate, memory lock rate, and transient difference in memory demand includes: The number of increments in the virtual memory allocation page table within the preset sampling window is counted, and then divided by the sampling window duration to calculate the memory page table allocation rate. The memory lock rate is obtained by dividing the number of locked video memory page tables in the current physical page table by the total number of virtual video memory request page tables. The transient difference in video memory demand is calculated by multiplying the video memory page table allocation rate by the first weighting coefficient and subtracting the product of the video memory lock rate, the preset base rate, and the second weighting coefficient.

5. The method according to claim 1, characterized in that, The triggering of the second-order adaptive expansion arbitration to allocate physical memory blocks includes: Calculate the elastic buffer quota based on the current memory bandwidth level of the target container, search for unallocated physical memory blocks in the physical memory of the graphics processor currently occupied by the target container, query the heterogeneous memory topology distance matrix, search for the secondary graphics processor with the smallest topology distance to the graphics processor and the number of remaining physical memory blocks reaching the elastic buffer quota, lock the corresponding number of physical memory blocks in the secondary graphics processor, allocate the physical memory blocks to the target container, and mount the virtual memory space of the target container; The elastic buffer quota is a dynamically adjusted video memory buffer quota configured for the target container; the secondary graphics processor is another graphics processor located on the same physical node as the graphics processor and not currently used by the target container.

6. The method according to claim 5, characterized in that, The process of triggering the second-order adaptive expansion arbitration to allocate physical memory blocks also includes: When searching for the physical video memory block and the virtual video memory space on which the target container is mounted, the frequency of the target container issuing kernel instructions is reduced, and the transferred computing power quota is allocated to high-priority containers on the same node. The physical video memory block and the mapped video memory page table are then mounted. The high-priority containers are those currently running core tasks smoothly.

7. The method according to claim 1, characterized in that, The control node reserves a safety buffer memory for each heterogeneous graphics processor; when the total allocation quota of the resource pool reaches the preset safety warning quota, and the transient difference in the memory demand of the target container exceeds the preset stability threshold, the safety buffer memory is mapped to the target container, including: Monitor the access frequency of the physical video memory blocks allocated to the target container in the computing node, and reserve a safe buffer video memory for each heterogeneous graphics processor; When the total allocation quota of the resource pool reaches the preset security warning quota, and the transient difference of the video memory demand of the target container exceeds the preset stability threshold, and the access frequency of the target container is lower than the preset access frequency threshold, the security buffer video memory is mapped to the target container. Mark the physical memory blocks in the target container that are below the access frequency threshold as swappable and cancel the memory page tables of the swappable physical memory blocks. Switch the memory page table mapping in the safe buffer memory to the physical memory block where the memory page table is undone, and release the safe buffer memory; When the target container initiates a write operation on the physical memory block in the swappable state, a memory page fault is triggered, the current memory page table mapping is locked, and the memory page table is restored.

8. A heterogeneous GPU cluster scheduling system based on granular partitioning and container awareness, applied to the heterogeneous GPU cluster scheduling method based on granular partitioning and container awareness as described in any one of claims 1 to 7, characterized in that, include: The heterogeneous GPU acquisition module acquires device information of physical video memory in the heterogeneous graphics processors of each computing node in the cluster; divides each physical video memory into multiple physical video memory blocks, aggregates the physical video memory blocks into a resource pool, and marks each physical video memory block; the device information includes: video memory capacity, video memory bandwidth, and bus topology attributes. The containerization awareness module, in response to receiving a video memory allocation request sent by the target container, parses the graphics processor requirement parameters carried by the target container and calculates the transient difference in video memory requirement, which is the degree to which the target container's video memory deviates from its steady state at an instant. The scheduling decision module, if the transient difference in video memory demand is lower than a preset stability threshold, triggers a first-order quota arbitration to allocate physical video memory blocks from the resource pool using the reference granularity set by the target container; if the transient difference in video memory demand is not lower than the preset stability threshold and the duration reaches a preset sampling window, it triggers a second-order adaptive expansion arbitration to allocate physical video memory blocks; the stability threshold is a limit for determining the stable state of video memory demand in the target container. The dynamic adjustment module controls the computing node to reserve a safety buffer memory for each heterogeneous graphics processor. When the total allocation quota of the resource pool reaches the preset safety warning quota, and the transient difference in the memory demand of the target container exceeds the preset stability threshold, the safety buffer memory is mapped to the target container. The safety warning quota is the total resource usage alarm threshold of the resource pool. The safety buffer memory is the memory resource reserved in the resource pool in advance to cope with the instantaneous memory burst of the task. The scheduling agent module, when the target container exhausts the physical video memory on the current graphics processor and there is no available physical video memory block in the computing node, overflows the pre-marked read-only tensor data in the target container to the physical video memory of the graphics processor on the same node, and copies the virtual page table during the mapping write; when the target container initiates a write operation on the overflowed read-only tensor data, the module suspends the instruction sending of the target container, asynchronously swaps the read-only tensor data back to the physical video memory block of the graphics processor, and then resumes the instruction sending to complete the heterogeneous GPU cluster scheduling.

Images