Cooperative scheduling method and system based on GPU topology and distributed cache

Abstract

The application discloses a kind of based on GPU topological structure and distributed cache cooperative scheduling method and system, including GPU topological information acquisition module, cache state monitoring module, topological perception scheduling engine and cache preheating coordinator, GPU topological information acquisition module is used to collect the topological connection information between physical GPU on node and generates topological matrix, cache state monitoring module is used to obtain the cache performance parameter of distributed cache group, topological perception scheduling engine is used to traverse cluster node and filters candidate node satisfying resource condition, the physical GPU on candidate node is grouped and topological score is calculated, the node score is determined by comprehensively considering GPU topological score and cache state score, and cache preheating coordinator is used to trigger cache preheating operation after scheduling decision is completed.For solving the problem that GPU scheduling ignores hardware topological structure, resulting in limited communication bandwidth, data cache and GPU scheduling are not coordinated, resulting in the delay of the first epoch start.

Description

Technical Field

[0001] This application relates to the field of GPU resource container scheduling technology in artificial intelligence training scenarios, specifically to a scheduling method that optimizes multi-GPU training performance by considering GPU hardware topology, and a technology that improves data loading efficiency by combining distributed file system caching mechanisms. In particular, it relates to a collaborative scheduling method, system, and storage medium based on GPU topology and JuiceFS cache. Background Technology

[0002] In artificial intelligence training scenarios, the efficient scheduling of GPU resources directly impacts model training efficiency. As the scale of deep learning models continues to increase, multi-GPU collaborative training has become the mainstream solution. To achieve flexible allocation and efficient utilization of GPU resources, containerization technology is widely used in GPU cluster management.

[0003] In existing technologies, Volcano, as a cloud-native batch scheduling system, has been widely used in GPU container scheduling scenarios. This system reports GPU device information to the Kubernetes cluster through the NVIDIA GPU Device Plugin, supporting the partitioning of physical GPUs into multiple virtual GPUs (vGPUs) to achieve fine-grained management of GPU resources. The Volcano scheduler makes Pod scheduling decisions based on resource requests, supports MIG hardware-level virtualization and vCUDA software-level virtualization, and provides a Gang scheduling mechanism to ensure that Pods for distributed training tasks start simultaneously. This technical solution can improve the utilization and resource isolation of a single GPU.

[0004] However, existing Volcano GPU scheduling technology primarily relies on round-robin and binpack scheduling strategies. When allocating multiple vGPUs, it only considers whether the number of GPU resources meets the request, without taking into account the physical connection methods between GPUs. Modern GPU servers have complex hardware topologies, with GPUs connected via different link types such as NVLink, PCIe Bridge, and Host Bridge. The communication bandwidth of these different link types varies greatly. For example, the bandwidth of an NVLink connection can reach 300-600 GB / s, while the bandwidth of a SYS connection across NUMA nodes is only a fraction of the PCIe Gen4 bandwidth. When the scheduler randomly allocates GPU combinations, it may lead to low-speed communication between GPUs, making data exchange between GPUs (such as All-Reduce and gradient aggregation) a training bottleneck, increasing overall training time, and decreasing GPU utilization.

[0005] In terms of data loading, JuiceFS, as a high-performance distributed file system, provides a three-level caching mechanism: kernel page cache, client cache, and local disk cache, and supports distributed cache group mode. In AI training scenarios, JuiceFS's caching mechanism can avoid repeated readings from object storage in subsequent epochs by caching accessed data blocks. However, in existing technologies, data caching strategies run independently of GPU container scheduling; data is only loaded into the cache after a container is scheduled to a node, resulting in a significant amount of GPU idle time waiting for data loading during the first epoch of training. Furthermore, the lack of intelligent cache sharing and warm-up mechanisms when multiple training tasks run simultaneously, and the caching strategy not considering GPU topology, can lead to GPUs with superior hardware connectivity becoming bottlenecks due to slow data loading.

[0006] Existing distributed AI job collective communication methods based on Kubernetes combine GPU emulation and proxy plugins with AI topology operators to construct GPU and RDMA network card resource topologies, utilizing NVLink or PCIe high-speed channels for data transmission. While this technology constructs resource topologies, it lacks a link type-based quantitative scoring mechanism, making it unable to select the GPU combination with optimal communication performance during scheduling decisions, and it does not involve a cache collaborative warm-up mechanism.

[0007] There are also batch scheduling systems and methods based on interference levels and communication costs. These systems optimize scheduling strategies using a hybrid frog-jumping algorithm by calculating the model synchronization cost and communication cost between GPUs. While this technique considers communication costs, its calculation is based on model synchronization costs rather than physical hardware topology link types. Therefore, it cannot directly reflect the communication performance differences between different links such as NVLink and PCIe, and it does not involve the coordinated optimization of data caching and scheduling.

[0008] Therefore, a new technical solution is needed that can perceive the physical topology connection type between GPUs during GPU container scheduling, quantitatively evaluate the communication performance of GPU combinations, and deeply couple data cache preheating with scheduling decisions to improve the overall efficiency of multi-GPU training. Summary of the Invention

[0009] The purpose of this invention is to provide a collaborative scheduling method and system based on GPU topology and distributed caching, so as to solve the problems in the prior art where GPU scheduling ignores hardware topology, resulting in limited communication bandwidth; data caching and GPU scheduling are not coordinated, resulting in delay in the first epoch start; and topology-aware scheduling is limited to between nodes and does not consider the physical topology of GPUs within a single node.

[0010] Specifically, the following issues need to be addressed: Addressing the issue of insufficient GPU topology awareness: By sensing the physical connection types between GPUs (NVLink / PIX / PXB / PHB / NODE / SYS), the communication performance of GPU combinations is quantified and scored, and the GPU combination with the optimal topology is prioritized for scheduling. This fully leverages the data exchange performance between multiple GPUs and improves the overall training efficiency of GPUs.

[0011] Addressing the issue of uncoordinated data caching and GPU scheduling: While scheduling GPU containers, the JuiceFS distributed cache is intelligently preheated to ensure that the required dataset is already in the cache when a container is scheduled to a node, shortening the startup time of the first epoch and improving GPU utilization.

[0012] Achieve topology-aware fine-grained scheduling: Within a single node, GPUs are grouped and scored based on the GPU topology; among multiple nodes, nodes with high cache hit rates and optimal GPU combinations are selected based on the topology distribution of the JuiceFS distributed cache group.

[0013] To achieve the above objectives, the present invention adopts the following technical solution: A collaborative scheduling system based on GPU topology and distributed caching, applied to container GPU clusters, includes a GPU topology information acquisition module, a cache status monitoring module, a topology-aware scheduling engine, and a cache preheating coordinator. The GPU topology information acquisition module collects topology connection information between physical GPUs on cluster nodes and generates a topology matrix. The cache status monitoring module obtains cache performance parameters of the distributed cache group. The topology-aware scheduling engine traverses cluster nodes and filters candidate nodes that meet resource conditions, groups the physical GPUs on the candidate nodes, calculates topology scores, and determines the node score by combining the GPU topology score and the cache status score. The cache preheating coordinator triggers cache preheating operations after the scheduling decision is completed.

[0014] In this application, the technical terms are uniformly explained as follows: The GPU topology information acquisition module is also called "topology collector" or "topology management module" and other variants.

[0015] Topology-aware scheduling engine, also known as "scheduler" or "scheduling module" or other variants.

[0016] The cache warm-up coordinator is also called a "warm-up module" or "warm-up manager" or other variants.

[0017] Link type, also known as "connection type" or "topology type" and other variants.

[0018] Cache hit rate, also known as "cache utilization" or "cache coverage" and other variations.

[0019] Cluster: Generally refers to a combination or joint operation of multiple computers in a computer system. In the context of Kubernetes, a cluster refers to a system consisting of a group of nodes that work together to run and manage containerized applications.

[0020] A container is a virtualization technology that allows an application and all its dependencies to run in an isolated environment. Containers are more lightweight than traditional virtual machines and are commonly used to deploy microservice architectures and cloud-native applications.

[0021] vGPU (virtual GPU): This is an abbreviation for "virtual GPU," referring to a virtualized graphics processing unit. vGPU allows the sharing of physical GPU resources within a virtualized environment, which is extremely useful for applications that require graphics acceleration.

[0022] Preferably, the GPU topology information acquisition module reports topology information through Kubernetes custom resource definitions, and the topology-aware scheduling engine obtains topology information through the Watch mechanism.

[0023] Preferably, in the topology-aware scheduling engine, the link type scoring rules are as follows: NVLink type gets 100 points, PIX type gets 50 points, PXB type gets 40 points, PHB type gets 30 points, NODE type gets 20 points, and SYS type gets 10 points.

[0024] Preferably, it also includes a dataset fingerprint recognition module, used to calculate the fingerprint information of the training dataset and store the mapping relationship between the fingerprint and the cache state. The fingerprint information includes the file list hash value, total size and modification time.

[0025] This invention also provides a collaborative scheduling method based on GPU topology and JuiceFS cache, applied to container GPU clusters, comprising the following steps: collecting topology connection information between physical GPUs on each node in the cluster; obtaining cache performance parameters of the distributed cache group; traversing cluster nodes and filtering candidate nodes that meet container resource requests; grouping the physical GPUs on the candidate nodes according to the number of virtual GPU usage requests, calculating the topology score for each group, and selecting the group with the highest topology score; calculating the node's comprehensive score based on the topology score and the cache performance parameters; selecting the node with the highest comprehensive score and scheduling the container to that node; and triggering a cache preheating operation after the scheduling decision is completed.

[0026] In the above technical solution, before collecting GPU topology information, a GPU resource usage request initiated by the container is received and the task is parsed. The request includes virtual GPU allocation information, video memory requirements, and the identifier of the dataset to be trained.

[0027] Specifically, the extracted task requirement parameters after parsing include: GPU resource requirements: number of virtual GPUs, video memory quota, computing power quota; Data requirements: Identifier of the dataset to be trained, storage path of the dataset, and data format type; Task type identifier: distinguishes between single-machine multi-GPU training, distributed multi-machine training, or inference tasks.

[0028] Specifically, task parsing also includes: parsing the parallel strategy identifier of the training task, including data parallelism (DP), model parallelism (MP), or pipeline parallelism (PP); determining the communication mode requirements between GPUs based on the parallel strategy identifier, prioritizing GPU combinations with NVLink high-speed interconnect if it is data parallelism, and prioritizing GPU combinations with matching memory capacity if it is model parallelism.

[0029] Specifically, task parsing also includes: parsing the access pattern characteristics of the dataset, including: Sequential read mode or random shuffle mode; Number of training epochs; Data preprocessing requirements; Generate a cache preheating strategy based on the access pattern characteristics. For the random shuffling mode, pre-calculate the data block access order for each epoch and preload hot data blocks.

[0030] In the above technical solution, the GPU topology connection information includes NVLink connection relationships, PCIe Bridge hierarchical structure, NUMA node affiliation, and QPI / UPI cross-slot connection status, which is collected through the topology hint interface of the NVIDIA NVML library or Kubernetes DevicePlugin.

[0031] Preferably, the step of collecting topology connection information includes constructing an N×N topology connection matrix, where matrix element T[i][j] represents the connection type between GPU i and GPU j, and the connection type includes NVLink type, PIX type, PXB type, PHB type, NODE type, and SYS type.

[0032] Preferably, the process first selects schedulable nodes with sufficient remaining idle virtual GPUs to meet the usage request, and then performs physical GPU grouping and comprehensive scoring on each schedulable node: calculating the topology score for each GPU combination based on GPU topology information, and calculating the cache status score based on JuiceFS cache status information; the sum of the two parts is the node's comprehensive score. That is: Node Comprehensive Score = max(Topology Score of All GPU Combinations) + Cache Status Score.

[0033] Preferably, schedulable nodes are filtered and selected based on the following resource conditions: Check if the number of idle vGPUs on the node meets the container request; check if the remaining GPU memory meets the request amount; check if the remaining GPU computing power meets the request amount; check if the GPU type matches; if any one of these conditions is not met, skip the node directly.

[0034] Preferably, for each GPU group, the following steps are taken: calculate the link type between all pairs of GPUs within the group, calculate the link type score according to the topology scoring rules, and calculate the score for each GPU group; select the group with the highest score as the node GPU group communication score or GPU group topology score.

[0035] Preferably, a weight score is assigned to the connection between each pair of physical GPUs in each group according to the link type, wherein NVLink connection is given the highest weight and SYS connection across NUMA nodes is given the lowest weight; the weight scores of all physical GPU pairs in the group are summed to obtain the topology score of the group.

[0036] Preferably, the topology score of the GPU combination is calculated in two modes based on the communication volume of the training task: Communication-sensitive mode: Based on the estimated inter-GPU communication volume for the task, the link type score of each pair of GPUs in the combination is multiplied by the communication volume coefficient and the weight coefficient, and then summed; GPU pairs with high communication volume are given higher weights, and high-speed links such as NVLink score higher. Default mode: Ignoring traffic differences, the link type scores of all GPU pairs within the combination are directly summed, with each link type scored according to a preset value; NVLink is the highest and cross-NUMA is the lowest; each GPU pair is only calculated once.

[0037] Preferably, the step of calculating the topology score further includes adding an extra score to the topology score when all selected GPU combinations are located on the same NUMA node. Preferably, an extra 50 points are added.

[0038] Preferably, calculating the cache status score based on JuiceFS cache status information includes: querying the proportion of data blocks that have been resident in the local disk cache and memory cache of the dataset to be trained in the candidate node, and recording it as the local hit rate; setting the tiered level according to the cache hit rate of the node in the JuiceFS cache group, and the tiered level adopts a step-by-step scoring rule: the higher the hit rate, the higher the score, and no score is added if the hit rate is less than half.

[0039] Specifically, the scoring rules for cache status are as follows: 30 points are added when the cache hit rate is greater than 90%, 20 points are added when the cache hit rate is between 70% and 90%, 10 points are added when the cache hit rate is between 50% and 70%, and no points are added when the cache hit rate is less than 50%.

[0040] The concise four-tier threshold has the following effects: >90% (30 points): Excellent cache state, ensuring that the GPU utilization reaches more than 98% in the first epoch; >70% (20 points): Good cache state, only a small amount of supplementary loading is needed to start; >50% (10 points): Passable cache state, there is IO risk but it is acceptable; ≤50% (0 points): Failed state, forced to trigger warm-up or avoid selection.

[0041] Compared to linear scoring, this invention adopts a tiered scoring rule, which can avoid scheduling jitter and enhance scheduling stability.

[0042] At the same time, it can also clearly define the cache quality threshold: 50% is used as a hard threshold for whether it is usable (no points are added if it is below), ensuring that only nodes with basic cache foundation can get the scoring advantage, and avoiding scheduling to half-hot and half-cold nodes, which will lead to IO bottlenecks.

[0043] Preferably, a JuiceFS cache warm-up operation is triggered before the container starts, preloading the dataset to be trained into the local cache of the target node.

[0044] Preferably, the step of triggering the cache preheating operation is implemented through the warmup command of the JuiceFS file system, using multi-threaded parallel preheating.

[0045] Preferably, the triggering of the JuiceFS cache preheating operation includes: sending a prefetch instruction to the JuiceFS client of the target node after the container scheduling decision is determined and before the container is actually started; the prefetch instruction loads data blocks according to the epoch order priority of the training dataset to ensure that the data required for the first epoch has been cached before the container starts; for multi-round training scenarios, the JuiceFS client cache sharing mechanism is used to maintain the resident state of the dataset in the cache for reuse in subsequent epochs.

[0046] Preferably, after the topology-aware scheduling engine or scheduler collects scores from all nodes that meet the resource conditions, it selects the target node by performing the following selection: Sort nodes in descending order of their scores; select the node with the highest score; if multiple nodes have the same score, prioritize the node with the larger cache space.

[0047] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described cooperative scheduling method.

[0048] The topology-aware scheduling engine of this invention is integrated into the Volcano scheduler and implemented as a custom scheduling plugin, GPU-topology-aware. This plugin is declared in the Volcano scheduler's configuration file, volcano-scheduler.conf.

[0049] This invention implements a custom scheduling plugin, `GPU-topology-aware`, based on the Volcano scheduler. The GPU-topology-aware plugin resides in the scheduler's tiers level, executing after the gang scheduling plugin and before the predicates filtering plugin, ensuring that topology-aware scoring is completed before resource filtering.

[0050] This invention also provides solutions for multi-round iterative AI training scenarios: A GPU topology-aware scoring method is characterized by: constructing a topology matrix based on the topology connection types between physical GPUs on a node; calculating the link type scores between all pairs of GPUs within an arbitrary GPU combination; calculating the total score of the GPU combination based on the base scores and weight coefficients corresponding to the link types; and evaluating the communication performance of the GPU combination based on the total score.

[0051] A GPU grouping selection method, applied to container GPU scheduling, is characterized by: selecting a list of candidate GPUs that meet resource conditions from all physical GPUs of a node; if the number of candidate GPUs is not less than the number of vGPUs requested by the container, then selecting the requested number of GPUs from the candidate GPUs to form a combination; calculating the total score for each GPU combination, the total score including the link type score between each pair of GPUs in the combination and the NUMA affinity bonus; and selecting the GPU combination with the highest total score as the optimal GPU allocation scheme for the node.

[0052] A cache co-warming method is applied to GPU container training scenarios, characterized by: receiving a virtual GPU usage request from the container, the request containing the path to the required training dataset; immediately triggering a cache warming operation for the dataset after completing the container scheduling decision and determining the target node; the cache warming operation is implemented through the warmup command of the JuiceFS file system, preloading the dataset data into the local cache or distributed cache group of the target node; the warming operation is executed in parallel with container startup, without blocking container startup.

[0053] A node comprehensive scoring method is applied to GPU container scheduling, characterized by: calculating the GPU topology score of a node, the GPU topology score being based on the link type score and NUMA affinity bonus of the optimal GPU combination on the node; querying the JuiceFS cache hit rate of the node, and adding cache bonus to the node according to the numerical range of the hit rate; the final score of the node is equal to the sum of the GPU topology score and the cache score; and selecting the optimal node for container scheduling based on the final score of the node.

[0054] A data caching optimization method for AI training scenarios, applied to a distributed file system, is characterized by: calculating a dataset fingerprint for the training dataset, the fingerprint including the file list hash value, total size, and modification time; storing the dataset fingerprint and corresponding cache status in the file system metadata; directly reusing the existing cache when a re-access request for the same dataset is detected; predicting the usage popularity of the dataset based on the periodicity of the training task, prioritizing cache eviction for cold datasets, and retaining cache for hot datasets.

[0055] Compared with the prior art, the beneficial effects of the present invention are: (1) Significantly improves GPU utilization and shortens training time.

[0056] To address the shortcomings of existing technologies that focus solely on GPU resource allocation while neglecting topology communication overhead, this invention senses the physical connection types between GPUs and quantifies and scores the communication performance of GPU combinations. It prioritizes scheduling GPU combinations with optimal topologies, fully leveraging the advantages of high-speed interconnects such as NVLink, and avoiding All-Reduce bottlenecks caused by cross-NUMA or low-speed PCIe bridging. This maximizes the data exchange performance between multiple GPUs, improving overall GPU training efficiency.

[0057] Actual tests show that this invention can increase GPU utilization from 50%-60% in existing technologies to over 85%, improve communication performance by 140% in 4-card training scenarios, and accelerate overall training by 22%.

[0058] (2) The training time of the first epoch is greatly shortened, realizing the collaboration between computation and data.

[0059] To address the GPU idleness issues caused by the independent operation of data caching and container scheduling in existing technologies, this invention innovatively introduces a cache preheating coordinator after the scheduling decision, completing JuiceFS data preloading before container startup. This ensures that the required dataset is already in the cache when the container is scheduled to the node, shortening the startup time of the first epoch and improving GPU utilization.

[0060] Compared to traditional scheduling methods that require waiting for the first epoch data to be loaded from object storage in real time (which takes several minutes to tens of minutes), this invention triggers the JuiceFS cache preheating operation immediately after the scheduling decision is completed, and uses the waiting time to load the dataset in parallel, so that "data waits for people" rather than "people wait for data". This can shorten the start time of the first epoch by more than 80%, and the first epoch can achieve a GPU utilization rate of 93%-98% (the first epoch of existing technologies is usually only 49%).

[0061] (3) Improve distributed caching efficiency, reduce storage layer pressure, and optimize storage bandwidth utilization.

[0062] To address the shortcomings of existing technologies in lacking cache collaborative scheduling, this invention improves the JuiceFS cache hit rate, reduces access to the underlying object storage, saves storage bandwidth, and increases overall training throughput through cache collaborative preheating and intelligent eviction strategies.

[0063] Existing technology: Cache hit rate is 70%-80% (according to JuiceFS documentation); This invention: Through intelligent preheating and eviction, the cache hit rate is stabilized at over 90%.

[0064] After the cache hit rate increased from 80% to 90%, the number of object storage accesses decreased by 50% (from 20% to 10%); assuming an object storage bandwidth of 100 GB / s, 10 GB / s of bandwidth was saved; more concurrent training tasks can be supported, improving the overall throughput of the cluster.

[0065] (4) Optimization of multi-node distributed training performance.

[0066] In multi-node distributed training scenarios, by combining the JuiceFS distributed cache group topology, node combinations with high cache hit rates and excellent GPU combinations are selected to avoid frequent cross-node access to object storage and improve distributed training efficiency.

[0067] Because the distributed cache group uses a consistent hashing algorithm, the data is evenly distributed; intelligent preheating ensures that each node has cached the required data blocks; and cross-node communication prioritizes high-bandwidth links (NVLink / high-bandwidth RDMA network).

[0068] Based on JuiceFS's practical test results for BERT training on a 1000 GPU scale: No topology awareness: GPU utilization dropped from 98% with 1000 cards to 95% with 500 cards; This invention features topology awareness: GPU utilization remains above 97% even with 500 GPUs.

[0069] In distributed training at a scale of 1000 GPUs, through topology awareness and cache collaboration mechanisms, GPU utilization is maintained at over 97%, which is 2% higher than the 95% of existing technologies.

[0070] (5) Achieve fine-grained scheduling of single nodes to make up for the blind spots of existing topology awareness.

[0071] Addressing the limitations of existing technologies that only focus on cross-node network topology or perform simple filtering, this invention fills the gap in physical topology awareness of multiple GPUs within a single node. By using fine-grained topology information such as PCI-E hierarchy and NUMA node affiliation, it achieves more accurate resource allocation than existing vGPU scheduling or simple grouping and scoring, fully leveraging the hardware potential of modern 8 / 16 GPU servers and avoiding resource mismatch due to superior hardware connectivity but poor topology combination.

[0072] (6) It is compatible with the existing cloud-native ecosystem and is easy to deploy and implement.

[0073] This invention is based on the Volcano scheduler and is a plug-in extension. It can be seamlessly integrated with existing GPU Operator, Device Plugin and JuiceFS CSI driver without modifying the Kubernetes core code or user training program (unlike CUDA API hijacking schemes). It has zero invasiveness and good engineering feasibility. Attached Figure Description

[0074] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of the collaborative scheduling system of the present invention.

[0075] Figure 2 This is a flowchart illustrating the process of nodes reporting topology information to GPU resource filtering in this invention.

[0076] Figure 3 This is a flowchart of the process from GPU resource selection to cache score calculation in this invention.

[0077] Figure 4 This is a process diagram of the present invention, from calculating cache scores to preheating and starting the cache container.

[0078] Figure 5 This is a flowchart of the first half of the GPU resource screening process in this invention (a detailed process of GPU topology-aware scoring).

[0079] Figure 6 This is a flowchart of the second half of the GPU resource screening process in this invention (a detailed process of GPU topology-aware scoring). Detailed Implementation

[0080] The specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other.

[0081] The collaborative scheduling method based on GPU topology and JuiceFS cache provided by this invention is applied to container GPU clusters. By sensing the physical connection type between GPUs and combining the distributed file system cache status, it makes intelligent scheduling decisions, which significantly improves GPU utilization and data loading efficiency in AI training scenarios.

[0082] Example 1 This embodiment provides a collaborative scheduling system and method based on GPU topology and JuiceFS cache, which fully implements GPU topology-aware scoring mechanism, cache status monitoring mechanism, comprehensive scheduling decision mechanism and cache preheating mechanism.

[0083] like Figure 1 As shown, the collaborative scheduling system in this embodiment includes a GPU topology information acquisition module, a JuiceFS cache status monitoring module, a topology-aware scheduling engine, a cache preheating coordinator, and a Volcano scheduling plugin extension module. The modules are electrically connected to each other through the internal network of the Kubernetes cluster.

[0084] The core components of the collaborative scheduling system architecture are introduced below: GPU Topology Information Acquisition Module: Responsible for collecting topology connection information between physical GPUs on the node, including link type, bandwidth, latency, etc.

[0085] JuiceFS Cache Management Module: Responsible for managing the JuiceFS distributed cache group, monitoring cache hit rate and hot data, and supporting cache preheating.

[0086] Topology-aware scheduling engine: The core scheduling algorithm, which combines GPU topology scoring and cache status to make optimal scheduling decisions.

[0087] Cache preheating coordinator: Intelligently triggers cache preheating operations after the container scheduling decision.

[0088] Volcano Scheduling Plugin Extension Module: Extends topology awareness and cache coordination capabilities on top of the existing Volcano scheduler.

[0089] The GPU topology information acquisition module is deployed on each node of the cluster, implemented through an extended Kubernetes DevicePlugin. Its output is electrically connected to the Kubernetes API Server for reporting GPU topology information. The JuiceFS cache status monitoring module is deployed in the cluster's control plane. Its input is electrically connected to the JuiceFS client on each node for collecting cache performance parameters. The topology-aware scheduling engine is integrated into the Volcano scheduler. Its inputs are electrically connected to both the GPU topology information acquisition module and the JuiceFS cache status monitoring module, and its output is electrically connected to the Kubernetes API Server for executing scheduling decisions. The cache preheating coordinator is deployed in the cluster's control plane. Its input is electrically connected to the topology-aware scheduling engine, and its output is electrically connected to the JuiceFS client on each node for triggering cache preheating operations.

[0090] The complete process of the method of the present invention is as follows: Figures 2-4 The flowchart of the cooperative scheduling method is shown below. It mainly proceeds in the following steps: (1) The workflow of the GPU topology information acquisition module for acquiring and reporting GPU topology information is as follows: Each node in the cluster periodically collects and reports GPU topology information through an extended GPU Device Plugin. The collection process (nodes in the cluster reporting physical GPU topology connection information to the cluster) includes three steps.

[0091] The first step is to construct the GPU link topology matrix. For N GPUs on a node, construct an N×N topology connection matrix. The matrix element T[i][j] represents the connection type between GPU i and GPU j. The connection type values ​​include: NVLink connection, PIX connection, PXB connection, PHB connection, NODE connection, SYS connection, and LOC connection.

[0092] NVLink connection indicates a direct connection via NVLink high-speed interconnect, with the specific link number marked as NV1 to NV18.

[0093] A PIX connection means that the connection is made through only one PCIe Bridge.

[0094] A PXB connection means that it spans multiple PCIe Bridges but does not pass through a Host Bridge.

[0095] A PHB connection indicates a connection via a PCIe Host Bridge.

[0096] NODE connection indicates a connection within the same NUMA node via Host Bridge and PCIe bus.

[0097] A SYS connection indicates a connection across NUMA nodes via QPI or UPI and PCIe.

[0098] A local connection (LOC) means that GPU i is equal to GPU j, i.e., they are on the same GPU.

[0099] Link performance parameters: The bandwidth and latency parameters corresponding to each link type can be queried using the nvidia-smi topo command or obtained using the NVTAGS tool.

[0100] NUMA affinity information: The NUMA node ID associated with each GPU, obtained through the nvidia-smi or hwloc library.

[0101] PCIe topology layers: Information on layers such as the PCIe Root Complex where the GPU resides and the PCIe Bridge.

[0102] Reporting method: Define GPU Topology resource types using Kubernetes CRD (Custom Resource Definition); The Device Plugin on the node creates and updates GPU Topology CRD instances; The Volcano scheduler watches these CRD resources to obtain the latest topology information in real time.

[0103] Communication performance varies significantly across different link types. NVLink connections typically have a bandwidth of 300 to 600 GB / s, offering the best relative performance. PIX connections typically have PCIe Gen4 bandwidth, providing excellent performance. PXB connections typically have PCIe Gen4 bandwidth, offering good performance. PHB connections typically have PCIe Gen4 bandwidth, offering moderate performance. NODE connections typically have PCIe Gen4 bandwidth, offering relatively poor performance. SYS connections typically have PCIe Gen4 bandwidth, offering the worst relative performance.

[0104] The second step is to collect link performance parameters. The bandwidth and latency parameters for each link type can be obtained by querying using the nvidia-smi topo command or the NVTAGS tool.

[0105] Specifically, executing the command `nvidia-smi topo -m` outputs a topology matrix between GPUs, where each matrix element displays the connection type and number of links between two GPUs. For example, the connection type between GPU0 and GPU1 is NV18, indicating a connection via 18 NVLink channels with a theoretical bandwidth of 900GB / s. The connection type between GPU0 and GPU4 is PHB, indicating a connection via PCIe Host Bridge with a theoretical bandwidth of 32GB / s.

[0106] The third step is to collect NUMA affinity information. The NUMA node ID associated with each GPU is obtained using the `nvidia-smi` command or the `hwloc` library. Specifically, executing the command `nvidia-smi topo -m` will simultaneously output the NUMA affinity information of the GPUs. For example, GPUs 0 to 3 are associated with NUMA node 0, and GPUs 4 to 7 are associated with NUMA node 1.

[0107] The reporting method uses a custom Kubernetes resource definition. The Device Plugin on the node creates and updates the GPUTopology CRD instance, and the Volcano scheduler obtains the latest topology information in real time through the Watch mechanism. The definition of the GPUTopology CRD is as follows.

[0108] Example CRD definition: apiVersion: topology.volcano.sh / v1beta1kind: GPUTopologymetadata:name: node-1-GPU-topologyspec:nodeName: node-1GPUCount: 8topologyMatrix:-from: 0to: 1linkType: NV18bandwidth: 900# GB / s- from: 0to: 4linkType:PHBbandwidth: 32- from: 0to: 7linkType: SYSbandwidth: 16numaAffinity:- GPUId: 0numaNode: 0- GPUId: 4numaNode: 1 In the above CRD definition, the nodeName field identifies the node name, the GPUCount field identifies the total number of GPUs on the node, the topologyMatrix field stores the topology connection information between GPUs in the form of an array, each array element contains a from field indicating the starting GPU number, a to field indicating the target GPU number, a type field indicating the link type, a bandwidth field indicating the theoretical bandwidth value, and a numaAffinity field storing the association between GPUs and NUMA nodes.

[0109] (2) The process of JuiceFS cache status monitoring module collecting cache status (JuiceFS distributed cache group topology awareness) is as follows: This invention incorporates the JuiceFS distributed cache group into the topology-aware scope.

[0110] Get the cache status of the JuiceFS distributed cache group, including the cache hit rate, hot data blocks, cache path and cache capacity of each node.

[0111] Cache group topology information collection includes: Get the IP address, network connection method, and network bandwidth of all nodes in the cache group; Analyze the network topology of the cache group (e.g., whether it is in the same rack or under the same switch); Monitor the cache hit rate and hot data distribution of the cache group.

[0112] Cache preheating strategies include the following modes: Active warm-up: Based on historical training task records, predict frequently used datasets and actively warm them up in the cache group during off-peak periods; Passive preheating: When a training task is submitted to the queue, the corresponding dataset is preheated in the cache while waiting for GPU resource allocation; Intelligent preheating order: Based on the node priority of the GPU topology score, the cache of high-scoring nodes is preheated first.

[0113] Example: For a distributed training task that requires the ImageNet dataset, the system makes the following decisions before scheduling: The path to the dataset required for the recognition task ( / mnt / juicefs / imagenet / train); Query the cache status of each node in the cache group; During GPU resource allocation, `juicefs warmup -p 32 / mnt / juicefs / imagenet / train` is triggered in parallel; Ensure that data is preloaded into the local or shared cache when the container starts.

[0114] The JuiceFS cache status monitoring module is deployed on the cluster's control plane. It connects to the JuiceFS client on each node via SSH and periodically executes the `juicefs stats` command to query cache statistics.

[0115] The specific implementation of the query process is as follows: For node-1 and the dataset path / mnt / juicefs / imagenet / train, execute the command ssh node-1 juicefs stats / mnt / juicefs / imagenet / train. The output of this command includes statistics such as cache hit rate, cache capacity, and used cache space.

[0116] The cache status monitoring module parses the above output, extracts the cache hit rate value of 85.3%, converts it to a floating-point number of 0.853, and stores it in the node's cache status record. This record includes fields such as node IP address, dataset path, cache hit rate, and update timestamp, stored in a Redis database as key-value pairs. The key format is cache:node-1: / mnt / juicefs / imagenet / train, and the value format is a JSON object containing a hit_rate field of 0.853, a capacity field of 100GB, a used field of 78GB, and an update_time field containing a Unix timestamp.

[0117] This query operation is executed asynchronously, without blocking the normal operation of the scheduling engine.

[0118] like Figures 5-6 As shown, the topology-aware scheduling engine's topology-aware workflow includes five steps: resource filtering, GPU grouping and topology scoring, NUMA affinity scoring, cache status scoring, node comprehensive scoring, and scheduling decision.

[0119] (3) GPU topology-aware scoring algorithm One of the core innovations of this invention is the GPU topology-aware scoring algorithm, which is used to quantitatively evaluate the communication performance of GPU combinations on a node.

[0120] The scheduler sequentially selects the virtual GPU allocation information and physical GPU topology information of one node in the cluster. It then traverses all nodes in the cluster and performs the following scoring process for each node: (3.1) Based on the nodes, filter the nodes whose remaining idle virtual GPUs can meet the usage requests.

[0121] First, basic resource filtering is performed: After receiving a virtual GPU usage request from a container, the scheduling engine iterates through all nodes in the cluster and performs the following checks on each node: First, check if the number of idle vGPUs on the node is not less than the number of vGPUs requested by the container. Second, check if the remaining GPU memory on the node is not less than the amount of GPU memory requested by the container. Third, check if the remaining computing power on the node's GPU is not less than the computing power requested by the container. Fourth, check if the node's GPU type matches the GPU type requested by the container (e.g., A100, A800, etc.). If any check fails, skip the node and do not proceed with further scoring.

[0122] (3.2) Then, based on the virtual GPU usage request, the physical GPUs on the node are grouped, and the topology connections within each group are scored. The group with the highest score is selected as the node topology score.

[0123] For nodes that meet the resource requirements, GPU grouping and topology scoring are performed. The implementation of the GPU grouping and topology scoring steps is as follows.

[0124] For nodes that pass resource filtering, the scheduling engine first constructs a candidate GPU list. It iterates through all physical GPUs on the node, checking if the sum of its used vGPUs and the vGPUs requested by the container does not exceed the total number of vGPUs for that GPU, if the sum of its used VGPUs and the VGPUs requested by the container does not exceed the total VGPUs for that GPU, if the sum of its used VRAM and the VGPUs requested by the container does not exceed the total VRAM for that GPU, if the sum of its used compute power and the compute power requested by the container does not exceed the total compute power for that GPU, and if the GPU type is equal to the GPU type requested by the container. If all the above conditions are met, the GPU is added to the candidate GPU list. If the length of the candidate GPU list is less than the number of vGPUs requested by the container, the node is skipped.

[0125] If the length of the candidate GPU list is not less than the number of vGPUs requested by the container, then the number of GPUs requested by the container is selected from the candidate GPU list to form a combination. The total number of combinations is C(n, k), where n is the length of the candidate GPU list and k is the number of vGPUs requested by the container. For example, if the candidate GPU list contains 8 GPUs and the container requests 4 vGPUs, then the total number of combinations is C(8, 4), which equals 70.

[0126] For each GPU combination, calculate the topology score for that combination. For example... Figure 2 and 3As shown, the detailed process of topology scoring is as follows: Initialize the scoring variable `score` to 0, initialize the NUMA affinity flag `inSameNUMA` to true, and obtain the NUMA node number of the first GPU in the combination as the reference NUMA node; traverse all GPU pairs in the combination. For GPU i and GPU j where i is less than j, query the link type between GPU i and GPU j from the node topology matrix, look up the corresponding score according to the link type, and accumulate the score into the `score` variable; check whether the NUMA node number of GPU j is equal to the reference NUMA node number. If they are not equal, set the `inSameNUMA` flag to false; if the `inSameNUMA` flag is true, add an extra 50 points to the `score` variable.

[0127] Topology-aware scheduling includes two modes: best-effort and strict. Best-effort mode selects the GPU combination with the optimal topology possible, but if the GPU topology score of all nodes is below a threshold, it degrades to traditional resource-first scheduling. Strict mode strictly requires the GPU topology score to be at least a threshold; if none of the nodes meet this condition, scheduling fails and the system waits for resources to be released. The threshold is set through the Volcano scheduler's configuration file, with a default value of 300.

[0128] GPU grouping algorithm: Assume a node has 8 physical GPUs, and a container requests 4 vGPUs, requiring the selection of 4 GPUs from the 8. The number of combinations is C(8, 4) = 70. For each combination: Calculate the link type between all pairs of GPUs within the combination; score based on link type; calculate the total topology score for the combination.

[0129] Topology scoring rules: Referring to the topology scoring mechanisms of NVIDIA NVTAGS and NCCL, the link type score is defined as shown in Table 1: Table 1 Link Type Topology Scoring Rules

[0130] Scoring formula: For a GPU combination G = {g1, g2, ..., gn}, calculate the score: Communication mode: Score(G) = Σ w_i × (Σ T_jk × link_score(link_type(g_j, g_k))); Simplified scoring method without traffic estimation: Score(G) = Σ link_score(link_type(g_j, g_k)), for any j <k; In both modes, in the GPU combination G = {g1, g2, ..., gn}, j and k are natural numbers from 1 to n, representing the GPU index, and n is a natural number; link_score is the link type score; and Score(G) is the GPU combination score.

[0131] Where: w_i is the weight of the GPU pair, taking into account the communication mode of the training task. For All-Reduce intensive tasks, the weight can be set to 1.0; for parameter server mode, different weights can be set for different GPU pairs; T_jk is the estimated communication volume between GPU j and GPU k, which can be derived from historical training task profile data or model architecture (such as the sparsity of the attention matrix of the transformer layer); link_score is the link type score.

[0132] Simplified rating (without traffic estimation): The following is an example of a node with 8 GPUs requesting a combination of 4 GPUs: Combination A: {GPU0, GPU1, GPU2, GPU3} GPU0 GPU1: NV18 → 100 points; GPU0 GPU2: NV18 → 100 points GPU0 GPU3: NV18 → 100 points GPU1 GPU2: NV18 → 100 points GPU1 GPU3: NV18 → 100 points GPU2 GPU3: NV18 → 100 points Total score: 600 points Combination B: {GPU0, GPU2, GPU4, GPU6} GPU0 GPU2: NV18 → 100 points GPU0 GPU4: PHB → 30 points GPU0 GPU6: PHB → 30 points GPU2 GPU4: PHB → 30 points GPU2 GPU6: PHB → 30 points GPU4 GPU6: NV18 → 100 points Total score: 320 points Clearly, combination A (600 points) is better than combination B (320 points).

[0133] (3.3) NUMA affinity bonus: An additional 50 points are awarded if all selected GPUs reside on the same NUMA node. Communication between GPUs across NUMA nodes requires traversing QPI / UPI, resulting in higher latency.

[0134] (3.4) Final comprehensive score of nodes: Node overall score = max(score of all GPU combinations) + cache status bonus; Cache status points are awarded according to the following rules: If the cache hit rate of the node in the JuiceFS cache group is greater than 90%, add 30 points; if the cache hit rate is greater than 70%, add 20 points; if the cache hit rate is greater than or equal to 50%, add 10 points; if the cache hit rate is less than 50%, no points are added.

[0135] (4) Comprehensive scheduling decision The node comprehensive scoring and scheduling decision-making steps are implemented as follows: A node's final score equals its GPU topology score plus its cache bonus. The scheduling engine collects the final scores of all nodes that have passed resource filtering and sorts them in descending order of score. The node with the highest score is selected as the target node. If multiple nodes have the same score, the node with the larger total cache capacity is prioritized.

[0136] The process of binding a container to a target node is as follows: The scheduling engine calls the Bind interface through the Kubernetes API Server to bind the Pod to the target node. Simultaneously, the scheduling engine allocates GPU devices from the optimal GPU combination to the container through the Allocate interface of the Kubernetes Device Manager. Specifically, the container's resource limit nvidia.com / GPU is set to the requested number of vGPUs, and the container's environment variable CUDA_VISIBLE_DEVICES is set to the list of GPU device numbers from the optimal GPU combination.

[0137] For example, if the optimal GPU combination is GPU0, GPU1, GPU2, and GPU3, then set CUDA_VISIBLE_DEVICES to 0, 1, 2, and 3. This setting ensures that containers can only access the allocated GPUs, avoiding interference with other containers.

[0138] (4.1) From all nodes whose remaining idle virtual GPUs can satisfy the usage request, select the node with the highest score and bind the container to that node: The scheduler collects scores from all nodes that meet the resource conditions: Sort the nodes in descending order of their scores; select the node with the highest score; if multiple nodes have the same score, prioritize the node with the larger cache space; schedule the container to that node.

[0139] (4.2) Group the physical GPUs bound to the container (highest score) and mount them to the target container according to the usage request.

[0140] After a container is scheduled to a node, the scheduler allocates the optimal GPU combination to the container: The Allocate interface (belonging to Kubernetes Device Manager) assigns a specific GPU ID to a container; sets the container's resource limits (nvidia.com / GPU) and GPU device visibility environment variables (CUDA_VISIBLE_DEVICES); and ensures that the container can only access the assigned GPU to avoid interfering with other containers.

[0141] (5) Cache preheating trigger The cache warming coordinator is deployed in the cluster's control plane, and its workflow is as follows: After the container scheduling decision is completed, the scheduling engine sends a warming request to the cache warming coordinator. The request includes the target node's IP address, dataset path, and warming concurrency parameters. Upon receiving the request, the cache warming coordinator connects to the target node via SSH and executes the `juicefswarmup` command to trigger the cache warming operation.

[0142] Ideally, 32 threads should be used to read files in the dataset in parallel and load the file data into the node's local cache or distributed cache group.

[0143] The warm-up operation is executed asynchronously, without blocking container startup. The cache warm-up coordinator returns immediately after initiating the SSH command, without waiting for the warm-up operation to complete. The warm-up operation continues in the background, so when the container starts and begins accessing the dataset, most of the data is already ready in the cache, significantly reducing the data loading time for the first epoch.

[0144] The cache preheating coordinator also supports the optional parameter `fill-group-cache`, which allows preheated data to participate in the sharing of the distributed cache group. When this parameter is enabled, the preheated data is not only stored in the local cache but also distributed to other nodes within the cache group using a consistent hashing algorithm, improving the overall hit rate of the cache group.

[0145] Volcano scheduling plugin extension: When users submit training tasks via PodGroup resources, they can declare topology-aware policies and cache warmup parameters in the PodGroup annotations. In the PodGroup configuration, the `minMember` field indicates that at least 4 Pods need to be started simultaneously, the `minResources` field indicates that a total of 4 GPUs are required, the `topology.volcano.sh / topology-policy` annotation indicates that the topology-aware policy is best-effort, the `cache.volcano.sh / dataset-path` annotation indicates that the dataset path is ` / mnt / juicefs / imagenet / train`, and the `cache.volcano.sh / warmup-parallelism` annotation indicates that the warmup concurrency is 32.

[0146] Example as follows: # Get the path to the dataset required by the container (read from Pod annotation or ConfigMap) DATASET_PATH= / mnt / juicefs / imagenet / train # Trigger cache warmup, using 32 threads in parallel juicefs warmup -p 32 $DATASET_PATH # Optional: Use the --fill-group-cache parameter to allow written data to participate in the cache group simultaneously #juicefs warmup -p 32 --fill-group-cache $DATASET_PATH The workflow of the collaborative scheduling system in this embodiment is as follows: The user submits a PodGroup and its corresponding Pod resources via kubectl. The PodGroup declares the topology-aware policy and cache preheating parameters. After receiving the request, the Volcano scheduler first performs a gang scheduling check to confirm that all Pods in the PodGroup can be scheduled simultaneously. After passing the check, it proceeds to the GPU-topology-aware plugin process.

[0147] The plugin retrieves the cluster node GPUTopology CRD resources from the Kubernetes API Server, constructs a GPU topology matrix, and queries the node cache status from Redis. It then iterates through the nodes and filters out candidate nodes that meet the resource criteria. For each candidate node, the plugin iterates through all GPU combinations, calculates a topology score including link type score and NUMA affinity bonus, selects the combination with the highest score as the optimal allocation scheme, and records it as the node topology score. It then combines this with the cache hit rate queried from Redis to calculate a cache bonus. The sum of the topology score and the cache bonus is used as the node's final score, and the target node is selected after sorting the scores in descending order.

[0148] The plugin binds the Pod to the target node, allocates the optimal GPU device to the container via Device Manager, and sends a warmup request to the cache warmup coordinator. The coordinator executes the `juicefs warmup` command asynchronously via SSH, without blocking container startup, ensuring that the data is cached and ready when the container accesses the dataset, significantly reducing the loading time of the first epoch. After the container starts, it accesses the GPU to perform training via `CUDA_VISIBLE_DEVICES`. The optimal topology improves communication bandwidth, and cache warmup ensures high GPU utilization, significantly improving training performance.

[0149] Testing and Verification: This embodiment is tested and verified on a Kubernetes cluster with 10 nodes. Each node is configured with 8 NVIDIA A800 GPUs, and the GPUs are connected in a hybrid topology using NVLink and PCIe. The cluster deploys the JuiceFS distributed file system, with MinIO as the object storage backend, and a distributed cache group covering all nodes.

[0150] The test task involved training a ResNet-50 model based on the ImageNet dataset (144GB), requesting 4 vGPUs. The performance differences between this embodiment and existing technologies were compared. The results showed that in the 4-GPU training scenario, GPU utilization increased from 60% to 85%, training time was reduced by 33.3%, and the start time of the first epoch was reduced from 82 minutes to 9 minutes (an 80% reduction).

[0151] In distributed training with 1000 GPUs, GPU utilization remained above 97%, verifying the applicability to large-scale clusters. In summary, this invention can achieve: GPU utilization increased to over 85%, training time reduced by 30%-50% under equivalent hardware, and the first epoch startup time reduced by over 80% through cache preheating.

[0152] Example 2 This embodiment, based on Embodiment 1, adds a dataset fingerprinting module and an intelligent cache eviction module to improve cache hit rate and space utilization. The dataset fingerprinting module is deployed in the control plane, extracts the dataset path from the PodGroup annotation, generates a JSON-formatted fingerprint by traversing files, calculating SHA-256 hash values, and calculating the dataset size and modification time. The fingerprint is stored in Redis, and a mapping between the fingerprint and the cache state is established. The cache can be reused during repeated accesses, adapting to multi-round iterative AI training scenarios.

[0153] The intelligent cache eviction module is also deployed in the control plane. It periodically scans the training task records in Redis, and combines the task status (running / completed / cancelled) and historical access data to mark the popularity of the dataset. When the cache is strained, data is evicted according to its popularity priority, which is superior to the traditional LRU strategy. Both modules are deeply integrated with the original collaborative scheduling system, prioritizing cache reuse when a task is submitted, and dynamically updating the popularity and performing intelligent eviction during execution.

[0154] The tests were conducted in the same environment as in Example 1, with the task being 10 epochs of ResNet-50 training on the ImageNet dataset (access order randomized). The results showed that the cache hit rate increased from 80% to 90%, and the cache space utilization improved by 20%. From the second to the tenth epoch, the cache hit rate stabilized above 88%, and when the cache space utilization reached 95%, the eviction module could ensure the retention of hot data and avoid performance degradation.

[0155] Comparative Example 1 This comparative example is used to verify the technical effectiveness of the GPU topology-aware scoring mechanism, comparing the existing random GPU allocation scheme (Comparison Scheme A) with the topology-aware scoring scheme of this invention (Comparison Scheme B).

[0156] Comparison scheme A uses the Volcano scheduler's default binpack plugin and round-robin strategy, without considering GPU topology relationships. When a container requests 4 vGPUs, it randomly selects one from the node's 8 GPUs without performing topology scoring. Comparison scheme B uses the GPU-topology-aware plugin from Example 1, which performs topology scoring on all GPU combinations on the node and selects the combination with the highest score for allocation.

[0157] The test platform was a 10-node Kubernetes cluster, with 8 NVIDIA A800 GPUs per node. GPUs 0-3 and 4-7 were connected via NVLink, while GPUs 3 and 4, and 7 and 0 were connected via PHB and SYS, respectively. The training task was a 144GB ImageNet dataset ResNet-50 training session, with 90 epochs and 256 batches. The container requested 4 vGPUs (100% computing power, 24GB VRAM). The comparison metrics were GPU utilization, training time, inter-GPU communication bandwidth, and training throughput. The comparison results are shown in Table 2.

[0158] Table 2: Performance Comparison of GPU Topology-Aware Scoring Mechanisms (ImageNet ResNet-50 Training, 4 GPUs, 90 epochs)

[0159] Table 2 shows that Scheme A, with random GPU allocation, achieves 60% utilization, 12 hours of training time, 45GB / s communication bandwidth, and 850 samples / second throughput. Scheme B, with topology-aware allocation, achieves 85% utilization, reduces training time to 8 hours, has 630GB / s bandwidth, and 1275 samples / second throughput. The performance improvement stems from topology optimization. Scheme A suffers from low-speed link bottlenecks, while Scheme B uses a fully connected NVLink combination to reduce GPU idle time. Actual measurements show a 33.3% reduction in training time, exceeding the theoretical value of Amdahl's Law and significantly improving training efficiency.

[0160] Comparative Example 2: Verifying the effectiveness of the cache preheating mechanism by comparing the existing cache-free preheating scheme (Comparative Example C) with the cache preheating scheme of this invention (Comparative Example D). In Example C, the dataset is read from JuiceFS only after container scheduling and startup. Because the data is not cached, it needs to be downloaded from object storage, resulting in a long loading time for the first epoch. In Example D, JuiceFS cache preheating is triggered immediately after scheduling, and the data is preloaded into the cache before container startup. The test platform is the same as Comparative Example 1, with a training task of 144GB ImageNet dataset ResNet-50 training, batch size of 256, and container requests of 4 vGPUs; JuiceFS is configured with 100GB local SSD cache, 32-thread preheating concurrency, object storage is MinIO, and network bandwidth is 10Gbps. The comparison metrics are the startup time of the first epoch, GPU utilization, and training time. The results are shown in Table 3.

[0161] Table 3: Performance Comparison of Caching Preheating Mechanisms (ImageNet ResNet-50 Training, 4 GPUs, First Epoch)

[0162] Table 3 shows that Solution C has a first epoch start time of 82 minutes, GPU utilization of 49%, and training time of 95 minutes; Solution D reduces these to 9 minutes, improves GPU utilization to 93%, and reduces training time to 15 minutes, respectively. The reason is that Solution C needs to download data from object storage, while Solution D triggers 32 threads for parallel warm-up after scheduling, completing loading in 48 seconds, eliminating GPU idle waiting, and significantly improving training efficiency and resource utilization.

[0163] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A collaborative scheduling system based on GPU topology and distributed caching, characterized in that... This system is applied to container GPU clusters and includes a GPU topology information acquisition module, a cache status monitoring module, a topology-aware scheduling engine, and a cache preheating coordinator. The GPU topology information acquisition module is used to collect topology connection information between physical GPUs on cluster nodes and generate a topology matrix. The cache status monitoring module is used to obtain cache performance parameters of distributed cache groups. The topology-aware scheduling engine is used to traverse cluster nodes and filter candidate nodes that meet resource conditions, group the physical GPUs on the candidate nodes and calculate topology scores, and determine the node score by combining the GPU topology score and the cache status score. The cache preheating coordinator is used to trigger the cache preheating operation after the scheduling decision is completed.

2. The collaborative scheduling system based on GPU topology and distributed caching according to claim 1, characterized in that... It also includes a dataset fingerprinting module, which is used to calculate the fingerprint information of the training dataset and store the mapping relationship between the fingerprint and the cache state. The fingerprint information includes the file list hash value, total size and modification time.

3. The collaborative scheduling system based on GPU topology and distributed caching according to claim 1, characterized in that... Deploy an intelligent cache eviction module in the cluster's control plane to predict the usage frequency of datasets based on the status of training tasks and periodicity. For high-frequency, high-training datasets with long average training times, prioritize their retention when cache space is limited. For low-frequency, low-training datasets with short average training times, prioritize their eviction when cache space is limited.

4. A collaborative scheduling method based on GPU topology and JuiceFS cache, characterized in that... The method, applied to container GPU clusters, includes the following steps: collecting topology connection information between physical GPUs on each node in the cluster; obtaining cache performance parameters of the distributed cache group; traversing the cluster nodes and filtering candidate nodes that meet the container resource requests; grouping the physical GPUs on the candidate nodes according to the number of virtual GPU usage requests, calculating the topology score for each group, and selecting the group with the highest topology score; calculating the node's comprehensive score based on the topology score and the cache performance parameters; selecting the node with the highest comprehensive score and scheduling the container to that node; and triggering a cache preheating operation after the scheduling decision is completed.

5. The collaborative scheduling method based on GPU topology and JuiceFS cache according to claim 4, characterized in that... The GPU topology connection information includes NVLink connection relationships, PCIe Bridge hierarchy, NUMA node affiliation, and QPI / UPI cross-slot connection status, which is collected through the topology hint interface of the NVIDIA NVML library or Kubernetes Device Plugin.

6. A GPU topology-aware scoring method, characterized in that: Construct a topology matrix based on the topology connection types between physical GPUs on the nodes; for any GPU combination, calculate the link type score between each pair of GPUs in the combination; calculate the total score of the GPU combination based on the base score and weight coefficient corresponding to the link type. The overall score is used to evaluate the communication performance of the GPU combination.

7. A GPU grouping selection method, applied to container GPU scheduling, characterized in that: From all physical GPUs on the node, a list of candidate GPUs that meet the resource conditions is selected; if the number of candidate GPUs is not less than the number of vGPUs requested by the container, the requested number of GPUs are selected from the candidate GPUs to form a combination; the total score of each GPU combination is calculated, the total score including the link type score between each pair of GPUs in the combination and the NUMA affinity score; the GPU combination with the highest total score is selected as the optimal GPU allocation scheme for the node.

8. A cache collaborative warm-up method, applied to GPU container training scenarios, characterized in that: The system receives a virtual GPU usage request from the container, which includes the path to the required training dataset. After completing the container scheduling decision and determining the target node, it immediately triggers a cache warm-up operation for the dataset. The cache warm-up operation is implemented through the warmup command of the JuiceFS file system, which preloads the dataset data into the local cache or distributed cache group of the target node. The warm-up operation is executed in parallel with the container startup and does not block the container startup.

9. A node comprehensive scoring method applied to GPU container scheduling, characterized in that: The system calculates the GPU topology score of a node, which is based on the link type score and NUMA affinity bonus of the optimal GPU combination on the node; queries the JuiceFS cache hit rate of the node, and adds cache bonus to the node according to the numerical range of the hit rate; the final score of the node is equal to the sum of the GPU topology score and the cache score; and selects the optimal node to schedule containers based on the node's final score.

10. A data caching optimization method for AI training scenarios, applied to a distributed file system, characterized in that: Calculate a dataset fingerprint for the training dataset, the fingerprint including the file list hash value, total size, and modification time; Store the dataset fingerprint and the corresponding cache state in the file system metadata; When a re-access request for the same dataset is detected, the existing cache is reused directly; Based on the periodic prediction of training tasks, the usage frequency of the dataset is determined, and the cache is evicted first for cold datasets and retained for hot datasets.

Images