An AI model training and prediction integrated system for heterogeneous computing power virtualization scheduling

Abstract

The application relates to the technical field of computing resource scheduling, and discloses an AI model training and pushing integrated system for heterogeneous computing power virtualization scheduling. A load-adapted heterogeneous computing power performance model and a multi-dimensional label index system are constructed, the standardization, unification and quantification of heterogeneous virtual computing power are realized based on a virtualization pooling computing power base, the differences between underlying hardware architectures are shielded, and mixed deployment scenes of domestic and overseas multi-brand acceleration chips are adapted. Through a hierarchical elastic mixed scheduling strategy combined with a hierarchical preemption yielding mechanism, the dynamic flow of training and pushing resources is realized based on a performance-matched scheduling initial scheme, the delay and availability SLA requirements of high-priority inference businesses are met through a dedicated guarantee pool, smooth cooperation of training and pushing resources in a tide scene is realized through an elastic shared pool, and the scheduling conflict problems of traditional hard isolation or full sharing modes are solved. An incremental state saving and lossless migration mechanism greatly reduces the task loss of resource switching.

Description

Technical Field

[0001] This invention relates to the technical field of computing power resource scheduling, specifically to an integrated AI model training and push system for heterogeneous computing power virtualization scheduling. Background Technology

[0002] With the large-scale deployment of technologies such as large language models, multimodal AI, and industry intelligent agents, the demand for AI computing power is growing exponentially. Computing infrastructure has fully entered the era of heterogeneous computing, with the coexistence of multiple architecture AI acceleration chips such as GPUs, NPUs, DCUs, and TPUs becoming the mainstream deployment form for intelligent computing centers and enterprise computing clusters. Simultaneously, AI business operation models are gradually shifting from training-pull separation to integrated collaboration. Enterprises expect to maximize the return on investment in computing power and reduce the construction and maintenance costs of multiple clusters by simultaneously supporting model training, fine-tuning, and online inference using a single computing cluster. However, existing heterogeneous computing power scheduling technologies have two core shortcomings in the integrated implementation of training and push: First, heterogeneous virtual computing power lacks a unified efficiency quantification system. Traditional scheduling often uses linear partitioning based on the nominal computing power and memory capacity of physical cards, ignoring the non-linear performance degradation caused by shared resource contention in virtualization scenarios. Furthermore, different chip architectures have varying performance characteristics for different types of... The computing power conversion efficiency of the AI ​​workload varies significantly, and there are frequent cases of computing power mismatch where resource parameters meet the standards but the actual performance of the task does not meet the standards, resulting in low overall utilization efficiency of heterogeneous resources. Secondly, the scheduling mechanism of the training and push hybrid workload is rigid, and the extreme modes of physical pool hard isolation or full sharing without isolation are generally adopted. The hard isolation mode cannot adapt to the business tidal characteristics, resulting in a large amount of computing power being idle. The full sharing mode is prone to long-cycle training tasks crowding out resources, causing the inference service SLA to exceed the standard. Moreover, the existing resource preemption mechanism is prone to task interruption and business jitter, and cannot achieve smooth coordination and efficient flow of training and push resources.

[0003] Chinese invention patent application CN121918935A, published on April 24, 2026, discloses an AI computing power enhancement method and system based on heterogeneous resource pooling and dynamic scheduling. This method divides physical GPU resources into multiple virtual computing units; integrates the host memory and device memory of physical GPU nodes into a unified elastic memory space pool; when an AI computing task is received, it matches the AI ​​computing task with a corresponding number and specifications of virtual computing units based on the task requirements; and based on the memory requirements of the AI ​​computing task within the virtual computing units... From the elastic memory space pool, a preset proportion of elastic memory is matched for each matched virtual computing power unit; a preset dynamic scheduler monitors the running virtual computing power units in real time, and reclaims or allocates elastic memory based on a preset control strategy, as well as migrates the AI ​​computing tasks of the virtual computing power units; this application has the effect of achieving dynamic coordination after decoupling computing power and video memory resources, thereby improving the energy efficiency of AI computing power; however, the above technical solutions cannot achieve accurate matching of heterogeneous computing power resources and efficient coordination of mixed loads in the AI ​​model training and inference process, thus reducing the efficiency and quality of the AI ​​model training and inference process. Summary of the Invention

[0004] (a) Technical problems to be solved To address the lack of a unified efficiency quantification system for heterogeneous virtual computing power, traditional scheduling often relies on linear partitioning based on the nominal computing power and memory capacity of physical cards. This ignores the non-linear performance degradation caused by contention for shared resources in virtualization scenarios. Furthermore, different chip architectures exhibit significant differences in computing power conversion efficiency for different types of AI workloads, frequently resulting in computing power mismatches where resource parameters meet standards but actual task performance falls short, leading to overall low utilization efficiency of heterogeneous resources. The scheduling mechanisms for training-push hybrid workloads are rigid, commonly employing extreme modes such as hard isolation of physical pools or full sharing without isolation. Hard isolation cannot adapt to the tidal characteristics of business operations, resulting in a large amount of idle computing power, while full sharing is prone to long training cycles. Training tasks consume resources, causing inference service SLAs to exceed limits. Furthermore, existing resource preemption mechanisms are prone to task interruptions and service jitter, failing to achieve smooth coordination and efficient flow of training and inference resources. To address this, a load-adaptive heterogeneous computing power efficiency model is constructed, achieving unified quantitative matching of computing power across multiple architectures and solving the computing power mismatch problem of traditional static scheduling. A hierarchical elastic hybrid scheduling strategy is adopted, balancing the stability of inference service SLAs with the coordination of training and inference resources in tidal scenarios, reducing resource switching losses. Combined with a closed-loop self-calibration mechanism, the system achieves adaptive evolution, significantly improving the overall resource utilization efficiency of heterogeneous clusters and supporting the goal of integrated, low-cost operation of training and inference services.

[0005] (II) Technical Solution This invention is achieved through the following technical solution: an integrated AI model training and push system with heterogeneous computing power virtualization scheduling, the system comprising: The heterogeneous computing power virtualization pooling module connects to heterogeneous acceleration hardware with multiple architectures. Through hardware-assisted virtualization and software abstraction layer, it performs fine-grained partitioning of computing power, unified abstraction of resources, and real-time status acquisition, and outputs a heterogeneous virtual computing power resource pool and a corresponding real-time running status dataset. The heterogeneous computing power performance modeling and tagging module, based on the heterogeneous virtual computing power resource pool and real-time running status dataset, and combined with multi-type AI load benchmark test data, constructs a load adaptation performance model of heterogeneous virtual computing power, performs multi-dimensional performance tagging on all virtual computing power units, and outputs a unified performance tag library and a three-level computing power matching index. The task parsing and scheduling decision module receives training and inference task requests, parses the task type, computing power requirements, priority and SLA level core parameters, performs joint retrieval based on the unified performance tag library and the three-level computing power matching index, filters candidate virtual computing power units that meet the requirements, and generates a hierarchical elastic scheduling initial scheme. The training and push task collaborative execution module, based on the aforementioned hierarchical elastic scheduling initial scheme, adopts a hierarchical elastic hybrid scheduling strategy to complete the resource allocation and dynamic flow of training and push tasks, connects with the deep learning training framework and inference engine to execute task deployment, state keep-alive and elastic scaling, and synchronously collects the running feedback performance data of all tasks. The closed-loop self-optimization module calculates the deviation between the actual performance and the expected value of the load adaptation performance model based on the task operation feedback performance data. If the deviation exceeds the preset fault tolerance threshold continuously, it triggers online incremental iteration and scheduling parameter calibration of the load adaptation performance model. If the deviation does not exceed the fault tolerance threshold, it maintains the current scheduling strategy and model parameters.

[0006] Preferably, the process of interfacing with multi-architecture heterogeneous acceleration hardware, and performing fine-grained allocation of computing power, unified abstraction of resources, and real-time status acquisition through hardware-assisted virtualization and software abstraction layers, to output a heterogeneous virtual computing power resource pool and corresponding real-time running status dataset, includes the following steps: S11 connects to AI acceleration chips with various architectures such as GPU, NPU, DCU, and TPU. Through SR-IOV hardware virtualization combined with computing power quota management technology, the physical acceleration card is divided into fine-grained virtual computing power units with three independently configurable parameters: computing power, video memory, and memory access bandwidth. S12. By using a unified resource abstraction interface to shield the differences between the underlying hardware drivers and programming architecture, standardized virtual computing power resource objects are output to the upper layer, enabling the upper layer business to make seamless calls to the underlying hardware. S13. The computing power utilization, video memory usage, memory access bandwidth, and communication bandwidth of each virtual computing unit are collected in real time using millisecond-level acquisition probes to generate the real-time operating status dataset. S14. Integrate all virtual computing power resource objects and corresponding real-time running status data to build an architecture-independent heterogeneous virtual computing power resource pool.

[0007] Preferably, the construction of the load adaptation performance model for heterogeneous virtual computing power specifically includes the following steps: S21. For the three core AI workloads of Transformer-type large models, CV computer vision models, and multimodal models, benchmark performance tests of training and inference scenarios are performed in the heterogeneous virtual computing power resource pool to obtain actual throughput, latency and memory usage data of virtual computing power units with different architectures and specifications, and to construct a benchmark performance dataset. S22. Combining the benchmark performance dataset with the real-time operating status data, a load adaptation performance model is constructed through a multivariate regression fitting algorithm, and the standardized performance values ​​of each virtual computing unit under different loads are output to achieve cross-architecture unified quantification of heterogeneous computing power.

[0008] Preferably, the step of performing multi-dimensional performance tagging on all virtual computing power units and outputting a unified performance tag library and a three-level computing power matching index specifically includes the following steps: S23. Label each virtual computing unit with a performance tag according to four dimensions: performance level, load adaptation type, memory specifications, and communication capability, and construct the unified performance tag library. S24. Generate a three-level computing power matching index of "load type-performance level-memory specification" based on performance tags to support fast and accurate matching of task requirements to computing power resources.

[0009] Preferably, the specific steps for receiving training and inference task requests, parsing the task type, computing power requirements, priority, and SLA level core parameters, performing joint retrieval based on the unified performance tag library and the three-level computing power matching index, filtering candidate virtual computing power units that meet the requirements, and generating a hierarchical elastic scheduling initial scheme include the following steps: S31. Receive training and inference task requests submitted by users, and obtain six core parameters in a structured parsing manner: task type, load model type, computing power performance requirements, memory requirements, task priority, and SLA level. S32. Call the three-level computing power matching index and unified performance tag library, and perform a joint search according to the rules of load adaptation priority, performance compliance, and memory satisfaction to filter out all candidate virtual computing power units that meet the requirements; S33. Combine task priority and SLA level to match the corresponding scheduling resource pool, and generate the current optimal hierarchical elastic scheduling initial scheme based on the distribution of candidate computing power units.

[0010] Preferably, the step of using a hierarchical elastic hybrid scheduling strategy to complete the resource allocation for training and push tasks specifically includes the following steps: S41. The heterogeneous virtual computing power resource pool is divided into three logically hierarchical pools: inference guarantee pool, elastic sharing pool, and training batch pool. The inference guarantee pool reserves fixed resources to carry high-priority inference services, the training batch pool carries long-cycle offline training tasks, and the elastic sharing pool is a dynamic scheduling buffer pool for training and inference sharing. S42. For high-priority online inference tasks, virtual computing power units with matching performance are allocated from the inference guarantee pool first to ensure inference latency and availability SLA; for long-cycle offline training tasks, aggregated computing power resources are allocated from the training batch pool first to ensure training throughput and parallel efficiency.

[0011] Preferably, the step of using a hierarchical elastic hybrid scheduling strategy to dynamically transfer training and push resources specifically includes the following steps: S43. When the inference service load increases and the inference guarantee pool resources are insufficient, the training tasks in the elastic shared pool are suspended in order of priority from low to high, the corresponding virtual computing power units are reclaimed to expand the inference resources, and the incremental state of the training tasks is saved synchronously. When the inference service load decreases and the elastic shared pool resources are idle, the idle computing power is automatically allocated to the training tasks in the waiting queue, and the execution of the training tasks is resumed. S44. Real-time monitoring of cluster inference SLA compliance rate, request concurrency, overall resource utilization and training queue length, dynamically adjusting the resource ratio of the three logical tier pools to balance inference service stability and cluster resource utilization.

[0012] Preferably, the process of connecting the deep learning training framework and inference engine to perform task deployment, state persistence, and elastic scaling, while simultaneously collecting runtime feedback performance data, specifically includes the following steps: S45, integrates with mainstream deep learning training frameworks such as PyTorch and MindSpore, as well as vLLM and Triton inference engines, to complete the deployment of the scheduling scheme. S46. For training tasks, incremental checkpoint saving and breakpoint resume training are supported. When resources are preempted and expanded, only the incremental training state is saved, reducing the computing power loss when tasks are interrupted. S47. For inference tasks, support model preloading and traffic canary migration to achieve service seamlessness when scaling up or down computing power and migrating nodes, ensuring the continuity of inference services. S48. During the task execution, continuously collect training throughput, inference latency, and inference QPS performance indicators to generate the task execution feedback performance data.

[0013] Preferably, based on the task operation feedback performance data, the deviation between the actual performance and the expected value of the load adaptation performance model is calculated; if the deviation continuously exceeds a preset fault tolerance threshold, online incremental iteration and scheduling parameter calibration of the load adaptation performance model are triggered; if the fault tolerance threshold is not exceeded, the current scheduling strategy and model parameters are maintained. The specific steps include the following: S51. Extract the actual performance indicators from the task operation feedback performance data, calculate the relative difference between the actual performance and the expected performance value of the load adaptation efficiency model, and obtain the matching deviation. S52. Preset fault tolerance threshold and continuous judgment period. If the matching deviation is greater than the fault tolerance threshold for three consecutive scheduling periods, it is determined that the hardware performance status or load characteristics have changed in a trend and the subsequent calibration process is triggered. If the continuous over-threshold condition is not met, it is determined to be a normal fluctuation and the current scheduling strategy and model parameters are maintained. S53. After triggering the calibration process, extract the multi-dimensional performance features and actual performance data within the corresponding period as incremental learning samples, and use the gradient descent method to perform online incremental iteration on the regression layer weights of the load adaptation performance model. S54. Before the model iteration is completed, the scheduling parameter calibration mechanism is enabled to force the computing power allocation efficiency threshold to be multiplied by the safety redundancy coefficient to ensure the stability of task operation; after the model iteration is completed and the matching deviation falls back to within the fault tolerance threshold, the safety redundancy coefficient is automatically removed.

[0014] (III) Beneficial Effects This invention provides an integrated AI model training and push system based on heterogeneous computing power virtualization scheduling. It offers the following advantages: I. By constructing a load-adaptive heterogeneous computing power performance model and a multi-dimensional tag index system, the standardization and unified quantification of heterogeneous virtual computing power are realized based on the virtualization pooled computing power base. This replaces the traditional static partitioning method based on hardware nominal parameters, effectively solving the problem of heterogeneous computing power mismatch, greatly improving the matching accuracy between task requirements and computing power resources, fully releasing the actual performance of computing power of different architectures, while shielding the differences in underlying hardware architecture, and adapting to the mixed deployment scenarios of domestic and overseas multi-brand acceleration chips.

[0015] Second, by combining a hierarchical elastic hybrid scheduling strategy with a hierarchical preemption and yielding mechanism, and based on the performance-matching initial scheduling scheme, the dynamic flow of training and push resources is realized. This not only meets the latency and availability SLA requirements of high-priority inference services through a dedicated guarantee pool, but also achieves smooth coordination of training and push resources in tidal scenarios through an elastic shared pool, thus solving the scheduling conflict problem of traditional hard isolation or full sharing modes. The incremental state preservation and lossless migration mechanism significantly reduce the task loss of resource switching and effectively improve the overall resource utilization efficiency of the cluster.

[0016] Third, through a complete logical link of computing power pooling, performance modeling, intelligent scheduling, task execution, and closed-loop optimization, data is interconnected and logic is linked in each link. By judging the threshold of the deviation, the performance model is triggered to iterate online and the parameters are calibrated for safety, so as to realize the system's adaptive closed-loop self-evolution. The whole solution supports the integrated collaborative operation of training and inference business on the same heterogeneous computing power cluster, effectively reducing the construction cost and operation and maintenance complexity of enterprise AI computing power, and adapting to the training and inference integrated computing power construction needs of intelligent computing centers, industry enterprises and other scenarios. Attached Figure Description

[0017] Figure 1 The flowchart of the operation of an AI model training and push integrated system with heterogeneous computing power virtualization scheduling provided by the present invention; Figure 2 This is a schematic diagram of a module of an AI model training and push integrated system with heterogeneous computing power virtualization scheduling provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] An example of an AI model training and push integrated system with heterogeneous computing power virtualization scheduling is as follows: Please see Figures 1-2 A heterogeneous computing power virtualization scheduling AI model training and push integrated system, the system comprising: The heterogeneous computing power virtualization pooling module connects to heterogeneous acceleration hardware with multiple architectures. Through hardware-assisted virtualization and software abstraction layer, it performs fine-grained partitioning of computing power, unified abstraction of resources, and real-time status acquisition, and outputs a heterogeneous virtual computing power resource pool and a corresponding real-time running status dataset. The heterogeneous computing power performance modeling and tagging module, based on the heterogeneous virtual computing power resource pool and real-time running status dataset, and combined with multi-type AI load benchmark test data, constructs a load adaptation performance model of heterogeneous virtual computing power, performs multi-dimensional performance tagging on all virtual computing power units, and outputs a unified performance tag library and a three-level computing power matching index. The task parsing and scheduling decision module receives training and inference task requests, parses the task type, computing power requirements, priority and SLA level core parameters, performs joint retrieval based on the unified performance tag library and the three-level computing power matching index, filters candidate virtual computing power units that meet the requirements, and generates a hierarchical elastic scheduling initial scheme. The training and push task collaborative execution module, based on the aforementioned hierarchical elastic scheduling initial scheme, adopts a hierarchical elastic hybrid scheduling strategy to complete the resource allocation and dynamic flow of training and push tasks, connects with the deep learning training framework and inference engine to execute task deployment, state keep-alive and elastic scaling, and synchronously collects the running feedback performance data of all tasks. The closed-loop self-optimization module calculates the deviation between the actual performance and the expected value of the load adaptation performance model based on the task operation feedback performance data. If the deviation exceeds the preset fault tolerance threshold continuously, it triggers online incremental iteration and scheduling parameter calibration of the load adaptation performance model. If the deviation does not exceed the fault tolerance threshold, it maintains the current scheduling strategy and model parameters.

[0020] The specific steps involved in connecting to multi-architecture heterogeneous acceleration hardware, performing fine-grained allocation of computing power, unified abstraction of resources, and real-time status acquisition through hardware-assisted virtualization and software abstraction layers, and outputting heterogeneous virtual computing power resource pools and corresponding real-time running status datasets include the following: S11 connects to AI acceleration chips with various architectures such as GPU, NPU, DCU, and TPU. Through SR-IOV hardware virtualization combined with computing power quota management technology, the physical acceleration card is divided into fine-grained virtual computing power units with three independently configurable parameters: computing power, video memory, and memory access bandwidth. Specifically, in this embodiment S11, the system is deployed at the intelligent computing center management node, and connects to three types of heterogeneous acceleration hardware in the cluster: NVIDIA A100 GPU, Ascend 910B NPU, and Hygon DCU. By adapting to the official virtualization drivers of each manufacturer and combining SR-IOV hardware virtualization technology and computing power quota management components, fine-grained partitioning of physical acceleration cards is achieved: a single physical acceleration card can be partitioned into virtual computing power units with a minimum of 1GB of video memory, 5% computing power quota, and 10% memory access bandwidth quota. Moreover, the three dimensions of computing power, video memory, and memory access bandwidth can be configured independently, covering the full range of resource requirements from lightweight inference to large-scale distributed training of models.

[0021] S12. By using a unified resource abstraction interface to shield the differences between the underlying hardware drivers and programming architecture, standardized virtual computing power resource objects are output to the upper layer, enabling the upper layer business to make seamless calls to the underlying hardware. Specifically, in this embodiment S12, the differences in driver and programming interfaces of different architectures such as CUDA, CANN, and ROCm are shielded by a unified resource abstraction interface, and standardized virtual computing power resource objects are output to the upper-layer scheduling system. The object attributes include four standardized fields: computing power specifications, video memory capacity, bandwidth quota, and running status. Upper-layer services do not need to be aware of the underlying hardware brand and architecture to complete the application and call of computing power resources.

[0022] S13. The computing power utilization, video memory usage, memory access bandwidth, and communication bandwidth of each virtual computing unit are collected in real time using millisecond-level acquisition probes to generate the real-time operating status dataset. Specifically, in S13 of this embodiment, a 10Hz frequency millisecond-level acquisition probe is deployed to collect four core operating indicators of each virtual computing unit in real time: computing power utilization, video memory occupancy, memory access bandwidth occupancy, and cross-node communication bandwidth. The resource status database is updated once per second to provide real-time data support for subsequent performance modeling and scheduling decisions.

[0023] S14. Integrate all virtual computing power resource objects and corresponding real-time running status data to build an architecture-independent heterogeneous virtual computing power resource pool.

[0024] The specific steps for constructing the load adaptation performance model of heterogeneous virtual computing power include the following: S21. For the three core AI workloads of Transformer-type large models, CV computer vision models, and multimodal models, benchmark performance tests of training and inference scenarios are performed in the heterogeneous virtual computing power resource pool to obtain actual throughput, latency and memory usage data of virtual computing power units with different architectures and specifications, and to construct a benchmark performance dataset. Specifically, in S21 of this embodiment, the Llama3-7B large model, ResNet50CV model, and StableDiffusion multimodal model are selected as benchmark test loads. The training and inference scenarios are stress tested on the full-specification virtual units of the heterogeneous virtual computing power resource pool, and the core indicators such as training throughput (samples / second), inference latency P99, inference QPS, peak memory usage, and actual computing power utilization of a single unit are recorded to form a benchmark performance dataset covering the full specifications and full load types.

[0025] S22. Combining the benchmark performance dataset with the real-time operating status data, a load adaptation performance model is constructed through a multivariate regression fitting algorithm, and the standardized performance values ​​of each virtual computing unit under different loads are output to achieve cross-architecture unified quantification of heterogeneous computing power.

[0026] Specifically, in this embodiment S22, based on the benchmark performance dataset and combined with the real-time collected virtual computing unit operation indicators, a load adaptation performance model is constructed through a multivariate regression fitting algorithm: for each type of training and push load, the performance of virtual computing units of different architectures is mapped to a unified standardized performance score. In this embodiment, the training performance of A100 single-card FP16 is set as the benchmark value of 100, and the corresponding output is the standardized performance value of Ascend 910B and Hygon DCU under the same type of load, realizing the cross-architecture unified quantification of heterogeneous computing power and solving the problem that computing power of different architectures is not comparable or interchangeable.

[0027] The process of performing multi-dimensional performance tagging on all virtual computing power units and outputting a unified performance tag library and a three-level computing power matching index specifically includes the following steps: S23. Label each virtual computing unit with a performance tag according to four dimensions: performance level, load adaptation type, memory specifications, and communication capability, and construct the unified performance tag library. Specifically, in this embodiment S23, each virtual computing power unit is labeled with four-dimensional performance tags: first, a performance level tag, which is divided into high, medium and low levels according to standardized performance values; second, a load adaptation tag, which indicates the optimal load type (large model training, large model inference, CV training, CV inference, etc.) for the unit; third, a memory specification tag, which indicates the actual available memory capacity; and fourth, a communication capability tag, which indicates the cross-card interconnection bandwidth level of the unit, ultimately constructing a unified performance tag library.

[0028] S24. Generate a three-level computing power matching index of "load type-performance level-memory specification" based on performance tags to support fast and accurate matching of task requirements to computing power resources.

[0029] Specifically, in this embodiment S24, a three-level computing power matching index of "load type-performance level-memory specification" is generated based on four types of tags. The three-level index can realize the second-level positioning of task requirements to computing power resources, and the retrieval response latency is controlled within 10ms, which meets the performance requirements of high-concurrency task scheduling.

[0030] The specific steps for receiving training and inference task requests, parsing task type, computing power requirements, priority, and SLA level core parameters, performing joint retrieval based on the unified performance tag library and the three-level computing power matching index, filtering candidate virtual computing power units that meet the requirements, and generating a hierarchical elastic scheduling initial scheme include the following: S31. Receive training and inference task requests submitted by users, and obtain six core parameters in a structured parsing manner: task type, load model type, computing power performance requirements, memory requirements, task priority, and SLA level. Specifically, in S31 of this embodiment, the system receives training and inference task requests submitted by users through the console or API, performs structured parsing on the request message, and extracts six core parameters: task type, load model type, computing power performance requirements, memory requirements, task priority, and SLA level. The task priority is divided into three levels: P0 (highest), P1 (medium), and P2 (lowest).

[0031] S32. Call the three-level computing power matching index and unified performance tag library, and perform a joint search according to the rules of load adaptation priority, performance compliance, and memory satisfaction to filter out all candidate virtual computing power units that meet the requirements; Specifically, in S32 of this embodiment, the three-level computing power matching index and unified performance tag library are called, and joint retrieval is performed according to the three-level rules of "load adaptation priority, performance standard, and video memory satisfaction": first, the optimal load adaptation type is matched, then the units whose performance values ​​meet the task requirements are filtered, and finally the video memory capacity is checked to ensure that all candidate virtual computing power units that meet the requirements are selected.

[0032] S33. Combine task priority and SLA level to match the corresponding scheduling resource pool, and generate the current optimal hierarchical elastic scheduling initial scheme based on the distribution of candidate computing power units.

[0033] Specifically, in this embodiment S33, P0-level high-priority inference tasks are matched with the inference guarantee pool, P1-level medium-priority training tasks are matched with the elastic sharing pool, and P2-level low-priority offline training tasks are matched with the training batch pool. Combining the topology distribution and communication distance of the candidate computing units, the current optimal hierarchical elastic scheduling initial scheme is generated.

[0034] The specific steps for allocating resources for training and push tasks using a hierarchical elastic hybrid scheduling strategy include the following: S41. The heterogeneous virtual computing power resource pool is divided into three logically hierarchical pools: inference guarantee pool, elastic sharing pool, and training batch pool. The inference guarantee pool reserves fixed resources to carry high-priority inference services, the training batch pool carries long-cycle offline training tasks, and the elastic sharing pool is a dynamic scheduling buffer pool for training and inference sharing. S42. For high-priority online inference tasks, virtual computing power units with matching performance are allocated from the inference guarantee pool first to ensure inference latency and availability SLA; for long-cycle offline training tasks, aggregated computing power resources are allocated from the training batch pool first to ensure training throughput and parallel efficiency.

[0035] Specifically, in S41 and S42 of this embodiment, the total cluster resources in the initial state are divided into three logical pools in a 3:4:3 ratio: the inference guarantee pool, the elastic sharing pool, and the training batch pool. The inference guarantee pool is fixed to carry P0-level online inference services, with an inference latency SLA threshold of 200ms and an availability of 99.9%. The training batch pool is fixed to carry long-cycle offline training and large model fine-tuning tasks, prioritizing the continuous computing power supply of multi-card aggregation. The elastic sharing pool serves as a dynamic buffer pool, simultaneously undertaking low-priority inference and medium-priority training tasks.

[0036] The specific steps for using a hierarchical elastic hybrid scheduling strategy to dynamically transfer training and push resources include the following: S43. When the inference service load increases and the inference guarantee pool resources are insufficient, the training tasks in the elastic shared pool are suspended in order of priority from low to high, the corresponding virtual computing power units are reclaimed to expand the inference resources, and the incremental state of the training tasks is saved synchronously. When the inference service load decreases and the elastic shared pool resources are idle, the idle computing power is automatically allocated to the training tasks in the waiting queue, and the execution of the training tasks is resumed. Specifically, in S43 of this embodiment, when the inference request concurrency exceeds the threshold for one minute and the inference guarantee pool utilization reaches over 90%, the system pauses the training tasks in the elastic shared pool according to the priority order from P2 to P1, and reclaims virtual computing power units to expand the inference resource pool; the training tasks synchronously perform incremental checkpoint saving, saving only the optimizer state, step number, and other incremental tensor data of the current training step, without repeatedly saving the full model weights, which improves the speed by 80% compared to full checkpoint saving, and the interruption recovery loss is controlled within 5 minutes. When the inference concurrency decreases for 5 minutes and the elastic shared pool idle rate exceeds 60%, the system automatically allocates idle computing power to the training waiting queue according to the training task priority from high to low, resumes the execution of training tasks, and maximizes the utilization of idle computing power.

[0037] S44. Real-time monitoring of cluster inference SLA compliance rate, request concurrency, overall resource utilization and training queue length, dynamically adjusting the resource ratio of the three logical tier pools to balance inference service stability and cluster resource utilization.

[0038] Specifically, in S44 of this embodiment, the system adjusts the pooling ratio in real time through a dynamic threshold scheduling algorithm: when the inference SLA compliance rate is lower than 99.9% for 10 consecutive minutes, the proportion of the inference guarantee pool is automatically increased by 5% each time, up to a maximum of 60% of the total cluster resources; when the overall average utilization of the cluster is lower than 30% for 10 consecutive minutes and the training waiting queue length is greater than 5, the proportion of the inference guarantee pool is automatically decreased by 5% each time, down to a minimum of 20%; at the same time, a 30-minute adjustment cooldown time is set to avoid frequent fluctuations in the pooling ratio affecting business stability and to achieve adaptive optimization of the scheduling strategy.

[0039] The specific steps involved in connecting the deep learning training framework and inference engine to perform task deployment, state persistence, elastic scaling, and synchronous collection of runtime feedback performance data include the following: S45, integrates with mainstream deep learning training frameworks such as PyTorch and MindSpore, as well as vLLM and Triton inference engines, to complete the deployment of the scheduling scheme. Specifically, in this embodiment S45, the system connects to the deep learning training framework through a standard custom resource interface and to the inference engine through a service registration and discovery mechanism, automatically converting scheduling decisions into task deployment instructions, thus completing the binding and deployment of computing resources and business loads without manual intervention.

[0040] S46. For training tasks, incremental checkpoint saving and breakpoint resume training are supported. When resources are preempted and expanded, only the incremental training state is saved, reducing the computing power loss when tasks are interrupted. Specifically, in S46 of this embodiment, when the training task starts, an incremental checkpoint hook component is automatically injected to save the incremental training state at a fixed period. When the task is resumed after being preempted, the component automatically merges the basic weights and the incremental state to realize breakpoint resume training, avoid the training progress from the beginning, and greatly reduce the waste of computing power caused by resource switching.

[0041] S47. For inference tasks, support model preloading and traffic canary migration to achieve service seamlessness when scaling up or down computing power and migrating nodes, ensuring the continuity of inference services. Specifically, in S47 of this embodiment, when expanding inference instances or migrating execution nodes, the system completes model loading and inference preheating in the target virtual computing unit in advance, and then gradually switches traffic in proportions of 10%, 30%, 50%, and 100%. The latency index is observed for 1 minute in each stage. If there are no abnormalities, the next stage is advanced. There is no business interruption or latency jitter throughout the process, ensuring the continuity of online inference services.

[0042] S48. During the task execution, continuously collect training throughput, inference latency, and inference QPS performance indicators to generate the task execution feedback performance data.

[0043] Specifically, in S48 of this embodiment, the system collects the core performance indicators of the entire task every 30 seconds, including the single-step throughput and memory usage of the training task, and the P99 latency, QPS, and error rate of the inference task. These are uniformly stored in the time series database to generate task operation feedback performance data, which serves as the input for the closed-loop self-optimization process.

[0044] Based on the task execution feedback performance data, the deviation between the actual performance and the expected value of the load adaptation performance model is calculated. If the deviation continuously exceeds a preset fault tolerance threshold, online incremental iteration and scheduling parameter calibration of the load adaptation performance model are triggered. If the deviation does not exceed the fault tolerance threshold, the current scheduling strategy and model parameters are maintained. The specific steps include the following: S51. Extract the actual performance indicators from the task operation feedback performance data, calculate the relative difference between the actual performance and the expected performance value of the load adaptation efficiency model, and obtain the matching deviation. Specifically, in this embodiment S51, for the training task, the relative difference between the actual training throughput and the expected throughput is taken, and for the inference task, the relative difference between the actual inference latency and the expected latency is taken. The calculation is uniformly performed according to the formula: Matching deviation = |Actual performance value - Expected performance value| / Expected performance value × 100%, and the maximum deviation between the two types of tasks is taken as the final judgment basis.

[0045] S52. Preset fault tolerance threshold and continuous judgment period. If the matching deviation is greater than the fault tolerance threshold for three consecutive scheduling periods, it is determined that the hardware performance status or load characteristics have changed in a trend and the subsequent calibration process is triggered. If the continuous over-threshold condition is not met, it is determined to be a normal fluctuation and the current scheduling strategy and model parameters are maintained. Specifically, in S52 of this embodiment, the preset fault tolerance threshold is 15%, and the continuous judgment period is 3 scheduling cycles (each scheduling cycle is 5 minutes): if the matching deviation is greater than 15% for 15 consecutive minutes, it is determined that the hardware performance has degraded or the load characteristics have changed in a trend, triggering the calibration process; if the threshold is exceeded only once or not continuously, it is determined to be normal random fluctuation, and the current scheduling strategy and model parameters are maintained. Further, the fault tolerance threshold is calculated based on the statistical distribution of prediction errors on the historical validation set: the matching deviation data of the performance model on the offline validation set is extracted, its mean and standard deviation are calculated, and the fault tolerance threshold is set to the mean + 2 times the standard deviation to distinguish between normal prediction fluctuations and model failure caused by substantial changes in hardware performance and load characteristics.

[0046] S53. After triggering the calibration process, extract the multi-dimensional performance features and actual performance data within the corresponding period as incremental learning samples, and use the gradient descent method to perform online incremental iteration on the regression layer weights of the load adaptation performance model. Specifically, in S53 of this embodiment, after the calibration process is triggered, the system automatically packages the multi-dimensional performance feature vector within the offset period and the real performance data into incremental learning samples, starts small-batch gradient descent operation in the background, and only fine-tunes the weights of the fully connected regression layer of the performance model. The learning rate is set to 0.0001 to avoid catastrophic forgetting caused by modifying the underlying feature extraction network, and to ensure that model iteration does not affect the core prediction capability. The iteration process is executed in the background and does not interfere with normal scheduling business.

[0047] S54. Before the model iteration is completed, the scheduling parameter calibration mechanism is enabled to force the computing power allocation efficiency threshold to be multiplied by the safety redundancy coefficient to ensure the stability of task operation; after the model iteration is completed and the matching deviation falls back to within the fault tolerance threshold, the safety redundancy coefficient is automatically removed.

[0048] Specifically, in S54 of this embodiment, online fine-tuning usually takes a few minutes. During this period, the system immediately activates the safety-backup calibration mechanism, forcibly multiplying the efficiency threshold of computing power allocation by a redundancy coefficient of 1.2, that is, allocating resources according to 80% of the expected efficiency, reserving computing power redundancy to ensure that the task performance meets the standards; after the model fine-tuning is completed and the matching deviation falls back to within 15% for three consecutive scheduling cycles, the redundancy coefficient is automatically removed and efficient scheduling is restored.

[0049] For example, consider a specific scenario where an enterprise's intelligent computing center supports integrated training and promotion services: The enterprise intelligent computing center is equipped with 30 A100 GPUs, 20 Ascend 910B NPUs, and 10 Hygon DCUs. It simultaneously handles four types of services: large-scale online inference, computer vision inference, large-scale model fine-tuning, and computer vision model training. It exhibits significant tidal characteristics: during the day from 8:00 to 22:00, online inference services dominate, accounting for approximately 75% of the load; at night from 22:00 to 8:00 the next day, offline training services dominate, accounting for approximately 70% of the load.

[0050] After system deployment, S1 first completes the fine-grained segmentation and unified abstraction of all hardware to build a heterogeneous virtual computing power resource pool; then S2 completes benchmark tests of three types of core loads based on the computing power pool to build a unified performance tag library and matching index; initially, the inference guarantee pool, elastic sharing pool, and training batch pool are divided in a 3:4:3 ratio.

[0051] During the peak inference period at 10:00 AM on weekdays, the concurrency of large model inference requests surged, leading to insufficient resources in the inference guarantee pool. The scheduling module automatically triggered the elastic preemption mechanism, suspending three low-priority CV training tasks in the elastic shared pool according to priority, and correspondingly reclaiming 12 virtual computing units to expand the inference resource pool. The training tasks executed incremental checkpoints and saved, with the interruption time being only 3 minutes. After the expansion, the SLA compliance rate of the inference business remained at 99.95%, meeting the business requirements.

[0052] During the low inference period at 23:00 at night, the inference concurrency dropped significantly, and the idle rate of the elastic shared pool reached 70%. The scheduling module automatically allocated the 28 idle virtual computing units to the large model fine-tuning tasks in the waiting queue, increasing the proportion of training resources to 72% of the total cluster resources. This fully utilized the idle computing power at night to complete the training tasks, and significantly shortened the training cycle.

[0053] After running for 3 months, due to aging hardware components and driver version iterations, the system detected that the training performance matching deviation exceeded 15% for 3 consecutive cycles. It automatically triggered online iteration of the performance model and activated a 1.2 times safety redundancy coefficient as a backup. After the model was fine-tuned, the matching deviation dropped to within 8%, and the system resumed normal and efficient scheduling without any manual intervention.

[0054] Through the operation of the entire system, the company has achieved unified management of multi-architecture heterogeneous computing power and integrated operation of training and promotion services, which has significantly reduced the cost of computing power construction and maintenance.

[0055] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A heterogeneous computing power virtualization scheduling AI model training and push integrated system, characterized in that, The system includes: The heterogeneous computing power virtualization pooling module connects to heterogeneous acceleration hardware with multiple architectures. Through hardware-assisted virtualization and software abstraction layer, it performs fine-grained partitioning of computing power, unified abstraction of resources, and real-time status acquisition, and outputs a heterogeneous virtual computing power resource pool and a corresponding real-time running status dataset. The heterogeneous computing power performance modeling and tagging module, based on the heterogeneous virtual computing power resource pool and real-time running status dataset, and combined with multi-type AI load benchmark test data, constructs a load adaptation performance model of heterogeneous virtual computing power, performs multi-dimensional performance tagging on all virtual computing power units, and outputs a unified performance tag library and a three-level computing power matching index. The task parsing and scheduling decision module receives training and inference task requests, parses the task type, computing power requirements, priority and SLA level core parameters, performs joint retrieval based on the unified performance tag library and the three-level computing power matching index, filters candidate virtual computing power units that meet the requirements, and generates a hierarchical elastic scheduling initial scheme. The training and push task collaborative execution module, based on the aforementioned hierarchical elastic scheduling initial scheme, adopts a hierarchical elastic hybrid scheduling strategy to complete the resource allocation and dynamic flow of training and push tasks, connects with the deep learning training framework and inference engine to execute task deployment, state keep-alive and elastic scaling, and synchronously collects the running feedback performance data of all tasks. The closed-loop self-optimization module calculates the deviation between the actual performance and the expected value of the load adaptation performance model based on the task operation feedback performance data. If the deviation exceeds the preset fault tolerance threshold continuously, it triggers online incremental iteration and scheduling parameter calibration of the load adaptation performance model. If the deviation does not exceed the fault tolerance threshold, it maintains the current scheduling strategy and model parameters.

2. The AI ​​model training and push integrated control system for heterogeneous computing power virtualization scheduling according to claim 1, characterized in that, The specific steps involved in connecting to multi-architecture heterogeneous acceleration hardware, performing fine-grained allocation of computing power, unified abstraction of resources, and real-time status acquisition through hardware-assisted virtualization and software abstraction layers, and outputting heterogeneous virtual computing power resource pools and corresponding real-time running status datasets include the following: S11 connects to AI acceleration chips with various architectures such as GPU, NPU, DCU, and TPU. Through SR-IOV hardware virtualization combined with computing power quota management technology, the physical acceleration card is divided into fine-grained virtual computing power units with three independently configurable parameters: computing power, video memory, and memory access bandwidth. S12. By using a unified resource abstraction interface to shield the differences between the underlying hardware drivers and programming architecture, standardized virtual computing power resource objects are output to the upper layer, enabling the upper layer business to make seamless calls to the underlying hardware. S13. The computing power utilization, video memory usage, memory access bandwidth, and communication bandwidth of each virtual computing unit are collected in real time using millisecond-level acquisition probes to generate the real-time operating status dataset. S14. Integrate all virtual computing power resource objects and corresponding real-time running status data to build an architecture-independent heterogeneous virtual computing power resource pool.

3. The AI ​​model training and push integrated control system for heterogeneous computing power virtualization scheduling according to claim 1, characterized in that, The specific steps for constructing the load adaptation performance model of heterogeneous virtual computing power include the following: S21. For the three core AI workloads of Transformer-type large models, CV computer vision models, and multimodal models, benchmark performance tests of training and inference scenarios are performed in the heterogeneous virtual computing power resource pool to obtain actual throughput, latency and memory usage data of virtual computing power units with different architectures and specifications, and to construct a benchmark performance dataset. S22. Combining the benchmark performance dataset with the real-time operating status data, a load adaptation performance model is constructed through a multivariate regression fitting algorithm, and the standardized performance values ​​of each virtual computing unit under different loads are output to achieve cross-architecture unified quantification of heterogeneous computing power.

4. The AI ​​model training and push integrated control system for heterogeneous computing power virtualization scheduling according to claim 1, characterized in that, The process of performing multi-dimensional performance tagging on all virtual computing power units and outputting a unified performance tag library and a three-level computing power matching index specifically includes the following steps: S23. Label each virtual computing unit with a performance tag according to four dimensions: performance level, load adaptation type, memory specifications, and communication capability, and construct the unified performance tag library. S24. Generate a three-level computing power matching index based on performance tags: "load type - performance level - memory specifications" to support fast and accurate matching of task requirements to computing power resources.

5. The AI ​​model training and push integrated control system for heterogeneous computing power virtualization scheduling according to claim 1, characterized in that, The specific steps for receiving training and inference task requests, parsing task type, computing power requirements, priority, and SLA level core parameters, performing joint retrieval based on the unified performance tag library and the three-level computing power matching index, filtering candidate virtual computing power units that meet the requirements, and generating a hierarchical elastic scheduling initial scheme include the following: S31. Receive training and inference task requests submitted by users, and obtain six core parameters in a structured parsing manner: task type, load model type, computing power performance requirements, memory requirements, task priority, and SLA level. S32. Call the three-level computing power matching index and unified performance tag library, and perform a joint search according to the rules of load adaptation priority, performance compliance, and memory satisfaction to filter out all candidate virtual computing power units that meet the requirements; S33. Combine task priority with SLA level to match the corresponding scheduling resource pool, and generate the current optimal hierarchical elastic scheduling initial scheme based on the distribution of candidate computing power units.

6. The AI ​​model training and push integrated control system for heterogeneous computing power virtualization scheduling according to claim 1, characterized in that, The specific steps for allocating resources for training and push tasks using a hierarchical elastic hybrid scheduling strategy include the following: S41. The heterogeneous virtual computing power resource pool is divided into three logically hierarchical pools: inference guarantee pool, elastic sharing pool, and training batch pool. The inference guarantee pool reserves fixed resources to carry high-priority inference services, the training batch pool carries long-cycle offline training tasks, and the elastic sharing pool is a dynamic scheduling buffer pool for training and inference sharing. S42. For high-priority online inference tasks, virtual computing power units with matching performance are allocated from the inference guarantee pool first to ensure inference latency and availability SLA; for long-cycle offline training tasks, aggregated computing power resources are allocated from the training batch pool first to ensure training throughput and parallel efficiency.

7. The AI ​​model training and push integrated control system for heterogeneous computing power virtualization scheduling according to claim 1, characterized in that, The specific steps for using a hierarchical elastic hybrid scheduling strategy to dynamically transfer training and push resources include the following: S43. When the inference service load increases and the inference guarantee pool resources are insufficient, the training tasks in the elastic shared pool are suspended in order of priority from low to high, the corresponding virtual computing power units are reclaimed to expand the inference resources, and the incremental state of the training tasks is saved synchronously. When the inference service load decreases and the elastic shared pool resources are idle, the idle computing power is automatically allocated to the training tasks in the waiting queue, and the execution of the training tasks is resumed. S44. Real-time monitoring of cluster inference SLA compliance rate, request concurrency, overall resource utilization and training queue length, dynamically adjusting the resource ratio of the three logical tier pools to balance inference service stability and cluster resource utilization.

8. The AI ​​model training and push integrated control system for heterogeneous computing power virtualization scheduling according to claim 1, characterized in that, The specific steps involved in connecting the deep learning training framework and inference engine to perform task deployment, state persistence, elastic scaling, and synchronous collection of runtime feedback performance data include the following: S45, integrates with mainstream deep learning training frameworks such as PyTorch and MindSpore, as well as vLLM and Triton inference engines, to complete the deployment of the scheduling scheme. S46. For training tasks, incremental checkpoint saving and breakpoint resume training are supported. When resources are preempted and expanded, only the incremental training state is saved, reducing the computing power loss when tasks are interrupted. S47. For inference tasks, support model preloading and traffic canary migration to achieve service seamlessness when scaling up or down computing power and migrating nodes, ensuring the continuity of inference services. S48. During the task execution, continuously collect training throughput, inference latency, and inference QPS performance indicators to generate the task execution feedback performance data.

9. The AI ​​model training and push integrated control system for heterogeneous computing power virtualization scheduling according to claim 1, characterized in that, Based on the task operation feedback performance data, calculate the degree of deviation between the actual performance and the expected value of the load adaptation efficiency model. If the matching deviation continuously exceeds the preset fault tolerance threshold, online incremental iteration and scheduling parameter calibration of the load adaptation performance model are triggered; if the fault tolerance threshold is not exceeded, the current scheduling strategy and model parameters are maintained. The specific steps include the following: S51. Extract the actual performance indicators from the task operation feedback performance data, calculate the relative difference between the actual performance and the expected performance value of the load adaptation efficiency model, and obtain the matching deviation. S52. Preset fault tolerance threshold and continuous judgment period. If the matching deviation is greater than the fault tolerance threshold for three consecutive scheduling periods, it is determined that the hardware performance status or load characteristics have changed in a trend and the subsequent calibration process is triggered. If the continuous over-threshold condition is not met, it is determined to be a normal fluctuation and the current scheduling strategy and model parameters are maintained. S53. After triggering the calibration process, extract the multi-dimensional performance features and actual performance data within the corresponding period as incremental learning samples, and use the gradient descent method to perform online incremental iteration on the regression layer weights of the load adaptation performance model. S54. Before the model iteration is completed, enable the scheduling parameter calibration mechanism to force the computing power allocation efficiency threshold to be multiplied by the safety redundancy coefficient to ensure the stability of task operation. Once the model iteration is complete and the matching deviation falls back to within the fault tolerance threshold, the safety redundancy coefficient is automatically removed.

Images