Quantization-aware multi-model online inference scheduling method for heterogeneous GPU cluster

Abstract

This invention discloses a quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters, relating to the field of distributed machine learning technology. The method includes: acquiring the remaining selectable DNN models, the DNN models already deployed on each GPU server, and GPU servers that do not meet computational resource and memory constraints; the scheduler selects a target combination and determines whether the system throughput is maximized when the selected DNN model is deployed on the target GPU server at the corresponding quantization level and performs parallel inference; if not, a new combination is selected; the DNN models in the target combination are deployed to the target GPU servers according to their corresponding quantization levels, and the status information is updated; after each DNN model deployment, it is determined whether the computational and memory resources of the corresponding GPU server still meet the deployment and inference requirements; the above process is repeated until the algorithm converges. This invention effectively improves GPU resource utilization and maximizes throughput.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning technology, and in particular to a quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters. Background Technology

[0002] With the rapid development of deep learning technology, the model representation capabilities of Deep Neural Networks (DNNs) have been significantly improved, further promoting the widespread application and rapid development of the artificial intelligence industry. Currently, DNN models have achieved breakthroughs in fields such as natural language understanding, image recognition, and voice interaction. For example, the successful application of large-scale artificial intelligence models like ChatGPT fully demonstrates the advantages of deep learning technology in handling complex tasks. Existing deep learning systems typically adopt an "offline training + online inference" operating mode. This means that first, based on a pre-processed large-scale dataset, the DNN model is pre-trained on a server cluster; then, according to different business task requests, the corresponding DNN model is dynamically loaded, and decision results are generated through real-time inference computation. Therefore, the real-time inference efficiency of DNN models has become one of the key factors affecting the performance of deep learning applications and user experience. To reduce the computational complexity and memory overhead during the DNN model inference process, model quantization technology is widely used in the deep learning inference optimization process. Model quantization typically reduces the precision of model parameters and intermediate activation values ​​by decreasing the data representation accuracy, such as compressing floating-point parameters into low-bit integer forms. This reduces model storage size and computational resource consumption, thereby improving model inference speed and lowering deployment costs. Different quantization strategies have varying impacts on model accuracy, inference latency, and GPU resource utilization. In multi-model parallel inference scenarios, resource contention and performance coupling between quantized models further increase the complexity of system scheduling optimization.

[0003] However, in practical deployment, the inference stage of DNN models still faces significant computational and scheduling challenges, especially in multi-model parallel inference scenarios, where different tasks typically require the use of different types of DNN models. For example, in autonomous driving applications, Convolutional Neural Networks (CNNs) are primarily used for object detection and environmental perception, while Transformer models are mainly employed in machine translation and natural language generation applications. Since different DNN models exhibit significant differences in computational characteristics, memory usage, quantization methods, and GPU resource requirements, how to efficiently combine, quantize, deploy, and coordinate multiple DNN models to improve the parallel execution efficiency of inference clusters has become an important research direction in the field of deep learning inference.

[0004] Related research indicates that during the DNN inference stage, reasonable scheduling of different DNN models and collaborative optimization using heterogeneous GPU resources can effectively improve the parallel execution efficiency of model inference and the overall throughput of the inference cluster. Furthermore, combining model quantization techniques can further reduce GPU memory usage and data transfer overhead, improving the utilization rate of heterogeneous GPU resources. However, existing technologies in heterogeneous GPU environments still suffer from problems such as low resource utilization, lack of adaptive model scheduling, insufficient matching between quantization strategies and hardware resources, and inadequate optimization of inference performance. Summary of the Invention

[0005] To overcome the aforementioned problems in the existing technology, this invention proposes a quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters.

[0006] The technical solution adopted by this invention to solve its technical problem is: a quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters, comprising the following steps: Step 1: Obtain current information, including the remaining selectable DNN models, the DNN models deployed on each GPU server, and the GPU servers that do not meet the computing resource and memory constraints. Step 2: Based on the information obtained in Step 1, the scheduler selects the target GPU server, the DNN model to be deployed, and the quantization level of the DNN model. Step 3: Determine whether the selected DNN model can maximize system throughput when deployed on the target GPU server and executed in parallel inference at the corresponding quantization level. If the system throughput is maximized, proceed to Step 4. If the current selection fails to maximize system throughput, return to Step 2 and select again until a target combination that maximizes system throughput is determined. Step 4: Deploy the DNN models in the target combination obtained in Step 3 to the target GPU server according to the corresponding quantization level, and update the remaining selectable DNN models, the model deployment information of each GPU server, and the GPU cluster resource status information. Step 5: After each DNN model deployment is completed, determine whether the computing resources and memory resources of the corresponding GPU server still meet the deployment and inference requirements. If not, stop deploying DNN models to the target GPU server. Step 6: Repeat steps 1-5 until the algorithm converges.

[0007] The above-mentioned quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters, specifically step 1, involves: letting... express Heterogeneous GPU servers express A collection of heterogeneous DNN models Represents each DNN model The degree of quantification can be selected. Indicates deployment on GPU server On A set of DNN models; the scheduler obtains the set of remaining available DNN models by observation. Each GPU server The collection of DNN models already deployed. And a collection of GPU servers that do not meet computing resource and memory constraints. .

[0008] The above-mentioned quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters, specifically step 4 is as follows: Step 4.1, configure each GPU server Treat it as an intelligent agent and configure it with a local deep recurrent Q-network. For the candidate DNN model set Each of the remaining optional models and the degree of quantization corresponding to this model. GPU server Through its corresponding deep recurrent Q-network Calculate the action value function ; Step 4.2: Update the relevant information, namely the set of remaining available DNN models. Each GPU server The collection of DNN models already deployed. .

[0009] The above-mentioned quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters, in step 4.1, selects a strategy that adopts... - Greedy strategy, the scheduler uses probability... GPU-based server The corresponding deep recurrent Q-network selects the action with the highest value for the current action: The degree of quantification is about to be DNN model Deployed on GPU server Parallel inference is performed on top; the scheduler uses probability... Perform random exploration from the candidate model set Randomly select a DNN model Choose a degree of quantification for it. and from the current set of available GPU servers Randomly select a GPU server Deployment is carried out.

[0010] The above-mentioned quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters, specifically step 5, involves: each GPU server... Given the constraints of computing resources and memory, after each DNN model is deployed to a GPU server, it is determined whether the computing resources and memory resources of that GPU server still meet the deployment and inference requirements. If not, the deployment of DNN models to that target GPU server is stopped, and the information of GPU servers in the GPU cluster that meet the deployment and inference requirements is updated.

[0011] The beneficial effects of the present invention are: (1) The present invention fully considers the heterogeneity of resources among GPU servers, and on this basis proposes a quantization-aware multi-model online inference scheduling mechanism based on deep reinforcement learning, so as to achieve optimal matching between different DNN models, quantization degree and heterogeneous GPUs, and effectively improve the overall parallel efficiency and resource utilization of inference tasks.

[0012] (2) The model parallel inference method proposed in this invention can automatically combine multiple DNN models and execute them concurrently in a heterogeneous GPU cluster, thereby adapting to the needs of diverse task scenarios, effectively reducing GPU idle rate, improving the throughput and energy efficiency of inference computing, and meeting the actual deployment needs of large-scale real-time inference systems. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the process of this invention; Figure 2 This is a comparison chart of throughput under different numbers of DNN models in the embodiments of the present invention; Figure 3 This is a comparison chart of convergence performance under different numbers of DNN models in the embodiments of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] This invention provides a quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters, such as... Figure 1 As shown, it includes the following steps: Step 1: Obtain current information, including the remaining selectable DNN models, the DNN models deployed on each GPU server, and GPU servers that do not meet the computational resource and memory constraints.

[0016] make express Heterogeneous GPU servers express A collection of heterogeneous DNN models Represents each DNN model The degree of quantification can be selected. Indicates deployment on GPU server On The scheduler obtains the set of remaining available DNN models by observation. Each GPU server The collection of DNN models already deployed. And a collection of GPU servers that do not meet computing resource and memory constraints. .

[0017] Step 2: Based on the status information, the scheduler selects the target GPU server, the DNN model to be deployed, and the quantization level of the DNN model. It then determines whether the selected DNN model, when deployed on the target GPU server at the corresponding quantization level and performing parallel inference, can maximize system throughput. If the current selection fails to maximize system throughput, it continues to search for other combinations of DNN models, quantization levels, and GPU servers until a target combination that maximizes system throughput is determined.

[0018] Step 3: Deploy the DNN models in the target combination to the target GPU server according to the corresponding quantization level, and update the remaining selectable DNN models, the model deployment information of each GPU server, and the GPU cluster resource status information.

[0019] (1) Each GPU server Treat it as an intelligent agent and configure it with a local deep recurrent Q-network. For the candidate DNN model set Each of the remaining optional models and the degree of quantization corresponding to this model. GPU server Through its corresponding deep recurrent Q-network Calculate the action value function To strike a balance between exploration and utilization, a decision-making approach is adopted. - Greedy strategy. Specifically, the scheduler uses probability... GPU-based server The corresponding deep recurrent Q-network selects the action with the highest value for the current action. The degree of quantification is about to be DNN model Deployed on GPU server Parallel inference is performed on top; the scheduler uses probability... Perform random exploration from the candidate model set Randomly select a DNN model Choose a degree of quantification for it. and from the current set of available GPU servers Randomly select a GPU server Deployment is carried out.

[0020] (2) Update relevant information, namely the remaining set of available DNN models. Each GPU server The collection of DNN models already deployed. The remaining available GPU server set .

[0021] Step 4: After each DNN model deployment is completed, determine whether the computing resources and memory resources of the corresponding GPU server still meet the deployment and inference requirements. If not, stop deploying DNN models to the target GPU server.

[0022] Each GPU server Given computational and memory constraints, after each DNN model deployment to a GPU server, it's necessary to determine whether the GPU server's computational and memory resources still meet the deployment and inference requirements. If not, deployment of DNN models to that target GPU server should be stopped, and the information on GPU servers in the GPU cluster that meet the deployment and inference requirements needs to be updated.

[0023] Step 5: Repeat steps 1, 2, 3, and 4 until the algorithm converges.

[0024] To comprehensively evaluate the performance of the proposed method, a systematic experiment was designed and conducted. Twenty typical DNN models were selected for the experiment, specifically including: BEiT, CAFormer, CaiT, CoaT, DaViT, DeiT, DenseNet121, DenseNet169, DenseNet201, EfficientFormer, MaxViT, MobileNetV2, MobileNetV3, ResNet50, ResNet101, ResNet152, VGG16, VGG19, Visformer, and ViT. To verify the adaptability and performance of the method in heterogeneous computing environments, the experiment was deployed and tested on five different types of NVIDIA GPUs, including: RTX 4090 (24 GB), Tesla V100 (32 GB), A100 (40 GB), H800 (80 GB), and H20 (96 GB). The dataset used was ImageNet, which is widely representative.

[0025] During the experiment, based on a fixed GPU resource setting, an increasing number of DNN models were gradually introduced to evaluate the adaptability and inference efficiency of the scheduling method under different model sizes. The specific model configurations are as follows: (1) 15 DNN models: CoaT, DeiT, DenseNet121, DenseNet169, DenseNet201, ResNet50, ResNet101, ResNet152, MobileNetV2 2), MobileNetV3, VGG16, VGG19, Visformer, EfficientFormer; (2) 20 DNN models: BEiT, CAFormer, CaiT, CoaT, DaViT, DeiT, DenseNet121, DenseNet169, DenseNet201, EfficientFormer, MaxViT, MobileNetV2, MobileNetV3, ResNet50, ResNet101, ResNet152, VGG16, VGG19, Visformer, ViT; (3) 25 DNN models: BEiT, CAFormer, CaiT, CoaT, DaViT, DeiT, DenseNet121, DenseNet169, DenseNet201, EfficientFormer, MaxViT, MobileNetV2, MobileNetV3, ResNet50, ResNet101, ResNet152, VGG16, VGG19, Visformer, ViT; Experimental results are as follows Figure 2 and Figure 3 As shown. HyMOD is the method proposed in this invention, and the other methods serve as comparative baselines, including non-learning methods (Random and Greedy) and reinforcement learning methods (IQL-based, VDN-based, QMIX-based, and MAPPO-based). Figure 2 This study demonstrates the impact of different scheduling algorithms on concurrent inference efficiency and overall system throughput when deploying multiple DNN models in a heterogeneous GPU cluster. Experiments were conducted in various settings with a fixed number of GPUs (RTX 4090, Tesla V100, A100, H800, H20), gradually increasing the number of models. Results show that the HyMOD method significantly outperforms the comparative methods MAPPO-based, QMIX-based, VDN-based, IQL-based, Greedy, and Random in all test scenarios, with average system throughput improvements ranging from 34.49% to 57.35%, fully validating the significant advantages of this invention in improving inference efficiency in heterogeneous environments. Figure 3 The training convergence process of HyMOD, MAPPO-based, QMIX-based, VDN-based, and IQL-based methods was further demonstrated under 20 model configurations. The training dynamics were analyzed by examining the trend of cumulative reward (i.e., concurrent inference throughput) over 1000 training epochs. Experimental results show that the HyMOD method achieves faster and more stable convergence under various settings, with a convergence speed approximately 1.60-2.65 times faster than MAPPO-based, QMIX-based, VDN-based, and IQL-based methods. Furthermore, it outperforms other methods in final throughput, fully demonstrating the efficiency and convergence stability of this invention in complex scheduling tasks.

[0026] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its scope and spirit, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A quantization-aware multi-model online inference scheduling method for a heterogeneous GPU cluster, characterized in that, Includes the following steps: Step 1: Obtain current information, including the remaining selectable DNN models, the DNN models deployed on each GPU server, and the GPU servers that do not meet the computing resource and memory constraints. Step 2: Based on the information obtained in Step 1, the scheduler selects the target GPU server, the DNN model to be deployed, and the quantization level of the DNN model. Step 3: Determine whether the selected DNN model can maximize system throughput when deployed on the target GPU server and executed in parallel inference at the corresponding quantization level. If the system throughput is maximized, proceed to Step 4. If the current selection fails to maximize system throughput, return to Step 2 and select again until a target combination that maximizes system throughput is determined. Step 4: Deploy the DNN models in the target combination obtained in Step 3 to the target GPU server according to the corresponding quantization level, and update the remaining selectable DNN models, the model deployment information of each GPU server, and the GPU cluster resource status information. Step 5: After each DNN model deployment is completed, determine whether the computing resources and memory resources of the corresponding GPU server still meet the deployment and inference requirements. If not, stop deploying DNN models to the target GPU server. Step 6: Repeat steps 1-5 until the algorithm converges.

2. The quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters according to claim 1, characterized in that, The step 1 is specifically: let represent a plurality of heterogeneous GPU servers, represent a plurality of heterogeneous DNN models, represent each DNN model a quantifiable degree of selection, represent a plurality of DNN models deployed on the GPU server ; the scheduler obtains the set of remaining selectable DNN models by observation , the set of DNN models deployed on each GPU server , and the set of GPU servers that do not meet the computing resource and memory constraints . .​ 3. The quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters according to claim 1, characterized in that, Step 4 specifically involves: Step 4.1, configure each GPU server Treat it as an intelligent agent and configure it with a local deep recurrent Q-network. For the candidate DNN model set Each of the remaining optional models and the degree of quantization corresponding to this model. GPU server Through its corresponding deep recurrent Q-network Calculate the action value function ; Step 4.2: Update the relevant information, namely the set of remaining available DNN models. Each GPU server The collection of DNN models already deployed. .

4. The quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters according to claim 3, characterized in that, In step 4.1, the strategy to be selected is adopted. - Greedy strategy, the scheduler uses probability... GPU-based server The corresponding deep recurrent Q-network selects the action with the highest value for the current action. The degree of quantification is about to be DNN model Deployed on GPU server Parallel inference is performed on top; the scheduler uses probability... Perform random exploration from the candidate model set Randomly select a DNN model Choose a degree of quantification for it. and from the current set of available GPU servers Randomly select a GPU server Deployment is carried out.

5. The quantization-aware multi-model online inference scheduling method for heterogeneous GPU clusters according to claim 1, characterized in that, Step 5 specifically involves: each GPU server Given the constraints of computing resources and memory, after each DNN model is deployed to a GPU server, it is determined whether the computing resources and memory resources of that GPU server still meet the deployment and inference requirements. If not, the deployment of DNN models to that target GPU server is stopped, and the information of GPU servers in the GPU cluster that meet the deployment and inference requirements is updated.

Images