A parallel inference method for deep neural network models on heterogeneous GPU clusters

Abstract

The present invention discloses a parallel inference method for deep neural network models for heterogeneous GPU clusters, which relates to the field of distributed machine learning. The method includes obtaining current information, remaining selectable DNN models, DNN models deployed on each GPU server, and GPU servers that do not meet the DNN model quantity constraint; a scheduler selects a DNN model to be deployed on the selected GPU server and calculates the throughput of executing parallel inference at this time; until the DNN model and GPU server combination with the highest throughput is found, and the relevant information is updated; it is determined whether the DNN model deployed on the GPU meets the quantity constraint, and the GPU cluster information is updated until all GPUs meet the specific DNN model quantity constraint; and the above steps are repeated until the algorithm converges. The present invention makes full use of limited heterogeneous GPU resources and selects DNN models with high compatibility for deployment and parallel inference execution to maximize throughput.

Description

Technical Field

[0001] The present invention relates to the field of distributed machine learning, and in particular to a parallel reasoning method for deep neural network models for heterogeneous GPU clusters. Background Art

[0002] With the rapid development of deep learning technology, the representational capabilities of deep neural network (DNN) models have continued to increase, significantly driving innovation and application expansion in the artificial intelligence industry. DNN models have achieved breakthroughs in key areas such as natural language understanding, image recognition, and autonomous driving, exemplified by the successful application of the large-scale language model ChatGPT. The implementation of these applications relies on an efficient model inference computational process: rapidly responding to and making decisions based on pre-trained models in response to external task requests within a large-scale server cluster. Currently, DNN model inference typically relies on data centers pre-deployed on servers. Upon receiving a task request, the corresponding model is dynamically loaded and executed for real-time inference computation. As task complexity and user concurrency continue to increase, improving the real-time performance and throughput of DNN model inference has become a core bottleneck limiting the service capabilities of deep learning systems.

[0003] To improve inference efficiency, the industry widely uses GPU clusters to perform parallel inference tasks for DNN models. However, GPU resources in real-world deployment environments are often heterogeneous, encompassing differences in hardware parameters such as computing power and video memory size, as well as differences in driver compatibility and model execution efficiency. In a heterogeneous GPU cluster environment, different types of DNN models have varying degrees of dependence on GPU resources. For example, convolutional neural networks (CNNs) rely more on tensor parallel computing capabilities for object detection tasks in autonomous driving, while Transformer models focus more on memory bandwidth and matrix operation efficiency for machine translation and text generation tasks. Therefore, in heterogeneous GPU clusters, intelligently matching and deploying different models based on their operational characteristics and the heterogeneity of GPU resources becomes a key challenge in improving model parallel inference efficiency.

[0004] Research has shown that, during the DNN inference phase, properly scheduling and combining different DNN models to efficiently utilize heterogeneous GPU resources can significantly improve the model's parallel execution efficiency and the overall cluster throughput. Therefore, for the parallel inference of DNN models in heterogeneous GPU clusters, a scheduling and deployment method with resource awareness and inference performance optimization capabilities is urgently needed to achieve efficient and scalable DNN inference services. Summary of the Invention

[0005] In order to overcome the above problems existing in the prior art, the present invention proposes a parallel reasoning method for deep neural network models for heterogeneous GPU clusters.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a parallel reasoning method for deep neural network models for heterogeneous GPU clusters, comprising the following steps:

[0007] Step 1: Obtain current information, including the remaining selectable DNN models, the DNN models deployed on each GPU server, and the GPU servers that do not meet the DNN model quantity constraint;

[0008] Step 2: The scheduler selects a DNN model to be deployed on the selected GPU server and calculates the throughput of the currently selected DNN model deployed on the selected GPU to perform parallel inference.

[0009] Step 3: Repeat step 2 until the DNN model and GPU server combination that maximizes throughput is found, deploy the DNN model to the GPU, and update the relevant information.

[0010] Step 4: Determine whether the DNN model deployed on the GPU meets the quantity constraint, and update the GPU cluster information until all GPUs meet the specific DNN model quantity constraint;

[0011] Step 5: Repeat steps 1-4 until the algorithm converges.

[0012] In the above-mentioned method for parallel inference of a deep neural network model for a heterogeneous GPU cluster, step 1 is specifically as follows: express Heterogeneous GPU servers, express A collection of heterogeneous DNN models, Indicates deployment on a GPU server on The scheduler obtains the set of remaining optional DNN models by observing , each GPU server A collection of DNN models deployed on And the collection of GPU servers that do not meet the DNN model quantity constraint .

[0013] The above-mentioned parallel reasoning method for deep neural network models in heterogeneous GPU clusters, the step 2 is specifically: each GPU server As an agent, there is a local deep recurrent Q network , through a deep recurrent Q network , against Each of the remaining optional DNN models in Calculate the corresponding action value function ;DNN model selection decision adopts -Greedy strategy.

[0014] The above-mentioned parallel reasoning method for deep neural network models oriented to heterogeneous GPU clusters, - Greedy strategy is as follows: the scheduler uses probability use Corresponding Select Action , the DNN model Deployed on GPU servers Optimal parallel reasoning is performed on , the scheduler then Select an optional DNN model to deploy to Select a GPU server superior.

[0015] The above-mentioned parallel reasoning method for deep neural network models of heterogeneous GPU clusters, wherein the updating of relevant information in step 3 specifically includes: the set of remaining optional DNN models Each GPU server A collection of DNN models deployed on .

[0016] The above-mentioned parallel reasoning method for deep neural network models in heterogeneous GPU clusters, the step 4 is specifically as follows: each GPU server There is a specific number constraint for deploying DNN models. After each DNN model is deployed to the GPU server, it is determined whether the DNN model deployed on the GPU meets the number constraint. If If the number constraints of the deployed DNN model are met, the information of the GPU servers that do not meet the constraints in the GPU cluster will be updated.

[0017] The beneficial effects of the present invention are: (1) The present invention fully considers the heterogeneity of resources among GPU servers, and on this basis proposes a multi-DNN model scheduling mechanism based on deep reinforcement learning, which achieves the optimal matching between different DNN models and heterogeneous GPUs, and effectively improves the overall parallel efficiency and resource utilization of inference tasks.

[0018] (2) The model parallel reasoning method proposed in this invention can automatically combine multiple DNN models and execute them concurrently in a heterogeneous GPU cluster, thereby adapting to the requirements of diverse task scenarios, effectively reducing the GPU idle rate, improving the throughput and energy efficiency of reasoning calculations, and meeting the actual deployment requirements of large-scale real-time reasoning systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Is a schematic flow diagram of the present invention;

[0020] Figure 2 This is a throughput comparison chart under different numbers of DNN models in an embodiment of the present invention;

[0021] Figure 3 This is a comparison chart of convergence performance under different numbers of DNN models in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] The present invention provides a parallel reasoning method for deep neural network models on heterogeneous GPU clusters. Figure 1 As shown, the following steps are included:

[0024] Step 1: Obtain current information, including the remaining selectable DNN models, the DNN models deployed on each GPU server, and the GPU servers that do not meet the DNN model quantity constraint.

[0025] make express Heterogeneous GPU servers, express A collection of heterogeneous DNN models, Indicates deployment on a GPU server on The scheduler obtains the remaining optional DNN model set by observation. , each GPU server A collection of DNN models deployed on And the collection of GPU servers that do not meet the DNN model quantity constraint .

[0026] In step 2, the scheduler selects a DNN model to deploy on the selected GPU server and determines whether the currently selected DNN model deployed on the selected GPU to perform parallel inference maximizes the throughput. This step is repeated until the DNN model and GPU server combination that maximizes the throughput is found. The DNN model is deployed on the GPU and the relevant information is updated.

[0027] (1) Each GPU server As an agent, there is a local deep recurrent Q network . Through the deep recurrent Q network , against Each of the remaining optional DNN models in Calculate the corresponding action value function To achieve a trade-off between exploration and exploitation, the DNN model selection decision adopts - Greedy strategy. Specifically, the scheduler uses the probability use Corresponding Select Action , the DNN model Deployed on GPU servers Optimal parallel reasoning is performed on , the scheduler then Select an optional DNN model to deploy to Select a GPU server superior.

[0028] (2) Update relevant information, that is, the set of remaining optional DNN models Each GPU server A collection of DNN models deployed on .

[0029] Step 3: After each DNN model is deployed to the GPU server, determine whether the DNN model deployed on the GPU meets the quantity constraint, and update the GPU cluster information until all GPUs meet the specific DNN model quantity constraint.

[0030] Each GPU server There is a specific number constraint for deploying DNN models. After each DNN model is deployed to a GPU server, it is necessary to determine whether the DNN model deployed on the GPU meets the number constraint. To meet the quantity constraints for deploying DNN models, you need to update the information of GPU servers in the GPU cluster that do not meet the constraints.

[0031] Step 4: Repeat steps 1, 2, and 3 until the algorithm converges.

[0032] To comprehensively evaluate the performance of the proposed method, a systematic experiment was designed and conducted. Five typical DNN models were selected for the experiment: convolutional neural networks (CNNs), recurrent neural networks (RNNs), Transformer networks, generative adversarial networks (GANs), and graph neural networks (GNNs). The selected models included: ResNet50, ResNet101, ResNet152, VGG16, VGG19, DenseNet20, LSTM, GRU, ConvLSTM, ViT, Visformer, VOLO, BEiT, DaViT, DeiT, Vanilla-GAN, CGAN, DCGAN, W-GAN, cGAT, GCN, SageGCN, and StackGCN.

[0033] To verify the adaptability and performance of the method in heterogeneous computing environments, experiments were conducted on five different types of NVIDIA GPUs: RTX2080 (8GB), RTX4090 (24GB), Tesla V100 (32GB), Tesla T4 (16GB), and A100 (40GB). The dataset used was ImageNet, which is widely representative.

[0034] During the experiment, based on a fixed GPU resource setting, we gradually introduced an increasing number of DNN models to evaluate the adaptability and inference efficiency of the scheduling method under different model sizes. The specific model configuration is as follows:

[0035] (1) 10 DNN models: ResNet50, VGG16, LSTM, GRU, ViT, VOLO, Vanilla-GAN, CGAN, cGAT, GCN;

[0036] (2) 15 DNN models: ResNet50, VGG16, DenseNet201, LSTM, GRU, Con-vLSTM, ViT, VOLO, BEiT, Vanilla-GAN, CGAN, DCGAN, cGAT, GCN, SageGCN;

[0037] (3) 17 DNN models: ResNet50, ResNet152, VGG16, DenseNet201, LSTM, GRU, Con-vLSTM, ViT, VOLO, BEiT, DEiT, Vanilla-GAN, CGAN, DCGAN, cGAT, GCN, SageGCN;

[0038] (4) 20 DNN models: ResNet50, ResNet152, VGG16, DenseNet201, LSTM, LSTM, GRU, Con-vLSTM, ViT, VOLO, BEiT, DEiT, Vanilla-GAN, CGAN, DCGAN, W-GAN, cGAT, GCN, SageGCN, StackGCN.

[0039] The experimental results are as follows Figure 2 and Figure 3 MoD-based represents the method proposed in this paper, and the remaining methods serve as comparison baselines, including non-learning methods (Random and Greedy) and reinforcement learning methods (VDN-based and QMIX-based). Figure 2 This paper demonstrates the impact of different scheduling algorithms on concurrent inference efficiency and overall system throughput when deploying multiple DNN models in a heterogeneous GPU cluster. Experiments were conducted in various settings with a fixed number of GPUs and a gradually increasing number of models. The results show that the MoD-based approach significantly outperforms the comparative methods QMIX-based, VDN-based, Greedy, and Random in all test scenarios, with average improvements in system throughput of 25%, 31%, 35%, and 51%, respectively. This fully demonstrates the significant advantages of this invention in improving inference efficiency in heterogeneous environments. Figure 3 The training convergence process of the MoD-based, QMIX-based, and VDN-based methods under 15 model configurations was further demonstrated. The training dynamics were analyzed by analyzing the changing trends of the cumulative reward (i.e., concurrent inference throughput) over 1000 training cycles. Experimental results show that the MoD-based method achieves faster and more stable convergence in all settings, with convergence speeds increasing by approximately 1.60 times and 2.00 times compared to the QMIX-based and VDN-based methods, respectively. The method also outperforms the other methods in terms of final throughput, fully demonstrating the efficiency and convergence stability of the present invention in complex scheduling tasks.

[0040] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the spirit and scope of protection of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present invention.

Claims

1. A parallel inference method for deep neural network models on heterogeneous GPU clusters, characterized by: The steps include: Step 1: Obtain current information, including the remaining selectable DNN models, the DNN models deployed on each GPU server, and the GPU servers that do not meet the DNN model quantity constraint; Step 2: The scheduler selects a DNN model to be deployed on the selected GPU server and calculates the throughput of the currently selected DNN model deployed on the selected GPU to perform parallel inference. Step 3: Repeat step 2 until the DNN model and GPU server combination that maximizes throughput is found, deploy the DNN model to the GPU, and update the relevant information. Step 4: Determine whether the DNN model deployed on the GPU meets the quantity constraint, and update the GPU cluster information until all GPUs meet the specific DNN model quantity constraint; Step 5: Repeat steps 1-4 until the algorithm converges; The step 1 is specifically as follows: express Heterogeneous GPU servers, express A collection of heterogeneous DNN models, Indicates deployment on a GPU server on The scheduler obtains the set of remaining optional DNN models by observing , each GPU server A collection of DNN models deployed on And the collection of GPU servers that do not meet the DNN model quantity constraint ; The step 2 is specifically as follows: As an agent, there is a local deep recurrent Q network , through a deep recurrent Q network , against Each of the remaining optional DNN models in Calculate the corresponding action value function ;DNN model selection decision adopts - Greedy strategy; described - Greedy strategy is as follows: the scheduler uses probability use Corresponding Select Action , the DNN model Deployed on GPU servers Optimal parallel reasoning is performed on , the scheduler then Select an optional DNN model to deploy to Select a GPU server superior.

2. The method for parallel reasoning of a deep neural network model for a heterogeneous GPU cluster according to claim 1, characterized in that: The updating of relevant information in step 3 specifically includes: the set of remaining optional DNN models Each GPU server A collection of DNN models deployed on .

3. The method for parallel reasoning of deep neural network models for heterogeneous GPU clusters according to claim 1, characterized in that: The step 4 is specifically as follows: each GPU server There is a specific number constraint for deploying DNN models. After each DNN model is deployed to the GPU server, it is determined whether the DNN model deployed on the GPU meets the number constraint. If If the number constraints of deployed DNN models are met, the information of GPU servers that do not meet the constraints in the GPU cluster will be updated.

Images