A heterogeneous cluster-oriented DNN inference task batch scheduling method

Abstract

The application discloses a DNN inference task batch scheduling method for a heterogeneous cluster. The method comprises a DNN inference task representation module, a heterogeneous cluster dynamic hardware processing capacity extraction module and a DNN inference task batch scheduling module. The DNN inference task representation module is used for classifying DNN inference tasks according to task types to construct a DNN inference task category set. The heterogeneous cluster dynamic hardware processing capacity extraction module is used for dividing computing nodes and constructing a heterogeneous computing node set according to a division result. The DNN inference task batch scheduling module is used for scheduling DNN tasks by a target search algorithm according to the DNN inference task division set and the heterogeneous computing node set. Through the above modules, the application realizes effective classification of DNN inference tasks, solves the measurement of different node computing powers of the heterogeneous cluster, realizes optimal scheduling of the DNN inference tasks, and realizes the optimization of the DNN inference task batch scheduling for the heterogeneous cluster.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to DNN inference task batch scheduling technology, in particular to a DNN inference task batch scheduling method for a heterogeneous cluster. BACKGROUND

[0002] In recent years, with the vigorous development of deep neural networks (DNN), artificial intelligence (AI) applications have been fully developed, and the application categories and the number of applications have increased significantly. Due to the outstanding large-scale parallel computing capability, general-purpose graphics processing units (GPGPU) have become the mainstream deep learning network accelerator.

[0003] Accelerating DNN networks on GPUs usually involves two stages. First, a large-scale training dataset is used to train a selected DNN model. This training process is very time-consuming and is usually placed on a high-performance computing cluster. After the training is completed, the network model is immediately deployed to a large-scale cluster for actual inference phase work. On the other hand, high-precision DNN networks have given rise to a large number of artificial intelligence applications based on this technology. These applications have a large number of user groups and correspond to different use targets. Usually, these applications send data collected on the terminal back to the cloud data center to use high-performance GPUs to perform inference work.

[0004] However, due to the differences between GPU hardware resources, most of the current clusters are heterogeneous, mainly in terms of computing power heterogeneity, that is, a large number of computing nodes with different computing power are often divided in the cluster to perform DNN application inference tasks. Task scheduling in the cluster usually needs to pay attention to the load balancing degree between different computing nodes and the average completion time of all tasks.

[0005] DNN application inference can be divided into three categories according to different application requirements: real-time tasks, interactive tasks, and background tasks. Real-time tasks require inference to be completed within a very small delay. Application scenarios such as security camera real-time monitoring and autonomous driving, etc. On the contrary, background tasks have little requirement for the time delay of inference but are very sensitive to the accuracy of inference. Interactive tasks are more complex. They can tolerate a certain inference delay and do not always require the highest inference accuracy, so the inference accuracy and inference rate can be reasonably reduced. In summary, the application requirements of the three types of tasks are different. Some require high inference rate and fast response. Others do not require fast response but require high inference accuracy. That is, inference tasks can be divided into many categories, and the models used, inference rate, and inference accuracy are different. Therefore, for tasks, the timeout amount of the inference task is an important measure of the quality of a scheduling result.

[0006] At the task flow level, a large number of users generate a large number of DNN application inference requests every day, and these inference requests arrive at the heterogeneous cluster at different time points for inference. Under big data, the task flow generated by a large number of users will show obvious tidal rules. For example, the density distribution of the number of tasks in a time period in a day is often high in the daytime and low at night. In traditional task scheduling, the distribution characteristics of tasks in the task flow are rarely studied. SUMMARY

[0007] The application provides a DNN inference task batch scheduling method for a heterogeneous cluster to realize optimization of DNN inference task batch scheduling on a cluster.

[0008] The technical solutions of the application mainly solve the following three problems:

[0009] (a) modeling the dynamic hardware processing capacity of the heterogeneous cluster

[0010] Most clusters are composed of various types of computing board cards with different computing power. To solve the measurement of the computing power of different nodes of the heterogeneous cluster, the application provides a modeling method of dynamic hardware processing capacity.

[0011] (b) modeling the DNN inference task through analysis of the DNN inference task

[0012] For different types of DNN inference tasks, the DNN model used, the required inference time constraint, and the required inference accuracy constraint are different. The application provides a DNN inference task modeling method, which can effectively classify the tasks in the DNN inference task set.

[0013] (c) scheduling the DNN inference task so that the scheduling result can reach the optimal value under the evaluation function composed of each evaluation index. The application provides a dimension reduction method based on static strategy and dynamic parameters, constructs a low-dimensional parameter solution space representing the scheduling result, and then introduces a meta-heuristic search algorithm to search for the optimal solution of scheduling.

[0014] Technical solutions of the application:

[0015] The embodiment provides a DNN inference task batch scheduling method for a heterogeneous cluster, which comprises the following steps:

[0016] a DNN inference task representation module, a heterogeneous cluster dynamic hardware processing capacity extraction module, and a DNN inference task batch scheduling module;

[0017] The DNN inference task representation module is used to classify the DNN inference tasks according to the task type to obtain a DNN inference task division set.

[0018] The heterogeneous cluster dynamic hardware processing capability extraction module is configured to divide the computing nodes and construct a heterogeneous computing node set according to the division result.

[0019] The DNN inference task batch scheduling module is configured to schedule the DNN inference tasks by a target search algorithm according to the DNN inference task division set and the heterogeneous computing node set.

[0020] Optionally, the DNN inference task representation module comprises:

[0021] A cluster deployment model statistic is configured to count the DNN models that can be inferred on each node in the cluster and determine a DNN model list.

[0022] A cluster external DNN application statistic is configured to count a set of external DNN applications corresponding to all the DNN models that can be inferred in the cluster.

[0023] A DNN inference task divider is configured to determine a DNN inference task division set according to the DNN model list and the set of external DNN applications.

[0024] Optionally, the DNN inference task division set comprises all the task types of DNN inference, and each task type corresponds to an allowed response time, a target use model and a target node set.

[0025] Optionally, the heterogeneous cluster dynamic hardware processing capability extraction module comprises:

[0026] A cluster computing node divider is configured to dynamically divide the boards in the cluster according to the computing power to obtain a heterogeneous computing node set.

[0027] A cluster inference task processing timer is configured to, according to the DNN inference task division set and the heterogeneous computing node set, pre-execute each type of DNN inference task on each processable heterogeneous computing node, count the time consumption of all the processes, generate a dynamic hardware processing capability matrix according to the time consumption result, and send the matrix to the DNN inference task batch scheduling module.

[0028] Optionally, a dimension reduction method based on static strategies and dynamic parameters is used to construct a dynamic parameter matrix representing the scheduling result.

[0029] Correspondingly, the DNN inference task batch scheduling module comprises:

[0030] A task scheduler is configured to segment the input task stream in the DNN inference task division set and generate a corresponding scheduling result stream.

[0031] A scheduling result evaluator is configured to evaluate each scheduling result in the scheduling result stream according to an evaluation function.

[0032] A searcher is configured to search in the dynamic parameter matrix through a meta-heuristic algorithm, and a solution obtained after convergence of the meta-heuristic algorithm is taken as an optimal scheduling solution.

[0033] Optionally, the evaluation function is:

[0034]

[0035] wherein σ1 is an average completion time of the tasks, σ2 is an average timeout amount of all the tasks, and σ3 is a load balancing degree; ξ is a certain scheduling result, and λ1, λ2 and λ3 are three different weights.

[0036] The DNN inference task can be effectively classified through the DNN inference task representation module, the measurement of the computing power of different nodes of the heterogeneous cluster is solved through the heterogeneous cluster dynamic hardware processing capability extraction module, the DNN inference task is scheduled through the DNN inference task batch scheduling module, the scheduling result can reach the optimal under each evaluation index composition evaluation function, and the optimization of the DNN inference task batch scheduling of the heterogeneous cluster is realized.

[0037]

[0038] BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is a framework design diagram of the DNN inference task batch scheduling method for the heterogeneous cluster in the application;

[0040] Figure 2 It is an implementation method schematic diagram of the DNN inference task batch scheduling in the application;

[0041] Figure 3 It is an example algorithm flow of the batch scheduling implementation method in the application;

[0042] Figure 4 It is a comparison diagram of scheduling results of different types of task streams in the application;

[0043] Figure 5 It is a comparison diagram of scheduling results of multiple heterogeneous cluster node configurations in the application;

[0044] Figure 6 It is a comparison diagram of scheduling results of multiple optimization objectives in the application;

[0045] Figure 7 It is a comparison diagram of convergence rates of multiple optimization algorithms in the application;

[0046] ​Figure 8 A heterogeneous cluster-oriented DNN inference task batch scheduling method framework scheduling example diagram in the present application. DETAILED DESCRIPTION

[0047] The application will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, and not to limit the present application. In addition, it should be noted that, for the sake of description, only the parts related to the present application are shown in the drawings, not all the structures.

[0048] In order to realize the heterogeneous cluster-oriented DNN inference task batch scheduling optimization method, the present application designs a scheduling optimization method based on a static-dynamic hybrid dimension reduction mechanism. Figure 1 The overall design framework diagram of the DNN-BS scheduling optimization method, including:

[0049] DNN inference task characterization module, heterogeneous cluster dynamic hardware processing capability extraction module and DNN inference task batch scheduling module;

[0050] The DNN inference task characterization module is used to classify DNN inference tasks according to task types to obtain a DNN inference task division set; wherein different DNN inference tasks are characterized by task feature vectors.

[0051] The heterogeneous cluster dynamic hardware processing capability extraction module is used to divide the computing nodes and construct a heterogeneous computing node set according to the division result.

[0052] The DNN inference task batch scheduling module is used to schedule DNN inference tasks according to the DNN inference task division set and the heterogeneous computing node set through a target search algorithm.

[0053] The specific composition and execution steps of the above three modules are described below.

[0054] I. DNN inference task characterization module

[0055] (a) Cluster deployment model statistic, used to count the DNN models that can be inferred on each node in the cluster, determine the DNN model list, see Table 1 for details, Table 1 lists the model set that can be inferred on each node.

[0056] Compute node 1 Compute node 2 …… Compute node n Model set 1 Model set 2 …… Model machine n

[0057] Table 1

[0058] (b) a cluster external DNN application statistic for counting a set of external DNN applications corresponding to all inferable DNN models in the cluster. Each application has a different task classification, i.e. real-time task, interactive task or background task. According to the cluster external DNN application statistic, the response time allowed for each DNN application to generate an inference request task can be obtained. And the used models are counted to form an application information matrix, such as Figure 2 .

[0059] DNN application 1 DNN application 2 …… DNN application n Allowed response Allowed response …… Allowed response Time 1 Time 2 …… Time n Use model 1 Use model 2 …… Use model n

[0060] Table 2

[0061] (c) a DNN inference task partitioner for determining a DNN inference task partition set according to the DNN model list and the external DNN application set. Specifically, the DNN inference task partitioner synthesizes the model statistics list and the application information matrix given by the cluster deployment model statistic and the cluster external DNN application statistic to obtain the DNN inference task partition set as shown in Table 3. This set gives all task types of DNN inference on the cluster, each type corresponding to the allowed response time, the used model and the node set that can handle this inference task.

[0062] DNN inference task type 1 DNN inference task type 2 …… DNN inference task type n Allowed response time 1 Allowed response time 2 …… Allowed response time n Use model 1 Use model 2 …… Use model n Processable node set 1 Processable node set 2 …… Processable node set n

[0063] Table 3

[0064] II. Heterogeneous cluster dynamic hardware processing capability extraction module

[0065] (a) a cluster computing node partitioner for dynamically partitioning the boards in the cluster according to the computing power to obtain a heterogeneous computing node set.

[0066] Specifically, a large cluster often has a large number of boards with heterogeneous computing power. The cluster computing node partitioner is mainly to dynamically partition the boards in the cluster into computing nodes with different computing power. After partitioning, a heterogeneous computing node set is obtained, which gives all heterogeneous computing nodes in the current cluster that can handle DNN inference tasks.

[0067] (b) a cluster inference task processing timer for counting the time consumed by all processing according to the DNN inference task partition set and the heterogeneous computing node set by actually executing each type of DNN inference task on each available heterogeneous computing node in advance. This time consumed includes the actual inference time and the communication time. Further, a dynamic hardware processing capability matrix is generated according to the time consumed result and sent to the DNN inference task batch scheduling module for generating a scheduling result stream.

[0068] III. DNN inference task batch scheduling module

[0069] (a) Batch scheduling implementation method

[0070] In this embodiment, the dimension reduction method based on static strategy and dynamic parameters constructs a low-dimensional parameter solution space representing the scheduling result, and then introduces a meta-heuristic search algorithm to search for the optimal solution of scheduling, so as to realize the optimization of the batch scheduling method of DNN inference task. For details, see Figure 2 In this scheduling implementation method, a scheduling method template is first constructed, which is composed of a set of static strategies and a set of all parameters in these strategies. This scheduling method template can be instantiated, that is, the static strategies are combined and all the parameter sets in them form a dynamic parameter vector. When a task stream comes, the short-term characteristics of task scheduling are used to dynamically segment the task stream, and each segment uses a different scheduling method instance to perform actual scheduling. The dynamic parameter vectors in all scheduling method instances in the entire task stream form a dynamic parameter matrix. This is a low-dimensional parameter matrix. Since the entire task stream can be scheduled through all scheduling method instances to obtain a scheduling result, and the dynamic parameter vectors in all scheduling method instances currently have current values. Therefore, finally, each overall value in this matrix corresponds to a scheduling result of a task stream. Next, the searcher searches for the value of the dynamic parameter matrix K in combination with the optimization target and dynamic hardware processing capacity. Each value corresponds to a scheduling result. When the search process converges, the optimal K is obtained. Finally, the optimal scheduling result can be generated according to the scheduling instance method of each segment of the task stream and the optimal value of the parameter matrix K. For details, see Figure 3 .

[0071] (b) Task scheduler

[0072] The scheduler segments the input task stream, and each segment uses a scheduling method instance to perform task scheduling operation. When the segmentation, all scheduling method instances and dynamic parameter matrix are determined, a scheduling result stream is generated for a given task stream input. The result stream gives the time when each DNN inference task is scheduled to a certain computing node for inference.

[0073] (c) Scheduling result evaluator, used to evaluate each scheduling result in the scheduling result stream according to the evaluation function.

[0074] There are three evaluation indexes in this scenario, the average completion time of all tasks σ1, the average timeout amount of all tasks σ2 and the load balancing degree σ3. The formula of the evaluation function is as follows:

[0075]

[0076] For a certain scheduling result ξ, the three are combined by using a linear function with three different weights λ1, λ2 and λ3 to form a score evaluation function. When a scheduling result is generated, the score of the score can be obtained. The pros and cons of the scheduling result. The weights here are all set by humans according to reality needs.

[0077] (d) searcher

[0078] By using a meta-heuristic algorithm (such as a particle swarm algorithm) to search in the dynamic parameter matrix K, the optimal scheduling solution can be obtained after convergence. The final scheduling result flow can be obtained through the scheduling solution.

[0079] The experimental verification of the present scheme is as follows:

[0080] The experiment of the present application is implemented on a heterogeneous inference cluster composed of heterogeneous devices such as NVIDIA A100, NVIDIA TITAN Xp, etc. In the above devices, some necessary operating environments such as Ubuntu 16.04, CUDA 10.2, CUDNN 7.5, PyTorch 1.7, etc. are uniformly installed, and DNN inference tasks are executed based on the PyTorch framework. The task set includes various classification tasks and target detection tasks, in which the backbone network adopts nine mainstream neural networks such as DenseNet121, DenseNet169, DenseNet201, ResNet50, ResNet101, ResNet152, VGG16, VGG19 and InceptionV3. This guarantees the heterogeneity of the platform environment and the diversity of the DNN inference task set. Five experiments are designed to evaluate the superiority of DNN-BS compared with existing algorithms from different aspects.

[0081] In the scheduling experiment, this experiment evaluated the proposed DNN inference task batch scheduling system, DNN-BS, based on the heterogeneous inference clusters mentioned above. In the comparative experimental setup, this experiment selected Minimum Completion Time (MCT), Min-min, Max-min, improved Min-min, improved Max-min, Genetic Algorithm (GA), and Particle Swarm Optimization (PSO) as the benchmarks for scheduling methods. Several comparative experiments were also set up, including: 1) comparing the scheduling results under different task flow sizes, multiple task flow types, various heterogeneous cluster node configurations, and various optimization objectives, using all seven comparison methods; 2) comparing the convergence rate and scheduling results under various task flow sizes and computing node ratios. Since the other five methods do not have convergence properties, GA and PSO were compared here. This experiment used the average completion time σ1 of all tasks, the average timeout of all tasks σ2, and the load balancing degree σ3 as observations to evaluate the scheduling performance. The default settings included 20 computing nodes and 8000 inference tasks.

[0082] 1. Comparison of scheduling results under different task flows

[0083] Experiments were conducted using three task flow scales: 3000, 5000, and 8000. Table 1 shows the average completion time σ1, average timeout σ2, load balancing σ3, and the final combined evaluation function score for all tasks. It can be seen that DNN-BS performs best across all metrics for different task flow scales. Since GA and PSO are global search algorithms, their results are also superior to the other five methods. See Table 4 for detailed results.

[0084]

[0085] Table 4

[0086] 2. Comparison of scheduling results under different types of task flows

[0087] The spatiotemporal characteristics of tasks within a task flow differ, such as the number of inference tasks arriving at different times and the distribution of different task categories within the task flow. This experiment selected task flows with five different task distributions for testing. Figure 4 As shown, each category, from left to right, represents the scheduling results of MCT, Min-min, modified Min-min, Max-min, modified Max-min, GA, PSO, and the proposed DNN-BS scheme. The results show that DNN-BS has excellent scheduling results for different types of task flows.

[0088] 3. Comparison of scheduling results under various heterogeneous cluster node configurations

[0089] In this experiment, different node computing power distributions in the cluster are set. The same task flow is used for the experiment, and the average score of ten different task flows is counted under each node computing power configuration. As shown in Figure 5 , from left to right in each category represent the scheduling results of MCT, Min-min, modified Min-min, Max-min, modified Max-min, GA, PSO and the present scheme DNN-BS, and the results show that DNN-BS has the best scheduling effect.

[0090] 4. Comparison of scheduling results under various optimization objectives

[0091] In the evaluation function composed of the average completion time of all tasks σ1, the average timeout amount of all tasks σ2 and the load balancing degree σ3, usually in practice, the adjustment of the weights of the three is adjusted according to the needs to experience which index is more important in scheduling. In this experiment, the weight parameters are adjusted according to the order of λ1, λ2, λ3 to form different scheduling optimization objectives. As shown in Figure 6 , from left to right in each category represent the scheduling results of MCT, Min-min, modified Min-min, Max-min, modified Max-min, GA, PSO and the present scheme DNN-BS, and the results show that DNN-BS still has superior scheduling results to the other seven methods.

[0092] 5. Comparison of convergence rates

[0093] In DNN-BS, due to the existence of global search link, it is necessary to evaluate its convergence rate. In this experiment, the commonly used PSO and GA are used as comparison experiment methods, and experiments are performed on different sizes of task flows. Selecting task flow sizes of 1000, 3000, 5000, 8000, 10000 and 20000 for experiments. As shown in Figure 7 , each category compares the convergence speed of GA, PSO and the present scheme DNN-BS. As can be seen from the figure, DNN-BS has very fast convergence rate in all cases, and the final convergence result also greatly exceeds the PSO and GA algorithms.

[0094] Abstract example:

[0095] Figure 8 The scheduling optimization method based on static and dynamic hybrid dimensionality reduction mechanism is shown, and the use example of DNN-BS scheduling optimization method is shown.

[0096] Embodiment 1:

[0097] Large data centers usually have very large heterogeneous computing clusters that can accommodate hundreds or thousands of computing nodes. A very large number of trained neural network models are deployed in them. There is a very wide user group outside the cluster using various DNN applications to send DNN inference task requests to the data center for inference. These tasks correspond to different inference models and allowed inference time delays. DNN task inference scheduling in this scenario includes the following three processes: cluster DNN inference request collection, DNN inference task initialization, and computing node DNN inference. The DNN-BS scheduling optimization method mainly acts on how to schedule each task to computing nodes with different computing power after the DNN inference tasks arrive at the cluster. DNN-BS can perform dynamic hardware processing capability analysis according to the current cluster heterogeneous computing node partitioning, and find the most suitable task-computing node matching result by running the DNN-BS scheduling algorithm.

[0098] Embodiment 2:

[0099] With the development of technologies such as the Internet of Things, artificial intelligence, and intelligent embedded devices, it is no longer necessary to deploy only a few large data centers to handle all DNN inference tasks. Instead, multiple small processing clusters are deployed in places where network edge devices are concentrated. In this case, local range DNN inference task scheduling needs to be performed on small processing clusters at the network edge.

[0100] For such scenarios, the DNN-BS scheduling method can analyze all DNN applications at the network edge, first classify the DNN inference tasks, and then divide all computing nodes in the small edge cluster to obtain a dynamic hardware processing capability matrix. When the local DNN inference tasks arrive at the small processing cluster, the weight parameters in the optimization objective function can be adjusted according to the scheduling target, and the complete DNN inference task scheduling process can continue.

[0101] Embodiment 3:

[0102] Smart homes also face such problems. In a smart home, there is usually a processing center that processes data from all sensors. After the data arrives at the processing center, it is converted into DNN inference tasks. There are usually more than one hardware processing unit in the processing center. Therefore, all types of data analysis in the smart home can be classified into a limited number of DNN inference task categories, and multiple hardware processing units in the processing center can be used for inference.

[0103] In this scenario, the DNN-BS scheduling optimization method can be deployed on the scheduling node of the home small processing center, and the dynamic hardware processing capability matrix is obtained by obtaining the inference time of all category DNN inference tasks on each hardware processing unit. Then all DNN inference tasks can be scheduled to the corresponding hardware processing unit for inference through the remaining DNN-BS scheduling algorithm process.

[0104] It should be noted that the above only describes the preferred embodiments of the present application and the principles of the technology used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A batch scheduling method for DNN inference tasks in heterogeneous clusters, characterized in that, include: The module consists of a DNN inference task representation module, a heterogeneous cluster dynamic hardware processing capability extraction module, and a DNN inference task batch scheduling module. The DNN inference task representation module is used to classify DNN inference tasks according to task type to obtain a DNN inference task partition set. The heterogeneous cluster dynamic hardware processing capability extraction module is used to divide the computing nodes and construct a heterogeneous computing node set based on the division results. The DNN inference task batch scheduling module is used to schedule DNN inference tasks based on the DNN inference task partitioning set and the heterogeneous computing node set using a target search algorithm. The DNN inference task batch scheduling module is specifically used to: construct a scheduling method template, which consists of a static policy set and a set of all parameters in the static policy set; Receive task flow and dynamically segment the received task flow according to the short-term characteristics of task scheduling, and use a different scheduling method instance for actual scheduling of each task flow. The dynamic parameter matrix is ​​constructed from the dynamic parameter vectors of all scheduling method instances in the entire task flow. The value of the dynamic parameter matrix is ​​searched by a searcher that combines the optimization objective and dynamic hardware processing capabilities. When the search process converges, the optimal dynamic parameter matrix is ​​obtained. The optimal scheduling result is generated based on the scheduling instance methods of each segment of the task flow and the optimal values ​​of the dynamic parameter matrix.

2. The method according to claim 1, characterized in that, The DNN inference task representation module includes: a cluster deployment model statistician, which is used to count the DNN models that can be inferred on each node in the cluster and determine the list of DNN models. The cluster external DNN application statistics tool is used to count the set of external DNN applications corresponding to all inference-enabled DNN models in the cluster. A DNN inference task partitioner is used to determine a set of DNN inference task partitions based on the list of DNN models and the set of external DNN applications.

3. The method according to claim 2, characterized in that, The DNN inference task partitioning set includes: all task types of DNN inference, the allowed response time, target model, and target node set for each task type.

4. The method according to claim 1, characterized in that, The heterogeneous cluster dynamic hardware processing capability extraction module includes: Cluster computing node partitioner is used to dynamically partition the boards in the cluster according to computing power to obtain a heterogeneous computing node set; The cluster inference task processing timer is used to divide the DNN inference task into a set and a heterogeneous computing node set. By pre-executing each type of DNN inference task on each processable heterogeneous computing node, the processing time of all processes is calculated. Based on the processing time results, a dynamic hardware processing capability matrix is ​​generated and the matrix is ​​sent to the DNN inference task batch scheduling module.

5. The method according to claim 1, characterized in that, A dimensionality reduction method based on static strategy and dynamic parameters is used to construct a dynamic parameter matrix representing the scheduling result; Accordingly, the DNN inference task batch scheduling module includes: The task scheduler is used to segment the input task flow in the DNN inference task partitioning set and generate the corresponding scheduling result flow. The scheduling result evaluator is used to evaluate each scheduling result in the scheduling result stream according to the evaluation function; the searcher is used to search in the dynamic parameter matrix using a metaheuristic algorithm, and the solution after the metaheuristic algorithm converges is taken as the optimal scheduling solution.

6. The method according to claim 5, characterized in that, The evaluation function is: Where σ1 is the average completion time of the task, σ2 is the average timeout of all tasks, σ3 is the load balancing degree, ξ is the scheduling result, and λ1, λ2 and λ3 are three different weights.

Images