A kernel function-based neural network model inference delay prediction method

Abstract

The application discloses a neural network model inference time delay prediction method based on a kernel function, and belongs to the field of deep learning. The application uses kernel function information to predict inference time delay, collects various parameters of the kernel function decomposed by neural network model inference, fits a parameter change curve of the kernel function with the amount of resource allocation of a graphic processing unit, and obtains a single model inference time delay prediction result through the fitted curve; and a four-dimensional surface related to inference time and time delay growth degree is constructed for each type of neural network model, and a time delay prediction result of the model coexisting with other models is obtained through the surface. The application solves the problem that the time delay is difficult to predict due to the multiple types of models and complex combination during multi-neural network execution, has high prediction accuracy, is more lightweight, and can improve the inference time delay prediction performance of a cloud server.

Description

TECHNICAL FIELD

[0001] The present application relates to a neural network model inference delay prediction method, in particular to a neural network model inference delay prediction method based on a kernel function, and belongs to the field of deep learning. BACKGROUND

[0002] With the proliferation of artificial intelligence applications, deep neural network inference workloads have become increasingly common in cloud servers. Due to the complexity of the structure of deep neural network models, a large amount of computing and memory resources are consumed during inference, so more and more Internet companies choose to use GPUs to perform inference to reduce inference delay and meet service targets. Since the utilization rate of a single inference task for a GPU is not high, in order to save costs, cloud servers will execute multiple inference tasks on the same GPU, which inevitably leads to an increase in inference delay due to the competition for GPU resources among multiple tasks. In order to improve this problem, Nvidia proposes a multi-process service technology, which allocates a specific proportion of streaming multiprocessor (SM) partitions to each inference task to reduce resource contention between processes. However, due to reasons such as the sharing of resources such as L2 cache and memory within the GPU by all processes, when multiple inference tasks are executed simultaneously, there will still be an increase in inference delay due to resource contention to some extent, which will further cause some tasks to fail to meet service targets. Therefore, accurately predicting the delay of multiple neural network model coexistence inference is the key to improving GPU utilization and ensuring service quality.

[0003] In view of the above problems, there are methods that use machine learning algorithms such as random forests for prediction. First, test the delay of model execution in pairs, and then use these delay data to train the model as a delay predictor for the coexistence of neural network models in pairs. Since the inference delay of a neural network model is affected by factors such as batch size and the amount of allocated resources, this method needs to test the delay under multiple combinations, and the number of tests increases exponentially with the number of models to be predicted, which has poor scalability. To alleviate this situation, a lighter prediction method has emerged, which determines the delay increase during coexistence through information such as L2 cache usage rate. Although it does not require a large amount of training data set, it still needs to execute multiple times for each new structure of neural network model to obtain the necessary information for prediction, and the prediction accuracy of multiple model coexistence inference is not high.

[0004] In summary, in order to more accurately predict the delay of various neural network model inference tasks executed simultaneously, a prediction method that is lighter, has strong scalability and is easier to deploy in cloud servers needs to be found. SUMMARY

[0005] The purpose of the present invention is to provide a neural network model inference delay prediction method based on kernel function, which decomposes neural network models of different structures as basic kernel functions, performs prediction based on basic kernel functions, enhances scalability through dimensionality reduction, and solves the problem of difficult prediction of delay caused by the variety of models and complex combinations when cloud servers execute neural network model inference, realizes the prediction of cloud server model inference delay, and improves the service quality of cloud server inference tasks.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] The present invention provides a method for predicting the time delay of a neural network model inference based on a kernel function, which specifically comprises the following steps:

[0008] Step 1: Obtain analysis data when the kernel function corresponding to the neural network model is executed;

[0009] Set different batches and allocated GPU resources for neural network model inference, and obtain the parameters of the kernel function execution in each case, including: the number of low-level assembly instructions N that will be decomposed when the kernel function is executed ins , the execution speed S of low-level assembly instructions during the execution of the kernel function ins , GPU clock rate S during kernel function execution GPU , GPU SM utilization during kernel function execution r sm , the number of cycles C that the GPU experiences during the execution of the kernel function GPU , the number of cycles C during which the SM is active during the execution of the kernel function sm ,during the execution of kernel function, the data exchange amount b1 between L1 cache and L2 cache, and the data exchange amount b2 between L2 cache and GPU memory;

[0010] Step 2: Fit the changing curve of kernel function parameters under different GPU resources;

[0011] The kernel function parameters of the fitting include: the execution speed of the instruction S ins , GPU clock rate S GPU , GPU SM utilization r sm and the number of cycles C that the GPU goes through GPU The number of cycles C during which the SM is active sm The product of the ratio α and the SM utilization rate α·r sm ;

[0012] As a preference, a linear function is used to fit the execution speed S of the instruction. ins , GPU clock rate S GPU And the product of the ratio of the total number of GPU cycles to the number of SM active cycles α and the SM utilization α·rsm Change curve under different GPU resources;

[0013] As a preference, an inverse proportional function is used to fit the GPU SM utilization r sm Change curve under different GPU resources;

[0014] As a preference, both of the above two fitting forms use nonlinear least squares method for actual fitting to obtain the function curve closest to the sample point. Finally, each kernel function will obtain four fitting curves.

[0015] Step 3: Build a single-model inference latency prediction model;

[0016] Determine all kernel function types that will be decomposed during model inference;

[0017] According to the amount of GPU resources allocated when the model is executed, the predicted value of each parameter of each decomposed kernel function is determined by fitting the kernel function parameter curve determined in step 2.

[0018] Determine the sum of the actual execution time T of all kernel functions decomposed during the single model inference process exec , as shown in formula (1):

[0019]

[0020] Among them, n is the number of kernel functions decomposed by the inference model, is the kernel function K i The number of low-level assembly instructions decomposed, Is the execution kernel function K i The GPU's instruction execution speed, Is the execution kernel function K i The GPU clock rate is C, which is a constant obtained by dividing the number of instructions of any kernel function by the instruction execution speed. i It's K i Total GPU cycles Number of SM active cycles The ratio of

[0021] By analyzing the kernel function type, And α is the kernel function K determined by step 2 i The fitting curve is obtained;

[0022] The total scheduling delay of the kernel function T sche It is related to the number of kernel functions, GPU resources and batch size, as shown in formula (2):

[0023]

[0024] Among them, t sche is the scheduling time of a single kernel function, bs is the batch size used for model inference, r gpu is the percentage of GPU resources allocated for model inference;

[0025] The single model reasoning delay includes the execution delay of the kernel function and the scheduling delay of the kernel function. The single model reasoning delay T infer As shown in formula (3):

[0026] T infer =T exec +T sche (3)

[0027] Step 4: Build a latency prediction model for multi-model coexistence reasoning;

[0028] Construct a latency prediction model for multi-model coexistence reasoning related to L2 cache and GPU memory, as shown in formula (4):

[0029]

[0030] Where P is the percentage increase in the latency prediction model of multi-model coexistence reasoning compared to the latency of a single model, and B predict as well as are the sum of the data exchange between the L1 cache and L2 cache of all kernel functions and the data exchange between the L2 cache and GPU memory during the independent execution of the neural network model to be predicted and the j-th coexisting neural network model, respectively. It is obtained by accumulating b1 and b2 of each kernel function obtained in step 1. M is the sum of the number of coexisting models other than the neural network model to be predicted. k1, k2, k3, k4 and C1 are parameters in the latency prediction model for multi-model coexistence reasoning;

[0031] Determining the parameters of the latency prediction model for multi-model coexistence reasoning includes the following substeps:

[0032] Step 4.1 collects the latency increase percentage P of the multi-model co-existence inference latency prediction model compared to running it alone, for the neural network model to be predicted and other neural network models under different batch sizes and GPU resource allocations;

[0033] Step 4.2 uses the data fitting method to determine the parameters k1, k2, k3, k4, and C1 in the delay prediction model of multi-model coexistence reasoning, completing the construction of the delay prediction model of multi-model coexistence reasoning;

[0034] As a preference, a nonlinear least squares fitting method is used to fit the P, B predict 、B co-located and rgpu The four-dimensional surface composed of the above parameters determines the five parameters k1, k2, k3, k4 and C1 in the delay prediction model of multi-model coexistence reasoning;

[0035] Step 5: Predict actual latency:

[0036] Deploy the single-model reasoning latency prediction model constructed in step 3 and the multi-model coexistence reasoning latency prediction model constructed in step 4 to the cloud server;

[0037] For the delay prediction of single model reasoning, obtain the kernel function type of the model to be predicted, use the delay prediction model of single model reasoning constructed in step 3, and obtain the delay prediction result of single model reasoning through formula (1), formula (2) and formula (3);

[0038] For the delay prediction of multi-model coexistence reasoning, obtain the kernel function type of each model to be predicted, use the delay prediction model of multi-model coexistence reasoning constructed in step 4, and use formula (4) to obtain the delay increase percentage of the model to be predicted under multi-model coexistence reasoning compared with the delay when running alone;

[0039] The coexistence reasoning delay prediction value is obtained by combining the delay prediction results of single-model reasoning and the delay increase percentage of the model to be predicted under multi-model coexistence reasoning compared to when it runs alone.

[0040] Beneficial effects

[0041] 1. The present invention provides a method for predicting the inference delay of a neural network model based on a kernel function. The method uses the delay of executing the underlying kernel function decomposed by the GPU under different GPU resources during the neural network model inference to predict the delay of the entire inference process. This method can improve the prediction accuracy of the neural network model inference delay and improve the quality of the cloud server inference service.

[0042] 2. The present invention provides a method for predicting the inference delay of a neural network model based on kernel functions. This method collects the values ​​of various kernel functions under different GPU resources in advance and fits their change curves. By predicting the delays of all kernel functions of a neural network model with any new structure, the inference delay of the entire network can be predicted, reducing the number of data sets that need to be prepared in advance. The method can be quickly deployed on any new type of GPU and has high versatility.

[0043] 3. The present invention provides a method for predicting the delay of neural network model inference based on kernel functions. It uses data such as the data exchange volume related to the L2 cache during the execution of the kernel function to lightweightly obtain the delay increase caused by the coexistence of multiple neural network models in reasoning. It uses the L2 cache data exchange volume for dimensionality reduction and can quickly obtain the prediction results of coexistence reasoning of multiple models of any type. It has strong scalability and is more suitable for actual reasoning services of cloud servers. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of a method for predicting time delay of a neural network model inference based on a kernel function according to the present invention;

[0045] Figure 2 This is a schematic diagram of the parameter fitting results of a certain type of kernel function in step 2 of a kernel function-based neural network model inference delay prediction method of the present invention;

[0046] Figure 3 This is a schematic diagram of the results of constructing a delay prediction model for multi-model coexistence reasoning in a kernel function-based neural network model reasoning delay prediction method of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be described in detail below with reference to the accompanying drawings and embodiments, and the technical problems solved by the technical solution of the present invention and the beneficial effects thereof will be discussed. It should be noted that the described embodiments are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0048] Example 1:

[0049] This embodiment discusses the application of a kernel function-based neural network model inference delay prediction method of the present invention in a scenario where four inference tasks are simultaneously executed on a GPU. Four classic neural network models are used as inference tasks received by a cloud server: ResNet50, VGG19, DenseNet201, and MobileNet_v2. The GPU model for performing inference is Nvidia RTX 2080Ti. Figure 1 The specific implementation steps are as follows:

[0050] Step 1: Obtain analysis data when the kernel function corresponding to the neural network model is executed;

[0051] A kernel function is a function code executed on the GPU. Each kernel function is executed simultaneously by multiple threads and has a large number of parameters. These parameter values ​​change with the amount of GPU resources allocated to the neural network model. Therefore, in order to more accurately predict the inference latency of various neural network models under different GPU allocations, it is necessary to analyze the kernel function execution of the possible predictive models under different batch sizes and different GPU resource conditions.

[0052] Set different batches and allocated GPU resources for neural network model inference, and obtain the parameters of the kernel function execution in each case, including: the number of low-level assembly instructions N that will be decomposed when the kernel function is executed ins , the execution speed S of the low-level assembly instructions decomposed when the kernel function is executedins , GPU clock rate S during kernel execution GpU , SM utilization rate r during kernel function execution sm , the number of cycles C that the GPU experiences during the kernel function execution process GPU , the number of cycles C during which the kernel function executes the SM sm , the data exchange amount b1 between L1 cache and L2 cache during kernel function execution, and the data exchange amount b2 between L2 cache and GPU memory during kernel function execution;

[0053] In the embodiment, the batch size is set to 1, 8, 16, and 32 respectively. For each batch size, the amount of GPU resources that can be used during inference is set to 10%, 25%, 50%, and 75% respectively, for a total of 16 combinations. The kernel function execution analysis of the 16 configurations of each model is completed to obtain the N under each configuration. ins , S ins , S CPU , r sm , C GPU , C sm , b1 and b2 eight parameter data;

[0054] Step 2: Fit the changing curve of kernel function parameters under different GPU resources;

[0055] The kernel function parameters of the fitting include: the execution speed of the instruction S ins , GPU clock rate S GpU , GPU SM utilization r sm and the number of cycles C that the GPU goes through GPU The number of cycles C during which the SM is active sm The product of the ratio α and the SM utilization rate α·r sm ;

[0056] In the embodiment, S ins 、S GPU and α·r sm Use a linear function to fit r sm The inverse proportional function form is used for fitting, and the fitting method uses the nonlinear least square method;

[0057] The fitting results of the four parameters of one kernel function are as follows Figure 2 As shown in the figure, after fitting is completed, for any type of kernel function in these four neural network models, the corresponding parameter value can be obtained by substituting the current GPU resource allocation;

[0058] Step 3: Build a single-model inference latency prediction model;

[0059] Determine all kernel function types that will be decomposed during model inference;

[0060] According to the amount of GPU resources allocated when the model is executed, the predicted value of each parameter of each decomposed kernel function is determined by fitting the kernel function parameter curve determined in step 2.

[0061] Determine the sum of the actual execution time T of all kernel functions decomposed during the single model inference process exec , as shown in formula (1):

[0062]

[0063] Among them, n is the number of kernel functions decomposed by the inference model, is the kernel function K i The number of low-level assembly instructions decomposed, Is the execution kernel function K i The GPU's instruction execution speed, Is the execution kernel function K i The GPU clock rate is C, which is a constant obtained by dividing the number of instructions of any kernel function by the instruction execution speed. i It's K i Total GPU cycles Number of SM active cycles The ratio of

[0064] In the embodiment, C is approximately 67.9 on Nvidia RTX 2080Ti;

[0065] By analyzing the kernel function type, And α is the kernel function K determined by step 2 i The fitting curve is obtained;

[0066] The total scheduling delay of the kernel function T sche It is related to the number of kernel functions, GPU resources and batch size, as shown in formula (2):

[0067]

[0068] Among them, t sche is the scheduling time of a single kernel function, bs is the batch size used for model inference, r gpu is the percentage of GPU resources allocated for model inference;

[0069] In the embodiment, for Nvidia RTX 2080Ti, the scheduling time of a single kernel function is t sche The time is 7us, which can be analyzed by Nsight System tools on any type of GPU.

[0070] The single model reasoning delay includes the execution delay of the kernel function and the scheduling delay of the kernel function. The single model reasoning delay T infer As shown in formula (3):

[0071] T infer =T exec +T sche (3)

[0072] In this embodiment, in order to evaluate the accuracy of the latency prediction model of single-model inference, for each neural network model, the batch size is set to 1, 8, 16, and 32 respectively. For each batch, the GPU allocation range is set to 10% to 100%, with each 10% as an interval, for a total of 40 sample points;

[0073] We tested the actual inference latency of 40 sample points for each neural network model, obtained the mean absolute error between the predicted and actual results, and compared them with the most advanced prediction method based on L2 cache usage. The results are shown in Table 1:

[0074] Table 1

[0075] Mean absolute error (ms) ResNet50 VGG19 DenseNet201 MobileNet_v2 Method of the present invention 1.6960 4.7971 2.6324 0.9293 Method based on L2 cache usage 2.9814 8.5877 4.5164 1.7940

[0076] The average absolute error percentage between the predicted results and the actual results is shown in Table 2:

[0077] Table 2

[0078] Mean absolute error percentage (%) ResNet50 VGG19 DenseNet201 MobileNet_v2 Method of the present invention 4.59 3.94 2.62 7.65 Method based on L2 cache usage 8.15 10.41 5.74 14.33

[0079] As shown in Tables 1 and 2, the prediction accuracy of the method of the present invention is higher. In terms of the mean absolute error of the prediction results, the average prediction error of the four networks is all less than 5 milliseconds, and the prediction error of MobileNet_v2 is only 0.9293 milliseconds. In terms of the average error percentage of the prediction results, the prediction error of the three networks is less than 5%, and the average error percentage of the most accurate DenseNet201 network is only 2.62%, achieving extremely high accuracy.

[0080] Step 4: Build a latency prediction model for multi-model coexistence reasoning;

[0081] Construct a latency prediction model for multi-model coexistence reasoning related to L2 cache and GPU memory, as shown in formula (4):

[0082]

[0083] Where P is the percentage increase in the latency prediction model of multi-model coexistence reasoning compared to the latency of a single model, and B predict as well as are the sum of the data exchange between the L1 cache and the L2 cache of all kernel functions during the separate execution of the neural network model to be predicted and the j-th coexisting neural network model, and the data exchange between the L2 cache and the GPU memory, which is obtained by accumulating b1 and b2 of each kernel function obtained in step 1. M is the sum of the number of other coexisting models except the neural network model to be predicted. In this embodiment, M=1, k1, k2, k3, k4 and C1 are parameters in the delay prediction model of multi-model coexistence reasoning;

[0084] In the embodiment, four-dimensional surfaces represented by Formula 8 are established for the four neural network models respectively;

[0085] Determining the parameters of the latency prediction model for multi-model coexistence reasoning includes the following substeps:

[0086] Step 4.1 Collect the latency prediction model of the neural network model to be predicted and other neural network models under different batches and GPU resource allocations. The latency increase percentage P of the multi-model co-existence inference compared to running alone:

[0087] In the embodiment, the following sub-steps are adopted:

[0088] 4.1.1 Select any model from the four models and set its batch size to 1, 8, or 16. For each batch size, set its GPU allocation to 10%, 25%, 50%, or 75% respectively.

[0089] 4.1.2 Select another model and set its batch size to 1, 8, or 16. For each batch size, set the GPU allocation to 10%, 25%, 50%, or 75%. This will generate multiple combinations of "Model A, Batch Size A, GPU Allocation Size A" and "Model B, Batch Size B, GPU Allocation Size B," ensuring that the GPU resources allocated to both models are less than or equal to 100%.

[0090] 4.1.3 Execute all combinations and collect the latency increase data for each model during model coexistence reasoning;

[0091] Step 4.2 uses the data fitting method to determine the parameters k1, k2, k3, k4, and C1 in the delay prediction model of multi-model coexistence reasoning, completing the construction of the delay prediction model of multi-model coexistence reasoning:

[0092] In the embodiment, the following sub-steps are adopted:

[0093] 4.2.1 After obtaining the delay growth data of the model coexistence inference of each combination, obtain B for each combination result. predict 、 r gpu . B predictis the total data exchange volume related to the L2 cache of the model to be predicted, which can be obtained by summing b1 and b2 of each kernel function of the model collected in step 1; r gpu is the GPU allocation of the model to be predicted, which is the known data; is the total data exchange volume related to the L2 cache of other models running simultaneously with the prediction model, which is also obtained from the kernel function information of the model collected in step 1. In this embodiment, M=1;

[0094] 4.2.2 Substitute the latency increase percentage P during model coexistence inference and the above three parameters into the latency prediction model for multi-model coexistence inference, and fit four-dimensional surfaces to the four neural network models. Preferably, in this embodiment, nonlinear least squares method is used for fitting.

[0095] Finally, the five unknown parameter values ​​of the surface corresponding to each model are obtained, as shown in Table 3:

[0096] Table 3

[0097] Neural Network / Parameters <![CDATA[k1]]> <![CDATA[k2]]> <![CDATA[k3]]> <![CDATA[k4]]> <![CDATA[C1]]> ResNet50 0.01097 30.03279 7.49720 0.17349 54.57114 DenseNet201 0.00051 0.93592 1.98348 2.39934 172.82499 VGG19 -0.00328 11.59961 2.11501 0.09744 -8.02221 MobileNet_v2 0.00013 0.19409 0.22946 1.56454 3.36774

[0098] Each model's corresponding surface contains three independent variables: the total data exchange associated with the predicted model's L2 cache, the total data exchange associated with the L2 cache of other coexisting models, and the GPU resource allocation of the predicted model; and one dependent variable: the percentage increase in latency of the predicted model. Each model's corresponding surface is a four-dimensional surface and cannot be represented by a three-dimensional image. Fixing the GPU resource allocation to 50% reduces the image dimension to three dimensions, as shown in the following example: Figure 3 As shown in the figure, the unit of data exchange volume is ten million bytes. Given a given total data exchange volume associated with the prediction model's L2 cache, the greater the total data exchange volume associated with the L2 cache of other coexisting models, the more affected the prediction model will be. In other words, the more resources a coexisting model occupies, the slower the prediction model's inference will be, manifested as a greater increase in latency.

[0099] Step 5: Predict actual latency:

[0100] Deploy the single-model reasoning latency prediction model constructed in step 3 and the multi-model coexistence reasoning latency prediction model constructed in step 4 to the cloud server;

[0101] For the delay prediction of single model reasoning, obtain the kernel function type of the model to be predicted, use the delay prediction model of single model reasoning constructed in step 3, and obtain the delay prediction result of single model reasoning through formula (1), formula (2) and formula (3);

[0102] For the delay prediction of multi-model coexistence reasoning, obtain the kernel function type of each model to be predicted, use the delay prediction model of multi-model coexistence reasoning constructed in step 4, and use formula (4) to obtain the delay increase percentage of the model to be predicted under multi-model coexistence reasoning compared with the delay when running alone;

[0103] The coexistence reasoning delay prediction value is obtained by combining the delay prediction results of single-model reasoning and the delay increase percentage of the model to be predicted under multi-model coexistence reasoning compared to when it runs alone.

[0104] In the embodiment, the coexistence of any two models, three models, and four models of the four neural network models is tested respectively;

[0105] Set the batch size of each neural network model to 1, 8, and 16 respectively. For each batch, set the GPU allocation to 10%, 25%, 50%, and 75% respectively. Each neural network model has 12 sample points;

[0106] When testing coexistence inference, we selected any two models and performed a Cartesian product of the sample points to obtain 144 inference configurations. We removed cases where GPU resource allocation was greater than 100%, then actually executed each inference configuration and collected the actual results for comparison with the predicted values. We obtained the mean absolute error of the prediction error percentage for all sample points coexisting with two, three, and four models, as shown in Table 4:

[0107] Table 4

[0108] Mean absolute error percentage (%) Coexistence reasoning of two models Three-model coexistence reasoning Four-model coexistence reasoning Kernel-based methods 4.91 6.65 9.38 Method based on L2 cache usage 12.77 13.41 22.11

[0109] As shown in Table 4, the four-dimensional surface fitted by the delay growth data when the two models coexist is able to well predict the delay growth percentage when multi-model coexistence reasoning is used, with the average error percentage of all cases being less than 10%.

[0110] In terms of accuracy, compared with existing prediction methods such as machine learning training and prediction and L2 cache utilization, the kernel function-based neural network model inference delay prediction method proposed in this invention is more accurate;

[0111] In terms of time complexity and scalability, the kernel function-based neural network model inference delay prediction method proposed in this invention only needs to collect the kernel function information and the delay increase when coexisting for any new model. The time complexity is linear, which is O(n), and it has strong scalability. The method based on L2 cache usage has a time complexity of O(n) and strong scalability, but its accuracy is much lower than this method. The time complexity of the data set acquisition of the machine learning training and prediction method is O(2 n), the time complexity is too high, it is difficult to scale, can only be used for a small number of model predictions, and cannot be deployed on the cloud;

[0112] Therefore, the kernel function-based neural network model inference delay prediction method proposed in the present invention is not only more accurate, lighter and more scalable, but also provides an effective solution for the neural network model inference delay prediction on cloud servers by obtaining the delay growth of any model when multiple models coexist in a lighter way.

[0113] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for predicting delay inference of a neural network model based on a kernel function, characterized by: By decomposing neural network models of different structures into basic kernel functions and performing predictions based on these basic kernel functions, and enhancing scalability through dimensionality reduction, this approach addresses the difficulty in predicting latency caused by the large variety of models and complex combinations when executing neural network model inference on cloud servers. This approach enables prediction of model inference latency on cloud servers and improves the service quality of inference tasks on cloud servers. The approach specifically includes the following steps: Step 1: Obtain analysis data when the kernel function corresponding to the neural network model is executed; Set different batches and allocated graphics processing unit (GPU) resources for neural network model inference, and obtain the parameters of the kernel function execution in each case, including: the number of low-level assembly instructions N that will be decomposed when the kernel function is executed ins , the execution speed S of low-level assembly instructions during the execution of the kernel function ins , GPU clock rate S during kernel function execution GPU , GPU streaming multiprocessor SM (StreamingMultiprocessor) utilization during kernel function execution r sm , the number of cycles C that the GPU experiences during the execution of the kernel function GPU , the number of cycles C during which the SM is active during the execution of the kernel function sm ,during the execution of kernel function, the data exchange amount b1 between L1 cache and L2 cache, and the data exchange amount b2 between L2 cache and GPU memory; Step 2: Fit the changing curve of kernel function parameters under different GPU resources; The kernel function parameters of the fitting include: the execution speed of the instruction S ins , GPU clock rate S GPU , GPU SM utilization r sm and the number of cycles C that the GPU goes through GPU The number of cycles C during which the SM is active sm The product of the ratio α and the SM utilization rate α·r sm ; Step 3: Build a single-model inference latency prediction model; Determine all kernel function types that will be decomposed during model inference; According to the amount of GPU resources allocated when the model is executed, the predicted value of each parameter of each decomposed kernel function is determined by fitting the kernel function parameter curve determined in step 2. Determine the sum of the actual execution time T of all kernel functions decomposed during the single model inference process exec , as shown in formula (1): Among them, n is the number of kernel functions decomposed by the inference model, is the kernel function K i The number of low-level assembly instructions decomposed, Is the execution kernel function K i The GPU's instruction execution speed, Is the execution kernel function K i The GPU clock rate is C, which is a constant obtained by dividing the number of instructions of any kernel function by the instruction execution speed. i It's K i Total GPU cycles Number of SM active cycles The ratio of By analyzing the kernel function type, And α is the kernel function K determined by step 2 i The fitting curve is obtained; The total scheduling delay T of the kernel function sche It is related to the number of kernel functions, GPU resources and batch size, as shown in formula (2): Among them, t sche is the scheduling time of a single kernel function, bs is the batch size used for model inference, r gpu is the percentage of GPU resources allocated for model inference; The single model reasoning delay includes the execution delay of the kernel function and the scheduling delay of the kernel function. The single model reasoning delay T infer As shown in formula (3): T infer =T exeC +T sche (3) Step 4: Build a latency prediction model for multi-model coexistence reasoning; Construct a latency prediction model for multi-model coexistence reasoning related to L2 cache and GPU memory, as shown in formula (4): Where P is the percentage increase in the latency prediction model of multi-model coexistence reasoning compared to the latency of a single model, and B predict as well as are the sum of the data exchange between the L1 cache and L2 cache of all kernel functions and the data exchange between the L2 cache and GPU memory during the independent execution of the neural network model to be predicted and the j-th coexisting neural network model, respectively. It is obtained by accumulating b1 and b2 of each kernel function obtained in step 1. M is the sum of the number of coexisting models other than the neural network model to be predicted. k1, k2, k3, k4 and C1 are parameters in the latency prediction model for multi-model coexistence reasoning; Determining the parameters of the latency prediction model for multi-model coexistence reasoning includes the following substeps: Step 4.1 collects the latency increase percentage P of the multi-model co-existence inference of the neural network model to be predicted and other neural network models under different batch sizes and GPU resource allocations compared to the latency of running them alone; Step 4.2 uses the data fitting method to determine the parameters k1, k2, k3, k4, and C1 in the delay prediction model of multi-model coexistence reasoning, completing the construction of the delay prediction model of multi-model coexistence reasoning; Step 5: Predict actual latency: Deploy the single-model reasoning latency prediction model constructed in step 3 and the multi-model coexistence reasoning latency prediction model constructed in step 4 to the cloud server; For the delay prediction of single model reasoning, obtain the kernel function type of the model to be predicted, use the delay prediction model of single model reasoning constructed in step 3, and obtain the delay prediction result of single model reasoning through formula (1), formula (2) and formula (3); For the delay prediction of multi-model coexistence reasoning, obtain the kernel function type of each model to be predicted, use the delay prediction model of multi-model coexistence reasoning constructed in step 4, and use formula (4) to obtain the delay increase percentage of the model to be predicted under multi-model coexistence reasoning compared with the delay when running alone; The coexistence reasoning delay prediction value is obtained by combining the delay prediction results of single-model reasoning and the delay increase percentage of the model to be predicted under multi-model coexistence reasoning compared to when it runs alone.

2. The method for predicting delay of a neural network model inference based on a kernel function according to claim 1, wherein: In step 2, a linear function is used to fit the execution speed S of the instruction. ins , GPU clock rate S GPU And the product of the ratio of the total number of GPU cycles to the number of SM active cycles α and the SM utilization α·r sm Change curve under different GPU resources; Use the inverse proportional function to fit the GPU SM utilization r sm Change curve under different GPU resources; Both of the above fitting forms use nonlinear least squares method for actual fitting to obtain the function curve closest to the sample point. In the end, each kernel function will obtain four fitting curves.

3. The method for predicting delay of a neural network model inference based on a kernel function according to claim 1, wherein: In step 4.2, the nonlinear least squares fitting method is used to fit the P, B predict 、B co-located and r gpu The four-dimensional surface composed of the five parameters k1, k2, k3, k4 and C1 in the delay prediction model of multi-model coexistence reasoning is determined.

Images