Deep neural network model reasoning delay prediction method suitable for heterogeneous hardware

Abstract

The invention relates to a deep neural network model reasoning delay prediction method suitable for heterogeneous hardware. The method comprises the following steps: constructing a general calculation graph of a target deep neural network model; creating a directed acyclic graph of a target deep neural network model according to the general calculation graph; extracting static features of the model according to the general calculation graph, and extracting hardware features through the current hardware parameters; and constructing a model reasoning delay prediction model by using the directed acyclic graph, the model static features and the hardware features. The beneficial effects of the invention are that the method can accurately predict the reasoning delay of the deep neural network on heterogeneous hardware, enables an enterprise to optimize the model deployment, and remarkably reduces the waste of computing resources and the operation cost. Meanwhile, hardware resource scheduling can be optimized through the accurate deep neural network model reasoning delay prediction, the hardware utilization rate is improved, and equipment purchase and maintenance expenses are reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and more specifically, to a method for predicting inference delay of a deep neural network model applicable to heterogeneous hardware. Background Art

[0002] With the widespread application of deep neural networks in multiple fields such as image recognition, natural language processing, and autonomous driving, the efficient deployment and real-time reasoning performance of models have become a hot topic of research. However, the reasoning delay of deep neural network models on different hardware platforms has significant differences, especially in heterogeneous hardware environments (such as CPU, GPU, FPGA, etc.). Accurately predicting the reasoning delay of deep neural network models on target hardware is of great significance for optimizing model structure, resource scheduling, and improving user experience. Existing methods for predicting the reasoning delay of deep neural network models mainly focus on a single hardware platform or optimize for a specific model structure. These methods often rely on a large number of empirical rules or statistical methods, and are difficult to adapt to diverse hardware configurations and complex model architectures. For example, traditional delay prediction methods may only consider macroscopic features such as the number of layers and the number of parameters of the model, while ignoring the specific operation types of each layer and its execution efficiency on different hardware. In addition, with the continuous evolution of DNN model structures, such as the diversification of operations such as convolutional layers, pooling layers, and activation functions, it seems inadequate to rely solely on static features for prediction. Especially in heterogeneous hardware environments, factors such as the computing power, memory bandwidth, and parallel processing capabilities of different hardware have a significant impact on inference latency. Existing methods find it difficult to fully capture and utilize these hardware characteristics, resulting in insufficient prediction accuracy, which limits their promotion and application in practical applications. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies in the prior art and propose a deep neural network model inference delay prediction method suitable for heterogeneous hardware.

[0004] First, a method for predicting the inference delay of a deep neural network model applicable to heterogeneous hardware is provided, including:

[0005] S1, construct a general computational graph for the target deep neural network model;

[0006] S2. Create a directed acyclic graph of the target deep neural network model according to the general computational graph;

[0007] S3, extracting model static features based on the general computational graph, and extracting hardware features through current hardware parameters;

[0008] S4. Utilize the directed acyclic graph, model static features and hardware features to build a model reasoning delay prediction model.

[0009] Preferably, S1 comprises:

[0010] S101, obtaining a target deep neural network model file and parsing it, identifying the types of layers in the model and their interconnected relationships, and calculating the size of the input and output tensors of each operation;

[0011] S102. Traverse each layer of the model in sequence according to the execution order of the model, identify the operation type of each layer of the model, extract the corresponding operation attributes, and record the output tensor size of each operation; construct the nodes and edges of the general computational graph according to the operation type, operation attributes and operation attributes.

[0012] Preferably, S2 comprises:

[0013] S201, constructing an empty node dictionary, and post-order traversing each node in the general computation graph; during the traversal process, assigning a unique code to each node using the number of nodes stored in the current node dictionary;

[0014] S202, sorting all nodes in the node dictionary to conform to the topological structure of the directed acyclic graph; in the sorting process, identifying the function call node, extracting the operation name, operation attribute and output size;

[0015] S203, using a one-hot encoding method to identify the operation type, and combining the operation attributes and output size to construct a node feature vector;

[0016] S204. Using the order of post-order traversal, all node feature vectors are stacked up to form a node feature matrix; and an adjacency matrix is ​​constructed according to the connection relationship between the nodes; and a directed acyclic graph is generated according to the node feature matrix and the adjacency matrix.

[0017] Preferably, S3 includes:

[0018] S301, converting the batch size of the input data in S1 into a feature vector, calculating the number of multiplication and addition operations of each node and accumulating the total amount of operations, and then converting it into an operation amount feature vector;

[0019] S302, traverse all operation nodes in the general computation graph constructed in S1, generate feature vectors of the number of operators of the convolutional layer, activation function and fully connected layer, and concatenate them with the batch size feature vector and the operation amount feature vector to form a static feature of the model;

[0020] S303: Normalize the hardware parameters of the device and combine them to form hardware features.

[0021] Preferably, S4 includes:

[0022] S401, constructing a node-edge encoder module, using a leaky linear rectifier activation function to perform nonlinear transformation on the hidden features of nodes and edges, wherein the edge features are updated through an attention mechanism;

[0023] S402. Construct a three-layer graph neural network layer transfer feature based on the model static features, hardware features, edge features and the directed acyclic graph constructed in S2.

[0024] Preferably, S402 includes:

[0025] S4021. Integrate the model static features and hardware features into the node feature vector to form enhanced node features;

[0026] S4022, transferring and aggregating the features and the directed acyclic graph through the three-layer graph neural network layer;

[0027] S4023, all node features are linearly transformed and nonlinearly activated through three fully connected layers;

[0028] S4024. Output the model through the Sigmoid activation function to predict the time required for inference under the current hardware configuration.

[0029] Preferably, in S4, the training process adopts the Huber loss function.

[0030] In a second aspect, a deep neural network model inference delay prediction system applicable to heterogeneous hardware is provided, which is used to execute any method described in the first aspect, including:

[0031] The first building block is used to build a general computational graph of a target deep neural network model;

[0032] A creation module, used to create a directed acyclic graph of a target deep neural network model according to the general computational graph;

[0033] The extraction module is used to extract the static features of the model based on the general computation graph and extract the hardware features through the current hardware parameters;

[0034] The second building module is used to build a model reasoning delay prediction model by using the directed acyclic graph, model static features and hardware features.

[0035] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0036] In a fourth aspect, an electronic device is provided, including:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute the computer program to implement any method as described in the first aspect.

[0039] The beneficial effects of the present invention are as follows: by accurately predicting the inference delay of deep neural networks on heterogeneous hardware, enterprises can optimize model deployment and significantly reduce the waste of computing resources and operating costs. At the same time, accurate prediction of inference delay of deep neural network models can optimize hardware resource scheduling, improve hardware utilization, and reduce equipment procurement and maintenance expenses. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of a method for predicting delay of a deep neural network model inference applicable to heterogeneous hardware in an embodiment of the present invention;

[0041] Figure 2 A flowchart of constructing a general calculation graph of a model in an embodiment of the present invention;

[0042] Figure 3 A flowchart for creating a directed acyclic graph of a model in an embodiment of the present invention;

[0043] Figure 4 This is a flow chart of extracting model static features and hardware feature vectors in an embodiment of the present invention;

[0044] Figure 5 A flowchart of constructing a model reasoning delay prediction model in an embodiment of the present invention;

[0045] Figure 6 Model architecture diagram designed for the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0047] Embodiment 1:

[0048] To solve the problems of the prior art, Example 1 of the present application provides a method for predicting inference delay of a deep neural network model suitable for heterogeneous hardware. By constructing a general computational graph and a directed acyclic graph, combining the static characteristics and hardware characteristics of the model, and using graph neural networks to perform accurate delay prediction, it helps to improve the deployment efficiency and operating performance of the deep neural network model, and also provides strong support for subsequent model optimization and hardware resource scheduling, and has broad application prospects and research value.

[0049] Specifically, Figure 1 and Figure 6 As shown, the method includes:

[0050] S1. Build a general computational graph for the target deep neural network model.

[0051] like Figure 2 As shown, S1 includes:

[0052] S101. Obtain the target deep neural network model file and parse it, identify the types of layers in the model and their interconnections, and calculate the size of the input and output tensors of each operation.

[0053] Specifically, use the API or tools provided by the target deep learning model framework to load the model file as an intermediate representation scripted model. After the intermediate representation scripted model is loaded, extract the model's input information and construct the model output tensor size in the format of [batch, channel, width, height], where batch represents the number of data samples input into the model for training or inference at one time, channel represents the channel dimension of the data, and width and height represent the horizontal and vertical dimensions of the input data, respectively. Based on the input data shape list inference, by parsing the connection relationship and parameters of each layer in the model, identify the type of each layer in the model and their interconnection relationship, and calculate the size of the input and output tensors of each operation.

[0054] The target deep neural network model is a common convolutional neural network (e.g., ResNet-50), and its model file is stored in ONNX format. When implementing the present invention, for example, the model file is first loaded using the API provided by ONNX. After the model is loaded, the size information of the input tensor is extracted. Assume that the input size of ResNet-50 is [1, 3, 224, 224], that is, batch=1, channel=3, width=224, height=224. Next, the connection relationship and parameters of each layer in the model are parsed. By traversing the graph of the model, the operation type and its parameters of each layer are identified, and the input and output tensor sizes of each operation are calculated.

[0055] S102. Traverse each layer of the model in sequence according to the execution order of the model, identify the operation type of each layer of the model, extract the corresponding operation attributes, and record the output tensor size of each operation; construct the nodes and edges of the general computational graph according to the operation type, operation attributes and operation attributes.

[0056] Specifically, after completing the reading and parsing of the model file and the shape inference, start traversing the various layers and components of the model to generate a general computational graph. The specific operations include traversing each layer in turn according to the execution order of the model, identifying the operation type of each layer, such as convolution, pooling or activation function, and extracting the corresponding operation attributes, including stride, padding, expansion, number of groups, number of channels, kernel size, data layout and kernel layout parameters. At the same time, record the output tensor size of each operation, that is, [batch, channel, width, height] extracted by S101. Based on this model information, construct the nodes and edges of the general computational graph, where each node represents an operation and its attribute information, and the edge represents the data flow path between operations, reflecting the data transfer relationship in the computational graph.

[0057] For example, after completing the reading and parsing of the model file and the shape inference, start traversing the various layers and components of the model to generate a general computational graph. For the ResNet-50 model, the nodes of the general computational graph include convolutional layers, batch normalization layers, activation functions, pooling layers, fully connected layers, etc. The generated general computational graph will contain all operation nodes and their attributes, and reflect the flow path of data between operations. For example, the initial layer of ResNet-50 may include a 7x7 convolutional layer connected to a batch normalization layer, which is then connected to a ReLU activation function, and so on.

[0058] S2. Create a directed acyclic graph of the target deep neural network model based on the general computational graph.

[0059] like Figure 3 As shown, S2 includes:

[0060] S201, construct an empty node dictionary, and traverse each node in the general computation graph in post-order; during the traversal process, a unique code is assigned to each node using the number of nodes stored in the current node dictionary.

[0061] Specifically, an empty node dictionary is created to store all nodes in the graph. Each node in the general computational graph constructed by S1 is traversed using the post-order traversal method. During the traversal process, a unique code is assigned to each node using the number of nodes stored in the current node dictionary. Since the number of stored nodes automatically increases each time a new node is added, the code of each new node is unique and continuous, avoiding the problem of code duplication.

[0062] For example, the first convolutional layer Conv1 of ResNet-50 is traversed and assigned codes in the general computation graph. For example, Conv1 may be assigned code 0 and BatchNorm1 code 1. This ensures that each operation node has a unique and continuous code to avoid code duplication.

[0063] S202, sorting all nodes in the node dictionary to conform to the topological structure of the directed acyclic graph; in the sorting process, identifying the function call node, extracting the operation name, operation attribute and output size.

[0064] Specifically, all nodes in the node dictionary are sorted to ensure that the order of the nodes conforms to the topological structure of the directed acyclic graph. During the sorting process, each node is checked to see if it is a function call node by checking whether the operation type of the node belongs to the three predefined function call categories (convolution, pooling, activation function). For nodes identified as function calls, their operation names and operation attributes are extracted, where the operation name is used for subsequent operation type identification, and the operation attributes include stride, padding, expansion, number of groups, number of channels, core size, data layout and core layout, and the output size is the output tensor size extracted in step S102.

[0065] For example, take the convolution layer Conv1 in ResNet-50 as an example, assuming its operation type is 'Conv', its related attributes are stride = 2, padding = 3, dilation = 1, groups = 1, channels = 64, kernel_size = 7, data_layout = 'NCHW', output_shape = [1, 64, 112, 112]. These attributes will be extracted and stored for use in subsequent steps.

[0066] S203: Use the one-hot encoding method to identify the operation type, and construct a node feature vector by combining the operation attributes and the output size.

[0067] Specifically, the operation type is identified using the One-Hot Encoding method, and a unit matrix is ​​generated as the node's operation attribute vector node_op. At the same time, the extracted operation attributes are constructed as the attribute vector node_attr, and the output size information is constructed as the node shape vector node_shape. Specifically, node_op represents different operation types by converting the operation type into a one-hot encoding vector; node_attr contains various operation attribute parameters extracted from the node; node_shape records the output size of each operation. The final node feature vector node_feature is formed by sequentially concatenating the three vectors node_op, node_attr, and node_shape.

[0068] For example, for the Conv1 node, its node_op is [1,0,0] (meaning 'Conv'), node_attr is [2,3,1,1,64,7], and node_shape is [1,64,112,112]. Therefore, node_feature is [1,0,0,2,3,1,1,64,7,1,64,112,112], which integrates the operation type, attributes, and output size information.

[0069] S204. Using the order of post-order traversal, all node feature vectors are stacked up to form a node feature matrix; and an adjacency matrix is ​​constructed according to the connection relationship between the nodes; and a directed acyclic graph is generated according to the node feature matrix and the adjacency matrix.

[0070] Specifically, all node feature vectors are stacked up in the order of post-order traversal to form a node feature matrix Matrix_node. At the same time, according to the connection relationship between nodes, the directed edge is recorded as [node_i.id, node_j.id], where node_i is the source node and node_j is the target node, indicating the flow path of data between node_i and node_j. Using this edge information, the adjacency matrix Matrix_adj is constructed, where each element is 1 to indicate that there is a connection relationship between nodes, and 0 to indicate that there is no connection relationship. Finally, a directed acyclic graph G containing a node feature matrix and an adjacency matrix is ​​generated.

[0071] For example, assuming that the first two function call nodes of ResNet-50 are Conv1 and BatchNorm1, Matrix_node will contain the feature vectors of these two nodes, and the feature vector length of each node is 13 (3 operation types unique hot encoding + 6 operation attributes + 4 output dimensions). The adjacency matrix Matrix_adj will reflect the connection relationship between nodes, for example, Conv1 is connected to BatchNorm1.

[0072] S3. Extract the static features of the model based on the general computational graph, and extract the hardware features through the current hardware parameters.

[0073] S4. Utilize the directed acyclic graph, model static features and hardware features to build a model reasoning delay prediction model.

[0074] Embodiment 2:

[0075] Based on Example 1, Example 2 of the present application provides a more specific method for predicting the inference delay of a deep neural network model applicable to heterogeneous hardware, including:

[0076] S1. Build a general computational graph for the target deep neural network model.

[0077] S2. Create a directed acyclic graph of the target deep neural network model based on the general computational graph.

[0078] S3. Extract the static features of the model based on the general computational graph, and extract the hardware features through the current hardware parameters.

[0079] like Figure 4 As shown, S3 includes:

[0080] S301, convert the batch size of the input data in S1 into a feature vector, calculate the number of multiplication and addition operations of each node and accumulate the total amount of operations, and then convert it into an operation amount feature vector.

[0081] Specifically, the batch size of the S1 input data is converted into a feature vector F batch , the batch size refers to the number of data samples that are input into the model for training or inference at one time. For each node i, the number of multiplication and addition operations is calculated as batch×channel×width×height, where [batch, channel, width, height] is the output tensor size extracted in step S101. Traverse the entire general computational graph, calculate the number of multiplication and addition operations of each relevant operation node according to the above formula and accumulate them to obtain the total number of multiplication and addition operations of the model, and convert it into billions of floating-point operations per second to obtain the feature vector F MAC .

[0082] For example, take the first two operation nodes of the ResNet-50 model as an example: 1) Conv1 layer: output size: [1,64,112,112]; MAC calculation: 1×64×112×112=802,816; 2) BatchNorm1 layer: output size: [1,64,112,112], MAC calculation: 1×64×112×112=802,816. Combining the MAC calculations of these two layers, the total number of multiplication and addition operations of the model is: 802,816+802,816=1,605,632

[0083] Convert total MACs to GFLOPs: MAC =1605632×10 -9 =0.0016.

[0084] S302, traverse all operation nodes in the general computation graph constructed in S1, generate feature vectors of the number of operators of the convolutional layer, activation function and fully connected layer, and concatenate them with the batch size feature vector and the computation amount feature vector to form static features of the model.

[0085] Specifically, we traverse all the operation nodes in the general computational graph constructed by S1, and for each operation node, we classify and count the number of operators in the convolutional layer, activation function, and fully connected layer, respectively, as feature vectors F conv 、F relu 、F dense The above feature vectors are concatenated to form the aggregated model static feature vector:

[0086]

[0087] Among them, the symbol Represents a vector concatenation operation.

[0088] S303: Normalize the hardware parameters of the device and combine them to form hardware features.

[0089] Specifically, normalize the hardware parameters of the device, including CPU operation frequency (CPU_freq), number of CPU cores (CPU_cores), GPU memory capacity (GPU_mem), number of GPU multiprocessors (GPU_proc), CUDA major version number (CUDA_major) and minor version number (CUDA_minor). After normalization, aggregate the different parameters into a hardware feature vector:

[0090]

[0091] Among them, the symbol Represents a vector concatenation operation.

[0092] Taking the NVIDIA Tesla V100 GPU platform as an example, the hardware feature vector extraction process is shown in Table 1 below:

[0093] Table 1

[0094]

[0095]

[0096] Therefore, the normalized hardware feature vector of the NVIDIA Tesla V100 GPU is:

[0097] F hardware =[0.5,0.4286,0.4286,0.75,0.6667,0]

[0098] S4. Utilize the directed acyclic graph, model static features and hardware features to build a model reasoning delay prediction model.

[0099] like Figure 5 As shown, S4 includes:

[0100] S401. Construct a node-edge encoder module and use a leaky linear rectification activation function to perform nonlinear transformation on the hidden features of nodes and edges, where the edge features are updated through an attention mechanism.

[0101] Specifically, we construct a node-edge encoder module with three learnable parameters, namely the node weight matrix: W node , edge weight matrix: W edge And the attention weight matrix W att The leaky linear rectifier activation function (LeakyReLU) is used to perform nonlinear transformation on the hidden features of nodes and edges and aggregate the node features.

[0102] The calculation formula of node characteristics is:

[0103] node_feature i =LeakyReLU(W node ×node_feature (i-1) )

[0104] Among them, node_feature (i-1) Represents the node features of the i-1th layer. The specific calculation of the LeakyReLU function is as follows:

[0105] LeakyReLU(x)=max(0,x)-α×max(0,-x)

[0106] Among them, α is the negative slope coefficient of the LeakyReLU function, which is used to impart nonlinear characteristics.

[0107] The update formula of edge features is:

[0108]

[0109] Among them, node_feature i (start) represents the feature vector of each edge source node in the directed acyclic graph of the model at the i-th layer, node_feature i (end) represents the feature vector of the destination node of each edge in the directed acyclic graph of the model at the i-th layer, W att is the attention weight matrix, Represents the concatenation operation of the vector, and softmax is used to normalize the attention weights.

[0110] S402. Construct a three-layer graph neural network layer transfer feature based on the model static features, hardware features, edge features and the directed acyclic graph constructed in S2.

[0111] S402 includes:

[0112] S4021, integrating the model static features and hardware features. static and hardware feature vector F hardware Integrate into the node feature vector to form enhanced node features:

[0113] S4022, pass each feature and the directed acyclic graph through three layers of graph neural network, each layer updates and aggregates the node features, and comprehensively considers the feature information of its adjacent nodes. The specific operations are as follows:

[0114]

[0115] Among them, l represents the current number of GNN layers, is the node weight matrix of the lth layer.

[0116] S4023. All node features are further linearly transformed and nonlinearly activated through three fully connected layers. The specific processing is as follows:

[0117] node_feature FC =Sigmoid(W FC3 ×W FC2 (W FC1 (node_feature)))

[0118] Among them, the weight matrix W of each layer FC1 ,W FC2 ,W FC3 All are learnable parameters, and the Sigmoid activation function is used to limit the output between 0 and 1.

[0119] S4024. Using the node features processed above, the Sigmoid activation function is used to output the model's prediction of the time required for inference under the current hardware configuration.

[0120] InferenceTime=log(Sigmoid(node_feature FC ))

[0121] In addition, during the training process, the Huber loss function is selected as the loss function to balance the regression accuracy and robustness to outliers. The Huber loss function is defined as follows:

[0122]

[0123] Among them, a is the prediction error and δ is the threshold, which is used to control the behavior of the loss function in different error ranges.

[0124] In addition, in order to verify the effectiveness of the inference delay prediction method of the deep neural network model suitable for heterogeneous hardware proposed in the present invention, the present invention uses inference delay data including multiple deep neural network models (such as ResNet-50, MobileNetV2, etc.) on different heterogeneous hardware platforms (such as NVIDIA Tesla V100 GPU, RTX 3090GPU, Intel Xeon CPU, etc.) for performance testing. The experimental data set contains information in multiple dimensions such as model characteristics, hardware parameters, and actual measured inference delay time. The data set is divided into a training set (70%) and a test set (30%) in a ratio of 7:3, wherein the training set is used for model training and parameter tuning, and the test set is used to evaluate the prediction performance of the model. The comparison method includes delay prediction techniques based on linear regression and traditional machine learning methods (such as support vector regression, random forest, etc.). Performance indicators include 1) root mean square error (RMSE), which is used to measure the average error between the predicted value and the actual inference delay; 2) coefficient of determination (R-squared), which measures the model's ability to explain the change in inference delay. The closer the value is to 1, the better the model fitting effect.

[0125] The experimental results are shown in Table 2 below:

[0126] Table 2

[0127]

[0128]

[0129] As can be seen from the table, the graph neural network method proposed in the present invention outperforms traditional linear regression and machine learning methods in terms of RMSE and R-squared indicators, indicating that this method can more accurately predict the inference delay of deep neural network models on different heterogeneous hardware platforms. Specifically, the GNN method has the lowest RMSE, which means that its prediction error is the smallest; at the same time, the R-squared value is the highest, indicating that it has the strongest explanatory power for inference delay. The experimental results show that the method of the present invention can effectively capture the complex relationship between the model structure and hardware characteristics, and improve the accuracy and robustness of the prediction. This enables the method of the present invention to provide reliable data support for the deployment and optimization of deep neural network models in practical applications, significantly improving the operating efficiency and resource utilization of the system.

[0130] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0131] Embodiment 3:

[0132] Based on embodiments 1 and 2, embodiment 3 of the present application provides a deep neural network model inference delay prediction system applicable to heterogeneous hardware, including:

[0133] The first building block is used to build a general computational graph of a target deep neural network model;

[0134] A creation module, used to create a directed acyclic graph of a target deep neural network model according to the general computational graph;

[0135] The extraction module is used to extract the static features of the model based on the general computation graph and extract the hardware features through the current hardware parameters;

[0136] The second building module is used to build a model reasoning delay prediction model by using the directed acyclic graph, model static features and hardware features.

[0137] Specifically, the system provided in this embodiment is a system corresponding to the methods provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be repeated in this application.

Claims

1. A method for predicting inference delay of a deep neural network model applicable to heterogeneous hardware, characterized in that: include: S1, construct a general computational graph for the target deep neural network model; S2. Create a directed acyclic graph of the target deep neural network model according to the general computational graph; S3, extracting model static features based on the general computational graph, and extracting hardware features through current hardware parameters; S4. Utilize the directed acyclic graph, model static features and hardware features to construct a model reasoning delay prediction model.

2. The method for predicting inference delay of a deep neural network model applicable to heterogeneous hardware according to claim 1, characterized in that: S1 includes: S101, obtaining a target deep neural network model file and parsing it, identifying the types of layers in the model and their interconnected relationships, and calculating the size of the input and output tensors of each operation; S102. Traverse each layer of the model in sequence according to the execution order of the model, identify the operation type of each layer of the model, extract the corresponding operation attributes, and record the output tensor size of each operation; construct the nodes and edges of the general computational graph according to the operation type, operation attributes and operation attributes.

3. The method for predicting the inference delay of a deep neural network model applicable to heterogeneous hardware according to claim 2, characterized in that S2 include: S201, construct an empty node dictionary, and traverse each node in the general computation graph in post-order; During the traversal process, a unique code is assigned to each node using the number of nodes stored in the current node dictionary; S202, sorting all nodes in the node dictionary to conform to the topological structure of the directed acyclic graph; During the sorting process, function call nodes are identified, and operation names, operation attributes, and output sizes are extracted; S203, using a one-hot encoding method to identify the operation type, and combining the operation attributes and output size to construct a node feature vector; S204, using the order of post-order traversal, stacking all node feature vectors to form a node feature matrix; And construct an adjacency matrix based on the connection relationship between nodes; A directed acyclic graph is generated according to the node feature matrix and the adjacency matrix.

4. The method for predicting the inference delay of a deep neural network model applicable to heterogeneous hardware according to claim 3, characterized in that S3 include: S301, converting the batch size of the input data in S1 into a feature vector, calculating the number of multiplication and addition operations of each node and accumulating the total amount of operations, and then converting it into an operation amount feature vector; S302, traverse all operation nodes in the general computation graph constructed in S1, generate feature vectors of the number of operators of the convolutional layer, activation function and fully connected layer, and concatenate them with the batch size feature vector and the operation amount feature vector to form a static feature of the model; S303: Normalize the hardware parameters of the device and combine them to form hardware features.

5. The method for predicting the inference delay of a deep neural network model applicable to heterogeneous hardware according to claim 4, characterized in that S4 include: S401, constructing a node-edge encoder module, using a leaky linear rectifier activation function to perform nonlinear transformation on the hidden features of nodes and edges, wherein the edge features are updated through an attention mechanism; S402. Construct a three-layer graph neural network layer transfer feature based on the model static features, hardware features, edge features and the directed acyclic graph constructed in S2.

6. The method for predicting inference delay of a deep neural network model applicable to heterogeneous hardware according to claim 5, characterized in that: S402 includes: S4021. Integrate the model static features and hardware features into the node feature vector to form enhanced node features; S4022, transferring and aggregating the features and the directed acyclic graph through the three-layer graph neural network layer; S4023, all node features are linearly transformed and nonlinearly activated through three fully connected layers; S4024. Output the model through the Sigmoid activation function to predict the time required for inference under the current hardware configuration.

7. The method for predicting inference delay of a deep neural network model applicable to heterogeneous hardware according to claim 6, characterized in that: In S4, the training process uses the Huber loss function.

8. A deep neural network model inference delay prediction system suitable for heterogeneous hardware, characterized in that: Used to perform the method according to any one of claims 1 to 7, comprising: The first building block is used to build a general computational graph of a target deep neural network model; A creation module, used to create a directed acyclic graph of a target deep neural network model according to the general computational graph; The extraction module is used to extract the static features of the model based on the general computation graph and extract the hardware features through the current hardware parameters; The second building module is used to build a model reasoning delay prediction model by using the directed acyclic graph, model static features and hardware features.

9. A computer storage medium, characterized in that The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 7.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 7.

Images