A deep neural network delay method based on graph attention mechanism and large language model

Abstract

The application provides a deep neural network delay method based on a graph attention mechanism and a large language model, which comprises structure modeling of a neural network model, device information acquisition and enhancement, and multilayer perception prediction; a graph attention network structure code is generated for a deep neural network model M to be inferred, so as to obtain a model structure representation vector; a device representation vector is obtained; the spliced model structure and device vector representation are used as the input of a predictor after model training convergence; and an inference delay value is output. Through a representation mechanism for capturing the parallel and sequence execution relationship in the model structure, and by fusing the device bottom layer performance index, an enhanced mechanism for automatic attribute acquisition is realized; high-precision prediction is still realized under a small sample, and a unified framework with inference performance and device factor interpretability is simultaneously provided, so that the inference precision, adaptability and interpretability in a heterogeneous device environment are comprehensively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and computer systems, and particularly relates to a deep neural network delay inference modeling and prediction method. BACKGROUND

[0002] With the wide application of deep learning technology in the fields of computer vision, natural language processing, recommendation system, network system, etc., deploying deep neural network model has become a core component of modern intelligent system. In the actual deployment process, model inference delay has become one of the key indicators affecting system performance, user experience, and resource scheduling efficiency.

[0003] In order to ensure that the model running on a given hardware platform (such as CPU, GPU, TPU, etc.) meets the delay constraint, it is of great significance to accurately evaluate or predict the inference delay of DNN. For example, in a cloud-edge collaborative environment, different devices have heterogeneous computing resources and performance bottlenecks. Only by accurately mastering the model delay performance can reasonable computing resource scheduling be carried out; in neural architecture search (NAS), the model structure may control the inference delay while optimizing the accuracy, so as to achieve the balance of performance-efficiency. If the inference delay is not accurately modeled, it may lead to ineffective scheduling strategy, unbalanced architecture selection, and even out-of-control system response.

[0004] Traditional DNN delay evaluation methods mainly rely on actual deployment testing, that is, loading the model to be tested to the target device for multiple rounds of execution, and then counting the average delay. However, this method has two main problems: (1) when the model-device combination space is large, the testing cost is too high. For example, if there are m devices and n model structures, the testing times reach orders of magnitude; (2) the device testing resources are limited, especially for edge devices or high-performance chips, the testing path is limited, the deployment cycle is long, and the sampling difficulty is great.

[0005] In order to reduce the deployment testing cost, researchers have proposed various low-cost delay inference methods, which can be roughly divided into three categories:

[0006] (1) Statistical index-based estimation method: Early research uses model parameter quantity (Parameters) or floating point operation number (FLOPs) as a proxy variable for model complexity to establish a delay estimation model. This method believes that the higher the FLOPs, the greater the delay, and the delay prediction can be approximated by a simple function. However, the real device running involves a large number of nonlinear factors such as memory access, parallel execution, cache hit rate, communication delay, etc., and there is a significant deviation between FLOPs and real delay;

[0007] (2) Aggregation method based on hierarchical prediction: This method decomposes the entire model into various layers (such as convolutional layers, pooling layers, etc.), builds a delay predictor for each layer, and then sums the delays of each layer to obtain the overall model delay. This method can train the estimation model of each layer with a small number of operation samples, and has good generality. However, it ignores the context-dependent relationship between layers and cannot model the synergistic effect between parallel structures, data reuse, pipeline execution, etc.

[0008] (3) End-to-end learning method based on graph structure: A new type of method developed in recent years represents the DNN model structure as a directed graph and uses graph neural networks (such as GCN, GraphSAGE) to extract model structure information. Through end-to-end training, a delay prediction model is established. These methods have stronger structure representation ability than the previous two methods, but they have the following limitations:

[0009] Most of the structure representation methods rely on discrete encoding or one-hot vectors, which cannot capture the functional association between similar operations.

[0010] The modeling ability of the dependence between nodes is limited, and it is difficult to assign different influence weights between different operations, which limits the delay discrimination ability of the model.

[0011] In addition to modeling the model structure, modeling the hardware device characteristics is also important. Different devices have significant differences in the execution delay of the same DNN model. For example, the number of stream processors, bandwidth, and memory size in GPU will directly affect the parallel execution efficiency; while in CPU, it is limited by the number of cores, cache hit rate, and instruction scheduling mechanism. Therefore, the delay inference model not only needs to understand the "model structure", but also must fully perceive the "device characteristics".

[0012] However, there are two significant problems in the existing device modeling methods: (1) Only "device category" (such as CPU / GPU / NPU) is used as a label or discrete variable, and specific performance parameters are not introduced; (2) Even if some methods introduce certain performance indicators, such as bandwidth, these indicators are selected in a rather one-sided manner, lack of systematicness, and are difficult to obtain manually. Existing methods have not yet established a comprehensive and lightweight device feature enhancement mechanism, resulting in insufficient generalization ability of the model to different hardware platforms, especially in the sparse sample scenario.

[0013] In addition, current delay inference methods generally lack explainable analysis of the impact of device factors. In a complex heterogeneous deployment environment, it is not enough to simply obtain delay prediction results, and it is more important to understand the key device indicators that cause delay changes, thereby guiding system optimization in reverse. For example, if the model can determine that the "core number" has the greatest impact on the current architecture, then high-core-number devices can be preferentially allocated in system scheduling. Such analysis methods are basically absent in existing research. In summary, the existing DNN delay inference methods have obvious shortcomings in model structure representation ability, device information utilization efficiency, and key factor explainability; in actual deployment, they face challenges such as device diversity, difficulty in obtaining samples, and high requirements for structure perception. SUMMARY

[0014] In view of the above deficiencies in the current deep neural network model delay inference modeling, the present application provides a deep neural network delay method based on a graph attention mechanism and a large language model, which can capture the representation mechanism of the parallel and sequential execution relationship in the model structure, and can integrate device bottom-level performance indicators to realize an enhanced mechanism for automatic attribute acquisition; it still achieves high-precision prediction under small samples, and has a unified framework of inference performance and device factor explainability, to comprehensively improve the inference accuracy, adaptability and explainability in a heterogeneous device environment.

[0015] To achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

[0016] A deep neural network delay method based on a graph attention mechanism and a large language model, comprising the following:

[0017] Model structure modeling: graph attention network structure encoding is performed on the deep neural network model M to be inferred to obtain a model structure representation vector ;

[0018] Device information acquisition and enhancement: inputting the constructed device prompt word to a pre-trained large language model, outputting a device corresponding key performance attribute field by the large language model, forming device enhancement information ; concatenating the device enhancement information and the basic information to obtain effective device information | ; inputting the effective device information | to a feedforward network encoder to generate a unified scale device representation vector ; wherein the key performance attribute field includes but is not limited to maximum turbo frequency, core number, thread number, cache capacity, and video memory bandwidth; wherein the feedforward network encoder adopts a single-layer linear transformation structure;

[0019] MLP prediction: a predictor is obtained after the trained convergence of the MLP model, and a model structure representation vector and a device representation vector The input vector obtained after splicing is input into the predictor, and the output is the inference delay value .

[0020] Further, the graph attention network structure encodes the deep neural network model M, and the specific steps are as follows:

[0021] The deep neural network model M is represented as a directed graph structure ; wherein, the node represents an operation unit, and the edge represents a data flow dependency;

[0022] Each node is mapped to an embedding vector to form an initial feature matrix ; wherein, the embedding vector is initialized by a trainable matrix , k is the total number of operation types, is the embedding dimension;

[0023] A global node is introduced in the graph structure, connected with other nodes to form an enhanced graph structure;

[0024] An L-layer graph attention network is constructed to encode the enhanced graph structure, and the update mode of each layer node embedding is as follows: ; wherein, is the attention weight of node to node , generated by activation function;

[0025] The embedding vector of the global node is output as the neural network structure representation vector .

[0026] Further, in the MLP training stage, the loss function is used for model optimization: , wherein, represents a training sample set, is a regularization coefficient.

[0027] Further, after the training of the MLP is completed, the obtained predictor model is subjected to gradient inversion and factor explanation as follows: for each device factor , its contribution to the delay output The partial derivative of the delay output with respect to the device factor z is defined as: ; wherein, represents the gradient magnitude of the delay output with respect to the device factor z;

[0028] When the device factor is , the difference change of the model loss function between the first round and the second round is represented as: ;

[0029] The first-order Taylor expansion is applied to the difference change of the loss function, and the high-order term is ignored, which is represented as:

[0030] ;

[0031] In the gradient descent process in the training process, the model parameter update is represented as:

[0032] ;

[0033] The difference change of the loss function .

[0034] The present application has the following advantages:

[0035] (1) Stronger neural network structure perception ability: The present application uses a graph attention mechanism to model the DNN structure, which can fully capture the topological features, structural dependencies and differences between serial / parallel execution paths in the model. Compared with the traditional FLOPs estimation or hierarchical summation modeling method, the graph structure expression combined with global nodes and multi-layer attention mechanism effectively enhances the model's representation ability for complex structures, thereby improving the delay prediction accuracy;

[0036] (2) Support for automatic and more informative device modeling mechanism: The present application first introduces a large language model as a device information generator to automatically obtain the core hardware indicators of the target device through natural language prompts, covering maximum turbo frequency, core number, thread number, cache capacity, video memory bandwidth and other attributes, solving the problem of insufficient device feature dimension and manual selection errors in the prior art. This mechanism has high universality and migration, and is suitable for various computing platforms;

[0037] (3) High-precision delay prediction ability under small sample conditions: The structure-device joint modeling method proposed by the present application can accurately infer the unmeasured device and structure combination under the premise of using only a small number of measured samples. Under the condition that the number of training samples is not more than 100, the normalized mean absolute error (NMAE) of delay prediction can still be controlled within 5%, which is significantly better than the existing method which relies on large-scale data training, reducing the deployment cost and system cold start difficulty;

[0038] (4) Introduce the device factor inversion mechanism of explainability: the present application can quantify the influence degree of each device index in the delay prediction result by constructing a gradient-based feature inversion module. Through partial derivative and first-order Taylor expansion, the system can output the marginal influence intensity of each hardware property on the model performance, thereby helping the developer to understand the performance bottleneck, providing data support for device selection and resource scheduling. This mechanism makes up for the shortcomings of the prior art in explainability;

[0039] (5) Good cross-platform generalization ability and application adaptability: the present application method supports unified modeling among multiple heterogeneous devices such as CPU, GPU, TPU, NPU, edge embedded platform, has structure independence and platform independence. At the same time, it can be seamlessly integrated into multiple application scenarios such as neural architecture search (NAS), intelligent inference deployment, edge computing scheduling, etc., has good industrial landing potential and engineering popularization value;

[0040] (6) Improve deployment efficiency and optimization feedback ability: the present application can not only predict model delay, but also analyze which device factor has the greatest impact on delay, support the system to give priority to "performance sensitive" resource configuration scheme at the scheduling level, realize the optimization of software and hardware, improve the inference efficiency and system stability, and promote the automation, controllability and intelligence of intelligent system deployment.

[0041] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and obtained by the structure particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiment or related art description. Obviously, the drawings in the following description are only the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0043] Figure 1 A deep neural network delay method flowchart based on graph attention mechanism and large language model is provided for the present application;

[0044] Figure 2 A comparison diagram of different delay modeling methods provided in the embodiment of the present application;

[0045] Figure 3 A general framework diagram of the delay inference system provided in the embodiment of the present application;

[0046] Figure 4 A schematic diagram of a neural network structure modeling process provided in an embodiment of the present application is shown in the figure.

[0047] Figure 5 A schematic diagram of a graph attention encoding module provided in an embodiment of the present application is shown in the figure.

[0048] Figure 6 A schematic diagram of a device information enhancement module based on a large language model provided in an embodiment of the present application is shown in the figure.

[0049] Figure 7 A schematic diagram of a delay module structure provided in an embodiment of the present application is shown in the figure.

[0050] Figure 8 A schematic diagram of evaluation test results of three different encoding methods provided in an embodiment of the present application is shown in the figure.

[0051] Figure 9 A schematic diagram of graph structure information modeling experimental results provided in an embodiment of the present application is shown in the figure.

[0052] Figure 10 A schematic diagram of large language model-based enhancement effect evaluation experimental results provided in an embodiment of the present application is shown in the figure.

[0053] Figure 11 A schematic diagram of systematic evaluation experimental results of the influence of the optimal configuration of key hyperparameters on the hidden dimension and the number of layers of the graph attention network in the representation learning process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] To make the purposes, technical solutions, and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0055] BRP-NASP: a prediction-based automated neural architecture search method that can improve search efficiency while accurately estimating hardware performance.

[0056] The design idea of the embodiments of the present application will be briefly introduced below.

[0057] As shown in Figure 1 A deep neural network delay method based on a graph attention mechanism and a large language model, including the following:

[0058] Step S01: Model structure modeling. Encode the deep neural network model M to be reasoned using a graph attention network structure to obtain the model structure representation vector. ;

[0059] The deep neural network model to be inferred Represented as a directed graph structure Among them, nodes Represents operation units (such as convolution, activation, pooling, etc.), edges This indicates the data flow dependencies between operations;

[0060] Each node Mapping to embedding vector Thus, the initial feature matrix is ​​constructed. The embedding vector is composed of a trainable matrix. Initialization, where, Let be the total number of operation types, and D be the embedding dimension. Similar operation types have similar representations in the embedding space.

[0061] Introducing global nodes into the graph structure This node is connected to all other nodes, forming an enhanced graph structure. The graph is encoded using an L-layer Graph Attention Network (GAT), where each layer embeds nodes... The update method is as follows:

[0062] ;

[0063] in, The attention weights of node j to node i are generated by the LeakyReLU activation function. Finally, the global node is output. Embedded vector As a representation vector of neural network structure .

[0064] Step S02: Device information acquisition and enhancement, including the constructed device prompt words. The data is input into a pre-trained large language model, which outputs key performance attribute fields corresponding to the device to form device enhancement information. ; splicing equipment enhances information and basic information To obtain valid device information ( | ); will include the device's valid information ( | The input is fed into the feedforward network encoder to generate a uniform-scale device vector representation. The key performance attribute fields include, but are not limited to, maximum turbo frequency, number of cores, number of threads, cache capacity, and memory bandwidth; the feedforward network encoder adopts a single-layer linear transformation structure.

[0065] Construction equipment prompts The data is input into a pre-trained Large Language Model (LLM) to obtain device enhancement information. This includes key performance indicators such as maximum turbo frequency, number of cores, number of threads, cache capacity, and memory bandwidth. Enhanced information will be provided. Basic Information The concatenated data is input into the encoder to generate a device vector representation with a uniform scale. , where Encoder is a single-layer feedforward network.

[0066] Step S03: As Figure 2 As shown, different from Figure 2 (a) The prediction method based on computational power and Figure 2 (b) Hierarchical prediction method, Figure 2 (c) Based on end-to-end prediction, multilayer perceptron prediction is used. After the multilayer perceptron model is trained and converged, the predictor is obtained, and the model structure representation vector is used. and device representation vector The concatenated input vector serves as the input to the predictor, and the output is the inference delay value. ;

[0067] Delayed prediction: Representing the structure vector Device representation vector After concatenation, the data is fed into a three-layer multilayer perceptron (MLP) for delay prediction. A ReLU activation function is used for nonlinear mapping, and the final output is the inference delay value. : ;

[0068] During the training phase, the following loss function is used to adjust the model parameters. Optimize:

[0069] ;

[0070] in, Represents the training sample set, for Regularization coefficient. During training, a small sample training scheme is adopted, and a prediction accuracy of no more than 0.05 normalized mean absolute error (NMAE) can still be obtained with no more than 100 samples.

[0071] Gradient inversion and factor interpretation: After model training is complete, by analyzing the model output... Regarding the device factor The partial derivative of the device factor with respect to the delay is calculated to quantify the degree of influence of the device factor on the delay: The influence of the change in the device factor on the prediction error can be further approximated by a first-order Taylor expansion:

[0072] ;

[0073] Thus, the sensitivity evaluation of the key hardware factor is achieved, providing a reference for device optimization and deployment.

[0074] In specific implementations, as shown in Figure 3 , a framework based on deep neural network delay inference is implemented, referred to as LLGAT; the structural description file of the deep neural network model to be tested (e.g., neural network structure M) and the specification information of the target computing device (denoted as D) are taken as inputs, and the corresponding inference delay prediction value is output. As shown in Figure 3 , it is composed of the following three main functional modules:

[0075] First, the DNN architecture encoding module; this module first parses the input deep learning model structure file into a directed graph structure, where the nodes in the graph represent operation units and the edges represent data flow dependencies. Then, through the embedding encoding layer, each operation type is mapped to an embedding representation in the vector space; next, the graph attention network (GAT) aggregates information on the operation nodes according to the graph topology and dynamically allocates attention weights to extract high-level semantic representations of the model structure. This module effectively captures the complex sequential, parallel, and skip connection relationships in the neural network structure;

[0076] Second, the computing device enhancement module; to fully represent the hardware characteristics of the device, in addition to the basic information (such as device name, type, etc.), this module further introduces a pre-trained large language model to generate enhanced structured attribute information for the device name, including but not limited to core number, main frequency, cache size, bandwidth, etc. After concatenating the basic information and the enhanced information, a continuous and trainable device representation vector is generated for subsequent inference tasks;

[0077] Third, the delay inference module. This module receives the model structure representation vector and the device representation vector generated by the previous two modules, and performs fusion and prediction through a multi-layer perceptron to finally output the corresponding delay prediction value . This module has end-to-end modeling capability and can effectively utilize the joint features of the model and the device to achieve accurate inference on unseen combinations.

[0078] In specific implementations, as shown in Figure 4 , precise modeling of the deep neural network (DNN) architecture and operation feature representation is achieved by first representing the neural network model structure M to be processed as a directed graph structure ; wherein each node in the graph corresponds to an operation unit (e.g. convolutional layer, pooling layer, etc.) in the network, and an edge between two nodes represents a data dependency or execution order between the operations.

[0079] To effectively distinguish different types of operations and preserve the potential structural relationship between similar operations, the present application provides an embedding encoding mechanism, which is different from the one-hot encoding or discrete vector encoding method (such as BRP-NAS, HELP or DNNPerf method) used in the prior art. By introducing a trainable embedding matrix, each type of operation is mapped to a continuous vector space, so that the model can automatically learn and capture the semantic proximity between different operations.

[0080] In a specific implementation, an operation embedding matrix is defined as follows: where k represents the total number of operation types, and D represents the dimension of the embedding vector. If a certain operation node belongs to the jth type of operation, its corresponding embedding vector is represented as In this embedding mechanism, operation nodes of the same type are represented by the same embedding vector to maintain consistency; while similar but different operation types (such as 1 1 convolution and convolution) are mapped to different vectors, but since the embedding space is a continuous and learnable space, the model can automatically learn the semantic similarity between these operations during training, so that it can more accurately capture their common computational features during delay inference. This embedding mechanism not only enhances the model's ability to express operation features, but also improves its generalization ability for low-sample situations, and provides high-quality input representation for subsequent graph attention network structure encoding and delay prediction.

[0081] To achieve accurate modeling of the complex dependency relationship between operations in the neural network structure, in one implementation as shown in Figure 5 , a graph attention network (GAT, Graph Attention Network) based encoding module is used to express the topological structure of the neural network. The neural network structure contains parallel and serial paths, and there are various calling relationships between the operation nodes of the paths, and the influence weight of different paths on the inference delay is also different, so the structure information needs to be differentiated and dynamically modeled.

[0082] In the graph attention mechanism, the information of the nodes in the graph is propagated layer by layer using a multi-layer structure. Let the graph encoder contain L layers in total, and there are N operation nodes in the neural network structure, and the node embedding initialization value is the vector representation extracted by the embedding encoding module . Then the i ​​The node embedding of a layer is calculated by the following update rule:

[0083] ;

[0084] wherein, is the embedding vector of the adjacent node, is the learnable weight matrix of the l-th layer, is a ReLU activation function, and LayerNorm is a standard layer normalization function, denotes the node The attention weight of the node is defined as follows:

[0085] ;

[0086] ;

[0087] wherein, | denotes a vector splicing operation, is a learnable parameter in the attention mechanism. Through the above attention mechanism, the importance of information transmission between different nodes can be dynamically adjusted according to the semantic relationship between the nodes, so as to capture complex calling dependencies in the network topology structure.

[0088] After L-layer graph attention aggregation, the final embedding representation of the global node is denoted as , and the vector is denoted as , which is the graph representation of the neural network structure M. The vector is one of the inputs of the delay reasoning module. In the embodiment, the graph attention encoding module has the following two main advantages:

[0089] (1) It has stronger graph structure representation capability. Through the L-layer information propagation mechanism, the node embedding can aggregate the neighbor information within the L-hop range. The global node can obtain the expression capability of the complete graph structure after aggregation by connecting with all nodes, which effectively improves the perception capability of the model to the global topology relationship;

[0090] (2) It has the ability to dynamically model complex calling dependencies: through the attention weight mechanism, the influence strength between operation nodes can be dynamically adjusted according to the structure position and context semantic information, the differential influence of different structure paths on reasoning delay is finely modeled, and the modeling accuracy of the calling relationship between operations is improved.

[0091] Through the structure encoding mode in the above embodiment, the application can effectively capture the complex serial and parallel dependency relationship in the deep neural network, and generate a high-quality structure representation vector by combining the global structure perception mechanism, thereby providing an accurate and generalizable feature input for subsequent delay reasoning.

[0092] In specific implementation, as Figure 6As shown, Figure 6 (a) Device attributes shown, Figure 6 (b) Attribute encoding. One embodiment provided by the present application effectively models device information to build a delay inference model. In which, the same neural network structure M may exhibit significantly different inference speeds on different computing devices, and this difference is mainly due to the underlying hardware indicators of the device. For example, Figure 6 (a) As shown, taking CPU as an example, the key factors affecting inference delay include frequency, maximum turbo frequency, core number, thread number, three-level cache capacity, memory bandwidth, etc.; while in GPU devices, the main parameters include stream processor number, core frequency, video memory capacity, video memory bit width, etc.

[0093] In specific implementation, by introducing a pre-trained language model, rich context attribute information is automatically extracted from existing device identifiers, and is encoded into a continuous vector to enhance the expression ability of device information. The enhancement mechanism includes the following two steps:

[0094] (1) Device attribute completion. The model number, name or other basic description information of the device is constructed as a natural language prompt (Prompt), which is input into the pre-trained language model, and the key performance attribute fields corresponding to the device are output by the language model to form enhanced information . This information includes but is not limited to core number, frequency, cache capacity, bandwidth, thread number, video memory parameters, etc.

[0095] (2) Information encoding generates embedding vector. As shown in Figure 6 (b), the basic device information and the enhanced device attribute information are spliced and input into the encoder (Encoder), which uses a single-layer linear transformation structure for feature projection, and outputs a device embedding vector with a dimension of as one of the inputs of the delay prediction module. The encoding formula is as follows: where, represents the basic attribute information of the device, is the enhanced information generated by the language model, and | represents the splicing operation. The generation process of the enhanced information can be represented as: . Where, The device prompt words are constructed for the device, and are used to guide the language model to generate complete attribute fields related to the device. Compared with the way of using only the basic information, the model can understand the device differences more comprehensively after introducing the enhanced information, and the delay prediction capability in the cross-device scene is significantly improved. The device information enhancement module has the characteristics of automation, high coverage, and controllable semantics, which not only reduces the cost of manually constructing device features, but also improves the adaptability of the inference model to complex hardware environments, and is particularly suitable for model deployment and performance evaluation scenes in a heterogeneous computing platform.

[0096] As shown in Figure 7 , in an embodiment, to realize accurate prediction of delay inference, a delay inference module and a training method are provided. The module takes the fused neural network structure representation vector and the device representation vector as input, and completes the regression inference task of the delay value through a multi-layer perception machine.

[0097] In a specific implementation, first, the structure representation vector obtained from the graph attention module is spliced with the device representation vector generated by the device information enhancement module to form a joint feature representation. The representation is input into a multi-layer perception network (MLP), and an inference delay prediction value is output. The expression is as follows:

[0098] , wherein the MLP is a three-layer feedforward neural network, a nonlinear activation function is used for feature conversion between layers, and finally a delay prediction result is output.

[0099] In terms of model training, the predictor parameters are optimized by using a supervised learning strategy, and the prediction error loss function is defined as: ,

[0100] , wherein is the actually observed inference delay, is the model prediction value, is the trainable parameter in the model, is a regularization coefficient for preventing overfitting (set to 0.1 in this embodiment). The loss function adopts a combination form of mean square error and regularization term, so as to guide the model to achieve a balance between accuracy and stability.

[0101] The whole training process includes the following steps:

[0102] (1) initialize the model parameters ;

[0103] (2) construct natural language prompts according to device information and enhanced information by a large language model generation device ;

[0104] (3) convert the neural network structure with reasoning into a graph structure G, and extract the structure representation through an embedding layer and a graph attention network ;

[0105] (4) input the enhanced device information to the encoder to obtain the device representation ;

[0106] (5) splice the structure representation and the device representation and input them to the MLP predictor to output the predicted delay value ;

[0107] (6) calculate the loss between the predicted result and the real delay value , and optimize the parameters through back propagation ;

[0108] (7) repeat the above process until the model converges, and finally obtain the trained delay reasoning model.

[0109] After completing the model training, the model can be deployed in the actual system, and the deep learning model of any structure and the target device description can be input to quickly predict the running delay without real deployment test. Since the model of the present application can deeply understand the interaction between the DNN structure topology and the device performance factor, even under the condition of limited number of training samples, high prediction accuracy and generalization ability can be obtained.

[0110] To further clarify the key device factors affecting delay prediction and improve the model interpretability, a gradient-based feature inversion method is proposed to quantify the contribution of device factors in the delay reasoning process.

[0111] In specific implementation, by means of the gradient propagation information in the model training process, first, the partial derivative of each device factor z to the delay output is calculated, which is defined as follows: , The gradient amplitude of the delay output to the device factor z, the larger the value, the more significant the influence of the factor on the reasoning result. In order to further demonstrate the effectiveness of the method, based on the first-order Taylor expansion, the model loss change is derived, the model parameters of the tth round and the t+1th round in the training process are and , then under the condition that the test sample device factor is , the change of the loss function can be expressed as: .

[0112] ​Applying first-order Taylor expansion to the above difference and ignoring high-order terms, we have

[0113] ;

[0114] In the gradient descent process during the training process, the model parameters are updated as:

[0115] ;

[0116] Substituting the update rule into the above equation, we have:

[0117] ;

[0118] Therefore, the loss change of the test sample is affected by the gradient size and direction of the device factor in the training sample. In other words, the greater the gradient of a device factor during the training process, the greater the impact of the loss function change on the model optimization process, thereby indirectly reflecting the importance of the factor to the delay prediction performance.

[0119] Experimental results

[0120] In an embodiment of the present application, the training and testing of the delay inference model are performed on a workstation running the Ubuntu 22.04 operating system, which is configured to include 64 GB of memory, an Intel Core i9-13900K central processor, and an NVIDIA GeForce RTX 4090 graphics processor. To achieve device information enhancement based on a large language model, the present application accesses a large language model service to generate device attribute text by expanding the API interface, and in specific implementation, a language model based on GPT-4o is used to enhance the semantic understanding and expression ability of the device. In terms of training parameter settings, the initial learning rate is set to 1×10 -3And adopt dynamic attenuation mechanism, after 35 consecutive rounds of model performance improvement, automatically reduce the learning rate to 0.5 times of the original. The optimizer selects the Adam method, and the batch size is set to 32, to ensure the stability of training while improving the training efficiency. To prevent overfitting, the present application limits the maximum training rounds to 1000, and introduces an early stopping mechanism (Early Stopping). If the performance of the validation set does not improve for 50 consecutive rounds, the training process is terminated. The final model used is the model parameter corresponding to the round with the best performance on the validation set. In the experiment, to ensure the fairness of the comparison, all the comparison baseline models are optimized in the hyperparameter space through grid search. The experiment uses 5 different random seed repeated running mode, 10% of the data is divided as the validation set in each training, and the final result is evaluated and reported based on the average inference result of the test set.

[0121] To evaluate the delay inference accuracy of the proposed method, two standardized error evaluation indicators are introduced: normalized mean absolute error (NMAE) and normalized root mean squared error (NRMSE). The above indicators are based on the difference between the predicted value and the true value of the unobserved delay sample, and can fully reflect the inference ability of the model on unseen data.

[0122] Let denote the index set of all unobserved delay items, denote the th predicted delay value, denote its corresponding true delay value, then the calculation method of NMAE and NRMSE is as follows:

[0123] ;

[0124] ;

[0125] Among them, NMAE is used to measure the average deviation of the overall error, and NRMSE reflects the deviation degree of the prediction result in the square sense. Both of the above two indicators are dimensionless quantities, the lower the value represents the higher the accuracy of the model delay inference, and have good comparability and scalability.

[0126] Table 1 Comparison of delay estimation model methods

[0127]

[0128] To verify the universality and accuracy of the proposed delay inference method under different sample sizes, first, a number of comparative experiments are designed to evaluate the inference performance of the method on CPU and GPU computing platforms. In the experiment, different numbers of training samples are selected to measure the adaptability of the method under low sample data conditions, while keeping consistent with the existing commonly used sample size to ensure the comparability of the results.

[0129] As shown in Table 1, under all sample sizes, the method of the present application is significantly better than the existing delay inference method in various performance indicators. Especially in the case of only 100 samples, the prediction error of the present application on the CPU platform can be reduced by 50.8%, and on the GPU platform, it also achieves a 27.4% error reduction, significantly better than the optimal baseline method. This shows that the structure-device joint modeling strategy proposed by the present application has good small sample learning ability. The method of the present application is compared with three types of mainstream delay modeling methods with higher accuracy:

[0130] (1) First, in comparison with traditional machine learning models, the linear estimation method characterized by FLOPs and the method based on gradient boosting tree (GBDT) are used as controls. It is found that FLOPs only provides the operation complexity index and cannot distinguish between parallel and serial structures within the model, resulting in a lack of structure information; GBDT has certain non-linear modeling ability, but cannot perceive the dependent call relationship between DNN operations. Therefore, under different sample sizes, both methods fail to achieve good performance. In contrast, the method of the present application effectively captures the topological information of the internal structure of the deep model by introducing graph structure modeling and attention mechanisms, showing higher inference accuracy;

[0131] (2) In comparison with neural network models, multilayer perceptron (MLP), gated recurrent unit (GRU), long short-term memory network (LSTM), and bidirectional RNN (BiRNN) are selected as representative methods for experiments. Related models convert DNN structures into sequence inputs, although RNN-based methods perform well in modeling sequential dependencies, they cannot effectively handle complex branching and fusion structures in the model. Therefore, they still show significant disadvantages in predicting delays. Unlike this, the present application, based on graph-level modeling, introduces context information generated by large language models to effectively integrate device features and structural interactions, improving the model's generalization ability and structural understanding ability, achieving the best results among all neural network models;

[0132] (3) In comparison with the graph neural network (GNN) type model, the BRP-NAS, HELP and DNNPerf three kinds of graph modeling delay reasoning methods are selected as references. Compared with these methods, the application achieves better performance under two types of device platforms. This advantage comes from the fact that the application uses a graph attention mechanism to replace the fixed adjacency aggregation mechanism in the traditional GNN, which can dynamically allocate attention weights between nodes according to the structure, and more flexibly capture the asymmetric dependency relationship between operations. At the same time, compared with the commonly used one-hot encoding method, the application uses a trainable embedding encoding layer to enhance the semantic association modeling ability between operations. In addition, by introducing the context information generated by the large language model, the device attribute dimension is further supplemented, and the utilization efficiency of device information is improved, so that the stable and high-precision reasoning performance can be maintained even with a small sample.

[0133] In summary, in the comparison experiments of multiple baseline methods, the application shows stable and superior performance, which fully verifies the technical advantages of the application in structure understanding ability, device information modeling ability, small sample reasoning precision and cross-platform adaptability.

[0134] In a preferred embodiment of the application, in order to verify the effectiveness of each key module in the proposed delay reasoning method, three types of experiments are designed to evaluate the influence of different encoding strategies on model performance, the modeling value of graph structure information and the gain of large language model enhancement module under different sample conditions. Related experiments are carried out under multiple sample sizes (for example ) conditions, and the reasoning performance under the normalized mean absolute error (NMAE) and normalized root mean square error (NRMSE) indicators is tested on CPU and GPU hardware platforms respectively.

[0135] The experimental results are shown in Figure 8 , Figure 8 (a) shows the relationship between the sample size and the normalized mean absolute error when using numerical encoding, one-hot encoding and embedding layer encoding based on the CPU dataset, Figure 8 (b) shows the relationship between the sample size and the normalized root mean square error when using numerical encoding, one-hot encoding and embedding layer encoding based on the CPU dataset, Figure 8 (c) shows the relationship between the sample size and the normalized mean absolute error when using numerical encoding, one-hot encoding and embedding layer encoding based on the GPU dataset, Figure 8(d) The relationship between the number of samples and the normalized mean square error when using value-based encoding, one-hot encoding, and embedding layer encoding based on the GPU dataset. First, in the encoding strategy evaluation experiment, the operation type was encoded in three different ways: value-based encoding, one-hot encoding, and embedding layer encoding. The experimental results show that embedding layer encoding performs best in all sample sizes, especially in the context of small sample sizes (such as ) on the CPU dataset, the NMAE and NRMSE are reduced by 32.99% and 30.90% respectively ( Figure 8 (a-b)), and on the GPU dataset, they are reduced by 25.71% and 22.35% respectively ( Figure 8 (c-d)). This performance improvement is due to the embedding encoding, which can map high-dimensional sparse operation types to low-dimensional dense representations, effectively capturing the potential semantic association of operations. On average, this strategy achieves an overall performance improvement of 42.98% in all sample sizes, verifying the superiority of embedding encoding in capturing operation semantic expression.

[0136] The experimental results are shown in Figure 9 , Figure 9 (a) The relationship between the number of samples and the normalized mean absolute error when using GCN, GraphSAGE, and GAT based on the CPU dataset, Figure 9 (b) The relationship between the number of samples and the normalized mean square error when using GCN, GraphSAGE, and GAT based on the CPU dataset, Figure 9 (c) The relationship between the number of samples and the normalized mean absolute error when using GCN, GraphSAGE, and GAT based on the GPU dataset, Figure 9 (d) The relationship between the number of samples and the normalized mean square error when using GCN, GraphSAGE, and GAT based on the GPU dataset. In the graph structure information modeling experiment, the graph attention network (GAT) used in the present application was compared with other graph neural network structures, including graph convolution network (GCN) and GraphSAGE, in terms of performance. Under the condition of keeping the number of network layers and parameter configuration consistent, the experimental results show that GAT significantly outperforms the other two model structures in the delay inference task. Especially when the sample size is , GAT improves the NMAE and NRMSE performance of GCN on the CPU dataset by 33.09% and 29.80% respectively ( Figure 9(a-b) ), which respectively improve 29.20% and 34.68% on GPU dataset. The performance advantage mainly comes from the ability of attention mechanism to give dynamic weights to the interaction between nodes, which can more effectively model the asymmetric dependence between operations and strengthen the expression ability of DNN complex topology ( Figure 9 (c-d) ). The graph structure modeling strategy improves the performance by an average of 41.79% under all sample sizes, fully verifying the modeling advantage of the application in structure understanding.

[0137] The experimental results are shown in Figure 10 , as shown in Figure 10 (a) shows the relationship between sample size and normalized mean absolute error when using and not using LLM based on CPU dataset, as shown in Figure 10 (b) shows the relationship between sample size and normalized mean absolute error when using and not using LLM based on CPU dataset, as shown in Figure 10 (c) shows the relationship between sample size and normalized mean absolute error when using and not using LLM based on GPU dataset, as shown in Figure 10 (d) shows the relationship between sample size and normalized mean absolute error when using and not using LLM based on GPU dataset. In the large language model enhancement effect evaluation experiment, the performance of the model with and without the introduction of LLM enhancement under different sample sizes is compared. The experimental results show that under various sample sizes, the error of the model with LLM enhancement is significantly reduced, especially under the condition of data scarcity. In case, the NMAE and NRMSE of the GPU dataset are reduced by 27.4% and 25.3% ( Figure 10 (c-d) ), and the CPU dataset is reduced by 50.8% and 45.9% ( Figure 10 (a-b) ). The results show that the context information generated by the language model can effectively supplement the device attribute information in the small sample environment, thereby enhancing the generalization ability and representation ability of the model. Although the performance gap between the two gradually narrows in the large sample context, on average, the LLM enhancement strategy brings an overall performance improvement of 24.33% under various sample conditions, further proving its significant advantage in the data insufficient scenario.

[0138] In summary, the application integrates the embedded coding mechanism, the graph structure modeling mechanism and the large language model enhancement mechanism in the delay reasoning framework, and exhibits stable and significant performance advantages under different sample sizes and device platforms, further proving the significant improvement effect of the application in accuracy, stability and generalization ability.

[0139] The experimental results are shown in Figure 11 , as shown in Figure 11(a) Normalized mean absolute error and normalized root mean square error on CPU dimensional data sets, as shown in Figure 11 (b) Normalized mean absolute error and normalized root mean square error on GPU dimensional data sets, as shown in Figure 11 (c) Normalized mean absolute error and normalized root mean square error on CPU layer data sets, as shown in Figure 11 (d) Normalized mean absolute error and normalized root mean square error on GPU layer data sets. In an embodiment of the present application, in order to further optimize the model performance and determine the optimal configuration of key hyperparameters, the influence of the hidden dimension size (denoted as D) used in the representation learning process and the number of graph attention network (GAT) layers is systematically evaluated.

[0140] First, as shown in Figure 11 (a-b), for the influence of the hidden dimension D, a plurality of sets of values of D are tested while keeping other hyperparameters unchanged, and the delayed inference performance of the model is evaluated on the CPU and GPU data sets, respectively. The experimental results show that as the hidden dimension increases, the prediction error (NMAE and NRMSE) of the model gradually decreases, indicating that a higher dimension helps to capture richer potential structural information. However, when D exceeds a certain threshold (such as more than 100), the model performance gradually tends to saturate, and even in some cases there is a slight decrease. The reason for this is that the overfitting problem caused by the excessively high dimension. Therefore, the hidden dimension D = 100 is finally selected as a unified setting in the present application, in order to balance the representation ability and model complexity.

[0141] Second, as shown in Figure 11 Figure 11 (c-d), in order to evaluate the influence of the number of GAT layers in the graph encoding module on the inference performance, the number of GAT network layers is varied between 1 and 5 while keeping other parameters unchanged, and the influence trend on the performance indicators is observed. The experimental results show that increasing the number of graph neural network layers can effectively improve the aggregation depth and expression ability of structural information, and the model performs best on various indicators when the number of layers is set to 4. When the number of layers is further increased to 5, the model performance decreases slightly, which may be due to the “over-smoothing” phenomenon in the graph neural network, i.e., as the propagation depth increases, the representations of different nodes tend to be consistent, resulting in a decrease in feature discrimination and affecting the model performance. Therefore, the number of layers of the graph attention network in the present application is finally determined to be 4, in order to achieve a balance between structural information propagation and model stability.

[0142] Through the above parameter tuning experiments, the stability and robustness of the proposed model under different structural configurations are verified, and the optimal combination of hyperparameters is selected based on experimental data, thereby ensuring the maximization of the delayed inference performance and the rationality of the system configuration.

[0143] In an embodiment of the present application, to further improve the model interpretability and identify the key device factors affecting the delay prediction, a device factor contribution analysis mechanism based on feature inversion is introduced. This mechanism calculates the gradient of each attribute of the device on the model output delay value, quantifies the influence of each hardware factor in the delay inference task, and thus assists in optimizing the device configuration and scheduling strategy.

[0144] Specifically, during the model training process, the present application calculates the gradient value of each dimension in the device vector on the predicted delay, and according to the gradient amplitude, the importance of the factor is represented. To facilitate comparison and visualization, the gradient amplitude is processed by percentage using the normalization formula, which is defined as follows:

[0145] ;

[0146] wherein, represents the normalized importance of the th device factor, represents the absolute value of the gradient of the factor on the current sample, is the total number of device attributes. The normalized gradient can be used as the influence of each attribute on the predicted delay, reflecting its relative contribution in the model decision.

[0147] Table II Feature inversion analysis experiment

[0148]

[0149] Based on the above method, the feature importance of the device factors in the CPU and GPU datasets is evaluated. Among them, for the CPU dataset, to respectively represent the running in the desktop system, the CPU processor type, the maximum Turbo Boost frequency, the core number, the thread number, the 3-level cache and the maximum memory bandwidth. For the GPU dataset, ~ respectively represent the desktop system, the GPU processor type, the stream processor number, the core clock frequency, the video memory and the memory bus width. As shown in Table II, the analysis results are as follows:

[0150] (1) In the CPU device, the "Number of Cores" is the most influential factor, with a contribution of 33.48%; in the GPU device, the "Stream Processor Count" is the dominant factor, with a contribution of 26.43%. This result shows that the model can effectively identify the most critical hardware resource indicators for delay in different platforms; ​

[0151] (2) Compared with the basic device information (such as the running platform type, processor model, etc.), the fine-grained performance information (such as the main frequency, cache, thread number, bandwidth, etc.) is found to have a higher influence weight in delay prediction. For example, in the CPU data, the basic attributes The total contribution is less than 10%, and the main frequency, thread number and core number account for more than 70%. This result verifies the necessity and effectiveness of introducing a large language model to generate detailed performance indicators in the present application;

[0152] (3) In existing methods, such as bandwidth is often regarded as the main reference index, but the experimental results of the present application show that the contribution of CPU bandwidth (such as in the table ) in the whole is only 8.84%, which is much lower than that of "core number", "thread number" and other factors. This further proves that the method of the present application has obvious advantages in identifying more representative key factors;

[0153] (4) Such a feature gradient-based inversion mechanism not only helps to explain the prediction source of the model, but also can be used to guide the optimization and resource scheduling strategy design on the device side. For example, according to the analysis results, it is known that on the CPU platform, high-core devices are preferred, and on the GPU platform, high-SP (stream processor) configuration is preferred, which will have a positive impact on the DNN software execution efficiency.

[0154] Through the feature contribution analysis mechanism, in addition to the delay reasoning accuracy, the present application further realizes the explainable modeling capability of key hardware factors, provides a scientific basis for model deployment, system scheduling and device selection, and has significant engineering practicality.

[0155] Compared with the prior art, the technical effects of the present application are:

[0156] (1) Stronger neural network structure perception ability: the present application uses a graph attention mechanism to model the DNN structure, which can fully capture the topological features, structural dependency relationships and differences between serial / parallel execution paths in the model. Compared with traditional FLOPs estimation or hierarchical summation modeling methods, the graph structure expression combined with global nodes and multi-layer attention mechanism effectively enhances the model's representation ability for complex structures, thereby improving the delay prediction accuracy;

[0157] (2) Support for automatic and more informative device modeling mechanism: the present application first introduces a large language model as a device information generator to automatically obtain the core hardware indicators of the target device through natural language prompts, covering maximum turbo frequency, core number, thread number, cache capacity, video memory bandwidth and other attributes, solving the problem of device feature dimension scarcity and errors caused by manual selection in the prior art. This mechanism has high universality and migration, and is suitable for various computing platforms;

[0158] (3) Achieve high-precision delay prediction ability under small sample conditions: The structure-device joint modeling method proposed by the present application can accurately infer the unmeasured device and structure combination under the premise of using only a small number of measured samples. In the case of no more than 100 training samples, the normalized mean absolute error (NMAE) of delay prediction can still be controlled within 5%, which is significantly better than the existing method relying on large-scale data training, reducing the deployment cost and system cold start difficulty;

[0159] (4) Introduce the device factor inversion mechanism of explainability: The present application can quantify the influence degree of each device index in the delay prediction result by constructing a gradient-based feature inversion module. Through partial derivative and first-order Taylor expansion, the system can output the marginal influence intensity of each hardware attribute on the model performance, helping developers understand the performance bottleneck and providing data support for device selection and resource scheduling. This mechanism makes up for the shortcomings of existing technologies in explainability;

[0160] (5) Good cross-platform generalization ability and application adaptability: The present application method supports unified modeling among CPU, GPU, TPU, NPU, edge embedded platform and other heterogeneous devices, with structure independence and platform independence. At the same time, it can be seamlessly integrated into neural architecture search (NAS), intelligent inference deployment, edge computing scheduling and other application scenarios, with good industrial landing potential and engineering promotion value;

[0161] (6) Improve deployment efficiency and optimization feedback ability: The present application can not only predict the model delay, but also analyze which device factor has the greatest impact on delay, support the system to prioritize "performance sensitive" resource configuration scheme at the scheduling level, realize software and hardware collaborative optimization, improve inference efficiency and system stability, and promote the automation, controllability and intelligence of intelligent system deployment.

[0162] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the scope of the disclosed technology can be easily thought of by those skilled in the art, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A deep neural network delay method based on graph attention mechanism and large language model, characterized in that, Comprise the following: Model structure modeling: a deep neural network model M to be reasoned is subjected to graph attention network structure coding to obtain a model structure representation vector ; Device information acquisition and enhancement: constructed device prompt word Input into a pre-trained large language model, output the corresponding key performance attribute field of the device by the large language model, form device enhancement information ; Stitching device augmentation information and base information , obtaining device effective information ( | ); inputting the device effective information ( | ) to a feedforward network encoder to generate a uniform scale device representation vector ; wherein the key performance attribute field includes maximum turbo frequency, core number, thread number, cache capacity and display memory bandwidth; wherein the feedforward network encoder adopts a single-layer linear transformation structure; Multi-layer perceptron prediction: a predictor is obtained after a multi-layer perceptron model is trained and converged, and a model structure representation vector and a device representation vector The input vector obtained after splicing is taken as the input of the predictor, and an inference delay value is output .

2. The deep neural network delay method based on graph attention mechanism and large language model according to claim 1, wherein, The specific steps of the graph attention network structure coding deep neural network model M are as follows: A deep neural network model M is represented as a directed graph structure ; where nodes represent operation units, and edges represent data flow dependencies; for each node mapped to an embedding vector , forming an initial feature matrix ; wherein the embedding vectors are initialized by a trainable matrix , k is the total number of operation types, is the embedding dimension; Introducing global nodes in graph structures Connected to other nodes, constituting an augmented graph structure; The L-layer graph attention network encodes the enhanced graph structure, and each layer of node embedding The update mode is as follows: ; wherein is the node The attention weight of the node ​ output global node embedding vector of the global node model structure representation vector as a neural network .

3. The deep neural network delay method based on graph attention mechanism and large language model according to claim 1, wherein, The multi-layer perceptron training phase uses a loss function Model optimization is performed: , wherein represents a training sample set, is a regularization coefficient.

4. The deep neural network delay method based on graph attention mechanism and large language model according to claim 1, wherein, The obtained predictor model is subjected to gradient inversion and factor explanation as follows: for each device factor The partial derivative of the delay output is calculated, defined as: ; wherein, denotes the gradient amplitude of the delay output to device factor z; When the device factor is , the difference between the model loss functions of the first round and the second round is expressed as: ; The difference value change of the loss function is applied to the first-order Taylor expansion, and the high-order terms are ignored, and is expressed as: ； During the gradient descent process in the training process, the model parameters are updated to represent ； then the difference change in the loss function .

Images