DNN inference delay prediction method and system for heterogeneous mobile equipment

Abstract

The invention relates to a DNN inference delay prediction method and system for heterogeneous mobile equipment, and the method comprises the steps: constructing a data set, carrying out the enhancement processing of the data set, and carrying out the normalization of a performance counter; constructing a structure layer-hardware layer-delay layer three-layer mapping retrieval mechanism; constructing a delay prediction model, and calculating final delay through a GNN double-branch architecture; the dynamic knowledge base and the delay prediction model are used for cooperative work, and a prediction result is stored in the dynamic knowledge base. The method has the beneficial effects that compared with a traditional hardware actual measurement delay acquisition scheme or a single prediction model scheme, the method does not need to depend on real-time availability of target hardware and manual deployment operation; and the problems of high DNN reasoning delay acquisition cost and difficulty in cross-platform adaptation under the heterogeneous mobile equipment can be effectively solved only through a query and prediction dual-module collaborative software architecture.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and in particular relates to a DNN inference latency prediction method and system for heterogeneous mobile devices. Background Technology

[0002] With the rapid development of deep learning technology, DNNs have been widely deployed on mobile edge devices such as smartphones and tablets, providing core support for applications in fields such as computer vision and natural language processing. Unlike resource-rich cloud servers, mobile devices are limited by hardware resources, making DNN inference speed a key bottleneck for application deployment. Accurately predicting the inference latency of DNNs on mobile devices is a core prerequisite for optimizing model architecture design, improving hardware resource utilization, and enhancing user experience.

[0003] Currently, DNN inference latency prediction for mobile devices faces three core challenges: First, the hardware environment is highly heterogeneous. Mobile devices are equipped with various accelerators such as CPUs, GPUs, and ASICs, and different devices differ significantly in chip models, core architectures, and storage configurations, greatly increasing prediction complexity. Second, there are diverse inference framework optimization techniques. Frameworks such as TensorRT and OpenPPL employ strategies such as operator fusion and adaptive algorithm selection, which improve inference efficiency but make it difficult to model the model execution process and quantify its impact on latency. Third, the diversity of neural architectures continues to increase. New architectures such as Transformer variants and lightweight convolutional networks differ significantly in the number of operations, types of operations, and inter-layer connections, further increasing the difficulty of prediction.

[0004] Existing latency acquisition and prediction methods have significant limitations: methods based on hardware testing rely on hardware availability and manual deployment; multi-platform adaptation requires the installation of dedicated frameworks (e.g., CUDA and TensorRT for GPUs), resulting in cumbersome processes, low data reuse rates, and a single data acquisition cost of up to 30 minutes; prediction models such as TPU-Perf and nn-Meter employ an "operator splitting" strategy, assuming the total latency is the sum of the latencies of each operator, ignoring factors such as hardware parallel computing and overlapping memory accesses, leading to prediction bias; methods such as BRP-NAS are only effective within a specific search space, lack a general graph embedding mechanism, cannot adapt to networks with significantly different topologies, and have limited generalization capabilities. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a DNN inference latency prediction method and system for heterogeneous mobile devices.

[0006] Firstly, a method for predicting delayed DNN inference (MELP) for heterogeneous mobile devices is provided, including:

[0007] S1. Construct a dataset and perform enhancement processing and performance counter normalization on the dataset;

[0008] S2. Construct a dynamic knowledge base and use the dynamic graph hashing encoding method to encode the dataset built based on S1, and construct a three-layer mapping retrieval mechanism of structure layer - hardware layer - delay layer;

[0009] S3. Based on the dynamic knowledge base, construct a delay prediction model, extract node embedding vectors using a graph neural network (GNN), and calculate the final delay using a GNN dual-branch architecture.

[0010] S4. The dynamic knowledge base and the delay prediction model work together. When a delay query request is received, the dynamic knowledge base module is used for matching first. If the matching is successful, historical delay data is returned. If the matching fails, the delay prediction model is triggered to generate a prediction result and the prediction result is stored in the dynamic knowledge base.

[0011] Preferably, S1 includes:

[0012] S101. Construct a dataset, perform augmentation processing on the dataset, adopt dynamic targeted uniform sampling, sort and divide the architectures in the training pool based on end-to-end latency, and select samples from each interval.

[0013] S102. Normalize the performance counter and calculate the relative rate based on the measured delay of the corresponding operation.

[0014] Preferably, S2 includes:

[0015] S201. Introduce hardware adaptive feature vectors, which include hardware architecture, platform configuration and dynamic state information. Embed the feature vectors into node features to form enhanced node features and construct a labeled dataset.

[0016] S202. Design hardware-aware attention weights to increase the hash weight of nodes with the same hardware during dynamic graph hash encoding, thereby achieving joint encoding of structure and hardware.

[0017] Preferably, S3 includes:

[0018] S301. Based on the dynamic knowledge base, extract data features to provide input for the node embedding vector of the graph neural network (GNN) module of the delayed prediction model;

[0019] S302. Calculate the final latency using a GNN dual-branch architecture; the GNN dual-branch architecture includes a base branch and a context branch; the base branch uses GNN to extract structural features to generate the base latency, and the context branch uses a timing modeling network to model hardware timing to generate an adjustment factor, and the base latency and the adjustment factor are fused by a formula to obtain the final latency.

[0020] Preferably, S4 includes:

[0021] S401. Receive DNN inference delayed query request, parse model topology and hardware configuration, and preferentially match model-hardware pairs through the three-layer mapping retrieval mechanism of the dynamic knowledge base;

[0022] S402. If the match is successful, retrieve and return the historical delay data; if the match fails, trigger the delay prediction model.

[0023] S403. The delayed prediction model loads the ONNX model, extracts and normalizes the five-dimensional node features, generates prediction results through the GNN dual-branch method, and stores the effective results in the dynamic knowledge base.

[0024] In a second aspect, a method and system for predicting DNN inference latency for heterogeneous mobile devices are provided, for performing any of the methods described in the first aspect, including:

[0025] The first building module is used to build the dataset and perform enhancement processing and performance counter normalization on the dataset;

[0026] The second building module is used to build a dynamic knowledge base. It uses a dynamic graph hashing encoding method to encode the dataset built by the first building module and builds a three-layer mapping retrieval mechanism of structure layer-hardware layer-delay layer.

[0027] The third construction module is used to construct a delay prediction model based on the dynamic knowledge base, extract node embedding vectors with the help of graph neural network (GNN), and calculate the final delay through the GNN dual-branch architecture.

[0028] The collaboration module is used to work collaboratively with the dynamic knowledge base and the delay prediction model. When a delay query request is received, the dynamic knowledge base module is used for matching first. If the matching is successful, historical delay data is returned. If the matching fails, the delay prediction model is triggered to generate a prediction result, and the prediction result is stored in the dynamic knowledge base.

[0029] Thirdly, a computer storage medium is provided, wherein a computer program is stored therein; when the computer program is run on a computer, the computer causes the computer to perform any of the methods described in the first aspect.

[0030] Fourthly, an electronic device is provided, comprising:

[0031] Memory, used to store computer programs;

[0032] A processor for executing the computer program to implement the method as described in any of the first aspects.

[0033] The beneficial effects of this invention are:

[0034] 1. Compared with traditional hardware-based latency acquisition schemes or single prediction model schemes, this invention does not rely on the real-time availability of the target hardware or manual deployment operations. It can effectively solve the problems of high cost of DNN inference latency acquisition and difficulty in cross-platform adaptation under heterogeneous mobile devices by using a software architecture that coordinates query and prediction modules, which greatly reduces the system deployment complexity and data acquisition cost.

[0035] 2. This invention adopts a collaborative scheme of "dynamic graph hashing encoding + multi-platform pre-training", which can deeply integrate hardware characteristics and model structure features. Through dynamic hash space mapping and cross-platform knowledge transfer, it covers heterogeneous hardware scenarios such as CPU, GPU, and ASIC, improving the system's generalization ability and adaptation efficiency in cross-platform scenarios.

[0036] 3. This invention utilizes a GNN dual-branch architecture as the core latency modeling tool to accurately capture operator interaction relationships and dynamic changes in hardware timing, simplifying the latency prediction modeling process under complex architectures. At the same time, after training, the system can independently adapt to multiple model scenarios without adjusting the core logic for specific architectures, achieving efficient inference latency prediction and stable performance output. Attached Figure Description

[0037] Figure 1 This invention provides an architecture diagram of a DNN inference latency prediction system for heterogeneous mobile devices;

[0038] Figure 2 This is a schematic diagram of the dynamic graph hash encoding and three-layer mapping retrieval mechanism of the dynamic knowledge base provided by the present invention;

[0039] Figure 3 This is a flowchart of the GNN dual-branch architecture of the delayed prediction (MELP) module provided by the present invention;

[0040] Figure 4 This is a schematic diagram of the dataset collection range provided by the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0042] Example 1:

[0043] To address the issues of insufficient cross-platform compatibility and high latency acquisition costs in existing technologies, Embodiment 1 of this application provides a DNN inference latency prediction method for heterogeneous mobile devices. Through the collaborative use of query and prediction modules and dynamic graph hashing encoding, it achieves efficient and accurate prediction of DNN inference latency in heterogeneous hardware environments. Specifically, it includes:

[0044] S1. Construct a dataset and perform enhancement processing and performance counter normalization on the dataset.

[0045] S1 includes:

[0046] S101. Construct a dataset, perform augmentation processing on the dataset, adopt dynamic targeted uniform sampling, sort and divide the architectures in the training pool based on end-to-end latency, and select samples from each interval.

[0047] Specifically, the DNN model architectures of the collected devices are collected, and the end-to-end latency values ​​of each device and architecture combination are obtained through hardware testing. All architectures are sorted in ascending order of the measured latency values. Secondly, the sorted architecture distribution is divided into N equally spaced intervals according to training requirements to ensure that the hierarchical architecture of each type of device is evenly distributed in the N intervals. The number of architectures contained in a single interval is calculated according to the formula K = (M × S) / N (where M is the number of training devices and S is the total number of architectures in the initial training set). Finally, one adaptive architecture is randomly selected from each interval as a training sample.

[0048] S102. Normalize the performance counter and calculate the relative rate based on the measured delay of the corresponding operation.

[0049] Specifically, the raw values ​​of the performance counters of each operator in the target DNN model and the measured latency values ​​of the corresponding operators are collected synchronously. Secondly, the raw value of each performance counter is divided by the measured latency of the corresponding operator to obtain relative rate indicators such as "CPU cycles per millisecond" and "cache hits per millisecond". Finally, the relative rate indicators are mapped to the [0,1] interval through the Min-Max normalization formula to eliminate the scale difference of hardware performance characteristics and output normalized hardware performance characteristic data that can be directly input into the model, thereby improving the accuracy of the model in capturing hardware features.

[0050] S2. Construct a dynamic knowledge base, use dynamic graph hashing encoding to encode the labeled data, and build a three-layer mapping retrieval mechanism of structure layer - hardware layer - delay layer.

[0051] In S2, the original data needs to be denoised by repeated measurements (the same model-hardware pair is measured 5 times, and the mean is taken after removing extreme values) and filtered for outliers (data that deviates from the mean by 3 times the standard deviation is removed); multi-dimensional annotation is carried out by combining the model topology features (operator type, input shape) and hardware configuration (device model, inference framework) to build an annotated dataset.

[0052] S2 includes:

[0053] S201. Introduce hardware adaptive feature vectors, which include hardware architecture, platform configuration and dynamic state information. Embed the feature vectors into node features to form enhanced node features and construct a labeled dataset.

[0054] Specifically, a hardware adaptive feature vector H is introduced. hw (Including GPU Tensor Core support rate, inference framework version, and real-time load rate), forming enhanced node features F. v ; Calculate the hardware-aware attention weight α u,v Through formula Achieve joint encoding; construct a three-layer mapping mechanism to generate a triplet of "structural fingerprint-hardware encoding-latency value", which reduces the cross-platform hash collision rate by 42% and improves the retrieval accuracy in hardware migration scenarios by 35%.

[0055] S202. Design hardware-aware attention weights to increase the hash weight of nodes with the same hardware during dynamic graph hash encoding, thereby achieving joint encoding of structure and hardware.

[0056] S3. Based on the dynamic knowledge base, construct a delay prediction model, extract node embedding vectors using a graph neural network (GNN), and calculate the final delay using a GNN dual-branch architecture.

[0057] S3 includes:

[0058] S301. Extract five-dimensional node features, including operator type, attributes, output shape, hardware configuration and execution context, and normalize the five-dimensional node features to provide input for the node embedding vector of the graph neural network (GNN) module of the delay prediction model.

[0059] Specifically, using the ONNX model as input, five-dimensional node features F are extracted. v ,in It includes memory locality ratio (proportion of continuous memory accesses), hardware affinity score (operator's hardware compatibility index), and historical execution mean / variance of similar operators; the features are normalized (pixel values ​​are scaled to [0,1]) to improve model training stability.

[0060] S302. Calculate the final latency using a GNN dual-branch architecture; the GNN dual-branch architecture includes a base branch and a context branch; the base branch uses GNN to extract structural features to generate the base latency, and the context branch uses a timing modeling network to model hardware timing to generate an adjustment factor, and the base latency and the adjustment factor are fused by a formula to obtain the final latency.

[0061] Specifically, GraphSAGE is used as the backbone network of the GNN, and the basic branches aggregate features from neighboring nodes to generate... The context branch models the GPU utilization sequence (past 50ms) using LSTM to generate β. v ;pass The final latency is calculated, and a 22.4% prediction error is dynamically corrected to cope with hardware load fluctuations.

[0062] S4. The dynamic knowledge base and the delay prediction model work together. When a delay query request is received, the dynamic knowledge base module is used for matching first. If the matching is successful, historical delay data is returned. If the matching fails, the delay prediction model is triggered to generate a prediction result and the prediction result is stored in the dynamic knowledge base.

[0063] S4 includes:

[0064] S401. Receive DNN inference delayed query request, parse model topology and hardware configuration, and preferentially match model-hardware pairs through the three-layer mapping retrieval mechanism of the dynamic knowledge base;

[0065] S402. If the match is successful, retrieve and return the historical delay data; if the match fails, trigger the delay prediction model.

[0066] S403. The delayed prediction model loads the ONNX model, extracts and normalizes the five-dimensional node features, generates prediction results through the GNN dual-branch method, and stores the effective results in the dynamic knowledge base.

[0067] Specifically, after receiving a delayed query request, the dynamic knowledge base module first matches the "model-hardware pair". If a match is found, historical data is returned (response time ≤ 5ms). If a match is not found, the delayed prediction model is triggered (time 17-20ms / iter) to generate a prediction result, which is then stored in the knowledge base. In the "unseen structure" scenario (the test model does not participate in training), compared with five mainstream methods such as FLOPs, nn-Meter, and TPU-Perf, as shown in Tables 1 and 2, the average MAPE of this system reaches 11.37% (3.98 percentage points lower than nn-Meter), and the average Acc (10%) reaches 74.88% (8.69 percentage points higher than nn-Meter).

[0068] Table 1

[0069] Model FLOPs FLOPs+MAC nn-Meter TPU-Perf BRP-NAS OUR AlexNet 18.20％ 10.55％ 7.20％ 12.50％ 14.50％ 7.62％ EfficientNet 17.50％ 16.74％ 18.93％ 11.80％ 13.80％ 7.38％ GoogLeNet 35.60％ 8.10％ 11.71％ 29.40％ 25.48％ 24.04％ MNASNet 19.80％ 11.61％ 10.69％ 15.60％ 17.26％ 8.98％ MobileNetV2 28.50％ 12.68％ 6.43％ 21.20％ 20.42％ 15.60％ MobileNetV3 25.60％ 9.97％ 35.27％ 18.90％ 58.13％ 10.07％ NasBench201 38.20％ 58.94％ 9.57％ 26.70％ 13.28％ 10.73％ ResNet18 21.50％ 20.05％ 15.58％ 14.50％ 15.84％ 9.56％ SqueezeNet 22.40％ 24.60％ 18.69％ 16.80％ 42.55％ 10.76％ VGG16 32.00％ 38.73％ 19.47％ 24.00％ 30.95％ 8.93％ AVG MAPE 25.93％ 21.20％ 15.35％ 19.14％ 25.22％ 11.37％

[0070] Table 2

[0071] Model FLOPs FLOPs+MAC nn-Meter TPU-Perf BRP-NAS OUR AlexNet 12.50％ 28.50％ 75.47％ 27.30％ 35.60％ 71.84％ EfficientB0 10.80％ 25.60％ 68.23％ 55.60％ 38.20％ 72.19％ GoogLeNet 5.20％ 12.30％ 42.17％ 68.90％ 27.40％ 66.70％ MNASNet 8.70％ 22.40％ 70.45％ 58.70％ 34.50％ 72.78％ MobileNetV2 15.30％ 35.70％ 85.60％ 42.30％ 45.60％ 77.34％ MobileNetV3 9.60％ 28.90％ 55.34％ 75.40％ 32.10％ 71.54％ NasBench201 3.20％ 18.60％ 72.40％ 32.50％ 41.20％ 94.94％ ResNet18 11.50％ 32.50％ 69.82％ 27.30％ 39.80％ 71.11％ SqueezeNet 14.20％ 30.40％ 58.90％ 12.60％ 36.70％ 75.32％ VGG16 4.80％ 16.70％ 63.50％ 53.60％ 29.50％ 75.08％ AVGAcc (10%) 9.60％ 24.30％ 66.19％ 54.90％ 35.30％ 74.88％

[0072] Example 2:

[0073] Based on Embodiment 1, Embodiment 2 of this application provides a DNN inference latency prediction system for heterogeneous mobile devices, comprising:

[0074] The first building module is used to build the dataset and perform enhancement processing and performance counter normalization on the dataset;

[0075] The second building module is used to build a dynamic knowledge base. It uses a dynamic graph hashing encoding method to encode the dataset built by the first building module and builds a three-layer mapping retrieval mechanism of structure layer-hardware layer-delay layer.

[0076] The third construction module is used to construct a delay prediction model based on the dynamic knowledge base, extract node embedding vectors with the help of graph neural network (GNN), and calculate the final delay through the GNN dual-branch architecture.

[0077] The collaboration module is used to work collaboratively with the dynamic knowledge base and the delay prediction model. When a delay query request is received, the dynamic knowledge base module is used for matching first. If the matching is successful, historical delay data is returned. If the matching fails, the delay prediction model is triggered to generate a prediction result, and the prediction result is stored in the dynamic knowledge base.

[0078] It should be noted that the system provided in this embodiment is the corresponding system of the method provided in embodiment 1. Therefore, the parts that are the same as or similar to those in embodiment 1 in this embodiment can be referred to each other, and will not be described again in this application.

[0079] Example 3:

[0080] Building upon Example 2, Example 3 of this application provides a more specific DNN inference latency prediction system for heterogeneous mobile devices, including:

[0081] Data processing module: Image denoising (5×5 Gaussian filter kernel) and outlier filtering (Z-score method) are implemented using Python OpenCV library. Annotated data (including model topology, hardware configuration, and latency values) are stored in JSON format, and a structured labeled dataset is output.

[0082] Encoding and retrieval module: Implements dynamic graph hashing encoding based on PyTorch, uses Elasticsearch to build a three-layer mapping index, supports 1000+ query requests per second, and outputs efficient retrieval results;

[0083] Feature extraction module: Uses ONNX Runtime to parse ONNX model, extracts five-dimensional node features, and outputs feature vectors through Min-Max normalization (scaled to [0,1]);

[0084] Delay prediction module: Implements a dual-branch GraphSAGE and LSTM based on PyTorch Geometric, uses CUDA 11.4 to accelerate computation, and outputs the delay prediction value (unit: ms);

[0085] Collaborative control module: It uses FastAPI to build the interface, receives HTTP query requests, schedules the work of the two modules, and uses MongoDB to record prediction results and knowledge base update logs to ensure system traceability.

[0086] Furthermore, Embodiment 3 of this application also provides a method for deploying and implementing transfer learning of the above system, including:

[0087] S1. System Deployment: A hardware testing platform was built using aluminum profiles, and 10 types of mobile devices were fixed in place (20cm apart, 1.5m high). TensorRT 7.1, OpenPPL 0.8 and other inference frameworks were installed, and CUDA 11.4 (GPU device), OpenCV 4.5 and other dependent libraries were configured. A dynamic knowledge base module (based on a MySQL database, with Redis-optimized indexes) and a MELP module (based on the PyTorch framework, with GNN implemented using PyTorch Geometric) were deployed to ensure that the data interaction latency between modules was ≤1ms.

[0088] S2, Transfer Learning Validation

[0089] S201, Pre-training: The system is trained using a dataset containing 20,000 models (covering 10 families including VGG and ResNet, excluding ConvNeXt) to learn general graph embedding and hardware-operator interaction rules;

[0090] S202, Fine-tuning: The dynamic graph hashing module was fixed, and the GNN dual branches and prediction heads were fine-tuned using 16, 32, 64, 128, 512, and 1000 samples from the ConvNeXt family. The results showed that when the sample size was 50, the Acc (10%) of this system reached 72.3%, while training from scratch required 500 samples to reach 70.1%, improving data efficiency by 10 times. The average Acc (10%) on 9 hardware platforms was 15-20 percentage points higher than that of training from scratch, verifying the effectiveness of transfer learning.

[0091] S203. Loss Function Optimization: Constructing the total loss function Loss = Loss reproj +λ1Loss disparity +λ2Loss reg Loss reproj Loss is the 3D point reprojection error (a measure of the deviation between predicted parameters and the actual projection). disparity For parallax consistency loss (satisfying binocular horizontal epipolar constraints), Loss reg The regularization term is used to prevent parameter anomalies. The Adam optimizer (learning rate 1e-4, weight decay 1e-5) is used to jointly optimize the network parameters. After training for 50 epochs, the model converges and has the best generalization ability.

[0092] It should be noted that the parts in this embodiment that are the same as or similar to those in Embodiment 2 can be referred to each other, and will not be repeated in this application.

[0093] In summary, this invention proposes a dual-module system of "query-prediction" that integrates a dynamic knowledge base and a graph neural network (GNN). It achieves joint representation of hardware-model features through dynamic graph hashing encoding, enhances cross-platform generalization capabilities by relying on pre-training and transfer learning, and ultimately achieves efficient and accurate prediction of DNN inference latency on heterogeneous mobile devices.

Claims

1. A DNN inference latency prediction method for heterogeneous mobile devices, characterized in that, include: S1. Construct a dataset and perform enhancement processing and performance counter normalization on the dataset; S2. Construct a dynamic knowledge base and use the dynamic graph hashing encoding method to encode the dataset built based on S1, and construct a three-layer mapping retrieval mechanism of structure layer - hardware layer - delay layer; S3. Based on the dynamic knowledge base, construct a delay prediction model, extract node embedding vectors using a graph neural network (GNN), and calculate the final delay using a GNN dual-branch architecture. S4. The dynamic knowledge base and the delay prediction model work together. When a delay query request is received, the dynamic knowledge base module is used for matching first. If the matching is successful, historical delay data is returned. If the matching fails, the delay prediction model is triggered to generate a prediction result and the prediction result is stored in the dynamic knowledge base.

2. The DNN inference latency prediction method for heterogeneous mobile devices according to claim 1, characterized in that, S1 includes: S101. Construct a dataset, perform augmentation processing on the dataset, adopt dynamic targeted uniform sampling, sort and divide the architectures in the training pool based on end-to-end latency, and select samples from each interval. S102. Normalize the performance counter and calculate the relative rate based on the measured delay of the corresponding operation.

3. The DNN inference latency prediction method for heterogeneous mobile devices according to claim 2, characterized in that, S2 include: S201. Introduce hardware adaptive feature vectors, which include hardware architecture, platform configuration and dynamic state information. Embed the feature vectors into node features to form enhanced node features and construct a labeled dataset. S202. Design hardware-aware attention weights to increase the hash weight of nodes with the same hardware during dynamic graph hash encoding, thereby achieving joint encoding of structure and hardware.

4. The DNN inference latency prediction method for heterogeneous mobile devices according to claim 3, characterized in that, S3 include: S301. Based on the dynamic knowledge base, extract data features to provide input for the node embedding vector of the graph neural network (GNN) module of the delayed prediction model; S302. Calculate the final latency using a GNN dual-branch architecture; the GNN dual-branch architecture includes a base branch and a context branch; the base branch uses GNN to extract structural features to generate the base latency, and the context branch uses a timing modeling network to model hardware timing to generate an adjustment factor, and the base latency and the adjustment factor are fused by a formula to obtain the final latency.

5. The DNN inference latency prediction method for heterogeneous mobile devices according to claim 4, characterized in that, S4 include: S401. Receive DNN inference delayed query request, parse model topology and hardware configuration, and preferentially match model-hardware pairs through the three-layer mapping retrieval mechanism of the dynamic knowledge base; S402. If the match is successful, retrieve and return the historical delay data; if the match fails, trigger the delay prediction model. S403. The delayed prediction model loads the ONNX model, extracts and normalizes the five-dimensional node features, generates prediction results through the GNN dual-branch method, and stores the effective results in the dynamic knowledge base.

6. A DNN inference latency prediction system for heterogeneous mobile devices, characterized in that, For performing the method according to any one of claims 1 to 5, comprising: The first building module is used to build the dataset and perform enhancement processing and performance counter normalization on the dataset; The second building module is used to build a dynamic knowledge base. It uses a dynamic graph hashing encoding method to encode the dataset built by the first building module and builds a three-layer mapping retrieval mechanism of structure layer-hardware layer-delay layer. The third construction module is used to construct a delay prediction model based on the dynamic knowledge base, extract node embedding vectors with the help of graph neural network (GNN), and calculate the final delay through the GNN dual-branch architecture. The collaboration module is used to work collaboratively with the dynamic knowledge base and the delay prediction model. When a delay query request is received, the dynamic knowledge base module is used for matching first. If the matching is successful, historical delay data is returned. If the matching fails, the delay prediction model is triggered to generate a prediction result, and the prediction result is stored in the dynamic knowledge base.

7. A computer storage medium, characterized in that, The computer storage medium stores a computer program; when the computer program is run on the computer, it causes the computer to perform the method described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1 to 5.

Images