A method and apparatus for evaluating the inference performance of edge-side deep learning models

Abstract

This disclosure provides a method and apparatus for evaluating the inference performance of an edge-side deep learning model, belonging to the field of artificial intelligence technology. The method includes: obtaining model parameters of the deep learning model to be evaluated; obtaining hardware parameters of the edge device used to run the deep learning model; obtaining environmental parameters of the deep learning model running on the edge device; collecting values ​​of multiple performance indicators of the edge device while the deep learning model is running on the edge device; calculating the value of each performance indicator using a normalization method to obtain a single performance score corresponding to each performance indicator; obtaining the performance indicator weight corresponding to each performance indicator based on the model parameters, hardware parameters, and environmental parameters; and obtaining the evaluation result of the deep learning model's inference performance based on the weights of the multiple performance indicators and the corresponding single performance score. The evaluation result can accurately indicate the running performance of the edge device when the deep learning model runs on the edge device.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method and apparatus for evaluating the inference performance of edge deep learning models. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning models are being deployed more and more widely on edge devices (including smartphones). To ensure user experience and system stability, it is crucial to accurately evaluate the inference performance of the model in a real edge environment.

[0003] Currently, the common method for evaluating the inference performance of deep learning models deployed on edge devices is as follows: While the deep learning model is running on the edge device, the device's built-in Performance Monitoring Unit (PMU) or tools provided by the operating system directly collect performance metrics such as processor utilization, memory usage, power consumption, and temperature rise. These independent performance metrics are then used directly as the evaluation result for inference performance. This method obtains accurate performance metrics, but because the evaluation result represents multiple performance metrics, it requires manual evaluation based on these metrics to arrive at a comprehensive assessment of the deep learning model's inference performance. This assessment, relying on human experience, has relatively low accuracy. Summary of the Invention

[0004] This disclosure provides a method and apparatus for evaluating the inference performance of a deep learning model on an edge device, which can obtain the inference performance of the deep learning model when running on an edge device, and the evaluation result of the inference performance is more accurate.

[0005] According to one aspect of the present disclosure, a method for evaluating the inference performance of an edge-side deep learning model is provided. The method includes: acquiring model parameters of a deep learning model to be evaluated, wherein the model parameters indicate parameters of the model's features; acquiring hardware parameters of an edge device used to run the deep learning model, wherein the hardware parameters indicate parameters of the device's hardware features; acquiring environmental parameters of the deep learning model running on the edge device, wherein the environmental parameters indicate parameters of the running environment features; collecting values ​​of multiple performance indicators of the edge device while the deep learning model is running on the edge device, wherein the performance indicators indicate the operating performance of the corresponding device; calculating a single performance score corresponding to each performance indicator by using a normalization method on the value of each performance indicator; acquiring a performance indicator weight corresponding to each performance indicator based on the model parameters, the hardware parameters, and the environmental parameters, wherein the performance indicator weight indicates the importance of the performance indicator in the inference performance of the deep learning model; and obtaining an evaluation result of the deep learning model's inference performance based on the multiple performance indicator weights and the corresponding single performance scores.

[0006] In some embodiments, obtaining the performance index weights corresponding to the performance indexes based on the model parameters, the hardware parameters, and the environmental parameters includes: inputting the model parameters, the hardware parameters, and the environmental parameters into a trained weight prediction model to obtain the performance index weights output by the weight prediction model.

[0007] In some embodiments, the training process of the weight prediction model includes: acquiring a training set, the training set including multiple sets of training data, each set of training data including model parameters of a sample model, hardware parameters of a sample device, environmental parameters of the sample model running on the sample device, and label data, wherein the category of the sample model is the same as the category of the deep learning model, the category of the sample device is the same as the category of the edge device, the hardware of the edge device executing the inference process of the deep learning model is the same as the hardware of the sample device executing the inference process of the sample model, and the label data includes a preset indicator weight for each performance indicator; iteratively training the weight prediction model based on the training set to obtain the trained weight prediction model; wherein any iteration in the iterative training includes: for each set of training data, calling the weight prediction model to make a prediction based on the model parameters, the hardware parameters, and the environmental parameters to obtain a predicted performance indicator weight for each performance indicator; calculating a prediction loss based on the predicted performance indicator weight and the preset indicator weight; and adjusting the model parameters of the weight prediction model based on the prediction loss.

[0008] In some embodiments, the ratio of the maximum to the minimum value of the total number of model parameters in different sample models is less than 1.2, and the ratio of the total number of model parameters in the deep learning model to the total number of model parameters in the sample model is 0.83-1.2.

[0009] In some embodiments, when the category of the deep learning model is a large language model, the performance metrics include memory usage, inference latency, power consumption, and temperature rise. The memory usage indicates the maximum memory occupied by the deep learning model on the edge device during operation. The inference latency indicates the time from input data to output result for the deep learning model. The power consumption indicates the average power consumption of the deep learning model during inference. The temperature rise indicates the temperature difference of the hardware before and after the inference process.

[0010] In some embodiments, the method further includes: obtaining a bottleneck indicator, wherein the bottleneck indicator indicates the performance indicator with the lowest single performance score or the highest weight of the performance indicator, and the bottleneck indicator is used as the basis for generating optimization suggestions.

[0011] According to another aspect of the present disclosure, an edge-side deep learning model inference performance evaluation apparatus is provided, the apparatus comprising: The model parameter extraction module is used to obtain the model parameters of the deep learning model to be evaluated, wherein the model parameters indicate the parameters of the model's features; A hardware parameter extraction module is used to obtain hardware parameters of the edge device used to run the deep learning model, wherein the hardware parameters indicate the hardware characteristics of the device. An environment parameter extraction module is used to obtain the environment parameters of the deep learning model when it runs on the edge device. The environment parameters indicate parameters corresponding to the characteristics of the running environment, and the performance indicators indicate the running performance of the device. The performance metric acquisition module is used to acquire the values ​​of multiple performance metrics of the edge device when the deep learning model is running on the edge device, and the performance metrics indicate the operating performance of the corresponding device. The normalization module is used to calculate the value of each of the performance indicators by normalization method to obtain the individual performance score corresponding to each of the performance indicators. The weight acquisition module is used to acquire the performance index weight corresponding to each performance index based on the model parameters, the hardware parameters and the environment parameters. The performance index weight indicates the importance of the performance index in the inference performance of the deep learning model. The comprehensive score acquisition module is used to obtain a comprehensive score based on the weights of the multiple performance indicators and the corresponding individual performance scores.

[0012] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: one or more processors; a memory for storing processor-executable program code; wherein the processor is configured to execute the program code to implement the above-described method.

[0013] According to another aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of a computer device, enables the computer device to perform the above-described method.

[0014] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the above-described method.

[0015] This disclosure provides a method for evaluating the inference performance of a deep learning model on an edge device. When the deep learning model runs on an edge device, it comprehensively considers the model parameters of the deep learning model, the hardware parameters of the edge device, and the runtime environment parameters, improving the accuracy of the obtained performance indicator weights. Simultaneously, it performs individual performance scoring on the collected performance indicator values. Based on the performance indicator weights and individual performance scores, a more accurate comprehensive score is calculated, providing an evaluation result (i.e., a comprehensive score) for the inference performance of the deep learning model running on the edge device. Since inference performance is the comprehensive performance of multiple performance indicators of the edge device during the inference process of the deep learning model, and each of these performance indicators is affected by the running performance of the edge device, the inference performance is also the running performance of the edge device. Therefore, this evaluation result can more accurately indicate the running performance of the edge device when the deep learning model runs on it. A high evaluation score indicates that the deep learning model has a small impact on the performance of the edge device, and the edge device performs well. This means that the deep learning model is suitable for deployment on the edge device and has good inference performance. Conversely, a low evaluation score indicates that the deep learning model has a significant impact on the performance of the edge device, and the edge device performs poorly, potentially posing security risks. This means that the deep learning model is not yet suitable for deployment on the edge device, and at least one of the deep learning model or the edge device needs to be optimized.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0018] Figure 1 This is a schematic diagram illustrating the implementation environment of an edge-side deep learning model inference performance evaluation method according to an exemplary embodiment.

[0019] Figure 2 This is a flowchart illustrating an edge-side deep learning model inference performance evaluation method according to an exemplary embodiment.

[0020] Figure 3 This is a flowchart illustrating another method for evaluating the inference performance of an edge-side deep learning model according to an exemplary embodiment.

[0021] Figure 4 This is a data relationship diagram illustrating another edge-side deep learning model inference performance evaluation method according to an exemplary embodiment.

[0022] Figure 5 This is a block diagram illustrating an edge-side deep learning model inference performance evaluation device according to an exemplary embodiment.

[0023] Figure 6 This is a block diagram illustrating a terminal according to an exemplary embodiment.

[0024] Figure 7 This is a schematic diagram of the structure of a server according to an exemplary embodiment. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0026] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0027] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0028] Figure 1 This is a schematic diagram illustrating an implementation environment for an edge-side deep learning model inference performance evaluation method according to an exemplary embodiment. Taking an electronic device as an example, see [link to example]. Figure 1 The implementation environment specifically includes: terminal 101, end-side device 102, and server 103.

[0029] Terminal 101 is at least one of the following devices: industrial control computer, embedded computer, integrated computer, desktop computer, laptop computer, and mobile laptop. An application supporting the evaluation method of this disclosure runs on terminal 101. Users can log in to this application through terminal 101 to access the services provided by the application. For example, users can use this application on terminal 101 to evaluate the inference performance of edge-side deep learning models.

[0030] The edge device 102 can be a smartphone, vehicle infotainment system (i.e., vehicle computer), tablet computer, laptop computer, industrial sensor, smart camera, etc., and the deep learning model to be evaluated is installed on the edge device 102.

[0031] Terminal 101 and edge device 102 can be directly or indirectly connected via wired or wireless communication, so that the application on terminal 101 can obtain the hardware parameters of edge device 102, obtain the model parameters of the deep learning model, obtain the environmental parameters of the deep learning model running on the edge device, obtain the performance index values ​​of edge device 102, etc.

[0032] A weight prediction model runs on terminal 101. The application on terminal 101 can input hardware parameters, model parameters and environmental parameters into the weight prediction model, obtain the performance index weight corresponding to each performance index output by the weight prediction model, and obtain the evaluation result of the deep learning model inference performance based on the performance index weight and the performance index value.

[0033] Server 103 provides backend services for training the weight prediction model. Server 103 can connect to terminal 101 via a wireless or wired network. Server 103 can provide database services to the terminal, and of course, other types of services, such as data preprocessing. In some embodiments, the number of servers may be more or fewer, and this disclosure does not limit this.

[0034] The aforementioned application can be a standalone application for implementing a method to evaluate the inference performance of an edge deep learning model, or it can be integrated into an AI development platform. This application integrates multiple data acquisition interfaces, enabling the collection of performance characteristic values ​​from edge devices 102 with different operating systems (Android, iOS, Linux, etc.) and hardware.

[0035] Figure 2 This is a flowchart illustrating an edge-side deep learning model inference performance evaluation method according to an exemplary embodiment. See also: Figure 2 In this embodiment, the method is executed by an electronic device, and the method includes the following steps: In step S201, the model parameters of the deep learning model to be evaluated are obtained. The model parameters indicate the parameters of the model's features.

[0036] In step S202, the hardware parameters of the edge device used to run the deep learning model are obtained. The hardware parameters indicate the parameters of the device's hardware characteristics.

[0037] In step S203, the environmental parameters of the deep learning model running on the edge device are obtained. The environmental parameters indicate the parameters of the running environment characteristics.

[0038] In step S204, when the deep learning model is running on the edge device, the values ​​of multiple performance indicators of the edge device are collected. The performance indicators indicate the running performance of the corresponding device.

[0039] In step S205, the value of each performance indicator is calculated using a normalization method to obtain the individual performance score corresponding to each performance indicator.

[0040] In step S206, based on the model parameters, hardware parameters, and environment parameters, the performance index weight corresponding to each performance index is obtained. The performance index weight indicates the importance of the performance index in the inference performance of the deep learning model.

[0041] In step S207, the evaluation results of the deep learning model inference performance are obtained based on the weights of multiple performance indicators and the corresponding individual performance scores.

[0042] In the method disclosed herein, the performance index weights corresponding to each performance index are obtained based on hardware parameters, model parameters, and environmental parameters. In the above process, the impact of the three aspects (deep learning model, edge device, and operating environment) on the performance index weights is comprehensively considered, resulting in a more comprehensive consideration and higher accuracy of the obtained performance index weights. This improves the accuracy of the evaluation results. Since the evaluation results indicate the inference performance of the deep learning model and the operating performance of the edge device, it also improves the accuracy of the inference performance of the deep learning model when running on the edge device, as well as the accuracy of the deep learning model's performance on the edge device. This provides more accurate guidance for optimizing the deep learning model and for determining whether the deep learning model can be deployed on the edge device.

[0043] Meanwhile, by using a normalization method, the values ​​of performance metrics are projected onto the same numerical range, thus unifying the quantification standards for performance metrics. Furthermore, for the same type of deep learning model, the number of attributes collected for the performance metrics is the same, resulting in a more unified evaluation dimension for the same type of deep learning model. In addition to providing the evaluation results, this method also provides the individual performance score for each performance metric and the score of each performance metric in the evaluation results (this score is the product of the individual performance score and the performance metric weight). This allows us to determine which performance metric(s) scores are lower or higher, leading to superior or inferior inference performance of the deep learning model on the edge device, thus enhancing the interpretability of this method. Moreover, considering specific application scenarios (a certain type of deep learning model running on a certain type of edge device), the performance metric weights for each performance metric are obtained based on the hardware parameters obtained from the edge device, the model parameters obtained from the deep learning model, and the environmental parameters. In other words, the performance metric weights in this method are obtained based on the corresponding application scenario, thus providing good flexibility and naturally supporting the needs of different application scenarios.

[0044] Figure 3 This is a flowchart illustrating another method for evaluating the inference performance of an edge-side deep learning model according to an exemplary embodiment. Figure 4 This is a data relationship diagram illustrating another edge-side deep learning model inference performance evaluation method according to an exemplary embodiment. See also: Figure 3 and Figure 4 In this embodiment, the method is executed by an electronic device, and the method includes the following steps: In step S301, the model parameters of the deep learning model to be evaluated are obtained. The model parameters indicate the parameters of the model's features.

[0045] The model parameters obtained in this step can describe the features of the deep learning model, which makes it easier to obtain the performance index weights corresponding to each performance index based on the model parameters. Furthermore, the influence of the model parameters on the performance index weights is taken into account, which helps to improve the accuracy of the performance index weights obtained in the subsequent steps.

[0046] The model file format of the deep learning model to be evaluated can be any format, such as .onnx or .tflite.

[0047] The model parameters include at least one of the following: structural data, operator type data, and computational graph topology data. Structural data includes parameters (number of parameters) and FLOPs (Floating-Point Operations Per Second). Operator type data includes the proportion of specific operators (e.g., the proportion of Conv (Convolutional) layers). In summary, the model parameters include: Width Multiplier (WM), number of parameters, FLOPs, number of layers, proportion of specific operators, quantization precision, and sparsity. Quantization precision includes BF16, FP16, and INT8, where BF16, FP16, and INT8 represent different numerical precision formats. For example, the model parameters include: 2.1M parameters, 0.5G FLOPs, and 15 Conv layers.

[0048] In step S302, the hardware parameters of the edge device used to run the deep learning model are obtained. The hardware parameters indicate the parameters of the device's hardware characteristics.

[0049] The hardware parameters obtained in this step can describe the hardware performance of the edge device, which facilitates the subsequent acquisition of the performance indicator weights corresponding to each performance indicator based on the hardware parameters. Furthermore, the influence of hardware parameters on the performance indicator weights is taken into account, which helps to improve the accuracy of the performance indicator weights obtained subsequently.

[0050] Among them, the end-side device can be a certain model of smartphone, in-vehicle system, tablet, laptop, etc.

[0051] The hardware parameters include: processor computing power (TOPS (Tera Operations Per Second)), memory bandwidth (GB / s), inference framework, memory capacity, cache size, and energy efficiency rating. The processor type must be at least one of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Network Processing Unit). For example, the hardware parameters include: CPU clock speed = 2.8GHz, memory bandwidth = 17.6GB / s, and NPU supporting INT8.

[0052] In step S303, the environmental parameters of the deep learning model running on the edge device are obtained. The environmental parameters indicate the parameters of the running environment.

[0053] The environmental parameters obtained in this step can describe the operating environment, which makes it easier to obtain the performance index weight corresponding to each performance index based on the environmental parameters. Furthermore, the influence of environmental parameters on the performance index weights is taken into account, which helps to improve the accuracy of the performance index weights obtained later.

[0054] The environmental characteristics include background load and ambient temperature. For example, the environmental parameters include: number of background applications = 3, ambient temperature = 25℃.

[0055] In step S304, when the deep learning model is running on the edge device, the values ​​of multiple performance indicators of the edge device are collected. The performance indicators indicate the running performance of the corresponding device.

[0056] This step collects measured performance metrics of the deep learning model running on edge devices. The obtained performance metrics accurately reflect the operating performance of the edge devices and serve as the basis for further evaluation of the deep learning model's inference performance.

[0057] In some embodiments, when the deep learning model is classified as a large language model, performance metrics include PMU (Peak Memory Usage), Lat (Latency), APC (Average Power Consumption), and ΔT (Temperature Rise). Memory usage indicates the maximum memory occupied by the deep learning model on the edge device during operation; inference latency indicates the time from input data to output result; power consumption indicates the average power consumption of the deep learning model during inference; and temperature rise indicates the temperature difference of the hardware before and after inference. For large language models, by comprehensively considering these performance metrics, a relatively accurate evaluation of the inference performance of the large language model can be achieved.

[0058] The performance metrics can be collected using system tools and sensors on the edge device. For example, when the edge device is a smartphone running Android, to collect memory usage data, the ` / proc / [pid] / status` file and the `dumpsys meminfo` command are used to monitor memory usage during deep learning model loading and inference, recording the maximum memory usage on the edge device. Similarly, when the edge device is a smartphone, to collect temperature rise data, the smartphone's internal temperature sensor reads data from the SoC (System On Chip), recording the SoC's temperature T0 at the start of the test and the highest temperature T1 after a set number of inference tests; therefore, the temperature rise is T1-T0.

[0059] In step S305, the value of each performance indicator is calculated using a normalization method to obtain the individual performance score corresponding to each performance indicator.

[0060] This normalization step converts performance metric values ​​into standardized scores (e.g., 0-100). These standardized scores serve as individual performance ratings, ensuring a unified standard for quantifying performance metric values. Furthermore, these individual performance ratings generally reflect the numerical performance of the corresponding metric; that is, better performance results in a higher score. For example, lower inference latency and lower power consumption both correspond to higher individual performance ratings.

[0061] For example, when the performance metric is memory usage, the normalization method includes: taking the difference between the memory usage value and the baseline memory as the excess memory value, where the baseline memory indicates the minimum memory value required for the deep learning model to run normally; obtaining a deduction value based on the excess memory value, a tolerance threshold, and a memory penalty coefficient, where the tolerance threshold is the maximum allowed excess memory, and the memory penalty coefficient indicates the score deducted for each unit of memory exceeding the tolerance threshold; if the deduction value is greater than 0, using the difference between the set maximum score value and the deduction value as the single performance score for memory usage; if the deduction value is less than 0, using the set maximum score value as the single performance score for memory usage, where the set maximum score value is the maximum score value for the single performance score.

[0062] In other words, the single performance score for memory usage, Score_mem, is calculated using the following formula: Formula 1: Score_mem=100-max(0,(Memory_used-Base_mem)÷Tolerance_mem×Penalty_factor) Here, `Base_mem` represents the baseline memory, for example, 50MB; `Memory_used` represents the memory usage value; `Tolerance_mem` is the tolerance threshold, for example, 10MB; `Penalty_factor` is the memory penalty coefficient, for example, 10 points / 10MB, meaning that for every 10MB the memory usage exceeds the baseline memory, 10 points are deducted; and the maximum score is set to 100 points. This method converts the memory usage value into a score between 0 and 100 points, achieving a unified quantification standard.

[0063] For example, when the performance metric is inference latency, the normalization method includes: obtaining a single performance score for inference latency based on the value of inference latency, the attenuation coefficient, and a set maximum score; or, obtaining a single performance score for inference latency based on the value of inference latency, the delay time corresponding to the set score benchmark, and a set maximum score.

[0064] That is, the following Formula 2 is used to calculate the individual performance score Score_lat for inference latency: Formula 2: Score_lat=100×exp(-k×Latency) Where Latency is the numerical value of inference latency; k is the decay coefficient, for example, k=0.02. This formula uses an exponential decay function, where 100 points are awarded when the inference latency is 0, and the score drops rapidly as the inference latency increases. This method takes into account the importance of low inference latency.

[0065] Alternatively, the following Formula 3 can be used to calculate the individual performance score Score_lat for inference latency: Formula 3: Score_lat = 100 ÷ (1 + exp(k × (Latency - BM))) Wherein, Latency is the numerical value of inference latency; BM is the delay time corresponding to the set score benchmark. For example, when BM is 30ms, it means that the delay time corresponding to the 50-point benchmark is 30ms, and this benchmark is extremely sensitive to inference latency; k is the attenuation coefficient. This formula uses the Sigmoid function to map any inference latency value to the (0,100) interval and improves the sensitivity to inference latency.

[0066] For example, when the performance metric is power consumption, the normalization method includes: obtaining the power consumption percentage based on the power consumption value and the maximum power of the edge device; or, taking the difference between the power consumption value and the target power consumption as the additional power consumption, obtaining the power consumption percentage based on the additional power consumption and the additional power consumption threshold; obtaining a single performance score for power consumption based on the power consumption percentage and a set full score value; the target power consumption indicates the theoretical power consumption when the deep learning model is running normally, and the additional power consumption threshold indicates the maximum value of the preset additional power consumption.

[0067] That is, the power consumption performance score Score_pwr is calculated using the following formula four: Formula 4: Score_pwr=100×(1-min(1,Power÷Max_power)) Where Power is the power consumption value; Max_power is the maximum power of the end-side device.

[0068] Alternatively, the following Formula 5 can be used to calculate the single performance score for power consumption, Score_pwr: Formula 5: Score_pwr=100×max(0,1-Power_extra÷TH) Where Power_extra is the extra power consumption, and TH is the extra power consumption threshold.

[0069] This method converts power consumption values ​​into scores between 0 and 100, achieving a unified quantification standard.

[0070] For example, when the performance indicator is temperature rise, the normalization method includes: obtaining a single performance score for temperature rise based on the value of temperature rise, temperature penalty coefficient, and a set full score value, wherein the temperature penalty coefficient indicates the score value deducted for each degree of temperature change in the temperature rise.

[0071] That is, the individual performance score Score_temp for temperature rise is calculated using the following formula six: Formula 6: Score_temp=max(0,100-ΔT×Penalty_per_degree) Where ΔT is the numerical value of temperature rise; Penalty_per_degree is the temperature penalty coefficient, for example, the temperature penalty coefficient can be set to 15 points / ℃, that is, 15 points are deducted for every 1℃ increase in temperature. Through this method, the numerical value of temperature rise is converted into a score between 0 and 100 points, realizing the standardization of quantification.

[0072] The above methods for obtaining individual performance scores for memory usage, inference latency, power consumption, and temperature rise are only descriptions of the principles of one or two implementation methods. There may be other implementation methods, which are not specifically limited in this disclosure.

[0073] In some embodiments, the method further includes: obtaining a first configuration file and adjusting the calculation formula for individual performance scores based on the information in the first configuration file. Using the first configuration file facilitates the updating and maintenance of the calculation formula for individual performance scores. This step can be performed before each implementation of the method to ensure that the calculation formula for the corresponding individual performance scores is already maintained when evaluating the inference performance of any deep learning model running on any edge device, thus ensuring the smooth operation of the method.

[0074] In some embodiments, the method further includes: displaying a graphical interface, which includes an editable formula for calculating individual performance scores. By modifying the editable formula for calculating individual performance scores, adjustments to the formula for calculating individual performance scores are achieved. Clicking on the editable formula allows for editing and modification of the formula, thus facilitating the updating and maintenance of the formula for individual performance scores through the graphical interface. This step can be performed before each implementation of the method to ensure that the corresponding formula for calculating individual performance scores is already maintained when evaluating the inference performance of any deep learning model running on any edge device, ensuring the smooth operation of the method.

[0075] In step S306, based on the model parameters, hardware parameters, and environment parameters, the performance index weight corresponding to each performance index is obtained. The performance index weight indicates the importance of the performance index in the inference performance of the deep learning model.

[0076] This step obtains performance metric weights by considering the combined effects of model parameters, hardware parameters, and environmental parameters, resulting in more accurate performance metric weights.

[0077] In some embodiments, the performance index weights corresponding to the performance index are obtained based on model parameters, hardware parameters, and environmental parameters, including: inputting model parameters, hardware parameters, and environmental parameters into a trained weight prediction model to obtain the performance index weights output by the weight prediction model.

[0078] In this step, during the training process, the weight prediction model establishes a relatively accurate mapping relationship between model parameters, hardware parameters, environmental parameters and performance index weights. By using the trained weight prediction model to obtain performance index weights, the accuracy of the obtained performance index weights is improved, as is the speed of obtaining performance index weights.

[0079] In some embodiments, the training process of the weight prediction model includes: acquiring a training set, which includes multiple sets of training data. Each set of training data includes model parameters of a sample model, hardware parameters of a sample device, environmental parameters of the sample model running on the sample device, and label data. The category of the sample model is the same as the category of the deep learning model, the category of the sample device is the same as the category of the edge device, and the hardware of the edge device executing the inference process of the deep learning model is the same as the hardware of the sample device executing the inference process of the sample model. The label data includes a preset indicator weight for each performance indicator. The weight prediction model is iteratively trained based on the training set to obtain a trained weight prediction model. Each iteration in the iterative training includes: for each set of training data, calling the weight prediction model to make predictions based on the model parameters, hardware parameters, and environmental parameters to obtain the predicted performance indicator weight for each performance indicator; calculating the prediction loss based on the predicted performance indicator weight and the preset indicator weight; and adjusting the model parameters of the weight prediction model based on the prediction loss.

[0080] In the training process of the weight prediction model described above, by setting the categories of sample devices and sample models, the accuracy of the obtained performance metric weights is improved for application scenarios where deep learning models run on edge devices.

[0081] In some embodiments, the ratio of the maximum to the minimum total number of model parameters in different sample models is less than 1.2, and the ratio of the total number of model parameters in the deep learning model to the total number of model parameters in the sample models is 0.83-1.2. By setting the total number of model parameters in the sample models and the deep learning models, the accuracy of the obtained performance metric weights is further improved.

[0082] The sample model category is the same as the deep learning model category. For example, both the sample model and the deep learning model are classified as large language models. The sample device category is the same as the edge device category, such as smartphones, tablets, in-vehicle computers, industrial sensors, smart cameras, or routers. The hardware used by the edge device to execute the inference process of the deep learning model is the same as the hardware used by the sample device to execute the inference process of the sample model. This same hardware includes the same processor type and number, specifically GPU, CPU, or NPU. For example, if both the edge device and the sample device are smartphones, and both use a single GPU to execute the inference process, then the hardware used for inference is the same. However, if both the edge device and the sample device are smartphones, but the edge device uses a single GPU while the sample device uses two CPUs, then the hardware used for inference is different.

[0083] The training data consists of historical data, and the labels for the historical data are obtained through model inference or set through expert experience.

[0084] In some embodiments, the method further includes: quantizing model parameters, hardware parameters, and environmental parameters into numerical or classification codes; combining the quantized model parameters, hardware parameters, and environmental parameters into a feature vector; and using the feature vector as input to the weight prediction model. During the inference phase of the weight prediction model, the quantized feature vector effectively compresses the data size, thereby improving the inference speed of the weight prediction model and reducing power consumption.

[0085] In some embodiments, the weight prediction model employs a classifier model. For example, the classifier model can be a Multilayer Perceptron (MLP). By inputting model parameters, hardware parameters, and environmental parameters into the MLP, the MLP outputs performance metric weights. MLPs can map input data to output data and learn complex patterns through nonlinear transformations of multiple layers of nodes (including an input layer, one or more hidden layers, and an output layer). Therefore, MLPs can predict more accurate performance metric weights based on multiple types of data (model parameters, hardware parameters, and environmental parameters).

[0086] In some embodiments, the method further includes: generating a second configuration file based on the output of the weight prediction model after the weight prediction model outputs the performance index weights; obtaining the second configuration file and adjusting the performance index weights based on the information in the second configuration file. The second configuration file facilitates updating and modifying at least one of the performance index weights. This step is performed immediately after the weight prediction model outputs the performance index weights, ensuring that the performance index weights output by the weight prediction model can be obtained and applied as quickly as possible.

[0087] In some embodiments, the method further includes: displaying a graphical interface, which includes editable performance metric weights. Adjusting the performance metric weights is achieved by modifying the editable performance metric weights. Clicking on an editable performance metric weight allows for modification, thus the graphical interface facilitates updating the performance metric weights. This step is performed after obtaining the performance metric weights to ensure that the performance metric weights are updated before calculating the evaluation results of the deep learning model's inference performance, ensuring the smooth operation of the method.

[0088] When the application scenario is different, the weights of the performance metrics obtained from step S306 above will be different, for example: The application scenario is when a deep learning model for real-time inference runs on a mobile device, with w_inference latency=0.5, w_power consumption=0.3, w_memory usage=0.2, and w_temperature rise=0.0.

[0089] The application scenario is that when a deep learning model for power optimization is running on a battery-powered IoT (Internet of Things) device, w_power consumption = 0.6, w_inference latency = 0.2, w_memory usage = 0.1, and w_temperature rise = 0.1.

[0090] In other words, since the performance index weights of this disclosure are obtained based on model parameters, hardware parameters and environmental parameters, the method of obtaining performance index weights in this disclosure naturally supports the needs of different application scenarios, thus improving the flexibility and adaptability of this method.

[0091] In step S307, the evaluation results of the deep learning model inference performance are obtained based on the weights of multiple performance indicators and the corresponding individual performance scores.

[0092] When calculating the overall score of the evaluation results in this step, the overall score is obtained by using the performance index weights that reflect the application scenario and the values ​​of performance parameters that truly reflect the operating performance of the edge device. This results in a more accurate overall score that reflects the inference performance of the deep learning model on the edge device, thus improving the accuracy of the evaluation results.

[0093] In some embodiments, the following Formula 7 is applied to calculate the final score of the evaluation results: Formula 7: Final_Score=Σ(Score_i×w_i) Where Score_i is the individual performance score; w_i is the performance indicator weight; where i is a positive integer, i∈[1,n], and n is the number of performance indicators. For example, when the performance indicators include memory usage, inference latency, power consumption, and temperature rise, n is 4; the sum of the performance indicator weights of all performance indicators is 1.

[0094] In some embodiments, performance metrics are classified according to subtypes, and performance metrics belonging to the same category are weighted to obtain subtype scores, and then the subtype scores are weighted a second time.

[0095] For example, the subtypes of performance metrics may include efficiency-related metrics, resource-related metrics, etc. For instance, efficiency-related performance metrics may include inference latency, power consumption, etc., while resource-related performance metrics may include memory usage, temperature rise, etc. The performance metric weight and subtype weight of each performance metric are obtained through step S306 (by adding preset subtype weights to the tag data).

[0096] This step also yields a comprehensive score for the evaluation results, as well as sub-type scores. Based on these sub-type scores, it becomes easier to analyze the model's strengths and weaknesses in terms of efficiency or resources.

[0097] In some embodiments, the method further includes: obtaining a third configuration file and adjusting the calculation formula for the comprehensive score based on the information in the third configuration file. The third configuration file facilitates the updating and maintenance of the calculation formula for the comprehensive score, and also allows the method to easily extend to new performance metrics (such as startup time, model size, and other new feature dimensions). This step can be performed before each implementation of the method to ensure that the calculation formula for the comprehensive score is already maintained when evaluating the inference performance of any deep learning model running on any edge device, thus ensuring the smooth operation of the method.

[0098] In some embodiments, the method further includes: displaying a graphical interface, which includes an editable formula for calculating the overall score. Clicking on the editable formula allows modification of the overall score calculation formula. Therefore, displaying a graphical interface facilitates the updating and maintenance of the overall score calculation formula, and also allows the method to easily extend to new performance metrics (such as startup time, model size, and other new feature dimensions). This step can be performed before each implementation of the method to ensure that the formula for calculating the overall score is already maintained when evaluating the inference performance of any deep learning model running on any edge device, thus ensuring the smooth operation of the method.

[0099] In step S308, the bottleneck index is obtained. The bottleneck index indicates the performance index with the lowest single performance score or the highest performance index weight. The bottleneck index is used as the basis for generating optimization suggestions.

[0100] This step yields the bottleneck metric, which is the item with the highest loss score in the overall evaluation result. Therefore, generating optimization suggestions based on the bottleneck metric and optimizing the deep learning model or edge device according to the optimization suggestions can most efficiently improve the value of the corresponding bottleneck metric, thereby improving the performance of the edge device when the deep learning model is running, that is, improving the inference performance of the deep learning model when running on the edge device.

[0101] In some embodiments, the method further includes generating optimization suggestions based on the bottleneck indicator.

[0102] In some embodiments, generating optimization suggestions based on the bottleneck indicator includes: searching a database based on the bottleneck indicator to obtain at least one of the model parameters and hardware parameters that affect the bottleneck indicator; sorting the model parameters and hardware parameters according to their degree of influence; and generating optimization suggestions based on the model parameters and hardware parameters ranked highest in influence. This step allows for the rapid acquisition of model parameters and hardware parameters that have a significant impact on the bottleneck indicator. Consequently, when optimization is performed based on the generated suggestions, the improvement in the bottleneck indicator is greater, and the operating performance of the edge device can be significantly improved.

[0103] For example, the bottleneck metric is inference latency, with a single performance score of 60 (the lowest). The optimization suggestion is to try quantization of deep learning models to improve speed. The bottleneck metric is power consumption, with a single performance score of 85, but a weight as high as 0.6 (the highest). Power consumption is still the main deduction item. The optimization suggestion is to optimize memory access mode.

[0104] For example, the bottleneck metric is inference latency. Inference latency has the lowest single performance score (38.3 points) and the highest weight (0.5), making it the main deduction item. The optimization suggestion is: the inference latency performance is poor. Based on the model features (containing 15 Conv layers), it is recommended to try channel pruning of the deep learning model or use a lighter backbone network (such as MobileNetV3) to reduce FLOPs, improve inference speed, reduce inference latency, and improve the operating performance of the edge device.

[0105] To illustrate the aforementioned method for evaluating the inference performance of edge-side deep learning models, two specific examples are given below.

[0106] Example 1: Evaluation of Mobile Image Classification Model This example uses the evaluation of the inference performance of a lightweight image classification model on a mainstream Android smartphone to illustrate a method for evaluating the inference performance of a device-side deep learning model disclosed herein. The lightweight image classification model is a convolutional neural network model (a simplified version of the MobileNet model) applied to mobile image classification. This example aims to provide mobile application developers with objective and quantitative model performance evaluation results and optimization directions.

[0107] 1. Assessment Object and Environmental Configuration The deep learning model to be evaluated is an image classification model. The model parameters of the image classification model include: the model class is MobileNetV1 (scaled with a width factor), the width factor is 0.75 (to further reduce the model size), the number of parameters is approximately 1.8 million (1.8M), the number of FLOPs is approximately 300 million (0.3G), the quantization precision is FP32 (32-bit floating-point), the model format is TensorFlow Lite (.tflite), the input of the model is a 224x224 RGB image, and the output of the model is the ImageNet classification results for 1000 classes.

[0108] The device on the device side is a common mid-range Android smartphone on the market. The smartphone's hardware specifications include: a Qualcomm Snapdragon 7xx series processor (model 778G, 6nm process, integrated Adreno GPU), 8GB LPDDR4x RAM, and 128GB UFS 2.2 storage; the operating system is Android 13, and the inference framework is TensorFlow Lite 2.12. Adreno is the integrated graphics processor of the Qualcomm Snapdragon mobile platform, and LPDDR4X is a low-power, double data rate synchronous dynamic random access memory.

[0109] The environmental parameters for the image classification model to run on a smartphone include: the device is in a state of full battery power (>80%), the screen brightness of the device is 50%, and in addition to the evaluation application, there are 3 regular applications running in the background on the device (such as WeChat, music player, and browser), and the ambient temperature is room temperature (approximately 25°C).

[0110] 2. Performance Data Acquisition Run the deep learning model 100 times on the aforementioned edge device, collect the values ​​of multiple performance indicators of the edge device, and take the average value according to the number of runs to obtain the performance indicator value of a single run. Specifically, obtain the following performance indicator values: memory usage, inference latency, power consumption, and temperature rise.

[0111] The methods for obtaining memory usage values ​​include: monitoring memory usage during deep learning model loading and inference processes using the Android system's ` / proc / [pid] / status` file and the `dumpsys meminfo` command. The recorded peak memory usage was 78MB, meaning the total memory usage included model weights, activation values, inference framework overhead, and intermediate buffers.

[0112] The method for obtaining the inference latency includes inserting high-precision timestamps (using `System.nanoTime()`) at the precise times when the deep learning model's input data is prepared and the final output result is generated. Measurements were performed on 100 inference attempts, the first 5 were discarded as warm-up attempts, and the average latency of the remaining 95 attempts was calculated to be 42 ms (Milliseconds). This inference latency represents the end-to-end processing time from the input image to the output classification result. `System.nanoTime()` is a Java method for obtaining the current time; it represents a call to the static method `nanoTime()` in the `System` class, which is used to obtain a high-precision time value.

[0113] One method for obtaining power consumption values ​​is to utilize Android's built-in battery statistics tool, batterystats. Statistics are reset before the inference test begins, and data is exported after 100 inference iterations. The average additional power consumption caused by model inference activity during this period is calculated to be 1.1W (Watt), meaning the power consumption value is 1.1W.

[0114] The method for obtaining the temperature rise value includes reading the temperature data of the phone's internal SoC. The SoC temperature was recorded as 28℃ at the start of the test, and reached a maximum of 31℃ after 100 inference tests, therefore the temperature rise value is 3.0℃.

[0115] 3. Individual performance index scoring Based on the calculation formula for a single performance score applicable to mobile devices as described above in this disclosure, the collected performance index values ​​are quantified and scored.

[0116] 3.1 Single performance score for memory usage (Score_mem) Formula 1 is used to calculate the single performance score for memory usage, where the baseline memory is set to 50MB and the memory penalty coefficient is 10MB / 10 points. That is, memory usage below 50MB gets full marks, and memory usage above 150MB gets 0 points.

[0117] The calculation result is: Score_mem=100-max(0,(78-50)÷10×10)=100-max(0,28)=72 points.

[0118] 3.2 Individual performance score for inference latency (Score_lat) Formula 2 is used to calculate the individual performance score of inference latency, where the attenuation coefficient k is 0.02.

[0119] The calculation result is: Score_lat=100×exp(-0.02×42)=100×exp(-0.84)≈100×0.432=43.2 points.

[0120] 3.3 Single performance score for power consumption (Score_pwr) Formula 4 is used to calculate the single performance score of power consumption. The maximum power of the end device is 2.0W. When the power consumption value is equal to 2.0W, the single performance score of power consumption is 0 points. The lower the power consumption, the higher the score. When the power consumption is 0, the score is 100 points.

[0121] The calculation result is: Score_pwr=100×(1-1.1 / 2.0)=100×(1-0.55)=100×0.45=45 points.

[0122] 3.4 Individual performance score for temperature rise (Score_temp) Formula 6 is used to calculate the individual performance score of temperature rise, where the temperature penalty coefficient is 15 points / ℃. When the temperature rise is 0℃, the individual performance score of temperature rise is 100 points, and when the temperature rise exceeds 6.67℃, it is 0 points.

[0123] The calculation result is: Score_temp=max(0,100-3.0×15)=max(0,100-45)=55 points.

[0124] 4. Weighted composite score calculation For the typical application scenario of "real-time interactive mobile applications", the above model parameters, hardware parameters, and environmental parameters are input into the trained weight prediction model, and the resulting weight prediction model outputs the following performance metrics: w_mem (memory usage) = 0.2, w_lat (inference latency) = 0.5, w_pwr (power consumption) = 0.2, w_temp (temperature rise) = 0.1.

[0125] The reason why inference latency has a relatively large weight in the performance metrics is that the image classification model in this example is a real-time interactive application, and real-time performance is the primary consideration. Therefore, inference latency is more important in the inference performance of the image classification model.

[0126] Formula 7 is used to calculate the overall score of the evaluation results. The overall score is: Final_Score=(Score_mem×w_mem)+(Score_lat×w_lat)+(Score_pwr×w_pwr)+(Score_temp×w_temp)=(72×0.2)+(43.2×0.5)+(45×0.2)+(55×0.1)=14.4+21.6+9.0+5.5=50.5 points.

[0127] 5. Evaluation Results Output and Optimization Suggestions 5.1 Evaluation Results Output The overall score of the evaluation result is 50.5 points. Under the premise of a full score of 100 points, this evaluation result indicates that the inference performance of the model is below average and there is considerable room for optimization.

[0128] The individual performance scores are as follows: memory usage is 72 points, which is good; inference latency is 43.2 points, which is poor and is a weak point; power consumption is 45 points, which is average; and temperature rise is 55 points, which is acceptable.

[0129] 5.2 Optimization Suggestion Generation The overall score was 50.5, indicating that the model's inference performance did not reach the ideal level.

[0130] Analysis of bottleneck indicators: Inference latency (43.2 points) scored the lowest and is the main bottleneck indicator affecting overall performance. Power consumption (45 points) also has room for improvement.

[0131] Optimization suggestions based on the weakest link indicator include: 1) Try smaller model variants, such as using MobileNetV1 with a width factor of 0.5 or exploring the more efficient EfficientNet-Lite series; 2) Enable hardware acceleration. Ensure that the XNNPACK acceleration backend is correctly configured and enabled in TensorFlow Lite to fully utilize the CPU's SIMD instruction set and significantly improve inference speed. 3) Considering model quantization, converting the FP32 model to an INT8 quantized model can significantly reduce computation and power consumption with almost no loss of accuracy, further improving latency and energy efficiency.

[0132] Example 2: Performance Evaluation of the Driver Status Monitoring Model on the Vehicle-to-Everything (V2X) Platform This example demonstrates the method of this disclosure by evaluating the edge-side inference performance of a DSM (Driver State Monitoring) model in a real-world operating environment, where the DSM is deployed on the IVI (In-Vehicle Infotainment) system of an intelligent vehicle. This example follows the method of this disclosure and fully considers the specific performance requirements of the in-vehicle environment, providing automotive electronic system developers with a scientific and comprehensive evaluation basis.

[0133] 1. Assessment Object and Environmental Configuration The model to be evaluated is the DSM model, a lightweight deep learning model used for real-time monitoring of driver fatigue and distraction. The DSM model parameters include: approximately 2.5M parameters, approximately 0.5G FLOPs, quantization precision of INT8 (quantized after training to optimize NPU inference efficiency), model architecture (customized based on a simplified version of YOLOv5s, specifically designed for facial landmark detection and behavior classification), model category ONNX Runtime (.onnx), and the model input is a 128x128 grayscale image of the driver's face captured by an infrared camera. The model output includes multi-task results such as fatigue state (duration of eye closure, yawning) and distraction state (direction of gaze, mobile phone usage).

[0134] The device on the device side is the in-vehicle infotainment system of a 2024 smart electric sedan from a high-end brand. The hardware parameters of the in-vehicle infotainment system include: 16GB LPDDR5 (Low Power Double Data Rate 5) memory, 128GB UFS (Universal Flash Storage) 3.1 storage, a Linux+QNX dual-system operating system (DSM function runs in the QNX security domain), an inference framework of ONNX Runtime 1.16 with Hexagon Delegate, a Qualcomm Snapdragon SA8295P SoC (5nm process, designed specifically for smart cockpits), a Kryo 680 CPU (8 cores, up to 3.0GHz), an Adreno 740 GPU, and a Hexagon 780 AI engine NPU (with 16 TOPS INT8 computing power). ONNX Runtime 1.16 is an inference engine used for the model inference process; Hexagon Delegate refers to the dedicated inference delegate of the Qualcomm Hexagon processor, used to accelerate the model execution of ONNX Runtime on Hexagon-enabled devices (such as Snapdragon chips).

[0135] The environmental parameters for the DSM model when running on the in-vehicle infotainment system include: vehicle status is that the vehicle is in motion (simulating urban road driving), and the engine / motor is running normally; system load is that the navigation system is running with high precision, the in-vehicle audio system is playing music, and the 5G T-Box (Telematics BOX, in-vehicle telematics processor) is continuously connected to the network; ambient temperature is 35°C outside the vehicle, and the air conditioning inside the vehicle is on with an initial temperature of 26°C.

[0136] 2. Performance Data Acquisition Using the evaluation tools provided in this disclosed system, the DSM model was run continuously for 15 minutes during vehicle operation to simulate the continuous monitoring requirements under real driving scenarios. After the continuous operation ended, the following performance indicators were obtained: memory usage, inference latency, power consumption, and temperature rise.

[0137] Under the aforementioned harsh operating environment, performance metrics are obtained through OBD-II (On-Board Diagnostic II) and the system's internal monitoring agent.

[0138] Specifically, methods for obtaining memory usage values ​​include: monitoring the model process using the `pidin mem` command in the QNX system. The peak memory usage during the 15-minute test period was recorded as 105MB, meaning the total memory usage was 105MB, which includes the model, runtime, intermediate feature maps, and safety redundancy buffers.

[0139] Specifically, the method for obtaining the inference latency value includes: inserting a high-precision timestamp (using clock_gettime()) at the precise time point between when the camera captures a frame of image and when the model outputs the final state judgment result. Inference is performed on 900 frames of images generated within 15 minutes. After removing the first 10 frames of warm-up data, the average latency of the remaining 890 frames is calculated to be 38ms, which is the inference latency value. This inference latency value directly affects the real-time performance of state monitoring.

[0140] Specifically, the method for obtaining power consumption values ​​includes: using a high-precision vehicle power consumption recorder to measure the power consumption difference of the entire cockpit domain controller in two states: "DSM function off" and "DSM function on". The average additional power consumption caused by the continuous operation of the DSM model was measured to be 1.8W, that is, the additional power consumption value is 1.8W.

[0141] Specifically, the method for obtaining the temperature rise value includes: reading the temperature sensor data near the main control chip on the cockpit domain controller PCB (Printed Circuit Board). The chip area temperature was recorded as 38°C at the start of the test, and the highest temperature reached 50°C after 15 minutes of driving test, therefore the temperature rise value is 12°C.

[0142] 3. Individual performance index scoring The collected data is quantitatively scored based on the refined scoring function for critical applications of vehicle infotainment systems as disclosed in this disclosure.

[0143] 3.1 Single performance score for memory usage (Score_mem) Formula 1 is used to calculate the single performance score for memory usage, where the baseline memory is set to 80MB and the memory penalty coefficient is 1MB / 1 point.

[0144] The calculation result is: Score_mem=100-max(0,(105-80)×1.0)=100-25=75 points.

[0145] 3.2 Individual performance score for inference latency (Score_lat) Formula 3 is used to calculate the individual performance score of inference latency, where the attenuation coefficient is 0.1 and the latency time corresponding to the score benchmark (50 points benchmark) is set to 30ms.

[0146] The calculation result is: Score_lat=100 / (1+exp(0.1×(38-30)))=100 / (1+exp(0.8))≈100 / (1+2.226)≈31.0 points.

[0147] 3.3 Single performance score for power consumption (Score_pwr) Formula 5 is used to calculate the individual performance score of power consumption, where the additional power consumption threshold is 2.5W.

[0148] The calculation result is: Score_pwr=100×(1-1.8 / 2.5)=100×(1-0.72)=100×0.28=28 points.

[0149] 3.4 Individual performance score for temperature rise (Score_temp) Formula 6 is used to calculate the individual performance score of temperature rise. Due to the high heat dissipation pressure in high temperature environments, the temperature penalty coefficient is 7 points / ℃, which means that 7 points are deducted for every 1℃ increase in temperature.

[0150] The calculation result is: Score_temp=max(0,100-12×7)=max(0,100-84)=16 points.

[0151] 4. Weighted composite score calculation For the application scenario of "critical safety applications on the vehicle-mounted system", the above model parameters, hardware parameters, and environmental parameters are input into the trained weight prediction model, and the resulting weight prediction model outputs the following performance metrics: w_mem (memory usage) = 0.1, w_lat (inference latency) = 0.5, w_pwr (power consumption) = 0.25, w_temp (temperature rise) = 0.15.

[0152] Among them, the performance metric of inference latency has a relatively large weight because the deep learning model in this example is used to monitor driver fatigue and distraction in real time, so real-time performance is related to driving safety, hence the highest weight of inference latency; while power consumption affects the vehicle's range and system stability, and high temperature may cause the vehicle's system to reduce frequency or malfunction, so power consumption and temperature rise also have high weights; the performance metric weights obtained above reflect the importance of real-time performance and stability.

[0153] Formula 7 is used to calculate the overall score of the evaluation results. The overall score is: Final_Score=(Score_mem×w_mem)+(Score_lat×w_lat)+(Score_pwr×w_pwr)+(Score_temp×w_temp)=(75×0.1)+(31.0×0.5)+(28×0.25)+(16×0.15)=7.5+15.5+7.0+2.4=32.4 points.

[0154] 5. Evaluation Results Output and Optimization Suggestions 5.1 Evaluation Results Output The overall score of the evaluation result is 32.4 points. Under the premise of a full score of 100 points, the evaluation result indicates that the model has poor inference performance, has potential safety hazards, and does not meet the requirements for mass production of pre-installed products.

[0155] The individual performance scores are as follows: memory usage is 75 points, which is good; inference latency is 31.0 points, which is poor and does not reach the safety threshold; power consumption is 28 points, which is poor; and temperature rise is 16 points, which is extremely poor and carries a high risk.

[0156] 5.2 Optimization Suggestion Generation The overall score is 32.4 points. The model's inference performance is seriously substandard, and there are risks to real-time performance and thermal safety, making it undeployable.

[0157] Analysis of bottleneck indicators: Inference latency (31.0 points) and temperature rise (16 points) are the main bottleneck indicators, and power consumption (28 points) also exceeds the standard. Excessive inference latency may lead to delayed response in dangerous situations.

[0158] Optimization suggestions based on the weakest link indicator include: 1) Deep optimization of model structure: Channel CP (pruning) and KD (Knowledge Distillation) are performed on YOLOv5s, with the goal of compressing FLOPs to below 0.3G; 2) Ensure NPU is running at full load: Check the ONNX Runtime Hexagon Delegate configuration to confirm that model operators are fully mapped to the Hexagon 780 NPU for execution, and avoid any operators falling back to the CPU; 3) Optimize input resolution: While ensuring the accuracy of key point detection, try reducing the input image resolution from 128x128 to 96x96, which can significantly reduce the amount of computation. 4) Enhance heat dissipation design: Assess the necessity of adding heat sinks or optimizing airflow to cope with continuous operation under high ambient temperatures.

[0159] This method employs a weighted scoring system (i.e., obtaining evaluation results based on performance indicator weights and individual performance scores) to uniformly quantify and integrate the values ​​of multiple performance indicators. This overcomes the limitations of traditional methods that rely on a single evaluation dimension, providing a comprehensive and quantifiable evaluation of inference performance. It achieves multi-dimensional comprehensive quantitative evaluation, resulting in assessments that more accurately indicate the performance of deep learning models on edge devices, as well as their inference performance. This provides more accurate guidance for optimizing deep learning models and determining whether they can be deployed on edge devices. The evaluation results are calculated from explicit individual performance scores and performance indicator weights, ensuring a transparent evaluation process and interpretable results. This method obtains performance indicator weights based on specific application scenarios, resulting in higher accuracy and making the evaluation results more targeted and practical. It provides interpretable and configurable evaluation standards. This method naturally supports the needs of different application scenarios, enhancing the flexibility and adaptability of the evaluation. This approach simplifies complex multi-indicator decision-making problems into a single, intuitive "comprehensive score," making it easy for non-experts to quickly understand model performance and lowering the barrier to entry for understanding and using the evaluation results, thus reducing evaluation complexity. By analyzing individual performance scores and performance indicator weights, it can quickly identify bottleneck indicators (i.e., the limitations of model inference performance), providing clear data support and directional guidance for model optimization and hardware selection. The weighted scoring system has clear logic, is simple to calculate, and is easy to implement and integrate into software tools or platforms, resulting in low engineering implementation costs and ease of implementation and integration.

[0160] In summary, this disclosure establishes a flexible, transparent, and quantifiable method for evaluating the inference performance of edge models by constructing a "weighted scoring system," effectively solving the core problem of low accuracy in existing technologies and providing a scientific and practical evaluation tool for the efficient development and deployment of edge deep learning models.

[0161] Figure 5 This is a block diagram illustrating an edge-side deep learning model inference performance evaluation device according to an exemplary embodiment. See also Figure 5 The edge-side deep learning model inference performance evaluation device includes: The model parameter acquisition module 501 is used to acquire the model parameters of the deep learning model to be evaluated. The model parameters indicate the parameters of the model's features. The hardware parameter acquisition module 502 is used to acquire the hardware parameters of the edge device used to run the deep learning model. The hardware parameters indicate the parameters of the hardware characteristics of the device. The environment parameter acquisition module 503 is used to acquire the environment parameters of the deep learning model when it runs on the edge device. The environment parameters indicate the characteristics of the running environment. The performance data acquisition module 504 is used to collect the values ​​of multiple performance indicators of the edge device when the deep learning model is running on the edge device. The performance indicators indicate the running performance of the corresponding device. The single performance scoring module 505 is used to calculate the value of each performance indicator using a normalization method to obtain the single performance score corresponding to each performance indicator. The weight acquisition module 506 is used to acquire the performance index weight corresponding to each performance index based on the model parameters, hardware parameters and environmental parameters. The performance index weight indicates the importance of the performance index in the inference performance of the deep learning model. The comprehensive scoring module 507 is used to obtain the evaluation results of the inference performance of the deep learning model based on the weights of multiple performance indicators and the corresponding individual performance scores.

[0162] In some embodiments, the weight acquisition module is used to input model parameters, hardware parameters, and environmental parameters into the trained weight prediction model to obtain the performance index weights output by the weight prediction model.

[0163] In some embodiments, the training process of the weight prediction model includes: acquiring a training set, which includes multiple sets of training data. Each set of training data includes model parameters of a sample model, hardware parameters of a sample device, environmental parameters of the sample model running on the sample device, and label data. The category of the sample model is the same as the category of the deep learning model, the category of the sample device is the same as the category of the edge device, and the hardware of the edge device executing the inference process of the deep learning model is the same as the hardware of the sample device executing the inference process of the sample model. The label data includes a preset indicator weight for each performance indicator. The weight prediction model is iteratively trained based on the training set to obtain a trained weight prediction model. Each iteration in the iterative training includes: for each set of training data, calling the weight prediction model to make predictions based on the model parameters, hardware parameters, and environmental parameters to obtain the predicted performance indicator weight for each performance indicator; calculating the prediction loss based on the predicted performance indicator weight and the preset indicator weight; and adjusting the model parameters of the weight prediction model based on the prediction loss.

[0164] In some embodiments, the ratio of the maximum to the minimum value of the total number of model parameters in different sample models is less than 1.2, and the ratio of the total number of model parameters in the deep learning model to the total number of model parameters in the sample models is 0.83-1.2.

[0165] In some embodiments, when the deep learning model is classified as a large language model, the performance metrics include memory usage, inference latency, power consumption, and temperature rise. Memory usage indicates the maximum memory occupied by the deep learning model on the edge device during operation. Inference latency indicates the time from input data to output result for the deep learning model. Power consumption indicates the average power consumption of the deep learning model during inference. Temperature rise indicates the temperature difference of the hardware before and after the inference process.

[0166] In some embodiments, the apparatus further includes: a bottleneck indicator acquisition module, used to acquire bottleneck indicators, wherein the bottleneck indicator indicates the performance indicator with the lowest single performance score or the highest performance indicator weight, and the bottleneck indicator is used as the basis for generating optimization suggestions.

[0167] It should be noted that the edge-side deep learning model inference performance evaluation device provided in the above embodiments is only illustrated by the division of the above functional units when evaluating the inference performance of edge-side deep learning models. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the edge-side deep learning model inference performance evaluation device and the edge-side deep learning model inference performance evaluation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0168] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0169] Figure 6 This is a block diagram illustrating a terminal 600 according to an exemplary embodiment. Figure 6 A structural block diagram of a terminal 600 provided in an exemplary embodiment of this disclosure is shown. The terminal 600 may be an industrial control computer, an embedded computer, an integrated computer, a desktop computer, or a laptop computer. The terminal 600 may also be referred to as user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0170] Typically, terminal 600 includes a processor 601 and a memory 602.

[0171] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as the CPU, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU, which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0172] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one computer program, which is executed by the processor 601 to implement the method provided in the method embodiments of this application.

[0173] In some embodiments, the terminal 600 may also optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 604, a display screen 605, an end-side device 606, and a power supply 607.

[0174] Peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0175] The radio frequency (RF) circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. In some embodiments, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0176] Display screen 605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 605, disposed on the front panel of terminal 600; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 600 or in a folded design; in other embodiments, display screen 605 may be a flexible display screen, disposed on a curved or folded surface of terminal 600. Furthermore, display screen 605 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 605 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0177] The edge device 606 is used to connect to the terminal 600 so that the terminal 600 can collect the hardware parameters of the edge device 606, collect the performance index values ​​of the edge device 606, etc. The edge device 606 can be a smartphone, a vehicle infotainment system (i.e., an in-vehicle computer), a tablet computer, a laptop computer, an industrial sensor, a smart camera, etc. The edge device 606 is used to run a deep learning model whose inference performance needs to be estimated.

[0178] Power supply 607 is used to power the various components in terminal 600. Power supply 607 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 607 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0179] When the electronic device is provided as terminal 600, the evaluation results of individual performance scores, performance index weights, and deep learning model inference performance can be displayed on the display screen 605 of terminal 600. A graphical interface can also be displayed on the display screen 605 to display the editable calculation formulas for individual performance scores and the calculation formulas for the overall score.

[0180] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on terminal 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0181] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device, such as a processor 601 of terminal 600, to complete the task execution method for the aforementioned network activities. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0182] The aforementioned computer equipment can also be implemented as a server. The structure of a server is described below: Figure 7 This is a schematic diagram of a server structure according to an exemplary embodiment. The server 700 can vary significantly due to different configurations or performance. It may include one or more processors 701 and one or more memories 702, wherein the one or more memories 702 store at least one computer program, which is loaded and executed by the one or more processors 701 to implement the methods provided in the various method embodiments described above. Of course, the server 700 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 700 may also include other components for implementing device functions, which will not be elaborated here.

[0183] When the electronic device is provided as a server, the evaluation results of individual performance scores, performance indicator weights, and deep learning model inference performance can be displayed on an external display screen.

[0184] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the methods in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0185] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. A processor of an electronic device reads the program code from the computer-readable storage medium and executes the program code, causing the electronic device to perform the method described above.

[0186] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0187] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for evaluating the inference performance of an edge-side deep learning model, characterized in that, The method includes: Obtain the model parameters of the deep learning model to be evaluated, wherein the model parameters indicate the parameters of the model's features; Obtain the hardware parameters of the edge device used to run the deep learning model, the hardware parameters indicating the hardware characteristics of the device; The environment parameters of the deep learning model during operation on the edge device are obtained, and the environment parameters indicate the characteristics of the operating environment. When the deep learning model runs on the edge device, the values ​​of multiple performance indicators of the edge device are collected, and the performance indicators indicate the operating performance of the corresponding device. The value of each performance indicator is calculated using a normalization method to obtain the individual performance score corresponding to each performance indicator. Based on the model parameters, hardware parameters, and environment parameters, obtain the performance index weight corresponding to each performance index, and the performance index weight indicates the importance of the performance index in the inference performance of the deep learning model. The evaluation result of the inference performance of the deep learning model is obtained based on the weights of the various performance indicators and the corresponding individual performance scores.

2. The method for evaluating the inference performance of an edge-side deep learning model according to claim 1, characterized in that, Based on the model parameters, the hardware parameters, and the environment parameters, obtain the performance indicator weights corresponding to the performance indicators, including: The model parameters, hardware parameters, and environmental parameters are input into the trained weight prediction model to obtain the performance index weights output by the weight prediction model.

3. The method for evaluating the inference performance of an edge-side deep learning model according to claim 2, characterized in that, The training process of the weight prediction model includes: Obtain a training set, which includes multiple sets of training data. Each set of training data includes model parameters of a sample model, hardware parameters of a sample device, environmental parameters of the sample model running on the sample device, and label data. The category of the sample model is the same as the category of the deep learning model. The category of the sample device is the same as the category of the edge device. The hardware of the edge device executing the inference process of the deep learning model is the same as the hardware of the sample device executing the inference process of the sample model. The label data includes a preset indicator weight for each performance indicator. The weight prediction model is iteratively trained based on the training set to obtain the trained weight prediction model. Each iteration in iterative training includes: For each set of training data, the weight prediction model is invoked to make predictions based on the model parameters, hardware parameters, and environmental parameters to obtain the predicted performance index weights for each performance index. Calculate the prediction loss based on the prediction performance index weights and the preset index weights; The model parameters of the weighted prediction model are adjusted based on the predicted loss.

4. The method for evaluating the inference performance of an edge-side deep learning model according to claim 3, characterized in that, The ratio of the maximum to the minimum value of the total number of model parameters in different sample models is less than 1.2, and the ratio of the total number of model parameters in the deep learning model to the total number of model parameters in the sample model is 0.83-1.

2.

5. The method for evaluating the inference performance of an edge-side deep learning model according to claim 1, characterized in that, When the category of the deep learning model is a large language model, the performance metrics include memory usage, inference latency, power consumption, and temperature rise. The memory usage indicates the maximum memory occupied by the deep learning model on the edge device during operation. The inference latency indicates the time from input data to output result of the deep learning model. The power consumption indicates the average power consumption of the deep learning model during inference. The temperature rise indicates the temperature difference of the hardware before and after the inference process of the deep learning model.

6. The method for evaluating the inference performance of an edge-side deep learning model according to claim 1, characterized in that, The method further includes: Obtain the bottleneck indicator, which indicates the performance indicator with the lowest single performance score or the highest weight of the performance indicator, and the bottleneck indicator is used as the basis for generating optimization suggestions.

7. A device for evaluating the inference performance of an edge-side deep learning model, characterized in that, The device includes: The model parameter extraction module is used to obtain the model parameters of the deep learning model to be evaluated, wherein the model parameters indicate the parameters of the model's features; A hardware parameter extraction module is used to obtain hardware parameters of the edge device used to run the deep learning model, wherein the hardware parameters indicate the hardware characteristics of the device. An environment parameter extraction module is used to obtain the environment parameters of the deep learning model when it runs on the edge device. The environment parameters indicate parameters corresponding to the characteristics of the running environment, and the performance indicators indicate the running performance of the device. The performance metric acquisition module is used to acquire the values ​​of multiple performance metrics of the edge device when the deep learning model is running on the edge device, and the performance metrics indicate the operating performance of the corresponding device. The normalization module is used to calculate the value of each of the performance indicators by normalization method to obtain the individual performance score corresponding to each of the performance indicators. The weight acquisition module is used to acquire the performance index weight corresponding to each performance index based on the model parameters, the hardware parameters and the environment parameters. The performance index weight indicates the importance of the performance index in the inference performance of the deep learning model. The comprehensive score acquisition module is used to obtain a comprehensive score based on the weights of the multiple performance indicators and the corresponding individual performance scores.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of a computer device, the computer device is enabled to perform the method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.

Images