Mobile terminal heterogeneous platform multi-inference framework-oriented delay overhead prediction method

Abstract

The invention discloses a delay overhead prediction method for a multi-inference framework of a mobile terminal heterogeneous platform, and the method comprises the steps: 1, obtaining an original model, obtaining convolution operators with similar configurations in the original model, and carrying out the grouping and merging of the convolution operators with similar configurations; 2, delay overhead prediction model modeling is carried out according to the convolution operator reasoning characteristics of different reasoning platforms; then, the overall delay of the model is deduced in equal proportion; 3, performing kernel self-definition, compiling and brushing on the heterogeneous hardware of the mobile terminal, and confirming a reasoning framework, a reasoning component, a core type and a core; and 4, quickly migrating the delay overhead prediction model among different configurations. According to the method, the problems of difficulty in delay overhead prediction modeling and low precision caused by diversity of software and hardware when the existing mobile terminal heterogeneous equipment faces various reasoning platforms, reasoning configurations and reasoning models are solved.

Description

TECHNICAL FIELD

[0001] The application relates to the field of mobile terminal edge intelligence, and particularly relates to a delay overhead prediction method for a mobile terminal heterogeneous platform multi-inference framework. BACKGROUND

[0002] With the rapid development of edge and terminal computing devices, cloud servers are no longer absolutely suitable for computing-intensive tasks. Especially for AI services, PC graphics cards led by Nvidia and their cuda platform are no longer the only choice for deep learning model inference. The rise of mobile Arm architecture and the heterogeneous collaborative design of processors have improved the computing power of edge terminals and reduced the deployment threshold and starting point of deep learning models. AI services are affected by factors such as computing power, cost, energy consumption, communication overhead, and privacy protection. The service mode has experienced three development stages, and developers and users can choose the appropriate computing mode according to actual needs. From cloud computing to edge computing to terminal computing, people's demand for privacy and real-time performance has led to the need for terminal computing to become a multi-dimensional optimization problem. Local deployment needs to face high computing overhead. Compared with the powerful GPU graphics card of the desktop, the energy consumption design of the mobile terminal leads to no such powerful performance release, resulting in a difference of tens of times in inference delay when running the same size model.

[0003] In order to realize the demand of model deployment on the mobile terminal, hardware manufacturers such as Mediatek, Qualcomm and Apple have developed powerful heterogeneous processors. The big.LITTLE size core heterogeneous design CPU can balance performance and energy consumption. The large core focuses on performance and the small core focuses on energy consumption. At the inference framework level, the ecosystem has gradually improved. Hardware manufacturers and operating system manufacturers (android-Google, ios-Apple, linux-Microsoft) actively develop deep learning inference frameworks and APIs for their own products to improve product experience and attract consumers to use them. Common ones include Google's LiteRT (former TensorFlow Lite), Qualcomm's SNPE, Microsoft's onnx and Tencent's ncnn, which have various new features such as multi-threading and mobile GPU support. Major manufacturers are actively providing model inference capabilities for their own software and hardware products, which provides a software framework basis for the deployment of edge intelligence applications.

[0004] In the model deployment and inference process commonly used in mobile terminals, first, a model structure of a large volume that can meet the accuracy requirements is obtained from the cloud. Then, the mainstream inference framework is used to train on the self-defined task dataset. The model is adjusted according to the inference delay constraint, the model structure is transformed, the complexity is reduced, and the redundant calculation amount is reduced, so as to reduce the delay. Finally, the model that can balance the accuracy and delay is determined. Next, the inference framework used for deep learning model inference on the mobile phone needs to be selected. The speed, characteristics and support for co-processors of different inference frameworks are different, and the appropriate inference framework needs to be selected according to the preference and platform. Then, the model format conversion is performed, and the preparation at the model level is completed. Then, the inference framework is compiled and deployed on the mobile phone according to the mobile terminal platform, system version, etc. Next, the inference component to be used is determined, and whether to use multi-thread optimization, the number of CPU cores, etc. are specified in advance. Many existing research works try to optimize the inference speed by parallel inference on heterogeneous processors, to fully utilize the computing power of each component, to maximize the performance release, and to realize inference acceleration.

[0005] Although the computing power of the mobile terminal is much stronger than that of the small microcontroller, it is still not enough to run the original large volume model to ensure high accuracy, low delay and good user experience. Therefore, the original large volume model generally needs to be fine-tuned and pruned, and redesigned to meet the high accuracy requirement under the specific delay requirement. During the pruning process, the real delay of the model on the specific device needs to be evaluated repeatedly, and the process of repeated fine-tuning is very time-consuming if the delay overhead is obtained by actual measurement each time.

[0006] In addition to the time-consuming problem of the actual measurement method, the diversity of inference configuration also leads to further expansion of the sample space of the actual measurement. It is not realistic to use the actual measurement method for model pruning in the face of a large search space.

[0007] The method based on prediction is still in the exploratory research stage for modeling the inference delay overhead of mobile smoke. The main research method is to predict the delay of diversified models under specific frequency and specific configuration. Through the configuration of operator granularity and the modeling of heterogeneous hardware computing power, the delay overhead is predicted by mathematical calculation or deep learning method. Although many methods consider the occurrence of complex situations such as operator fusion, and achieve good results under specific configurations, there is a big problem that the overhead model needs to be modeled by complex systems and deep learning methods to obtain strong representation ability. The modeling scheme needs to be rebuilt for other configurations each time, which is too expensive. And this kind of method usually ignores the characteristics of the inference platform, and only designs the prediction method for a specific inference framework. The universality and efficiency need to be strengthened.

[0008] Therefore, how to design a simple and low-overhead method that can adapt to different inference frameworks and also realize cross-inference configuration migration and reuse of mobile terminal delay efficient sampling and overhead modeling is an important problem to be solved at present.

[0009] Therefore, the existing delay prediction and overhead modeling scheme of the mobile terminal heterogeneous platform still has great limitations and cannot meet the actual needs of efficient sampling and rapid evaluation of model delay under diversified inference frameworks, bringing difficulties to model pruning, model compression and other localized deployment strategies. SUMMARY

[0010] The application provides a delay and overhead prediction method for a mobile terminal heterogeneous platform multi-inference framework to solve the problems of excessive overhead of the existing edge terminal model fine-tuning and the low efficiency and accuracy of the prediction-based method when deploying the model on a mobile terminal real machine.

[0011] To achieve the above purpose, the technical scheme adopted by the application is as follows:

[0012] The delay and overhead prediction method for the mobile terminal heterogeneous platform multi-inference framework has the following process:

[0013] Step 1, obtaining an original model, performing configuration analysis on the original model to obtain convolution operators with similar configurations in the original model as the main basis for subsequent sampling and overhead model modeling, and then grouping and merging the convolution operators with similar configurations to reduce repeated convolution operator configurations and reduce the number of convolution operators to be sampled;

[0014] Step 2, delay and overhead prediction model modeling is performed on the convolution operator inference characteristics of different inference platforms, wherein:

[0015] For the inference platform onnx, a simple linear fitting is used to model the delay and overhead prediction model of all group convolution operators obtained in step 1;

[0016] For the inference platform TensorFlow Lite, a linear step function is used to model the delay and overhead prediction model of all group convolution operators obtained in step 1;

[0017] For the inference platform ncnn, since the inference platform ncnn uses NEON optimization, according to the bilinear characteristics of ncnn, a piecewise linear fitting composed of a piecewise function is used to model the delay and overhead prediction model of all group convolution operators obtained in step 1;

[0018] Then, according to the calculation complexity of the convolution operator part in the prediction model, and the convolution operator prediction delay obtained by the delay and overhead prediction model under each inference platform, the overall delay of the prediction model is proportionally inferred.

[0019] Further, the step 1 process is as follows:

[0020] Step 101, obtain the model layer information and parameters of the original model, judge whether the model layer is a convolution operator according to the model layer information and parameters, thereby divide the model layer L into two parts, L = L oth + L conv , wherein L conv is a convolution operator, and L other is a non-convolution operator; then parse the configuration of the convolution operator L conv , and obtain a similar convolution operator;

[0021] Step 102, except for the input channel number and the output channel number, group the convolution operators in the model with the same configuration, thereby grouping the convolution operators according to similar configurations; then, determine the sampling base of each group as cout max · cin max , wherein cout max is the maximum output channel number in the group, and cin max is the maximum input channel number in the group.

[0022] Further, in step 2, for the inference platform onnx, the modeling process of simple linear fitting is as follows:

[0023] Fix the output channel number cout, fit the input channel number cin as the independent variable, and the convolution operator delay l as the dependent variable, use interval step sampling to obtain the convolution operator delay under different input channel numbers, to reduce the sampling overhead, and perform simple linear fitting on the convolution operator delay overhead curve, to establish a delay overhead prediction model l as shown in the following formula:

[0024] l = ax i + b

[0025] x i is cin in Group i s.t.cout dimension fixed

[0026] Wherein: a is the slope of the linear fitting curve; x i is the cin size allowed by the Group i , which needs to be less than cin max , and this curve is applicable when cout is fixed at a certain value from 1 to cout max ; b is the intercept of the linear fitting curve.

[0027] Further, in step 2, the modeling process of the linear step function for the inference platform TensorFlow Lite is as follows:

[0028] Based on the equal-width segmentation and the equidifference step property in the delay curve, by fixing the input channel number cin, fitting the output channel number cout as the independent variable, and the convolution operator delay l as the dependent variable, the operator delay is sampled, the continuous sampling point data is smoothed, the jump point is found, and the step width is estimated, that is, the parameters of the linear step function can be fitted, and the delay overhead prediction model l(x) is established as follows:

[0029]

[0030] Where: x i is the Group i convolution allowed cout independent variable size; Δheight is the step height, height0 is the starting height, and width is the step width;

[0031] When modeling the delay overhead prediction model for the first time, the step width width is determined first, the noise is processed by moving average, the jump point is found to obtain the width, and the smoothed sequence The first-order difference is calculated All first-order differences d i that satisfy |d i |>θ are detected, where θ is a manually set hyperparameter threshold, and these points are the step jump points, denoted as

[0032] The step width width is calculated by the following formula:

[0033]

[0034] Where: m is the number of step jump points in the delay curve; is the horizontal coordinate of the i+1th step jump point; is the horizontal coordinate of the i th step jump point;

[0035] Then, for different steps The corresponding step height is obtained, where S b ={i | b i =b} represents all points belonging to the b th step;

[0036] The parameters Δheight and height0 of the linear step function are calculated according to the step height, as shown in the following formula:

[0037]

[0038] height0 = c min = c0

[0039] wherein: c max is the maximum step height; c min is the minimum step height; |{c0, c1, c2, …}| is the number of steps.

[0040] Further, in step 2, the modeling process of the piecewise linear fitting of the inference platform ncnn segment function is as follows:

[0041] Using piecewise function for two-dimensional linear fitting, by fixing the output channel number cout, fitting the input channel number cin as the independent variable, and the convolution operator delay l as the dependent variable, the coefficient straight line and the dense straight line are classified and modeled. Since the coefficient bifurcation line and the dense bifurcation line both have obvious linear nature, equal interval step sampling is used to reduce the sampling overhead, and the delay overhead prediction model I(x i ) is established as shown in the following formula:

[0042]

[0043] wherein: x i is the cin independent variable size allowed by the Group i convolution; a1 is the sparse linear curve slope; a2 is the dense linear curve slope; b1 is the sparse line intercept; b2 is the dense line intercept; imod 8 = 0 means that the sparse line is suitable for sparse points whose cin can be divided by 8.

[0044] Further, in step 2, the process of proportionally inferring the overall delay of the model is as follows:

[0045] Taking the convolution operator as the main part of the delay, based on FLOPs complexity or MACs, the proportion conversion is carried out, wherein FLOPs = 2MACs, thereby realizing the conversion of operator granularity overhead modeling to model granularity overhead prediction, as shown in the following formula:

[0046]

[0047] wherein: L model is the predicted model delay; L conv is the predicted convolution operator delay; MACs model is the multiplication-addition operation number of the target model; MACs conv is the multiplication-addition operation number of the convolution operator in the target model.

[0048] Further, it also includes:

[0049] Step 3, customizing, compiling and flashing the Linux kernel for mobile heterogeneous hardware, and confirming the inference framework, inference components, core type, core number, and core running frequency required for inference;

[0050] Step 4, based on the inference framework, inference components, core type, core number, and core running frequency, the delay overhead prediction model is quickly migrated between different inference configurations;

[0051] Wherein, in the case of homogeneous cores, different inference frameworks use different correction points to correct parameters, thereby migrating different core numbers; and in the case of differences between inference frequency and core type, the cpu capicity of the linxu kernel is read for proportional conversion.

[0052] Further, step 3 is as follows:

[0053] An Android GKI kernel with userspace cpufreq scheduler is constructed, a custom design is made on the public kernel, custom permissions are enabled, the userspace scheduler is restarted to enable custom device frequency adjustment, the CONFIG_CPU_FREQ_GOV_USERSPACE = m option in the linux kernel is enabled, the userspace scheduler is compiled into the custom kernel as an external module, the bootloader and vab partition verification of the device are closed, and the new kernel is flashed, and the hardware level can control the running frequency gear of the CPU by writing scaling_setspeed.

[0054] Further, in step 4, for the case where the core number changes in the homogeneous core inference, the correction point sampling varies with the platform, wherein:

[0055] For the inference platform onnx, l min The left endpoint and As the right endpoint, only one sampling l min The left endpoint can be refitted to a straight line, l max is the right endpoint; core_num is the core number;

[0056] For the inference platform TensorFlow Lite, when the width is determined, only the leftmost and rightmost platform values need to be sampled for direct fitting, and the sampling point output channel number of the rightmost platform is a multiple of width, and |{c0, c1, c2, …}| mod 8 = 0;

[0057] For the inference platform ncnn, the curve becomes bilinear fitting, which requires two curve starting point samplings, and the sparse fitting line l AThe correction points of (x) are fitted from the output of 8.

[0058] Further, in step 4, for migration between heterogeneous cores or different frequencies, when the core type and frequency change, the cpu capacity is converted, and the cpu capacity conversion formula is as follows:

[0059]

[0060] Wherein: l conf1 is the predicted delay under configuration 1; l conf is the predicted delay under configuration 2; capicity core1 is the normalized computing power index of core1 core; capicity core2 is the normalized computing power index of core2 core; freq core1 is the inference delay of the inference core under configuration 1; freq core2 is the inference delay of the inference core under configuration 2; freq core_max is the maximum frequency at which the inference core used in configuration 2 can run; freq core1_max is the maximum frequency at which the inference core used in configuration 1 can run.

[0061] Compared with the prior art, the present application has the following advantages:

[0062] (1) The present application proposes a mobile terminal delay overhead modeling method combined with the inference delay characteristics of multiple inference frameworks, which can efficiently and lowly sample and model the delay of multiple inference frameworks, effectively reducing the number of redundant samples and a large amount of measurement time overhead.

[0063] (2) The present application realizes the frequency locking of the mobile terminal device chip through the self-defined GKI Android kernel, further guarantees the accuracy and stability of the sampling, and cooperates with the modeling design of multiple platforms, which can reduce the delay prediction model construction overhead while guaranteeing the prediction accuracy.

[0064] (3) The present application uses linear gradient fitting, piecewise function modeling design, and uses sliding average to remove noise, and models the delay overhead of onnx, TensorFlow Lite, ncnn three kinds of mobile terminal commonly used inference platforms, designs the overhead model migration method between different inference configurations, realizes the service of existing parameters, and further reduces the system overhead of sampling.

[0065] The present application solves the problems of existing mobile heterogeneous devices in the face of various inference platforms and inference configurations, inference models, delay overhead prediction modeling difficulty, low precision caused by software and hardware diversity. By fully utilizing the characteristics of different inference framework platforms found, the platform related delay curve modeling and efficient sampling method is designed, and the fast migration method between device configurations is designed. It can guarantee the accuracy of delay overhead prediction while reducing the number of sampled operators and overhead as much as possible, and is suitable for mobile model deployment, model adaptation, end-side inference and other application scenarios that require a large amount of manpower and device cost to deploy models on the end side. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is a mobile end heterogeneous platform multi-inference framework delay accurate sampling and overhead modeling framework schematic diagram of the present embodiment;

[0067] Figure 2 is a process flow diagram for operator granularity model export optimization for multiple platforms of the present embodiment;

[0068] Figure 3 is a flowchart of operator delay accurate sampling and delay overhead modeling under a mobile end heterogeneous processor of the present embodiment;

[0069] Figure 4 is a three-dimensional delay schematic diagram of a multi-size similar configuration convolution operator model under the onnx inference framework of the present embodiment;

[0070] Figure 5 is a two-dimensional delay schematic diagram of a multi-size similar configuration convolution operator model under the onnx inference framework of the present embodiment after fixing the number of input or output channels;

[0071] Figure 6 is a three-dimensional delay schematic diagram of a multi-size similar configuration convolution operator model under the TensorFlow Lite (LiteRT) inference framework of the present embodiment;

[0072] Figure 7 is a two-dimensional delay schematic diagram of a multi-size similar configuration convolution operator model under the TensorFlow Lite (LiteRT) inference framework of the present embodiment after fixing the number of input or output channels;

[0073] Figure 8 is a three-dimensional delay schematic diagram of a multi-size similar configuration convolution operator model under the ncnn inference framework of the present embodiment;

[0074] Figure 9 is a two-dimensional delay schematic diagram of a multi-size similar configuration convolution operator model under the ncnn inference framework of the present embodiment after fixing the number of input or output channels. DETAILED DESCRIPTION

[0075] The application will be further described below in combination with the drawings and examples.

[0076] The embodiment is based on the analysis of the hardware heterogeneity and computing power level of mobile heterogeneous platforms, as well as the delay characteristics of various mobile inference frameworks. By utilizing the discovered platform characteristics, different delay overhead modeling schemes and efficient sampling methods are designed for three different inference frameworks (onnx, TensorFlowLite, and ncnn), realizing efficient sampling and overhead modeling methods for delay under mobile heterogeneous platform multi-inference framework.

[0077] Figure 1 The real scene and overall process to which the embodiment is directed are shown in the cloud server. The cloud server uses the powerful computing power of the cloud to use the mainstream deep learning framework TensorFlow or PyTorch to pre-train the model and push it to the edge server for model compression and model pruning, fine-tuning or model adaptation. Due to the differences in computing power, storage, and inference framework selection of mobile devices, the same model cannot be used on all platforms, and simple structured pruning cannot well adapt to the diverse characteristics of mobile devices. Therefore, in this process, the performance (inference delay) of the model on a specific hardware device, specific inference platform and configuration needs to be measured, and the model structure needs to be adjusted repeatedly to send it to the mobile phone for real machine deployment to obtain the real performance of the delay evaluation model.

[0078] The method based on the actual measurement has too much overhead, and the method based on prediction is not accurate and efficient, and has poor portability. Figure 1 The overall technical framework of the embodiment is shown in the figure. The obtained model structure usually only performs structural pruning on the width of the convolution layer, and other configurations (such as step, convolution kernel size, etc.) will not change. In order to efficiently collect as few delay samples as possible and adapt to the platform characteristics, the model is first converted to a format suitable for mobile platforms, and then according to the inference framework of the platform to be deployed, an efficient delay sampling decision is made to facilitate subsequent accurate and efficient delay curve modeling and fitting.

[0079] The sampling decision part here is mainly designed for the characteristics of the three platforms. Onnx uses simple linear fitting, TensorFlow Lite uses a linear step function model, and ncnn uses a segmented function composed of segmented linear fitting due to the use of NEON optimization according to its bilinear characteristics. At the same time, considering the differences in inference configuration (mainly the core types used - large, medium and small cores, the number of homogeneous cores, and CPU / GPU frequency levels), low-overhead migration is performed. For example, previously inferred configuration options can be corrected by sampling a small number of points to correct the existing overhead model parameters, which can efficiently migrate the existing predictor.

[0080] Figure 1 After determining the operator granularity convolution model configuration that needs to be sampled, the operator model can be generated. Then the terminal device uses a specific inference framework to perform inference on a specific co-processor (CPU / GPU) with a specific number of cores and configuration, and a specific frequency level. Due to the integration of the linux kernel in the android AOSP, the new generation of mobile phones using the GKI kernel cannot be sampled at a fixed frequency. In this embodiment, the GKI linux kernel is customized to make it possible to run at a fixed hardware frequency, ensuring the stability of the model inference performance during sampling and providing a basis for building a relatively accurate delay model in the future.

[0081] After model inference of the sampled convolution operator, if all configurations are measured using the actual measurement method, the delay measurement is acceptable, and the delay lookup table method is used. If the number of samples is too large, a prediction scheme is used to build a delay overhead model.

[0082] Finally, the delay overhead model is built. If the actual measurement method is used, the sum of all operator delays can be obtained by looking up the table directly. If the prediction method is used, a new predictor is built or an old similar predictor is calibrated according to the platform characteristics and sampling decision logic.

[0083] As shown in the execution mode of Figure 1 , this embodiment builds a delay modeling mechanism that is adaptive to mobile heterogeneous hardware platforms and software inference frameworks, and can quickly migrate different inference configurations and inference frameworks to achieve efficient and low-overhead end-side inference delay modeling in a multi-source heterogeneous edge computing environment.

[0084] The delay overhead prediction method of this embodiment for mobile heterogeneous platform multi-inference framework is as follows:

[0085] Step 1, obtain the original model, parse the configuration of the original model, and analyze the similar convolution operators in the original model. The original model has layers L = {l0, l1, l2, l3, …, l n-1 , n}Separate all convolution operators from the n-layer model as the main basis for subsequent sampling and delay overhead prediction model modeling; then group similar configurations of convolution operators in the convolution operator, reduce repeated convolution operator configurations, and reduce the number of convolution operators that need to be sampled.

[0086] Step 1 specific process as follows:

[0087] Step 101, obtain the model layer information and parameters of the original model, and determine whether the model layer is a convolution operator according to the model layer information and parameters, thereby dividing the model layer L into two parts, L = L other + L conv , where L conv is a convolution operator, and L othe is a non-convolution operator.

[0088] Convolution operator L conv Since it is the main contribution part of the deep neural network inference delay, the configuration of the convolution operator L conv is parsed to obtain the similar convolution operator. The delay of other non-convolution operators L otheer is subsequently converted according to the specific proportion of the convolution operator delay calculation complexity FLOPs to obtain the overall inference delay of the model granularity. As shown in the following formula:

[0089] L conv = {l j | l j ∈ L and l j is convlayer}

[0090] L other = {l i | l i ∈ L and l i is not convlayer}

[0091] Where: L conv is a convolution operator, and L other is a non-convolution operator.

[0092] Step 102, in addition to the input channel number, the output channel number, the same convolution operator in other configurations (convolution kernel size, padding size, input resolution, etc.) in the model is divided into a group, and the convolution operator is divided into multiple groups according to similar configurations; Then, the maximum input channel number and the maximum output channel number of all convolution operators in each group are counted, so as to determine the sampling base in each group as cout max ·cin max , wherein cout max is the maximum output channel number in the group, and cin max is the maximum input channel number in the group.

[0093] As shown in Figure 2 , the convolution operators in the original model structure are configured and grouped, the matrix height and width, the step length, and the convolution kernel size are grouped into a group, which is called a convolution operator with similar configuration; All convolution operators in the model are analyzed into multiple groups Group i , and the configuration inside each group is analyzed to obtain the maximum input channel cin max and the maximum output channel cout max .

[0094] Group i ={l j ∣l j ∈L conv and have similar config i}

[0095] Among them: Group i is a group, which is composed of all convolution operators with similar configurations.

[0096] The base of sampling in each convolution group is cout max ·cin max , as shown in Figure 2 , there are two sampling methods, which correspond to the modeling schemes of measurement and prediction respectively: if the time overhead brought by the sampling number is acceptable or the precision needs to be as high as possible, the measurement method is used to derive and measure all size convolution operators; If the measurement overhead is unacceptable or a high-efficiency modeling method is needed, the sample number of the sampling can be greatly reduced according to the subsequent prediction-based modeling method.

[0097] Step 2, the delay overhead prediction model is modeled according to the convolution operator inference characteristics of different inference platforms, wherein:

[0098] For the inference platform onnx, a simple linear fitting is used to predict the delay overhead of all group convolution operators obtained in step one. For the inference platform TensorFlow Lite, a linear step function is used to predict the delay overhead of all group convolution operators obtained in step one. For the inference platform ncnn, since the inference platform ncnn uses NEON optimization, according to the bilinear characteristics of ncnn, a segmented function composed of segmented linear fitting is used to predict the delay overhead of all group convolution operators obtained in step one.

[0099] Then, according to the proportion of the convolution operator part in the overall complexity of the prediction model, and combined with the predicted delay of the convolution operator obtained by the delay overhead prediction model under each inference platform, the overall delay of the prediction model is proportionally inferred.

[0100] The specific process of step 2 is as follows:

[0101] Step 201, for the onnx inference platform, a simple linear fitting method is used for delay overhead prediction model modeling.

[0102] Specifically, by fixing the output channel number cout, fitting the input channel number cin as the independent variable, and the convolution operator delay l as the dependent variable, since there is obvious linearity, interval step sampling is used to obtain the convolution operator delay under different input channel numbers to reduce the sampling overhead, and a simple linear fitting is performed on the convolution operator delay overhead curve.

[0103] As shown in Figure 4 , for a convolution operator of a specific group, x i ranges from cout min to cout max or cin min to cin max , combined with Figure 5 analysis, it is found that when the input channel is fixed, the output channel will have a significant jump when it is 1, and the fitting curve is not flat. Only by fixing the output channel can linear fitting be performed on all conditions to obtain better prediction accuracy and universality.

[0104] Therefore, the overhead modeling under the onnx framework is as follows: the input of the linear function is the input channel, and the output is the convolution layer delay. The output channel is fixed, and the delay overhead prediction model l is established as shown in the following formula:

[0105] l=ax i +b

[0106] x i is cin in Group is.t.cout dimension fixed

[0107] where: a is the slope of the linear fit; x i is the Group i convolution allowed cin size, needs to be less than cin max , this curve is applicable when cout is fixed to a certain value from 1 to cout max ; b is the intercept of the linear fit.

[0108] The goal of the fit is to minimize the sum of the squares of the residuals, as shown in the following equation:

[0109]

[0110] where, l i is the delay of the i-th operator, x i is the input channel size in Group i .

[0111] Therefore, the parameters a, b of the linear fit can be obtained using the following equation:

[0112]

[0113] where: is the average input channel size of n operators; is the average delay of n operators; n is the maximum input channel size cin i of the operators in Group max .

[0114] Step 202, for the TensorFlow Lite inference platform, a linear step function fitting is used for delay overhead prediction model modeling.

[0115] Specifically, based on the discovered equal-width segmentation and equal-difference step properties in the delay curve, a linear step function is designed for fitting. By fixing the input channel size cin, the output channel size cout is taken as the independent variable, and the convolution operator delay l is taken as the dependent variable, the operator delay is sampled and the continuous sampling point data is smoothed, the step width is estimated after finding the jump point, and the parameters of the linear step function can be fitted.

[0116] Through Figure 6 and Figure 7 observation and analysis, it can be found that when the output channel is fixed, the linear gradient and the later gradient difference is large when the input channel number is low, and the fitting precision is reduced by linear fitting. When the input channel is fixed, it shows obvious step property, with equal-width segmentation and equal-difference step phenomenon, so fixing the input channel can maximize the fitting precision.

[0117] Therefore, the latency overhead prediction model l(x) established for the TensorFlow Lite platform is as follows:

[0118] Where: x i For x i It is a group i The number of output channels in the value; Δheight is the step height, height0 is the starting height, and width is the step width (usually 8).

[0119] First, sampling and smoothing are performed. If this is the first time cost modeling is done, noise needs to be eliminated before finding transition points to determine the step width. Fluctuations can interfere with the judgment, so smoothing is performed first. Noise is handled using moving average (MAD). For each point l... i Define its smoothing value The mean of the surrounding 2k+1 points is given by the following formula:

[0120]

[0121] Then, we need to find the transition points to determine the step width for the smoothed sequence. Calculate the first difference Detect all d i In the middle, satisfy |d i The points where |>θ, where θ is a manually set hyperparameter threshold, are called step transition points, denoted as θa. The horizontal distance between the transition points is the step width.

[0122] To ensure the highest possible accuracy when initially calculating the step width, the average value is typically used to determine the width, as shown in the formula below.

[0123]

[0124] Where: m is the number of step transition points in the delay curve; Let x be the x-coordinate of the (i+1)th step transition point; Let x be the x-coordinate of the i-th step transition point.

[0125] When performing overhead modeling on specific hardware devices for the first time, the step width can be reused without the above steps. Only the following steps are needed to determine Δheight and height0:

[0126] For ease of understanding, all data is arranged according to step number b. i Grouping: Each x i Belongs to the b iThere are 10 steps. For each step numbered b, the corresponding average height is calculated using the following formula:

[0127]

[0128] Where S b ={i∣b i Let {b} represent all points belonging to the b-th step. Solving using the formula yields the average height of a set of steps {c0, c1, c2, ...}. Then, the parameters Δheight and height0 of the linear step function are calculated based on the step heights, as shown in the following equation:

[0129]

[0130] height0 = c min =c0

[0131] Where: c max c is the maximum step height. min The minimum step height is denoted by |{c0,c1,c2,…}|, which represents the number of steps.

[0132] c b The solution process does not require calculating the height of all steps; it only needs to determine c based on the already determined step width. max and c min This allows for a significant reduction in the number of samples while achieving high fitting accuracy.

[0133] Step 203: For the ncnn inference platform, a piecewise function is used to perform two-dimensional linear fitting to model the delay overhead prediction model.

[0134] Specifically, by fixing the number of output channels cout, fitting the number of input channels cin as the independent variable, and the convolution operator delay l as the dependent variable, we classify and model the coefficient straight line and dense straight line. Since both the coefficient bifurcation line and the dense bifurcation line have obvious linearity, we sample the actual operator delay in the delay curve at equal intervals to reduce sampling overhead.

[0135] like Figure 8 and Figure 9 As shown, observation and analysis reveal that when the input channel is fixed, the delay curve fluctuates repeatedly and cannot be well fitted, resulting in an uneven fitted curve. However, when the output channel is fixed, it exhibits a clear two-dimensional linear property. Points where the input width is divisible by 8 can be fitted as a sparse straight line, while other points can be fitted as a dense curve. By linking these points together using a piecewise function, better prediction accuracy and universality can be achieved.

[0136] Therefore, the latency overhead prediction model established for the ncnn platform is shown in the following formula.

[0137]

[0138] Where: l(x i ) is the predicted delay obtained from the fitting, l A (x)=a1x i +b1、l B (x)=a2x i +b2 corresponds to the fitting results of the sparse line and the principal line, respectively; x i For Group i The allowed size of the cin independent variable for group convolution; a1 is the slope of the sparse linear curve; a2 is the slope of the dense linear curve; b1 is the intercept of the sparse line; b2 is the intercept of the dense line; imod 8 = 0 means that the sparse line applies to sparse points where cin is divisible by 8.

[0139] Based on the observed phenomena, the data is divided into subsets A and B, respectively. and Corresponding to two straight lines l respectively A (x)=a1x+b1 and l B (x) = a2x + b2.

[0140] Among them, the relative coefficient of curve A only conforms to the fitting law of the coefficient line when the number of input channels is divisible by 8. For other dense curves, similarly to the ONNX fitting method, a step-by-step sampling method can be used to reduce sampling overhead and maintain prediction accuracy.

[0141] Step 304: Based on the computational complexity (MACs / FLOPs) of the convolution operator in the model to be predicted under each inference platform, and combined with the prediction latency of all convolution operators in the model to be predicted under each inference platform, the overall latency of the model to be predicted is proportionally inferred.

[0142] Since the delay in the model consists of both non-convolution operators and convolution operators, we have the following equation:

[0143]

[0144] Where: model_latency represents the overall latency of the model; Indicates the delay of the convolution operator; This indicates the delay of the non-convolution operator.

[0145] As the main component of the delay, the convolution operator can be proportionally converted based on the computational complexity of FLOPs or MACs, where FLOPs = 2MACs. This realizes the transformation from operator-level cost modeling to model-level cost prediction, and the calculation formula is shown below.

[0146]

[0147] wherein: L model is the model granularity latency to be predicted; L conv is the latency of all convolution operators in the prediction model; MACs model is the total complexity of the prediction model; MACs conv is the complexity of all convolution operators in the prediction model.

[0148] Step 3, customizing, compiling and flashing the Linux kernel for mobile heterogeneous hardware, ensuring the stability of the frequency and the accuracy of the sampling results in the model inference process; at the same time, confirming the inference framework to be used, the inference component device_type = {cpu, gpu}, the core type core_type = {big, middle, little}, and the core number core_num, the core running frequency freq, providing operating system kernel basis and inference configuration overhead modeling for fine sampling, facilitating accurate and stable sampling and subsequent delay overhead cross-configuration migration. The specific process is as follows:

[0149] Step 301, build an Android GKI kernel with userspace cpufreq governor, as shown in Figure 3 Since Android 12 has begun to force all Android device OEM manufacturers to integrate driver configurations, custom design is done on the public kernel, so many custom permissions are closed as public configurations, and userspace governor is no longer provided to allow users to adjust device frequency. Therefore, after downloading the official kernel source code, use Kleaf to manually enable the CONFIG_CPU_FREQ_GOV_USERSPACE = m option, compile the userspace governor as an external module into the custom kernel, close the bootloader and vab partition verification of the device, and then flash the new kernel. By writing scaling_setspeed, the running frequency gear of the CPU can be controlled.

[0150] Step 302, model the hardware configuration of inference, as shown in the following hardware_conf j formula, so as to facilitate subsequent cross-configuration migration of the delay overhead model, as shown in Figure 3 The device_type, core_num, core_type and freq of the inference component are configured.

[0151] Therefore, it can be modeled as hardware_conf j , where the core type has big, middle and small cores, as shown in Figure 3As shown, mobile end CPU mostly adopts heterogeneous design, taking the CPU in the figure as an example, which adopts three cluster design, consisting of one A77 big core, three A77 medium cores and four A55 small cores, and the highest computing power from high to low is as follows:

[0152] hardware_conf j ={device_type,core_type,core_num,freq}

[0153] Step 4, the inference configuration of the mobile terminal may change according to different inference tasks and models, and a fast migration scheme for different configurations is designed in this embodiment, based on the inference framework, inference components, core type, core number and core running frequency, the delay overhead prediction model for different inference configurations (different frequencies, different core numbers and types used) is migrated quickly.

[0154] Among them, in the case of isomorphic core core_type, different inference frameworks use different correction points to correct parameters, thereby migrating different core numbers core_num; and in the case of difference between inference frequency freq and core type core_type, the cpu capicity of linxu kernel is read for proportional conversion. The specific process is as follows:

[0155] If the isomorphic core is used for inference, only the core number changes, and since the product of the delay l(x i ) and the core number core_num is not directly inversely proportional, it cannot be directly multiplied by the coefficient as a new curve, because the communication overhead between cores is very common, so it needs to be corrected based on the original modeling, and the correction point sampling varies with the platform.

[0156] For the inference platform onnx, take l min left end point and as the right end point to re-linearly fit, because when the delay is large, the synchronization overhead is low, and only when the inference delay itself is low, the communication overhead will become a bottleneck, which will have a greater impact on performance, so only one sampling of l min left end point can be re-fitted. Wherein l max is the right end point (maximum delay).

[0157] For the inference platform TensorFlow Lite, since the width is determined, only the leftmost and rightmost platform values need to be sampled to directly fit, and the sampling point output channel number of the rightmost platform is a multiple of width, which is convenient to confirm the platform height, because |{c0, c1, c2, …}| mod 8 = 0 at this time.

[0158] The approach for the inference platform ncnn is similar to onnx, except that the curve becomes bilinear fitting, which requires sampling of the starting point of the curve twice, and the sparse fitting line l A The correction point of (x) is fitted from the output of 8, because this is the starting point of the sparse fitting line.

[0159] For migration between heterogeneous cores or different frequencies, the core type core_type and the frequency freq change, which are converted through capicity (in linux / sys / devices / system / cpu / cpu* / cpu_capacity) (only cpu has heterogeneous design, which involves cross-core migration problem, and gpu can only cross frequency), and the conversion formula is as follows:

[0160]

[0161] Where: l conf is the predicted delay under configuration 1; l conf is the predicted delay under configuration 2; capicity co is the normalized computing power index of core1 core; capicity core2 is the normalized computing power index of core2 core; freq co is the inference delay of the inference core under configuration 1; freq core is the inference delay of the inference core under configuration 2; freq cor_max is the maximum frequency that the inference core used in configuration 2 can run; freq core1_max is the maximum frequency that the inference core used in configuration 1 can run.

[0162] The embodiment also discloses a kernel modification scheme for fixing comments of a GKI kernel mobile terminal device, which modifies kernel configuration and scaling_setspeed virtual file writing to support custom frequency modulation, and ensures the stability and precision of delay overhead sampling.

[0163] The framework builds a delay modeling mechanism with adaptability to mobile terminal heterogeneous hardware platforms and software inference frameworks, can efficiently sample and quickly migrate for different inference configurations and inference frameworks, and realizes efficient and low-overhead end-side inference delay overhead modeling and prediction in a multi-heterogeneous edge computing environment, optimizes based on the characteristics of different inference frameworks, and balances sampling overhead and prediction accuracy.

[0164] The embodiment also discloses a kernel modification scheme for fixing comments of a GKI kernel mobile terminal device, and custom frequency modulation is supported by modifying kernel configuration and writing into a scaling_setspeed virtual file, so that stability and precision of delay overhead sampling are ensured.

[0165] The preferred embodiments of the present application are described in detail above with reference to the drawings, and the embodiments described in the present application are merely used to describe the preferred embodiments of the present application, and do not limit the concept and scope of the present application. In the above specific embodiments, various specific technical features are described, and in the case of no contradiction, any suitable combination can be combined, and the combination shall be regarded as the disclosure of the present application as long as it does not deviate from the concept of the present application. In order to avoid unnecessary repetition, various possible combinations are not described again in the present application.

[0166] The present application is not limited to the specific details described in the above embodiments, and various modifications and improvements of the technical solutions of the present application made by those skilled in the art within the scope of the technical concept of the present application and without departing from the design concept of the present application shall fall within the protection scope of the present application, and the technical content of the present application claimed for protection has been fully recorded in the claims.

Claims

1. A latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms, characterized in that, The process is as follows: Step 1: Obtain the original model and perform configuration parsing on the original model to obtain convolution operators with similar configurations in the original model, which will serve as the main basis for subsequent sampling and overhead model modeling; then, group and merge convolution operators with similar configurations to reduce duplicate convolution operator configurations and reduce the number of convolution operators that need to be sampled. Step 2: Model latency overhead prediction models for the inference characteristics of convolution operators on different inference platforms, where: For the inference platform ONNX, a delay overhead prediction model is built for all groups of convolution operators obtained in step one using simple linear fitting. For the inference platform TensorFlow Lite, a linear step function is used to model the latency overhead prediction model for all groups of convolution operators obtained in step one. For the inference platform ncnn, since the inference platform ncnn uses NEON optimization, based on the bilinear features of ncnn, a piecewise function composed of piecewise linear fitting is used to model the delay overhead prediction model for all groups of convolution operators obtained in step one. Then, based on the computational complexity of the convolution operator part in the prediction model as a percentage of the overall prediction model, and combined with the prediction latency of the convolution operator obtained from the latency overhead prediction model under each inference platform, the overall latency of the prediction model can be inferred proportionally.

2. The latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms according to claim 1, characterized in that, Step 1 is as follows: Step 101: Obtain the model layer information and parameters of the original model. Based on the model layer information and parameters, determine whether the model layer is a convolution operator. Therefore, divide the model layer L into two parts, with L = L other +L conv L conv For the convolution operator, L other For non-convolution operators; then for convolution operators L conv The configuration is parsed to obtain convolution operators with similar configurations; Step 102: Except for the number of input channels and the number of output channels, group the convolution operators with the same configuration in the model together, thus dividing the convolution operators into multiple groups based on similar configurations; then, count the maximum number of input channels and the maximum number of output channels for all convolution operators within each group, thereby determining the sampling base of each group as cout. max ·cin max where cout max cin is the maximum number of output channels within the group. max This represents the maximum number of input channels within the group.

3. The latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms according to claim 1, characterized in that, In step 2, the modeling process using simple linear fitting for the inference platform ONNX is as follows: With a fixed number of output channels (cout), the number of input channels (cin) is fitted as the independent variable, and the convolution operator delay (l) is the dependent variable. Interval step-by-step sampling is used to obtain the convolution operator delay under different numbers of input channels to reduce sampling overhead. A simple linear fit is performed on the convolution operator delay overhead curve, and the established delay overhead prediction model (l) is shown in the following equation: l=ax i +b x i is cin in Group i s.t.cout dimension fixed Where: a is the slope of the linearly fitted curve; x i For Group i The allowed size of cin for group convolutions must be less than cin. max This curve is applicable when cout is fixed at 1 to cout max The case of a specific value in the curve; b is the intercept of the linearly fitted curve.

4. The latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms according to claim 1, characterized in that, In step 2, the modeling process using a linear step function for the TensorFlow Lite inference platform is as follows: Based on the properties of the delay curve, such as its moderately wide segmentation and arithmetic step, and by fixing the number of input channels (cin), fitting the number of output channels (cout) as the independent variable, and the delay of the convolution operator (l) as the dependent variable, the operator delay is sampled and the continuous sampled data is smoothed. After finding the jump point, the step width is estimated, and the parameters of the linear step function can be fitted. Thus, the delay overhead prediction model l(x) is established as shown in the following equation: Where: x i For Group i The allowed size of the cout argument for group convolution; Δheight is the step height, height0 is the starting height, and width is the step width; When modeling the delay overhead prediction model for the first time, the step width is determined first. After processing noise through moving average, the jump point is found to obtain the width, and then the smoothed sequence is processed. Calculate the first-order difference Detect all first-order differences d i In the middle, satisfy |d i The points that are |>θ, where θ is a manually set hyperparameter threshold, are called step transition points, denoted as θa. The initial calculation of the step width is performed using the following formula: Where: m is the number of step transition points in the delay curve; Let x be the x-coordinate of the (i+1)th step transition point; Let x be the x-coordinate of the i-th step transition point; Then, for different steps Get the corresponding step height Where S b ={i∣b i =b} represents all points belonging to the b-th step; The parameters Δheight and height0 of the linear step function are calculated based on the step height, as shown in the following formula: height0=c min =c0 Where: c max c is the maximum step height. min The minimum step height is denoted by |{c0,c1,c2,…}|, which represents the number of steps.

5. The latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms according to claim 1, characterized in that, In step 2, the modeling process for the piecewise function composed of piecewise linear fitting of the inference platform ncnn is as follows: A piecewise function is used for two-dimensional linear fitting. The number of output channels (cout) is fixed, the number of input channels (cin) is fitted as the independent variable, and the convolution operator delay (l) is the dependent variable. Modeling is performed for both linear and dense linear coefficients. Since both linear and dense linear coefficients exhibit significant linearity, equal-interval step-by-step sampling is used to reduce sampling overhead. The resulting delay overhead prediction model l(x) is then established. i The formula is shown below: Where: x i For Group i The allowed size of the cin independent variable for group convolution; a1 is the slope of the sparse linear curve; a2 is the slope of the dense linear curve; b1 is the intercept of the sparse line; b2 is the intercept of the dense line; imod 8 = 0 means that the sparse line applies to sparse points where cin is divisible by 8.

6. The latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms according to claim 1, characterized in that, In step 2, the process of proportionally inferring the overall delay of the model is as follows: Using the convolution operator as the main delay component, a scaling factor is applied based on FLOPs (Functional Operating Procedures) or MACs (Macroscopy and Computational Costs), where FLOPs = 2MACs. This achieves the transformation from operator-level cost modeling to model-level cost prediction, as shown in the following formula: Where: L model The predicted model delay; L conv Delay for the predicted convolution operator; MACs model For predicting the multiply-accumulate operations of the target model; MACs conv To predict the number of multiply-accumulate operations of the convolution operator in the target model.

7. The latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms according to any one of claims 1-6, characterized in that, Also includes: Step 3: Customize, compile, and flash the Linux kernel for heterogeneous mobile hardware, and confirm the inference framework, inference components, kernel type, number of kernels, and kernel operating frequency required for inference. Step 4: Based on the inference framework, inference components, core types, number of cores, and core operating frequency, perform rapid migration of latency overhead prediction models between different inference configurations; In cases where the cores are homogeneous, different inference frameworks use different correction points to correct the parameters, thereby migrating to cases with different numbers of cores; while in cases where there are differences between inference frequency and core type, the CPU capacity of the Linux kernel is read for proportional conversion.

8. The latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms according to claim 7, characterized in that, Step 3 is as follows: An Android GKI kernel with a userspace cpufreq scheduler is built. Custom designs are made on the public kernel, custom permissions are enabled, the userspace scheduler is restarted to enable custom device frequency adjustment, and the CONFIG_CPU_FREQ_GOV_USERSPACE=m option in the Linux kernel is enabled. The userspace scheduler is compiled as an external module into the custom kernel. After disabling the device's bootloader and VAB partition verification, the new kernel is flashed. At the hardware level, the CPU's operating frequency level can be controlled by writing scaling_setspeed.

9. The latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms according to claim 7, characterized in that, In step 4, for the case where only the number of cores changes in isomorphic core inference, the sampling of correction points varies depending on the platform, where: For the inference platform ONNX, take l min left endpoint and As the right endpoint, re-linear fitting only requires one sampling. min The left endpoint can then be used to refit the line. max The right endpoint is `core_num`, which represents the number of cores. For the inference platform TensorFlow Lite, once the width is determined, we only need to sample the values ​​of the leftmost and rightmost platforms to directly perform fitting. The number of output channels of the sampling points of the rightmost platform is a multiple of the width, at which point |{c0,c1,c2,…}|mod8=0; For the inference platform ncnn, the curve becomes a bilinear fit, requiring two samplings of the curve's starting point, resulting in a sparse fitting line l A The correction point for (x) is fitted starting from the output of 8.

10. The latency overhead prediction method for multiple inference frameworks on heterogeneous mobile platforms according to claim 7, characterized in that, In step 4, migration is performed between heterogeneous cores or between different frequencies. When the core type and frequency change, CPU capacity is used for conversion. The CPU capacity conversion formula is as follows: Where: l conf For prediction delay under configuration 1; l conf2 For prediction latency under configuration 2; capicity core1 The normalized computing power index for core 1; capicity core2 The normalized computing power index for core2; freq core1 To configure the inference latency of the inference core under configuration 1; freq core2 To configure the inference latency for two inference cores; freq core_max The maximum frequency at which the inference cores used in configuration 2 can run; freq core_max The maximum frequency at which the inference core used for configuration 1 can run.

Images