Neural network dynamic exit lightweight method and system for multiple consecutive inferences

Abstract

The application provides a neural network dynamic exit lightweight method and system for multiple continuous inferences, comprising: step 1: constructing a neural network-based inference model, predicting the position of network exit for each inference within a preset small time range, and accordingly predicting the calculation configuration, the calculation configuration comprising frequency and voltage; for multiple inferences within a preset large time range, performing processor frequency and voltage calibration through residual inference workload and time constraints; step 2: performing the neural network according to the predicted and calibrated calculation configuration, thereby realizing dynamic voltage and frequency adjustment. Compared with a classic deep learning network, the application can realize energy saving up to 63.8%, while ensuring that multiple neural network inferences are completed within a specified time, the inference can be terminated in advance through early exit to obtain accurate results, and the calculation and energy costs are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, in particular to a neural network dynamic exit lightweight method and system for multiple continuous inferences. BACKGROUND

[0002] Deep learning methods, such as convolutional neural networks, have achieved great success in multi-purpose applications. However, one of the challenges of deploying deep learning models on resource-constrained systems is their huge energy cost. As a dynamic inference method, early exit adds an exit layer to the network, which can terminate inference early and obtain accurate results to save energy. The passive decision of energy regulation for early exit cannot adapt to the ongoing inference state, different inference workloads and time constraints, not to mention guiding the reasonable configuration of the computing platform during inference to save potential energy.

[0003] Patent document US20210056357A1 discloses a system and method for implementing a flexible, input-adaptive deep learning neural network; patent document US20210012178A1 discloses a system, method and device for early exiting convolution; patent document EP3997621A1 discloses a system, method and device for early exiting convolution. The above documents propose methods for early exit, but cannot realize prediction of the early exit point.

[0004] Patent document CN114997370A discloses a low-power neural network system based on prediction exit and an implementation method thereof. Although the patent document proposes prediction of the early exit point, it can only predict single neural network inference and reduce computational load and system energy consumption, and the prediction accuracy is low, cannot accurately predict from the first few layers of the network or even the first time, and cannot accurately predict the early exit point for large network models. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a neural network dynamic exit lightweight method and system for multiple continuous inferences.

[0006] The neural network dynamic exit lightweight method for multiple continuous inferences provided by the present application comprises:

[0007] Step 1: Construct a neural network-based inference model, for each inference within a preset small time range, predict the position of network exit, and accordingly predict the calculation configuration, the calculation configuration including frequency and voltage;

[0008] Step 2: For multiple inferences within a preset large time range, processor frequency and voltage calibration is performed through residual inference workload and time constraints;

[0009] Step 3: configure the calculation of the neural network according to the prediction and calibration to perform the neural network, so as to realize dynamic voltage and frequency adjustment.

[0010] Preferably, the coordination period of the inference is set during the inference process, that is, the inference task deadline, expressed as:

[0011]

[0012]

[0013] Wherein, λ represents the tightness of the task deadline, 0 < λ ≤ 1, the smaller λ is, the tighter the deadline is; represents the actual completion time of each inference; is the coordination period; M is the number of inference tasks; represents the coordination deadline of the i-th inference task.

[0014] Preferably, the power consumed in the calculation process during the execution of the inference task is the active power , the dynamic power , the static power , and the constant power ;

[0015] The power consumed when the calculation platform is idle is the idle power , which is the sum of the static power and the constant power ;

[0016]

[0017]

[0018] Wherein, C is the capacitance of the switch logic gate; V represents the supply voltage of the processor; f represents the clock frequency of the processor; is the number of logic gates; is the normalized static current of each logic gate;

[0019] The energy consumption of each inference in the calculation platform is:

[0020]

[0021] The network layer located before the early exit runs at , , after the network exits at point, the calculation platform reduces the voltage and frequency to the lowest level , by the dynamic voltage frequency adjustment technology DVFS, therefore, the inference task Energy consumption is expressed as:

[0022]

[0023] Representational reasoning task Energy consumption; This indicates the high power supply voltage of the processor; Indicates a high processor clock frequency; This indicates the processor's low power supply voltage.

[0024] Preferably, in a reasoning task First, predict the network exit position and establish the remaining layers to complete the inference; then, based on the remaining layers and the worst-case inference time, execute the inference. Until the current inference ends, establish (V,f) for prediction; based on The remaining inference workload is used to enable the cross-inference calibrator through a feedback control strategy to calibrate (V,f) based on multi-inference progress, total inference workload, and time constraints; based on prediction and calibration, the processor is updated to the appropriate compute configuration at runtime via DVFS.

[0025] Preferably, in reasoning tasks In the existing layers, there are BoF pooling layers and FC layers. This is the intermediate result of the existing layer i. The BoF pooling layer is used to calculate the number of object classes in the dataset. The features extracted are aggregated and, in BoF pooling, described using a set of feature vectors called the codebook. The weight of each codebook is determined by measuring the codebook and... The codebook weights are generated based on the similarity between the codebooks, and the weights are greater than the similarity between the codebooks. Therefore, the FC layer is further used as a classifier to adjust the BoF pooling result to Thus, the final output of the existing layers is estimated, and the estimation result of the classifier is:

[0026]

[0027] in, The intermediate result of the i-th layer; x is the initial network input; The parameters for the i-th exit layer are the BoF pooling layer and the FC layer. It is the estimation result of the classifier;

[0028] Train the parameters in the existing layers offline, during training Choose the same cross-entropy loss function;

[0029]

[0030] Where p(i) and q(i) are the actual and predicted class distributions for each object, respectively; Cross-entropy;

[0031] When training the existing layers, the training set data {x1, x2, ..., xN} is input into the model, where N is the number of training data points. The parameters of the existing layers are optimized using batch gradient descent. Let r be the number of layers in the network, and let the target vector set be {r1, r2, ..., rN} ∈ For correct results; first, train the model using the following formula without any existing layers:

[0032]

[0033] Where W' represents the updated overall network parameters; η is the learning rate, which is adjusted if the accuracy does not meet the requirements; and j represents the vector position. Represents the gradient of the parameters; Represents the elements of the training data vector; Represents the elements of the target data vector;

[0034] With the original model W fixed, the model is retrained based on the existing layers, optimizing the parameters of the BoF pooling layer and the FC layer. The expression is as follows:

[0035]

[0036] After training the exit layer, calculate the average feature weights. The expression for the parameter used in the exit decision is:

[0037]

[0038] Where k represents the kth classifying target;

[0039] Initial network input During the inference process, at each early exit layer, the weight ratio Divide the maximum feature weight by Multiplying by the user-specified hyperparameter β, the larger the maximum feature weight, the more confident the classification is. If the result is greater than 1, the inference terminates, and the result of this early exit layer is used as the final result, expressed as:

[0040] .

[0041] Preferably, the predictor starts from the first... The prediction begins at the first layer, with the predictor input being the first layer. The execution result vector of a layered neural network and its corresponding exit layer execution result vector ;

[0042] Expanding the dimensions of the above results by vector Adding zeros to both ends of a vector Depend on Dimensional extension Wei, and denoted as ,in For vectors Length is also the number of categories of the reasoning results. The number of feature vectors in the feature pooling group is shown in the following formula:

[0043]

[0044] Indicates the first One element;

[0045] Output vector after zero padding A one-dimensional convolution operation will be performed to obtain a new feature weight vector. The convolution weights are of length . a one-dimensional vector The specific calculation process is shown in the following formula:

[0046]

[0047] The intermediate estimation results are recursively processed. Replace with And repeat the calculation to get The prediction result of the discharge layer at position +2 is denoted as ;

[0048] Place the prediction results of the exit layer into At any subsequent level, the PREDICT function is obtained, described as follows:

[0049]

[0050] Where, ζ∈ ,express The minimum value of the predicted point;

[0051]

[0052] In the formula, for The predicted exit confidence of the +ζ layer, therefore +ζ is the exit point of the prediction function. If ζ cannot be found, a hyperparameter τ is introduced, i.e., ζ∈[1,τ], τ≤ The upper bound of τ indicates that prediction results outside the last layer are prohibited; if there are no integers in [1, τ] that satisfy the condition, then let ζ = τ;

[0053] Convert the existing PREDICT prediction result ζ into an intermediate frequency. For reasoning tasks In the beginning of reasoning Within the predicted time interval, a relatively high-level conservative approach is applied. Complete the model in a timely manner Without exiting, the expression is:

[0054]

[0055] In the formula, This is the highest default frequency for the computing platform, which is available on this computing platform. The reasoning is completed at each level, and is not affected by time. Early exit; frequency Depend on calibration;

[0056] In prediction At that time, based on the predicted exit point, f is adjusted to an intermediate frequency. and run the network until in Exit, given a value with For networks with multiple layers and early prediction ζ, the intermediate frequency calculation formula is as follows:

[0057]

[0058] in, It is the predicted transit time. The lowest frequency at which inference is completed.

[0059] Preferably, for reasoning tasks Cross-inference DVFS strategy in completing including Provide calibration recommendations after all previous tasks, including those mentioned above. This calibration is achieved by a discrete incremental proportional-integral (PI) regulator, expressed as:

[0060]

[0061] in, and These are the proportional coefficient and the integral coefficient, respectively; index i represents the i-th inference task within the current coordination cycle; Inference The relative completion time from the start of the coordination cycle; For reasoning tasks and The time interval between completions;

[0062] Input deviation of PI controller It is based on the total inference workload and the inference execution progress since the start of the self-coordination period. The evaluation of the reasoning progress is expressed as: The first term is the reference speed for inference workload balancing, and the second term is the processing speed to date.

[0063] Preferably, the frequency and corresponding voltage are configured through the DVFS speed controller, and the prediction and calibration are executed based on both small and large time scales. At the start of each inference, the cross-inference calibration is derived. The DVFS controller sets the computation configuration to... Conservatively complete the reasoning without prematurely exiting, at the reasoning level. When performing early exit prediction, based on the prediction derived by the intrinsic inference predictor And the calibration derived from cross-inference The DVFS controller establishes an appropriate frequency configuration. The expression is:

[0064]

[0065] Therefore, the energy consumption is:

[0066]

[0067] in, For prediction layer and the network layers before it Completion time; For a given The corresponding voltage; since the choice of f is based on the remaining workload predicted in inference, in-inference prediction can... Depend on Completed; cross-inference calibration target passed. Complete M sequential inferences.

[0068] The lightweight system for dynamic exit of neural networks for multiple consecutive inferences provided by the present invention includes:

[0069] Module M1: Constructs a neural network-based inference model. For each inference within a preset small time range, it predicts the location where the network exits and predicts the computational configuration accordingly, including frequency and voltage. For multiple inferences within a preset large time range, it calibrates the processor frequency and voltage based on the remaining inference workload and time constraints.

[0070] Module M2: Based on prediction and calibration calculations, a configuration is executed to perform a neural network, thereby enabling dynamic voltage and frequency regulation.

[0071] Preferably, a coordination period, or reasoning task duration, is set during the reasoning process, expressed as:

[0072]

[0073]

[0074] Where λ represents the tightness of the task deadline, 0 < λ ≤ 1, the smaller λ is, the tighter the deadline; Indicates the actual time taken to complete each reasoning step; The coordination period is M; M is the number of reasoning tasks. Indicates the coordination period for the i-th reasoning task;

[0075] The power consumed during the calculation process when performing inference tasks is active power. , for dynamic power Static power Constant power The sum;

[0076] The power consumed by the computing platform when it is idle is called idle power. , is the static power Constant power The sum;

[0077]

[0078]

[0079] Where C is the capacitor of the switching logic gate; V represents the processor's power supply voltage; and f represents the processor's clock frequency. The number of logic gates; Normalized static current for each logic gate;

[0080] The energy consumption for each inference step is:

[0081]

[0082] The network layer located before early exit runs in ( , ),exist After the network exits, the computing platform uses Dynamic Voltage and Frequency Adjustment (DVFS) technology to reduce the voltage and frequency to the lowest possible level. , Therefore, reasoning tasks Energy consumption is expressed as:

[0083]

[0084] Representational reasoning task Energy consumption; This indicates the high power supply voltage of the processor; Indicates a high processor clock frequency; Indicates the processor's low supply voltage;

[0085] In a reasoning task First, predict the network exit position and establish the remaining layers to complete the inference; then, based on the remaining layers and the worst-case inference time, execute the inference. Until the current inference ends, establish (V,f) for prediction; based on The remaining inference workload is used to enable the cross-inference calibrator through a feedback control strategy to calibrate (V,f) based on multi-inference progress, total inference workload, and time constraints; based on prediction and calibration, the processor is updated to the appropriate compute configuration at runtime via DVFS.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] This invention predicts the location where the network will exit during each inference and adjusts the computational configuration (i.e., frequency and voltage) accordingly within a small time range. Simultaneously, based on multiple inferences over a large time range, combined with the remaining inference workload and time constraints, it provides suggestions for the computational configuration (i.e., frequency and voltage). The neural network is executed according to the predicted and calibrated configuration, thereby achieving dynamic voltage and frequency regulation. Compared to classic deep learning networks, this invention can achieve up to 63.8% energy savings while ensuring that multiple neural network inferences are completed within a specified time. Early exit allows for premature termination of inference and obtaining accurate results, reducing computational and energy costs. Attached Figure Description

[0088] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0089] Figure 1 This is a diagram of the neural network structure based on early exit in this invention. Detailed Implementation

[0090] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0091] Example 1:

[0092] like Figure 1 For neural network applications that complete multiple neural network inferences within a certain period of time, this invention provides a lightweight method for dynamic exit of neural networks with multiple consecutive inferences. By combining the inference state with dynamic voltage and frequency adjustment of the processor during runtime, the computational power and power consumption of the neural network are reduced. Specifically, it includes the following steps:

[0093] Step 1: In each inference over a short time period, predict where the network will exit and predict the computation configuration accordingly within the short time range, the computation configuration including frequency and voltage; for multiple inferences over a long time range, provide processor frequency and voltage calibration through remaining inference workload and deadline constraints;

[0094] Step 2: Configure the neural network based on the prediction and calibration calculations to achieve dynamic voltage and frequency regulation.

[0095] A more detailed explanation will follow.

[0096] The expression for the coordination period of reasoning, i.e. the duration of the reasoning task, is as follows:

[0097]

[0098]

[0099] λ represents the tightness of the task deadline, 0 < λ ≤ 1, the smaller λ is, the tighter the deadline; Indicates the actual time taken to complete each reasoning step; The coordination period is M; M is the number of reasoning tasks. This represents the coordination period for the i-th reasoning task.

[0100] The power consumed during the calculation process when performing inference tasks is active power. , for dynamic power Static power Constant power The sum;

[0101] The power consumed by the computing platform when it is idle is called idle power. , is the static power Constant power The sum;

[0102]

[0103]

[0104] Where C is the capacitor of the switching logic gate; V represents the processor's power supply voltage; and f represents the processor's clock frequency. The number of logic gates; Normalized static current for each logic gate;

[0105] The energy consumption for each inference step is:

[0106]

[0107] Because neural networks exhibit early deployment, the network layers located before early exit operate in (…). , ), After the point network exits, the computing platform reduces the voltage and frequency to the lowest level via DVFS. , ),until Therefore, reasoning task Energy consumption is expressed as:

[0108]

[0109] Indicates energy consumption; This indicates the high power supply voltage of the processor; Indicates a high processor clock frequency; Indicates the processor's low supply voltage;

[0110] Establish appropriate (V,f) to adjust the remaining workload and time constraints, based on the predicted network exit location and the deadline. The remaining inferences completed previously are adjusted at runtime (V,f).

[0111] In a reasoning task In the process, the exit position of the inference network is predicted first, and the remaining number of layers is established to complete the inference; then, based on the remaining layers and the worst-case execution time... Until the current inference ends, establish appropriate (V,f) forecasts to reduce energy costs on small timescales; considering The remaining inference load is calibrated by the cross-inference calibrator, enabled by a feedback control strategy, based on multi-inference progress, total inference workload, and time constraints (V,f) to balance workload, energy, and time costs over a large timescale. Based on prediction and calibration, the DVFS regulator will update the processor to the appropriate compute configuration (i.e., frequency and voltage) at runtime to save energy while meeting deadlines.

[0112] In reasoning tasks In this design, the existing layers are based on existing work, and all existing layers are further modified to share the same topology. It includes BoF (Bag-of-Features) pooling layers and FC (fully-connected) layers. This is an intermediate result of the i-th layer. The number of object classes in the dataset. The BoF pooling layer acts as a buffer from... The features extracted are aggregated. In BoF pooling, a set of feature vectors called the codebook is used to describe... The weight of each codebook is determined by measuring the codebook and... It is generated based on the similarity between them. Since the codebook weights are typically larger than... Therefore, the FC layer is further used as a classifier to adjust the BoF pooling result to This allows us to estimate the final output of the layered structure.

[0113] The functions of the existing layers are denoted as:

[0114]

[0115] in, The intermediate result of the i-th layer is given by x, and the initial network input is given by x. These are the parameters for the i-th exit layer (BoF and FC); It is the estimation result of the classifier.

[0116] The parameters in the existing layers need to be trained offline, as the existing layers function by estimating the final network output. Choose the same cross-entropy loss function.

[0117]

[0118] Where p(i) and q(i) are the actual and predicted class distributions for each object, respectively; Cross-entropy;

[0119] When training the existing layers, the training set data {x1, x2, ..., xN} is input into the model, where N is the number of training data points. The parameters of the existing layers are optimized using batch gradient descent. Let r be the number of layers in the network, and let the target vector set be {r1, r2, ..., rN} ∈ For correct results; first, train the model using the following formula without any existing layers:

[0120]

[0121] Where W' represents the updated overall network parameters, η is the learning rate (η can be adjusted if the accuracy is insufficient), and j represents the vector position. Represents the gradient of the parameters; Represents the elements of the training data vector; Represents the elements of the target data vector;

[0122] Next, the original model W is fixed, and the model is retrained based on the existing layers to optimize the parameters of the BoF pooling layer and the FC layer. The expression is:

[0123]

[0124] After training the exit layer, calculate the average feature weights. The expression for the parameter used in the exit decision is:

[0125]

[0126] k represents the k-th classifying target.

[0127] Initial network input During the inference process, at each early exit layer, the weight ratio Divide the maximum feature weight by Multiply by the user-specified hyperparameter β. The larger the weight of the largest feature, the more confident the classification is. If the result is greater than 1, the inference terminates, and the result of this early exit layer is used as the final result.

[0128]

[0129] Suppose the predictor starts from the first... The prediction begins at the first layer, with the predictor input being the first layer. The execution result vector of a layered neural network and its corresponding exit layer execution result vector .

[0130] The zero-padding module expands the dimensions of the above results by adding dimensions to the vector. Adding zeros to both ends of a vector Depend on Dimensional extension Wei, and denoted as ,in For vectors Length is also the number of categories of the reasoning results. This represents the number of feature vectors in the feature pooling group. The specific calculation process is shown in the following formula:

[0131]

[0132] Indicates the first Each element.

[0133] Output vector after zero-padding module A one-dimensional convolution operation will be performed to obtain a new feature weight vector. The convolution weights are of length . a one-dimensional vector The specific calculation process is shown in the following formula:

[0134]

[0135] The intermediate estimation results are recursively processed. Replace with And repeat the calculation to get The prediction result of the discharge layer at position +2 is denoted as Following the steps above, the calculation can be performed. The prediction results for any existing layer thereafter.

[0136] Place the prediction results of the exit layer into At any subsequent level, we will obtain the PREDICT function, described as follows:

[0137]

[0138] Where, ζ∈ ,express The minimum value of the predicted point;

[0139]

[0140] In the formula, for +ζ layer predicted exit confidence. Therefore, +ζ is the exit point of the prediction function. If ζ cannot be found, a hyperparameter τ is further introduced, i.e., ζ∈[1,τ], τ≤ The upper bound of τ indicates that predictions outside the last layer are prohibited. If no integer in [1, τ] satisfies the condition, then let ζ = τ. The time complexity of the prediction is O(τ). Meanwhile, hyperparameters are introduced into the prediction. β and τ can be tuned by advanced users to balance prediction accuracy and computational cost for different application scenarios.

[0141] Frequency Prediction: Convert the existing PREDICT prediction result ζ into an appropriate "intermediate" frequency. For reasoning tasks In the beginning of reasoning Within the predicted time interval, a relatively "high-level" conservative approach is applied. Complete the model in a timely manner Without exiting, the expression is:

[0142]

[0143] In the formula, This is the highest default frequency for the computing platform, which can be used... The layer completes the reasoning and will not be affected by time. Exit early. Frequency Depend on calibration.

[0144] In prediction At that time, based on the predicted exit point, f is adjusted to a suitable "middle layer". and run the network until in Exit. Given a... For networks with more layers and earlier prediction ζ, the computation frequency is reduced to:

[0145]

[0146] in, Through time It can make predictions based on the lowest frequency of inference.

[0147] Since our goal is to perform reasoning tasks in a spontaneous manner, for reasoning tasks... Cross-inference DVFS strategy in completing including Provide calibration recommendations after all previous tasks, including those mentioned above. Considering a larger time scale The workload and time constraints are as follows. This calibration is implemented using a discrete incremental proportional-integral (PI) regulator, expressed as:

[0148]

[0149] in, and These are the proportional coefficient and the integral coefficient, respectively. Index i represents the i-th inference task within the current coordination cycle. Inference The relative completion time from the start of the coordination cycle. For reasoning tasks and The time interval for completion.

[0150] Input deviation of PI controller It is based on the total inference workload and the inference execution progress since the start of the self-coordination period. The evaluation of the reasoning progress is expressed as: The first term is the reference speed for inference workload balancing, and the second term is the processing speed to date.

[0151] In practical applications, a smaller time period is used. ,0< ≤1. and The intervals are a margin reserved through feedback control to eliminate the effects of execution time overshoot. A proper selection can strike a balance between energy conservation and meeting deadlines. The output of the cross-inference corrector is the DVFS speed controller configuration computing platform. The next reasoning Frequency calibration.

[0152] Finally, the DVFS speed controller is configured with frequency and corresponding voltage, and the network is executed based on predictions and calibrations at both small and large time scales, further saving energy and meeting the network inference deadline. The DVFS controller scales with operations (V,f) at a finer granularity than inference. At the start of each inference iteration, a cross-inference calibration is derived. The DVFS controller sets the compute configuration to... Conservatively complete the reasoning without prematurely exiting. At the reasoning level... When performing early exit prediction, the DVFS controller establishes an appropriate frequency configuration. Derived from the intrinsic inference predictor And the calibration derived from cross-inference The expression is:

[0153]

[0154] Therefore, the resulting energy consumption will be:

[0155]

[0156] in, For prediction layer and the network layers before it Completion time. For a given The corresponding voltage. Since the choice of f is based on the remaining workload predicted during inference, in-inference prediction can... Depend on Completed. Cross-inference calibration target passed. Complete M sequential inferences. Since V and f are linear metrics of computational performance (inversely proportional to inference), and for... and The cubic and linear scales of the dual-time-scale power management effectively reduce E.

[0157] Processor voltage and frequency adjustments can be implemented at the application level via system calls. Based on our observations, the transient time for each (V,f) change is approximately 1ms to 3ms. The transient process of changing processor (V,f) was incorporated into our actual system evaluation.

[0158] For each inference at runtime, the computational complexity of the inference exit prediction is O(n). Both cross-inference calibration and DVFS speed controller are .

[0159] We evaluate the inference accuracy and temporal performance of this invention using VGG-19 and ResNets-18 as backbone models on commonly used datasets such as CIFAR-10, CIFAR-100, SVHN, and CINIC. The evaluation was performed on an NVIDIA Jetson TX2 GPU. =1.30050 GHz, =0.11475 GHz. Serial and I / O operations are performed on the CPU, while a large number of parallel and computationally intensive segments are offloaded to the GPU, automatically managed by the PyTorch library. Energy costs were assessed using a Tektronix MDO32 oscilloscope and a TCP2020 current probe at a sampling frequency of 50 kHz. Predicted and calibration parameters for different benchmarks are listed in Table 1.

[0160] Table 1

[0161]

[0162] During inference, the frequency is calculated based on predictions of early exit points. The accuracy of these predictions directly determines the task's time and energy consumption. Therefore, we tested different... The dataset and model prediction accuracy were evaluated. Based on the prediction and inference accuracy results, the remaining parts of Vgg-19 and ResNet-18 were set separately. =6 and 7.

[0163] The inference accuracy was then evaluated, and the results are shown in Table 2. This invention achieved the same accuracy as other early exit methods on both models, but 1%-3% lower than classic CNNs. This is because in early exit prediction, even if the prediction is incorrect, the network continues with the next prediction instead of forcing an inference exit, thus avoiding additional loss of inference accuracy. The remaining power management settings have no impact on inference accuracy because the existing layer predictions of ζ are determined first to achieve high inference accuracy, and power management is built upon this foundation.

[0164] Table 2

[0165]

[0166] We evaluated the timing performance, and this invention demonstrates a more focused and shorter execution time than other methods due to efficient dual-timescale DVFS management. α strikes a balance between energy conservation and meeting deadlines. α=0.75 results in a slightly longer total execution time than α=0.5, while saving more energy.

[0167] Finally, we evaluated the deadlines under different λ values. Satisfaction rate. As a stress test, we compared timing performance under tighter deadlines, i.e. 0 < λ ≤ 1. When λ = 1, all methods can meet the deadline.

[0168] To measure power consumption during operation, the Jetson TX2 board was directly connected to a Keysight E36231A power supply at 19 V. A current probe and oscilloscope monitored the current fed to the board. Table 3 summarizes the average energy consumption for each task. Energy measurements on a real-world platform show that this invention achieves 63.8% energy savings compared to classic deep learning networks and 21.5% energy savings compared to early exit under state-of-the-art exit strategies. In most cases, α=0.75 consumes less energy than α=0.5.

[0169] Table 3

[0170]

[0171] The primary source of overhead is early exit from PREDICT. Therefore, we evaluated the time cost of 1750 early exits from PREDICT for each benchmark. The maximum time cost for all benchmarks was below 300 μs. The ResNets-18 box is wider than the VGG-19 box because ResNets-18 is expected to exit with a larger ζ. The time cost demonstrates the applicability and scalability of the online power management of this invention.

[0172] Example 2:

[0173] The present invention also provides a lightweight system for dynamic exit of neural networks for multiple consecutive inferences. The lightweight system for dynamic exit of neural networks for multiple consecutive inferences can be implemented by executing the process steps of the method for dynamic exit of neural networks for multiple consecutive inferences. That is, those skilled in the art can understand the method for dynamic exit of neural networks for multiple consecutive inferences as a preferred embodiment of the lightweight system for dynamic exit of neural networks for multiple consecutive inferences.

[0174] The lightweight system for dynamic exit of neural networks for multiple consecutive inferences provided by the present invention includes: module M1: constructing a neural network-based inference model, predicting the exit position of the network for each inference within a preset small time range, and predicting the computational configuration accordingly, the computational configuration including frequency and voltage; for multiple inferences within a preset large time range, calibrating the processor frequency and voltage based on the remaining inference workload and time constraints; module M2: executing the neural network according to the predicted and calibrated computational configuration, thereby realizing dynamic voltage and frequency adjustment.

[0175] In the reasoning process, a coordination period, or the time limit for the reasoning task, is set, expressed as:

[0176]

[0177]

[0178] Where λ represents the tightness of the task deadline, 0 < λ ≤ 1, the smaller λ is, the tighter the deadline; Indicates the actual time taken to complete each reasoning step; The coordination period is M; M is the number of reasoning tasks. Indicates the coordination period for the i-th reasoning task;

[0179] The power consumed during the calculation process when performing inference tasks is active power. , for dynamic power Static power Constant power The sum;

[0180] The power consumed by the computing platform when it is idle is called idle power. , is the static power Constant power The sum;

[0181]

[0182]

[0183] Where C is the capacitor of the switching logic gate; V represents the processor's power supply voltage; and f represents the processor's clock frequency. The number of logic gates; Normalized static current for each logic gate;

[0184] The energy consumption for each inference step is:

[0185]

[0186] The network layer located before early exit runs in ( , ),exist After the network exits, the computing platform uses Dynamic Voltage and Frequency Adjustment (DVFS) technology to reduce the voltage and frequency to the lowest possible level. , Therefore, reasoning tasks Energy consumption is expressed as:

[0187]

[0188] Representational reasoning task Energy consumption; This indicates the high power supply voltage of the processor; Indicates a high processor clock frequency; Indicates the processor's low supply voltage;

[0189] In a reasoning task First, predict the network exit position and establish the remaining layers to complete the inference; then, based on the remaining layers and the worst-case inference time, execute the inference. Until the current inference ends, establish (V,f) for prediction; based on The remaining inference workload is used to enable the cross-inference calibrator through a feedback control strategy to calibrate (V,f) based on multi-inference progress, total inference workload, and time constraints; based on prediction and calibration, the processor is updated to the appropriate compute configuration at runtime via DVFS.

[0190] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0191] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for dynamically exiting lightweight of a neural network for multiple continuous inferences, characterized in that, Comprising: Step 1: constructing a neural network-based inference model, for each inference within a preset small time range, predicting the position of network exit, and accordingly predicting the computing configuration, the computing configuration including frequency and voltage; for multiple inferences within a preset large time range, through the remaining inference workload and time constraints, processor frequency and voltage calibration is performed; Step 2: performing the neural network according to the predicted and calibrated computing configuration, thereby realizing dynamic voltage and frequency adjustment; In the inference task , the existing layers include a BoF pooling layer and an FC layer, set as the intermediate result of the i-th layer of the existing layers, as the number of object classes in the data set, the BoF pooling layer is aggregated as a feature extracted from , in the BoF pooling, a set of feature vectors called codebooks is used to describe , the weight of each codebook is generated by measuring the similarity between the codebook and , the size of the codebook weight is greater than , so the FC layer further adjusts the result of the BoF pooling as , so as to estimate the final output of the existing layer, and the estimation result of the classifier is: wherein, is the intermediate result for the i-th layer; x is the initial network input; is the parameter for the i-th exit layer, the exit layer being the BoF pooling layer and the FC layer; is the estimation result of the classifier; The parameters in the existing layers are trained offline, and the same cross-entropy loss function is selected during the training ​ where p(i) and q(i) are the actual and predicted class distribution for each object, respectively; is the cross-entropy; In training the existing layer, the training set data {x1, x2, …, xN} is input into the model, N is the number of training data, and the existing layer parameters are optimized by batch gradient descent, and let is the number of layers of the network, and the target vector set is {r1, r2, …, rN} ∈ is the correct result; first, without any existing layer, the model is trained by the following formula: Wherein, W' is the updated overall network parameter; η is the learning rate, if the accuracy cannot meet the requirements, adjust η; j represents the vector position; represents the parameter gradient; represents the training data vector element; represents the target data vector element; Fixing the original model W, retraining the model on the basis of the existing layers, optimizing the parameters of the BoF pooling layer and the FC layer, the expression being: After training the exit layer, the average feature weights are computed As a parameter for the exit decision, the expression is Wherein, k represents the kth overall classification target; In the inference process of the initial network input At each early exit layer, the weight ratio is the maximum feature weight divided by multiplied by a user-specified hyperparameter β, the greater the maximum feature weight, the more confident the classification is considered, if is greater than 1, the inference terminates, and the result at this early exit layer is applied as the final result, expressed as: ； Let the predictor start predicting from the first layer of the neural network, with the predictor input being the execution result vector of the first layer of the neural network. and its corresponding exit layer execution result vector .

2. The method of claim 1, wherein, In the inference process, the coordination period of inference, that is, the inference task deadline, is set, the expression being: Wherein, λ represents the tightness of the deadline of the task, 0<λ≤1, the smaller λ is, the tighter the deadline is; denotes the actual completion time of each inference; is the coordination period; M is the number of inference tasks; denotes the coordination deadline of the i-th inference task.

3. The method of claim 2, wherein, The power consumed in the computation when performing the inference task is active power , the sum of dynamic power , static power , constant power . The power consumed by the computing platform when idle is the idle power , the sum of the static power , the constant power , and the dynamic power where C is the capacitance of the switching logic gate; V represents the supply voltage of the processor; f represents the processor clock frequency; is the number of logic gates; is the normalized static current of each logic gate; The energy consumption for each inference is: The network layer located before the early exit runs at , ), after the network exits at point, the computing platform reduces the voltage and frequency to the lowest level by dynamic voltage and frequency scaling technology DVFS , , therefore, the energy consumption of the inference task is represented as: representing the inference task energy consumption; representing a high supply voltage for the processor; representing a high clock frequency for the processor; representing a low supply voltage for the processor.

4. The method of claim 3, wherein, In one inference task , the network exit position is predicted first, and the remaining number of layers is established to complete the inference; then the remaining number of layers and the worst inference execution time are executed until the current inference ends, and (V, f) is established for prediction; according to the remaining inference amount, the cross-inference calibrator is enabled through the feedback control strategy, and (V, f) is calibrated according to the multi-inference progress, the total inference workload and the time constraint; according to the prediction and calibration, the processor is updated to the appropriate computing configuration at runtime through DVFS.

5. The method of claim 4, wherein, To extend the dimension of the above results, by adding 0 to the vector at both ends, the vector is obtained. is extended to dimension, and is denoted as , where is the vector length, and also the number of classification results of reasoning, is the number of feature vectors in the feature pooling group, and the specific calculation process is shown in the following formula: represents the first element; output vector after zero padding a one-dimensional convolution operation is performed and a new feature weight vector is obtained where the convolution weight is a one-dimensional vector of length The specific calculation process is shown in the following formula:​ By recursion, the intermediate estimates are replaced by and the calculation is repeated to obtain the prediction of the output layer at position +2, denoted by ; The prediction result of the exit layer is placed in any subsequent layer, resulting in the PREDICT function, described as follows: where ζ∈ , denotes the minimum value of the predicted out-point after In the formula, For The predicted exit confidence of the +ζ layer, therefore, +ζ is the exit point predicted by the prediction function, and if ζ cannot be found, a hyperparameter τ is introduced, that is, ζ∈[1,τ], τ≤ The upper bound of τ indicates that the prediction result outside the last layer is prohibited; if there is no integer in [1,τ] that satisfies the condition, set ζ=τ; transforming existing prediction results ζ of PREDICT into mid-level frequencies for the inference task at the start of the inference to within the time interval of the prediction, conservatively applying a relatively high level of completing the model's without exiting, expressed as: In the formula, The highest frequency that the computing platform can use The layer completes the inference, and does not cause time Exit early; frequency By Calibration; At the time of prediction f is adjusted to the intermediate frequency according to the predicted exit point and the network is run until exit, given a network with number of layers and early prediction ζ, the intermediate frequency is calculated as: wherein, is the predicted transit time the lowest frequency at which the inference is completed.

6. The method of claim 5, wherein, For reasoning tasks Cross-inference DVFS strategy in completing including Provide calibration recommendations after all previous tasks, including those mentioned above. This calibration is achieved by a discrete incremental proportional-integral (PI) regulator, expressed as: wherein, and are a proportional coefficient and an integral coefficient, respectively; the index i represents the i-th inference task in the current coordination period; denotes an inference relative completion time from the start of the coordination period; is the inference task and the time interval of completion; Input deviation of pi regulator is evaluated according to the total reasoning workload and the reasoning execution progress since the self-coordination period starts, expressed as: where the first term is the reference speed of reasoning workload balancing, and the second term is the processing speed so far.​ 7. The method of claim 6, wherein, The frequency and corresponding voltage are configured by the DVFS governor, and the network is executed according to the prediction and calibration of the small time scale and the large time scale respectively, at the beginning of each inference, the cross-inference calibration based on The DVFS governor sets the computation configuration to The conservative inference is completed without early exit, and the early exit prediction is performed at the inference layer According to the derived by the inner inference predictor, and the derived by the cross-inference calibration, the DVFS governor establishes the appropriate frequency configuration The expression is: Therefore, the energy consumption generated is: wherein, is a prediction layer and the network layer preceding it is completed; is given the corresponding voltage; since the selection of f is based on the predicted remaining workload in the inference, the intra-inference prediction can enable by is completed; the cross-inference calibration target is completed by M sequential inferences.

8. A lightweight system for dynamic exit of neural networks with multiple consecutive inferences, characterized in that, Comprising: Module M1: constructing a neural network-based inference model, for each inference within a preset small time range, predicting the position of network exit, and accordingly predicting the computing configuration, the computing configuration including frequency and voltage; Module M2: for multiple inferences within a preset large time range, through the remaining inference workload and time constraints, processor frequency and voltage calibration is performed; Module M3: performing the neural network according to the predicted and calibrated computing configuration, thereby realizing dynamic voltage and frequency adjustment; In the inference task The existing layers include a BoF pooling layer and an FC layer, and Let the intermediate result of the i-th existing layer be Let the number of object classes in the data set be C, the BoF pooling layer aggregates features extracted from In BoF pooling, a set of feature vectors called codebooks is used to describe The weight of each codebook is generated by measuring the similarity between the codebook and The size of the codebook weight is greater than Therefore, the FC layer further adjusts the result of the BoF pooling to Thus, the final output of the existing layers is estimated, and the estimation result of the classifier is: wherein, is the intermediate result for the i-th layer; x is the initial network input; is the parameter for the i-th exit layer, the exit layer being the BoF pooling layer and the FC layer; is the estimation result of the classifier; The parameters in the existing layers are trained offline, and the same cross-entropy loss function is selected during the training ​ where p(i) and q(i) are the actual and predicted class distribution for each object, respectively; is the cross-entropy; In training the existing layer, the training set data {x1, x2, …, xN} is input into the model, N is the number of training data, and the existing layer parameters are optimized by batch gradient descent, and let be the number of layers of the network, and the target vector set is {r1, r2, …, rN} ∈ be the correct result; first, without any existing layer, the model is trained by the following formula: Wherein, W' is the updated overall network parameter; η is the learning rate, if the accuracy cannot meet the requirements, adjust η; j represents the vector position; represents the parameter gradient; represents the training data vector element; represents the target data vector element; Fixing the original model W, retraining the model on the basis of the existing layers, optimizing the parameters of the BoF pooling layer and the FC layer, the expression being: After training the exit layer, the average feature weights are computed As a parameter for the exit decision, the expression is: Wherein, k represents the kth overall classification target; In the inference process of the initial network input At each early exit layer, the weight ratio is the maximum feature weight divided by multiplied by a user-specified hyperparameter β, the greater the maximum feature weight, the more confident the classification is considered, if is greater than 1, the inference terminates, and the result at this early exit layer is applied as the final result, expressed as: ； Let the predictor start predicting from the first layer of the neural network, with the predictor input being the execution result vector of the first layer of the neural network and its corresponding exit layer execution result vector .

9. The neural network dynamic pruning system for multiple consecutive inferences of claim 8, wherein, In the inference process, the coordination period of inference, that is, the inference task deadline, is set, the expression being: Wherein, λ represents the tightness of the deadline of the task, 0<λ≤1, the smaller λ is, the tighter the deadline is; denotes the actual completion time of each inference; is the coordination period; M is the number of inference tasks; denotes the coordination deadline of the i-th inference task; The power consumed in the computation when performing the inference task is active power , the sum of dynamic power , static power , and constant power . The power consumed by the computing platform when idle is the idle power , the sum of the static power , the constant power ​ where C is the capacitance of the switching logic gate; V represents the supply voltage of the processor; f represents the processor clock frequency; is the number of logic gates; is the normalized static current of each logic gate; The energy consumption for each inference is: The network layer located before the early exit runs at , ), after the network exits at point, the computing platform reduces the voltage and frequency to the lowest level by dynamic voltage and frequency scaling technology DVFS , , therefore, the energy consumption of the inference task is represented as: representing the inference task energy consumption; representing a high supply voltage for the processor; representing a high clock frequency for the processor; representing a low supply voltage for the processor; In one inference task , the network exit position is predicted first, and the remaining number of layers is established to complete the inference; then the remaining number of layers and the worst inference execution time are established until the current inference ends, and (V, f) is established for prediction; according to the remaining inference amount, the cross-inference calibrator is enabled through the feedback control strategy, and (V, f) is calibrated according to the multi-inference progress, the total inference workload and the time constraint; according to the prediction and calibration, the processor is updated to the appropriate computing configuration at runtime through DVFS.

Images