A neural network dynamic early retirement accelerated inference method for edge devices

Abstract

The present invention discloses a method for accelerating inference of dynamic early exit of neural networks for edge devices, which belongs to the field of edge computing. A heterogeneous exit header structure is used to improve the feature extraction capability of the model, achieving higher model accuracy, faster inference speed and lower model operation complexity; a two-stage training strategy is used to train a dynamic early exit model based on a heterogeneous exit header, which gives full play to the potential of the exit header, further improves model accuracy and reduces model operation complexity; a dynamic inference method is used to dynamically execute part of the network to save computing resources; at the same time, the model output path is dynamically selected according to the complexity of the input data to adapt to different scenarios and achieve a trade-off between accuracy and efficiency. The present invention is applicable to the field of edge computing, improves model accuracy, speeds up inference speed and reduces model operation complexity, and provides support for the deployment of intelligent applications based on complex models on edge devices.

Description

Technical Field

[0001] The present invention relates to a neural network inference acceleration method, and in particular to a neural network dynamic early-retirement acceleration inference method for edge devices, belonging to the field of edge computing. Background Art

[0002] With the rapid development of theoretical technologies such as big data and high-performance computing, deep neural network (DNN) models have achieved superhuman performance in tasks such as computer vision and natural language processing. The Vision Transformer (ViT) model, a foundational model in computer vision, boasts excellent performance, making its deployment on edge devices such as smartphones and watches crucial for intelligent applications. However, the ViT model is complex and computationally intensive, requiring extremely high storage requirements and computational complexity. Edge devices have far fewer computing resources than desktop computers and are constrained by limited battery capacity, making it difficult to directly deploy the ViT model on edge devices for intelligent applications. For example, the ViT-L model, a classic ViT model, requires 190.7 GB floating-point operations per second for a single inference. However, the computing power of the Raspberry Pi 4B, an edge device, is only 13.5 GB floating-point operations per second.

[0003] To solve the above problems, some researchers have proposed inserting an internal classifier (also called an exit head) into the middle layer of the original DNN model. During the inference process, once the prediction result of the exit head meets the exit criteria, the inference is terminated in advance and the calculation of the subsequent neural network layer is no longer performed. This method can effectively reduce the inference delay. This method is called dynamic early exit. This method can dynamically execute part of the neural network model, save computing resources, and has the advantages of high efficiency and flexibility. At the same time, it can dynamically select the model output position according to the complexity of the input data to adapt to different scenarios. However, the current research on dynamic early exit methods in terms of exit head structure design and training strategies is still insufficient. There are two main problems: 1. The exit head design of most methods is simple and rough, and the balance between inference accuracy and inference delay is not achieved. 2. The training strategy of the current method is not designed for the characteristics of the exit head, and the maximum potential of the exit head cannot be brought into play, which limits the improvement of model accuracy. Summary of the Invention

[0004] In view of the current dynamic early exit method, which fails to balance the inference accuracy and inference delay, and the problem that the training strategy limits the improvement of model accuracy, the main purpose of this invention is to propose a dynamic early exit accelerated inference method for neural networks for edge devices. By constructing a dynamic early exit model based on heterogeneous exit heads and combining it with a two-stage training strategy, the inference speed is improved while ensuring a slight loss of accuracy.

[0005] The main purpose of the present invention is achieved through the following technical solutions:

[0006] This paper discloses a method for accelerating inference by dynamic early exit of neural networks for edge devices. First, two heterogeneous exit head structures are constructed: a local perception head based on convolution and a global aggregation head based on a self-attention mechanism. Then, a dynamic early exit model based on the heterogeneous exit heads is established on the ViT backbone network. The dynamic early exit model is then trained using a two-stage training strategy. Finally, the trained model is deployed on edge devices for dynamic inference, accelerating inference on edge devices and reducing inference latency. This method improves inference speed while ensuring minimal accuracy loss.

[0007] The present invention discloses a method for accelerating inference of a neural network using dynamic early retirement for edge devices, which specifically includes the following steps:

[0008] Step 1: Construct two heterogeneous exit heads: a local perception head based on convolution and a global aggregation head based on self-attention mechanism;

[0009] The convolution-based local perception head uses 1×1 convolution to expand the dimension of the input feature. The expanded feature is passed to the position weighted depth convolution, whose convolution kernel size is k×k. The position weighted depth convolution PDConv is:

[0010]

[0011] Among them, PDConv(·) represents the output of position weighted depth convolution; X represents the input feature; DWConv k×k Represents a depthwise convolution with a kernel size of k×k; f(·) is a decreasing linear mapping function used to determine the kernel size of PDConv and reduce the additional computational overhead caused by the introduction of exit heads, that is, k=f(m), m≤M / 2, where M represents the number of exit heads. When k=0, the expanded features will bypass PDConv and enter the subsequent part directly; otherwise, the expanded features of the mth exit position are passed to the k×k depthwise convolution. After PDConv, the expanded features are remapped to the original dimension using 1×1 convolution and passed to the average pooling layer. The output of the local perception head LPH is:

[0012]

[0013]

[0014] Where LPH(·) represents the output of the local perception head; For a given input feature, N represents the number of blocks, D represents the dimension of the hidden layer; X CLS Represents classification tokens; Conv 1×1(·) represents a 1×1 convolution, and Pool(·) represents average pooling. After each convolution, Gaussian Error Linear Unit and batch normalization are used. Finally, the output of the local perception head is passed into a linear classifier to obtain the classification result.

[0015] The global aggregation head based on the self-attention mechanism consists of self-attention modules. The global aggregation head adopts a position-weighted feature aggregation module, namely PFC, to further extract key features. In the PFC module, the input feature is reshaped into and downsampled with a window of size s×s. The downsampled feature is restored to the original dimensional format N'×D, where To avoid introducing additional learnable parameters, average pooling with a stride of s is used to achieve downsampling with a window size of s×s. The window size of PFC depends on the exit position m. Larger window sizes are used at deeper positions of the exit points, thus significantly reducing the additional computational amount. An increasing linear mapping function g(·) is used to determine the window size of PFC, that is, s = g(m), M / 2 < m ≤ M. The position-weighted feature aggregation module PFC is expressed as:

[0016] PFC(X feature ,m) = Pool g(m) (X feature ),

[0017] where PFC(·) represents the output of the position-weighted feature aggregation module, X feature is the given input feature, and Pool g(m) represents average pooling with a stride of g(m). The output of the global aggregation head GAH is:

[0018]

[0019] where GAH(·) represents the output of the global aggregation head, MHSA(·) represents multi-head self-attention, Pool(·) represents average pooling, and softmax(·) represents the normalized exponential function; d is the vector dimension. In MHSA(X), the input X is linearly transformed into query vector, key vector, and value vector in the self-attention mechanism through learnable transformation matrices W Q , W K and W V .

[0020] Step 2: Establish a dynamic early-exit model based on heterogeneous exit heads on the backbone network;

[0021] Using the ViT model as the backbone network, M exit heads are inserted between ViT's encoding layers to form a dynamic early-exit network. M / 2 convolution-based local perception heads are placed at exit points shallow in the network, and M / 2 global aggregation heads based on the self-attention mechanism are placed at exit points deep in the network. The positions of the exit heads are determined based on an approximately equidistant distribution, meaning that the amount of neural network computation required between adjacent exit points is essentially the same. The output of the encoding layer is fed into the corresponding exit head and classifier, and the resulting classification result is subjected to prescribed computations to obtain a prediction confidence. If the confidence level is less than or equal to a pre-set threshold, computations in subsequent neural network layers continue. If the confidence level is greater than the threshold, inference ends prematurely, discontinuing computations in subsequent neural network layers, thereby accelerating inference.

[0022] The final prediction value of the inference phase of a dynamic early-exit network consisting of M exit heads and 1 final classifier is:

[0023]

[0024] in, represents the probability that the i-th exit head predicts the input category to be c, τ i Represents the exit threshold of the i-th exit head. When the above formula is satisfied or the reasoning is executed to the final classifier, the final prediction value is obtained.

[0025] Step 3: Use a two-stage training strategy to train a dynamic early exit model based on heterogeneous exit heads;

[0026] In the first stage, the alternating weighted training method is used to rapidly improve model performance.

[0027] The backbone network has L layers, which have M exit heads and 1 final classifier. From shallow to deep, the exit point of the i-th classifier is m i , where m≤L,i≤M+1, the final exit point of the classifier is m M+1 = L. The alternating weighting method alternates between odd and even numbers of training iterations using two objective functions:

[0028]

[0029] Among them, θ represents the model parameters of the backbone network, w i Indicates the model parameters of the i-th exit head. In the image classification task, the loss function is the cross entropy function.

[0030] In the second stage, the self-distillation training method freezes the parameters of the backbone network and the final classifier, and only updates the parameters of the exit header. This ensures that the model accuracy is not reduced while further improving the model efficiency. The overall loss function includes heterogeneous distillation loss, homogeneous distillation loss, and prediction loss function.

[0031] The heterogeneous distillation loss function is used to promote learning between heterogeneous exit heads. In order to reduce the conflict between multiple losses and the model performance is generally better as the network deepens, only the features of the last layer are used as reference features for the first and last exit heads of the local perception head and the global aggregation head. Since the output of the exit head and the output of the final layer have different dimensions, an alignment module is used to match the dimensions. This module includes depthwise convolution, GELU, and BN activation functions. The feature map of the last layer The dimension of is first reconstructed as The reconstructed feature map is reduced in dimension by the corresponding module and then restored to its original size N'×D. The heterogeneous distillation loss function is expressed as:

[0032]

[0033] in, is the Kullback-Leibler divergence function, and Align(·) represents the alignment module.

[0034] The isomorphic distillation loss function is used to further improve the performance between exit heads with similar structures. In each type of exit head, the last exit head is used as a teacher model to help the previous isomorphic exit head learn, that is, among all local perception heads, the features of the last local perception head are used as reference features for the previous local perception heads. Given the feature maps from the first to the mth exit head The isomorphic distillation loss function between local perception heads is expressed as:

[0035]

[0036] in, is the mean square error function. Since the feature maps of the global aggregation head have different shapes, a dot product operation is performed on the feature maps to match the dimensions. Given the feature map of the global aggregation head at the mth exit point By calculation Convert its shape to D × D. The loss function of isomorphic distillation between global aggregation heads is expressed as:

[0037]

[0038] The total loss function of isomorphic distillation is expressed as:

[0039]

[0040] The prediction distillation loss function can further ensure that the model performance is not degraded. The final classifier is used as the reference label of the M / 2 and M exit points where the last local perception head and global aggregation head are located. The label sample of an input sample is y, and the predicted probability values ​​of the M / 2 and M exit points are respectively and The loss function for prediction distillation is expressed as:

[0041]

[0042] in, is the loss function of general knowledge distillation, Represents the predicted probability value of the final classifier:

[0043]

[0044] in, is the cross entropy function, and They represent the predicted probability values ​​of the student classifier and the teacher classifier in knowledge distillation respectively; T is a temperature value used to control the smoothness of the output before normalization; γ is a hyperparameter used to adjust the two loss functions.

[0045] The total loss function of the second stage self-distillation training is:

[0046]

[0047] Among them, α and β are hyperparameters used to adjust different loss functions.

[0048] Step 4: Deploy the trained model to the edge device for dynamic inference to improve the inference speed.

[0049] The trained model is deployed to the edge device and the image classification task is performed on the edge device. Confidence is used as the exit indicator, which represents the maximum probability among all classification categories. The prediction confidence c of the mth exit point is m Defined as:

[0050]

[0051] Among them, p mis the predicted distribution of the mth exit point, and C is the set of classification labels. After the mth exit head outputs the classification result, the confidence level is calculated. If the confidence level is greater than a pre-set threshold τ, inference is terminated early, and the classification result of the mth exit head is output; otherwise, inference continues with subsequent neural network layers. During inference, the model can derive different exit metrics based on the sample, thereby dynamically adjusting the exit path. The threshold τ can be adjusted based on the model's computational overhead, edge device hardware performance, and application requirements to achieve a trade-off between accuracy and efficiency. Lowering the threshold increases inference speed, but at the expense of model accuracy. If the exit conditions are not met, the model outputs the final classifier result.

[0052] Beneficial effects

[0053] 1. The present invention discloses a method for accelerating inference of a neural network with dynamic early exit for edge devices. The method uses a heterogeneous exit header structure. Compared with the traditional exit header structure, it can improve the feature extraction capability of the model, achieve higher model accuracy, faster inference speed and lower model operation complexity.

[0054] 2. The present invention discloses a method for accelerating inference of dynamic early exit of neural networks for edge devices. It uses a two-stage training strategy to train a dynamic early exit model based on heterogeneous exit heads. Compared with traditional training strategies, it can more fully exert the potential of the exit head, further improve the model accuracy, and reduce the model operation complexity.

[0055] 3. This invention discloses a method for accelerating neural network inference with dynamic early retirement for edge devices. Using dynamic inference, it can dynamically execute parts of the network, saving computing resources. It can also dynamically select the model output path based on the complexity of the input data to adapt to different scenarios and achieve a trade-off between accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of the framework of the heterogeneous exit head in this embodiment;

[0057] Figure 2 This is a schematic diagram of a dynamic early exit framework based on a heterogeneous exit header structure in this embodiment;

[0058] Figure 3 This is a performance-efficiency trade-off diagram of a neural network dynamic early retirement accelerated inference method for edge devices disclosed in this example;

[0059] Figure 4 This is a flowchart of a method for accelerating inference of a neural network dynamic early retirement for edge devices disclosed in the present invention. DETAILED DESCRIPTION

[0060] The present invention will be described in detail below with reference to the accompanying drawings and embodiments, and the technical problems solved by the technical solution of the present invention and the beneficial effects thereof will be discussed. It should be noted that the described embodiments are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0061] Example 1

[0062] This embodiment discloses a method for accelerating inference of a neural network with dynamic early retirement for edge devices, which is applied to the ViT model. The specific steps are as follows:

[0063] Step 1: Construct two heterogeneous exit heads: a local perception head based on convolution and a global aggregation head based on self-attention mechanism;

[0064] This embodiment adopts a heterogeneous exit head structure, such as Figure 1 As shown in the figure, two local perception heads with convolution kernel sizes of k=3 and k=0 are used, and two global aggregation heads with window sizes of s=2 and s=3 are used. Due to the use of convolution modules, the local perception head can improve the ability to obtain local information and the feature extraction ability of the backbone network; for the local perception head with a convolution kernel size of k=0, it skips PDConv, thereby significantly reducing the additional computational complexity brought by the local perception head. Due to the use of self-attention modules, the global aggregation head can effectively capture key semantic information; for the global aggregation head with a window size of 2×2 or 3×3, the input feature In the PFC module, the input features are reshaped to 768×14×14 size, and the downsampled features are or It is then restored to 49×768 or 25×768 size, and the downsampling process significantly reduces the additional computational effort caused by the global aggregation head.

[0065] Step 2: Establish a dynamic early exit model based on heterogeneous exit headers on the ViT backbone network;

[0066] like Figure 2 As shown, using the ViT-B model as the backbone network, the input image is first segmented into several equal-sized blocks. Each block and a learnable classification token are then passed through the block embedding module to obtain a block embedding vector. The block embedding vector then passes through 12 Transformer encoding layers, enabling the learning of rich semantic features in the image. Eight exit headers are inserted between the ViT-B encoding layers to form a dynamic early exit network. The exit points are {4, 5, 6, 7, 8, 9, 10, 11}, and the placement of each exit header is shown below:

[0067] Exit header structure Placement Local perception head with convolution kernel size k=3 {4，5} Local perception head with convolution kernel size k=0 {6，7} Global aggregation header with window size r=2 {8，9} Global aggregation header with window size r=3 {10，11}

[0068] The exit threshold is set to 0.8, that is, τi =0.8 (i=1,2,...8), so the final prediction value in the inference phase is:

[0069]

[0070] in It represents the probability that the i-th exit head predicts the input category to be c. When the above formula is satisfied or the reasoning is executed to the final classifier, the final prediction value is obtained.

[0071] Step 3: Use a two-stage training strategy to train a dynamic early exit model based on heterogeneous exit heads;

[0072] After building a dynamic early exit model based on heterogeneous exit headers, we employed a two-stage training strategy to train the entire model, fully leveraging the potential of the exit headers. The model was trained on the CIFAR-100 dataset, with 100 iterations on the dataset, a batch size of 64, and an initial learning rate of 5e-5. The learning rate was linearly decreased after each iteration until it reached 0. Training was divided into two stages:

[0073] The first phase is alternating weighted training, where the number of iterations on the dataset is set to 50. The backbone network has 12 layers, with 8 exit heads and 1 final classifier. The alternating weighted method uses two objective functions alternately between odd and even training iterations:

[0074]

[0075] The exit point m = {4, 5, 6, 7, 8, 9, 10, 11, 12}, from shallow to deep, the exit point of the i-th classifier is m i For example, the exit point of the first classifier is m1=4, and the exit point of the ninth classifier is m9=12.

[0076] The second stage is self-distillation training. The number of iterations on the dataset during this stage is also set to 50. The loss function of the general knowledge distillation in the self-distillation prediction distillation is:

[0077]

[0078] The total loss function during self-distillation is:

[0079]

[0080] Step 4: Deploy the trained model to the edge device for dynamic inference

[0081] After training the model, deploy it on the NVIDIA Jetson Nano edge device and use the CIFAR-100 test dataset to perform dynamic inference testing on the model. Assume that f0 represents the output of image preprocessing, that is, the block embedding vector obtained after block and embedding operations, and f i represents the output of the i-th encoding layer, z i Represents the classification probability distribution, and the dynamic inference algorithm is as follows:

[0082]

[0083] Top-1 classification accuracy and speedup are used as performance and efficiency indicators. For a ViT with L layers, the speedup is calculated by comparing the number of layers actually executed in the forward propagation with the number of complete layers. The speedup is defined as:

[0084]

[0085] Among them, m i is the number of samples exiting at the i-th layer of ViT. In addition, the average number of cumulative multiplication and addition operations performed on the entire test dataset is also used as one of the evaluation metrics.

[0086] In addition, two exit header structures, the traditional exit header structure and the ViT exit header structure, were used for comparison. The traditional exit header structure is relatively simple, that is, a pooling layer is followed by a fully connected layer. The ViT exit header structure directly uses the ViT encoding layer as the exit header. The placement of the traditional exit header is the same as that of the heterogeneous exit header. However, the ViT exit header structure is only placed at one exit point {6} because the exit header itself has a large amount of computation. Placing too many ViT exit headers will result in a huge increase in the amount of additional computation and parameters. The training strategy uses a conventional training method rather than a two-stage training strategy. The other training hyperparameter settings are consistent with the heterogeneous exit header. The final measurement results are as follows:

[0087] method Accuracy Multiply-add accumulate operands Speedup Original Model 90.8％ 16.93G 1.00× Traditional exit head structure 86.5％ 10.16G 1.64× ViT Exit Header Structure 87.5％ 11.65G 1.65× Heterogeneous exit header structure 88.5％ 9.76G 1.87×

[0088] Compared with traditional exit head structures and conventional training methods, the method disclosed in this embodiment can achieve higher accuracy, higher acceleration ratio and lower number of multiplication and addition cumulative operations. Compared with the original model, the method disclosed in this embodiment achieves a 1.87-fold acceleration ratio with only 2.3% accuracy loss, reduces the number of multiplication and addition cumulative operations by more than 40%, effectively improves model performance and computational efficiency, and reduces model complexity.

[0089] In addition, by adjusting the exit threshold τ, different accuracy and acceleration ratios can be obtained, thereby achieving a trade-off between performance and efficiency. Figure 3As shown, the method disclosed in the present invention can obtain a higher acceleration ratio under the same accuracy and a higher accuracy under the same acceleration ratio compared with the traditional method.

[0090] Therefore, the present invention discloses a neural network dynamic early exit accelerated inference method for edge devices, which uses a heterogeneous exit head structure to improve the feature extraction capability of the model, and uses a two-stage training strategy to fully tap the potential of the exit head, effectively improving the model accuracy while reducing the model operation complexity and accelerating the inference speed, providing support for the deployment of intelligent applications based on complex models on edge devices.

[0091] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A neural network dynamic early-retirement accelerated inference method for edge devices, characterized by: The following steps are included: Step 1: Construct two heterogeneous exit heads: a local perception head based on convolution and a global aggregation head based on self-attention mechanism; Step 2: Establish a dynamic early exit model based on heterogeneous exit headers on the backbone network; Step 3: Use a two-stage training strategy to train a dynamic early exit model based on heterogeneous exit heads; The implementation method of step three is: In the first stage, the alternating weighted training method is used to rapidly improve the model performance; The backbone network has L layers, which have M exit heads and 1 final classifier. From shallow to deep, the exit point of the i-th classifier is m i , where m≤L,i≤M+1, the final exit point of the classifier is m M+1 = L; the alternating weighting method alternates between odd and even numbers of training iterations using two objective functions: Among them, θ represents the model parameters of the backbone network, w i represents the model parameters of the i-th exit head; in the image classification task, the loss function is the cross entropy function; In the second stage, the self-distillation training method freezes the parameters of the backbone network and the final classifier and only updates the parameters of the exit header. This method further improves model efficiency while ensuring that the model accuracy does not decrease. The overall loss function includes heterogeneous distillation loss, homogeneous distillation loss, and prediction loss function. The heterogeneous distillation loss function is used to promote learning between heterogeneous exit heads. In order to reduce the conflict between multiple losses and the model performance is generally better as the network deepens, only the features of the last layer are used as reference features for the first and last exit heads of the local perception head and the global aggregation head. Since the output of the exit head and the output of the final layer have different dimensions, an alignment module is used to match the dimensions. This module includes deep convolution, GELU and BN activation functions. The feature map of the last layer is The dimension of is first reconstructed as The reconstructed feature map is reduced in dimension by the corresponding module and then restored to its original size N ′ ×D; the heterogeneous distillation loss function is expressed as: in, is the Kullback-Leibler divergence function, Align(·) represents the alignment module; The isomorphic distillation loss function is used to further improve the performance between exit heads with similar structures. In each type of exit head, the last exit head is used as a teacher model to help the previous isomorphic exit head learn, that is, among all local perception heads, the features of the last local perception head are used as reference features of the previous local perception heads. Given the feature mapping from the first to the mth exit head The isomorphic distillation loss function between local perception heads is expressed as: in, is the mean square error function; since the feature maps of the global aggregation head have different shapes, the dot product operation is performed on the feature maps to match the dimensions; given the feature map of the global aggregation head of the mth exit point By calculation Convert its shape to D×D; the loss function of isomorphic distillation between global aggregation heads is expressed as: The total loss function of isomorphic distillation is expressed as: The prediction distillation loss function can further ensure that the model performance is not degraded; the final classifier is used as the reference label of the M / 2 and M exit points where the last local perception head and global aggregation head are located; the label sample of an input sample is y, and the predicted probability values ​​of the M / 2 and M exit points are respectively and The loss function of prediction distillation is expressed as: in, is the loss function of general knowledge distillation, Represents the predicted probability value of the final classifier: in, is the cross entropy function, and They represent the predicted probability values ​​of the student classifier and the teacher classifier in knowledge distillation respectively; T is a temperature value used to control the smoothness of the output before normalization; γ is a hyperparameter used to adjust the two loss functions; The total loss function of the second stage self-distillation training is: Among them, α and β are hyperparameters used to adjust different loss functions; Step 4: Deploy the trained model to the edge device for dynamic inference to improve the inference speed.

2. The method for accelerating inference of a neural network using dynamic early retirement for edge devices according to claim 1, wherein: The implementation method of step one is: The convolution-based local perception head uses 1×1 convolution to expand the dimension of the input feature. The expanded feature is passed to the position weighted depth convolution with a convolution kernel size of k×k; the position weighted depth convolution PDConv is: Among them, PDConv(·) represents the output of position weighted depth convolution; X represents the input feature; DWConv k×k represents a depthwise convolution with a kernel size of k×k; f(·) is a decreasing linear mapping function used to determine the kernel size of PDConv and reduce the additional computational overhead caused by the introduction of exit heads, i.e., k=f(m), m≤M / 2, where M represents the number of exit heads; when k=0, the expanded features will bypass PDConv and directly enter the subsequent part; otherwise, the expanded features of the mth exit position are passed to the k×k depthwise convolution; after PDConv, the expanded features are remapped to the original dimension using 1×1 convolution and passed to the average pooling layer; the output of the local perception head LPH is: Where LPH(·) represents the output of the local perception head; For a given input feature, N represents the number of blocks, D represents the dimension of the hidden layer; X CLS Represents classification tokens; Conv 1×1 (·) represents 1×1 convolution, and Pool(·) represents average pooling. After each convolution, Gaussian error linear unit and batch normalization are used. Finally, the output of the local perception head is passed to the linear classifier to obtain the classification result. The global aggregation head based on the self-attention mechanism consists of self-attention modules; the global aggregation head adopts a position-weighted feature convergence module, namely PFC, to further extract key features; in the PFC module, the input features are reshaped into and downsampled using a window of size s×s; the downsampled features are restored to the original dimensional format N ′ ×D, where To avoid introducing additional learnable parameters, average pooling with a stride of s is used to achieve downsampling with a window size of s×s; the window size of PFC depends on the exit position m, and a larger window size is adopted for the exit points at deeper positions, thus significantly reducing the additional computational amount; an increasing linear mapping function g(·) is used to determine the window size of PFC, that is, s = g(m), M / 2 < m ≤ M; the position-weighted feature convergence module PFC is expressed as: PFC(X feature ,m)=Pool g(m) (X feature ), Among them, PFC(·) represents the output of the position weight feature aggregation module, X feature For a given input feature, Pool g(m) represents average pooling with a step size of g(m); the output of the global aggregation head GAH is: GAH(X feature ,m)=Pool(MHSA(PFC(X feature ,m)))+X CLS , Where GAH(·) represents the output of the global aggregation head, MHSA(·) represents multi-head self-attention, Pool(·) represents average pooling, and softmax(·) represents the normalized exponential function; d is the vector dimension; in MHSA(X), the input X is transformed by the learnable transformation matrix W Q 、W K and W V is linearly transformed into the query vector, key vector, and value vector in the self-attention mechanism.

3. The method for accelerating inference of a neural network using dynamic early retirement for edge devices according to claim 2, wherein: The implementation method of step 2 is: Using the ViT model as the backbone network, M exit heads are inserted between the encoding layers of ViT to form a dynamic early exit network, of which M / 2 local perception heads based on convolution are placed at the exit points in the shallow layer of the network; M / 2 global aggregation heads based on the self-attention mechanism are placed at the exit points in the deep layer of the network; the positions of the exit heads are determined according to the approximate equidistant calculation distribution, that is, the amount of neural network calculation between adjacent exit points is basically the same; the output of the encoding layer is input into the corresponding exit head and classifier, and the classification result is obtained through the prescribed calculation to obtain the prediction confidence; if the confidence is less than or equal to the pre-set threshold, the calculation of the subsequent neural network layer is continued; if the confidence is greater than the threshold, the reasoning is terminated early and the calculation of the subsequent neural network layer is no longer executed, thereby accelerating the reasoning; The final prediction value of the inference phase of a dynamic early-exit network consisting of M exit heads and 1 final classifier is: in, represents the probability that the i-th exit head predicts the input category to be c, τ i Represents the exit threshold of the i-th exit head. When the above formula is satisfied or the reasoning is executed to the final classifier, the final prediction value is obtained.

4. The method for accelerating inference of a neural network using dynamic early retirement for edge devices according to claim 1, wherein: The implementation method of step four is: The trained model is deployed to the edge device and the image classification task is performed on the edge device. The confidence is used as the exit indicator, which represents the maximum probability of all classification categories. The prediction confidence c of the mth exit point is m Defined as: Among them, p m is the predicted distribution of the mth exit point, and C is the set of classification labels; after the mth exit head outputs the classification result, the confidence level is calculated; if the confidence level is greater than the pre-set threshold τ, the reasoning is terminated early and the classification result of the mth exit head is output; otherwise, the reasoning of subsequent neural network layers is continued; during the reasoning process, the model can obtain different exit indicators based on the samples, thereby dynamically adjusting the exit path; the size of the threshold τ can be adjusted according to the model computing overhead, edge device hardware performance and application requirements to achieve a trade-off between accuracy and efficiency; lowering the threshold will increase the reasoning speed, but will correspondingly sacrifice a certain amount of model accuracy; if the exit condition is not met, the model will output the result of the final classifier.

Images