Hardware-aware early-quit neural network optimization method
Through the hardware-aware early regression neural network optimization method, the NSGA-II algorithm is used to optimize the early regression network. Combined with the hardware characteristics, the problems of high latency and large energy consumption of deep neural network in edge computing environments are solved, and more efficient resource utilization and accuracy are achieved.
Patent Information
- Application Number
- CN202411715787.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-14
AI Technical Summary
In resource-constrained edge computing environments, the deep neural network model has high inference latency and high energy consumption, and traditional early regression neural networks have challenges in controlling accuracy and efficiency.
The hardware-aware early retreat neural network optimization method is adopted to optimize the early retreat network through the NSGA-II algorithm. Combined with the hardware characteristics, the optimization goals include accuracy, delay and energy consumption. After freezing the backbone network weights, the early retreat branches are added to ensure the accuracy of the model.
On the basis of ensuring model accuracy, inference delay and energy consumption are reduced, and more efficient resource utilization is achieved, suitable for edge computing scenarios.
Smart Images
Figure CN119623529A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of early-departure neural network, and in particular to a hardware-aware early-departure neural network optimization method. Background Art
[0002] In the current digital age, the widespread use of deep learning models has brought significant progress in many fields, especially in computer vision, natural language processing, and speech recognition. The success of deep neural networks lies in their powerful representation learning capabilities, but this also leads to the complexity and size of the model. In mobile devices, embedded systems, and edge computing environments, this large-scale model places strict requirements on computing resources, bringing challenges to mobile devices, embedded systems, and edge computing environments. Traditional deep neural network structures usually require a lot of computing power and storage resources, which may lead to performance bottlenecks in resource-constrained environments. Mobile devices usually have limited processing power and storage space, and embedded systems also face the balance between power consumption and performance. The edge computing environment requires real-time reasoning locally, requiring the model to have higher real-time performance and efficiency.
[0003] Early exit networks are able to make data-dependent decisions during inference, based on confidence. Many early exit networks are able to output "simple" samples when exiting early if a certain confidence-based criterion is met. These methods usually require estimating the confidence of the intermediate predictions and comparing them to a predefined threshold for the decision. In classification tasks, confidence is usually represented by the maximum element of the SoftMax output. [5,6] , confidence-based criteria are easy to implement and usually do not require special training methods. The trade-off between accuracy and efficiency is controlled by manipulating the threshold, which is usually adjusted on a validation dataset. However, the literature [11,12] The problem of early exit based on confidence is also pointed out. Since misclassified samples can obtain high confidence when exiting early, it will affect the effectiveness of decision-making based on confidence.
[0004] In addition, the location and network structure of the early-falling branch are still hyperparameters in many studies. The location of the early-falling branch is usually determined by the researchers themselves, but the structure of the early-falling branch layer is relatively uniform in many studies, including two convolutional layers, batch normalization layers, activation layers, and linear layers, but the specific parameters of the convolutional layer, such as the convolution kernel, will be different. [13,14,10] In addition to architecture design, training is also crucial for dynamic networks. [10,15]It is usually trained by minimizing the weighted cumulative loss of the intermediate classifiers. One challenge in training such models is the joint optimization of multiple classifiers, which may interfere with each other. There is no unified view on how to set the weights of multiple classifiers. Some studies believe that higher weights should be given to the later classifiers.
[16] Some studies have shown that giving higher weights to the front classifiers will gain more accuracy. [5] , and many studies have concluded that giving each classifier equal weight can achieve higher energy efficiency. [6,10] . Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention provides a hardware-aware early-retirement neural network optimization method, which solves the problems raised in the above-mentioned background technology.
[0006] To achieve the above objectives, the present invention is implemented by the following technical solutions: a hardware-aware early-dead neural network optimization method, specifically comprising the following steps: S1 model input and S2 optimization target, wherein the S1 model input and the S2 optimization target are connected through a post-genetic algorithm;
[0007] In S1, the model input includes a data set, a backbone network and hardware parameters, and the backbone network is extended to connect to the early-falling branch;
[0008] In S2, the optimization target branch is provided with accuracy, delay and energy consumption modules;
[0009] The model input in S1 and S2 consists of a data set, a backbone network with early-departure branches added, and hardware parameters. The generated early-departure network is optimized through the NSGA-Ⅱ algorithm. There are three optimization objectives, namely accuracy, latency, and energy consumption. Before the early-departure branch is added to the backbone network, it is in a trained state. Adding the early-departure branch after freezing the weight helps to ensure the accuracy score of the entire early-departure network and finally obtain the Pareto frontier.
[0010] Preferably, the backbone network is obtained by any convolutional neural network model, and after training to reach a certain accuracy, it is used as the backbone network of the early-leaving network. If there are positions in the backbone network where early-leaving branches can be inserted, a vector is defined to represent the insertion of early-leaving branches.
[0011] B=[B1,B2,B3,…,B M-1 ],B i ∈{0,1}:
[0012] Among them, 0 means that no branch is inserted at this position, and 1 means that an early-retirement branch is inserted at this position.
[0013] Preferably, in S1 and S2, according to the underlying hardware, data flow, shared buffer size, private buffer size and the number of PEs can be used to construct P, the data flow includes WS, OS and RS, according to the actual device parameters and the settings of the relevant literature, the size of the shared buffer ranges from 54KB to 4MB, the size of the private buffer ranges from 512B to 2KB, and the number of PEs ranges from 168 to 1024. [25–30,28] ,Under different data flows, different reused tensors, combined with different buffer sizes, will lead to different times of data movement, thus affecting energy consumption.,The size and number of PEs are reflected in the parallelism of the accelerator.,The larger the number and PE size, the higher the parallelism, thus effectively reducing the inference latency.
[0014] Preferably, in S1 and S2, the (E, H) combination subspace is searched by the NSGA-II algorithm to randomly generate an initial population P of size N. X,H , after non-dominated sorting, selection, crossover and mutation, the offspring population Q is generated X,H , and combine the two populations to form a population R of size 2N X,H , perform fast non-dominated sorting, and calculate the crowding degree of individuals in each non-dominated layer. According to the non-dominated relationship and the crowding degree of individuals, select suitable individuals to form a new parent population, generate a new child population through the basic operation of the genetic algorithm, and merge the new parent population with the new child population to form a new population. Repeat the above operation until the last generation, and return to the Pareto optimal set (e * ,h * ), in each generation, a population is generated from the encoding of the combinatorial subspace for dynamic fitness evaluation.
[0015] Preferably, the simulation software for obtaining energy consumption and time delay is MAESTRO.
[0016] Preferably, the MAESTRO input includes a layer description file of the network structure, a hardware parameter file and a data flow description, wherein the hardware parameter file and the data flow description are directly provided, but only supports the generation of a layer description file of the network structure based on Keras, thus combining the tool
[31] The provided method is optimized to support the generation of various types of layer description files based on Pytorch. Tensor analysis is used to extract tensor names, tensor types, coupling dimensions, and the size of each coupling dimension; cluster analysis is used to extract data flow instructions at each cluster level, extract the size of data dimensions at each cluster level, infer missing instructions, application strides, etc., and analyze the number of sub-clusters at each cluster level; reuse analysis is used to analyze the unit mapping size and the reuse of cross-unit steps for each possible combination of data iteration position situations, that is, computing time reuse, and merge the unit step reuse analysis results to calculate the reuse of the entire sub-cluster array at each cluster level, that is, computing space reuse; performance analysis is used to identify all possible data iteration situations and the number of times the situation occurs, consider data reuse, calculate ingress / egress traffic, calculate communication and computing delays, consider delay hiding, and determine prominent delays; cost analysis is used to calculate the number of buffer accesses and MAC operations for each data iteration situation, and determine the buffer size requirements based on the worst-case analysis.
[0017] Preferably, the ResNet34
[32] For the backbone network, after adding the early-retirement branch, under the same hardware settings, three data streams, WS, OS, and RS, are used for inference, and the runtime latency and energy consumption of the convolutional layer and the fully connected layer.
[0018] Preferably, the delay and energy consumption cannot be optimized on the same data stream. At the same time, introducing more PEs will lead to high energy consumption, but will also greatly reduce the delay.
[17] ,Therefore, it is necessary to perform multi-objective optimization on data flow and related hardware parameters, balancing latency and energy consumption.
[0019] The present invention provides a hardware-aware early-exit neural network optimization method, which has the following beneficial effects: This project deeply explores an innovative hardware-aware early-exit neural network optimization strategy, aiming to achieve coordinated optimization of the early-exit network and the underlying hardware design in response to the efficient reasoning needs in edge computing scenarios. Traditional neural network models often face the challenges of high reasoning latency and high energy consumption when running on resource-constrained edge devices. This method effectively reduces these bottlenecks by closely combining the early-exit mechanism with hardware characteristics. The early-exit network can terminate the reasoning process of some samples in advance while ensuring the accuracy of the model, thereby reducing unnecessary computing overhead. In order to further improve the performance of the early-exit network, this project also proposes a new An algorithm is specially used to search for the optimal exit location of the early-exit network. This algorithm transforms the selection of the early-exit network exit and the design space exploration problem of accelerator hardware parameters into a complex multi-objective optimization problem. By introducing an evolutionary genetic engine, the algorithm can intelligently search in a huge candidate space to find the exit configuration that can maximize model accuracy while minimizing inference latency and energy consumption. At the same time, the algorithm also takes into account the specific constraints of the underlying hardware, such as the parallelism of computing units and memory bandwidth, to ensure that the generated hardware design can efficiently support the operation of the early-exit network. This comprehensive optimization strategy provides new ideas for the deployment of neural networks on edge devices and helps promote the widespread application of artificial intelligence technology in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is an overview diagram of the model of the present invention;
[0021] Figure 2 This is a flow chart of the model of the present invention;
[0022] Figure 3 It is the structure diagram of the simulation software of the present invention;
[0023] Figure 4 This is the network delay and energy consumption of the early withdrawal based on ResNet34 of the present invention. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] See also Figure 1-4The embodiment of the present invention provides a technical solution: a hardware-aware early-departure neural network optimization method, wherein the model input consists of a data set, a backbone network with early-departure branches added, and hardware parameters. The generated early-departure network is optimized by the NSGA-Ⅱ algorithm. There are three optimization objectives, namely, accuracy, delay, and energy consumption. Before the early-departure branch is added to the backbone network, it is in a trained state. Adding the early-departure branch after freezing the weight helps to ensure the accuracy score of the entire early-departure network, and finally obtains the Pareto frontier. The backbone network is obtained by any convolutional neural network model. After training to reach a certain accuracy, it is used as the backbone network of the early-departure network. Assuming that there are M positions in the backbone network where the early-departure branch can be inserted, a vector B is defined to represent the insertion of the early-departure branch.
[0026] B=[B1,B2,B3,…,B M-1 ],B i ∈{0,1}
[0027] Among them, 0 means that no branch is inserted at this position, and 1 means that an early-retired branch is inserted at this position;
[0028] According to the underlying hardware, data flows, shared buffer size, private buffer size and the number of,PEs, can be used to construct P,data flows including,WS,,OS,and,RS,. According to the actual device parameters and the settings of,related literature, the shared buffer size ranges from 54KB to 4MB, the private buffer size ranges from 512B to 2KB, and the number of,PEs ranges from 168 to 1024. [25–30,28] ,Among them, different reused tensors under different data flows, combined with different buffer sizes, will lead to different times of data movement, thus affecting the energy consumption.,The size and number of PEs are manifested as the parallelism of the accelerator.,The larger the number and PE size, the higher the parallelism, which effectively reduces the latency of inference.
[0029] Through,NSGA-Ⅱ,algorithm search (E,H) combination subspace,randomly generate the initial population P of size N X,H , after non-dominated sorting, selection, crossover and mutation, the offspring population Q is generated X,H , and combine the two populations to form a population R of size 2N X,H , perform fast non-dominated sorting, and calculate the crowding degree of individuals in each non-dominated layer. According to the non-dominated relationship and the crowding degree of individuals, select suitable individuals to form a new parent population, generate a new child population through the basic operation of the genetic algorithm, and merge the new parent population with the new child population to form a new population. Repeat the above operation until the last generation, and return to the Pareto optimal set (e * ,h *), in each generation, a population is generated from the encoding of the combined subspace for dynamic fitness evaluation. MAESTRO, the input includes the layer description file of the network structure, the hardware parameter file and the data flow description. The hardware parameter file and the data flow description are directly provided, but only Keras is supported to generate the layer description file of the network structure. Therefore, the tool is combined
[31] The provided method is optimized to support the generation of various types of layer description files based on Pytorch. Tensor analysis is used to extract tensor names, tensor types (input / output), coupling dimensions, and the size of each coupling dimension; cluster analysis is used to extract data flow instructions at each cluster level, extract the size of data dimensions at each cluster level, infer missing instructions, application strides, etc., and analyze the number of sub-clusters at each cluster level; reuse analysis is used to analyze the unit mapping size and the reuse of cross-unit steps for each possible combination of data iteration location situations, that is, computing time reuse, and merge the unit step reuse analysis results to calculate the reuse of the entire sub-cluster array at each cluster level, that is, computing space reuse; performance analysis is used to identify all possible data iteration situations and the number of times the situation occurs, considering data reuse, calculating ingress / egress traffic, calculating communication (ingress / egress) and computing delays, considering delay hiding, and determining prominent delays; cost analysis is used to calculate the number of buffer accesses and MAC operations for each data iteration situation, and determine the buffer size requirements based on the worst-case analysis,
[0030] In this paper, ResNet34
[32] For the backbone network, after adding early-retirement branches, under the same hardware settings, three data streams, WS, OS, and,RS, are used for inference, respectively, to obtain the runtime latency and energy consumption of the convolutional layer and the fully connected layer. The latency and energy consumption cannot be optimized on the same data stream. At the same time, introducing more,PE,numbers will lead to high energy consumption, but will also greatly reduce the latency.
[17] ,Therefore, it is necessary to perform multi-objective optimization on data flow and related hardware parameters, balancing latency and energy consumption.
[0031] The first is to reduce the energy consumption of the early-departure network, and the second is to reduce the delay of the early-departure network. Due to the particularity of data flow and hardware parameters, finding an optimal solution to minimize the delay and energy consumption of the early-departure network is a complex task. Therefore, this project uses the multi-objective optimization algorithm NSGA-II to deal with this problem. By using the simulation software MAESTRO to calculate the energy consumption and delay of the early-departure network, and obtain the Pareto frontier, it helps to provide a reasonable reference for the network structure setting and hardware equipment selection of the reasoning process in edge scenarios.
[0032] The exploration of the hardware design space is a key means to solve this problem. It helps to find the best solution for the performance of early-exit networks. By combining different hardware settings, early-exit branch locations and numbers, the optimal design can be found to improve the energy efficiency of the reasoning process. In addition, in-depth exploration of the hardware design space can also identify directions that are not yet covered by current research, and provide important clues for guiding research directions such as the design of accelerators suitable for early-exit networks and the optimization of early-exit networks. This will help promote the development of more innovative and efficient design and optimization methods.
[0033] In practice, the use of the multi-objective optimization algorithm NSGA-II helps to comprehensively consider the two goals of reducing energy consumption and reducing latency, and obtain a series of Pareto frontier solutions. These solutions provide comprehensive information on the hardware design space, allowing experiments to have a more comprehensive and in-depth understanding of the performance of early-departure networks under different conditions. This systematic hardware design space exploration method is expected to provide a more effective solution for accelerating the dynamic network reasoning process in edge scenarios, and promote development and innovation in this field.
[0034] The early-retirement network is jointly optimized through a hardware-aware method, aiming to improve the performance of the inference process on the accelerator while maintaining accuracy, and achieve a large gain in latency and energy consumption, especially near the Pareto frontier. Considering the hugeness of the entire design exploration space, this project decomposes it into two subspaces for more targeted experiments and joint optimization.
[0035] For the optimization of early-exit branches, the location and network structure of the early-exit branches need to be determined in the experiment. The network structure usually adopts a combination of convolutional layers, batch normalization layers, activation layers, and fully connected layers. The settings of these structures can refer to their universality. Therefore, in the subspace, only the location design of the early-exit branches is focused. Secondly, the optimization of hardware parameters is a key part. This subspace needs to configure accelerator-related hardware parameters to improve the resource efficiency of the early-exit network. Considering how different computing workloads utilize the underlying hardware components, the early-exit network can obtain maximum resource efficiency under different accelerator settings. In actual scenarios, edge devices may involve various heterogeneous accelerators, so hardware parameters include data flow, shared buffer size, private buffer size, and the number of processing units (PEs).
[0036] In order to ensure the effectiveness of the experiment, the backbone network of the early-withdrawal network can adopt various convolutional neural network (CNN) models. During the experiment, the backbone network is first trained to achieve a satisfactory accuracy, and then the early-withdrawal branch is added and the experiment is carried out in combination with the hardware parameters. It is worth noting that during the training process, the parameters of the backbone network need to be frozen to prevent the training of the early-withdrawal branch from affecting the functions of the early layers of the network, thereby causing the accuracy of the early-withdrawal network to decrease.
[0037] This joint optimization method enables the systematic exploration of the performance of the early-departure network under different hardware configurations and structural schemes during the experiment, thereby providing more effective solutions for practical applications. Through careful adjustment of hardware perception, efficient and high-performance early-departure networks can be deployed on edge devices.
Claims
1. A hardware-aware early-retirement neural network optimization method, characterized in that; Specifically, the following steps are included: S1 model input and S2 optimization target, wherein the S1 model input and S2 optimization target are connected through a post-genetic algorithm; In S1; The model input includes a data set, a backbone network and hardware parameters, and the backbone network is extended to connect with early-falling branches; In S2, the optimization target branch is provided with accuracy, delay and energy consumption modules; The model input in S1 and S2 consists of a data set, a backbone network with early-departure branches added, and hardware parameters. The generated early-departure network is optimized through the NSGA-Ⅱ algorithm. There are three optimization objectives, namely accuracy, latency, and energy consumption. Before the early-departure branch is added to the backbone network, it is in a trained state. Adding the early-departure branch after freezing the weight helps to ensure the accuracy score of the entire early-departure network and finally obtain the Pareto frontier.
2. The hardware-aware early-dead neural network optimization method according to claim 1, characterized in that: The backbone network is obtained by any convolutional neural network model. After training to reach a certain accuracy, it is used as the backbone network of the early-leaving network. Assuming that there are positions in the backbone network where early-leaving branches can be inserted, a vector is defined to represent the insertion of early-leaving branches. B=[B1,B2,B3,…,B M-1 ],B i ∈{0,1}: Among them, 0 means that no branch is inserted at this position, and 1 means that an early-retirement branch is inserted at this position.
3. The hardware-aware early-dead neural network optimization method according to claim 1, characterized in that: In S1 and S2, according to the underlying hardware, data flow, shared buffer size, private buffer size and the number of PEs can be used to construct P. The data flow includes WS, OS and RS. According to the actual device parameters and the settings of relevant literature, the size of the shared buffer ranges from 54KB to 4MB, the size of the private buffer ranges from 512B to 2KB, and the number of PEs ranges from 168 to 1024. [25–30,28] ,Among them, different reused tensors under different data flows, combined with different buffer sizes, will lead to different times of data movement, thus affecting energy consumption.,The size and number of PEs are expressed as the parallelism of the accelerator.,The larger the number and PE size, the higher the parallelism, which effectively reduces the inference latency.
4. The hardware-aware early-dead neural network optimization method according to claim 1, characterized in that: In S1 and S2, the NSGA-Ⅱ algorithm is used to search the (E, H) combination subspace and randomly generate an initial population P of size N. X,H , after non-dominated sorting, selection, crossover and mutation, the offspring population Q is generated X,H , and combine the two populations to form a population R of size 2N X,H , perform fast non-dominated sorting, and calculate the crowding degree of individuals in each non-dominated layer. According to the non-dominated relationship and the crowding degree of individuals, select suitable individuals to form a new parent population, generate a new child population through the basic operation of the genetic algorithm, and merge the new parent population with the new child population to form a new population. Repeat the above operation until the last generation, and return to the Pareto optimal set (e * ,h * ), in each generation, a population is generated from the encoding of the combinatorial subspace for dynamic fitness evaluation.
5. The hardware-aware early-dead neural network optimization method according to claim 1, characterized in that: The simulation software for obtaining energy consumption and time delay is MAESTRO.
6. The hardware-aware early-dead neural network optimization method according to claim 5, characterized in that: The MAESTRO input includes the layer description file of the network structure, the hardware parameter file and the data flow description, where the hardware parameter file and the data flow description are directly provided, but only supports the layer description file of the network structure generated based on Keras, so it is combined with the tool [31] The provided method is optimized to support the generation of various types of layer description files based on Pytorch. Tensor analysis is used to extract tensor names, tensor types, coupling dimensions, and the size of each coupling dimension; cluster analysis is used to extract data flow instructions at each cluster level, extract the size of data dimensions at each cluster level, infer missing instructions, application strides, etc., and analyze the number of sub-clusters at each cluster level; reuse analysis is used to analyze the unit mapping size and the reuse of cross-unit steps for each possible combination of data iteration position situations, that is, computing time reuse, and merge the unit step reuse analysis results to calculate the reuse of the entire sub-cluster array at each cluster level, that is, computing space reuse; performance analysis is used to identify all possible data iteration situations and the number of times the situation occurs, consider data reuse, calculate inlet / outlet traffic, calculate communication and computing delays, consider delay hiding, and determine prominent delays; Cost analysis is used to calculate the number of buffer accesses and MAC operations for each data iteration case and determine the buffer size requirement based on the worst-case analysis.
7. The hardware-aware early-dead neural network optimization method according to claim 6, characterized in that: The ResNet34 【32】 For the backbone network, after adding the early-retirement branch, under the same hardware settings, three data streams, WS, OS, and RS, are used for inference, and the runtime latency and energy consumption of the convolutional layer and the fully connected layer.
8. The hardware-aware early-dead neural network optimization method according to claim 7, characterized in that: The latency and energy consumption cannot be optimized for the same data stream. At the same time, introducing more PEs will lead to high energy consumption, but will also greatly reduce latency. [17] ,Therefore, it is necessary to perform multi-objective optimization on data flow and related hardware parameters, balancing latency and energy consumption.