A method and device for designing a multi-output neural network for edge scenario dynamic resources

Abstract

A kind of multi-outlet neural network design method and device for edge scene dynamic resource, its method includes: (1) given pre-training neural network, generate effective branch library, and screening is carried out;(2) further reduce memory occupation to the branch in branch library by self-merging, the accuracy of branch is quickly recovered by retraining through few-sample knowledge distillation after self-merging, and form branch candidate library;(3) based on priority, the optimal branch combination that satisfies memory and time requirement is found out to branch candidate library by depth-first search;(4) after selected branch combination is received by on-device scheduler, the branch of pre-training neural network is updated in real time by SBRAN component.The multi-outlet neural network with heterogeneous and dynamic characteristics is converted from traditional single-outlet neural network, and neural network can more flexibly adapt to the change of memory resource, so as to improve running efficiency and accuracy on edge device.

Description

Technical Field

[0001] This invention provides a multi-exit neural network design method and apparatus for dynamic resources in edge scenarios, with particular focus on the application of edge devices in environments with dynamically changing memory resources. Background Technology

[0002] With the rapid development of edge computing in modern technology, the demand for efficiently utilizing the computing power of edge devices is constantly increasing. The core advantage of edge computing lies in its ability to process data close to the data source, thereby reducing latency and improving response speed. However, edge devices are typically limited by lower computing power and limited memory resources, which significantly restricts their ability to handle complex tasks. In this environment, the management and optimization of memory resources become particularly critical.

[0003] Edge devices are filled with a large amount of unpredictable computing resources, including but not limited to CPU / GPU cores and VRAM. The unpredictable nature of these resources means they can be affected by various factors, such as parallel tasks and energy constraints, leading to fluctuations in their performance at different points in time. Against this backdrop, the application of deep neural networks (DNNs) faces a highly challenging task: maintaining efficient inference performance under constantly changing memory resources. Multi-exit neural networks are considered a potential solution because they allow for the output of results in advance during inference, providing greater flexibility and accuracy under resource constraints.

[0004] While multi-exit neural networks have shown promise in handling nondeterministic computational resources, existing techniques remain limited in several key areas. In particular, the accuracy of intermediate inference results and the isomorphism and static structure of the exit models restrict the effectiveness of these networks in dynamic environments. Therefore, for a given pre-trained neural network, it becomes particularly important to efficiently determine the structure, number, and layout of heterogeneous branches and to dynamically update them based on changes in device memory. Summary of the Invention

[0005] To overcome the limitations of existing technologies, this invention proposes an innovative multi-exit neural network design method and apparatus, specifically targeting scenarios with dynamically changing memory resources. The core of this method lies in designing and implementing a system capable of automatically transforming a traditional single-exit neural network into a heterogeneous and dynamic multi-exit neural network. Essentially, this method enables the neural network to adapt more flexibly to changes in memory resources through this transformation, thereby improving its operating efficiency and accuracy on edge devices.

[0006] The first aspect of this invention proposes a multi-exit neural network design method for dynamic resources in edge scenarios, comprising the following steps:

[0007] (1) Given a pre-trained neural network, generate an effective branch library and perform filtering;

[0008] (2) The branches in the branch library are self-merged to further reduce memory usage. After self-merging, the accuracy of the branches is quickly restored by retraining through few-sample knowledge distillation to form a branch candidate library.

[0009] (3) Perform a depth-first search on the branch candidate library based on priority to find the optimal combination of branches that meets the memory and time requirements;

[0010] (4) After the scheduler on the device receives the selected branch combination, it uses the SBRAN component to update the branches of the pre-trained neural network in real time.

[0011] Step 1 specifically includes:

[0012] Step 1.1: Pre-select several convolutional layer structures and fully connected layer structures that can be combined to form effective branches, and form a branch library based on the differences between the convolutional layer structures and fully connected layer structures and the different exit points.

[0013] Step 1.2: Train the branches in the branch library on the given pre-trained neural network to obtain the branch configuration information, including weights, number of parameters, inference accuracy and execution completion time.

[0014] Step 1.3: Filter the branch library. Filtering criteria: delete branches whose execution accuracy does not meet the requirements, and delete branches whose time is greater than the inference time of the pre-trained neural network but whose accuracy is lower.

[0015] In step 2, weight sharing further reduces the memory usage of branches in the branch library, specifically including:

[0016] Step 2.1: Traditional weight-sharing schemes select several branches for weight sharing, but due to differences in the number of layers and neurons among the branches, the sharing benefits are not ideal. This invention proposes branch self-merging, which can maximize the saving of memory space with almost no loss of accuracy.

[0017] Step 2.1.1: Calculate the error between different neurons in the branch. Unlike the Hessian error calculation method, which consumes a lot of memory, this invention uses L2-Norm for error calculation to enable more efficient self-merging of all branches.

[0018] Step 2.1.2: Select neuron pairs to merge based on error. Pair all neurons according to their error magnitude, sort them according to the difference between the neuron pairs, and merge the top 1 / 3 of neuron pairs with lower errors.

[0019] Step 2.1.3: Calculate the shared weights for each neuron pair to be fused. Again, instead of using the Hessian method, we directly calculate the average of the weights of the two neurons as the shared weights, which is simple yet effective.

[0020] Step 2.2: Branch self-merging inevitably leads to a certain degree of accuracy loss. We use few-sample knowledge distillation retraining to quickly restore the accuracy of branches.

[0021] Step 2.2.1: Construct a suitable loss function for retraining. The loss function L consists of three parts: the supervision information MSE from the labels, the self-distilled KL divergence, and the L1-Norm loss with shared weights, where y is the label. It is the prediction for the i-th branch, y c This is the final prediction, W s , The weights correspond to the shared weights and the weights updated via backpropagation, respectively. α and β are adjustable parameters. Preferably, this invention suggests setting α = 0.5 and β = 1.

[0022]

[0023] Step 2.2.2: For self-merged branches, filter them according to the same selection criteria and add them to the branch library.

[0024] In step 3, the memory budget M and time budget T of the given device are... budget This invention proposes a priority-based depth-first search (DFS) to select the optimal branch combination from the branch library, specifically including:

[0025] Step 3.1: This invention proposes the area enclosed by accuracy and time, p = Acc·Time, as a performance indicator, where Acc is the branch accuracy and Time is the deadline minus the branch inference completion time. This represents the performance improvement of the current branch combination per unit of parameters in each branch. As the priority of this branch, the priority of the same branch is different under different current branch combinations, and needs to be calculated according to the current branch combination.

[0026] Step 3.2: With a branch transmission rate of R, we can calculate that DFS has two limitations: memory budget M and search time T. search The branch selection order for each level of DFS is determined according to the branch priority. During search time T... searchUltimately, the optimal branch combination B that meets the memory requirements is found. selected .

[0027]

[0028] In step 4, the branch combination B is performed using an SBRAN component. selected Update to memory to enable memory elastic inference.

[0029] A second aspect of the present invention relates to a multi-exit neural network design apparatus for dynamic resources in edge scenarios, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the multi-exit neural network design method for dynamic resources in edge scenarios of the present invention.

[0030] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the multi-exit neural network design method of the present invention for dynamic resources in edge scenarios.

[0031] The core of this invention lies in designing and implementing an innovative system that can transform a traditional single-exit neural network into a multi-exit neural network with heterogeneous and dynamic characteristics. Through this transformation, the neural network can more flexibly adapt to changes in memory resources, thereby improving operational efficiency and accuracy on edge devices.

[0032] The advantages of this invention are: it can automatically convert single-exit neural networks into multi-exit neural networks, making it particularly suitable for edge computing environments. This innovation solves the challenge of designing and deploying efficient heterogeneous branches under varying memory conditions and significantly reduces the memory overhead caused by dynamic branches. By achieving high-precision inference while reducing memory consumption, this invention provides a powerful solution for efficiently utilizing nondeterministic computing capabilities on resource-constrained devices, and has broad application prospects. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the process of the method of the present invention.

[0035] Figure 2 This is a data structure diagram in the method of the present invention.

[0036] Figure 3 This is a priority-based DFS flowchart in the method of this invention.

[0037] Figure 4 This is a schematic diagram of priority-based DFS in the method of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0039] Example 1

[0040] Reference Figure 1 This invention proposes a multi-exit neural network design method for dynamic resources in edge scenarios. The core of this method lies in designing and implementing an innovative system that can transform a traditional single-exit neural network into a multi-exit neural network with heterogeneous and dynamic characteristics. Through this transformation, the neural network can more flexibly adapt to changes in memory resources, thereby improving operating efficiency and accuracy on edge devices.

[0041] Specifically, the following steps are included:

[0042] Step 1: Given a pre-trained neural network, generate branch candidates, such as... Figure 2 As shown.

[0043] Step 1.1: Pre-select several convolutional layer structures and fully connected layer structures that can be combined to form effective branches, and form a branch library based on the differences between the convolutional layer structures and fully connected layer structures and the different exit points.

[0044] Step 1.2: Train the branches in the branch library on the given pre-trained neural network to obtain the branch configuration information, including weights, number of parameters, inference accuracy and execution completion time.

[0045] Step 1.3: Filter the branch library. Filtering criteria: delete branches whose execution accuracy does not meet the requirements, and delete branches whose time is greater than the inference time of the pre-trained neural network but whose accuracy is lower.

[0046] Step 2: Weight sharing can further reduce the memory usage of branches in the branch library.

[0047] Step 2.1: Traditional weight-sharing schemes select several branches for weight sharing, but due to differences in the number of layers and neurons among the branches, the sharing benefits are not ideal. This invention proposes branch self-merging, which can maximize the saving of memory space with almost no loss of accuracy.

[0048] Step 2.1.1: Calculate the error between different neurons in the branch. Unlike the Hessian error calculation method, which consumes a lot of memory, this invention uses L2-Norm for error calculation to enable more efficient self-merging of all branches.

[0049] Step 2.1.2: Select neuron pairs to merge based on error. Pair all neurons according to their error magnitude, sort them according to the difference between the neuron pairs, and merge the top 1 / 3 of neuron pairs with lower errors.

[0050] Step 2.1.3: Calculate the shared weights for each neuron pair to be fused. Again, instead of using the Hessian method, we directly calculate the average of the weights of the two neurons as the shared weights, which is simple yet effective.

[0051] Step 2.2: Branch self-merging inevitably leads to a certain degree of accuracy loss. We use few-sample knowledge distillation retraining to quickly restore the accuracy of branches.

[0052] Step 2.2.1: Construct a suitable loss function for retraining. The loss function L consists of three parts: the supervision information MSE from the labels, the self-distilled KL divergence, and the L1-Norm loss with shared weights, where y is the label. It is the prediction for the i-th branch, y c This is the final prediction, W s , Corresponding to the shared weights and the weights updated by backpropagation, this invention suggests setting the parameters α = 0.5 and β = 1.

[0053]

[0054] Step 2.2.2: For self-merged branches, filter them according to the same selection criteria and add them to the branch library.

[0055] Step 3: Given the device's memory budget M and time budget T budget This invention proposes a priority-based depth-first search (DFS) to select the optimal branch combination from the branch library, such as... Figure 3 , Figure 4 As shown.

[0056] Step 3.1: This invention proposes the area enclosed by accuracy and time, p = Acc·Time, as a performance indicator, where Acc is the branch accuracy and Time is the deadline minus the branch inference completion time. This represents the performance improvement of the current branch combination per unit of parameters in each branch. As the priority of this branch, the priority of the same branch is different under different current branch combinations, and needs to be calculated according to the current branch combination.

[0057] Step 3.2: With a branch transmission rate of R, we can calculate that DFS has two limitations: memory budget M and search time T. search The branch selection order for each level of DFS is determined according to the branch priority. During search time T... search Ultimately, the optimal branch combination B that meets the memory requirements is found. selected .

[0058]

[0059] Step 4: The on-device scheduler predicts dynamic changes in memory and time using the memory budget component and latency budget component. Based on steps 1-3, the AI ​​task on the device performs dynamic memory inference.

[0060] Example 2

[0061] This embodiment relates to a multi-exit neural network design device for dynamic resources in edge scenarios, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the multi-exit neural network design method for dynamic resources in edge scenarios of Embodiment 1.

[0062] Example 3

[0063] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the multi-exit neural network design method for dynamic resources in edge scenarios as described in Embodiment 1.

[0064] This invention takes into account the dynamic changes in memory of edge devices, enabling branch updates from the server to the device. Even with dynamic memory changes, weaker devices can still effectively perform model inference, thus achieving efficient and reliable AI task execution in resource-constrained edge computing environments.

[0065] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A multi-exit neural network design method for dynamic resources in edge scenarios, comprising the following steps: (1) Given a pre-trained neural network, generate an effective branch library and perform filtering; (2) The branches in the branch library are self-merged to further reduce memory usage. After self-merging, the accuracy of the branches is quickly restored through few-sample knowledge distillation and retraining, forming a branch candidate library. The specific steps are as follows: Step 2.1: Propose branch self-merging to maximize memory savings with almost no loss of precision, specifically including: Step 2.1.1: Calculate the error between different neurons in the branch, using L2-Norm for error calculation; Step 2.1.2: Select neuron pairs to be merged based on error; group all neurons into pairs according to the magnitude of error, sort them according to the magnitude of the difference between neuron pairs, and merge the first 1 / 3 of neuron pairs with lower errors; Step 2.1.3: Calculate the shared weight of each neuron pair to be fused; directly calculate the average of the weights of the two neurons as the shared weight; Step 2.2: To avoid the accuracy loss caused by branch self-merging, few-sample knowledge distillation retraining is used to quickly restore the accuracy of the branches; Step 2.2.1: Construct a suitable loss function for retraining; loss function It consists of three parts: the supervision information MSE from the labels, the self-distilled KL divergence, and the L1-Norm loss with shared weights. It's a tag. It is the first Prediction of each branch, This is the final prediction. , These correspond to the shared weights and the weights updated via backpropagation, respectively. α and β are adjustable parameters that can be set. （1） Step 2.2.2: For self-merging branches, filter them according to the same selection criteria and add them to the branch library; (3) Perform a depth-first search on the branch candidate library based on priority to find the optimal branch combination that meets the memory and time requirements; the device's memory budget is Time budget is The process of finding the optimal combination of branches that meets the memory and time requirements is as follows: Step 3.1: Determine the area enclosed by accuracy and time. As a performance indicator, For the accuracy of the branch, Subtract the branch inference completion time from the deadline; performance improvement of the current branch combination per unit of parameters per branch. As the priority of this branch, the priority of the same branch is different under different current branch combinations, and needs to be calculated according to the current branch combination; Step 3.2: Assuming the branch transfer rate is R, there are two limitations in calculating DFS: memory budget. and search time The branch selection order for each level of DFS is determined according to the branch priority; during the search time... Ultimately, the optimal combination of branches that meets the memory requirements is found. ; （2） (4) After the scheduler on the device receives the selected branch combination, it implements the real-time update of the branches of the pre-trained neural network through the SBRAN component.

2. The multi-exit neural network design method for dynamic resources in edge scenarios as described in claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Pre-select several convolutional layer structures and fully connected layer structures that can be combined to form effective branches, and form a branch library based on the differences between the convolutional layer structures and fully connected layer structures and the different exit points; Step 1.2: Train the branches in the branch library on the given pre-trained neural network to obtain the branch configuration information, including weights, number of parameters, inference accuracy and execution completion time; Step 1.3: Filter the branch library. Filtering criteria: delete branches whose execution accuracy does not meet the requirements, and delete branches whose time is greater than the inference time of the pre-trained neural network but whose accuracy is lower.

3. The multi-exit neural network design method for dynamic resources in edge scenarios as described in claim 1, characterized in that, Step 2.2.1 settings , .

4. The multi-exit neural network design method for dynamic resources in edge scenarios as described in claim 1, characterized in that, In step 4, the branches are combined using an SBRAN component. Update to memory to enable memory elastic inference.

5. A multi-exit neural network design device for dynamic resources in edge scenarios, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the multi-exit neural network design method for dynamic resources in edge scenarios as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the multi-exit neural network design method for dynamic resources in edge scenarios as described in any one of claims 1-4.

Images