Deep learning load mapping optimization method and system on multi-accelerator heterogeneous SoC platform

Abstract

The invention discloses a deep learning load mapping optimization method and system on a multi-accelerator heterogeneous SoC platform, relates to the technical field of computer performance and energy consumption collaborative optimization, and aims to improve the performance of deep learning reasoning application and reduce energy consumption at the same time. According to the technical key points, the method comprises the following steps: firstly, extracting a sub-network with performance or energy consumption optimization potential; then measuring performance and energy consumption data of the fusion node and the sub-network; establishing an analytic performance and energy consumption prediction model to predict the running time and energy consumption of the inference engine under different mapping configurations; searching mapping configuration with better performance-energy consumption by using a mapping optimization algorithm based on a variable-length sliding window; and finally, an inference engine is generated, and the optimal CUDA stream number is traversed and searched. According to the method, user intervention is not needed, mapping configuration with better performance-energy consumption can be fully automatically found, and the inference engine is generated. Compared with existing JEDI methods and the like, the method has the advantages that the performance is improved more, more energy is saved, and the time required by the optimization process is shorter.

Description

Technical Field

[0001] The present invention relates to the technical field of computer performance-energy consumption coordinated optimization, and in particular to a deep learning load mapping optimization method and system on a multi-accelerator heterogeneous SoC platform. Background Art

[0002] Deep learning applications are now common on heterogeneous SoC (System on a Chip) platforms. Currently, many heterogeneous SoC platforms are equipped with various accelerators such as GPU (Graph Processing Unit) and NPU (Neural Processing Unit, NPU) to accelerate deep learning applications. For example, the widely used NVIDIA Jetson series platform includes CPU, GPU, DLA (deep learning accelerator). The Jetson series heterogeneous SoC platform is a computing platform released by NVIDIA specifically for robots and embedded edge AI applications. DLA is a hardware accelerator developed by NVIDIA for deep learning operations. DLA is designed to fully hardware accelerate convolutional neural networks. DLA supports neural network layers such as convolution, deconvolution, full connection, activation, pooling, batch normalization, etc. NVIDIA provides TensorRT development tools to help users deploy deep learning reasoning applications on the Jetson platform, which can generate reasoning engines that can run on the Jetson platform from deep learning models. TensorRT mainly focuses on optimizing the performance of deep learning applications on a single accelerator (GPU or DLA), and cannot automatically map different layers of the neural network to the most suitable accelerator. TensorRT only supports two mapping strategies by default: ① Use only the GPU, that is, map all deep learning network layers to the GPU; ② Use DLA as much as possible, that is, map as many neural network layers as possible to DLA. TensorRT provides Python and C++ API interfaces, which support users to map deep learning network layers to GPU and DLA according to customized strategies to generate a GPU-DLA hybrid engine.

[0003] Both GPU and DLA are accelerators that are more suitable for processing deep learning loads. Generally speaking, GPU has stronger absolute performance and better versatility, and can run almost all types of neural network layers. Generally speaking, for highly compute-intensive neural network layers, GPU may have better energy efficiency (for example, lower performance delay product EDP = Energy·Time), and GPU should also be used for scenarios that pursue low latency. DLA is designed for running specific types of neural network layers. It has better energy efficiency on some neural network layers, but its absolute latency is not necessarily lower than GPU. In addition, it cannot run neural network layers outside the design range. It is also necessary to consider that deep learning loads have additional overhead when switching between GPU and DLA, which cannot be ignored. Therefore, compared with the mapping strategy of using only GPU or using DLA as much as possible, if deep learning loads are reasonably mapped between GPU and DLA, there is an opportunity to improve performance (i.e., increase the number of inferences per unit time or reduce latency) while reducing energy consumption.

[0004] In recent years, many studies have optimized the mapping of deep learning workloads to multiple accelerators to improve performance and reduce energy consumption. Early methods manually traversed the mapping search space and could not automatically optimize. The optimization process was cumbersome and time-consuming. Some semi-automatic methods manually established empirical models to guide the mapping optimization process. These methods require users to have a lot of expertise in heterogeneous computing and deep learning, and the establishment of empirical models also requires a lot of time and data. Moreover, these empirical models are only applicable to specific applications or specific platforms and have poor versatility.

[0005] The most advanced methods currently use evolutionary algorithms or heuristic algorithms to find the optimal layer mapping configuration. Due to the characteristics of evolutionary algorithms and heuristic algorithms, and the lack of performance and energy consumption models to guide the optimization process, these automated methods must repeatedly generate deep learning inference engines under different layer mapping configurations and measure performance and energy consumption. Therefore, these automated methods usually take more than ten hours or even several days to find the optimal configuration. For example, the JEDI method with better optimization effect [1] It is based on genetic algorithms and heuristic algorithms.

[0006] In summary, existing deep learning load mapping optimization methods are not efficient. In addition, existing methods do not fully consider the topological structure of deep neural networks and may miss many optimization opportunities. Summary of the invention

[0007] Therefore, the present invention proposes a deep learning load mapping optimization method and system on a multi-accelerator heterogeneous SoC platform, which is used to implement an automatic optimization system for deep learning load mapping to improve the performance of deep learning reasoning applications while reducing energy consumption.

[0008] According to one aspect of the present invention, a method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform is provided. The method comprises: for a heterogeneous SoC platform equipped with a GPU and a DLA accelerator:

[0009] Extract candidate subnetworks with performance or energy consumption optimization potential in deep learning model networks;

[0010] The deep learning model network is split into multiple fusion nodes, and the running time and energy consumption data of each fusion node on the GPU and DLA accelerator are measured separately. Then, a single-input single-output sub-network is split from the deep learning model network, and the running time and energy consumption data of each single-input single-output sub-network are measured separately; the single-input single-output sub-network is a connected sub-network with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node;

[0011] Based on the measured data, a performance and energy consumption prediction model is established to deduce the running time and energy consumption of the inference engine with different mapping configurations from the fused node and sub-network data; and a mapping optimization algorithm based on a variable-length sliding window is used to find the mapping configuration with the best performance and energy consumption;

[0012] Generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, search for the optimal number of CUDA streams, and complete deep learning load mapping optimization.

[0013] Furthermore, the extraction of candidate subnetworks with performance or energy consumption optimization potential in the deep learning model network includes:

[0014] Generate a pure GPU inference engine from the deep learning model and record the layer fusion information; use the layer fusion information of the pure GPU engine to build a layer fusion node, and each group of fused layers constitutes a fusion node Node i ;

[0015] Initialize the data structure of the entire deep learning model network Net = {Node i}, topologically sort all fusion nodes so that the fusion node sequence number id i Satisfy the topological order; initialize the candidate subnetwork set setSeqNet, setSglIONet and the processed node set OldNodes; all entry nodes Node of the deep learning model i The set of intermediate nodes that make up the breadth-first search BFSNodes;

[0016] If BFSNodes is not empty, process each unprocessed fusion node in BFSNodes separately; if the current node has been processed, skip it; use the breadth-first search algorithm and intersection operation to match the only entry as Nodei Candidate subnetwork subNet of SglIONet type l , and save the result to setSglIONet; if the SglIONet type subnetwork cannot be matched, use the depth-first search algorithm and intersection operation to match the only entry as Node i SeqNet type subnetwork subNet l , and save the result in setSeqNet; if the subNet is successfully matched l , then subNet l The exit node l,out Storing in BFSNodes new , and subNet l The other nodes are stored in OldNodes, otherwise Node i All child nodes are stored in BFSNodes new Node i Store in OldNodes; then use BFSNodes new Update BFSNodes and proceed to the next cycle; when BFSNodes is empty, return the result to obtain the deep learning model network Net and two sets of candidate sub-networks; among them, SglIONet is a single-input and single-output sub-network, and SeqNet is a sequential sub-network. The sequential sub-network is a connected sub-network with a unique entry node and a unique exit node, and there is only one path from the entry node to the exit node.

[0017] Further, the deep learning model network is split into multiple fusion nodes, and the running time and energy consumption data of each fusion node on the GPU and DLA accelerator are measured separately, and then a single-input single-output sub-network is split from the deep learning model network, and the running time and energy consumption data of each single-input single-output sub-network are measured separately, including:

[0018] Measuring the no-load static power pwr of the Jetson platform s ; For each Node in Net i and each SglIONet in setSglIONet i , measure and collect performance and energy consumption data on the GPU and DLA, including:

[0019] From Node i or SglIONet i Generates an inference engine on the GPU or DLA and outputs the engine runtime data to a JSON file containing the runtime of each fusion layer; if the DLA engine fails to be generated, isDLA iSet to false and skip the current loop; otherwise, isDLA i Set to true; then extract the performance data, first extract the average running time from the JSON file Run the current engine several times in succession to measure the average power pwr of the heterogeneous computing platform t ; Then, calculate the average dynamic power and dynamic energy consumption. If the accelerator type is DLA, also obtain the average time and average dynamic energy consumption of the input and output stages. Use the average power of the DLA engine To approximate the dynamic energy consumption of the input and output stages and Finally, we obtain the deep learning model fusion node set Net containing performance and energy consumption data and two sets of candidate sub-networks setSeqNet and setSglIONet.

[0020] Furthermore, the performance and energy consumption prediction model is established as follows:

[0021] min F=f obj (Eng map ,Time map )

[0022]

[0023] In the formula, min means minimization; f obj (Eng map ,Time map )=Eng map Time map , f obj represents the performance-energy optimization objective function, Eng map represents the average energy consumption of a single run of the GPU-DLA hybrid engine under the deep learning load mapping configuration; Time map Time represents the average time of one run of the GPU-DLA hybrid engine under the deep learning load mapping configuration; gpu It represents the average running time of a pure GPU inference engine. Map represents the deep learning load mapping configuration.

[0024] Furthermore, the average running time of the GPU-DLA hybrid engine under the mapping configuration m for the deep learning load in the performance and energy consumption prediction model is It is calculated by the average running time and the average pre- and post-processing time of multiple sub-networks under the mapping configuration m. Including GPU corresponding Corresponding to DLA Respectively expressed as:

[0025]

[0026] but:

[0027] In the formula, represents the average running time of the k-th sub-network on the GPU, represents the average running time of the lth sub-network on DLA, and Respectively represent the pre-processing and post-processing time of the l-th sub-network on the GPU, represents the average running time of the load mapped to the GPU under the mapping configuration m; It represents the average running time of the load mapped to DLA under the mapping configuration m.

[0028] Furthermore, the average energy consumption of a single operation of the GPU-DLA hybrid engine under the mapping configuration m in the performance and energy consumption prediction model is The average energy consumption of the operation and the average energy consumption of the pre- and post-processing corresponding to multiple sub-networks under the mapping configuration m is calculated and expressed as:

[0029]

[0030] In the formula, represents the average dynamic energy consumption of the k-th subnetwork on the GPU, pwr s represents the average static power of a heterogeneous SoC platform, represents the average dynamic energy consumption of the lth sub-network on DLA, and They represent the average dynamic energy consumption of pre-processing and post-processing of the lth sub-network on the GPU respectively.

[0031] Furthermore, the method of finding the mapping configuration with the best performance and energy consumption by using a mapping optimization algorithm based on a variable-length sliding window includes:

[0032] First, find candidate subnets with performance or energy consumption optimization potential m , and then use a variable-length sliding window to determine the mapping configuration with the best performance-energy consumption; specifically, the two sets of candidate sub-networks setSeqNet and setSglIONet are merged into a set setSubNet; for each candidate sub-network subNet m Determine whether it has the potential for performance or energy consumption optimization, and add the potential subnetwork to the candidate set: If a candidate subnetwork subNet m It is of SgllONet type, and the calculation only converts subNet m Mapping configuration property tuple when mapping to DLA MapInfom If the average running time of a reasoning is not greater than the reasoning time of using only GPU and the objective function value of a reasoning is also less than the objective function value of using only GPU, then the subnetwork subNet m There is potential for performance or energy optimization, and the subnet m Add to the candidate set candSubNet; if a candidate subnet subNet m It is of SeqNet type. First, use the objective function value when only GPU is used to initialize the local objective function value F. lcl , calculate each subnetwork subNet m Mapping configuration attribute tuple MapInfo [j:k] k , and then the subnet with better performance and energy consumption m [j:k] Save as subNet lcl And update the local objective function value F lcl ; If subNet lcl If valid, subNet lcl Add the candidate set candSubNet;

[0033] In the fusion node set Net, select candidate subnets subNet according to their performance or energy consumption optimization potential. m The minimum fusion node number id contained in i Sort candSubNet in ascending order; use a variable-length sliding window to create a mapping configuration Map of adjacent candidate subnetworks j ; Then calculate the corresponding mapping configuration attribute tuple MapInfo j , and find out the j The number of connected subgraphs composed of fused nodes contained in NumGraphs. If the conditions are met, the current mapping configuration Map j Performance-energy consumption is better, Mqp j Save as Map and update F lcl ; Finally, the optimal mapping configuration Map is obtained.

[0034] Furthermore, in the mapping optimization algorithm based on a variable-length sliding window to find the mapping configuration with the best performance and energy consumption, the inference time and energy consumption using only the GPU are calculated by the following formula:

[0035]

[0036] Furthermore, in the mapping optimization algorithm based on the variable-length sliding window to find the mapping configuration with the best performance and energy consumption, if the following three conditions are met, the current mapping configuration Map jBetter performance and energy consumption: ① In the map j ① The time for one inference does not exceed the time for one inference when only GPU is used; ② The current optimization objective function value is less than the historical optimal value; ③ The number of connected subnetworks mapped to DLA does not exceed the maximum number of subnetworks that a DLA can support on average.

[0037] Furthermore, generating the optimal inference engine for deep learning applications according to the optimal mapping configuration and searching for the optimal number of CUDA streams includes: using the TensorRT API to generate the TensorRT inference engine for the deep learning model according to the mapping configuration Map with optimal performance and energy consumption, and then gradually increasing the number of CUDA streams, and running the inference engine on these CUDA streams, and measuring the number of inference queries per second (qps) curr , until qps curr Reduction means that the optimal number of CUDA streams is found; finally, the optimal TensorRT inference engine and the matching optimal number of CUDA streams are obtained.

[0038] According to another aspect of the present invention, a deep learning load mapping optimization system on a multi-accelerator heterogeneous SoC platform is proposed. For a heterogeneous SoC platform equipped with a GPU and a DLA accelerator, the system includes:

[0039] A subnetwork extraction module configured to extract candidate subnetworks with performance or energy consumption optimization potential in a deep learning model network;

[0040] A performance-energy consumption measurement module, which is configured to split the deep learning model network into multiple fusion nodes, measure the running time and energy consumption data of each fusion node on the GPU and DLA accelerator separately, and then split a single-input single-output sub-network from the deep learning model network, and measure the running time and energy consumption data of each single-input single-output sub-network separately; the single-input single-output sub-network is a connected sub-network with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node;

[0041] A mapping optimization module is configured to establish an analytical performance and energy consumption prediction model based on the measurement data, derive the running time and energy consumption of the inference engine of different mapping configurations from the fusion node and sub-network data; and use a mapping optimization algorithm based on a variable-length sliding window to find the mapping configuration with the best performance-energy consumption;

[0042] The engine generation module is configured to generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, and search for the optimal number of CUDA streams to complete the deep learning load mapping optimization.

[0043] The beneficial technical effects of the present invention are:

[0044] The present invention proposes a method and system for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform. First, subnetworks with performance or energy consumption optimization potential are extracted; then the performance and energy consumption data of fusion nodes and subnetworks are measured; then, an analytical performance and energy consumption prediction model is established to predict the running time and energy consumption of the inference engine under different mapping configurations; then, a mapping optimization algorithm based on a variable-length sliding window is used to search for a mapping configuration with better performance-energy consumption; finally, an inference engine is generated and traversed to search for the optimal number of CUDA streams. The present invention does not require user intervention and can automatically find a mapping configuration with better performance-energy consumption and generate an inference engine. Compared with existing methods such as JEDI, the present invention has greater performance improvement and energy savings, and the optimization process also takes less time.

[0045] The present invention can automatically optimize the mapping of different layers in a deep learning network to multiple heterogeneous accelerators (GPU and DLA), automatically find the mapping configuration that has both performance improvement and energy saving and generate an inference engine, and automatically determine the optimal number of CUDA streams. Compared with existing methods, the present invention has greater performance improvement, greater energy saving, and a shorter optimization process time. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification, and are used to further illustrate the preferred embodiments of the present invention and explain the principles and advantages of the present invention.

[0047] Figure 1 This is a framework diagram of the deep learning load mapping optimization system proposed in the present invention.

[0048] Figure 2 It is a schematic diagram of the sequential subnetwork in the present invention.

[0049] Figure 3 This is a schematic diagram of a single-input single-output sub-network in the present invention.

[0050] Figure 4 This is an example diagram of possible execution timing and energy consumption of the reasoning task in the present invention.

[0051] Figure 5 This is a diagram of the performance and energy consumption optimization results in the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the scheme of the present invention, exemplary implementations or embodiments of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described implementations or embodiments are only implementations or embodiments of a part of the present invention, not all of them. Based on the implementations or embodiments of the present invention, all other implementations or embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0053] An embodiment of the present invention provides a method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform. The method includes: for a heterogeneous SoC platform equipped with a GPU and a DLA accelerator:

[0054] S1. Extract candidate sub-networks with performance or energy consumption optimization potential in deep learning model networks;

[0055] S2. Split the deep learning model network into multiple fusion nodes, measure the running time and energy consumption data of each fusion node on the GPU and DLA accelerator separately, and then split a single-input single-output sub-network from the deep learning model network, and measure the running time and energy consumption data of each single-input single-output sub-network separately; the single-input single-output sub-network is a connected sub-network with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node;

[0056] S3. Based on the measured data, establish a performance and energy consumption prediction model for analysis, deduce the running time and energy consumption of the inference engine with different mapping configurations from the fusion node and sub-network data; and use the mapping optimization algorithm based on the variable-length sliding window to find the mapping configuration with the best performance-energy consumption;

[0057] S4. Generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, search for the optimal number of CUDA streams, and complete the deep learning load mapping optimization.

[0058] The present invention is described in detail below.

[0059] 1. Deep learning load mapping optimization system framework

[0060] The deep learning load mapping optimization system framework proposed in this invention is as follows Figure 1As shown. First, extract subnetworks with performance or energy optimization potential (①): traverse and analyze the deep learning model network, extract sequential subnetworks and single-input single-output subnetworks, which are candidate subnetworks with performance or energy optimization potential; then, measure and collect the performance and energy consumption data of each fusion node and subnetwork on the GPU and DLA (②): split the deep learning model network into multiple fusion nodes, measure the running time and energy consumption data of each fusion node on different accelerators (GPU and DLA) separately, and then split the single-input single-output subnetwork from the deep learning model network, and measure the running time and energy consumption data of each single-input single-output subnetwork separately; then, based on the measured data, establish the performance and Energy consumption prediction model, evaluates the performance and energy consumption of different mapping configurations, and uses a mapping optimization algorithm based on a variable-length sliding window to find the mapping configuration with the best performance-energy consumption (③): For sequential subnetworks, a variable-length sliding window is used to determine the subsequence with the best performance-energy consumption. These optimal subsequences and all single-input and single-output subnetworks form candidate subnetworks, and then a variable-length sliding window is used to determine the candidate subnetwork subsequence with the best performance-energy consumption, and the optimized mapping configuration can be obtained; finally, the optimal inference engine is generated according to the optimal mapping configuration, and the optimal number of CUDA streams is searched (④): According to the optimized mapping configuration, a deep learning application inference engine is generated, and the number of CUDA streams is set incrementally from small to large, and the number of CUDA streams with the best performance is searched traversally. Among them, CUDA is a parallel computing platform and programming model provided by NVIDIA for its GPU, DLA and other devices. CUDA stream is used to manage the characteristics of tasks on GPU and DLA. Tasks in the same CUDA stream are executed in sequence, and tasks between different CUDA streams can be executed overlappingly.

[0061] 2 Formalization of Deep Learning Load Mapping Optimization Problem

[0062] This section formalizes the problem of mapping neural network layers to appropriate accelerators (GPU and DLA) to achieve optimal performance and energy consumption. Table 1 lists the symbols and meanings related to the formalization of the problem. The measured performance and energy consumption data of the entire neural network Net consists of many fusion nodes Node i A fusion node i Contains one or more layers. This is because TensorRT performs layer fusion optimization when generating the inference engine. Naturally, Node i It is the basic granularity for collecting performance and energy consumption data and mapping deep learning workloads. In addition, some subnets with performance and energy consumption optimization potential can be selected. iAs the granularity for measuring performance and energy consumption data and mapping deep learning loads. When TensorRT processes a neural network containing multiple fused nodes, it may perform additional optimizations between fused nodes. Therefore, measuring the entire sub-network as a whole can obtain more direct and accurate performance-energy consumption data than the indirect results derived from the performance-energy consumption data of multiple fused nodes.

[0063] Formula (1) defines a fusion node Node i , setPnt i and setChd i Save topology information. All types of neural network layers can run on the GPU, but due to the limitations of DLA and TensorRT, some types of layers cannot run on DLA. Use isDLA i Marking Node i Can it be run on DLA. The remaining 10 variables ( arrive ) is the performance-energy consumption data that needs to be measured for a fusion node. The dynamic power refers to the power consumption minus the static power pwr s Due to the characteristics of the Jetson platform and TensorRT, there are additional GPU cores before and after the DLA workload to copy and format the input and output data. These additional costs are express.

[0064]

[0065] Formula (2) defines a subnet subNet i . SubNet i Contains multiple fusion nodes Node j , these fusion nodes form a set subNet i (Formula (3)). Subnet i The symbols included are also shown in Table 1.

[0066]

[0067] setNode i ={Node in , ..., Node j , ..., Node out} (3)

[0068] Table 1 Formal notation for deep learning load mapping optimization problem

[0069]

[0070]

[0071] Based on the above definition, a formal representation of the deep learning load mapping performance and energy consumption optimization problem is given:

[0072]

[0073] The goal of the mapping optimization problem is to find the optimal mapping configuration (Map) to optimize the performance-energy consumption objective function (f obj ) is minimal, and the inference engine performance is no less than that when using only the GPU ($Time map ≤Time gpu ). The present invention selects the energy delay product EDP as the performance-energy optimization objective function, as shown in formula (5). Using the energy delay product EDP as the only optimization objective can simplify the optimization process and can simultaneously consider the optimization of performance and energy consumption. Users can also specify other optimization objective functions as needed, such as total energy consumption, energy delay square product EDP 2 P, etc. Fusion Node in Map i are mapped to DLA, while other fusion nodes of deep learning models are mapped to GPU by default.

[0074] f obj (Eng map ,Time map )=Eng map Time map (5)

[0075] 3. Performance and energy consumption data measurement

[0076] This section describes each fusion node i and subnets with performance or energy optimization potential i Measure performance and energy consumption data. i The included layers can be determined by parsing the output log file of the trtexec tool. However, it is not easy to extract subnetworks with performance or energy consumption optimization potential. Therefore, this section first proposes a candidate subnetwork extraction algorithm, and then designs an algorithm to measure the performance of each fusion node. i And candidate subnet subNet i .

[0077] 3.1 Candidate Subnetwork Extraction Algorithm

[0078] Since there are additional GPU cores before and after the DLA workload to copy and format input and output data, which is very expensive, a subNet with as few ingress and egress nodes as possible and enough internal nodes is needed. i It is more likely to have performance or energy optimization potential. We focus on two common sub-network types that meet this principle.

[0079] Definition 1 SeqNet is a connected subnet. i , and there is a unique entry node and a unique exit node, and there is only one path from the entry node to the exit node, such as Figure 2 shown.

[0080] Definition 2: A single-input single-output subnetwork SglIONet is a connected subnetwork subNet i , and there is a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node, such as Figure 3 shown.

[0081] The sequential subnetwork SeqNet and the single-input single-output subnetwork SglIONet are considered as candidate subnetworks with performance or energy consumption optimization potential. Both SeqNet and SglIONet can be expressed as formula (2). The only entry node and exit node are denoted as Node in and Node out .

[0082] The candidate subnetwork extraction algorithm (Algorithm 1) is designed below to extract candidate subnetworks with performance or energy optimization potential. First, use the trtexec tool to generate a pure GPU inference engine from the deep learning model and record the layer fusion information (Line 1). At this time, trtexec performs layer fusion optimization for the GPU; therefore, the layer fusion configuration is more favorable to the GPU. If the layer fusion configuration is favorable to the GPU, it can still be done by adding the fusion node Node i Mapping to DLA to find performance or energy consumption optimization potential indicates that the optimized GPU-DLA hybrid engine will outperform the pure GPU engine. Therefore, for conservative considerations, the layer fusion configuration of the pure GPU engine is used to construct the layer fusion node, and each group of fused layers constitutes a fusion node Node i After that, initialize the data structure of the entire deep learning model network Net = {Node i}, at this time, all fusion nodes will be topologically sorted so that the fusion node sequence number id iSatisfy the topological order (line 1). Next, initialize the candidate subnetwork set (setSeqNet and setSglIONet) and the processed node set OldNodes (line 2). All entry nodes of the deep learning model constitute the intermediate node set of breadth-first search (line 3).

[0083] If BFSNodes is not empty, process each unprocessed fusion node in BFSNodes separately (lines 4-27). If the current node has been processed, skip it (lines 7-9). Use the breadth-first search algorithm and intersection operation to match the only entry as Node i Candidate subnetwork subNet of SgllONet type l , and save the result to setSglIONet (lines 10-12). If the SglIONet type subnetwork cannot be matched, the depth-first search algorithm and intersection operation are used to match the only entry as Node i SeqNet type subnetwork subNet l , and save the result in setSeqNet (lines 13-18). If the subNet is successfully matched l , then subNet l The exit node l,out Storing in BFSNodes new , and subNet l The other nodes are stored in OldNodes (lines 19-21). Otherwise, Node i All child nodes are stored in BFSNodes new Node i Store in OldNodes (lines 22-25). Then use BFSNodes new Update BFSNodes and proceed to the next while loop (line 27). When BFSNodes is empty, return the result (line 29).

[0084]

[0085]

[0086]

[0087] 3.2 Performance and energy consumption data measurement algorithm

[0088] This section designs an algorithm to accurately measure performance and energy consumption data and prepare basic data for subsequent performance and energy consumption modeling. iIt is the basic granularity for performance and energy efficiency modeling. Each needs to measure performance and energy consumption data. According to experimental data and observation analysis, for the sequential subnetwork SeqNet, from the fusion node Node i The data can be used to deduce the performance and energy consumption of the sequential subnetwork SeqNet. For the single-input and single-output subnetwork SglIONet, there are multiple paths between the inlet and outlet. TensorRT will perform more optimizations when generating the engine, which results in i The performance and energy consumption results of the single-input single-output subnetwork SglIONet derived from the data are inaccurate, and often overestimate the energy consumption and execution time. Therefore, it is necessary to measure the single-input single-output subnetwork SglIONet as a whole to obtain sufficiently accurate performance and energy consumption data.

[0089] The following is a performance and energy consumption data measurement algorithm (Algorithm 2) to collect the performance and energy consumption data of each fusion node. i and the single-input single-output subnetwork SglIONet i First, use Algorithm 1 to obtain the deep learning model network Net and two sets of candidate sub-networks (row 1). Then measure the no-load static power pwr of the Jetson platform. s In order to calculate the dynamic power and energy consumption later (line 2). Then, for each Node in the Net i and each SgllONet in setSglIONet i Measure and collect performance and energy consumption data on the GPU and DLA respectively (lines 3-19). First, use the trtexec tool to i or SglIONet i Generate an inference engine on the corresponding device (GPU or DLA) and output the engine runtime data to a JSON file (line 5). The JSON file contains the runtime of each fusion layer, as shown in lines 3-5 of Table 2. Before and after running the load on DLA, the input and output data may need to be copied and formatted on the GPU. The input and output times are shown in lines 2 and 6 of Table 2, respectively. Note that when multiple Node i When running continuously on DLA, these Node i There is no data copying and formatting between them. If the DLA engine generation fails, isDLA i Set to false and skip the current loop (lines 6-9); otherwise, isDLA i Set to true (line 10).

[0090]

[0091] Next, extract the performance data. First, extract the average running time from the JSON file. That is, the average time sum of rows 3-5 in Table 2 (row 11). Then run the current engine several times in succession to measure the average power pwr of the heterogeneous computing platform. t (Line 12). Then, the average dynamic power and dynamic energy consumption are calculated (Line 13). If the accelerator type is DLA, the average time and average dynamic energy consumption of the input and output stages are also obtained (Lines 14-17). Due to the limitations of the Jetson heterogeneous computing platform, it is not possible to measure the input or output power separately. The average power of the DLA engine can be used To approximate the dynamic energy consumption of the input and output stages and (Lines 15ω16). Finally, we can obtain the deep learning model fusion node set Net containing performance and energy consumption data and two sets of candidate sub-networks setSeqNet and setSglIONet (Line 20).

[0092] Table 2 JSON file example

[0093]

[0094] 4. Performance and energy consumption modeling and deep learning load mapping

[0095] Based on the performance and energy consumption data of the fusion nodes and candidate sub-networks measured in the previous section, this section first designs a performance and energy consumption model to evaluate the performance and energy consumption of the TensorRT engine generated under a specific mapping configuration. Then, under the guidance of the performance and energy consumption model, an algorithm is designed to search for the mapping configuration with the best performance and energy consumption and the matching optimal number of CUDA streams.

[0096] 4.1 Performance and energy consumption model design

[0097] First, we design a performance model, i.e., a runtime model, to evaluate the performance of the TensorRT engine generated under a specific mapping configuration. Without loss of generality, assume that there are m sub-networks mapped to DLA, and these sub-networks form a mapping configuration denoted as Map m , as shown in formula (6). The remaining fusion nodes in the deep learning model are mapped to the GPU by default.

[0098] Map m ={subNet l} (6)

[0099] All candidate subnetworks subNet l All fused nodes in Node k Composition of the MapNodes collection m , as shown in formula (7).

[0100]

[0101] Mapping Configuration m All the entry fusion nodes and exit fusion nodes of the neutron network form two sets EnNodes respectively m and ExNodes m , as shown in formula (8).

[0102]

[0103] If a subnet l is a single-input single-output subnetwork SglIONet that has been measured, and its running time data has been recorded. For subnetworks that have not been measured as a whole, the running time data can be calculated using formula (9). l DLA runtime All the Nodes it contains k DLA runtime The sum of subNet l Input phase time It is the fusion node of all its entry points enNode i Input phase time The sum of subNet l Output phase time All its export fusion nodes exNode i Output phase time The sum of .

[0104]

[0105] Based on each fusion node i And each subnet l The runtime data can be derived from the mapping configuration Map m Next, the running time of GPU and DLA in one inference is shown in formulas (10) and (11). The GPU running time is equal to the sum of the running time of the fusion node mapped to the GPU and the input and output stage time of the fusion node mapped to the DLA; the DLA running time is equal to the sum of the DLA running time of the fusion node mapped to the DLA.

[0106]

[0107] In order to derive the average running time and energy consumption of the reasoning task, we first analyze the possible execution timings of multiple consecutive executions of the reasoning task, such as Figure 4As shown. Without loss of generality, it can be assumed that a TensorRT inference engine consists of four stages: GPU load, DLA input stage, DLA load, and DLA output stage. The Jetson Orin NX 16GB platform has two DLAs (DLA0 and DLA1), which supports two DLA workloads running simultaneously, such as D1 and D2. When multiple CUDA streams are used, the three GPU activity stages (GPU load, DLA input stage, DLA output stage) and DLA load can also run overlappingly, such as D1, D2, G3, and I3. When running the TensorRT inference engine in multiple CUDA streams, the TensorRT runtime can automatically schedule and run these four stages overlappingly.

[0108] Based on the above analysis, the running time of 2n inference tasks using the same mapping configuration inference engine can be derived, as shown in formula (12).

[0109]

[0110] in It represents the total overlap time of GPU and DLA in one inference, which is the minimum value between 2 times GPU running time and DLA running time, as shown in formula (13).

[0111]

[0112] Substituting formula (13) into formula (12) can obtain the average running time $Time^{map_m}$ of one inference, as shown in formula (14).

[0113]

[0114] Design an energy consumption model based on the above runtime model. l is a single-input single-output subnetwork SglIONet that has been measured, and its energy consumption data has been recorded. For subnetworks that have not been measured as a whole, formula (15) can be used to calculate their energy consumption data. l DLA dynamic energy consumption All the Nodes it contains k DLA dynamic energy consumption The sum of subNet l Dynamic energy consumption in the input stage It is the fusion node of all its entry points enNode i Dynamic energy consumption in the input stage The sum of subNet l Dynamic energy consumption in the output stage All its export fusion nodes exNodei Dynamic energy consumption in the output stage The sum of .

[0115]

[0116] Based on each fusion node i And each subnet l The energy consumption and running time data can be derived in the mapping configuration Map m Next, the average energy consumption of one inference and the performance-energy consumption optimization objective function are shown in formulas (16) and (17).

[0117]

[0118] So far, it has been deduced that the deep learning model is in the mapping configuration Map m The performance and energy consumption model of the inference engine generated below is formula (14), (16), (17). For the sake of brevity, define the mapping configuration Map m Attribute tuple MapInfo m , which contains the mapping configuration itself and the next inference time of the mapping configuration Performance-energy consumption objective function value As shown in formula (18).

[0119]

[0120] 4.2 Subnetwork Mapping Algorithm for Performance and Energy Optimization

[0121] Under the guidance of the performance and energy consumption model designed in the previous section, this section designs a subnet mapping algorithm for performance and energy consumption optimization (Algorithm 3) to determine the mapping configuration with the best performance-energy consumption. First, find the candidate subnets with performance or energy consumption optimization potential. m (Lines 1-21), and then a variable-length sliding window is used to determine the mapping configuration with the best performance-energy consumption (Lines 22-34).

[0122] Merge the two sets of candidate sub-networks setSeqNet and setSglIONet into a set setSubNet (line 2). For each candidate sub-network subNet m Determine whether it has the potential for performance or energy consumption optimization, and add the potential subnetwork to the candidate set (lines 3-25). m It is of SglIONet type, and only subNet m Mapping configuration property tuple when mapping to DLA MapInfo m(Lines 4-6); If the average running time of a reasoning is not greater than the reasoning time using only the GPU and the objective function value of a reasoning is also less than the objective function value using only the GPU, then the subnetwork subNet m There is potential for performance or energy optimization, and the subnet m Add the candidate set candSubNet. The inference time and energy consumption using only GPU are calculated by formula (19).

[0123]

[0124] If a candidate subnetwork subNet m It is of SeqNet type. First, use the objective function value when only GPU is used to initialize the local objective function value F. lcl (Line 11). Calculate each subnet m Mapping configuration attribute tuple MapInfo [j:k] k (Lines 12-15), and then the subnet with better performance and energy consumption is subNet m [j:k] Save as subNet lcl And update the local objective function value F lcl (Lines 16-18). If subNet lcl If valid, subNet lcl Join the candidate set candSubNet (lines 21-23).

[0125]

[0126]

[0127]

[0128] In the fusion node set Net, all fusion nodes Node i All follow the topological sorting. Therefore, according to each candidate subnet with performance or energy consumption optimization potential, m The minimum fusion node number id contained in i Arrange candSubNet in ascending order (line 26) to make all subNet m All follow the topological sorting. At this time, sort the adjacent candidate subnetworks subNet m More likely to be adjacent in topology. m Mapping to DLA as a whole is more likely to eliminate these candidate subnetworks mTherefore, a variable-length sliding window can be used to create a mapping configuration Map of adjacent candidate subnetworks. j (Lines 28-30). Then calculate the corresponding mapping configuration attribute tuple MapInfo j , and find out the j The number of connected subgraphs composed of fused nodes contained in NumGraphs (lines 31-32). If the following three conditions are met, the current mapping configuration Map j Performance-energy consumption is better, Map j Save as Map and update F lcl (Lines 33-35). ① In the Map j ② The time for one inference is no longer than the time for one inference when only the GPU is used. ② The current optimization objective function value is less than the historical optimal value. ③ The number of connected subnetworks mapped to DLA does not exceed the maximum number of subnetworks that a DLA can support on average (10 subnetworks on the Jetson Orin NX 16GB platform). This ensures that the inference task can use all DLAs at the same time (2 DLAs on the Jetson OrinNX 16GB platform). Finally, the optimal mapping configuration Map is obtained (line 38).

[0129] 4.3 Optimal CUDA Stream Number Search Algorithm

[0130] Integrate the above algorithms and design the JDIMO algorithm framework (Algorithm 4) to generate the best performance-energy consumption TensorRT inference engine and search for the optimal number of CUDA streams. First, use Algorithms 2 and 3 to determine the best performance-energy consumption mapping configuration (lines 1-2). Then use TensorRTAPI to generate the TensorRT inference engine of the deep learning model according to the best performance-energy consumption mapping configuration Map (line 3).

[0131] Then gradually increase the number of CUDA streams and run the inference engine on these CUDA streams to measure the number of inference queries per second (qps). curr , until qps curr The optimal number of CUDA streams is found (lines 4-12). These CUDA streams will use all DLAs in a loop (DLA0 and DLA1 are used alternately on the Jetson Orin NX 16GB platform) to fully utilize the computing power of the heterogeneous computing platform. Finally, the optimal TensorRT inference engine and the matching optimal number of CUDA streams are obtained (line 13).

[0132]

[0133] The technical effect of the present invention is further verified through experiments.

[0134] Optimization goal: Minimize the energy consumption delay product energy consumption, that is, f obj (Eng map ,Time map )=Eng map Time map (Formula (5));

[0135] Experimental hardware platform: Jetson Orin NX 16GB multi-accelerator heterogeneous SoC computing platform; tested deep learning models: VGG-16, VGG-19, MobileNet-v2, RetinaNet, YOLOv4.

[0136] Comparison method:

[0137] ①GPU-only, TensorRT default strategy 1, only using GPU;

[0138] ②DLA-possible, TensorRT default strategy 2, try to use DLA;

[0139] ③JEDI [1] , an existing mapping method with good optimization effect, using GPU and DLA at the same time, this method divides the deep learning model into multiple stages, generates multiple engines correspondingly, and runs them in a pipeline;

[0140] ④ The method of the present invention - JDIMO.

[0141] It should be noted that all deep learning models (ONNX files) are downloaded from the ONNX Model Zoo code repository (https: / / github.com / onnx / models\#onnx-model-zoo).

[0142] Performance and energy consumption optimization results are as follows Figure 5 As shown in the figure, the normalized energy-delay product EDP of an inference (the lower the better), the energy consumption of an inference (the lower the better), the number of inference queries per second (QPS, indicating performance, the higher the better), and the average power (the lower the better) are shown based on GPU-only data.

[0143] It can be seen that the method JDIMO of the present invention exhibits the lowest EDP, the lowest energy consumption, and the highest QPS, while the average power is basically the same as that of the GPU-only method.

[0144] The present invention (JDIMO): average normalized EDP = 0.5, average normalized energy consumption = 0.72, average normalized QPS = 1.42, average normalized power = 1.0. JEDI: average normalized EDP = 0.68, average normalized energy consumption = 0.82, average normalized QPS = 1.26, average normalized power = 1.02. DLA-possible: average normalized EDP = 2.12, average normalized energy consumption = 0.96, average normalized QPS = 0.86, average normalized power = 0.72. GPU-only: average normalized EDP = 1.0, average normalized energy consumption = 1.0, average normalized QPS = 1.0, average normalized power = 1.0.

[0145] Compared with the GPU-only method, the JDIMO method of the present invention reduces the EDP by 50%, saves 28% of energy consumption, and increases the QPS by 42%, with basically the same power. Compared with the DLA-possible method, the JDIMO method of the present invention reduces the EDP by 76.4%, saves 25% of energy consumption, increases the QPS by 65.1%, and increases the power by 38.9%. Although the DLA-possible has lower power, its normalized QPS is only 0.2, and the performance loss is serious. Its normalized energy consumption is 1.8, and the energy consumption is significantly increased rather than saved. Compared with the JEDI method, the JDIMO method of the present invention reduces the EDP by 26.5%, saves 12.1% of energy consumption, increases the QPS by 12.7%, and reduces the power by 2.0%.

[0146] The optimized mapping configurations and optimization process times of the four methods on five deep learning models are shown in Table 3. It can be seen that overall, JDIMO maps more neural network layers to DLA than the JEDI method, and compared with the DLA-possible method alone, combined with the above performance and energy consumption optimization result data, it shows that JDIMO has found a mapping configuration with a better balance between performance and energy consumption.

[0147] Although the optimization process time of the JDIMO method is longer than that of the GPU-only and DLA-possible methods, it is shorter than that of the JEDI method. Compared with the JEDI method, the JDIMO method of the present invention shortens the optimization process time by 93.6% on average.

[0148] The above data and analysis prove that the method of the present invention can automatically find a more optimal mapping configuration of deep learning loads to multiple heterogeneous accelerators and generate an inference engine, can find the optimal number of CUDA streams, can reduce energy consumption while improving performance, and can also significantly shorten the optimization process time.

[0149] Table 3 Optimized mapping configuration and optimization process time

[0150]

[0151] 1. "GPU:", "DLA0:", and "DLA1:" indicate mapping to GPU, DLA0, and DLA1, respectively. "DLA:" indicates mapping to any one or both DLAs. "-" indicates omitting consecutive serial numbers. The numbers in the circle correspond to the engines generated by JEDI.

[0152] 2. For simplicity, only nodes mapped to DLA are shown. The output log of trtexec does not show the intermediate nodes; "..." indicates the omitted intermediate nodes.

[0153] Another embodiment of the present invention provides a deep learning load mapping optimization system on a multi-accelerator heterogeneous SoC platform. For a heterogeneous SoC platform equipped with a GPU and a DLA accelerator, the system includes:

[0154] A subnetwork extraction module configured to extract candidate subnetworks with performance or energy consumption optimization potential in a deep learning model network;

[0155] A performance-energy consumption measurement module, which is configured to split the deep learning model network into multiple fusion nodes, measure the running time and energy consumption data of each fusion node on the GPU and DLA accelerator separately, and then split a single-input single-output sub-network from the deep learning model network, and measure the running time and energy consumption data of each single-input single-output sub-network separately; the single-input single-output sub-network is a connected sub-network with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node;

[0156] A mapping optimization module is configured to establish an analytical performance and energy consumption prediction model based on the measurement data, derive the running time and energy consumption of the inference engine of different mapping configurations from the fusion node and sub-network data; and use a mapping optimization algorithm based on a variable-length sliding window to find the mapping configuration with the best performance-energy consumption;

[0157] The engine generation module is configured to generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, and search for the optimal number of CUDA streams to complete the deep learning load mapping optimization.

[0158] The functions of a deep learning load mapping optimization system on a multi-accelerator heterogeneous SoC platform according to an embodiment of the present invention can be described by the aforementioned deep learning load mapping optimization method on a multi-accelerator heterogeneous SoC platform. Therefore, for the parts not described in detail in the system embodiment, please refer to the above method embodiment and will not be repeated here.

[0159] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments are contemplated within the scope of the invention thus described. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

[0160] The documents cited in the present invention are as follows:

[0161] [1]Jeong EJ, Kim J, Ha S. Tensorrt-based framework and optimization methodology for deep learning inference on jetson boards [J]. ACM Transactionson Embedded Computing Systems (TECS), 2022, 21(5):1-26.

Claims

1. A method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform, characterized in that: include: For heterogeneous SoC platforms equipped with GPU and DLA accelerator: Extract candidate subnetworks with performance or energy consumption optimization potential in deep learning model networks; The deep learning model network is split into multiple fusion nodes, and the running time and energy consumption data of each fusion node on the GPU and DLA accelerator are measured separately. Then, a single-input single-output sub-network is split from the deep learning model network, and the running time and energy consumption data of each single-input single-output sub-network are measured separately; the single-input single-output sub-network is a connected sub-network with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node; Based on the measured data, a parsed performance and energy consumption prediction model is established to derive the runtime and energy consumption of the inference engine with different mapping configurations from the fused node and sub-network data; And use the mapping optimization algorithm based on variable-length sliding window to find the mapping configuration with optimal performance and energy consumption; Generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, search for the optimal number of CUDA streams, and complete deep learning load mapping optimization.

2. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 1, characterized in that: The candidate subnetworks extracted from the deep learning model network with performance or energy consumption optimization potential include: Generate a pure GPU inference engine from the deep learning model and record the layer fusion information; use the layer fusion information of the pure GPU engine to build a layer fusion node, and each group of fused layers constitutes a fusion node Node i ; Initialize the data structure of the entire deep learning model network Net = {Node i }, topologically sort all fusion nodes so that the fusion node sequence number id i Satisfy the topological order; initialize the candidate subnetwork set setSeqNet, setSglIONet and the processed node set OldNodes; all entry nodes Node of the deep learning model i The set of intermediate nodes that make up the breadth-first search BFSNodes; If BFSNodes is not empty, process each unprocessed fusion node in BFSNodes separately; if the current node has been processed, skip it; use the breadth-first search algorithm and intersection operation to match the only entry as Node i Candidate subnetwork subNet of SglIONet type l , and save the result to setSglIONet; if the SglIONet type subnetwork cannot be matched, use the depth-first search algorithm and intersection operation to match the only entry as Node i SeqNet type subnetwork subNet l , and save the result in setSeqNet; if the subNet is successfully matched l , then subNet l The exit node l，out Storing in BFSNodes new , and subNet l The other nodes are stored in OldNodes, otherwise Node i All child nodes are stored in BFSNodes new Node i Store in OldNodes; then use BFSNodes new Update BFSNodes and proceed to the next cycle; when BFSNodes is empty, return the result to obtain the deep learning model network Net and two sets of candidate sub-networks; among them, SglIONet is a single-input and single-output sub-network, and SeqNet is a sequential sub-network. The sequential sub-network is a connected sub-network with a unique entry node and a unique exit node, and there is only one path from the entry node to the exit node.

3. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 2, characterized in that: The method of splitting the deep learning model network into multiple fusion nodes, measuring the running time and energy consumption data of each fusion node on the GPU and DLA accelerator separately, and then splitting the single-input single-output sub-network from the deep learning model network, and measuring the running time and energy consumption data of each single-input single-output sub-network separately includes: Measuring the no-load static power pwr of the Jetson platform s ; For each Node in Net i and each SglIONet in setSglIONet i , measure and collect performance and energy consumption data on the GPU and DLA, including: From Node i or SglIONet i Generates an inference engine on the GPU or DLA and outputs the engine runtime data to a JSON file containing the runtime of each fusion layer; if the DLA engine fails to be generated, isDLA i Set to false and skip the current loop; otherwise, isDLA i Set to true; then extract the performance data, first extract the average running time from the JSON file Run the current engine several times in succession to measure the average power pwr of the heterogeneous computing platform t ; Then, calculate the average dynamic power and dynamic energy consumption. If the accelerator type is DLA, also obtain the average time and average dynamic energy consumption of the input and output stages. Use the average power of the DLA engine To approximate the dynamic energy consumption of the input and output stages and Finally, we obtain the deep learning model fusion node set Net containing performance and energy consumption data and two sets of candidate sub-networks setSeqNet and setSglIONet.

4. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 3, characterized in that: The performance and energy consumption prediction model is established as follows: min F=f obj (Eng map ,Time map ) In the formula, min means minimization; f obj (Eng map ,Time map )=Eng map Time map , f obj represents the performance-energy optimization objective function, Eng map represents the average energy consumption of a single run of the GPU-DLA hybrid engine under the deep learning load mapping configuration; Time map Time represents the average time of one run of the GPU-DLA hybrid engine under the deep learning load mapping configuration; gpu It represents the average running time of a pure GPU inference engine. Map represents the deep learning load mapping configuration.

5. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 4, characterized in that: The average running time of the GPU-DLA hybrid engine under mapping configuration m for the deep learning workload in the performance and energy consumption prediction model It is calculated by the average running time and the average pre- and post-processing time of multiple sub-networks under the mapping configuration m. Including GPU corresponding Corresponding to DLA Respectively expressed as: but: In the formula, represents the average running time of the k-th sub-network on the GPU, represents the average running time of the lth sub-network on DLA, and Respectively represent the pre-processing and post-processing time of the l-th sub-network on the GPU, represents the average running time of the load mapped to the GPU under the mapping configuration m; It represents the average running time of the load mapped to DLA under the mapping configuration m.

6. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 5, characterized in that: The average energy consumption of a single run of the GPU-DLA hybrid engine under mapping configuration m for the deep learning workload in the performance and energy consumption prediction model The average energy consumption of the operation and the average energy consumption of the pre- and post-processing corresponding to multiple sub-networks under the mapping configuration m is calculated and expressed as: In the formula, represents the average dynamic energy consumption of the k-th subnetwork on the GPU, pwr s represents the average static power of a heterogeneous SoC platform, represents the average dynamic energy consumption of the lth sub-network on DLA, and They represent the average dynamic energy consumption of pre-processing and post-processing of the lth sub-network on the GPU respectively.

7. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 6, characterized in that: The method of using a mapping optimization algorithm based on a variable-length sliding window to find a mapping configuration with optimal performance and energy consumption includes: First, find candidate subnets with performance or energy consumption optimization potential m , and then use a variable-length sliding window to determine the mapping configuration with the best performance and energy consumption; specifically, the two sets of candidate sub-networks setSeqNet and setSglIONet are merged into a set setSubNet; for each candidate sub-network subNet m Determine whether it has the potential for performance or energy consumption optimization, and add the potential subnetwork to the candidate set: If a candidate subnetwork subNet m Is the SglIONet type, the calculation only subNet m Mapping configuration property tuple when mapping to DLA MapInfo m If the average running time of a reasoning is not greater than the reasoning time of using only GPU and the objective function value of a reasoning is also less than the objective function value of using only GPU, then the subnetwork subNet m There is potential for performance or energy optimization, and the subnet m Add to the candidate set candSubNet; if a candidate subnet subNet m It is of SeqNet type. First, use the objective function value when only GPU is used to initialize the local objective function value F. lcl , calculate each subnetwork subNet m Mapping configuration attribute tuple MapInfo [j:k] k , and then the subnet with better performance and energy consumption m [j:k] Save as subNet lcl And update the local objective function value F lcl ; If subNet lcl If valid, subNet lcl Add the candidate set candSubNet; the inference time and energy consumption using only GPU are calculated by the following formula: In the fusion node set Net, select candidate subnets subNet according to their performance or energy consumption optimization potential. m The minimum fusion node number id contained in i Sort candSubNet in ascending order; use a variable-length sliding window to create a mapping configuration Map of adjacent candidate subnetworks j ; Then calculate the corresponding mapping configuration attribute tuple MapInfo j , and find out the j The number of connected subgraphs composed of fused nodes contained in NumGraphs. If the conditions are met, the current mapping configuration Map j Performance-energy consumption is better, Map j Save as Map and update F lcl ; Finally, the optimal mapping configuration Map is obtained.

8. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 7, characterized in that: In the mapping optimization algorithm based on the variable-length sliding window to find the mapping configuration with the best performance-energy consumption, if the following three conditions are met, the current mapping configuration Map j Better performance and energy consumption: ① In the map j ① The time for one inference does not exceed the time for one inference when only GPU is used; ② The current optimization objective function value is less than the historical optimal value; ③ The number of connected subnetworks mapped to DLA does not exceed the maximum number of subnetworks that a DLA can support on average.

9. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 8, characterized in that: Generating the optimal inference engine for deep learning applications according to the optimal mapping configuration and searching for the optimal number of CUDA streams includes: using the TensorRT API to generate the TensorRT inference engine for the deep learning model according to the mapping configuration Map with the best performance-energy consumption, then gradually increasing the number of CUDA streams, running the inference engine on these CUDA streams, and measuring the number of inference queries per second (qps) curr , until qps curr Reduction means that the optimal number of CUDA streams is found; finally, the optimal TensorRT inference engine and the matching optimal number of CUDA streams are obtained.

10. A deep learning load mapping optimization system on a multi-accelerator heterogeneous SoC platform, characterized in that: For a heterogeneous SoC platform equipped with a GPU and a DLA accelerator, the system includes: A subnetwork extraction module configured to extract candidate subnetworks with performance or energy consumption optimization potential in a deep learning model network; A performance-energy consumption measurement module, which is configured to split the deep learning model network into multiple fusion nodes, measure the running time and energy consumption data of each fusion node on the GPU and DLA accelerator separately, and then split a single-input single-output sub-network from the deep learning model network, and measure the running time and energy consumption data of each single-input single-output sub-network separately; the single-input single-output sub-network is a connected sub-network with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node; A mapping optimization module is configured to establish an analytical performance and energy consumption prediction model based on the measurement data, derive the running time and energy consumption of the inference engine of different mapping configurations from the fusion node and sub-network data; and use a mapping optimization algorithm based on a variable-length sliding window to find the mapping configuration with the best performance-energy consumption; The engine generation module is configured to generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, and search for the optimal number of CUDA streams to complete the deep learning load mapping optimization.

Images